Nate Swenson

A remote that reads the TV back before it says done

Apple TV · No. 137 · 13 min read ·v0.1.0

Shipped

The first release of appletv: a Claude Code skill that finds the Apple TVs on the network, pairs with one, and drives it from chat. Power, playback, apps, the on-screen keyboard, volume, “what’s playing”, and a real screenshot when nothing else will do. The one rule it was built around is that a command is never reported as done until the TV’s state has been read back and agrees with it. That rule turned out to be the whole product, because the first thing the TV taught me is how often a command “succeeds” and nothing happens. This guide is about building that read-back loop yourself.

Why a remote needs a loop

Every remote-control library I looked at treats a command as fire-and-forget. You call pause(), the call returns, and the library considers its job done. On a TV over Wi-Fi that is the wrong contract. The keypress can land on a sleeping box, the wrong app can have focus, the app can ignore the command outright. Home Assistant’s Apple TV integration says it plainly in its own docs: turning the entity off “only toggles the power state in Home Assistant”, and asked whether it can tell if the device is on without interacting with it, the answer is no.

Control engineers solved this a century ago. Åström and Murray describe the basic pattern as a controller that senses the operation of a system, compares it against the desired behavior, computes corrective actions and actuates the system, and they call that loop of sensing, computation and actuation the central concept in control. A remote that only actuates is an open loop. What we want is the closed one: read the state, send, read the state again, and only then decide what to say.

For an LLM agent driving the TV this matters twice over. The model will happily narrate “paused” over a return value. The read-back is what keeps it honest.

Prerequisites

You need pyatv, which speaks the Apple TV’s protocols from a Mac or Linux box, and Python 3.9 or newer. Put it in a virtual environment; do not install it globally.

python3 -m venv .venv
.venv/bin/pip install pyatv

One thing I hit immediately: pyatv’s bundled atvremote command crashes on Python 3.12 and newer with “There is no current event loop”, because it calls asyncio.get_event_loop() at startup. The library API is fine. Everything below uses the API directly and runs its own loop.

You also need to pair once. On tvOS 15 and later, pair AirPlay (metadata and playback ride inside it) and Companion (apps, keyboard, power). pyatv stores the credentials in ~/.pyatv.conf so you never pass them again:

.venv/bin/python -c "
import asyncio, pyatv
from pyatv.const import Protocol
from pyatv.storage.file_storage import FileStorage

async def pair(proto):
    loop = asyncio.get_running_loop()
    storage = FileStorage.default_storage(loop)
    await storage.load()
    conf = (await pyatv.scan(loop, storage=storage))[0]
    pairing = await pyatv.pair(conf, proto, loop, storage=storage)
    await pairing.begin()
    pairing.pin(input('PIN on the TV: '))
    await pairing.finish()
    await storage.save()
    await pairing.close()

asyncio.run(pair(Protocol.AirPlay))
asyncio.run(pair(Protocol.Companion))
"

Stand in front of the TV when you run that. Each protocol shows its own four-digit code the moment pairing begins, and the code changes on every attempt.

Step 1: read everything the device will tell you

The read-back is only as good as the state you can observe, so the first function collects every observable field into one dictionary. Anything the device cannot report gets recorded as unsupported rather than silently missing; that distinction matters later.

# atv.py
import asyncio
from dataclasses import dataclass, field


@dataclass
class State:
    power: str | None = None          # "on" | "off" | "unknown"
    app: str | None = None            # bundle id of the now-playing owner
    playback: str | None = None       # "playing" | "paused" | "idle" | ...
    title: str | None = None
    position: int | None = None
    unsupported: dict = field(default_factory=dict)


async def read_state(atv) -> State:
    """Read every observable field once. A field the device cannot report
    is named in `unsupported`, never left as a silent None."""
    st = State()
    try:
        st.power = atv.power.power_state.name.lower()
    except Exception as e:
        st.unsupported["power"] = type(e).__name__
    app = atv.metadata.app
    st.app = app.identifier if app else None
    try:
        p = await asyncio.wait_for(atv.metadata.playing(), timeout=3)
        st.playback = p.device_state.name.lower()
        st.title = p.title
        st.position = p.position
    except Exception as e:
        st.unsupported["playback"] = type(e).__name__
    return st

Two things are worth knowing about that app field before you trust it. It is not the app on screen. pyatv’s maintainer described it this way in the issue asking for a current-app API: from the traffic logs it is only possible to see which app is currently playing something, so you cannot see an app that is just open on screen. That single fact shapes half the verdict logic below.

Step 2: send, then poll the read-back until it moves

The naive version reads once after a fixed sleep. That either wastes seconds on commands that landed instantly or misses ones that take a while (power takes a few seconds to reflect). The better version knows which field each command is supposed to change, polls that field every half second, and stops as soon as it moves or a per-command ceiling passes.

# atv.py (continued)
import time

OBSERVES = {
    "play": "playback", "pause": "playback",
    "turn_on": "power", "turn_off": "power",
    "launch_app": "app",
    "skip_forward": "position",
}
CEILING = {"turn_on": 6.0, "turn_off": 6.0, "launch_app": 3.0}
KEYPRESSES = {"up", "down", "left", "right", "select", "menu"}


async def dispatch(atv, command: str, arg=None):
    if command in ("turn_on", "turn_off"):
        return await getattr(atv.power, command)(await_new_state=False)
    if command == "launch_app":
        return await atv.apps.launch_app(arg)
    return await getattr(atv.remote_control, command)()


async def press(atv, command: str, arg=None) -> dict:
    """Read before, send, read after until the observed field moves.
    Returns a capture the verdict function can judge offline."""
    before = await read_state(atv)
    try:
        await asyncio.wait_for(dispatch(atv, command, arg), timeout=6)
        sent = {"ok": True}
    except Exception as e:
        sent = {"ok": False, "error": type(e).__name__}

    what = OBSERVES.get(command)
    ceiling = 0.0 if command in KEYPRESSES else CEILING.get(command, 4.0)
    reads = []
    started = time.monotonic()
    while True:
        await asyncio.sleep(0.5)
        now = await read_state(atv)
        reads.append(now)
        moved = what and getattr(now, what) != getattr(before, what)
        if moved or time.monotonic() - started >= ceiling:
            break
    return {"command": command, "arg": arg, "sent": sent,
            "before": before, "reads": reads, "after": reads[-1]}

Keypresses get a ceiling of zero on purpose. up has no readable effect, so waiting for one is pure sleep. The capture is a plain dictionary so the next step can run without a TV at all.

Step 3: exactly three verdicts, and what each one means

This is the part the whole thing hangs on. A capture gets one of three verdicts, and only the first one is ever reported as done:

  • verified: the read-back shows the effect the command names.
  • mismatch: the read-back shows something else, or the device refused.
  • unverifiable: the command has no readable effect, or the device cannot report the field.

The verdict function is pure. It looks at a capture and nothing else, which means the same code that runs live also runs in CI over frozen captures, and a rule change shows up as a red test rather than a quiet drift.

# verify.py
from atv import State, KEYPRESSES

TV_APP = "com.apple.TVWatchList"


def frozen(cap) -> bool:
    """Did the read-back move at all? A number that never changed is not
    evidence of anything."""
    sig = lambda s: (s.power, s.app, s.playback, s.position)
    first = sig(cap["before"])
    return all(sig(r) == first for r in cap["reads"])


def verdict(cap) -> tuple[str, str]:
    cmd, before, after = cap["command"], cap["before"], cap["after"]
    if not cap["sent"]["ok"]:
        return "mismatch", f"device refused: {cap['sent']['error']}"

    if cmd in ("turn_on", "turn_off"):
        want = "on" if cmd == "turn_on" else "off"
        if after.power in (None, "unknown"):
            return "unverifiable", "device does not report power"
        if after.power == want and before.power == want:
            return "unverifiable", f"already {want} before the send, nothing to prove"
        if after.power == want:
            return "verified", f"read-back is {want}"
        return "mismatch", f"expected {want}, read-back is {after.power}"

    if cmd in ("play", "pause"):
        want = "playing" if cmd == "play" else "paused"
        if after.playback is None:
            return "unverifiable", "device does not report playback"
        if after.playback == want and before.playback == want:
            return "unverifiable", f"already {want} before the send, nothing to prove"
        if after.playback == want:
            return "verified", f"read-back is {want}"
        if frozen(cap) and after.app == TV_APP:
            return "unverifiable", "read-back never changed; the TV app stops reporting at skip points"
        return "mismatch", f"expected {want}, read-back is {after.playback}"

    if cmd == "launch_app":
        if after.app is None:
            return "unverifiable", "device does not report the now-playing owner"
        if after.app == cap["arg"] and before.app == cap["arg"]:
            return "unverifiable", "already the now-playing owner; foreground unknown"
        if after.app == cap["arg"]:
            return "verified", f"now-playing owner became {after.app}"
        if after.app == before.app:
            return "unverifiable", "foreground unknown until it plays something; look at the screen"
        return "mismatch", f"expected {cap['arg']}, owner became {after.app}"

    if cmd in KEYPRESSES:
        return "unverifiable", "a keypress has no readable state of its own"
    return "unverifiable", f"no rule for {cmd}"

Three rules in there cost me a wrong answer each before they existed. A state that already matched before the send is not verified: turning on a TV that reads on proves nothing, and my first version cheerfully reported it as a success. A read-back that never moved is not a mismatch when the app is known to freeze its report; more on that below. And a launch is verified only when the now-playing owner changed to the target, because an unchanged owner is unknowable, not a failure.

Step 4: run it, then run the verdicts without a TV

Live use is a short coroutine that connects, presses, and prints the verdict:

# run.py
import asyncio, sys
import pyatv
from pyatv.storage.file_storage import FileStorage
from atv import press
from verify import verdict


async def main(command, arg=None):
    loop = asyncio.get_running_loop()
    storage = FileStorage.default_storage(loop)
    await storage.load()
    conf = (await pyatv.scan(loop, storage=storage, timeout=2))[0]
    atv = await pyatv.connect(conf, loop, storage=storage)
    try:
        cap = await press(atv, command, arg)
        v, why = verdict(cap)
        print(f"{command}: {v} ({why})")
    finally:
        atv.close()


asyncio.run(main(sys.argv[1], sys.argv[2] if len(sys.argv) > 2 else None))

.venv/bin/python run.py turn_off

You should see something like:

turn_off: verified (read-back is off)

The verdict function needs no device, so exercise it directly with captures shaped like the ones press produces. This is the check that belongs in your test suite:

# test_verify.py
from atv import State
from verify import verdict

def cap(command, before, after, arg=None, ok=True):
    return {"command": command, "arg": arg, "sent": {"ok": ok},
            "before": before, "reads": [after], "after": after}

on, off = State(power="on"), State(power="off")
assert verdict(cap("turn_off", on, off))[0] == "verified"
assert verdict(cap("turn_off", on, on))[0] == "mismatch"
assert verdict(cap("turn_on", on, on))[0] == "unverifiable"       # already on
assert verdict(cap("up", on, on))[0] == "unverifiable"            # a keypress
nf = State(app="com.netflix.Netflix")
assert verdict(cap("launch_app", State(app="x"), nf, arg=nf.app))[0] == "verified"
assert verdict(cap("launch_app", nf, nf, arg=nf.app))[0] == "unverifiable"
tv = State(app="com.apple.TVWatchList", playback="paused", position=101)
assert verdict(cap("play", tv, tv))[0] == "unverifiable"          # frozen TV app
print("verify.py: 7 verdicts as expected")

python3 test_verify.py

verify.py: 7 verdicts as expected

That last file imports nothing from pyatv, so it runs anywhere.

When there is no state to read: get eyes

Some tasks have no readable end state at all. “Open Netflix and play episode three” is five keypresses through menus the network cannot see. My first attempt at that navigated blind, and a menu press from Netflix’s home screen exited the app; the next four presses opened a different app in front of the household. That is the failure the verdict rule prevents for single commands, and it needs its own answer for navigation.

The answer on tvOS is a real screenshot. Starting with iOS 17, Apple moved developer services to CoreDevice and RemoteXPC, so developer commands need an RSD tunnel to the device, as the pymobiledevice3 docs explain; tvOS follows the same rules. Pair the box once with the TV sitting on Settings › Remotes and Devices › Remote App and Devices, keep a tunnel up, and a capture takes a couple of seconds:

.venv/bin/pip install pymobiledevice3
.venv/bin/pymobiledevice3 remote pair           # 6-digit code on the TV
sudo .venv/bin/pymobiledevice3 remote tunneld --no-usb --no-usbmux --no-mobdev2 --wifi
.venv/bin/pymobiledevice3 developer dvt screenshot screen.png --tunnel ''

With that, navigation becomes look, press, look: one screenshot, one press whose result you can predict from the picture, another screenshot. The end state is still read back over the network (Netflix is the now-playing owner and it is playing), and the picture covers the part the network cannot. Playing DRM video renders black in a capture, which is itself a reliable tell that protected video is on screen.

One design rule I would keep even if I rebuilt this: the agent asks before anything with a cost. The Alexa design guidance splits confirmations by consequence, explicit approval when the consequences for an error are high, contextual confirmation for low-consequence steps. For a TV, pausing and skipping need no question; turning it off or switching apps while something is playing does.

Gotchas

The app field is the now-playing owner, not the foreground app. Launching Netflix to its home screen leaves metadata.app on whatever last played, so a launch with an unchanged read-back looked like a failure and I first classified it as mismatch. The escape is in the verdict function: unchanged owner means unverifiable, and only a change to a different app than the target is evidence against the launch. This is the documented limit of the protocol, not a bug in the library.

The Apple TV app freezes its report at skip points. Playing an Apple TV+ episode, the position sat at 101 seconds and the state read paused for over a minute while the episode played, right after a “Skip Recap” press. Two screenshots eight seconds apart were clean black with no scrubber (a paused player keeps its scrubber on screen). The escape is the frozen check above: when every read is identical and the owner is the TV app, the verdict is unverifiable with that reason, and the screen decides.

Netflix does not honour deep links on tvOS any more. Eight link forms were ignored during the first run; one produced an “Open in Netflix?” dialog whose default button is Cancel. The Home Assistant community’s maintained list marks both the title and watch forms as not working as of the September 2025 Netflix update, while Disney+ and YouTube links still work. The escape is to refuse Netflix URLs up front and go through the app’s own search with screenshots.

sudo in an agent’s shell has no terminal. The tunnel needs root, and an agent-driven shell cannot answer a password prompt, so the command failed before I stopped pasting it into chat. The escape is to open a real terminal window with the command already typed (on macOS, osascript telling Terminal to run it) and let the person enter the password there.

Sources

Changelog

  • appletv 0.1.0 — control Apple TVs from chat, every command verified by read-back (#234) (43bcbdf)