The Agent That Watched a Log File for 30 Hours
In short
- One AI subagent waited 30 hours on a background job and spent 46M weighted tokens doing it, 5% of a week’s use.
- It polled a log file in a shell loop. Each check outlasted the subagent’s five-minute prompt cache, so each of its 60 wakes rewrote at least 171k tokens of context.
- It wasn’t rare: in the month before, subagents waited past five minutes 112 times, and 108 of those wakes found the cache gone.
- A pre-tool hook now refuses those waits. In its first six weeks, no wait it can see ran past five minutes.
One of my coding agents spent 30 hours watching a log file. It was a subagent, spawned to run a self-play experiment for a game engine and report the result. It started the experiment in the background, chose on its own to wait by polling the log, and nothing stopped it. Over 330 model calls it spent 46M weighted tokens, 5% of a week’s use.
I found it by breaking that week’s spend down by the session that spawned each subagent. My agents run on a flat-rate plan with a weekly allowance, and a flat rate hides waste. Nothing arrives as a charge, so the cost shows up as work that didn’t fit in the week. On a metered bill, the same loop costs money.
Weighted tokens. Each kind of token is priced against new input at list price. A cache read counts 0.1, a cache write 1.25 and output 5. A raw count adds them at face value, so an agent re-reading a warm cache all day looks huge and costs little. The plan’s own accounting may weigh them differently.
Each wake rebuilt the cache
The experiment ran in the background, and the agent waited for it:
until grep -q 'done ' logs/selfplay.log; do sleep 30; done
A shell call can block for at most ten minutes. When it times out, the model takes another turn, sees the job still running, and starts the loop again. This agent did that for 29.7 hours.
Claude Code keeps a subagent’s prompt cache for five minutes and a main session’s for an hour. My logs match. Nearly every main-session cache write used the hour, and every subagent write used five minutes.
That gap is what made the loop expensive. The loop itself costs nothing. The turn after each timeout does, because every model call re-reads the agent’s whole context, and the cache makes that cheap only while it’s warm. So at every wake the cache had expired, and the call paid to write the full context again, 60 times. The context grew while it waited, from 171k tokens at the first wake to 584k.
The agent couldn’t have known. Nothing in its context shows what a turn cost or that its cache expired, so each wake looked like the one before.
Subagents did it most days
I scanned a month of transcripts, September 1 to October 1, for shell calls with a while, until or for loop around a sleep. Subagents held a foreground wait past five minutes 112 times, more than three a day, and 108 of those wakes found the cache gone. Main sessions did it 62 times and never paid, because their cache lasts an hour.
The wakes after those long waits cost 47M weighted tokens over the month, 30.5M of it from the one agent. With a warm cache at every wake, the month would have cost 40M less.
The agent stops instead of waiting
The agent starts the long job in the background, writes down where the result will appear, and stops. When the job is done, a new agent with a small context reads the result. A main session can hand the wait to Claude Code instead, which runs it in the background and wakes the session when the job exits. Either way the model runs once when the job finishes, not once every ten minutes.
Two alternatives cost more:
- Check more often. A warm read costs a tenth of a write, but 30 hours at four-minute intervals is still 450 wakes. At an average context of 300k tokens, that’s 13.5M weighted to learn “not done yet”.
- Keep subagent caches for an hour. Claude Code allows it (
subagentPromptCacheTtl), but a one-hour write costs 2× new input against 1.25×, and every subagent write pays it. My subagents have written 550M tokens to cache since September. The longer lifetime would have added about 410M weighted, against 40M of cold wakes in the month before the fix.
A hook refuses the wait
Since the agent can’t see the cost, the rule has to come from outside it. A line in the agent instructions would ask for this. A pre-tool hook enforces it. Before each shell command runs, the hook refuses a loop that sleeps five seconds or more, unless the loop runs in the background, is detached (nohup, setsid, a trailing &), or is capped under four minutes. The detection is one regex:
LOOP = re.compile(
r"\b(?:until|while|for)\b.*?\bdo\b.*?\bsleep\s+(\d+(?:\.\d+)?)", re.S
)
An & counts only after the loop’s done. node harness.js & until grep -q done run.log; do sleep 30; done detaches the job and then babysits it in the foreground, which is the incident itself.
The refusal says what to do instead, so the agent’s retry is already the right shape. Replayed on the month before it went live, the hook blocks 97% of the waits that ran past five minutes, and a quarter of the shorter ones, each of which costs one retry. In its first six weeks it refused 60 loops, and no wait it can see ran past five minutes.
What the hook misses
It reads the command text for a shell loop around sleep. A loop inside a script file passes, and so do a Python time.sleep loop, gh run watch, wait $pid, tail -f piped to grep, and one long sleep 300 per turn. Each can babysit a job as surely as the incident did. The hook’s tests pin each one as a known miss, and the zero above is counted with the same regex, so it can’t see them either.
A subagent’s idle time is cheap only while its cache is warm, and that lasts five minutes. Any longer wait should stop the agent and leave the result for a fresh one, and the same arithmetic applies to an agent polling CI, a deploy or a queue. To check your own agents, search their transcripts for this pattern. Claude Code keeps them as JSONL files under ~/.claude/projects/, with each subagent’s in a subagents/ folder, and the regex above finds the loops. Any that sat past five minutes paid the full context again on waking.