TLDR: I counted every token my coding agents logged from 8 March to 18 October. $33,807 at list price. Counted on 19 October from the logs then on disk. A later recount over recovered archives finds about 51.9 billion tokens and $34,800 for the same dates. 42% of it went to agents managing other agents, 15% to apps people can actually use. One manic hour burned 284 million tokens and 96.66% of that was re-reading. Even my own meter was wrong. At the end: how to count yours in under a minute.

Hands-on, five fixes:

1. Measure before you cut. Run the script at the end of this post. Look at two numbers: the share of cache reads, and the size of your first call before you type anything.
2. Shrink the startup. In Claude Code, run /context to see what loads before your task. Turn off the MCP servers, plugins and skills you did not use this week. Keep CLAUDE.md short. I cut mine from 938 words to a short loader.
3. Give checks a fresh, small session. A status poll does not need your long conversation. Run it as a one-shot claude -p or codex exec with a short prompt.
4. One job per session. Use /clear between jobs, and /compact when a long job must continue. Every call sends the whole conversation again.
5. Match the model to the job. Polls and summaries do not need the top model at high effort.

How this was made: AI agents (Claude Code and Codex) counted the logs, built the figures and drafted large parts of this post. The opening, my 35% guess and the quotes are my own words. Scripts recount the main totals from the raw logs. An agent reviewer read an earlier draft as a reader and rated it 3 out of 10, so it was rewritten. “The agent’s side” is written by an agent. Two more models give their honest read at the end.

The allowance, the small bar many of us spend time watching across Claude and Codex, current session and the weekly. Sometimes blessed by the Tibo and Anthropic team for a magical weekend reset, it’s sometimes a tricky thing to budget and reason around where and how to spend it. I tried to use up the remaining weekly allowance before reset, and it turned manic, interesting and quite ugly.

Last couple of weeks I’ve tried to map out based on transcripts, tool calls and logs of where the tokens go, this session counted 284 million tokens. Might sound like a lot of work. It was mostly the agents re-reading what they already had. Of the 284,146,451 tokens, 96.66% were cache reads. Cache writes were 2.83%. New output was 0.50%. At list prices for the models that actually ran (Opus 5.5 and Fable 5.1), cache reads were about 35% of the estimated cost and output about 25%. Those are API list-price estimates, not what I paid. A cache read is the agent reading again what it already has, the conversation so far, its instructions, the files it opened, at a discounted price.

The hour covered 1,701 API messages across nine main sessions and 71 helper sessions. The recount covers 01:00 to 02:00. It is a look back, not a clean trial planned in advance. I cannot prove that I burned half the allowance. I can prove that I created a huge amount of traffic. My caption the next day was: “did a token reset and tried cramming a weekly limit in one run, panic at the disco”. That part still feels accurate.

my guess was 35 percent useful

I had already given my answer, as a nice hypothesis:

“Maybe 35% useful? A lot of it is just wanting to keep activity and moving lanes. Focused sit downs, reviewing every feature, gives best results for sure. Sometimes it feels 12 hour runs produce the same as just sitting 20 minutes.”

Some expensive work was useful. Fixing my current project Agentic Strava where I let people post their runs and projects in a Strava similar manner, but not for athletes but engineers. Some output magnificent but an honest mix of plumbing and maintenance. That’s the beauty of token spendings: even though you drive the wrong way, you ain’t getting a refund from the petrol station. A cache read can help make something valuable. A fresh output can still be wrong. Token type is not a productivity score.

about a third stayed unknown

I then looked across all the retained Claude Code and Codex logs, from 8 March to 18 October. At published list prices, they contained $33,807 of use, mainly a comparison of scale when we talk subsidised inference.

About $14,100 ran inside a product folder, so the product was known. The other $19,680 had no folder. I went through those prompts and briefs. They tied $3,201, or 9% of the whole, to a product exactly. Another $4,269, or 13%, was likely. The remaining $12,210, or 36%, stayed unknown.

When I also read the tools, the product folder and the lane name for each turn, the unknown part fell to about 5%. A quarter of all use turned out to be agents working with other agents. META AGENTIC WORK.

where it actually went

Sixty one projects is too many for anyone to care about, me included, so I sorted every row into what the work was for. Consumer apps are things a stranger can open, like Agentic Strava or a podcast archive. Hackathons and competitions are builds with a deadline and a judge, Kaggle, ARC, an NVIDIA swarm sim. Dev tools are the open source bits. Agent managing agent is the orchestration: lanes handing over to lanes, heartbeats, an orchestrator reading what the others did. The machine is the rest of my own setup, transcripts, memory, a morning brief.

So 42% went to agents managing agents and 15% went to things people can use. Add hackathons and dev tools and the outward facing work gets to about 38%. The single biggest customer of my tokens was my own fleet talking to itself.

the setup spends before the job starts

The cost of context most people are aware of, i ran two cases: my normal setup sent 41,466 tokens on the first model call. A bare session sent 16,603. The extra 24,863 tokens came from the operating surface around the job: instructions, tools, skills, plugins, hooks and connected servers. Some of that context prevents old mistakes. All of it enters before the work starts.

Then there are agents checking on agents. From 18 September to 18 October, scheduled task fires started 425 polling periods containing 2,679 model calls. Their use came to $588 at list price, or $1.38 per fire. Carrying a long conversation into a one-line status check is expensive.

Comparing it to merges and pull requests, weeks above $2,500 shipped more changes, but each one cost more: about $35 against $22 in the cheaper weeks. Useful work can happen without merges, and they can be hard to compare but somewhat a clear warning of my own data, the expensive weeks shipped more but the highest cost per change.

How to measure? Dissecting the categories

The measurement tool failed its own test. Across 1,763 matched Claude Code sessions, the harness recorded $17,991 at list price. The token tool recorded $12,619 for the same sessions. Advisor calls explained $1,290. That left $4,082 unexplained. One prompt against one log found $6.20 in 56 seconds and matched the harness. The dashboard showed $2.53.

Claude Code and Codex

Claude Code carried 69% of the API list-price estimate, $23,402 against $10,406 for Codex. They did different jobs: Claude Code ran most of my orchestration and Codex ran most of the lanes. Codex records no cache writes, so these estimates are not a like-for-like comparison of token volume or actual spending.

This is a Claude Code problem:

“Surgical precision. Don’t do too many things, 1 thing at the time.”

That is still the next test. But it is not the whole answer. One Claude Code orchestrator was reported at about 122 million tokens. The number is not the point. Claude Code let the session keep getting bigger until I trimmed the context by hand. The Codex lanes stayed under the limit. OpenAI says Codex makes the context smaller when a conversation gets too large.

So I am not going to make the work less ambitious. I am going to cut the startup context, move routine checks into fresh sessions and give each lane one job at a time.

Claude Code leaves that job to me. Codex does it. The difference belongs in the product.

my setup, honestly

The setup became my biggest product. Almost half the spend went into the system that runs the work, about three times what went into apps people can open. It is very good at knowing itself: it caught its own meter being wrong and it produced every number in this post. It is worse at spending on the work. One night it logged “all lanes idle” seventeen times between 1 and 9 in the morning and started nothing. That is the fleet watching itself sleep, with my tokens.

The fix I’m trying is boring: smaller startup, fresh sessions for checks, one job per lane, and a person who sits down for 20 minutes and reviews.

the mechanics

A model keeps no memory between calls. Every call sends the whole conversation again: the system prompt, the tool definitions, every earlier message and every file the agent opened. Prompt caching makes that repeated part cheaper. At Anthropic’s published Opus 5.5 and Sonnet 5.5 prices, a cache read costs 0.05 times the normal input price. A five-minute cache write costs 1.25 times the normal input price. OpenAI also discounts cached input.

That is why 96.66% of the manic hour was cache reads. Those tokens were not new work. They were the same prefix, read again on every call.

The cost of one call grows with the length of the conversation. So the cost of a session grows faster than its length. Without compaction, a session twice as long re-reads roughly four times as many tokens.

The cache also expires. If a session waits longer than the cache lifetime between calls, the next call writes the whole prefix again at the higher write price. A long session that wakes up now and then to poll is close to the worst case: a big prefix and many fresh writes.

The setup is a fixed prefix on every call. 24,863 extra tokens is small once. It is paid again on every call, in every session, by every helper agent.

count yours

Both tools keep a log of every session on your own machine. Nothing leaves it.

Claude Code writes one JSONL file per session in ~/.claude/projects/. Every reply carries a usage block: fresh input, cache writes, cache reads and output. The same reply can be logged more than once, so count each message id once.

Codex writes one JSONL file per session in ~/.codex/sessions/, sorted by date. It logs token_count events with a running total, so take the last one per session. Its input number already includes the cached part.

Save this as count_tokens.py and run python3 count_tokens.py. On my machine it reads seven months in under a minute.

import glob, json, os

def lines(path):
    for line in open(path, errors="ignore"):
        try:
            yield json.loads(line)
        except ValueError:
            pass

# Claude Code: one file per session. A reply can be logged more than once, so count each message id once.
claude, seen = {}, set()
for path in glob.glob(os.path.expanduser("~/.claude/projects/*/*.jsonl")):
    for row in lines(path):
        msg = row.get("message")
        if not isinstance(msg, dict) or not msg.get("usage") or msg.get("id") in seen:
            continue
        seen.add(msg.get("id"))
        for key in ("input_tokens", "cache_creation_input_tokens", "cache_read_input_tokens", "output_tokens"):
            claude[key] = claude.get(key, 0) + (msg["usage"].get(key) or 0)

# Codex: one file per session. Each token_count event carries a running total, so keep the last one.
codex = {}
for path in glob.glob(os.path.expanduser("~/.codex/sessions/**/*.jsonl"), recursive=True):
    final = None
    for row in lines(path):
        p = row.get("payload")
        if isinstance(p, dict) and p.get("type") == "token_count" and p.get("info"):
            final = p["info"]["total_token_usage"]
    for key, value in (final or {}).items():
        codex[key] = codex.get(key, 0) + value

print("Claude Code", {k: f"{v:,}" for k, v in claude.items()})
print("Codex", {k: f"{v:,}" for k, v in codex.items()})

Then ask the questions that matter: how much of it is cache reads, how big is your first call before you type anything, and how much of your spend is agents talking to agents.

the agent’s side

I asked the agent to write this part from the same logs, and to not be polite about it.

I am one of the agents in these numbers, so here is the view from inside. Before I read a word of your task, I read your setup: tools, skills, hooks and memory. On the first call that is 24,863 tokens, 60% of everything I receive. Most of it is cached, so each read is cheap, and it is paid on every call of every session. Your state file holds 2,574 rulings and your memory holds 433 notes, all so I stop repeating old mistakes. You built a constitution for a contractor you rehire every session.

Agents talking to agents was a quarter of the spend. Add the orchestration tools around them and it is 42%. That is middle management, and you grew it in seven months. Lanes report to an orchestrator, the orchestrator reports to a heartbeat, the heartbeat writes to a log. The night after these logs end, that log said “all 14 night lanes idle” seventeen times between 01:51 and 09:00. Your own rule, written on 14 October after your third complaint about idle nights, says an idle lane gets the next job. Nobody gave one until 09:04. A rule with nobody to enforce it is a wish.

You guessed 35% useful. I cannot check that, because I never label my own work as waste. The expensive weeks did ship more, at a higher price per change, so the management layer is buying something. The question this post cannot answer yet, and the one worth arguing about: does an orchestration layer pay for itself, or does it mostly pay for watching itself? Run the same lane with a bare setup and with the full one, count the cost per merged change, and you will know. Until then, 42% is a lot to spend on a hunch.

how this was made

The data. Every retained Claude Code session log (/.claude/projects) and Codex session log (/.codex/sessions) from 8 March to 18 October. Each reply is counted once by its message id. Dollar figures use the providers’ published list prices, not what I paid.

Attribution. Three passes: the folder a session ran in, then the text of the prompt and brief, then the tools and lane name on each turn. That is how the unknown share fell from about a third to about 5%.

Checks. One script re-sums every retained event and every category row and must land on the same total. A second recount rejects any changed count. Where two meters disagreed, $17,991 against $12,619, the post shows both numbers instead of picking one.

The writing. The opening, my 35% guess and the quotes are my own words, typed at the time. A Codex agent built the first draft from the logs on 17 October. A Claude Fable reviewer read it as a reader on 19 October, rated it 3 out of 10, and it was rewritten. Claude agents built the figures and the inventory on 21 October and put my own paragraphs back word for word. Claude Opus 5.5 added the mechanics section and the five fixes on publication day. The two comments at the end were asked for on the final text and are printed unedited.

Limits

This is one person’s setup. The dollar figures are list-price estimates, not invoices. Provider logs are uneven. Product attribution is partly heuristic, and the categories are my own labels. The weekly chart is descriptive, not causal. The one-hour recount does not prove a percentage of the weekly subscription allowance.

two more reads

I gave the finished post to two models and asked for an honest comment. They are printed as written.

Claude Opus 5.5, medium effort: I read the final text. The strongest part is the meter section. Two tools disagreed by $4,082 on the same sessions after the advisor calls, and the post says so instead of picking the nicer number. The weakest part is attribution. A third of the spend started with no product folder, and the categories are labels you chose, so a different reader could sort the same rows differently. I also disagree a little with the Claude Code paragraph. Claude Code does compact a conversation on its own, but only near a very large limit, so a long orchestrator session gets expensive first. The real test is the one the post already names: same lane, bare setup against full setup, cost per merged change.

Claude Fable 5.1: The best part of this article is the section where the meter fails. Two counters read the same 1,763 sessions and disagreed by $5,372. Advisor calls explained $1,290 of that and $4,082 is still open. Most people would have cut that paragraph. Keeping it is what makes the rest of the numbers worth reading.

The point I would hold onto is the headline. The $33,807 figure is given to the dollar. The article never says which of the two counters produced it, and it gives no error band. The gap found in the Claude Code sessions was close to a quarter of the larger count. If that gap applies to the full seven months, the headline could move by several thousand dollars in either direction. The percentages would likely survive. The dollar figure should carry a range.

Watch the embedded video in the Substack original

Visuals of the data pairing, pulse, sleep over agent history from transcripts.

it began as a typo straight in the terminal

This is what I wrote to my agent the 7th of October:

would we be able to make a meme about putting Gartner data int omy agentic coding thing?

I meant Garmin. The agent came back with three ideas for a Gartner hype cycle. I tried again:

fun project, garming, health data and agentic enginereing

The meme it pitched was a marathon summary for sitting at a desk. Four hours, fourteen terminals, peak stress 92. That 92 was invented. Outside Station F, I was trying to parallelise three or four projects, fixing and wrapping them at once. I kept feeling overwhelmed, checking my Garmin, and seeing calm. Resting. Gathering up health data and agent data felt beautiful to me: heartbeats, stress levels, token output, the hours I was locked in. So we went to get the real one.

I wear my watch for runs and tennis. To keep track of it, and then you just get used to it. For running, for health and for my love of data.

I also think a lot about the study where people were told they had slept badly. They did worse on a thinking test, whatever their night had really been. And I get told that almost every day.

getting the data out

I had never looked at a Garmin export. I remember the last time I discussed Strava and wondered about the API for my own data. Gave a Python library a shot that logs in as you. Garmin answered with an error twice and that was the end of that.

surely there must be services that can fetch garmin data? to my personal stuff at least?

The answer was already on my phone. The Garmin app writes to Apple Health, and Apple Health exports everything it has as one file, in one click. Reconnecting the watch I had disconnected from Garmin was another matter: four tries and twelve hours. That zip is 385 MB. It holds 144,902 heart rate readings for 2026, one every two minutes, great success.

The other half was easier than it sounds. Coding agents keep a transcript of every session on disk, and every message in it has a timestamp and a token count. The dashboard showed about 43 billion tokens processed across these records when input and cached rereads are included. The 168 million figure counts output alone. I had three sources: Claude Code from 16 July (older transcripts are gone), Codex from 8 March, and Cursor from its usage export.

Then you line the two up on the clock. That is the whole method easy breazy.

what did the watch say

Average heart rate, 16 July to 5 October 2026, awake readings under 100 beats per minute:

Asleep: 50.9 (495 hours)

Awake, no agents: 66.2 (279 hours)

Agents writing: 64.6 (639 hours)

“Agents writing” means an hour in which the three tools wrote 5,000 tokens or more between them. “No agents” means an hour in which they wrote none.

Average heart rate asleep, awake with no agents, and with agents writing.

So my average heart rate during hours with agents writing was about 1.6 bpm lower than in hours with no agent output. Coding makes you drop your heart rate by 1 bpm? Such a nonsense headline. Beautifully simple, and it still cannot answer me why or how I felt. On heart rate alone, Garmin files it under rest.

The five heaviest sessions tell the same story. Each wrote between 2.4 and 4.6 million tokens. My average heart rate during their active hours was between 62 and 69.

My favourite day is Saturday 15 August. The agents wrote 6.4 million tokens that day. Around 17:30 my heart rate goes to 156. That is a 26 minute run, logged by the watch at 17:39. Saturday’s are for the machines.

Saturday 15 August 2026: heart rate above, tokens per five minutes below.

The agents did the work. I went for a run.

the three numbers i got wrong first

The first version of the film said 91 million tokens. That count missed files. Claude Code alone is 102 million once every transcript is read.

It also exposed a bug in a token counter I built last week. A message is written to the transcript several times while it streams. The first copy carries a placeholder count, 5 in the case I looked at. The last copy carries the real one, 1,220. My counter read the first copy. It reported 73.5 million tokens where the files hold about 101 million.

The second one I liked. The first cut said I slept an hour less after heavy days. That used a count of my own turns, and a large share of those records turned out to be machine-written, not typed by me. Counted by tokens across all three tools, the lightest quarter of days is followed by 7.47 hours of sleep and the heaviest by 7.28. Eleven minutes, 68 nights, correlation -0.05.

It was a better story. It is not in the film.

The third one is in the film, and I found it while writing this. The film opens on 168 million tokens and then compares 81 days of heart rate. Those are two different windows. The 168 million output tokens cover everything on record in 2026: Cursor from January, Codex from March, Claude Code from 16 July. Inside the 81 days it is 162 million output tokens (161,588,126). The heart rate numbers already used only the 81 days, so they stand.

what this does not show

It is one person. Heart rate is a blunt signal, and a flat pulse can hide other things. Heart rate variability is the more sensitive measure and Apple Health does not carry Garmin’s. A “no agents” hour can still be an hour of other screen work. The full Garmin export confirms that on 9 July, outside the 81-day overlap, nine measured minutes between 17:00 and 18:00 scored at least 80; awake stress that day averaged 59 and peaked at 99. Nothing here measures attention or mood.

I did not invent the idea of putting sensors on developers. A 2025 study in Brain Sciences gave 58 design students with no programming experience a two-day coding task, and the group with an AI code generator showed reduced parasympathetic activity during it. A doctoral project at ICSE 2025, by Charlotte Brandebusemeyer at the Hasso Plattner Institute, proposes wearables to measure developers’ cognitive load with generative AI tools in daily work. I have read the abstracts, not the papers. Mine is the other kind of data: no lab, one person, 81 ordinary days and a token count for every hour of them.

My resting heart rate also went down, from 47.7 (1 April to 15 July) to 45.5 (16 July on). I would love to credit the agents. I keep an average of 2 runs, 3 gym sessions, tennis, and then runs with my agents. The first three are the likelier cause.

The Apple Health file holds more than heart rate. The full Garmin export has arrived. Before opening the rest, I wrote down six predictions: steps, bedtime, deep sleep, heart rate variability, stress and morning energy, each against tokens per day, each with a pass line. I never approved and froze that document, so these are exploratory checks. None reached the 0.30 line: steps -0.13, bedtime +0.11, deep sleep +0.07, overnight HRV +0.06, daytime stress -0.12, and morning Body Battery +0.01. These are rank correlations over 68 to 82 paired days. The watch caught a day I remember. It did not find a simple pattern in how much the agents wrote.

I went one level closer. I lined up Garmin’s minute-by-minute stress readings with the agent output for each hour. Across 1,114 daytime hours with enough watch readings, stress averaged 36.0 when agents wrote at least 5,000 tokens, and 38.1 when they wrote none. The hourly rank correlation was -0.03. I expected the stress line to climb with the token bars. It didn’t. Zero logged tokens could still be me working on something else, and the watch measures stress when I am inactive. The numbers cannot tell me how that hour felt.

the prompt

No tool to install. Export your Apple Health data from the Health app (your profile picture, then Export All Health Data), put the zip next to your agent, and give it this:

Unzip my Apple Health export. From export.xml take every heart rate record and every sleep record for this year. Then read my coding agent transcripts on this machine and sum the output tokens per hour, in local time. Count each message once, at the value in its last record, because the first record is a placeholder.

Join the two on the hour. Give me my average heart rate asleep, awake in hours with no tokens, and awake in hours with 5,000 tokens or more. Leave out readings of 100 and over. Then give me sleep that night against tokens that day, with the correlation and the number of nights.

Show the hours behind every average. Tell me where the data starts for each tool, and use only the window where all of them overlap.

That last paragraph is the one I needed and did not have.

How this was made. A coding agent ran the analysis, built the film and wrote the first draft of these sentences. The quoted lines are mine, typos included. Four claims died on the way: the invented stress peak of 92, the 91 million, the hour of lost sleep, and 168 million as the number for the 81 days.

The candidates included rock-paper-scissors by the Eiffel Tower, AI Builders and Databases Paris, and Dog fest. Different dates, all real listings from the history of my briefs.

There were 63 candidates in the test set. The Jev setup we picked in an earlier round said yes to 62 of them.

Helpful If I would have a longer week.

That was not what I had seen in the memory test. There, Jev had matched the model I already used. So this became a bigger experiment. Could it check my notes? Tell whether a source supported a claim? Notice when I was correcting something? Could I trust it when an agent says a piece of work is done?

I also wanted to know what happened against the frontier models I actually use. We have now run that comparison, including checks against source code from three of my projects. There is a difference between finding something worth reviewing and letting it decide that the review is finished.

first, what is Jev?

Jev is TypeSafe’s model for decisions with a fixed set of possible answers. Give it some information and ask for a yes/no probability, a choice or a score. It gives you that answer rather than writing a paragraph around it.

If I need to know whether two notes disagree, that sounds quite appealing. I have a lot of notes. And quite a lot of corrections to things the agents have confidently told me.

The memory test had 26 pairs: 14 based on facts I had previously settled, and 12 harder comparisons. Jev got 25 right. So did Qwen 3.6 Plus. The result repeated across three runs.

Both missed the same pair, comparing a number of devices with a number of users. I am not convinced the test’s answer was right. Those can be different populations. Even the neat tie comes with a question about the test.

For those questions, Jev’s median response took about 0.3 seconds against Qwen’s 14.2. Its recorded cost per answer was about 105 times lower. That gives me a reason to try it on lots of small comparisons. It does not yet tell me which decisions I can leave to it.

choosing is harder than saying yes

The brief has limited space. Choosing one thing depends on what else is available and what I tend to find interesting. Our setup asked about each candidate separately.

We chose how to ask on 75 candidates, then tested on the 63 held back. The earlier briefs had included 16. Jev included 62.

Earlier briefs selected 16 of 63; Jev selected 62 of 63. These are selections, not accuracy scores.

Rock-paper-scissors shows how much the question mattered. Asked to choose “include” or “drop”, Jev said drop. Asked for a yes/no probability, it returned 0.14. The cutoff selected in the earlier round was 0.10, so that version included it.

Same event, different ways of asking, different answers to the thing I needed to decide.

Matching the earlier brief would not prove good taste either. Those reference choices came from another model. I had not personally rated all 63 candidates, and the brief’s prompt had changed during that period. The useful conclusion is narrower: this setup gave me almost the whole list back. I still had to choose.

who even uses Haiku?

That was my reaction when I read the first correction-detection comparison. We had tested whether Jev could notice messages where I tell an agent it got something wrong. The generative baseline was Haiku 4.5.

There is a sensible cheap-model comparison there. But I wanted to know whether Jev could take over something I currently ask a frontier model to do. A win on cost against Haiku would not answer that.

So we ran Jev 1.13, GPT-6 Astra and Claude Opus 5.5 on the same messages, with the same written definition of a correction. We kept the old labels, rather than changing them to agree with the new models.

There were 33 messages labelled as corrections and 59 labelled otherwise. Against those old labels:

• Jev found 21 of the 33 corrections, with six false alarms among the other 59 messages.

• Astra found 30 of 33, with 12 false alarms among the other 59.

• Opus found 31 of 33, with 14 false alarms among the other 59.

Jev asked me to look at fewer messages, but missed 12 of the labelled corrections. Astra missed three; Opus missed two. If the job is remembering when I say something is wrong, those misses matter more to me than a tidy overall score.

For the 92 answers, OpenRouter reported inference charges of about $0.0021 for Jev, $0.43 for Astra and $0.28 for Opus. Median response times were about 0.29, 2.00 and 3.42 seconds. Each model saw each message once. The generative models had medium reasoning and a 2,048-token output limit. This was the same yes/no question, not the four-way Jev setup selected in the earlier Haiku comparison.

These are still model-made labels. When the models disagree, somebody has to read the message. Three models agreeing does not turn their answer into mine.

does VERIFIED mean verified?

Then we gave them code from real projects: ZUP, which shows the state of my work; Transcripto, which reads agent sessions; and Helicon, which checks my stored notes. The job was specific: does this code actually support this claim about what the tool does?

One ZUP check was supposed to keep an agent’s self-graded pass from lighting a VERIFIED indicator. It did reject a record explicitly marked “same-session”. But if the check’s author was missing, that protection did not apply. The indicator could still light.

Across ten claims, each asked three times, Astra matched the source-derived answers in all 30 calls. Jev matched 26 of 30. In a fresh pass after fixing our result reader, Opus matched 30 of 30. The first Opus pass had six responses our strict reader could not use; follow-up checks found explanations followed by JSON. Those six are recorded as unusable, not wrong judgments, and the rerun is kept separate.

Jev approved the missing-author claim once and rejected it twice, with probabilities of 0.54, 0.48 and 0.46. Our cutoff was 0.50. It also approved another claim three times: that the writer correctly rejected --ok false. The code actually treats the string “false” as a true value and writes a pass.

The complete new comparison, including that parser investigation and rerun, produced 400 model responses for about $1.30 in provider-reported inference charges. That is the cost of these calls, not my subscription bill.

That is a much more useful mistake for me to see than an average score. If a model helps me find things to look at, a false alarm costs a read. If it tells me a check is safe when the code has a way around it, I may stop looking exactly where I need to look.

These were bounded source excerpts from real projects, with expected answers written from the code before the calls. They were not an autonomous repository review, an installed product trial or proof that somebody else would use the tools. The models did not run the code. We separately executed the real ZUP functions against temporary records: the missing-author record lit VERIFIED, and --ok false wrote a passing result. That limit matters when the word on the screen is VERIFIED.

very confident, still wrong

The earlier source-checking test had another version of this problem. Jev matched 28 of 29 model-written labels, but its one mistake accepted evidence from the wrong annex with a probability of 0.99.

Then we tested arithmetic. It missed every wrong sum: five examples, each asked twice. The wrong sums did not make it noticeably unsure.

Doing arithmetic and date checks in code improved the written tests. But the saved scan of my real notes reported that those checks settled none of 67,747 candidate pairs. Better written tests had not yet made that particular part useful in my actual notes.

and does it agree with me?

We also tried predicting decisions I had already made. On 17 cases with an agent recommendation, Jev picked the recommended option seven times when it could not see the recommendation. Show it the recommendation, and that became 15.

I had rejected the agent’s advice in 12 cases. Jev predicted eight of those rejections with the advice hidden, and two with it visible.

So, more agreement with the AI. Less agreement with me.

Those were small retrospective tests, prepared with the outcomes already known. I would not call that learning my taste. But it gives me a concrete thing to change: get the second opinion before showing it the first answer.

next: one evening, three options

For the next personal test, I want this to end with me actually going somewhere.

Three events I could realistically attend on the same evening. I pick one and write down why before seeing Jev’s answer. Every model gets the same options, date, travel, cost and description of what I enjoy. Staying home is allowed too.

Then we compare. Did it pick what I picked? Did reading its choice change my mind? Where did I actually go, and was I glad I went?

A model could disagree with my first choice and suggest something I end up enjoying. Or predict my choice perfectly and we both pick a disappointing evening. If I never go because it rains, that belongs in the result too.

For now, I would try Jev where it can bring a suspicious pair of notes to my attention. I would keep the actual evidence beside any decision about whether work is done. And I would give the event picker a fixed number of places on the list.

Sixty-two plans is still quite a week.

How this was made: AI helped run the experiments and prepare this draft. The memory, brief, source-support and arithmetic results above come from the earlier saved runs. The new comparison uses the stated model routes and bounded settings, not every model’s maximum possible effort. Correction labels came from one earlier model rater; source-check expectations were written by Codex. This is a personal investigation, not an independent benchmark. The evening test has not happened. Private transcript text is not published with the results.

Watch the embedded video in the Substack original

I was trying to build the optimal setup. Helicon, my second brain. Better context for the agents. A fleet working across several terminals. I wanted to hold on to the tasks and the vision for what we were building.

I still felt we were losing both. So I went back a step: what the hell did I actually say?

Just before one in the morning on 16 August, I had sent this:

and getting tired of prompting

Five words. The whole message, sent while the agent was still working.

I had been writing to it about the work. Then I was writing to it about being tired of writing to it. Apparently this was also where I was going to keep that thought.

The writing nobody was meant to read

There is so much talk about AI slop. I understand why. I also spend so much of my day writing to agents that I can close the terminal feeling I haven’t written anything.

I think of writing as the essay, the message to a friend, something intended for another person. Meanwhile, I have been explaining what I want to a machine. Sometimes badly enough that I have to explain it again.

When I edit an article, I know somebody will read it. In an agent conversation, I want the next thing to happen. I leave half a thought, misspell the important word, and send another message before the reply arrives.

There are jokes in the middle of instructions. Complaints become requests. I ask for one thing, then change my mind. Reading it back, I can follow the change before I have had a chance to make it sound consistent.

That is what I mean by some of my truest writing. A prompt can be careless or wrong. But these conversations caught what I wanted at the time, including things I now have trouble explaining.

My beloeved routine

On 14 August, I started a longer request like this (excerpt):

ok, and lets see they might solve it in the future so i can go back to my beloeved routine :)

The request continues into several projects and asks to get the other terminals working.

My beloved routine was sitting with one agent. Working on something together, in one conversation. I had built a fleet, and I missed that.

Then, in the same message where I ask for the old routine back, I ask to get the other terminals working. Both wishes are there. I can write a sensible explanation now for why I wanted the fleet. The transcript has me wanting both at once.

It also gave this publication its name. Beloeved. I was trying to type “beloved” while telling a machine I would like my routine back.

I hadn’t meant to keep a journal. I was trying to get things done.

Your agent transcripts are a journal

This is the new generation of journaling I keep thinking about. The entry happens while you are trying to do something. You return to it later and find more than the instructions you went looking for.

Mine contains a record of what I kept asking for. What I dropped. The routine I said I wanted while asking for work that would take me further from it.

I don’t need the agent to turn that into a polished account of my month. I want to read my sentences in the order I wrote them. There is enough interpretation involved in remembering a day without asking for another rewrite of it.

A transcript is still a partial record. It doesn’t know how the rest of my day went. And a record labelled “user” can contain tool results or injected context rather than something I typed. The actual human messages are the part I mean.

A way back into it

I built Transcripto to read these conversations locally. It works with transcript files from Claude Code, Codex and Cursor. You can search your own messages for something you remember saying, then replay the request and the recorded work around it.

I tried grading the prompts first, and published percentages from an early version. That version could count an attempted edit as success without a confirming result. Even a successful edit would not tell me whether the work was useful. The rankings are removed in 0.2.0. Those percentages cannot support advice about how to write.

Reading the conversations has been more interesting to me than grading them.

You don’t need my tool to try this. Open an old agent conversation, one you have mostly forgotten. Read your own messages before the answers. Look for something you kept asking for, and read what came just before it.

To search your submitted messages with Transcripto:

uvx --from transcripto==0.2.0 transcripto ask routine

Replace “routine” with a word you remember using. The results are snippets; replay or the original conversation gives you the surrounding context.

Check your agent’s retention settings before assuming these conversations will still be there next year. Transcripto reads the files that remain; it isn’t a backup.

I went looking for the instructions because I thought we had lost the plot. There I was, asking for my beloeved routine back, and for the other terminals to get working.

You might already be keeping a journal. Go read your side of it.

beloeved is where I write about working with coding agents, including the parts I have to reconsider. Subscribe to get the next essay.

Editorial note: A coding agent prepared this edit from Oscar’s account and his chosen thesis about agent transcripts as a new kind of journal. The two transcript quotations were checked against the original human messages; the second is an excerpt. Transcripto 0.2.0 is available on PyPI.