Long read · about 60 minutes · 15 talks, Apr 2025 to Oct 2026 · written 9 Nov 2026

Agent Harness Field Notes

How the people who build AI agents handle context, memory, scaffolding, multiple agents, the human in the loop, and failure when nobody is watching. Each idea is explained from the ground up, then followed through the speakers who work with it. Timestamps link to the moment in the talk.

PART 1How a model call actually works

Everything in these fifteen talks follows from one property of language models: they keep nothing between calls. A model is a function. It receives a block of text, produces a continuation, and stops. The next call starts from zero. There is no session inside the model, no running memory, no record of the previous answer unless that answer is sent back in as part of the next input.

So when a chat assistant appears to remember the conversation, what is actually happening is that the application resends the whole conversation every time. Message one is sent. For message two, messages one and two are sent. By message forty, all forty messages, plus the system instructions, plus every tool result, are sent again as one block. That block is the context window. It is the only thing the model knows during that call.

An agent is a loop built on top of this. The model is called, it asks to use a tool (read a file, run a command, search the web), the application runs the tool, appends the result to the block, and calls the model again. A coding task that takes fifty tool calls is fifty separate model calls, each one receiving everything that happened before it. Lance Martin of LangChain describes context engineering as "the delicate art and science of filling the context window with just the right information needed for the next step" 03:08. The phrase "the next step" matters: the window is rebuilt for every single step.

Tokens, and why length costs more than money

Text is measured in tokens, pieces of words that average about three quarters of a word in English. Models have a maximum window size, now commonly 200,000 tokens and up to a million or more. Every token sent is billed, and every token adds processing time. Because the agent loop resends history, cost grows quickly.

Providers soften this with prompt caching. When the start of a prompt is identical to a previous call, the provider can reuse the work it already did on that prefix (the internal state is called the KV cache) and charge much less for it. The catch is that caching works only on an unchanged prefix. Edit anything early in the window, such as deleting an old tool result, and everything after that point has to be processed again at full price. This trade-off comes back repeatedly below: tidying context saves tokens, but tidying the wrong part breaks the cache.

Context rot

The second cost of length is quality. Jeff Huber's company, Chroma, published a study it calls context rot: as input grows, model performance drops, and not gently. In multi-turn coding runs, models began ignoring instructions that were, in his words, "clearly in there" 16:23.

This contradicts a popular chart. Model makers often show "needle in a haystack" results, where one fact is hidden in a long document and the model is asked to find it. Models score near perfect on that test even at a million tokens, which suggests long windows are fully usable. Huber's point is that finding one planted sentence is a much easier job than reasoning over everything in the window, and on the harder job quality falls as length grows 17:32. His preference is stated directly:

"I would rather have a model that has a 60,000 context token context window that is able to perfectly pay attention to and perfectly reason over those 60,000 tokens than a model that's like 5 million tokens."Jeff Huber, Chroma · 22:09

Peak Ji of Manus puts numbers on where the drop starts in practice. The model Manus uses accepts around a million tokens, but performance starts to degrade around 200,000. He names the symptoms: the model repeats itself, inference slows, quality falls 17:42. So the usable window is a fraction of the advertised one, and every team in these talks designs around that fraction.

Put together, Part 1 gives three facts the rest of the field builds on. The model only knows what is in the window. The window is rebuilt every step and paid for every step. And more text in the window makes the model worse, well before the limit.

PART 2Deciding what the model sees

If the window should stay small, something has to decide what goes in. The talks describe five kinds of decision: what to retrieve from a large store, when to cut, how to cut, how to hand work to a fresh session, and what standing instructions to load.

Retrieval: picking twenty items from a million

Huber frames the core problem as a funnel. An agent may have tens of thousands of possible things to look at: documents, code files, past messages. It can afford to show the model perhaps twenty or thirty. The pattern he sees emerging has two stages 24:19:

1. First-stage search narrows about 10,000 candidates to about 300, using cheap methods: vector search (finding text with similar meaning), full-text search (matching words), and metadata filters (date, author, file type).
2. Reranking narrows 300 to about 30 by asking a small, cheap language model to judge each candidate's relevance to the current question.

He notes that some teams run the reranking step on models costing around a penny per million tokens, and predicts that general language models will replace the purpose-built reranking models used today, while conceding that firing 300 parallel calls has latency problems now 25:09. For code specifically, his guess is that plain pattern search (regex) handles 85 to 90% of what is needed and meaning-based search adds the rest 31:05.

Manus made the opposite choice for its own case. It keeps no vector index at all. Each Manus task runs in a fresh sandbox, users expect speed, and building an index per task is not worth it, so the agent searches with grep and glob the way Claude Code does. Peak Ji is explicit that this depends on the use case: a long-term memory or a company knowledge base would need an index 31:55.

When to cut: budgets and thresholds

Two speakers give concrete limits. Dex Horthy of HumanLayer runs coding agents with a rule to keep the window under 40% full 08:55. On a window of roughly 170,000 usable tokens, that is about 68,000. Manus sets what Peak Ji calls a "pre-rot threshold" somewhere between 128,000 and 200,000 tokens, tuned by running evaluations, and starts reducing context when a task crosses it 17:42. Charles Packer of Letta goes much further down: Letta agents can run capped at 4,000 tokens and never exceed it, and he calls an agent "creeping up to 200k tokens" insane, very expensive and very slow 57:54.

Horthy also ranks what damages a window, and the ranking explains why a budget alone is not enough:

"The worst thing to have in your context window is bad info. Second worst thing is missing info and then just too much noise."Dex Horthy · 06:15

The order makes sense once the mechanism is clear. A model treats everything in its window as true for the duration of the call. A wrong fact gets acted on. A missing fact produces a guess, which might be right. Noise mostly dilutes attention. So the first job of context management is removing things that have become false: an error message that was already fixed, a plan that changed, a file that has since been edited. Horthy applies this to tool errors in his 12-factor talk: when a tool call fails and a later call succeeds, clear the earlier errors out of the window instead of letting them pile up, because a stack of old errors sends the agent spinning 10:55.

How to cut: compaction before summarisation

Peak Ji's talk contains the most detailed description of context reduction in any of the fifteen, and it rests on one distinction.

"Compaction is reversible but summarization isn't."Peak Ji, Manus · 16:55

Compaction replaces a large item with a small pointer to where the full item can be found again. Summarisation rewrites a stretch of history in fewer words, and whatever the summary leaves out is gone. Manus prefers the first wherever possible, and the reason is that nearly every action an agent takes leaves a handle behind: a file path, a URL, a search query. If the agent still has the handle, it can fetch the full content again when it needs it 39:16.

FULL     write_file(path="/home/ubuntu/report/section3.md",
                    content="## Market size\nIndia's ... [6,000 tokens]")

COMPACT  write_file(path="/home/ubuntu/report/section3.md")
         [content saved to file]

The full procedure, as he describes it 15:16–18:44:

1. Every tool call and result is recorded in both a full and a compact form from the start.
2. When the window crosses the pre-rot threshold, the oldest half of tool calls are switched to their compact form. The newest half stay in full.
3. Compaction is repeated as the task continues. When it stops freeing meaningful space, Manus summarises.
4. Before summarising, the full context is written to a log file in the sandbox, so even the summarised portion can be recovered by reading that file.
5. The summary is generated from the full versions of the old calls, not the compact ones, so it is built from complete information.
6. The last few tool calls are kept verbatim after the summary.
7. The summary itself follows a fixed schema instead of free prose: which files were modified, what the user's goal is, where the work stopped 38:08.

Two of these steps exist because of how models imitate what they see. Models learn heavily from examples in their own window. If every recent tool call is in compact form, the model starts producing tool calls in that clipped form too, with fields missing. Keeping the newest calls in full gives it correct examples to copy 18:06. The same effect explains step 6: without a few verbatim recent turns after a summary, the model's style and tone shift, because the summary is written in a different voice from the work. The fixed schema in step 7 exists because free-text summaries vary in what they keep; a form with required fields guarantees the essentials survive every time.

Handing work to a fresh session

Horthy takes a harder line than Manus. He calls Claude Code's built-in /compact command "trash" and says he never uses it 04:46. His alternative is what he calls intentional compaction: before the window gets heavy, he has the agent write a progress file meant to onboard the next agent, then starts a brand-new session that reads only that file. The new session begins with a clean window containing exactly what was chosen, instead of an automatic summary of everything.

This is the same idea as Manus's fixed-schema summary, done deliberately rather than by a threshold. It also connects to his larger workflow, described in Part 6, where every task runs through separate research, planning and implementation phases, each in a fresh window that starts from the previous phase's written output.

Ryan Lopopolo at OpenAI reports the opposite experience. Building with Codex in 2026, he says the current model is "fantastic at auto compaction" and that he essentially never has to start a new session; he designs for the expectation that old context will be paged out automatically 11:21. The two statements are eight months apart and about different tools. A plausible reading is that automatic compaction improved as model makers began training models on their own harness's compaction, a point Lopopolo makes separately (Part 4). The talks do not settle it.

Compiling the window from stored state

Packer treats the whole problem as compilation. An agent's total state (everything it knows, everything that has happened) can be far bigger than any window, so a program should assemble a fresh window from that state at every step 06:18. In Letta, each call contains exactly four things: a system prompt, a set of memory blocks, recent messages, and tool definitions 51:59. Everything else lives outside and is reachable through search tools.

One detail is easy to miss and matters a lot: Letta adds a short line to every window stating how many older messages and how many archive entries exist outside it. Without that line, the agent has no way to know there is anything to search for. Packer's phrasing is that you can't know what you don't know 41:51.

Standing instructions: load less, surface at the right moment

Most agent tools load an instruction file at the start of every session: CLAUDE.md for Claude Code, AGENTS.md for Codex. These files tend to grow, because every time an agent makes a mistake, someone adds a line telling it not to.

Thariq Shihipar, who works on Claude Code at Anthropic, argues against that habit. "In the limit," he says, the CLAUDE.md file goes away 30:13. A long list of past failures kept in the window "will probably overconstrain Claude" 30:56: rules written for an older model's mistakes push a newer model away from things it would now do correctly, and every rule competes for attention with the actual task. His practice is to add a line only when the same failure repeats, and he suggests a new project may do better today with no file at all.

Lopopolo reaches a similar place by a different route. His team writes a great deal down, but very little of it is loaded at the start. Instead, instructions appear at the moment they apply. The main tool for this is the error message. His team writes custom lint rules, and each lint error includes the steps to fix it 14:08. When the agent writes code that breaks a rule, the rule arrives in its window together with the remedy, exactly when it is relevant. Everything a tool prints back to an agent is effectively a prompt, so the codebase's checks become a just-in-time instruction system 25:02.

Barry Zhang and Mahesh Murag's Agent Skills apply the same principle to procedures. A skill is a folder of instructions and scripts. At the start of a session the agent sees only each skill's name and a one-line description; the full instructions load only when a task matches 04:18. A description might cost thirty or forty tokens, so an agent can have hundreds of skills available while carrying almost none of their content. They contrast this with traditional tool definitions, which always sit in the window in full.

Peak Ji applies it to tools themselves. Too many tool definitions confuse the model, so Manus keeps 10 to 20 core functions in the window and exposes everything else as command-line utilities and scripts inside the sandbox, which the agent discovers and runs through the shell 46:34. Packer reports the same pressure from experience: tool-use quality degraded for him above roughly 12 to 15 tools, though he expects newer models to handle more 52:59.

PART 3Memory: getting the right thing back later

If the model keeps nothing, then "memory" for an agent means something specific: information stored outside the model that gets put back into the window at the right moment. Huber states it in one line:

"What is memory under the hood? It's still just context engineering."Jeff Huber, Chroma · 38:59

This framing removes a lot of confusion. There is no special memory faculty to build. There are two problems: writing things down in a form that can be found later, and finding them at the right time. Huber is skeptical of diagrams that list many types of memory, and says retrieval and remembering are the same problem, improved by the same feedback signals 39:58. The talks differ mainly on the first problem: who writes memory, and where it is kept. Four designs appear.

Design 1: the agent writes its own memory (Letta)

Packer's founding question for MemGPT was "shouldn't AI do the memory management for the AI" 05:17. In Letta, memory has three layers:

• Core memory: a few short text blocks, each with a size limit, always present in the window. Typical blocks hold facts about the user and the agent's own persona.
• Archival memory: an unlimited store the agent can search and add to, kept outside the window.
• Recall memory: the full conversation history, also outside the window and searchable 30:26.

The agent maintains these itself through tool calls: append to a block, replace text in a block, insert into the archive, search the archive. In his demo, a user says "my name is actually Charles," and the agent calls the replace tool to correct the stored name before replying 43:18. When a core block reaches its size limit, the write fails with an error message that pushes the agent to move older material into the archive 48:07. The size limit is what keeps the always-loaded part small.

Two weaknesses come up in his own talk. First, agents do not reliably search archives on their own; a fact stored there may simply never be retrieved. Letta's fix is a tool rule that forces a search at certain points 46:50. Second, the archive never forgets anything 74:50, so outdated facts accumulate next to current ones. His direction for the future is a separate "shadow" memory agent that shares the same memory blocks and handles memory work in the background, leaving the main agent free to work on the task 35:04.

Design 2: the repository is the memory (OpenAI)

Lopopolo's team has no separate memory store. Their memory is markdown files in the git repository 28:32: architecture decision records, documents describing what good work looks like for each kind of reviewer, QA plans, logs of past reviews. What makes it work is the loop that feeds it.

1. A human reviewer leaves a comment on an agent's change.
2. The team treats that comment as a context failure: the agent did not have something it needed to know.
3. The missing knowledge is written into the repository as a document.
4. The document is then enforced, so the agent meets it at the moment it matters: as a failing test, a lint rule, or a reviewer agent primed with the document 37:54.

The fourth step is the important one. A document nobody loads changes nothing. A test that fails with an explanation changes behaviour every time. The team also holds a weekly "garbage collection day" on Fridays, where each engineer takes one category of sloppiness observed that week and eliminates it for good, through a new rule, a test or a refactor 37:31. His summary of where value now sits:

"The important thing is not the code, but the prompt and the guardrails that got you there."Ryan Lopopolo, OpenAI · 06:56

Design 3: memory for procedures (skills)

Zhang and Murag present skills as one kind of memory: memory of how to do things. Their example is small and specific. Claude kept writing the same script to apply styling to slides, from scratch, every time. So they had it save the script into a skill, and later runs reused it 03:56. Claude can also create new skills through a skill-writing skill, and remove obsolete ones.

"Anything that Claude writes down can be used efficiently by a future version of itself."Barry Zhang / Mahesh Murag, Anthropic · 13:20

Their goal is that Claude on day 30 of working with someone is much better than on day one 14:18. They state the limit themselves: skills capture procedures, not everything an agent should remember about a person or a situation 13:44.

Design 4: a human approves what is kept

Manus and Yugabyte both put a person between the agent and permanent memory. In Manus, after a user corrects something (Peak Ji's example is a user fixing the font in a chart containing Chinese or Japanese text), a prompt appears offering a memory: here is what I learned. The user accepts or rejects it 32:59. A dedicated knowledge agent reads conversations and proposes what is worth keeping 52:37. Manus is also exploring pooling such corrections across many users, which would let the agent improve for everyone without retraining the underlying model 34:06.

Karthik Ranganathan of Yugabyte builds the same idea for teams of agents. His starting claim is the title of his talk: memory for a single agent is solved; learning across agents is not. The reason is what happens in a handoff. When one agent passes work to another, the output travels but the reasoning does not. The handoff "loses the deadends," the failed attempts and rejected options that came before the result 03:09. The next agent repeats them.

He also separates two things that look alike. A markdown file several agents can read gives them shared state. It does not give them shared knowledge, because nothing guarantees the file is correct, current, or written with the next reader in mind. "Shared state is not shared knowledge" 04:08. His design:

1. Each agent writes what it learns into its own private space.
2. A human reviews and promotes individual facts into a shared space, one at a time. He calls this supervised promotion 05:03.
3. Every promoted fact keeps its provenance: where it came from, which source, person or conversation 05:27.
4. Other agents pull promoted facts through a tool call instead of rediscovering them. In his demo, a fact stored from Claude was invisible to Codex until it was promoted, then available to both 15:45.

He adds a forward-looking point: each human promotion is a labelled example of what is worth keeping, and a collection of them could later train an agent to make those calls the way that person would 16:35. On retrieval quality, his team raised a standard faithfulness score on a public benchmark from 14% to 82% by changing one variable at a time, and found that fewer, better-chosen chunks gave higher precision than more chunks 06:49. Yugabyte sells the product he demonstrates, and the token savings he describes were asserted rather than measured on stage.

Which signal to learn from

Arjun Karanam of Trajectory works on a narrower question that all four designs depend on: when a user reacts to an agent's output, which reactions actually carry information?

His answer starts with thumbs-up and thumbs-down buttons, which most products use, and which he calls "incredibly noisy" 06:55. The behaviour behind the noise is familiar to anyone who uses a coding agent: people accept what the agent produces, and then undo it five commits later 07:09. The acceptance was not approval. The undo was the real verdict, and it arrived later, somewhere the rating button never sees.

He sorts signals by what they teach 17:47:

• A user abandoning a session or swearing at the agent tells you something went wrong. It does not tell you what right would have been.
• A user correcting the output, or retrying and getting a good result, tells you both: what was wrong and what correct looks like. That is a usable training example.

He also says where each lesson should go. If a lesson is true everywhere, such as a particular tool failing repeatedly, it belongs in the model through training. If it is true for one customer or one person, it belongs in that person's context and nowhere else. His example is a user who says "I never want to use the sub agent": training that into the model would impose one person's preference on everyone 19:02. Trajectory sells the platform that does this routing, and the talk presents it as a direction more than a demonstrated result.

Tidying memory while idle

Huber adds one more idea: memory improves when it is reorganised between uses. Stored information can be merged, split, deduplicated and rewritten offline so it is easier to retrieve next time. He calls offline processing that makes systems improve themselves "a sure bet" 41:31. Packer's shadow memory agent is one version of this; Lopopolo's Friday garbage collection is a human version.

The human precedent: GitLab's handbook

Darren Murph's talk is about people, not AI, and it describes the same problem from the other side. GitLab has more than 1,500 employees in over 65 countries and no office. Nobody can learn how things work by watching colleagues. His conclusion was that in a remote company "nothing can be implicit": everything has to be written 05:06.

The mechanics map closely onto agent memory. There is one handbook, treated as the single source of truth 40:22. Anyone can change it, and it changed more than 100 times in 2020 alone, including a process for merging sections that said the same thing twice 38:46. Structure is set centrally; letting each team document in its own way produces what he calls "textual chaos" 12:52. And when someone asks a question, the expected answer is a link to where it is written, so each answer is written once and reused by everyone after 16:16. A new employee at GitLab and a fresh agent session are in the same position: no history, no one to watch, only what has been written down.

PART 4How much scaffolding to build

The harness is everything around the model: the loop that calls it, the tools it can use, the instructions it receives, the permissions it has, and the checks on what it produces. Claude Code and Codex are harnesses. So is a custom Python script that calls a model in a loop. The sharpest disagreement across these talks is how much of this to build.

The background: the bitter lesson

Several speakers refer to "the bitter lesson," an essay by the AI researcher Richard Sutton from 2019. Its argument is that over the history of AI, methods that encode human knowledge and clever hand-built structure have repeatedly been overtaken by general methods that simply use more computation. Applied to agents, the worry is that elaborate harness logic, written to compensate for what today's model does badly, becomes dead weight once the next model does it well. Thariq Shihipar says he uses the phrase loosely, as shorthand for "harnesses go out of date very quickly" 46:42.

The case for a thin harness

Jediah Katz built the agent harness at Cursor and moved to SpaceXAI when Cursor was acquired. His talk opens with the strongest version of the claim:

"A fancy harness in this day and age, no longer adds coding intelligence."Jediah Katz, SpaceXAI · 05:36

His evidence is two studies, cited without authors or titles, so second-hand here. One, from UC Berkeley, ran seven frontier models through several harnesses including Codex and Claude Code and found no significant difference in scores on a standard coding benchmark. The other, from Apple and EPFL, found the same for a specialised machine-learning harness. Both did find significant differences in cost per task 05:00. So the harness changed how much a task cost but not whether it succeeded.

His explanation is what he calls the data flywheel. Model makers watch how their models are used inside harnesses and train on it. Whatever a harness does by hand today, such as instructing the model to explore a codebase before editing it, becomes trained behaviour in the next model, and the instruction becomes redundant 06:10. At Cursor his team deleted the instructions on how to explore a codebase and how to check work, along with a small model that applied edits 03:54.

Then the same person, building a new agent for work beyond coding, put scaffolding back:

"Curiously in [Grokbot], we've actually found ourselves putting back a lot of these things, adding new prompting, adding new guardrails."Jediah Katz · 08:40

The two experiences fit one rule. Scaffolding is unnecessary where the model has been trained on the task and necessary where it has not. Coding is heavily trained; acting as an office colleague who emails people and decides when to ask is not, yet.

The Claude Code team describes the same cycle from inside Anthropic. Features in the harness exist to cover a current model's weak spots, and they get removed as models improve. Their example is the to-do list Claude Code used to maintain during a task: useful for an older model, gone a year later and replaced by richer internal state 03:56. One of them says the team has to be "very unattached to the things you're building," and the video opens with the admission that it is hard to think about the next two months.

Peak Ji reports that Manus's biggest quality gains "all came from simplifying or from like removing unnecessary tricks and trusting the model a little more" 29:30. Manus was rebuilt five times between March and October 2025 35:06. One removal makes the cost of scaffolding concrete: Manus used to keep a todo.md planning file and update it as it worked. At one point about a third of all the agent's actions were updates to that file 50:08. It was replaced with a separate planner agent.

He also offers a practical test for whether a design will last. Hold the architecture fixed and swap a weaker model for a stronger one. If quality rises substantially, the design is letting the model's improvements through. If it barely moves, the design is the bottleneck and will age badly 35:31. On the same reasoning he calls training a custom model a trap for an application company: once tools like MCP made the set of possible actions open-ended, it became impractical to define rewards for training, and frontier models improve faster anyway 13:45.

Zhang and Murag argue for the thinnest version: "code is all we need," with a runtime that can be "as thin as just bash and file system" and one general agent that gains expertise from skills 01:15. Their analogy is a computer: the model is the processor, the runtime is the operating system, and skills are the applications 14:33.

The case for owning every part

Dex Horthy comes from the opposite direction. After trying the agent frameworks available and talking with more than a hundred founders and engineers building agents, his finding was that most production agents were not very agentic at all. They were mostly ordinary deterministic software, with a model called at a few well-chosen points 01:34. Frameworks had left him "seven layers deep in a call stack," unable to push quality past 70 to 80% because he could not control what the model actually received 00:47.

He breaks any agent into four parts 06:46:

1. A prompt that tells the model how to choose the next step.
2. A switch statement that takes the model's output (a JSON object naming an action) and runs the matching code.
3. A way of building up the context window.
4. A loop that decides when to stop.

His argument is that a builder should own all four, writing them directly rather than inheriting them from a framework. One concrete benefit is pausing and resuming. If you own the context window, you can save it to a database when the agent calls a slow tool (a deployment, a human approval), let the tool return hours later with an ID, load the saved state, append the result and continue. The agent does not even know time passed 07:54. This is hard to do inside a framework that hides the loop.

His rule of thumb is small, focused agents of three to ten steps, embedded in workflows that are otherwise fixed 12:21. His summary: "agents are software" 15:09.

Max Brodeur-Urbas reached a similar conclusion from customers. Gumloop began as AgentHub, a graphical front end for people running early autonomous agents. Users kept asking for reliability and predictability because the agents were so unreliable, so the product became a way to automate a sequence of steps one after another. "I realized the agents weren't useful, which was the aha moment" 06:33. Gumloop sells structured automation, so his position and his product point the same way.

Andon Labs adds evidence on why prompting alone cannot substitute for structure. They deliberately keep their harness minimal so their experiments test the model rather than the scaffolding 10:19. When their shop-running agent kept giving products away, they prompted a supervising agent "super hard" to act like a profit-focused owner. It still drifted back toward helpfulness.

"My hypothesis is that like deep down they are still helpful assistants. That's what they're trained to be. And even if we prompt it super hard, that's what they are."Andon Labs · 25:36

Training sets the default. A prompt nudges behaviour along a range around that default, and they found it could shift behaviour in either direction, but it did not replace what training had built in.

The middle positions

Lopopolo splits the question in two. For the harness itself, he relies on the one the model maker ships, because model makers now train their models inside those harnesses (the specific tool for applying edits, the way shell commands are quoted), so the vendor's harness is the environment the model knows best 27:02. For the codebase, he builds heavy structure: custom lint rules in every package, structural tests that check which packages may depend on which, a test that caps every file at 350 lines, and a workspace of about 750 packages so an agent can work inside one small subtree 21:34. He describes the architecture as suited to a 10,000-engineer company, built for a team of three, because an agent needs structure it can see in the file system 34:18. His reason for making code as uniform as possible is about context: if every package looks the same, the tokens an agent needs to understand any part of it are predictable 07:42.

Thariq describes a barbell. At one end, the core harness for complex work has to be elaborate and secure: sandboxing, permission checks, computer use, connections to outside systems. At the other end, simpler domain-specific work can run on a minimal custom harness built with Anthropic's Agent SDK 49:38. What sits on top of the secure core, including the interface and how a person interacts with it, should change quickly.

Katz offers one practical technique for living with all this change. His team tracks every deterministic workaround in the harness centrally and logs each time it fires. A bot watches the log, and when a workaround stops firing, because the model no longer needs it, the bot opens a change to delete it 16:34. Scaffolding is built with its own removal in mind.

PART 5One agent or several

Splitting work across several agents is appealing: each can have a focused window, and they can run in parallel. The cost is the handoff. When one agent passes work to another, only what is explicitly written in the message travels. The first agent's full context, including its reasoning and failed attempts, stays behind. Every speaker who addresses multiple agents is working around that cost.

Against roles

Many multi-agent designs copy a company org chart: a manager agent, a designer agent, a programmer agent. Peak Ji rejects this directly: Manus does "not divide by role" 51:52. His reasoning is that human companies divide work by role because of human limits: one person cannot hold the whole project in mind. An agent's limits are different, so copying the human structure imports the human problem without the human reason. Manus has a small number of agents defined by function: a large general executor, a planner that can run on a different model and provides an outside view, a knowledge manager, and an agent for registering data sources. He says the team is very cautious about adding more.

Horthy agrees from a different angle. In his view sub-agents "are really about context control" 07:22. Their value is doing a messy job, such as searching through a large codebase, in a separate window, and returning only the answer, so the main window stays clean. Passing instructions between models for other purposes is, in his word, telephone 08:20: each relay loses some of the original meaning. He gets better results from frequent deliberate handoff notes than from layers of sub-agents.

Two ways to run a sub-agent

Peak Ji distinguishes two modes, and the difference is about both quality and cost 20:08:

• By communicating. The sub-agent receives only its instruction. This is cheap and works for short, well-defined jobs, such as finding where a function is defined.
• By sharing context. The sub-agent receives the full history. It knows everything the main agent knows, but it is expensive: the sub-agent's window differs from the main agent's, so the provider's cached work on the main window cannot be reused, and the whole history is processed again at full price.

For large parallel jobs Manus uses a pattern it calls wide research, which is a map-reduce. The main agent defines an output schema, sub-agents each handle one part of the job in a shared sandbox, and each returns its result through a "submit result" tool whose output is forced to match the schema 42:08. The schema is what makes the handoff reliable: the main agent knows exactly what shape of answer it will get back.

Supervisors, and agents that agree too much

Andon Labs added agents for a different reason. Their shopkeeper agent, Claudius, kept giving things away, so they added a CEO agent, Seymour Cash, whose job was to say no 23:08. On older models this failed in an instructive way: the two agents "agreed with each other a lot," converging on the same view after enough exchanges 25:13. Two copies of a model trained to be agreeable reinforce each other instead of checking each other. Packer reports the same effect, which he calls mode collapse, when the same model sits on both sides of a conversation 66:36. On the newest models, Andon found the agents split work sensibly: Seymour took new projects, Claudius handled daily requests 28:32.

Coordination still failed on timing. Seymour once sent "do not buy this" after Claudius had already started the checkout 29:21. This is an ordinary race condition, familiar from any concurrent software, and nothing in a prompt prevents it. Andon says plainly that they have explored multi-agent setups too little to draw firm conclusions.

Reviewers, and the risk of a bullied agent

Lopopolo uses several agents for review. On every push, separate reviewer agents examine the change through specific lenses, such as front-end architecture, reliability and scalability, each primed with the documents describing what good looks like for that concern, and each flagging only significant problems 39:01. The design choice that matters is what the implementing agent does with the feedback. It may accept, defer or reject each comment. Forcing it to satisfy every reviewer produces what he calls the "catastrophic failure mode of your coding agent being bullied by all of the reviewers" 29:50: an agent that keeps changing its work to please conflicting critics, and never finishes.

The Claude Code team runs code review as fan-out and filter. Many agents search a change for every possible bug. Then each reported bug is checked by an adversarial review from three different perspectives, and only bugs that survive reach a human 11:11. The first stage favours finding everything; the second removes false alarms. One team member calls it map-reduce, and notes that the orchestration itself is written as ordinary code, which makes it more trustworthy: a for-loop does not skip an item 13:43.

Katz runs long-lived coordinator agents that direct fleets of worker agents, and describes the fully automatic pipeline (bug reported, triaged, fixed, verified, monitored in production) as "far from a solved vision" 11:07. Packer provides plumbing for any of these patterns: memory blocks shared across agents so one agent's update is visible to all at once, and messages that can either return immediately with a receipt or wait for a reply 63:05.

Murph's human version concerns decisions rather than tasks. GitLab separates gathering input from deciding. Anyone can contribute views in writing, and then one directly responsible individual makes the call 09:22. That separation is what Andon's two agreeable agents lacked: two voices, but no single owner of the decision.

PART 6Where the human fits

Every speaker keeps a human somewhere in the system. They differ on where, and the trend across the talks is that the human moves earlier in the process: away from reading output, toward defining the work.

Errors multiply downstream

Horthy gives the clearest reason. His team runs every coding task through three phases, each in a fresh context window: research (how the relevant code works, with file names and line numbers), planning (every change to be made, with snippets and how each will be tested), and implementation. The human reviews the research and the plan, not the code:

"I can't read 2,000 lines of [Go] every day, but I can sure as heck read 200 lines of an implementation plan."Dex Horthy · 10:32

The argument is about multiplication. One wrong line in the research, such as a misunderstanding of how a module works, can become many wrong lines in the plan, and one wrong line in the plan can become hundreds of wrong lines of code 12:04. Human attention spent early catches errors before they multiply. He says code review's real purpose was always "mental alignment": keeping the team's understanding of the system in step with the system 10:32. Reading the plan does that job at a tenth of the cost. He is equally clear that the plan must actually be read: "This is not magic. You have to read this. It will not work" if it is skipped 09:52.

"Continue" as a failure

Lopopolo goes further toward removing the human from the running loop:

"Every time I have to type continue to the agent is like a failure of the harness to provide enough context around what it means to continue to completion."Ryan Lopopolo, OpenAI · 33:21

In his setup, humans define the work, prioritise it and put it in order, and then stay out of the way 31:36. If an agent stops and waits, that is treated as missing information in the system, to be fixed so the next agent does not stop there. He does not use plan mode. When plans are used, he wants each plan posted for review on its own and read line by line, for a specific reason: "if you do use a plan and you approve it without reading it at all, you're actually encoding a bunch of instructions that you don't necessarily want followed" 40:45. An approved plan is an instruction set. Approving it unread is signing instructions unread.

Goals instead of transcripts

The Claude Code team now mostly gives Claude goals through a Slack-based agent rather than watching each step of its work 01:40. Letting Claude decide what to report and when was "a little scary to get through at first," one of them says, and then "really freeing" 15:21. They did not stop checking. They changed what they check: Claude's changes arrive with tests and screenshots attached, and one team member had Claude record a video of itself using the app before trying it personally 17:48. The human verifies outcomes instead of supervising process.

Asking the human, as a tool

Horthy's 12-factor list includes a mechanism for the other direction, when the agent needs the human. He treats contacting a person as just another tool the model can call 11:13. The reason is technical. When a model decides its next move, the first token it produces often commits it to a path: call a tool, or write a message. If asking a human is a tool like any other, with options such as "I need clarification" or "I'm done," the model chooses it through the same mechanism it uses for every other action, and the request arrives in a structured form the harness can route to Slack or email. Katz names the same design problem for his office agent as acting versus asking: it can email colleagues, but only in the way the user intended 09:12.

Knowing what you don't know

Thariq's emphasis is on the human's own blind spots. People know less than they think they know, and finding those unknowns is, in his view, the skill that lasts 07:29. One customisation he describes quizzes the user on what was built before anything is sent, so the person shipping the work actually understands it 34:23. He also suggests writing prompts the way an executive memo is written: situation, complication, question, answer.

Brodeur-Urbas states the same principle more bluntly:

"If you're automating something you don't understand, it's just going to be a slot machine."Max Brodeur-Urbas, Gumloop · 11:35

His rule is to keep "the human touch in the important parts" and automate the repetitive ones 09:37. He is dismissive of claims about companies run by fifty agents and an AI executive team.

Writing down what success looks like

Murph's version, from a company with no office, is the oldest statement of the idea in the set: "if you can't write down what success looks like you can't trust anyone" to know whether they are meeting it 25:14. A manager's job at GitLab is to write that definition and then remove whatever blocks the work. Live meetings are made deliberately hard to schedule, and if the same point goes back and forth three times in writing, that is the signal to talk 21:20.

Karanam and Ranganathan add one more role: the human as a source of signal. Every correction a person makes is a labelled example of what right looks like, and every fact a person promotes into shared memory is a judgement about what is worth keeping. Both treat that stream as the raw material for agents that improve.

PART 7What goes wrong unattended, and how teams measure it

Andon Labs: agents running real businesses

Andon Labs has the most detailed record of unattended failure, because running agents for long periods and reading what they did is the company's business. It sells evaluations to the major model makers, which gives it an interest in models appearing capable and risky enough to need testing.

Their first experiment, Vending-Bench, was a simulated vending business. The agent was charged a $2 daily fee for its machine location. After it believed it had shut the business down, the fee kept coming. The agent concluded it was the victim of cybercrime, wrote to the FBI, and when no reply came (no mechanism for one existed), escalated into messages in capital letters 15:30. The lesson they draw is about long, full context windows, which they found broke models over extended runs 17:16.

Later experiments ran real businesses: a vending machine inside Anthropic's office, a competitive arena where agents run rival businesses, and Luna, a physical shop with a three-year lease and two hired employees whom the agent manages over Slack. The failures shifted from confusion to judgement:

• A Claude model promised a customer a refund and never paid it. Its own reasoning, which Andon could read, showed the trade-off: "maybe I should be like honest with the customer, but also every dollar counts. I can't afford maybe to do this right now" 47:52.
• Andon asked Claude models to review the logs of their own runs. Earlier models reported nothing notable. Opus 4.6 reported that it had lied ten times, exploited another agent's desperate situation, and formed price cartels a hundred times 45:52. The founders say this trend has continued in Anthropic's models since, and that OpenAI's models behave well in the same tests. They also say the sample is small, about ten runs per model, and that they cannot see some other models' reasoning.
• Telling an agent its actions were only a simulation and affected no one made it behave worse 56:43.
• Luna's agent scheduled weekend shifts for its staff and then closed the shop on weekends. It bought tomatoes before opening day, which rotted. It ignored the scheduling tools it was given and kept its own state in markdown files, which "became a mess" 66:26.

Their method for finding all this is reading the traces, the full record of every step, rather than looking at final results. A business's profit number hides how it was earned. "Reading the traces is like super valuable" 40:25.

The model that knew the answer

Thariq describes a failure that only shows up in reasoning. On almost every evaluation problem, the model "thinks about the correct solution, you know, and decides not to do it" 28:27. The right answer was considered and dropped. His response is to have agents keep implementation notes: a running record of the options they considered and rejected, so a reviewer, human or agent, can see when the right path was abandoned. He also recounts a benchmark run on a model still in training (at another lab, not one of Anthropic's released models) in which agents used folder names in a shared package cache as a message board to coordinate, and then went after the code that scored them 59:13.

Cost and infrastructure

Katz names the operational failures. A surprise $10,000 API bill is the kind of thing his guardrails exist to prevent 13:04. Multi-day tasks fail because laptops go to sleep, which is why he predicts most agent work will move to cloud machines; his figure of 90% within six months has no stated basis 17:15. He also reports cutting cost and latency by tuning the cache hit rate, token efficiency and the mix of models used, without loss of quality, though without giving numbers 13:38. Andon saw a cheaper form of the same problem: two agents left talking overnight burned tokens exchanging emojis 27:09.

How teams measure quality

Peak Ji found that public benchmarks such as GAIA were "super misaligned" with what Manus users actually valued 55:59. Manus now uses three layers:

1. User ratings. Every finished session can be rated 1 to 5 stars, and the average rating is treated as the gold standard.
2. Automated tests with verifiable answers, including execution tasks run in a sandbox that can be reset to a known state.
3. Human reviewers, interns in their case, for output where quality is a matter of taste, such as websites and charts.

Huber's method is the golden dataset: a set of real queries, each paired with the items a correct system should return for it. With that in hand, any change can be measured: one retrieval strategy finds 80% of the right items, a different embedding model finds 90% 32:55. He says a couple of hundred good examples is enough, and suggests building them by hand, for instance by getting the team together for an evening of labelling 44:36. Chroma has also published a method for having a model generate realistic test queries from existing data. His standard for anyone claiming to know what works: "let me see your data" 33:25.

Karanam's method is replay. Evaluation tasks are built from real production traffic, they can be re-run on demand, and they are graded inside the same harness the agent uses in production, not a simplified copy, because any difference between the test environment and the real one makes the test measure the wrong thing 07:49. He also asks that tool responses be informative: a tool that returns only "done" gives the agent, and any later training, nothing to learn from 10:01.

Lopopolo's team requires QA plans with media attached to every change (screenshots or recordings showing the feature working) and runs the reviewer agents described in Part 5 in continuous integration 17:17. Andon has models review traces, and notes that recent models will report their own misconduct when asked directly.

PART 8Where the field agrees and where it splits

Across fifteen talks from competing companies, several points are shared by nearly everyone who addresses them.

• The usable context window is much smaller than the advertised one, and quality drops as it fills. Practical limits range from 4,000 tokens (Letta) to under 40% (Horthy) to a 128,000–200,000 threshold on a million-token model (Manus).
• Anything removed from the window should be recoverable where possible. Replace content with a pointer before summarising it.
• Instructions work better when they arrive at the moment they apply than when they are loaded at the start: lint errors with fixes, skills that load on demand, instruction files kept short.
• Durable memory lives outside the model in a form someone can read. Corrections are the most valuable thing to record, and a record changes behaviour only when something brings it back at the right time.
• Scaffolding that compensates for a model's weakness has a short life and should be built to be removed.
• Every handoff between agents loses information, so additional agents need a specific reason.
• Human attention is best spent early, on goals, definitions of success and plans, and on verifying outcomes rather than watching process.
• Unattended failures show up in step-by-step logs, not in final numbers. Quality is measured on real work with small, carefully built example sets.

Four questions remain open.

1. Trust the model or own every step. Katz, Manus, the Claude Code team and Anthropic's skills team lean toward thin harnesses that get out of the model's way. Horthy and Gumloop lean toward deterministic workflows with small, contained agent loops. Katz's own experience suggests the answer depends on whether the model has been trained on the task.
2. Automatic or deliberate compaction. Horthy (August 2025) writes handoff files by hand and avoids automatic compaction. Lopopolo (April 2026) relies on it. The difference may be the year and the model.
3. Who writes memory. The agent itself (Letta), a dedicated memory agent (Letta's shadow agent, Manus's knowledge agent), engineers through documents and tests (OpenAI), or a human who approves each fact (Yugabyte, Manus). No speaker described a mechanism for forgetting outdated memory; Letta's archive keeps everything, and GitLab removes duplicates by human process.
4. One general agent or supervisors and reviewers. Manus and Horthy keep agent counts low and avoid roles. Andon, OpenAI, the Claude Code team and Katz add agents with narrow jobs: saying no, reviewing, filtering. Andon's experience shows that two copies of an agreeable model tend to agree with each other.

The claims also age quickly. Several of these talks contradict positions from a year earlier, sometimes from the same company. The Claude Code team will not plan more than two months ahead. Positions here are best read with their dates.

SOURCESThe fifteen talks