Note
The software factory stack
Build on two things every workflow needs. See what happened and continue from the last verified state.
A lot of people are saying to build a software factory lately. Some are warning against it. If you choose to build one, here’s the best thinking I’ve found on it, laid out as a map.
My bet is the bottom layer. If I can’t see what a run did, or pick up from the last verified step after a worker dies, I don’t trust anything built on top.
This is part of the Engineered Intelligence digital garden, a collection of connected pieces I revisit as I learn more.
Stage: Growing
For this piece: A working map. The recovery replay hasn’t run yet, so the foundation layer is still a claim I’m testing.
Last updated: 11/15/26
How it builds
Read from the bottom up. Each layer adds a capability.
Improve
Verify
Run with boundaries
Foundation
One case ID connects the trace, saved checkpoint, and independent verdict.
What each part owns
| Responsibility | What it does |
|---|---|
| Observability | Send OpenTelemetry traces to a receiver over OTLP. Show what triggered the work, which checks ran, what blocked progress, and where to find the evidence. Ellis · Lloyd |
| Resumability | Save the last verified candidate, any unresolved action, and the next safe step outside the disposable worker. Ellis · Yegge |
| Work definition | Version the desired outcome, acceptance criteria, and behavior that must survive the change. Ellis · Horthy |
| Execution boundaries | Declare each worker’s files, tools, permissions, and budget. Keep the evaluator’s expected answers outside its workspace. Yegge |
| Verification | Show fail → fix → pass through the real interface. Verify the candidate independently. Keep later changes manageable and test that across successive steps. Tan · Horthy |
| Improvement and scale | Replay the same cases, varying one thing at a time. Test on-demand history before loading it by default. Add workers when verified output justifies cost and review time. Lloyd · Martin · Yegge |
Layered layout inspired by Zach Lloyd’s Factory Stack. This stack reflects the Docker replay research.
Where a team should start
Tan’s smallest loop is write code, run the code, find the issues, write better code. Agents can already write code. In most setups a person still runs it, so it isn’t a loop yet. The first thing to get working is the agent running the app the way a real user would, with traces and performance numbers, so it can check its own work.
She crossed “good” off “write good code” on purpose. The loop gets you code that does what you asked. It might still be slow or wasteful. Getting agents to write good code by default is the engineer’s job. Once the inner loop works, an alert can start it. On the whiteboard, a Datadog alert has an agent reproduce the issue on its own machine and run the same loop.
Inner loop Write code → run the code → find the issues → write better code, and repeat
The missing piece A person usually runs the code. Teach the agent that step first.
Outer trigger Datadog alert → reproduce the issue → enter the same inner loop
Before you connect Datadog, prove the loop on demand. Pick a known failure. The worker needs to reproduce it through the real product path, make a bounded repair, and show the same check failing before the change and passing after it. Before I’d let alerts start that workflow, I’d also check that the OpenTelemetry trace shows both results and that a fresh worker can resume from a saved checkpoint. A named person owns unclear outcomes and approvals for changes that are hard to reverse.
The Summit pyramid explains where to invest so that the inner loop produces work you can trust. Spend effort first on the codebase the agent will copy and the checks that make mistakes cheap to catch. Then give the worker the tools to verify its repair.
Applying it to a team
Take a recurring support ticket where CSV export sometimes ignores the filter. Keep the scope small enough that one person can explain what passed.
- Give the job an owner and an outcome. Name the person who can settle an ambiguity. Write down which rows the export should include and which existing behavior it must preserve.
- Fix the path agents will copy. Make the CLI and export share the filtering logic. Remove the bypass and add a regression check that fails when a filtered export includes an extra row.
- Let a worker prove the repair. Give it the command or product path it needs. Ask it to reproduce the failure, make the change, and show the same check passing. A separate evaluator checks the exact candidate.
- Watch it run and stop it once. Run it by hand and find the failure and passing rerun in OpenTelemetry. On another run, stop the worker after a verified step and see whether a fresh one can continue from that checkpoint. If it is waiting on an uncertain effect or decision, the case record should show who can resolve it.
- Connect the alert after the on-demand run passes. Have Datadog supply the information the worker needs to reproduce the failure, then start the same bounded workflow. Compare accepted results and review time before adding another worker.
Lightcone, Tan on the smallest loop at 40:22 · Tan’s full Summit talk
Where the sources disagree
When can an agent merge its own fix?
Lauren Tan lets some Grok Bot fixes merge once the worker can reproduce the problem in the app, show the fix working, and pass the codebase’s checks. Infrastructure and deployment still warrant a person.
Dex Horthy shut down his lights-off factory after about four months. The code got harder to maintain even though the system could keep shipping. He still reads his overnight cleanup PRs before merging. A test can tell you the fix works today. Dex wants to know what the third change will be like for the next person.
Tan’s Summit example · Tan on higher-risk changes · Horthy’s shutdown and overnight reviews · Horthy on successive changes
My call. Start with a reviewer. Try auto-merge on one reversible class of fixes once an independent check passes on the exact candidate and a three-change replay shows the code stays easy to change.
How much should projects share?
Zach Lloyd puts workflow definitions under version control, records runs centrally, and replays real tasks to compare changes. A shared platform makes those comparisons easier to run and inspect.
Steve Yegge moved from Gas Town to a factory built around one game. He calls the merge queue and CI the mechanical layer. His Wyvern players asked for a roadmap and a slower pace. The tooling could keep producing changes, but someone had to decide which ones the community could use.
Lloyd on versioned workflows and replay · Yegge on the pace players could absorb · Yegge on the two layers
My call. Share OpenTelemetry traces, checkpoint recovery, and the replay runner. Let each project set its own work, permissions, acceptance checks, and release pace.
What should a new worker remember?
Yegge gives a fresh worker access to its predecessors’ notes and the current policy so it can pick up an established role. He also describes a night when accumulated rules left his workers idle despite a backlog.
Lance Martin showed a different failure in a controlled test. A wrong note persisted across sessions and kept steering Claude back to the wrong answer. Letting Claude revisit and correct the note recovered performance in that test. The choice is how much history to put in front of a worker before it checks the present case.
Yegge on predecessor notes · Yegge on rules blocking work · Martin’s stale-memory experiment
My call. Start a replacement worker with the last verified checkpoint and current instructions. Make older notes available on demand. Replay one case where history helps and one where it’s wrong before loading more by default.
First recovery replay
Stop the ticket-filter replay after its second verified change. Restore the saved candidate in a fresh worker and finish the CSV export. Check the final result independently and find both linked attempts in the OpenTelemetry receiver.
Where it stands
Docker command isolation, failure propagation, and timeout cleanup passed. Live trace delivery and recovery still need a run.