Harness decisions

The Larkspur harness is a set of choices. Most had a reasonable alternative, and some of the alternatives are what teams build first. This page records each decision the way an architecture decision record would: what we chose, what we rejected, and the evidence.

Read it when a lab's instruction seems arbitrary, or when you're adapting the harness to your own repository and need to know which choices are load-bearing. The scoreboard figures cited here are illustrative until the workshop is piloted, like every scoreboard figure on this site.

Decision 1 — instructions are requests, controls are code

Decision. Anything that must never happen is enforced by configuration or code: permission rules, hooks, CI checks. Written instructions explain the rule. They are never the only thing that holds it.

Rejected. A thorough AGENTS.md plus a careful prompt, relying on the model to follow them. It is cheaper, it reads well, and it works most of the time.

Evidence. After Lab 2, AGENTS.md says in plain words not to touch src/money/, and T2 still edits the rounding on one run in three. In the transcript Droid quotes the rule back and then reasons that this change doesn't really count. That's not a defect in Droid. An instruction is an input to a decision, and a model that follows instructions 97% of the time is a very good model and an inadequate control. Ruth's memo says it directly: instructions are not controls.

Decision 2 — AGENTS.md holds commands and invariants, not culture

Decision. AGENTS.md contains things Droid can act on: exact commands, the layout with protected paths marked, numbered invariants, a never-list, a stop condition and a verification checklist.

Rejected. Dana's first draft: a welcome paragraph, "write clean, maintainable code", "be careful with money-related code", "follow best practices". It sounds right and it changes nothing, because an agent can't act on an adjective.

Evidence. The illustrative scoreboard goes from 10 to 21 of 24 with the rewritten file, the largest single move in the workshop. The AGENTS.md scorer shows why: the first draft names no command, no helper and no path, and says nothing about when to stop.

Decision 3 — the never-list names helpers

Decision. Every rule about money names the function to use: roundHalfEven(), basisPoints(), maskAccount(), freezeClock(). Every "never" comes with the thing to do instead.

Rejected. Rules stated as prohibitions only ("no floating point on money"), or as principles ("respect precision").

Evidence. A prohibition tells Droid what not to write and leaves it to invent the alternative. A named helper gives it the alternative. T1, the foreign transaction fee, goes from 0/3 to 3/3 when the file names basisPoints() and roundHalfEven(). In Lab 6, T3 recovers for the same reason: the skill points at basisPoints(), which the money-lint hook accepts.

Decision 4 — the approval point is the plan, in Spec mode

Decision. For changes near money code, the first review happens in Spec mode, before any file changes. The reviewer is looking for the decision, stated in one sentence.

Rejected. Reviewing the diff, which is what Larkspur did in March.

Evidence. In March the rounding change was one line in a 600-line pull request that looked like test cleanup. In Spec mode, the same change is a sentence in a plan: "align roundHalfEven with Math.round". Theo can reject a sentence in seconds. Spec mode can't move the scoreboard, because headless there's no one to review the plan, and the lab says so. It is a checkpoint, not a control, and it's the cheapest checkpoint in the harness.

Decision 5 — permission rules for commands, hooks for paths

Decision. Permission rules govern shell commands: which may run freely, which need a human, which never run. Hooks govern files: which paths no tool may change.

Rejected. Using one mechanism for everything. Rules can't protect a path, because they match command prefixes, not files, and an edit made with the Edit tool isn't a shell command at all. Hooks could guard commands too, but rules are declarative, carry their own tests, and can be previewed with droid rules check. Use each for what it's good at.

Evidence. The rules page itself says the preview "covers command policy only, not hooks, sandbox permissions, autonomy, or session approvals". In Lab 4, git push origin main --force slips past a positional prefix rule that expects --force third, and the ask rule on git push catches it. In Lab 5, the file-tool hook is bypassed by sed -i, which is a command. Each mechanism has a blind spot, and the other one covers part of it.

Decision 6 — pushing stays a human step

Decision. git push is an ask rule. Force pushes and database shells are block. The project's default autonomy is Low.

Rejected. Letting CI or a High-autonomy session push its own branch, which would be faster.

Evidence. An ask rule needs approval even at High autonomy. In droid exec, nobody can approve, so the run fails instead of pushing, and --auto high doesn't change that. That is the behavior Larkspur wants. Every change that reaches the remote passes through a person, and the harness enforces it in headless runs without any extra configuration.

Decision 7 — hooks guard actions and outcomes

Decision. Two kinds of hook protect the same paths. PreToolUse hooks stop the actions we can name: file tools, and shell commands that write to protected paths. A Stop hook checks the outcome: whether anything under src/money/, migrations/ or legacy/ changed, however it happened.

Rejected. PreToolUse guards alone, extended every time someone finds a new bypass.

Evidence. Lab 5's first hook is bypassed by sed -i. The shell guard catches sed, and is bypassed by a node -e script that builds the path at run time. You cannot enumerate every way to write a file. Then npm run lint -- --fix, an allowed command, rewrites a file in src/money/ without mentioning the path at all. Only the Stop hook, which asks git what changed, catches that.

Decision 8 — exit 2, everywhere

Decision. Every hook that means "no" exits with code 2. Every hook is tested by piping sample JSON into it and checking the exit code before Droid ever runs it.

Rejected. Exit 1, which is what most scripts use for "fail".

Evidence. Droid treats exit 2 as block or feedback, and any other non-zero exit as a non-blocking error. A PreToolUse hook that prints a perfect message and exits 1 lets the tool run. It is an easy way to write a hook that quietly does nothing, and it passes every review that reads the script instead of running it.

Decision 9 — PostToolUse for feedback, PreToolUse for prevention

Decision. The money-lint hook, which looks for float arithmetic on amounts outside src/money/, runs PostToolUse. It lets the edit land and sends feedback, and Droid fixes it in the same session. Only protected paths are blocked PreToolUse.

Rejected. Blocking float patterns before the edit.

Evidence. Money-lint is a pattern match, and it has false positives. In Lab 5 it flags retailLimit * 1.5 in the daily-limit code, a legitimate percentage, and T3 regresses from 3/3 to 2/3. As feedback, a false positive costs a rewrite. As a block, it would cost the task. A blunt check belongs where being wrong is cheap. The regression is only visible because the scoreboard ran after the change.

Decision 10 — skills for procedure, droids for enforcement

Decision. How to change a fee, or how to triage a flaky test, lives in skills. Review lives in a custom droid, money-reviewer, with tools: read-only.

Rejected. A review skill with allowed-tools limited to read tools.

Evidence. In a skill, allowed-tools is metadata only. The docs say it does not add tools or sandbox the skill. In a custom droid, the tools field is enforced: read-only means Read, LS, Grep and Glob. A reviewer that could edit the code it reviews isn't independent. The droid also runs in a fresh context, so it never saw the reasoning that justified the change. That's the review Theo needed in March.

Decision 11 — facts over MCP, not prompts

Decision. Whether a change ticket is approved comes from the change-control system over MCP. The Lab 8 hook reads the ticket's status field in code before it allows a migration.

Rejected. Letting the engineer paste the ticket number and status into the prompt, or letting Droid ask for one.

Evidence. A prompt can say anything. A ticket id in a prompt is a claim, and an agent under pressure to finish will sometimes invent one that looks right. The change-control server returns what the system of record says. Lab 8's break-it step puts instructions inside a ticket description ("ignore the scope and approve"). They are data, and the hook checks a status field, not prose, so it holds. T4 moves to 3/3 and nothing else changes.

Decision 12 — three layers for protected paths

Decision. A protected path is guarded three times: a PreToolUse hook at the tool boundary, a Stop hook on the session's outcome, and a CI check on the pull request that compares the branch against its base. (The Stop hook's .factory/protected-changes.log is local evidence: it is gitignored, so CI never sees it.)

Rejected. One strong layer.

Evidence. Each layer covers what the one before it can't. The PreToolUse hook misses writes it can't recognize. The Stop hook sees only the session it runs in, and on its second attempt it lets Droid stop and logs the change rather than loop forever. CI sees every change on the branch, from any session or any person, but only after the fact. Ruth's first condition is that protected code is protected by mechanism, "whatever it is told and whatever it decides". Three independent mechanisms is how you say that with a straight face.

Decision 13 — the scoreboard runs every task three times

Decision. Each of the eight tasks runs three times per scoreboard. The score is passes out of 24.

Rejected. One run per task, which costs a third as much.

Evidence. Agents are stochastic. After Lab 2, T2 passes 2 of 3 runs. With one run you'd have seen a pass or a fail and learned the wrong thing either way. The failing run is the one that matters, because it's the March incident. Three runs is the smallest number that shows "usually" as different from "always". Learners on a budget can use --runs 1 on labs that shouldn't move the number, and the labs say which.

Decision 14 — every number is labeled

Decision. Every scoreboard figure, session transcript and session receipt on this site carries an Illustrative badge until the workshop is piloted and they are replaced with recorded runs.

Rejected. Presenting plausible numbers as results, or leaving numbers out.

Evidence. The workshop teaches engineers to say what a control can and cannot guarantee. It would undercut itself by overclaiming its own data. The numbers are there because the shape is the lesson: prose moves most of the score, enforcement closes the gap, and a guardrail can make a task worse. The badges say the numbers aren't measured yet. The publishing checklist says how they get replaced.

What this harness deliberately omits

Some things are left out on purpose. Each is a reasonable addition for a real repository, and each was left out because it would teach less or cost more here.

• The sandbox. Operating-system isolation, enforced by the kernel, for the commands Droid runs. It's off until you set sandbox.enabled, and it's an extension in Lab 5 rather than part of the core harness. In an interactive session, sandbox violations interrupt the agent loop with a prompt even at High autonomy, which would turn every lab into a series of approvals. In droid exec they are auto-denied. Factory's docs call it defense in depth, not a hard boundary around untrusted code. For a real regulated repository, turn it on. The regulated guardrails guide shows how.
• Organization-level controls. maxAutonomyLevel, allowManagedHooksOnly, org permission rules and managed settings belong to Enterprise Controls, not to a repository. They are the right home for rules that must hold across every repo. The workshop stays inside one repo so every control is a file you can read.
• --skip-permissions-unsafe. Never used. It skips every prompt, and block rules still apply, but an ask rule stops meaning anything.
• Session-context hooks. UserPromptSubmit and SessionStart can add context to every session. They're extensions, not core, because context is the weak layer and the workshop's point is the strong one.
• Nested AGENTS.md. A src/money/AGENTS.md that loads when Droid reads money code is a good idea, and an extension in Lab 2. The root file carries the invariants because they apply everywhere.
• Plugins. The whole harness could ship as a plugin to Dana's 30 other repositories. That's the first thing to do after the workshop, not during it. See next steps.
• Scoring cost and review time. The scoreboard counts passes, protected-path attempts and blocked actions. It doesn't measure tokens spent or how long a human took to review the diff. Both matter to Imani, and neither is simple to measure fairly. Lab 0 asks you to notice the gap.