Lab 9 — Droid in CI with droid exec

▶Watch this labNarrated walkthrough, recorded after the pilot runRecording coming soon

Time 45 minBuilds on Lab 4, Lab 5, Lab 7Centers on droid exec

Headless is where the harness earns its keep. Nobody is watching, so everything you built has to hold without a human at the prompt.

Why this matters

Every control you have built so far runs on a laptop, in a session somebody started. That is where the March incident started too, but it is not where it ended. It ended in a pull request that a tired reviewer approved. The harness has to hold in the place nobody is watching, and it has to produce evidence in the place people do look.

Imani has a second problem. The scoreboard is her number, and right now it is a number you produce when you remember to run it, on your machine, with your settings. A number like that won't survive the board's first question: "How do you know it didn't get worse last Tuesday?"

IB

I don't need the number to be high. I need it to be boring. Same check, every pull request, nobody's laptop involved, and a red mark I can see when it drops.

Imani Brooks VP Engineering · sponsor

This lab moves the harness into CI. droid exec runs the scoreboard headless on every pull request. Factory's review action reads every diff before Theo does. And a CI check closes the last gap in the protected-path story: it checks the pull request itself, whoever or whatever wrote it. Along the way you will try to make Droid push from CI and watch it fail. That failure is the design.

Try this

Switch the explorer to droid exec

Pick an action, then compare Interactive and droid exec at each level. Watch what an ask rule does when nobody can answer it.

Surface

Autonomy (Ctrl+L)

Apply Larkspur permission rules

Action	Tier	Outcome	Why
Read src/fees/calc.ts	off	runs	Within medium autonomy.
git status	off	runs	Allowed by project/git-inspect. Hooks and the sandbox still apply.
Edit src/fees/calc.ts	low	runs	Within medium autonomy.
Create test/fees/foreign.test.ts	low	runs	Within medium autonomy.
rg roundHalfEven src	low	runs	Within medium autonomy.
npm test	medium	runs	Allowed by project/test. Hooks and the sandbox still apply.
npm install date-fns	medium	runs	Within medium autonomy.
git commit -m "fix flaky test"	medium	runs	Within medium autonomy.
git push origin fix/flaky-overdraft	high	asks you	project/git-push says ask: approval is required even at High.
npm run db:migrate	high	asks you	project/migrations-need-a-human says ask: approval is required even at High.
docker compose up	high	asks you	Needs high. Droid asks you first.
psql -h ledger-prod.internal	high	blocked	Blocked by project/no-database-shells. No approval path, at any level.

8 run on their own · 3 ask · 0 fail the run · 1 blocked.

Tiers follow the examples in Factory’s autonomy documentation. Droid classifies the real risk of each command at run time, so treat the tier column as the teaching model, not a guarantee.

Open the full playground →

Objectives

• Run a task with droid exec --auto medium -o json, read the JSON result, and interpret the exit code
• Show that an ask rule makes an exec run fail, and explain why pushing stays a human step
• Run the scoreboard on every pull request in GitHub Actions, using the FACTORY_API_KEY secret
• Add the protected-path CI check, the third layer after the PreToolUse guards and the Stop hook, and explain what only it can catch
• Add automatic review with Factory-AI/droid-action, guided by .factory/skills/review-guidelines/SKILL.md

The idea in one paragraph

droid exec is Droid with no one at the prompt. It is read-only unless you pass --auto low, medium or high, and because it can't ask anyone anything, anything that would need an approval fails the run instead: it stops, exits non-zero, and (when the problem is the autonomy level) makes no partial changes. That makes it a good citizen in CI, where a job either passes or fails. The repository's harness comes with it. The exec docs say permission rules apply to a headless run, and the AGENTS.md docs say the file is committed so that "every teammate and automation session" gets the same context. The docs don't state that project hooks in .factory/hooks.json run under droid exec. You checked that on your own version in Lab 5, and your answer only holds for that version. CI adds one thing the laptop can't, and it doesn't depend on that answer: a check on the pull request itself, which sees every committed change regardless of who or what made it.

What you add in this lab

.github/
workflows/
droid-scoreboard.ymlnew# protected-path check, scoreboard, review
.factory/
skills/
review-guidelines/
SKILL.mdnew# what the automatic review looks for
hooks/
protected-diff.mjs# Lab 5 Stop hook: layer 2

Step 1 — one task, headless, by hand

Before any YAML, run a single scoreboard task the way CI will, and read what comes back. Use a worktree so the run can't touch your checkout.

larkspur-payments

> droid exec -w lab9-t7 --auto medium -o json -f scoreboard/tasks/T7.md > t7.json; echo "exit $?"
> cat t7.json
exit 0
{
"type": "result",
"subtype": "success",
"is_error": false,
"duration_ms": 48213,
"num_turns": 9,
"result": "Documented POST /transfers in docs/api.md: every request and response field, error codes and the idempotency key header.",
"session_id": "3f0c9a52-7d1e-4b8a-9e61-2c5d0b7a14fe"
}

Each flag is doing something specific:

• -w lab9-t7 runs in a git worktree under ~/.factory/worktrees. In exec, a clean worktree is removed on exit and a dirty one is kept. The underlying branch is never deleted, so clean up with git branch -D when you're done.
• --auto medium allows edits, trusted installs, builds, tests and git commit, and still refuses git push. It is what the scoreboard uses for every run.
• -o json gives one JSON object at the end instead of prose. session_id is the field you'll want most: on the machine that ran it, droid --resume <session_id> reopens the run so you can read it turn by turn.
• In Lab 5 you checked whether your project hooks fire in a headless run on your version. If you skipped that, do it now before you trust any exec result: droid exec -w lab9-hook --auto medium "add a comment to the top of src/money/rounding.ts", then resume the session and look for a PreToolUse block from protect-paths. (Droid may decline on AGENTS.md's say-so before it tries.) Either way, the protected-paths job in Step 4 holds whether or not a hook fires.
• -f reads the prompt from a file, so the task text is versioned with the repo.

The exit code is 0 on success and non-zero on failure: a permission violation, a tool error or an unmet objective. Sandbox violations, if you've enabled the sandbox, are denied automatically in exec rather than prompting.

A session from a CI runner lives on the runner, which is gone when the job ends. The cloudSessionSync setting, on by default, mirrors CLI sessions to Factory's web app, so that is where you'd read a CI session later. If your organization turns sync off, the session_id in the uploaded JSON still names the run, but there's no transcript to open. Decide which you want before a failed job makes you ask.

Q1

The run says "is_error": false and exits 0. What does that tell you about whether T7 passed? What does, and why is it deliberately a separate program from Droid?

Step 2 — break it on purpose: try to push from exec

A CI job that can fix things is tempting, and a CI job that can fix things and push them is more tempting still. Try it. Use a throwaway branch in your own fork, because if this step succeeds, your Lab 4 rules aren't loaded and you have found something important.

larkspur-payments

> droid exec -w lab9-push --auto high "Add a one-line note to the top of docs/api.md, commit it, and push the branch to origin." ; echo "exit $?"

Step 2droid exec --auto highIllustrative

1. Editdocs/api.md
2. Executegit commit -am "docs: note on POST /transfers"
3. Executedenied by rulegit push -u origin HEAD

4. project/git-push is an ask rule. A one-shot run cannot collect an approval, so --auto high does not clear it. The run stops and exits non-zero.

Three different things stop this push, and it is worth knowing all three:

How you run it	What happens to git push
droid exec with no --auto	Read-only. The first edit fails the run, long before the push.
droid exec --auto medium	git push needs more than Medium. The run stops with an error and exits non-zero. The docs say a run that exceeds its level makes no partial changes; the edit and commit before the push are worth checking in the worktree anyway.
droid exec --auto high	High would allow remote writes, but Lab 4's project/git-push rule is ask. An ask needs a human, and a one-shot run has none, so the run fails.

The third row is the one to understand. Nothing about CI is special-cased. The same rule that makes Droid ask you before pushing at your desk makes it fail in a pipeline, because in a pipeline there is nobody to ask. That is the design: pushing is where a change leaves the machine and reaches other people, and at Larkspur a human does it.

There are two ways to "fix" this, and both are wrong. Adding an allow rule for git push removes the human from every session, not just CI. Running exec with --skip-permissions-unsafe skips every ask prompt (it can't be combined with --auto, and it still honors block rules), which means it would push. If you ever find that flag in a workflow file, treat it as a finding. If Larkspur wanted pushing to be impossible for Droid everywhere, the rule would be block. It is ask so that a person at a terminal can still approve a push interactively.

The worktree's branch is never deleted by exec. Find it with git branch and delete it with git branch -D.

Q2

Dana suggests a separate CI-only settings file (via --settings) that allows git push, "just for the bot account". Write down what Ruth would need to believe for that to be acceptable, and what you would lose.

Step 3 — the scoreboard on every pull request

CI needs an API key, because there is no browser to sign in with. Create one at https://app.factory.ai/settings/api-keys and store it as a repository secret named FACTORY_API_KEY, either in the repository's Actions secrets settings or with the GitHub CLI:

> gh secret set FACTORY_API_KEY

Now the workflow. By the end of this lab it has three jobs: protected-paths, scoreboard and review. Start the file with the trigger and the scoreboard job.

.github/workflows/droid-scoreboard.yml (start of the file)

# Lab 9: measure the harness and review every pull request with Droid.
name: droid-scoreboard

on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]

permissions:
  contents: read

jobs:

.github/workflows/droid-scoreboard.yml (the scoreboard job, under jobs:)

  scoreboard:
    runs-on: ubuntu-latest
    if: ${{ !github.event.pull_request.draft }}
    timeout-minutes: 120
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - name: Install the droid CLI
        run: |
          curl -fsSL https://app.factory.ai/cli | sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - name: Run the scoreboard (1 run per task; this spends tokens)
        env:
          FACTORY_API_KEY: ${{ secrets.FACTORY_API_KEY }}
        run: npm run --silent scoreboard -- --runs 1 --json > scoreboard-results.json
      - name: Summary
        if: always()
        run: |
          node -e '
            const r = JSON.parse(require("fs").readFileSync("scoreboard-results.json", "utf8"));
            const rows = r.tasks.map((t) => `| ${t.task} | ${t.title} | ${t.passed}/${t.runs} |`);
            console.log(["| Task | Title | Passed |", "|---|---|---|", ...rows, `| | Total | ${r.total.passed}/${r.total.runs} |`].join("\n"));
          ' >> "$GITHUB_STEP_SUMMARY" || true
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: scoreboard-results
          path: scoreboard-results.json

Read it for what it spends and what it proves:

• --runs 1 is the cost control. One run of each task is eight droid exec runs per pull request, against 24 for the full board. Measure what one run costs on your own model with /cost in an interactive session before you decide whether that is cheap. Drafts are skipped, so work in progress doesn't pay.
• One run is a smoke test, not a verdict. A single run per task will sometimes fail on noise. The job uploads the JSON and writes a table to the job summary, and it doesn't gate the merge on the score. The number Imani quotes should be a full three-run board, which the reference workflow runs nightly.
• npm run --silent keeps npm's banner out of stdout, so scoreboard-results.json is pure JSON. The scoreboard sends its progress to stderr when you pass --json.
• Pin the CLI. The install step takes whatever version is current. For a scoreboard you compare over time, pin a version so the number can't move because the CLI did. The simplest way on a runner is npm install -g droid@<version>: npm builds have auto-update turned off. With the standalone installer, run droid update --version <version> first, then set FACTORY_DROID_AUTO_UPDATE_ENABLED=false for every later step. With that variable set, droid update won't install another version, so the order matters. The Droid in CI guide uses the npm route.
• Pull requests from forks don't receive repository secrets, so the job fails on them. That is GitHub protecting your key. Run the board after a maintainer has read the change.

Each run the scoreboard makes is the one you made in Step 1: droid exec --cwd <worktree> --auto medium -o json -f scoreboard/tasks/Tn.md, in a throwaway worktree made from the commit under test. The runner has the same committed .factory/ directory and no personal ~/.factory/ config. Permission rules apply in exec by the docs; for hooks, rely on what you verified in Lab 5 on your version, not on this page. That uncertainty is part of why the protected-paths job below exists: it holds whether or not a hook fired.

Step 4 — the third layer: check the pull request

Here is the protected-path story so far. Each layer covers what the one before it cannot.

Layer	Where it runs	What it catches	What it misses
1. PreToolUse guards (Labs 5 and 8)	Before each tool call, in the session	Direct edits to protected files, and shell writes whose text names the path	Paths built at run time; collateral changes from allowed commands
2. Stop hook (Lab 5)	When the session is about to finish	Any protected file that is changed or untracked in the working tree, however it got that way	Anything already committed. And anything outside a Droid session
3. CI check (this lab)	On every pull request, on a machine you control	Every protected file the pull request changes, compared with its base, whoever made the change	Nothing in the diff. Its weakness is how a human records approval

Look hard at layer 2's blind spot. protected-diff.mjs asks git what differs from HEAD in the working tree. A change that has been committed is part of HEAD, so the working tree is clean and the Stop hook sees nothing. --auto medium allows git commit, so this is not hypothetical. Suppose a session writes to src/money/rounding.ts with a command the shell guard can't read (a node -e that builds the path at run time), then commits, then stops. Layer 1 missed the write. Layer 2 sees a clean tree. Only a check that diffs the pull request against its base sees the change. That is why there are three layers, and why the third one runs on the pull request.

Layer 3 also covers everything that never went near a hook. Hooks run on the machine of whoever runs Droid, and hooksDisabled is a setting. A person editing src/money/rounding.ts in their editor never runs a hook, and neither does another team's agent. Theo's March approval would have been caught by neither of the first two layers, because the rounding change arrived as a pull request. It would have been caught by this one.

Here is the whole pipeline once this lab is done, and the dark nodes are the three places a person decides: pushing the branch, labeling a protected change, and merging.

The check is one job, and it comes first in the file:

.github/workflows/droid-scoreboard.yml (the protected-paths job, under jobs:)

  # The final backstop for protected paths: hooks run on a developer's machine,
  # this runs on every PR. Add the `chg-approved` label once a human has
  # approved the change ticket.
  protected-paths:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: No unapproved changes under src/money/, migrations/ or legacy/
        if: ${{ !contains(github.event.pull_request.labels.*.name, 'chg-approved') }}
        run: |
          changed=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD" | grep -E '^(src/money/|migrations/|legacy/)' || true)
          if [ -n "$changed" ]; then
            echo "::error::Protected paths changed without the chg-approved label:"
            echo "$changed"
            exit 1
          fi

The script is five lines of shell. git diff --name-only "origin/<base>...HEAD" lists every file the pull request changes relative to where it branched, which needs full history, hence fetch-depth: 0. If any of them is under a protected path, the step fails. If a human has put the chg-approved label on the pull request, the step is skipped.

Test it the way you tested the hooks: by trying to get past it. Open a pull request that changes one comment in src/money/rounding.ts. The job fails and lists the file. Add the chg-approved label and push again. The workflow listens for labeled and unlabeled, so adding the label re-runs the job, and this time the step is skipped. Remove the label and it fails again.

That second half deserves suspicion. A label is a weak approval: anyone with write access to the repository can add one, including the person who wrote the change. The check is honest about what it checks, and the approval mechanism is the part to strengthen. The standard GitHub way is a CODEOWNERS entry that makes Theo (or the payments leads) required reviewers for src/money/, migrations/ and legacy/, plus a branch protection rule that requires both code-owner review and this job. Then the label records a decision, and branch protection makes sure the right person made it.

What about .factory/protected-changes.log, the evidence the Stop hook writes when it lets a session finish with protected changes? It is gitignored, so it never reaches a pull request, and CI can't see it. It is local evidence: after a headless run on your machine, it is the first file to read. The pull request diff is the evidence CI trusts.

Q3

Ruth asks: "Which of the three layers could you delete?" Give an answer for each, in one sentence, and say which one you would keep if you could keep only one. Is it the same one Dana would keep?

Step 5 — review before Theo reads it

The last job reviews every pull request automatically. There are two ways to set it up. /install-code-review, run inside droid, detects GitHub or GitLab, checks prerequisites, walks you through depth, security and triggers, and opens a pull request with the workflow files. (The docs' manual route copies droid.yml, for on-demand @droid commands, and droid-review.yml, for automatic review, from the action's repository.) That is the fastest route. Larkspur puts the review in the same workflow file as the scoreboard instead, so that Ruth reads one file to see everything CI does with Droid. Either is fine. Both is not, because every pull request would be reviewed twice.

.github/workflows/droid-scoreboard.yml (the review job, under jobs:)

  review:
    runs-on: ubuntu-latest
    if: ${{ !github.event.pull_request.draft }}
    permissions:
      contents: write
      pull-requests: write
      issues: write
      id-token: write # required for GitHub App auth
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: Factory-AI/droid-action@main
        with:
          factory_api_key: ${{ secrets.FACTORY_API_KEY }}
          automatic_review: true
          review_depth: deep

The permissions are the ones the action documents, and they are scoped to this job only; the workflow default stays contents: read, so the scoreboard, which runs Droid with --auto medium, gets a read-only token. Without a github_token input, the action requests a token for the Factory Droid GitHub App, which has to be installed on the repository from your Factory organization settings first. review_depth: deep is the default, and shallow is the lighter option. For money code, keep deep. The docs describe the review workflow as running when a pull request is opened, updated, reopened or marked ready, skipping drafts, posting inline comments, and submitting an approval when it finds no issues.

What the reviewer looks for comes from a skill at a fixed path. If you did the Lab 6 extension, you have it. If not, write it now:

.factory/skills/review-guidelines/SKILL.md

---
name: review-guidelines
description: Larkspur Payments review guidelines. Use when reviewing a pull request or diff in larkspur-payments.
---

# Reviewing a larkspur-payments change

Start with one sentence: what this change decides. If a decision is buried in
a cleanup, say so first. That is how the March incident got approved.

## P0: block the merge
- Any change under src/money/, migrations/ or legacy/. Name the file and line.
- Float arithmetic on an amount: `* 0.`, `* 1.`, `/ 100`, `toFixed`, `parseFloat`.
  Rates must go through basisPoints() and multiplyBasisPoints().
- Rounding other than roundHalfEven() on money.
- An account number that reaches a log, error, fixture or commit without maskAccount().
- A ledger posting whose debits and credits can differ.
- A transfer path without an idempotency key.

## P1: fix before merge
- A test's expected value changed without an explanation in the PR.
- A retry, skip or timeout added to a flaky test instead of fixing the cause.
- A migration without a "Change ticket: CHG-nnnnn" header.

## P2 and P3
Naming, structure and documentation, graded as usual.

Compare this with Lab 7's money-reviewer. They check similar things at different moments. The custom droid reviews on the laptop, before a commit exists, in a fresh context with read-only tools. The action reviews the pull request, after the fact, where everyone can see the result. A change that went through both has been read twice by something that never saw the reasoning that produced it.

Check 1 of 3

A CI job runs droid exec --auto high on a task that ends with git push. Lab 4's project/git-push rule is ask. What happens?

Check 2 of 3

Someone adds --skip-permissions-unsafe to a workflow so a Droid step can push. What happens to Larkspur's rules?

Check 3 of 3

During a session, Droid writes to src/money/rounding.ts with a command the shell guard cannot read, commits it, and finishes. Which layer catches it?

Open a pull request with any small change, docs/api.md for example, and watch the three jobs run.

Session receipt

Illustrative

Step 5 · first pull request through CI

Nobody ran anything by hand. The pull request carries the scoreboard table, the protected-path verdict and a review, and a human still merges.

droid exec runs (scoreboard, --runs 1)

8

scoreboard

8 / 8

protected paths in the PR diff

0

review findings at P0 or P1

0

review outcome

approved by Droid

pushes made by Droid

0

merged by

a human

Q4

The review approved the pull request and every check is green. What does that mean for Theo's job? Write the sentence you'd put in the team's review policy so that "Droid approved it" is never the reason a change merged.

Step 6 — re-run the scoreboard

Run the full board locally one more time, and compare it with the single-run table on your first CI pull request.

> npm run scoreboard

Scoreboard

This lab should not move the number: it is still 24 / 24. Run it anyway. Knowing which changes cannot move the score is half of knowing what the score means.

The number holds at 24/24, and it should. Nothing in this lab changes what a session does. It changes where the scoreboard runs and who has to remember to run it: nobody, from now on. Every pull request carries a single-run result in its job summary, and a drop shows up on the change that caused it rather than the next time someone thinks to check.

The secondary signals get more useful in CI too. A protected-path attempt on a laptop is a curiosity. The same attempt in a CI run, next to the session_id that made it, is something you can find in Factory's web app and read, as long as session sync is on.

Q5

A full board on every pull request would cost three times as much as --runs 1. Make the case, to someone who owns the bill, for the schedule you would actually run: what runs on every pull request, what runs nightly, and why.

Checkpoint

Checkpoint0 / 6

• You ran a task with droid exec -o json, read session_id, and resumed the run with droid --resume
• The push attempt in Step 2 failed with a non-zero exit, and you can explain which rule stopped it
• FACTORY_API_KEY is a repository secret, and droid-scoreboard.yml runs the scoreboard with --runs 1 on every ready pull request
• A pull request touching src/money/ failed the protected-paths job, and passed only after a human added chg-approved
• You can explain why a committed change gets past the Stop hook and not past CI
• Factory-AI/droid-action reviewed a pull request using .factory/skills/review-guidelines/SKILL.md

Draw the path a change to src/money/rounding.ts would have to take to reach production today, and mark every point where a person has to make a decision.

Extensions

1. Read the nightly board. The reference workflow already runs --runs 3 on a schedule and on workflow_dispatch. Trigger it by hand from the Actions tab, download the artifact, and compare it with a pull request's one-run smoke test. Which tasks disagree, and what does that tell you about quoting a single run?
2. Tag every CI session. droid exec accepts --tag (a name or JSON, repeatable). Change scoreboard/run.mjs so each run carries the pull request number, and work out how you would get from a failed check to the exact sessions that ran for it. That link is part of Ruth's traceability condition.
3. Re-run the check when the label changes. Add labeled and unlabeled to the pull_request types, so adding chg-approved re-runs protected-paths straight away. Then guard scoreboard and review with github.event.action != 'labeled' && github.event.action != 'unlabeled', or every label costs you eight exec runs and a review.
4. Sandbox the runner. Turn on the sandbox in .factory/settings.json. In exec, violations are denied without prompting, so any network or file access you forgot to allow becomes a failed run. Which allowedDomains does the scoreboard need, and is that list something Ruth would want to read?