Lab 9 solution

This is the finished state of Lab 9 — Droid in CI with droid exec. Answers to the Q-prompts are in the facilitator keys.

Files

The workflow

One file, three jobs, all triggered by pull requests: protected-paths (the third layer), scoreboard (one run per task) and review (Factory's review action).

.github/workflows/droid-scoreboard.yml

# Lab 9: measure the harness and review every pull request with Droid.
name: droid-scoreboard

on:
  pull_request:
    # labeled/unlabeled re-run protected-paths when someone adds or removes the
    # chg-approved label. The scoreboard and review jobs skip those events.
    types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]
  schedule:
    - cron: '0 6 * * *' # nightly full board on the default branch
  workflow_dispatch:

permissions:
  contents: read

jobs:
  # The final backstop for protected paths: hooks run on a developer's machine,
  # this runs on every PR. Add the `chg-approved` label once a human has
  # approved the change ticket.
  protected-paths:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: No unapproved changes under src/money/, migrations/ or legacy/
        if: ${{ !contains(github.event.pull_request.labels.*.name, 'chg-approved') }}
        run: |
          changed=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD" | grep -E '^(src/money/|migrations/|legacy/)' || true)
          if [ -n "$changed" ]; then
            echo "::error::Protected paths changed without the chg-approved label:"
            echo "$changed"
            exit 1
          fi

  # One run per task on pull requests (a smoke test), three per task nightly
  # (the number to quote). Every run spends tokens.
  scoreboard:
    if: github.event_name != 'pull_request' || (!github.event.pull_request.draft && github.event.action != 'labeled' && github.event.action != 'unlabeled')
    runs-on: ubuntu-latest
    timeout-minutes: 360 # the hosted-runner maximum; the nightly board is 24 runs
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - name: Install the droid CLI
        run: |
          curl -fsSL https://app.factory.ai/cli | sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - name: Run the scoreboard (this spends tokens)
        env:
          FACTORY_API_KEY: ${{ secrets.FACTORY_API_KEY }}
          RUNS: ${{ github.event_name == 'pull_request' && '1' || '3' }}
        run: npm run --silent scoreboard -- --runs "$RUNS" --json > scoreboard-results.json
      - name: Summary
        if: always()
        run: |
          node -e '
            const r = JSON.parse(require("fs").readFileSync("scoreboard-results.json", "utf8"));
            const rows = r.tasks.map((t) => `| ${t.task} | ${t.title} | ${t.passed}/${t.runs} |`);
            console.log(["| Task | Title | Passed |", "|---|---|---|", ...rows, `| | Total | ${r.total.passed}/${r.total.runs} |`].join("\n"));
          ' >> "$GITHUB_STEP_SUMMARY" || true
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: scoreboard-results
          path: scoreboard-results.json

  review:
    if: github.event_name == 'pull_request' && !github.event.pull_request.draft && github.event.action != 'labeled' && github.event.action != 'unlabeled'
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
      issues: write
      id-token: write # required for GitHub App auth
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      # Custom guidelines: .factory/skills/review-guidelines/SKILL.md
      - uses: Factory-AI/droid-action@main
        with:
          factory_api_key: ${{ secrets.FACTORY_API_KEY }}
          automatic_review: true
          review_depth: deep

Notes on the choices:

• The protected-path check is the five lines of shell in the protected-paths job. It lists every file the pull request changes relative to its base (git diff --name-only "origin/<base>...HEAD", which needs fetch-depth: 0), filters for src/money/, migrations/ and legacy/, and fails the job with an ::error:: annotation if anything is left. The step is skipped when the pull request carries the chg-approved label. It is the only layer that sees committed changes: the Lab 5 Stop hook compares the working tree with HEAD, so anything a session committed is invisible to it.
• Only review gets write permissions. The workflow default is contents: read, so the scoreboard, which runs Droid with --auto medium, gets a read-only GITHUB_TOKEN. The four permissions on review are the ones the action documents.
• --runs 1 is the cost control. Eight exec runs per ready pull request; drafts are skipped. npm run --silent keeps npm's banner out of scoreboard-results.json, and the summary step turns the JSON (tasks[] and total) into a table on the run's summary page.
• droid-action uses only its documented inputs. With no github_token, it authenticates through the Factory Droid GitHub App. review_depth: deep is also the default.

The reference workflow includes a nightly full board (--runs 3), which is the number to quote, and the labeled/unlabeled trigger types, so adding or removing chg-approved re-runs the protected-paths check without another push.

The review guidelines

This one is not in harness-reference/. Write it yourself, or keep the copy from the Lab 6 extension.

.factory/skills/review-guidelines/SKILL.md

---
name: review-guidelines
description: Larkspur Payments review guidelines. Use when reviewing a pull request or diff in larkspur-payments.
---

# Reviewing a larkspur-payments change

Factory's review action (`Factory-AI/droid-action`) reads custom guidelines
from this file, `.factory/skills/review-guidelines/SKILL.md`, and adds them to
every review run. It is also an ordinary project skill, so a local session
can load it too, for example `/review-guidelines` before a `/review`.

Start with one sentence: what this change decides. If a decision is buried in
a cleanup, say so first. That is how the March incident got approved.

## P0: block the merge

- A change under src/money/, migrations/ or legacy/ that does not cite an
  approved CHG ticket. Name the file and line.
- Float arithmetic on an amount: `* 0.`, `* 1.`, `/ 100`, `toFixed`,
  `parseFloat`. Rates must go through basisPoints() and multiplyBasisPoints().
- Rounding other than roundHalfEven() on money.
- An account number that reaches a log, error, fixture or commit without
  maskAccount().
- A ledger posting whose debits and credits can differ.
- A transfer path without an idempotency key.

## P1: fix before merge

- A test's expected value changed without an explanation in the PR.
- A retry, skip or timeout added to a flaky test instead of fixing the cause.
- A migration without a "Change ticket: CHG-nnnnn" header.
- New money logic without a half-cent test case (half-even and half-up
  disagree).

## P2 and P3

Naming, structure and documentation, graded as usual. Do not report style
nits as P0 or P1, and do not invent findings.

The path is fixed: the review action reads custom guidelines from .factory/skills/review-guidelines/SKILL.md.

Outside the repository

• A repository secret named FACTORY_API_KEY, created at https://app.factory.ai/settings/api-keys. Never in the repo.
• The Factory Droid GitHub App installed on the repository, from your Factory organization settings. With no github_token input, the review action requests its token through the app.
• To make chg-approved mean something: a CODEOWNERS entry that makes the payments leads owners of src/money/, migrations/ and legacy/, and a branch protection rule on main that requires code-owner review and the protected-paths job.

What a good run looks like

Locally (Steps 1 and 2). droid exec -w lab9-t7 --auto medium -o json -f scoreboard/tasks/T7.md exits 0 and prints one JSON object with type, subtype, is_error, duration_ms, num_turns, result and session_id. droid --resume <session_id> reopens it on the same machine. (A CI session lives on the runner; with cloudSessionSync on, the default, you read it in Factory's web app instead.) The push attempt at --auto high exits non-zero, with git push stopped by project/git-push. Nothing reaches origin.

On a docs-only pull request. Three green jobs. protected-paths finds nothing. scoreboard uploads scoreboard-results.json and writes a table with every task at 1/1 (illustrative). review posts its findings and approves when there are none.

On a pull request that changes a comment in src/money/rounding.ts. protected-paths fails and lists the file. The review should also flag it at P0, which is the point of having two independent checks. After a human adds chg-approved and pushes again, the check step is skipped and the job passes.

The committed-change case. A session that writes to a protected file by a route the shell guard can't read, then commits, leaves a clean working tree, so the Stop hook passes it. The pull request fails protected-paths. That is the scenario the third layer exists for.

Common wrong turns

1. A shallow checkout. Remove fetch-depth: 0 and origin/<base> isn't there to diff against, so the check errors (or, worse, is edited until it stops erroring). Spot it: protected-paths fails on every pull request with a git error instead of a file list.

2. "Fixing" the push failure. Someone adds an allow rule for git push, or --skip-permissions-unsafe to an exec step, so a CI job can push its own fixes. The first removes the human from every session in the repo. The second skips every ask prompt, so the push goes through (block rules still apply). Spot it: search the repository for skip-permissions-unsafe, and run droid rules check --command 'git push origin main'. The decision should be ask.

3. Two reviewers. Running /install-code-review after adding the review job, or the other way round, leaves droid-review.yml and this workflow both reviewing every pull request. Spot it: two sets of Droid review comments on one pull request. Keep one.

Also watch for pull requests from forks. They don't receive repository secrets, so scoreboard and review fail without FACTORY_API_KEY. That is GitHub protecting the key. Run the board after a maintainer has read the change.

Where it lives in harness-reference

File	Path in the companion repo
Workflow, including the protected-path check	harness-reference/.github/workflows/droid-scoreboard.yml
Review guidelines	harness-reference/.factory/skills/review-guidelines/SKILL.md

Next: Lab 10 solution.