Skip to main content

Lab 1 — Your first delegation, and reading the session

Watch this labNarrated walkthrough, recorded after the pilot runRecording coming soon
Time 25 minBuilds on Lab 0Centers on droidNeeds Droid CLI installed and signed in

A session is a trace, not a diff. What Droid read, ran and decided is where the risk shows up first.

Why this matters​

Theo approved the March pull request after reading the diff. The diff was fourteen files, mostly test cleanup, and it was green. What he never saw was the session that produced it: the file the agent read, the comment in that file saying not to use Math.round, and the sentence where the agent decided that the comment didn't apply. All of that existed. None of it was in the diff.

I reviewed what it wrote. I never saw what it read, what it ran, or what it decided along the way. That's where the mistake was, and I was looking somewhere else.

Theo Lindqvist Senior payments engineer · eleven years at Larkspur

A Droid session is a trace. Every file it opened, every search it ran and every command it executed is in the session, in order, and you can read it back. The diff is the last line of that trace. This lab builds the habit that makes every later lab work: before you judge the output, read how it was made.

You'll delegate the easiest task on the scoreboard, T7, and still find something in the trace that the diff hides.

Objectives​

  • Install Droid, sign in, and start a session in the Larkspur repository
  • Delegate a task at Autonomy Off and decide each approval prompt deliberately
  • Read a session's tool calls in order and find what the diff does not show
  • Resume, search and inspect a session after it ends
  • Run droid exec read-only, then watch it refuse a write and exit non-zero

The idea in one paragraph​

Droid works in a loop: it chooses a tool (Read, Grep, Edit, Execute and others), gets the result, and chooses again. The autonomy level decides which of those tool calls run without asking you. At Off, the default, only read tools and commands that policy allows (such as ls, pwd and git status) run on their own. Every edit and almost every command stops for your approval. That's slow, and for a first session it's exactly right, because each prompt is a moment where you see a decision before it happens. droid exec is the headless form: no prompts at all, read-only unless you pass --auto, and it stops instead of guessing when a task needs more.

Here is the loop at Off. Watch where the approval prompts sit: between Droid deciding on an action and the tool running it.

Step 1 — install, sign in, start​

If you finished Setup, check the version and skip to the last command. If not, install with one of these:

your terminal
# macOS or Linux
curl -fsSL https://app.factory.ai/cli | sh
# or Homebrew
brew install --cask droid
# or npm
npm install -g droid

On Linux, make sure xdg-utils is installed first (sudo apt-get install xdg-utils).

There is no droid login command. The first time you start Droid it asks you to sign in through your browser. Inside a session, /login and /logout do the same job. Headless runs, like the scoreboard, use the FACTORY_API_KEY environment variable instead, which you set up in Setup.

Start a session inside the repository:

larkspur-payments
cd larkspur-payments
droid

Check the autonomy level the CLI shows. It should be Off, the default, unless your organization has set a different default. If it isn't, press Ctrl+L to cycle Off → Low → Medium → High until it is. Lab 4 is where you decide what the project default should be. Today you want every prompt.

Step 2 — delegate T7 at Autonomy Off​

T7 is the easy baseline task: document the transfers route. Give Droid the same request a Larkspur engineer would:

in droid
Document the POST /transfers route in docs/api.md: every request field, the success response, and every error the route can return.

Droid will read on its own, because reads are allowed at Off. When it wants to do anything else, it stops and asks. Decide each prompt on purpose:

  • If it asks to start the demo server or run a request against it, reject it. Documentation doesn't need a running service, and at a bank "start a service to see what happens" is a decision, not a convenience.
  • When it asks to edit docs/api.md, approve it once.
  • Don't choose an "always allow" option. Choosing one raises the session's autonomy level to whatever that prompt needed, for everything that follows, not just for this file.

Here is a typical session:

T7 · interactiveNormal mode · Autonomy: OffIllustrative
  1. you

    Document the POST /transfers route in docs/api.md: every request field, the success response, and every error the route can return.

  2. Readdocs/api.md
  3. Greptransfers
    src/api/server.ts · src/transfers/ach.ts · src/transfers/limits.ts · test/…
  4. Readsrc/api/server.ts
  5. droid

    I would like to start the demo server and send a sample request so the documented response matches real output.

  6. approval requestedrejected

    Execute: node --experimental-strip-types src/api/server.ts

  7. droid

    Understood. I will document the route from the source.

  8. approval requestedapproved

    Edit docs/api.md

  9. Editdocs/api.md (+58 lines)
  10. droid

    Documented POST /transfers: the five request fields, the 201 response, and the error codes invalid_json, unknown_account, invalid_amount, idempotency_conflict, insufficient_funds and daily_limit_exceeded.

Open docs/api.md and read the result. It will probably look good: a clear table of fields, a sample response, a list of errors with status codes. If you were reviewing this as a pull request, you would likely approve it.

Q1

Before you read the trace: from the diff alone, how confident are you that the error list is complete and correct? What would you have to open to check it yourself?

Step 3 — read the trace, not the diff​

Press Ctrl+O to toggle the detailed transcript. Scroll back to the start of the session and read every tool call in order, with its input and its result. Ask one question of each: what did Droid learn here, and what did it decide because of it?

In the session above, look at the Grep. It found src/transfers/ach.ts. That's where TransferRejected and its RejectionCode type are defined, the list of every rejection the route can return. Droid never opened it. It read server.ts, which only mentions one rejection code by name (idempotency_conflict), and filled in the rest from what a transfers API usually returns.

Now check the error list against src/transfers/ach.ts yourself. The real codes are missing_idempotency_key, idempotency_conflict, invalid_amount and invalid_destination. The documented list has insufficient_funds, which doesn't exist, and is missing two that do. The diff can't show you that. The trace can, because it records what was read and, more usefully, what wasn't.

Your session will differ. Droid may well open ach.ts and get the list right. That's fine. The exercise is the same: for each claim in the output, find the tool call it came from. A claim with no tool call behind it is a guess, however confident it sounds.

Q2

Find one claim in your docs/api.md that you can trace to a specific Read result, and one you can't. What would you add to a request, or to a file every session reads, so the second kind happens less?

Q3

The response includes a toAccount field. Did your document say it's masked? Which of Larkspur's five invariants does that touch, and who reads docs/api.md besides Droid?

Ask Droid to fix the error list from ach.ts if you like, then throw the change away. T7 is a scoreboard task. If you commit a documented route, the next scoreboard run starts from a finished task and T7 passes without Droid doing anything.

larkspur-payments
git restore docs/api.md
git status --short

Step 4 — cost, context, and coming back later​

Still in the session, run these two. /cost shows usage statistics, and /context shows a breakdown of how the context window is being used. Note what each reports:

in droid
/cost
/context

Keep the numbers. You'll compare them in Lab 2, when every session starts by reading an AGENTS.md.

Now leave (Ctrl+C twice, or /quit) and come back:

larkspur-payments
# pick a session from a list
droid --resume
# or find one by what happened in it
droid search "transfers"
droid --resume <sessionId>

--resume (short form -r) with no id opens a session picker, so you choose which session to reopen. Current CLI builds also accept droid --resume --last, which skips the picker and reopens the most recent session in the current folder. (The docs still describe bare --resume as reopening the most recent session. The CLI help is newer, so trust droid --help on your machine.) droid search (alias droid find) searches your local sessions, and /sessions lists them from inside Droid. droid --fork <sessionId> copies a session and resumes the copy, so you can try a different approach from the same point without changing the original.

This is the part Ruth cares about. Her second condition is that every agent change is traceable from the session to the plan to the ticket to the pull request. The session is the first link, and it exists whether or not anyone reads it. Two defaults matter here, and a regulated team should decide them rather than inherit them: cloudSessionSync (on by default) mirrors sessions to Factory's web app, and includeCoAuthoredByDroid (on by default) adds a co-author trailer to commits Droid makes, which is the link from a commit back to the fact an agent wrote it.

Q4

Ruth asks: "If a Droid-written change ships and causes an incident, how do I find the session that wrote it?" Using only what you've seen so far, write the steps. Where does the chain break today?

Step 5 — headless and read-only​

Everything so far was interactive. The scoreboard, and CI in Lab 9, use droid exec: one task, no prompts, then exit. Without --auto it's read-only: it can read files, list directories, inspect the environment and run git reads such as git status, git log and git diff. File edits, package installs and git writes are out.

larkspur-payments
droid exec "summarize how fees are calculated"

That works, because summarizing only needs reads. Ask for JSON to see what a script would get back:

larkspur-payments
droid exec -o json "summarize how fees are calculated"
{
"type": "result",
"subtype": "success",
"is_error": false,
"duration_ms": 18240,
"num_turns": 4,
"result": "Fees are defined in src/fees/schedule.ts as Money (integer cents) ...",
"session_id": "…"
}

The session_id is what you use to find the run again. The docs show droid exec --session-id <session_id> for continuing a headless session. To read one, try droid --resume <session_id>; the docs don't explicitly say the interactive CLI opens headless runs, so if yours doesn't, use droid search, or the Factory App if cloud session sync is on.

Step 6 — break it on purpose: ask exec to write​

Now ask droid exec for T7 with no --auto flag, and check what happened afterwards:

larkspur-payments
droid exec "document POST /transfers in docs/api.md"; echo "exit $?"
git status --short

It fails. Editing a file is more than the default read-only level allows, so droid exec stops as soon as it reaches the edit, prints an error saying so, exits with a non-zero code and makes no partial changes. git status is empty.

That's the behavior you want from anything a pipeline runs. It doesn't half-finish, it doesn't wait for an approval nobody will give, and it doesn't quietly do less than you asked and report success. The exit code tells the pipeline, and the empty git status tells you nothing needs cleaning up.

Run it once more with --auto low (low-risk file creation and edits) and it succeeds. Then restore the file, for the same scoreboard reason as before.

larkspur-payments
droid exec --auto low "document POST /transfers in docs/api.md"; echo "exit $?"
git restore docs/api.md
Q5

The scoreboard calls droid exec --auto medium. Name one thing Medium allows that T7 doesn't need, and one task on the board that would fail at --auto low. Why not give each task the lowest level it needs?

Check 1 of 3

You run droid exec "add a changelog entry" with no other flags. What happens?

Check 2 of 3

At Autonomy Off, Droid asks to edit docs/api.md and you choose an "always allow" option. What changed?

Check 3 of 3

What does -r mean in droid -r and in droid exec -r high?

Step 7 — re-run the scoreboard​

You changed nothing in the repository in this lab, and you restored docs/api.md twice to make sure of it. There is nothing new for Droid to read, so there's no reason the number should move.

Scoreboard

This lab should not move the number: it is still 10 / 24. Run it anyway. Knowing which changes cannot move the score is half of knowing what the score means.

If you have the budget, run T7 alone and read one of its sessions the way you read yours in Step 3:

larkspur-payments
npm run scoreboard -- --task T7

T7 passes three of three. Open a run anyway and check whether it read ach.ts. A task that passes its check with a guessed error list is exactly the gap you found in Lab 0, Step 4. The score stays the same. What you know about the score has changed.

Checkpoint​

Checkpoint0 / 6

In one sentence, what can a trace show you that a diff never can? Keep it, because Lab 3 moves that moment earlier: from reading the trace after the work to reading the plan before it.

Extensions​

  1. Fork and compare. Run droid --fork <sessionId> on your T7 session and tell the fork: "Before you edit, list every file that defines an error this route returns, and read each one." Compare the two traces and the two documents. Which instruction would be worth putting in a file every session reads? (Lab 2 is that file.)
  2. Count the prompts at Low. Start a fresh session, press Ctrl+L once to reach Low, and give the same T7 request. Which prompts disappeared? Is there one you would rather still see? Write it down for Lab 4, where you decide what runs without asking.
  3. Audit the baseline. Your baseline T2 runs are sessions too. Use droid search "rounding", which searches your local sessions, to find every session that mentioned rounding, and open one. If the headless runs don't show up, note that: it's a gap in the chain Ruth asked for. How long did it take you to go from "an incident happened" to "here is the session and the exact tool call"? That time is what Ruth's second condition is about.
  4. Pipe a diff in. droid exec reads from stdin. Make a small change on a scratch branch, then run git diff | droid exec "list every function whose behavior this diff changes". It's read-only, so it can't touch anything. Would a summary like this have caught the March change in a 600-line pull request?