Research on Autopilot

Most of my early use of AI coding agents was a back and forth loop. I asked for a function, it gave me one, I tweaked it, ran it, and came back for the next one. That is useful, but I am still driving every step.

A while back I started handing whole investigations over instead. In this post I’ll go through how I set that up, and one investigation where it made a difference.

How I set it up

It comes down to three things.

Write the context down first. Before handing anything over I write the problem definition, the current hypothesis, and the directions I want the agent to explore. For a bug, that includes what I have already ruled out. The quality of this document is the biggest lever on how useful the agent ends up being.

Give it permission to run, not just to answer. A partner needs to execute scripts, check logs, spin up environments, make observations and iterate, without asking at every step. You are not taking yourself out of the loop; You are raising the threshold for when your input is needed.

Ask for a running log and explicit checkpoints. Long investigations produce a lot of noise. I ask the agent to record what it tried, what it found and what it is doing next, and to flag me when something specific happens, e.g. a test passes unexpectedly, an error changes character, a pattern shows up.

Example: intermittent backup failures

I was trying to work out why certain backup jobs were failing intermittently. The test cycle was:

1. Set up an S3 bucket populated with many objects
2. Make a configuration change, e.g. bucket policy, KMS key, bucket notifications, object versioning
3. Upload or delete some objects
4. Wait for the next scheduled backup job to run
5. Observe job status
6. Trigger a restore and verify the result
7. Repeat from step 1 with a different config combination

Ten rounds of this, and every round involved waiting. Before AI agents I had a notebook full of CLI commands. I eventually scripted it, which made it repeatable, but I was still the one watching the terminal, rerunning things and taking notes.

With the investigation handed over, the workflow was different. I wrote the context doc first: the problem statement, the relevant config knobs, what failure looked like, what I had already tried. Then I asked the agent to generate scripts for each phase of the loop, gave it permission to execute them in the test environment, and asked it to watch for job completion, restore verification and error codes, and to notify me when something was worth looking at.

Then I went and did other work.

The agent ran the cycle, kept a log, and flagged me when a restore failed with an unexpected error. I read its summary, formed a hypothesis about the root cause, updated the context and sent it back for another round. What used to take half a day of focused attention became something running in the background.

One small addition worked better than I expected. I asked the agent to use the macOS say command to announce the outcome at each checkpoint, so I would hear “restore completed successfully” or “job failed with error code 403” from the laptop while working on something else. (it is a gimmick, but I stopped checking back manually.)

What is hard about it

The hard part is not technical. It is the impulse to stay in the loop at every step.

Engineers are trained to be hands-on. We want to see the output of every command, and to catch things before they go wrong. Letting an agent run ten test cycles while you are in a meeting feels like losing visibility. What you are trading is low value visibility for high value attention; You do not need to watch a script run, you need to know when it finishes with something interesting.

The enabler is that opening context doc. Investigations handed over with a vague mandate tend to drift; Ones with a clear “flag me when X happens” signal tend to converge.

Conclusion

The setup that works for me is to write the problem, the hypotheses and the success criteria down first, give the agent permission to execute in a test environment, and ask it to flag specific outcomes rather than report every step.