Auditing codebases for AI agent readiness

I have been building a skill that audits a codebase for AI agent readiness: point an agent at a repo, and it comes back with an evidence-cited report of where the next agent (or the next engineer) will waste tokens, guess wrong, or get misled by the repo’s own documentation. In this post, I’ll explain the idea behind it, walk through what it found on a fixture repo you can clone and rerun yourself, and on one real codebase, and show how to run it on yours. The skill is at github.com/nicholasren/agent-readiness.

Background

Teams starting with AI agents usually ask which model to use or how to write better prompts. After spending real time delegating work to agents in a few codebases, I think that is the wrong end to start from. When an agent wastes tokens exploring, or writes code in the wrong layer, or “fixes” a function that is never called, the cause is usually not the model; it is that the project’s knowledge is illegible. The run/test commands live in six overlapping bash scripts, the README documents a setup that stopped working after a dependency bump, and the design conventions live in reviewers’ heads.

A new engineer hits exactly the same walls; they just struggle quietly, over weeks, and nobody writes it down. The agent produces a transcript. Every place it had to guess is recorded, turn by turn. Your agent’s flailing is a free audit of your codebase; the skill just makes the audit systematic.

I wrote about one slice of this in Three Consumers, One Interface: the CLI you built for developers and CI turns out to have a third consumer. This skill widens that argument and makes it runnable.

How the audit works

The skill drives an agent through three steps.

First, an orientation probe. Before reading any checklist, the agent tries to answer four questions using only what the repo offers, the way a new hire would: how do I run the tests, how do I run the app, where would a new business rule go, how does a change reach production. It keeps a friction log; any moment of guessing, contradictory documentation, or a documented command failing goes in it. Where a discovered command looks safe, it actually runs it; “README says X, X fails with Y” is stronger evidence than “README looks stale”.

Second, a scan against a catalogue of eight symptoms, e.g. no discoverable way to run anything, no trustworthy “done” signal, CI and laptop running different code, every session re-learning the repo. Each confirmed symptom links to a fix reference with a recipe, a worked example, and a what-not-to-do list.

Third, a report, AGENT_READINESS.md, committed at the repo root. Three rules keep it honest. Every finding must cite a file, a line, or a command output; a finding without evidence does not go in. The agent must never invent conventions; if it cannot tell whether something is a convention or an accident, that becomes a question for the team, not a confident sentence. And every recommended fix must be additive and small: wrap the existing scripts behind a task runner instead of rewriting them, add an architecture test with an exemption list that grandfathers today’s violations instead of fixing all 200 of them, quarantine the flaky tests instead of overhauling the suite. The upside of additive fixes is they ship in an afternoon without anyone’s permission; the downside is your repo still looks messy underneath. (it already did.)

A fixture you can rerun it on

orderly is a small order-management API I use as the example repo for this series. Its main branch is well prepared; its messy branch is the same app aged on purpose, the shape of a repo that grew for three years: six overlapping bash scripts, a stale README, business logic inline in route handlers. The defects are planted, so I know the ground truth, and you can clone the branch and rerun the audit to see the same findings come back.

On the messy branch the audit confirmed, among others:

• The README’s first run instruction, make dev, fails outright; there is no Makefile anywhere. Its fallback, ./scripts/run.sh --local, silently ignores the flag; run.sh is one line with no argument handling.
• Test invocation has four different spellings across README, two scripts, and CI; the one captured as an executable script, scripts/test.sh, is the broken one. The audit ran it: FileNotFoundError: pytest.ini, a config file that does not exist.
• Plain pytest reports 3 passed, but one of the three silently returns when PAYMENT_API_KEY is unset; the money-moving code path is never exercised, and the summary line does not say so.
• app/utils.py defines a complete charge() function that is never called; the live payment call is inline in app/main.py:confirm(), with a comment admitting the confusion. An agent asked to change charging behaviour has a coin-flip chance of editing dead code.
• CI installs dependencies with an inline unpinned pip install that bypasses requirements.txt, runs Python 3.11 against the README’s claimed 3.8, and lints at a different line length than the local check script.

Each finding came with the recommended smallest fix; for this branch, a thin task runner wrapping the existing scripts unchanged, and a short AGENTS.md naming the dead code. None of the fixes is a rewrite.

What it found on OpenClaw

Fixtures prove the skill finds defects where I planted them, so the more interesting test is a large repo I had no hand in. I ran the audit on a checkout of OpenClaw (main @ e797e699698), a TypeScript monorepo with a 2.8GB working tree and 176 workspace projects. Its agent-readiness infrastructure is further along than anything else I have audited: 22 scoped AGENTS.md files with their CLAUDE.md symlinks all correctly pointed, dozens of recorded skills for release and PR work, and every script referenced in its docs actually existing on disk.

The audit still confirmed two findings. The root AGENTS.md is 388 lines that place Repair Doctrine, Product Doctrine, and bot-review policy before the Commands section, so a fresh session reads roughly 150 lines of policy before learning how to run a test; the index forgot to be an index. And the documented pnpm test entry point triggers a workspace install of 1,402 packages before a single test runs, with the suite’s real footguns (“bare vitest never exits”) recorded as prose warnings rather than guards; there are 8,852 test files across 115 Vitest configs, and no cheap honest subset of them.

That is a repo that invested seriously in agent readiness, and its gap was economy rather than accuracy: the docs are correct, and buried. I would not have guessed that from the outside, which is roughly the argument for auditing instead of guessing.

Does it generalise

The fixture doubles as the skill’s benchmark. Against the messy branch, the audit hit 16/16 scored assertions with the skill loaded, against 77% for the same model without it. The gap was exactly where it should be: the unaided runs drifted into recommending rewrites, and in one case confidently asserted which of the two duplicate payment functions was “the intended path” (the code never says; that is an invented convention). Against the well-prepared main branch the skill reported zero findings, which matters more than it sounds; an auditor that pads is an auditor you stop reading.

To measure beyond repos I built, the skill also ships scripts/inject_symptoms.py: it plants six defined defect classes into a copy of any real repository and writes a ground-truth manifest outside the copy, so recall is measurable on codebases with their natural mess. Injected defects are fresher than organically aged ones, so treat the recall number as an optimistic bound; each report also ends by asking the reader to grade every finding, and those verdicts are the evaluation that actually matters.

Running it on your repo

git clone https://github.com/nicholasren/agent-readiness.git ~/.claude/skills/agent-readiness

Then, in a fresh session in your repo: Assess this repo for AI agent readiness. The audit took about four minutes and 51k tokens on the orderly fixture, and about eight minutes and 154k tokens on the OpenClaw monorepo. Repo size, not model choice, seems to drive the cost.

The report names one highest-leverage fix. Confirm the findings you agree with, then: Fix the next thing in the readiness report. The skill implements exactly one fix, verifies it with the finding’s own measure, and logs it in the report’s History section; git log AGENT_READINESS.md becomes the record of your repo’s readiness over time. When the next finding is fixed, the report says what comes after that.

Caveats

The audit is only as good as its ability to run things; in a repo where no documented command works (e.g. a decade-old snapshot with a dead toolchain, which is what my first field test accidentally ran against), it degrades to reading, and says so in a “Not audited” section. Findings are model-dependent; the runs in this post used Sonnet, and I would not expect identical reports on a different day. A report is a claim about one commit on one date, which is why it lives in git next to the code it describes.