Auditing codebases for AI agent readiness

I have been building a skill that audits a codebase for AI agent readiness. You point an agent at your repo, and it comes back with an evidence-cited report of where the next agent, or the next engineer, will waste tokens, guess wrong, or be misled by the repo’s own documentation. Each finding comes with the smallest fix that removes it.

In this post, I’ll explain the idea behind it, show what it found on a real codebase, and how to run it on yours. The skill is at github.com/nicholasren/agent-readiness.

Background

Teams starting with AI agents usually ask which model to use or how to write better prompts. After spending real time delegating work to agents in a few codebases, I think that is the wrong end to start from. When an agent wastes tokens exploring, or writes code in the wrong layer, or “fixes” a function that is never called, the cause is usually not the model; it is that the project’s knowledge is illegible. The run/test commands live in six overlapping bash scripts, the README documents a setup that stopped working after a dependency bump, and the design conventions live in reviewers’ heads.

A new engineer hits exactly the same walls; they just struggle quietly, over weeks, and nobody writes it down. The agent produces a transcript. Every place it had to guess is recorded, turn by turn. Your agent’s flailing is a free audit of your codebase; the skill just makes the audit systematic.

I wrote about one slice of this in Three Consumers, One Interface: the CLI you built for developers and CI turns out to have a third consumer. This skill widens that argument and makes it runnable.

How the audit works

The skill drives an agent through three steps.

First, an orientation probe. Before reading any checklist, the agent tries to answer four questions using only what the repo offers, the way a new hire would: how do I run the tests, how do I run the app, where would a new business rule go, how does a change reach production. It keeps a friction log; any moment of guessing, contradictory documentation, or a documented command failing goes in it. Where a discovered command looks safe, it actually runs it; “README says X, X fails with Y” is stronger evidence than “README looks stale”.

Second, a scan against a catalogue of eight symptoms, e.g. no discoverable way to run anything, no trustworthy “done” signal, CI and laptop running different code, every session re-learning the repo. Each confirmed symptom links to a fix reference with a recipe, a worked example, and a what-not-to-do list.

Third, a report, AGENT_READINESS.md, written at the repo root. Three rules keep it honest. Every finding must cite a file, a line, or a command output; a finding without evidence does not go in. The agent must never invent conventions; if it cannot tell whether something is a convention or an accident, that becomes a question for the team, not a confident sentence. And every recommended fix must be additive and small: wrap the existing scripts behind a task runner instead of rewriting them, add an architecture test with an exemption list that grandfathers today’s violations instead of fixing all 200 of them, quarantine the flaky tests instead of overhauling the suite. The upside of additive fixes is they ship in an afternoon without anyone’s permission; the downside is your repo still looks messy underneath. (it already did.)

What it found on OpenClaw

I ran the audit on a checkout of OpenClaw (main @ e797e699698), a TypeScript monorepo with a 2.8GB working tree and 176 workspace projects. Its agent-readiness infrastructure is further along than anything else I have audited: 22 scoped AGENTS.md files with their CLAUDE.md symlinks all correctly pointed, dozens of recorded skills for release and PR work, and every script referenced in its docs actually existing on disk.

The audit still confirmed two findings. The root AGENTS.md is 388 lines that place Repair Doctrine, Product Doctrine, and bot-review policy before the Commands section, so a fresh session reads roughly 150 lines of policy before learning how to run a test; the index forgot to be an index. And the documented pnpm test entry point triggers a workspace install of 1,402 packages before a single test runs, with the suite’s real footguns (“bare vitest never exits”) recorded as prose warnings rather than guards; there are 8,852 test files across 115 Vitest configs, and no cheap honest subset of them.

That is a repo that invested seriously in agent readiness, and its gap was economy rather than accuracy: the docs are correct, and buried. I would not have guessed that from the outside, which is the argument for auditing instead of guessing.

Running it on your repo

git clone https://github.com/nicholasren/agent-readiness.git ~/.agents/skills/agent-readiness

Then, in a fresh session in your repo: Assess this repo for AI agent readiness. It took about eight minutes and 154k tokens on the OpenClaw monorepo, less on a normal service repo; repo size, not model choice, drives the cost.

What comes back is a report with a verdict naming the single highest-leverage fix, a “what already works” section (worth as much as the findings; it tells you what to build on), the findings themselves with their evidence and their smallest fix, and two sections most reports skip: questions for the team, where the agent refused to guess, and what it could not audit and why.

Then the loop. Confirm the findings you agree with, and: Fix the next thing in the readiness report. The skill implements exactly one fix, verifies it with that finding’s own measure, and logs it in the report’s History section. One afternoon per fix is the method, not a limitation; committing the report next to the code means git log AGENT_READINESS.md becomes the record of your repo’s readiness over time, and the next session picks up where the last one stopped.

Every report ends by asking you to grade its findings. Corrections, especially the ones marking a finding wrong, are how the symptom catalogue gets better; the issue tracker on the repo is the place for them.

Does it generalise

Against a fixture repo with seeded defects, the audit hit 16/16 scored assertions with the skill loaded, against 77% for the same model without it. The gap was where it should be: the unaided runs drifted into recommending rewrites, and one of them confidently asserted which of two duplicate functions was “the intended path” when the code never says. Against a well-prepared repo the skill reported zero findings, which matters more than it sounds; an auditor that pads is an auditor you stop reading.

Fixtures only prove it finds defects where I planted them, so the skill also ships scripts/inject_symptoms.py: it plants six defined defect classes into a copy of any real repository and writes a ground-truth manifest outside the copy, so recall is measurable on codebases with their natural mess. Injected defects are fresher than organically aged ones, so treat that number as an optimistic bound. The graded findings coming back from real repos are the evaluation that actually matters.

Caveats

The audit is only as good as its ability to run things; in a repo where no documented command works (e.g. a decade-old snapshot with a dead toolchain, which is what my first field test accidentally ran against), it degrades to reading, and says so in a “Not audited” section. Findings are model-dependent; the runs in this post used Sonnet, and I would not expect identical reports on a different day. A report is a claim about one commit on one date, which is why it lives in git next to the code it describes.