How do you actually test if an AI understands your codebase

Writing correct code is largely a solved benchmark problem; the harder question is whether an AI agent understands a codebase the way a senior engineer does — knowing what depends on what, and what silently breaks when you touch it. The usual checks fail: asking a model to explain a file only tests local reading, architecture trivia tests memorized recall, and passing tests miss exactly the silent regressions that matter, because a confident wrong answer looks identical to a correct one.

One engineer built a concrete alternative: a 'change-impact teardown audit.' Pick the hub model a real app leans on, then ask an agent to find every dependent before that model is torn down — a pure structure question with no single-file answer. The method's rigor comes from three choices: a hand-built, file-and-line-precise answer key built before any run; grading against that key rather than plausible-sounding prose; and running each case twice, since non-deterministic inference can hide real variance. Tested across thirteen real Ruby codebases with and without a queryable structural map, the map made a clear difference on large applications and little difference on small, tightly colocated ones.

The author is candid about the gaps: the test only catches static call-and-dependency edges, missing dynamic wiring like pub-sub or runtime dispatch; it covers only one shape of task; the answer key inherits the builder's own blind spots; and it measures recall of dependents, not judgment about which ones are actually risky.

For engineers, this offers a practical, repeatable way to sanity-check an agent's structural understanding before letting it near load-bearing code — while also being a useful case study in how hard 'understanding' is to define and measure honestly.