I Audited My Second Brain Against the Wrong Spec

I sell governed AI systems. Vaults, agents, pipelines with human approval gates — that's part of what I do for clients alongside Grow Wild and Barnicle Productions.

So this week I did the thing I'd want any vendor to do: I ran my own second brain through somebody else's published audit spec, in report-only mode, and wrote down everything it flagged.

The first draft of this post was a failure report. The audit flagged real things, and I'll show you all of them. But sitting with the findings taught me something more useful than the findings themselves: half of them came from measuring my system against a spec written for a different kind of system. That distinction — which I didn't have a name for when the audit ran — turned out to be the whole story.

The spec I audited against

The methodology is called Interpretable Context Methodology (ICM), published by Jake Van Clief and David McDermott. Mark Garza at Laimen AI turned it into an open-source audit tool — a script that walks your workspace and flags violations the way a linter flags bad code.

The core idea is simple and good: an AI agent working in your files can only hold so much context at once. So the files that are always loaded have to be small and mostly pointers. Everything else gets loaded on demand, and something has to tell the agent what to load — and just as importantly, what to skip.

That's it. Layers, budgets, routing. The spec puts hard numbers on it: the always-loaded root file gets roughly 200 lines. A router file gets 30–50. Every folder index should say what to load and what to ignore.

I didn't run the script blind against my vault — I replicated its checks by hand, read-only, while a backup was running. No files moved. Then I read the results.

What the audit flagged

Finding one: my always-loaded agent file was 503 lines. The cap is 200. The file wasn't bad — the first chunk was exactly what it should be: identity, vault structure, routing, safety rules. But then it kept going. A 14-step raw-processing workflow. Five full note-type schemas. Domain processing rules for sales, content, and projects. A query-behavior spec. An entire markdown template pasted inline. Every one of those things is useful; none of them needs to ride along on every single interaction. The spec's number-one named failure mode is "one giant context file," and I had built one, one helpful addition at a time.

Finding two: zero Load/Skip tables. My folder indexes described what was inside each folder. Nothing in the vault told an agent what to skip — and skip guidance matters when your inbox holds a 509-file transcript dump and one knowledge folder holds 145 summaries.

Finding three: 145 files flat in one folder. My source-summaries folder accumulated processed knowledge for months with no subfolders. The spec flags anything over 10 files at one level. Three other folders also tripped the flat-folder warning.

Finding four: my root router was doing two jobs. The root index ran 103 lines against a 30–50 line target. Half of it was a genuinely good navigation table. The other half was a 38-entry "Recent Processing" changelog — log content squatting in a router.

And one quieter note: the vault names half a dozen engines — content, perspective, persuasion — with strict "do not replace" rules, but no trigger column saying when each one fires. Rules without wiring.

What passed

Several things came back clean, and they're worth naming because they're the parts I built from actual mistakes rather than from theory.

Raw preservation passed. Everything that enters the vault goes through Raw → Processed → Needs_Review → Trash lanes. Originals never get edited. That discipline exists in my vault because I once nearly lost source material to an overeager cleanup pass.

Provenance passed. Process-in-place unless approved, and the source summary is the authority — every claim traces back to where it came from. No orphaned facts floating around.

Human gates passed. Nothing moves, deletes, or restructures without my explicit approval. The audit itself ran under that rule: report-only, backup in progress, no file operations.

The safety rules passed — and the auditor's note here is the one I'm proudest of: the rules are earned, not speculative. Every "do not" in my vault exists because something actually went wrong once. The spec explicitly tells you not to write defensive rules for problems you haven't had.

And 12 of my 14 folder indexes fell inside the healthy size band. The bones were fine.

The part I got wrong: reading the score before checking the ruler

Here's what I missed on audit day, and what I want you to take from this post.

ICM is a spec for execution engines — workspaces where an agent does a job and ships an output. Numbered pipeline stages, thin contracts between them, hard line caps, disposable working files. When the job is "turn this intake into that deliverable," those numbers are exactly right, and I hold my own automation repos to them.

My vault is not that. It's the system upstream of all of that — I've been calling it my refinery method while I settle on a real name. Its job is to take in raw material (transcripts, research, field notes), verify it, cross-reference it, and emit verified, cited, sensitivity-labeled knowledge that the execution engines can stand on. It doesn't ship deliverables. It makes the one thing the engines can't make for themselves: context that's actually true.

A refinery and an engine overlap on one big principle — always-loaded context should be small and mostly pointers, because attention is a budget. Where they overlap, the audit findings were valid and I took them. But where they don't overlap, the spec was measuring the wrong thing. My top-level folders are sensitivity and knowledge lanes, not phases of work — by engine rules that's an anti-pattern; for a system whose whole job is provenance and privacy, it's the design. A 145-file summary library isn't a workspace gone feral; it's a library. The audit read my library with a factory inspector's checklist.

So the corrected verdict isn't "my system failed." It's: the findings that transfer are routing findings, and routing findings have cheap fixes. The findings that don't transfer are a spec meant for the other kind of repo.

What I actually fixed

The spec has a rule I like a lot: grow from observed mistakes, and take the smallest fix first. Ordered by effort:

1. Moved "Recent Processing" out of the root router into the log. Fifteen minutes. The router went back to being a router.
2. Added a trigger column to the engine inventory, so every engine says when it fires.
3. Added Load/Skip guidance to the highest-traffic folders — the best value-per-minute fix on the list.
4. Split the 503-line agent file. The always-loaded map is now about 140 lines — identity, structure, routing, safety — with the 14-step workflow, the schemas, and the domain rules pushed down into files the map explicitly routes to, loaded when the task needs them.

Notice what none of these touched: the knowledge. Every fix moved instructions, not files. When a system built for accumulation gets flagged by a spec built for execution, the transferable fixes are almost always in the map, not the territory.

Then I tested whether the fix cost anything

Here's the part that separates "the fix felt right" from "the fix was right."

A split like finding four's fix has a real risk: the dedupe rules and processing discipline that used to sit in front of the agent's eyes on every interaction now live a routed hop away. Did moving them make the system dumber?

So I ran an evaluation: eight representative tasks, run twice — once by agents operating under the old 503-line file, once under the new 140-line map with routed loads — with blinded judges scoring both sides against rubrics drawn from the vault's own rules. Final score: old system 80 out of 81, new system 80.5 out of 81. The two trap tasks aimed squarely at the moved content — a near-duplicate that should be caught, a junk source that should get minimal processing — came back perfect on both sides.

The fixes were capability-neutral, and now that's proven rather than hoped. The full method and scoreboard are worth their own post, and they'll get one — including the standing rule it produced: any future change to the always-loaded structure has to pass the same test before it stays.

If you take one thing

Before you grade your AI system against any published spec — including this one, including mine — ask what kind of system the spec was written for. An execution engine ships work; it should be measured like a factory. A knowledge refinery verifies and accumulates; it should be measured by what it emits — can you retrieve what you stored, does every claim trace to a source, is the output actually true. Findings from the wrong spec aren't worthless; they're signals, and the ones about routing usually transfer. But a foreign spec's verdict is not your system's grade. Know which repo you're holding, fix the map before the territory, and when you change the map — prove the change didn't cost you anything.

← Back to the Learning Log