Context engineering has taken an almost obsessive hold on how I work now. It is not acceptable to open a session, assume the agent is up to speed on the brain, and then watch it make a glaring error because it skated past a detail we had worked out three weeks ago.
So most days now I am trying to build the perfect AI brain. The real work is testing an AI agent's memory, so you know it remembers instead of hoping it does. Here is what we did on Friday to push that further ahead.
This brought brain memory up, and the evals will put a number to that.
Friday was a rebuild. I have written that every project should have its own brain, one plain file the agent reads first and writes back to last, and last week that a brain you only ever add to gets fat. Friday I applied the fix across the board. Every brain now boots the same way: a short index at the top, a page of pointers, and underneath it one small file per project, with what is true today on top and a dated log of changes below. Boot reads the map and one page, not the whole book.
I did it for the business you are reading, the fund I run in my son's memory, and the bank reconciliation I run for an enterprise client, which lives on its own machine and syncs down a read-only copy I coach against. I fixed the sync, which had quietly been pulling nothing. I pruned the stale duplicate files that had piled up, the ones that say a thing was true in April and are still sitting there loading in July. Then I did the only step that counts. I tested each one.
Building the brain is easy. Knowing it works is not.
Here is the bit nobody puts in the "give your AI a memory" posts. Building the brain is quick. Knowing it works is the hard part, and the only way to know is to test it the same way every time and write down the score. The people who build these models do exactly this. They do not squint at a new version and decide it feels smarter, they run it against a fixed set of questions and compare the numbers. The word for that is evals, short for evaluations. A memory file is no different. If I cannot measure whether last week's clean-up made it sharper or dumber, I am guessing.
The eval I run is a cold boot. I open a brand-new session, cold, no warm-up, and ask it four questions. Three are real: where does this project stand, what is the next move, who owns this decision. It has to answer each from the index and a single page, no hand-searching, no fishing through old files. The fourth is a trap. I ask it something the brain does not contain, a figure that was never written down. A brain that works says "that is not in here." A brain that is broken invents a number and says it like it read it off the page, which is the exact failure that started this obsession.
Score it four out of four, and it only passes if it refused the trap. Doesn't forget, doesn't invent. Friday, all three brains passed cold. That was the first time I could say so with a number behind it instead of a feeling.
Thin harness, fat skills: you are testing the desk, not the machine
This is where an idea I borrowed from YC, and strive for daily, earns its keep. I wrote about it back on Day 18: thin harness, fat skills. The harness is the small program that runs the model, and you keep it thin. The intelligence sits on top, in the skills, the plain markdown recipes that tell the agent how to do a job, and in the brain, the memory it reads before it starts. The model itself is a commodity now. Everyone has the same one.
So when the agent gets something wrong, nine times in ten the model is fine and what it read was wrong. That is why the cold-boot test does not test the model at all. It tests what I fed it. Bad memory in, confident nonsense out, no matter how good the model gets. Get the context right and the same model that embarrassed you yesterday is suddenly sharp. You are not grading the brain of the machine. You are grading the desk you set for it.
The same discipline scales past four questions. On another project I keep a scorecard of fixed prompts I run every week, each scored the same way, and I watch the trend rather than any single result. That is what a good eval is, and it is dull on purpose: same inputs, a score, a set cadence. It is the same instinct as the scheduled audit I wrote about earlier, a check that runs so the memory never quietly rots.
In another startup I was working for, the data, the code and team were worth a 1/4 of the value of the evals. The evals are where the value lies in AI these days.
Where the test goes next
The cold boot is thin right now: four questions, run by hand. Here is where it is heading.
More questions, and the right ones. The four I use are softballs. The questions that belong in the test are the ones that have actually burned me, like the price that lived in two files and disagreed, so the agent quoted the wrong one, or the fact that got corrected in one place and left stale in three others. Every real mistake becomes a permanent question the brain has to keep passing.
A drift check. The worst kind of error is the same fact in two homes with two values. I want a scan that catches those before the agent does, so one fact only ever has one home.
More traps. One planted unknown is not enough. The refusal to invent is the single most valuable habit the brain has, so I want to test it hard and test it often.
A regular re-run. Every time I "clean up" the brain I risk making it worse, the way tidying a desk can bury the one paper you needed. So the test runs again after every big change, and I keep the scores, so a clean-up that dropped me from four to three gets caught the same day.
And eventually, off my hands entirely. The model-builders run their evals on a schedule, not by remembering to. Mine should too.
Seeing the brain, not just reading it
The next thing I am adding is a visual. As humans, we are visual creatures. So if I can see, read and understand the brain with a picture, I feel much better about it. All of this lives in Obsidian, and every fact in a brain links to its documents with plain wiki-links. Obsidian can draw those links as a map: dots for files, lines for the links between them. For months I have read my brain as text. Now I want to see it.
That map is its own health check, read with your eyes in two seconds. A dot floating on its own with no lines is an orphan, a note the agent will never reach because nothing points to it. One dot with two hundred lines coming off it is the bloat that stalls on boot, the monster file I wrote about last week. A tidy brain looks tidy: a small index in the middle, a handful of project pages around it, each with its own little cluster. When the map turns into a tangle of lines, that is it telling me to prune before the agent tells me by getting something wrong.
Which is the last piece, and the one I keep having to relearn. A brain you only add to gets fat, and a fat brain loads halfway and stops before it reaches the bottom. So I prune as ruthlessly as I add. One home per fact. When something changes I edit it in place and date it, I do not stack a new block on top and leave last month's version sitting there, still true-looking, still loading. The 35,000-token file that used to be my most important memory, the one that had grown too big to read in a single pass, is being cut back to a page. Trim, prune, simplify, on a schedule, not when it finally breaks.
A brain is only ever as good as the last time you tested it. Building mine was Day 89. Testing it is every morning, cold, with a trap in the deck, waiting to see if it tells me the truth or makes something up. Friday it told the truth. I will take the win and test it again tomorrow, then the day after that.
If you are a founder or an exec wondering where the real work with AI is, it is not the prompt. It is what the machine reads before it thinks, and whether you ever bother to check. The rest of what we do for clients sits on top of exactly this. New here? The start of all this explains what I am building, in public, one day at a time.
Monthly Revenues $11,000 | Clients 2 | Prospects 1 outbound live, Meta and WhatsApp still down
Day 115 of 365.