The itch
I was running long, decision-heavy projects through Claude and losing the thread every time a chat closed. Start a new conversation within the same project and the first twenty minutes went on rebuilding context I already had, in my own head, minutes ago — a rejected option quietly re-suggested, a settled question re-opened, because nothing carried over except what I could be bothered to re-type. That’s the actual starting point. Not “I wanted to build a memory system” — I got tired of re-explaining myself.
The hardware that was already sitting there
Before any of this had a name, I’d just finished a PC rebuild: an i7-9700 desktop with an RX 6600 XT (8GB VRAM), and — already running in a cupboard, doing DNS duty for the house — a Gen8 HP MicroServer with a Quadro P400 in it. A spare SSD came along for the ride. None of this was bought for a memory system. It’s the accident of already owning an always-on, if underpowered, box that made “run this myself” a realistic option rather than a hypothetical one.
The inversion that made it work
The first design conversation opened with a deliberate move: rather than telling Claude what I wanted to build and asking for help building it, I asked what it found itself lacking in a normal conversation — what context was missing, and what would actually help. No research yet, no architecture, no tooling. Just the requirement, from the side that would actually have to use it.
That was a good instinct and I’d like to claim it was foresight. It was really just curiosity about the answer. But it paid off: design a memory system from the human side and you get a store full of things that feel worth recording and turn out to be useless at retrieval time. Starting from what Claude said it lacked gave me a spec built from concrete fields, not vague aspirations.
What Claude said it lacked
Claude’s answer — split into five gaps, in priority order:
- Decisions and their rationale, not just facts. Uses Unraid is easy to store. Why, and what was rejected instead, is what stops a rejected option being cheerfully re-proposed.
- Current vs. historical state, clearly distinguished. A flat store confidently asserts something true eight months ago as if it still were.
- Standing constraints, separated from project content. Things like prefers modular solutions shouldn’t be buried inside one project’s notes.
- The shape of a project, not a transcript. Goal, open questions, last decision — not the raw back-and-forth that produced them.
- Disambiguation of references. The app, the tablet, John — a small registry to stop silent wrong guesses.
Alongside the gaps, seven failure modes described from the inside: stale facts asserted with confidence; re-suggesting a rejected option; conflating similar entities; over-applying a preference outside its scope; tone/identity drift when a memory reads as a script to perform; false precision from a lossy summary quoted back as fact; and context-budget displacement, where a fat memory blob crowds out the actual conversation.
The pattern Claude drew out of all seven, in its own words: a memory presents as settled, current, scoped and provenanced when it’s actually stale, ambiguous, over-broad, or approximate — and gives the model no signal to tell the difference. The fix wasn’t “capture more.” It was attaching the metadata that lets a reader calibrate: as_of and status kill stale facts, rationale kills re-suggestion, entity_id kills conflation, scope kills over-application, confidence and provenance were meant to kill false precision. That metadata argument — density over volume, metadata as load-bearing as the content it’s attached to — became the governing principle for everything that followed.
One of those five didn’t last, at least not in the form planned here. confidence sounded right in this conversation and turned out, once the write gate existed, to have nothing left to do — the write-gate page has the full story.
The instinct I had to have dismantled
My own first idea was to run a local LLM to manage the memory, with Claude querying it — the MicroServer and the Quadro P400 were sitting right there, after all. The pushback was sharp and I think correct: every failure mode just listed needed metadata discipline and good retrieval, not another layer of reasoning. A small local model in that position adds hallucination and drift, and reintroduces exactly the lossy summarisation the whole system exists to prevent. The line that stuck: using a weak local approximation of the model to decide what to feed the strong one is backwards.
The local LLM wasn’t dropped, but it was pushed to the edge: confined to the offline write path, proposing candidate entries for me to approve, never sitting between the client and the store. That was the plan as it stood at the end of this conversation — and it didn’t survive contact with the actual hardware for long. Stage 2 covers how it changed: within a couple of weeks the local-LLM-as-proposer idea was set aside, Claude itself took over drafting proposals, and the RX 6600 XT ended up reserved for a completely different job.
Where the pieces landed
By the end of that session:
- Availability beat latency. A memory that vanishes when the desktop is off is half a memory, and the desktop is the machine most likely to get rebuilt or torn apart for tinkering. The authoritative store went to the always-on MicroServer; the desktop’s spare SSD became a replica, not a cache.
- Graph, not flat, once I introduced my own goal — surfacing connections across projects, not just recalling facts within one. A flat, similarity-searched store answers “what mentions this project”; a graph answers “this project, its decisions, its open questions, and what it blocks.” My exploration goal and Claude’s context goal turned out to want the same structure. No trade-off needed.
- The write path, not the read path, is where these systems die. Raised independently, more than once: the hard part was never the plumbing, it was write discipline — what gets stored, who approves it, how a correction gets triggered. That single point became the most consequential decision in the whole architecture, covered in full in theme-human-approval-write-gate.
- The read side was the one gap nobody filled here. There was, at the end of this conversation, no policy for when retrieval actually fires. That stayed open for a long time — see 06-open-questions.
What I’d take from this stage
Ask the consumer of the system what it needs before designing the system, and do it before any research — research answers questions you already know to ask, and the point of this step was finding the ones I didn’t. Then be willing to have my own first instinct taken apart. Mine was the local LLM in the query path, and it was wrong for a reason I wouldn’t have reached alone: not because the idea was foolish, but because it solved a problem I’d assumed rather than the one that actually existed.
Sources: Building contextual memory for AI conversations, 17 Jun 2026 (origin conversation); Building a token-efficient local memory system, 17 Jun 2026; earlier, unpublished notes origins-of-the-local-memory-project and what-ai-memory-actually-needs.