How I Built a Shared AI Memory System Using Markdown and GitHub

For a while, every new AI session started with me re-explaining things: who I am, what I’m working on and which documents matter, often by uploading the same PDFs again. Now any AI I use starts with all of that.

This is a story about how I moved my notes out of Google Drive and into a folder of plain text files that every AI I use can read and update, and about the handful of rules that keep that folder from turning into a mess. I call it my memex.

If you read my last post, you’ve already met it. The bot that reads our kids’ school email keeps its rules there. This post is about the memex itself: what’s in it, how it stays accurate when several AIs are writing to it, and how you could start one.

What Was Wrong With Google Drive?

In short, AIs kept tripping over my documents. When I pointed Claude at my Google Drive and asked it to edit a Google Doc, it would create a new copy instead of changing the one I had. Claude has its own documents that it can edit, but other AIs can’t easily read them. And most AIs couldn’t search my Drive well enough to find things, so I often ended up exporting documents to PDF and uploading them by hand.

I wanted one place that any AI could read, search and change, without me copying things around. A note-taking app wasn’t the answer either, since you’re limited to the structure and features the app gives you.

What Is the Memex?

The name comes from Vannevar Bush’s 1945 essay “As We May Think”, where a memex is a desk that stores a person’s books, records and communications so they can be consulted quickly. Mine is a git repo of markdown files. Markdown is plain text with a little formatting, e.g. a # in front of a heading. A git repo is a folder that records every change, so you can see what changed, when and why, and undo any of it. Mine lives on my laptop and in a private repo on GitHub.

The files are organized by what they’re about:

1. knowledge/ holds the notes: my career history, projects, reference material, the household (school schedules, the cars, recipes) and a daily log.
2. context/ holds a few files that tell an AI who I am and how I like to work, e.g. “ask me instead of guessing” and “never commit directly to main”.
3. bots/ holds a profile for each bot, which is what my last post was about.
4. scripts/ and skills/ hold the small programs that check and maintain everything, with their tests.

I created the repo on March 1, 2026 from a template and then mostly forgot about it. The real work started on October 13, 2026. As of October 31, it has 131 notes and about 157,000 words.

How Did My Drive Get Into It?

On October 13, I downloaded my Google Drive to my laptop and had Claude read it. It extracted the text from about 780 files: PDFs, Word documents, slide decks, spreadsheets and plain text. Google Docs were the one snag. They sync to a laptop as pointer files with no text in them, so I downloaded those folders again as Word and Excel files. Claude then read the converted files too and summarized everything it had read into notes, mostly about my career and past projects, rather than copying the documents in.

Some things I didn’t want in the memex at all, so before anything was summarized we agreed on what to leave out: tax and financial records, anything that looked like it held a password, medical and identity documents, and the folders about our kids. The memex keeps a ledger of the import that lists what was read, what was skipped and why. That way a later session doesn’t redo the work, or worse, import something I had already decided to exclude.

The import surfaced something I hadn’t expected. When you read 25 years of someone’s documents side by side, they disagree with each other. For example, one of my old self-reviews said our hotfixes had dropped from 8 a week to 1.5, while my other records said 7. Rather than pick one, Claude listed every conflict like that and asked me. I answered them in batches that day, and each answer became an entry in a decisions log.

How Does It Stay Accurate?

A pile of notes that several AIs write to will drift. One session restates a fact a little differently, another copies an old number from a document, and a month later nobody knows which version is right. So the memex has a few rules, and a program checks most of them on every change.

An example header. The index is built from these lines.

1. Every note starts with a short header: when it was created and last updated, some tags, whether it’s active or a draft, a one-line summary and how private it is. Private notes never leave the memex. Usable notes can inform things I write, like this post. Public notes are already public.
2. A script builds an index from those headers, one line per note. An AI reads the index first and then opens only the notes it needs. That matters because an AI can only hold so much text at once (its “context window”), and 157,000 words is far more than you’d want it to load for every question.
3. The decisions log records each of my rulings with its date, 42 of them so far. Old decisions are never edited. A new one says which one it replaces.
4. Some decisions are backed by a rule in the checker, so a corrected fact can’t creep back in. If a note says hotfixes went from 8 to 1.5 a week, the check fails and points at the decision.
5. The checker, or linter, also makes sure every note has its header, every link points to a real note, file names follow one pattern, nothing looks like a password or an access key, and the index is current. It runs on every change, along with 196 automated tests for the scripts.
6. Every change arrives as a pull request, which is a proposed change I can read before it’s accepted. Nothing is committed straight to the main copy. For the memex, I let the AI merge its own pull request once the checks pass.
7. Nothing is deleted. Notes that are out of date move to an archive folder.

None of this is clever. The rules are simple enough that any AI can follow them, and the checker catches it when one doesn’t.

What Is the Dream?

The checker can tell you a link is broken, but it can’t tell you that two notes say the same thing in different words, or that a plan from three weeks ago is stale. For that, there is a cleanup pass called a “dream”, loosely after the way sleep is thought to consolidate memories.

I didn’t come up with the idea. Anthropic added dreaming to Claude Managed Agents in May 2026: a scheduled process that reviews an agent’s past sessions and memory, pulls out patterns and tidies the memory, either on its own or with you reviewing the changes first. Their Code with Claude talk, “Memory and dreaming for self-learning agents”, covers how they got there. OpenClaw has its own version that runs in light, REM and deep phases and writes a diary of what it changed, and Hermes Agent has a community plugin modeled on OpenClaw’s. Mine borrows from all of them. The difference is that my memory is a git repo, so a dream’s changes arrive as a pull request I can read.

A dream has two halves. The mechanical half runs the checker and rebuilds the index. The reflective half is an AI working through a written checklist: promote anything durable from the daily log into the right note, find facts stated in more than one place and keep one owner for each, flag notes that have gone stale, and write down the questions only I can answer. It lands as a pull request with a short report.

What the AI is allowed to do on its own depends on the change:

1. Mechanical fixes that don’t change meaning, like a broken link or a stale index, it just does.
2. Moves, merges and new notes, it makes on the branch and calls out in the report, so I can accept or revert them.
3. Anything that changes what a fact means, which version of something is right, or a privacy rule, it asks me about first and doesn’t touch until I answer.

There have been four dreams so far: October 13, October 15, October 25 and November 1. The last two were run by a bot on its own weekly schedule. On October 25, lint was already clean, so the main results were notes for the two people mentioned most often who didn’t have one yet (notes on people are private and stay in the memex), and one place where the repo’s instructions contradicted a newer decision. The November 1 dream was lighter: one lint warning fixed, the bot’s own log brought up to date and no questions for me.

Who Writes to It?

More than one AI, which was the whole idea. From October 13 to 31, the memex had 190 commits and 122 merged pull requests. Most came from Claude Code sessions on my laptop. Others came from Grok Bot, the agent I use to run bots on a cloud computer, and from the bots themselves, e.g. the school bot updating its profile and a memex bot that does the weekly upkeep.

Two things fall out of that. First, I’m not tied to any one AI. If I switch tools tomorrow, the notes, the rules and the history come with me, because they’re just files. Second, every session starts with context. I don’t paste my background into a prompt or explain how something works, because the AI reads the index and finds it.

In practice, I start most things with that same memex bot. When I want to add to something or work on it, like a new project idea, a blog post I should write or a book I should read, I ask the memex bot to add the details. It already has the context of many things in my life, both current and old. And when I’m working in another AI and I think my memex would help, I give that AI read and write access to the memex’s GitHub repo, and it has the same context right away. A bot gets its own access token that works only on that one repo, can only read and write files and open pull requests, and expires within 90 days.

Pull requests turned out to be useful for more than catching an AI’s mistakes. When I want feedback on a document, I open it as a pull request and leave comments on specific lines in GitHub, the way you’d review code. The AI answers each comment and pushes a new version. This post was reviewed that way.

Is It Safe?

I think it’s relatively safe, as long as you don’t have a lot of automations that send out or post content unattended. The only bot of mine that emails anyone sends to one family list, from its own account; the others open pull requests or message me in a private Slack channel. That said, there are risks, and you certainly don’t want to store anything sensitive in your memex, e.g. credit card numbers. The checker flags anything that looks like a password or an access key, but it can’t catch everything, so the better habit is to leave those things out in the first place.

What Would I Do Differently From Day One?

Not much, and I wouldn’t try to design it all up front. Everyone’s preferences differ, and mine weren’t clear until I’d used it for a while. A memex changes shape as you use it, and the goal is that over time it holds all of your context and preferences, written down.

That said, three things would have saved me some cleanup:

1. Start the decisions log on day one, not after the import.
2. Decide what stays out, e.g. financial and medical records, before an AI reads anything.
3. Add the index before the notes pass a few dozen.

How Could You Start One?

You don’t need my scripts to get most of the benefit. A folder of markdown files in a git repo, a short header on every note and an AI that can read files will get you a long way. I’d add the rest in roughly the order it paid off for me:

1. A context file that says who you are and how you like to work.
2. The header on every note, and an index built from the headers.
3. A decisions log, as soon as you catch yourself correcting the same fact twice.
4. A checker that runs on every change.
5. Changes by pull request, so you see what an AI did before it sticks.
6. A cleanup pass once the notes outgrow what you can review by hand.

If you’d like a head start, I’ve put the memex’s structure and tooling on GitHub as a template, with none of my notes in it: memex-template. It has the folders, the note header, the index builder, the checker, a decisions log with an example rule, and the dream protocol with its /dream command. The bot template from my last post lives there too, since the bots deliver into the memex.

Thanks for reading.