I'm a self-taught robotics engineering student in Bengaluru. I built Neo, a fully offline AI assistant that remembers facts about you across sessions. It runs llama3.1 through Ollama, stores vectors in LanceDB and facts in SQLite, and does web search through a self-hosted SearXNG. No cloud, no API keys.

The problem

When you tell the assistant "I stopped liking chess", it has to decide what that means. Is it a new fact, an update to an old one, or a retraction?

My first design used one rule: if cosine similarity with an old fact is above 0.8, treat it as a contradiction.

What I measured

I tested 10 real sentence pairs with my actual embedding model through Ollama. The 0.8 rule missed paraphrases and updates. Unrelated pairs scored only 0.29 to 0.41, so the useful range sat much lower than I had assumed.

I also found an earlier mistake. My retrieval eval was measuring L2 distance, not cosine. After fixing the metric, Recall@k came out at 0.95 and MRR at 0.925.

The fix

I stopped asking cosine to make the decision. Now it only finds candidates, using a low threshold of about 0.5. Then the flow is:

new fact ↓ find candidates (cosine ≥ 0.5, active facts only) ↓ no candidates → ADD candidates found → LLM returns {action, target_id} ↓ code validates the answer ↓ write to SQLite + LanceDB

The LLM picks one of ADD, UPDATE, DELETE or NONE, plus a target id. Plain code then checks the answer:

  • An unknown or malformed target id falls back to ADD.
  • An UPDATE whose target text is identical to the new fact becomes NONE.

The model proposes and the code decides what is allowed. This mattered because llama3.1 sometimes returned UPDATE for a fact that already existed, or picked the wrong target. Prompt tweaks helped, but the guards in code are what made it reliable.

What I'd tell others

  • Test your threshold on real sentence pairs before trusting it.
  • Check which metric your eval actually uses (L2 vs cosine).
  • Keep the LLM's output narrow and validate it in code.
  • Write tests with fake LLM calls so you can check the gate logic without Ollama running.

Honest notes

v1.9 was mostly written with Claude and turned out unstable, so I rebuilt v1.9.5 by hand from v1.8.5 to understand every part. The README credits the AI help. Known limits are listed in KNOWN_ISSUES.md. For example, comparative preferences aren't extracted, and follow-up web queries lose context. It also needs hardware that can run an 8B model locally.

Repo: https://github.com/Neo-X7/neo-AI

How would you handle contradicting facts in a memory system? I'd like to hear other approaches and how can I improve things in the future.