Build Buddy, Day 5: my vector search's best match was a wrong answer

Day 5 of MLH Global Hack Week: Builders Week. All week, Build Buddy has been collecting explanations from AI models about C# compiler errors. Some of them are right. Some are confidently wrong. Today I put them in a vector database to see what happens when you search them like a RAG system would.

The setup

• 12 real explanations of compiler errors from Gemma 3 4B and Grok 4.7, collected during the week
• Each one reviewed by hand: correct, wrong, or inconsistent
• Embedded locally with nomic-embed-text through Ollama, so no text left my laptop
• Stored in a Tiger Data hypertable with a VECTOR(768) column and an HNSW index

CREATE TABLE fix_notes ( noted_at TIMESTAMPTZ NOT NULL, model TEXT NOT NULL, error_code TEXT NOT NULL, explanation TEXT NOT NULL, review TEXT NOT NULL, embedding VECTOR(768) NOT NULL ); SELECT create_hypertable('fix_notes', by_range('noted_at', INTERVAL '1 day')); CREATE INDEX ON fix_notes USING hnsw (embedding vector_cosine_ops); 

Then I asked questions in plain English and ranked every explanation by cosine distance (lower means closer in meaning).

When it works, it works well

"An async call returns a Task and is assigned to a string without await."

The top match was Grok's explanation: "GetStringAsync returns Task<string>, so the call is awaited." Distance 0.245. The next best was 0.338.

It even told apart two errors with the same code. Gemma had also explained a CS0029, but that one was about assigning a string to an int. It ranked #7. Matching on the error code alone would have treated them as the same.

The best match was wrong

"A statement is missing its semicolon at the end of the line."

The closest match, by a clear margin, was Gemma's explanation that the semicolon "wasn't required." That's the opposite of the real error.

Vector search found exactly the right topic. It has no idea whether the content is true. A RAG system would have retrieved this and repeated it with confidence.

Two filters, two different jobs

Filter 1: trusted sources only. Adding WHERE review = 'correct' removed the wrong answer. But there was no correct semicolon explanation in my data, so the search returned the closest leftovers instead: notes about missing return values, at 0.318 and up. Still an answer, just an irrelevant one.

Filter 2: a distance threshold. Every real match in my data scored between 0.186 and 0.261. Everything unrelated scored 0.318 or more. So a threshold around 0.30 catches "nothing relevant found."

But the threshold alone isn't enough. The wrong semicolon answer scored 0.231, comfortably inside it.

So retrieval needs both:

1. Trusted sources, so wrong answers can't be retrieved
2. A relevance threshold, so unrelated answers aren't used
3. And when nothing passes both: "I don't have a trusted answer."

Time filtering worked the same way. WHERE noted_at >= '2026-11-09' removed Grok's Nov 8 async explanation, and the remaining results were all weak.

Limits

12 explanations, 5 questions, one embedding model, and reviews done by me. The 0.30 threshold fits this data only. It's a demonstration, not a benchmark.

Full rankings, data, and scripts: db/fix-notes-results.md

Next

Builders Week ends Thursday. After that, issue #1: behaviour checks, so a fix only counts if it builds and behaves the same. Today's lesson applies there too: just because the AI's answer looks right doesn't mean it is.

How it was built: the schema, embedding script, and analysis were worked through with Claude; I ran everything on my laptop and in the Tiger Data console, and every number comes from those runs.