I Built a Japanese PDF RAG Demo Where You Can See What the Model Looked At

This article was originally published in Japanese on Zenn.

Introduction

Hi, I'm Kohei Matsunobe, a second-year student in the online (correspondence) program at ZEN University. I'm working toward becoming an AI engineer, and I learn by building LLM-related systems with my own hands.

When you ask an LLM about a document, you usually get back an answer in fluent, plausible language. What bothered me was the question: "Which part of the document is that answer actually based on?" I didn't want to judge an answer by whether it sounded right. I wanted to build something where it's clear what the model looked at when it answered.

So I built a RAG demo: you upload PDFs, ask questions in Japanese, and get answers with sources (file name and page number).

https://github.com/koheip/rag-doc-qa

This post isn't a usage guide. Instead, it focuses on:

• What I decided, and where, in order to show sources accurately
• What I learned once I evaluated it and looked at the numbers

Setup instructions are in the README, so please check there.

What I built

It's a Streamlit app. You register PDFs and ask a question; it searches for relevant passages and answers using only those passages as evidence. Answers carry numbers like [1], and each number maps to "file name p.page". You can also see the retrieved chunks and their similarity scores in the UI.

• Retrieval: multilingual-e5-small + FAISS (IndexFlatIP)
• Generation: Google Gemini (official SDK google-genai). The default is gemini-3.5-flash-lite, and you can switch to gemini-3.8-flash from the sidebar
• Without an API key, it runs in "extractive mode," which shows the retrieved passages as-is

Here's the flow:

flowchart LR A[PDF] -->|pypdf| B[Text per page] B -->|Japanese-aware chunking| C[Chunks] C -->|"passage: " + e5| D[(FAISS)] Q[Question] -->|"query: " + e5| D D --> R[top-k chunks + sources] R -->|with key| L[Answer with Gemini] R -->|no key| E[Extractive mode]

Decision 1: Chunks never cross page boundaries

The decision that mattered most for showing accurate page numbers was how I split text into chunks.

I wrote the chunker myself instead of using a library. Japanese has no spaces between words, so I measure length in characters. Then I split into roughly 200-character pieces, preferring sentence boundaries like 「。」, and overlap a bit of the previous chunk's tail so context isn't cut off at the seams.

And chunks never cross pages. If a chunk spans two pages, you can't say which single page it came from. Since the whole point of the app is showing sources, I prioritized citation accuracy over retrieval convenience here.

I wrote it myself because I wanted Japanese-specific splitting in a form I could actually explain. If you leave it to a library, it's harder to explain in your own words why the text was cut where it was.

Decision 2: Make it say "I don't know," and cite only what it used

In the generation prompt, I number the retrieved chunks, attach their sources, and give instructions like this (from src/llm.py; the prompt itself is in Japanese):

SYSTEM_PROMPT = """あなたは社内ドキュメントに関する質問に答えるアシスタントです。 以下のルールを必ず守ってください。 1. 回答は【参考資料】に書かれている内容だけを根拠にしてください。あなた自身の知識で補ってはいけません。 2. 【参考資料】に答えが書かれていない場合は、推測せずに「わからない」とだけ答えてください（この場合は資料番号を付けないでください）。 3. 回答の文末には、根拠にした資料の番号を [1] [2] のように付けてください。 4. 日本語で、簡潔かつ丁寧に答えてください。""" 

Roughly translated, the rules are:

1. Base your answer only on the provided reference material; don't fill in with your own knowledge.
2. If the answer isn't in the material, don't guess — just say 「わからない」 ("I don't know"), with no reference numbers.
3. End the answer with the numbers of the passages you relied on, like [1] [2].
4. Answer in Japanese, concisely and politely.

The key point is that I don't list every retrieved chunk as a "source." Only the numbers the answer actually references are shown as sources. Even if the top 4 chunks are passed in, if the answer used just one, only one source is shown. When the answer is 「わからない」, no sources are shown.

Temperature is set to 0.1, because I wanted answers that stay faithful to the material and don't wobble between runs.

Decision 3: Let people try it without an API key

Without an API key, the app runs in "extractive mode," showing the top chunks directly as the answer.

I added this so anyone could try it right away, but once I built it I found another benefit. Extractive mode is retrieval only, so before generation even happens, you can check whether the right passages were found at all. When the RAG answer is off, it became much easier to tell whether the problem is in retrieval or in generation.

Other choices

E5 expects a passage: prefix on documents and a query: prefix on questions, so I follow that. Vectors are 384-dimensional and normalized to length 1, so FAISS's inner product is exactly cosine similarity.

At this data size, a numpy matrix multiplication would be plenty. But with FAISS, when data grows I can swap the brute-force IndexFlatIP for an approximate index. The vector store is isolated in one file, so I can change it without affecting the rest.

What the evaluation taught me

I didn't want to stop at "it kind of works," so I made retrieval and answers measurable. The results below are what I wrote up in the README on 2026-10-21.

• Eval data: 30 questions on the bundled sample PDF (employment rules and an internal FAQ for a fictional company, 11 pages)
• Of the 30, 26 are answerable from the document and 4 are not. It also includes paraphrased questions that use different words from the document
• Retrieval is dense vector search (e5-small, top-k=4)

Retrieval: Hit@3 is high, Hit@1 is still low

With a chunk size of 200 characters (the one I adopted), retrieval on the 26 answerable questions looked like this:

Every correct passage is within the top 4, but it's ranked first only 73.1% of the time. In other words, the right answer often lands at rank 2–4.

Right now I pass the top 4 chunks to the LLM together, so this doesn't affect the answers. But if I want to pass fewer chunks to lower cost, Hit@1 needs to improve. The clearest thing the numbers showed me was that "the answer is correct" and "retrieval is good enough" are two different things.

Bigger chunks aren't automatically better

I compared chunk sizes of 150, 200, 300, and 400 characters, and 200 gave the best balance. The surprise was 300: Hit@3 on paraphrased questions dropped from 92.3% to 69.2%. My read is that one chunk ends up mixing several rules (articles), so the meaning of the vector gets blurry.

If I had chosen the size based only on the intuition that "more context should be better," I would have missed this.

Generation: this dataset doesn't separate models

With gemini-3.5-flash-lite (the default), all 26 answerable questions were answered correctly (26/26), and all 4 unanswerable questions were correctly refused with 「わからない」 (4/4). Latency was 0.56s at p50 and 0.72s at p95. Grading was done by a different LLM than the answering model, comparing each answer with a reference answer and scoring it 0/1 (LLM-as-a-judge).

I don't read this as "the model is perfect." I read it as the current dataset is too easy and has hit the ceiling. To see differences, I need harder questions — ones that span multiple passages, or that require calculation.

Limits of this evaluation

• 30 questions is small; a single question moves the score by about 4 points
• Both the dataset and the grading come from one person (me) and one LLM
• I also tried comparing against gemini-3.8-flash, but it was under heavy load (503) during evaluation and didn't respond, so I couldn't compare

Known limitations

Here are the limits of this demo that I found while building it:

• E5 similarity scores tend to cluster around 0.8–0.9, so it's hard to set a threshold like "below X means unrelated." That's why, for questions not covered by the document, I leave the decision to the LLM via the prompt and have it say 「わからない」
• With paraphrases like 給料 vs. 給与 (both meaning "salary"), the correct passage sometimes isn't ranked first
• Scanned-image-only PDFs need OCR, so they're not supported
• The index lives only in memory and disappears when the session ends (I prioritized simplicity for a demo)

Tests

Tests are written with pytest and cover chunking, retrieval, the LLM-related logic, and the evaluation metrics. Retrieval tests use a lightweight fake embedder built from character counts (FakeEmbedder), and generation tests use a fake LLM (FakeLLM), so you can check the main flow without downloading the model and without calling the API.

Next: hybrid search

To improve Hit@1, my next step is hybrid search that combines BM25 (keyword search) with vector search. Keyword search is better at proper nouns, numbers, and article numbers, so the idea is to pair it with vector search, which handles paraphrases better.

I haven't implemented it yet. I've made the retrieval method swappable and prepared the function specs and tests for src/bm25.py first. I plan to measure whether Hit@1 and MRR go up on the same 30 questions.

Wrap-up

• To show accurate page numbers, chunks never cross pages, and I wrote my own Japanese-aware chunker
• Only the numbers the answer actually references are shown as sources, and when the document doesn't cover the question, the model says 「わからない」
• In evaluation, Hit@3 was 96.2% but Hit@1 was 73.1%. Widening chunks to 300 characters dropped paraphrase Hit@3 from 92.3% to 69.2%
• "The answer is correct" and "retrieval is good enough" are different things, and I only saw what to do next once I put numbers on it

When building RAG, I found it helps to be able to check "what did it look at to answer?" both in the UI and in numbers, not just how the answer looks. All the code is public, so feel free to try it out.

https://github.com/koheip/rag-doc-qa