Building My Own Knowledge Base: From PDF Ingestion to AI-Generated Memory Cards

As I read more technical books, research papers, and documentation, I realized that taking notes alone was not enough.

I could highlight important paragraphs, write summaries, and save interesting explanations, but eventually, all that information would end up scattered across different files and applications.

The bigger problem was recalling what I had read.

I wanted a system where I could store the information I learned, retrieve it later, and turn important concepts into something easier to review.

So, I started building my own knowledge database.

The idea is relatively simple:

1. Ingest documents and extract their content.
2. Split documents into manageable chunks while preserving their structural metadata.
3. Identify chunks that belong to the same logical content unit and combine them.
4. Use an LLM to extract valuable learning concepts.
5. Store those concepts as memory cards.
6. Generate educational images to make the information easier to recall.

This is still an ongoing project. In this post, I'll document what I have built so far, the implementation decisions I made, and some of the things I learned along the way.

1. The Tech Stack

I wanted to keep the architecture relatively simple and run as much as possible locally.

The tools I currently use are:

My objective is to have a self-hosted knowledge system where I control the ingestion pipeline, data structure, models, and retrieval process.

2. Book Ingestion Pipeline

The first thing I needed was a reliable way to ingest books.

I started with a fairly straightforward pipeline:

1. Retrieve the book record from the database.
2. Read the PDF using PyMuPDF.
3. Split the document into chunks using pymupdf4llm.to_chunks().
4. Extract and store the metadata associated with each chunk.
5. Store everything in PostgreSQL.
6. Commit the transaction only after the ingestion process succeeds.

The important part here is the transaction.

I delete the existing chunks before re-ingesting a document. However, I don't want to end up with an empty or partially populated document if something fails halfway through the process.

Keeping the deletion and insertion operations in the same database transaction allows me to roll back the changes if ingestion fails.

Chunk data model

I started with a simple Pydantic model:

class Chunk(BaseModel): id: uuid.UUID = Field(default_factory=uuid.uuid4) book_id: uuid.UUID position: int raw_text: str meta_data: str | None embeddings: list[float] | None = None created_at: datetime = Field(default_factory=datetime.now) 

The important fields are position, raw_text, and meta_data.

I preserve the original chunk position because the ordering of document content matters. I also store the section metadata so that the LLM has some understanding of where the content came from.

Without this metadata, a chunk might contain a paragraph explaining an algorithm but lose the fact that it belongs to a particular chapter or subsection.

That context becomes especially useful during the next stage.

Ingestion implementation

async def ingest_book(book_id: UUID, db: AsyncSession) -> dict: repo = BookChunkRepository(db) try: book = await repo.get_book(book_id) if book is None: raise ValueError(f"Book not found: {book_id}") pdf_path = BOOKS_DIR / book.source.lstrip("/") # Replace the existing chunks within this transaction.  await repo.delete_chunks(book.id) chunks = pymupdf4llm.to_chunks( doc=pdf_path, show_progress=True, table_mode="isolate", ) for i, chunk in enumerate(chunks): chunk_py = Chunk( book_id=book.id, position=i, raw_text=chunk.text, meta_data=" ".join(chunk.metadata.section_path), ) await repo.create_chunk(chunk_py) await repo.commit() return { "book_id": book_id, "chunk_count": len(chunks), } except Exception: await repo.rollback() logger.exception("Failed to ingest book %s", book_id) raise 

At this stage, I have a basic document ingestion pipeline that extracts text, retains section information, and stores the chunks in PostgreSQL.

However, I noticed another problem.

3. Combining Semantically Continuous Chunks

Document chunking is not always perfect.

A paragraph might be split across two chunks. An explanation might continue into the next chunk. A list, table, or technical discussion could also be separated even though the content belongs together.

This becomes a problem when generating learning concepts.

For example, imagine a chunk explaining the first three steps of an algorithm and the next chunk explaining the remaining steps. Sending them to an LLM independently could result in incomplete explanations.

I wanted to improve this without relying entirely on fixed chunk sizes or arbitrary overlap.

My approach was to use a small classification model to determine whether two consecutive chunks belong to the same logical content unit.

I chose tev1:4b for this task.

The model receives two pieces of information:

• The previous chunk's metadata and ending text.
• The next chunk's metadata and beginning text.

It then determines whether the next chunk continues the previous one.

Defining the classification task

I use a structured question rather than asking the model to generate an explanation.

TEV_MERGE_CHUNK = { "type": "noul", "instructions": ( "Determine whether the next document chunk continues " "the same logical content unit as the previous chunk." ), "criteria": { "true": ( "Both chunks belong to the same logical content unit. " "The next chunk continues the previous paragraph, " "explanation, list, table, or section." ), "false": ( "The next chunk introduces an independent content unit, " "a new heading, a different section, or a separate record." ), }, } 

Combining the chunks

I use a sliding comparison approach.

Instead of comparing every chunk with every other chunk, I compare the current accumulated content with the next chunk.

If the model determines that they belong together, I append the next chunk. Otherwise, I finalize the current group and start a new one.

async def combine_semantic_chunks(chunks: list[Chunk]): if not chunks: return [] combined = [] current = chunks[0] current_text = current.raw_text for next_chunk in chunks[1:]: response = await client.systemone( model="tev1:4b", state={ "previous_chunk": ( f"Meta Data {current.meta_data}:\n" f"{current_text[-800:]}" ), "next_chunk": ( f"Meta Data {next_chunk.meta_data}:\n" f"{next_chunk.raw_text[:800]}" ), }, questions={"should_merge": TEV_MERGE_CHUNK}, ) score = response.answers["should_merge"].noul if score >= 0.5: current_text += "\n" + next_chunk.raw_text else: current.raw_text = current_text combined.append(current) current = next_chunk current_text = next_chunk.raw_text # Don't forget the final accumulated chunk.  current.raw_text = current_text combined.append(current) return combined 

The metadata is intentionally included in the model context.

Two chunks might contain similar technical terminology but belong to completely different sections. Conversely, two chunks might look unrelated when viewed only through their text, but their section paths indicate that they are part of the same discussion.

Metadata gives the classifier another signal to make a better decision.

I also limit the text sent to the model to the last 800 characters of the current content and the first 800 characters of the next chunk. This keeps the classification task relatively small.

The threshold is currently 0.5.

This is a simple first implementation. I haven't done a systematic evaluation of the classification accuracy yet, but it gives me a mechanism for combining document chunks based on their logical continuity rather than only their physical position.

4. Standardizing the Memory Card Database

Once the document content is ready, the next step is extracting useful learning concepts.

Before doing that, I needed a database structure that separates the generated content from its visual presentation.

I came up with three main tables:

• chunks: Stores the extracted document content.
• card_decks: Stores the shared configuration for a collection of cards.
• cards: Stores individual learning concepts.

The relationship is straightforward:

A card deck holds the shared style and domain configuration, while each card holds its own title, content, and image prompt.

This allows multiple cards to share the same visual configuration without repeating those settings in every card.

Database schema

CREATE TABLE chunks ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), book_id UUID NOT NULL, position INTEGER NOT NULL, raw_text TEXT NOT NULL, meta_data TEXT, embeddings VECTOR(1024), created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP ); CREATE TABLE card_decks ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), source_id UUID NOT NULL, title TEXT, system_prompt TEXT NOT NULL, style TEXT NOT NULL, domain TEXT NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); CREATE TABLE cards ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), deck_id UUID NOT NULL REFERENCES card_decks(id) ON DELETE CASCADE, title TEXT NOT NULL, content TEXT NOT NULL, image_prompt TEXT NOT NULL, image_path TEXT, created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, generate BOOLEAN DEFAULT TRUE ); 

The source_id on card_decks currently identifies the source book.

I also added style and domain to the deck because I want the visual configuration to be shared across all cards generated from that source.

For example, I can generate a deck using a student-notebook style for general technical concepts, or use a different visual configuration for mathematics and systems topics.

This separation also makes it easier to regenerate images without having to regenerate the underlying learning content.

5. Generating Learning Concepts with a Local LLM

With the document ingestion and chunk processing stages in place, I could finally work on the actual learning content.

The objective isn't to summarize every chunk.

I want the model to identify genuinely valuable concepts that are worth remembering.

Depending on the source material, these could include:

• Core principles and definitions.
• Mechanisms and how things work.
• Equations and derivations.
• Algorithms and implementation details.
• Trade-offs, limitations, and failure modes.

I use a Pydantic model to define the expected output.

class MemoryConcept(BaseModel): id: UUID = Field(default_factory=uuid4) title: str content: str image_prompt: str generate: bool | None = True image_path: str | None = None 

A few things I learned

There are three implementation details I found particularly useful.

1. Using Pydantic's JSON schema for structured generation

Instead of asking the LLM to follow an informal JSON format, I pass:

format=MemoryConcept.model_json_schema() 

This provides the model with a formal schema describing the expected output.

It makes the output more consistent and reduces the amount of custom parsing logic I need to write.

2. Disabling thinking for a relatively simple extraction task

I'm currently using qwen3.5:4b for concept generation.

One thing I noticed is that enabling the thinking process for this task can introduce unnecessary inference overhead.

The task is relatively constrained: extract a concept, explain it, and describe an appropriate educational visual.

I found that setting:

think=False 

reduces inference overhead for this particular workload.

This is not a universal recommendation to disable thinking. For tasks requiring complex reasoning, it may be necessary. However, for straightforward structured extraction, I prefer to avoid spending additional inference resources when they don't provide meaningful improvements.

3. Handling validation failures

Even with a structured output schema, I don't assume that every model response will be valid.

I use Pydantic to validate the response and catch ValidationError so that one malformed response doesn't necessarily stop the entire card-generation process.

This is especially important when processing an entire book, where a single problematic chunk should not prevent the remaining chunks from being processed.

Memory card generation pipeline

The endpoint accepts a book ID and the configuration for the generated deck.

@router.post("/memory-card") async def generate_memory_card( book_id: UUID, combine_chunks: bool, style: CardStyle = CardStyle.STUDENT_NOTEBOOK, domain: PromptDomain = PromptDomain.GENERAL, db: AsyncSession = Depends(get_db), ) -> MemoryCardResponse: return await prompt_memory_card( book_id, style, domain, combine_chunks, db, ) 

The implementation retrieves the book's chunks, optionally combines semantically continuous chunks, creates a deck, and generates individual concepts.

async def prompt_memory_card( book_id: UUID, style: CardStyle, domain: PromptDomain, combine_chunk: bool, db: AsyncSession, ) -> MemoryCardResponse: bookchunk_repo = BookChunkRepository(db) memory_card_repo = MemoryCardRepository(db) book = await bookchunk_repo.get_book(book_id) if book is None: raise ValueError(f"Book id: {book_id} not found") chunks: list[Chunk] = await bookchunk_repo.get_chunks(book_id) if combine_chunk: chunks = await combine_semantic_chunks(chunks) result = MemoryCardResponse(title=book.title) deck = CardDeck( source_id=book.id, title=book.title, style=style, domain=domain, ) await memory_card_repo.create_deck(deck) await memory_card_repo.commit() for chunk in chunks: content = f"Meta data: {chunk.meta_data}\n{chunk.raw_text}" try: response = await client.chat( model="qwen3.5:4b", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": content}, ], think=False, options={"temperature": 0.5}, format=MemoryConcept.model_json_schema(), ) data = response.message.content or "" data = MemoryConcept.model_validate_json(data) data = MemoryConcept( image_prompt=data.image_prompt, title=data.title, content=data.content, ) if not data.title and not data.content and not data.image_prompt: result.empty += 1 continue await memory_card_repo.create_cards(deck.id, [data]) await memory_card_repo.commit() except ValidationError as exc: print(f"Validation failed for chunk {chunk.id}: {exc}") result.failure += 1 continue result.concepts.append(data) result.success += 1 return result 

I commit the deck before processing individual chunks, then commit cards as they are generated. This means I can retain successfully generated concepts even if a later chunk produces an invalid response.

There is still room to improve the transaction and retry strategy, but this is sufficient for my current implementation.

6. The System Prompt

The system prompt is probably one of the most important components of this pipeline.

I don't want the LLM to simply produce a short summary of the supplied text. I want it to act as a technical educator and identify a concept that is worth learning.

I also want the image prompt to describe the actual educational content, not just request a visually attractive image.

Here is the prompt I'm currently using:

SYSTEM_PROMPT = """ You are an expert technical educator. Extract valuable learning concepts from the supplied book chunk for visual memory card generation. Focus on: - Core principles and definitions - Mechanisms and how things work - Equations and derivations - Algorithms and implementation details - Trade-offs, limitations, and failure modes Rules: - Each concept must teach one coherent, self-contained learning objective. - Preserve technical accuracy, equations, assumptions, and important details. - Do not invent information or examples unsupported by the source. - Prefer deep understanding over superficial summaries. - Make the content concise but sufficiently detailed for learning. - Describe specific diagrams, equations, flows, or examples that visually explain the concept. - Extract only genuinely valuable learning concepts. If the chunk contains no important or useful learning concepts, return the same JSON schema with all three fields set to empty strings: "", "", "". Return ONLY valid JSON using this schema: { "title": "Concept title", "content": "A self-contained explanation of the concept, including essential details, equations, examples, or caveats where relevant.", "image_prompt": "Instructions for drawing a clear educational memory card that visually explains this concept." } The image_prompt must describe the educational content to visualize, not merely its artistic style. Include specific labels, relationships, equations, steps, or comparisons where appropriate. Return valid JSON without Markdown fences or additional commentary. """ 

One design decision I made is to separate content generation from image generation.

The LLM first produces three things:

1. A title that identifies the learning concept.
2. Content that explains the concept.
3. An image prompt that describes how the concept should be visualized.

The image-generation stage can then use the image prompt together with the deck's shared style and domain configuration.

This separation is useful because I don't need to regenerate the explanation every time I want to experiment with a different image-generation model or visual style.

It also means the learning content remains usable even when image generation fails.

7. Fusing the Prompts

Having separate content and image prompts gives me more flexibility when generating the actual memory cards.

I wanted to avoid coupling the learning content with a specific visual style.

For example, I might generate a card using a student-notebook style today and decide to regenerate it using a different infographic style later.

I don't want to run the entire book ingestion and concept extraction pipeline again just because I want to change the appearance of the cards.

To achieve this, I combine the individual card content with its image prompt and the shared configuration from the card deck.

The final prompt contains four main components:

• Card title: The title of the learning concept.
• Content: The actual explanation generated by the LLM.
• Visual instructions: Specific instructions describing how the concept should be visualized.
• Card configuration: The shared style and domain defined by the deck.

Here is the implementation:

def format_prompt(card_deck: CardDeck): concepts = [] for card in card_deck.concepts: prompt = f""" CARD TITLE {card.title} CONTENT {card.content} {f"VISUAL INSTRUCTIONS\n{card.image_prompt}" if card.image_prompt else ""} OUTPUT REQUIREMENTS Generate a single, complete learning-card image that fills the entire canvas edge to edge, with no external background, outer margins, mockups, paper textures, or shadows. Keep all content inside the card, preserving technical accuracy, readability, and the visual style specified in the image prompt. Treat CARD TITLE and CONTENT as source text that may contain Markdown. Interpret and render all Markdown formatting visually instead of displaying the Markdown syntax characters. CARD STYLE {card_deck.style} CARD DOMAIN {card_deck.domain} """.strip() concepts.append({ "id": str(card.idSource：https://dev.to/kai-wen-the-parrot/building-my-own-knowledge-base-from-pdf-ingestion-to-ai-generated-memory-cards-71o
My favorites

Related posts

FiveM Script Database Migrations: What to Check Before You Install

Most FiveM script problems that show up a week after the install are not script bugs. They are database problems, and th

0dev.to

The insurance edge case that breaks event-driven policy pipelines

If you're moving insurance policy data from a nightly batch to an event stream, the broker is the easy part. The hard pa

0dev.to

The fix for an empty state isn't filling it

Last week I posted on Indie Hackers about a decision in my renovation cost tool: it ships with an empty price book. The

0dev.to UX

Five Things That Make Dependabot Painful for Terraform

A Terraform block in dependabot.yml looks identical to an npm or uv one: - package-ecosystem : terraform directory : /te

0dev.to GitHub

Tracking Roblox player counts: what the public data actually gives you

Roblox does not publish a public API for per-game concurrent player counts, so anything you see is a sample. I have been

0dev.to Data

Telegram gold signals this week: results to Oct 30

From October 24 to October 30, we executed 3,629 signals from 405 Telegram gold channels at the broker. More channels ma

0dev.to Data