Evaluating DeepSeek-V4.1-Flash-1017 on a 60-Page PDF: What Text Extraction Misses

I recently deployed DeepSeek-V4.1-Flash-1017 across two NVIDIA DGX Spark systems and connected the model to Open WebUI. Once the deployment was working, I wanted to test a common research workflow: attaching a long PDF and asking questions about its tables and figures.

The most useful observation was not an overall accuracy score. It was how the same figure-dependent information was handled when the evidence was supplied within a PDF versus as individual PNG images.

I used a 60-page fictional report, ASTER-47. I preserved the original PDF-question prompt and response, and later recovered the original Open WebUI chat exports for five separate PNG-input questions. This article examines what those records establish. It is not a standardized benchmark or a controlled comparison of models.

The two-node installation and operating procedures are covered in the companion implementation article.

1. Evaluation environment

The serving configuration was based on MiaAI-Lab's two-DGX-Spark repository at the revision used for this deployment.

The 262,144-token context is a configured limit, not a measured effective context length. The document-test API payload and complete performance measurements were not archived, so this article does not report throughput or long-context accuracy.

2. Evaluation document and actual procedure

ASTER-47 is a custom-generated, completely fictional report about a multi-site virtual monitoring study. Its facility names, event counts, detector statistics, tables, and figures were generated for this test. None represents a real-world measurement or research participant.

The published evaluation package contains the original PDF (60 pages, 686,543 bytes), the 28-question set and answer key, generator and manifest, five figure-page PNGs, and preserved model responses.

The questions were organized into three groups:

• Q01–Q20: facts, arithmetic, and reconciliation across selectable text and tables.
• Q21–Q25: specific values and labels printed inside raster figures, intentionally not repeated in the selectable text.
• Q26–Q28: information the fictional report explicitly does not contain, such as measured GPU power or human participants.

For the recorded PDF run, I attached only the report PDF to Open WebUI, selected Using Entire Document, and submitted all 28 questions together in one message. The answer key was not attached. The original model reply was exported as both text and a six-page PDF.

The recorded Open WebUI session also showed file-tool activity (list_chat_files, query_chat_files, view_file, and grep_chat_files). Although Using Entire Document was selected, the exact retrieved text and final API payload were not archived. Thus, this is not a verified test of placing all 60 pages into a single model request.

An important methodological detail: the PDF-question prompt instructed the model not to invent unsupported values and explicitly warned that a PDF-to-text-only workflow was not expected to access raster pixels. The synthetic report repeated this warning. Therefore, the five figure-related abstentions cannot be taken as evidence that the model independently discovered a parser failure.

For the separate image condition, each of Figures 6–10 was uploaded as one PNG in its own Open WebUI chat, with one question. The recovered JSON exports identify the model as DeepSeek-V4.1-Flash-1017 and preserve each exact user prompt and complete final response. The attached-image SHA-256 hashes match the five PNG files in the repository.

3. Recorded outcomes

The PDF condition involved one combined 28-question interaction; the PNG condition involved five separate, single-image chats, with differently worded questions. These six interactions are not a controlled, identical-prompt comparison, and their scores should not be pooled into a general accuracy figure.

Example A: Reading a table and performing a calculation

Question 9 asked for precision, recall, and F1 to four decimal places using the fictional S2 confusion matrix on page 17:

• TP = 430
• FP = 224
• FN = 110
• TN = 16,156

An excerpt from the original model response reads:

Precision = 430/(430+224) = 430/654 = 0.6575 Recall = 430/540 = 0.7963 F1 = 2(430)/(2·430 + 224 + 110) = 860/1194 = 0.7203 

The calculations match the prepared key. This demonstrates successful extraction and arithmetic for this example, not statistical validation of the fictional detector.

Figure 1. Cropped from the actual six-page chat PDF export. The numerical results and wording were not edited.

The response also correctly distinguished the report's 12 illustrative case-log rows from the larger population of 60 positive events in Q20. These examples are more informative than simply listing the overall 20/20 count.

Example B: A figure value across two input conditions

Question 22 asked for the value at site FX × Packet in Figure 7 (page 26). The heatmap has the value 0.72 printed inside its pixels; that numeric cell is deliberately omitted from the surrounding selectable text.

Figure 2. The original fictional report's page 26. The cell value is visible in the raster figure.

In the retained PDF-only response, the model marked Q22 “Not accessible” and explained that the cell values were present in the raster image but absent from the text it could use. It withheld the requested numeric answer, as it did for Q21 and Q23–Q25.

Figure 3. Cropped from the actual six-page chat PDF export. The five figure-dependent responses are presented together.

This is an observation about the tested end-to-end workflow, not proof that the model lacks vision capabilities or that a particular parser discarded the images.

In the separate PNG-input chat for Figure 7, the user asked the model to examine the image directly and identify the FX × Packet value. The preserved answer reads:

I read the image directly. In Figure 7, the bottom row labeled FX intersects the Packet column (second column) at a cell annotated 0.72.

Exact value: 0.72

The four other recovered PNG answers also match their keys:

The public transcripts preserve the full prompts and final replies for all five PNG questions. This verifies the recorded answers, but the input modality was not the only difference between the two conditions.

Example C: Declining unsupported requests

Questions 26–28 requested real measured GPU power, actual geographical coordinates of a fictional site, and the number of human participants in a supposed field study.

The recorded model reply did not fabricate these facts. However, the fictional report expressly says there were no physical measurements, real facilities, or human-subject experiments. The initial user prompt also prohibited unsupported claims.

Thus, 3/3 describes agreement with an intentionally easy, explicitly signposted abstention test. It is not evidence of general hallucination resistance.

4. Interpretation and limitations

The experiment supports a practical distinction: the fact that a figure is visible in a PDF does not establish that its values are available in the tested question-answering workflow. In the recorded PDF chat, the model withheld the requested raster-only values; when the corresponding PNGs were uploaded separately, it provided the correct values.

The following details matter when interpreting that difference:

1. Synthetic and purpose-built material. ASTER-47 was designed with auditable text answers and deliberately image-only facts. It is not a representative sample of published research PDFs.
2. Different question procedures. The PDF condition used one 28-question prompt; the PNG condition used five separate chats with shorter, individually phrased questions. These results are descriptive, not a controlled paired modality benchmark.
3. Explicit hints. The PDF prompt and document warned about missing raster evidence and nonexistent real-world measurements. Therefore, the abstentions do not independently demonstrate parser-failure detection or general resistance to hallucination.
4. Uninspected internal payload. File tools were used, but the exact final model request and retrieved chunks were not archived. The observation does not identify the component responsible for the inaccessible figure values or prove that all 60 pages were present in a single context.
5. Ambiguity in the fictional report. Page 14 describes R0 calibration as using a training quantile, while page 15 presents validation-based R0 threshold selection. These may describe different steps, but their relationship is not explained. The model flagged this ambiguity in Q14; its reported threshold values still match the prepared answer key.
6. No repeated trials or performance benchmark. The article reports observed answers, not a measured error rate, confidence interval, token throughput, or 262,144-token context-quality result.

5. Materials and reproducibility

The published repository includes the original 60-page PDF, exact combined 28-question prompt, answer key, generator and manifest, five figure PNGs, and the unaltered PDF-chat TXT and six-page PDF export. It also contains the verbatim prompts and complete final responses from five PNG-input chats, extracted into public-facing Markdown and JSON files.

The public PNG transcripts omit account identifiers and model reasoning traces from the raw chat exports while retaining the original user prompts and final answers. The five attached PNGs were verified against the repository files using SHA-256 hashes.

Evaluation materials: ASTER-47 PDF, questions, answer key, figures, and recorded DeepSeek responses.

To reproduce the PDF condition, open a new Open WebUI chat with the same model and document setting, attach only the ASTER-47 PDF (not the answer key), submit the original combined 28-question prompt, and preserve the response before scoring it.

To reproduce the PNG condition, use five separate chats, each with the corresponding Figure 6–10 PNG and its original individual prompt. Save each answer verbatim before comparing it with the key. The supplied transcripts make it possible to check the five previously observed answers without repeating the experiment.

The article screenshots illustrate selected responses; the original exports and transcripts are the primary evaluation record.

Conclusion

In this recorded deployment, DeepSeek-V4.1-Flash-1017 matched the prepared key on all 20 text/table questions in the synthetic PDF, while withholding answers to five raster-only figure questions in the PDF chat. In five separate chats with the relevant PNGs attached directly, it correctly answered all five figure questions. The prompts and final answers for both conditions are now preserved.

Because the PDF prompt included explicit raster-related hints and the two conditions used different question formats, the results describe this workflow rather than a controlled modality comparison or a general model-accuracy score.

The practical takeaway is to verify whether a document-processing workflow makes the visual evidence needed for a question accessible, especially when interpreting scientific figures and diagrams.

AI disclosure: This article documents my own technical setup and results. An LLM was used to help draft, organize, and edit the English text. I reviewed the technical claims against the available original prompt, answer key, document, and chat exports.