PDF Parser Shootout
Data & Processes • Open-source benchmark
Repo: github.com/miraculix95/pdf-parser-shootout
What it is
A hands-on benchmark of three modern document parsers — LlamaParse, Mistral OCR and Docling — on the case most parser comparisons avoid: a real European magazine cover (vertical rotated text, hand-lettered headlines, multi-column layout, mixed languages). It matters because a parser’s output is the input to a RAG pipeline: if it drops headings or scrambles reading order, no amount of downstream tuning recovers it. Two of the three parsers broke on the cover.
What it found
| Parser | Cover text recall | TOC reading order | Vertical text | Latency (10 pp) | Cost (10 pp) |
|---|---|---|---|---|---|
| LlamaParse Agentic | complete | correct | extracted | ~30 s | $0.13 |
| Mistral OCR | most | correct | missed | ~5 s | $0.01 |
| Docling (cpu) | lost | scrambled | missed | ~216 s | $0 |
The deliverable isn’t just a table — it’s a routing rule: for editorial / brand / photo-heavy PDFs, LlamaParse is the only one that survives; for standard documents (contracts, invoices, papers) Mistral OCR is the price-performance sweet spot (fast, cheap, EU-hosted). The repo ships every raw parser output so anyone can rerun the test in five minutes.
Why it’s here
It shows the part of AI work that isn’t the model: choosing the right tool with evidence, on real-world data, and writing the finding up clearly enough that a team can act on it. (It also directly informs the Doc → Insight demo’s OCR path.)
Stack
Python · LlamaParse · Mistral OCR · Docling · reproducible test harness