Exporting PDFs to Clean Markdown: Bridging Structured PDFs and LLMs

PDFs remain the most common format for business reports, research papers, and documentation. However, they are also the most notoriously difficult to parse. Traditionally, developers have written complex parsers to dump PDFs to JSON or plain text.

But with the explosion of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG), a new standard format has emerged: Markdown.

Here is why we built the PDF-to-Markdown export capability in Datagrunt, the value it brings to modern AI pipelines, and how it works under the hood.

Why Markdown? The LLM Connection

When preparing PDF data for an LLM or vector database, raw text extraction is usually insufficient. If you strip away headers, lists, tables, and images, you lose the semantic structure of the document. A table of financial metrics becomes a meaningless jumble of floating numbers; headers blend into body paragraphs; reading order is lost.

HTML preserves structure, but it introduces massive token overhead and noise.

Markdown is the sweet spot:

1. Semantic Hierarchy: It uses clean, lightweight syntax (#, -, **) to represent headers, lists, and emphasis, which LLMs understand natively.
2. Table Preservation: It represents tables in plain ASCII grid layouts, allowing LLMs to trace rows and columns accurately.
3. Multimodal References: It inserts clean ![image](path) links where graphics exist, allowing multimodal pipelines to associate text with extracted visual assets.
4. Token Efficiency: It is extremely lightweight, leaving more of the LLM’s context window available for actual content rather than markup boilerplate.

How It Works in Datagrunt

Using Datagrunt, exporting a PDF to Markdown (complete with extracted, deduplicated images) requires only a few lines of code:

from datagrunt import PDFWriter

# Initialize the PDFWriter (using the default, permissive PDFium engine)
writer = PDFWriter("quarterly_report.pdf")

# Export to Markdown and write images to a dedicated directory
writer.write_markdown(
    export_filename="output.md",
    image_output_dir="extracted_images"
)

Behind this simple API, a multi-stage pipeline is executed:

[ PDF Document ] 
       │
       ▼
[ Page Elements Extraction ] (PDFium / PyMuPDF)
       │
       ▼
[ PageLayoutSorter (XY-Cut) ] (Column & reading-order reconstruction)
       │
       ▼
[ Unified Schema Assembler ] (Classifying paragraphs, headers, and lists)
       │
       ▼
[ Image MD5 Deduplication ] (Collapsing duplicate assets)
       │
       ▼
[ Markdown Formatter ] (Building tables, list tags, and image links)
       │
       ▼
[ Markdown Output ] (.md file)

1. Layout-Aware Column Reconstruction

Most PDF engines extract text line-by-line horizontally. In a two-column PDF, this reads the first line of column one, then the first line of column two, turning the text into alphabet soup.

Datagrunt’s PageLayoutSorter runs a recursive XY-Cut algorithm to split columns, detect vertical gutters, and isolate spanning headers. This ensures the output Markdown reads in natural, human-like column order.

2. Table Formatting

When a table is identified, the parser extracts cells using a line-and-spacing detector. Instead of flattening this data, the Markdown formatter constructs a clean markdown table:

| Faction | Difficulty | Complex Rules |
| :--- | :--- | :--- |
| Marquise de Cat | Easy | Keep |
| Eyrie Dynasties | Medium | Decree |
| Woodland Alliance | Hard | Sympathy |

3. Smart Image Deduplication

Documents frequently reuse header graphics, logos, or icons. Saving these repeatedly clutters the images directory.

Datagrunt reads each extracted image as raw bytes and checks its uniqueness using an MD5 checksum. If an image is byte-identical to one already extracted, the duplicate file is deleted from disk, and the markdown image link ![image](path) is updated to point to the existing file.

Performance Parity

To verify this capability, we benchmarked the Markdown exporter over three highly complex, graphically dense multi-page rulebooks (76 pages total, with image saving enabled):

• Speed: pdfium_structured compiled the 76 pages into formatted Markdown files and extracted all images in 20.41 seconds (averaging ~260ms per page), running 3x faster than PyMuPDF (63.69 seconds).
• Completeness: On complex documents, Datagrunt’s structured layout assembler extracted 10% to 65% more text than PyMuPDF (which encountered call stack recursion limits on multi-column pages).
• Quality: Structured tables were successfully preserved, and duplicate background textures were collapsed using MD5 checks to keep the image folders clean.

By converting PDFs to structured Markdown in seconds, Datagrunt provides the cleanest, fastest pathway to feed complex manuals, reports, and books directly into RAG search indexes and LLM prompt windows.