PDF Parsing

PDFReader and PDFWriter parse a PDF into in-memory Python objects or write the results to disk. Both require the pdf extra. See the Readers and Writers reference for full signatures.

reader = PDFReader('report.pdf')                    # String path (PDFium engine)
reader = PDFReader('report.json')                    # Load pre-parsed JSON path directly (instant)
reader = PDFReader(doc_dict)                         # Load in-memory parsed dict directly (instant)

Each reader or writer parses the source once and caches the result, so mixed calls on the same object (to_dicts() then to_dataframe(), or write_json() then write_markdown()) never re-parse the PDF.

Encrypted (password-protected) PDFs raise a clear ValueError on every engine. A zero-byte PDF never raises: readers return empty objects ({} / empty DataFrame) and writers produce empty output files.

Choosing an Engine

PDFReader and PDFWriter accept an engine argument (case-insensitive; an unrecognized name raises a ValueError listing the valid engines). PDFium is the default — it is permissively licensed (BSD-3 / Apache-2.0) and faster on text-heavy documents — with PyMuPDF (AGPL-3.0 / commercial) available as an alternative. Both engines produce the same unified element schema, so output is interchangeable.

# Default: PDFium, unified element schema.
reader = PDFReader('report.pdf')

# Lean, fast PDFium mode: text + positioned text objects + images, no table
# detection (~20-80x faster when you don't need structured tables).
reader = PDFReader('report.pdf', native=True)

# Alternative engine.
reader = PDFReader('report.pdf', engine='pymupdf')

native (PDFium only, default False) switches to the lean native schema instead of the default unified element schema. The workers argument applies to the PDFium engine only; PyMuPDF always runs sequentially (see Concurrency & Environments).

The Unified Element Schema

to_dicts() returns the full parsed document. The envelope:

{
  "document": {
    "source": "report.pdf",
    "total_pages": 12,
    "processing_id": "proc_py_1752861600",
    "pipeline_type": "pure_python_local_v1",
    "errors": null,
    "pages": [ ... ]
  }
}

errors is null, or a list of "Page N: <message>" strings for pages that failed to parse — a bad page is recorded here and skipped, never aborting the batch (see Error isolation).

Each page:

{
  "page_number": 1,
  "width": 612.0,
  "height": 792.0,
  "classification": "text_only",
  "elements": [ ... ]
}

• classification is "text_only" (text layer, no images or tables), "scanned" (no text layer, has images — OCR runs), or "mixed" (everything else).
• A warnings key appears only when something soft-failed on the page (e.g. OCR unavailable); the page and its other content are always kept.
• Coordinates throughout are PDF points (1/72 inch) with the origin at the top-left of the page.

Each element:

{
  "id": "elem_01_001",
  "type": "header",
  "content": "Quarterly Report",
  "page": 1,
  "position": {"x": 72.0, "y": 80.1, "w": 210.5, "h": 24.0},
  "confidence": 1.0,
  "metadata": { ... }
}

id is elem_<page>_<n>, both 1-based and zero-padded. Elements within a page are layout-sorted into reading order — a recursive column detector (XY-cut) recognizes multi-column layouts and orders each column’s content top-to-bottom before moving to the next, instead of naively interleaving across columns.

Element types and their metadata

Headings are classified relative to each page’s median font size, so the same absolute size can be a header in one document and body text in another. confidence is 1.0 for native-text elements and the Tesseract confidence (0–1) for OCR text.

Text that falls inside a detected table region is not duplicated as separate text elements — it appears only in the table’s content.

The Native PDFium Schema

With native=True, PDFium’s raw output is kept — full page text, positioned text objects, and images — with no text-block classification and no table detection. This is ~20–80× faster when you don’t need structured tables:

{
  "document": {
    "source": "report.pdf",
    "page_count": 12,
    "errors": null,
    "pages": [{
      "page_number": 1,
      "width": 612.0,
      "height": 792.0,
      "text": "full page text...",
      "text_objects": [{"text": "...", "bbox": [72.0, 80.1, 282.5, 104.1], "position": {"x": 72.0, "y": 80.1, "w": 210.5, "h": 24.0}, "font_size": 11.0}],
      "images": [{"file": null, "bbox": [...], "position": {...}, "px_width": 800, "px_height": 600, "extracted": false}],
      "ocr": false
    }]
  }
}

The OCR fallback still applies in native mode: a page with no text layer is OCR’d, its ocr flag is set true, and the recovered lines are appended to text_objects (with font_size: null).

Flattened Output (DataFrame / Arrow / JSONL)

to_dataframe(), to_arrow_table(), and write_json_newline_delimited() flatten the document to one row per element. The columns depend on the schema:

Non-scalar values are JSON-encoded strings in the flat form — a unified table’s content is its rows as a JSON string, and metadata is always a JSON string; parse them back with json.loads when needed.

OCR for Scanned Pages

OCR runs automatically on a page only when the page has no text layer and does contain images (classification: "scanned"), on either engine. It requires the Tesseract system binary (see optional PDF support); native-text PDFs never invoke it.

• Render DPI is chosen per page: 150 DPI normally, dropping to 75 DPI for large-format pages (over 1,500 points on a side) to bound memory.
• OCR text lands as body_text elements whose confidence is Tesseract’s per-line confidence (0–1) and whose metadata records ocr_engine: "tesseract" and word_count.
• If OCR is unavailable or fails, the page is never discarded — it keeps its already-extracted images and tables, and a page-level warnings entry records what happened.

Table Detection

Table detection (shared by both engines, via pdfplumber) runs in the default unified mode — never in native mode — and only on pages where it can matter: pages with vector line drawings, or scanned pages. Extracted tables carry their cell grid in content and rows / columns / has_header_row in metadata.

Dropping layout-artifact tables

On graphically dense PDFs, line-based detection can pick up decorative boxes and rule lines as spurious “tables”. Pass drop_layout_tables=True to any conversion or writer method to discard tables smaller than 2×2 (i.e. 1×N or N×1):

reader.to_dicts(drop_layout_tables=True)
writer.write_json('out.json', drop_layout_tables=True)

It is off by default — Datagrunt preserves everything unless told otherwise.

Filtering Small Embedded Images

By default, embedded images smaller than 40 px on either side are dropped as layout artifacts (rule lines, separators, icon slivers). Pass the keyword-only min_image_dimension argument to PDFReader or PDFWriter to change that threshold:

reader = PDFReader('report.pdf', min_image_dimension=15)   # keep logos, signatures
reader = PDFReader('report.pdf', min_image_dimension=200)  # keep only large figures
reader = PDFReader('report.pdf', min_image_dimension=0)    # keep every image

The threshold applies to both PDF engines and every conversion/output method. It must be a non-negative int — negative values, non-integers, and booleans raise immediately at construction.

Error Isolation & Robustness

PDF extraction is always 100% complete with per-page error isolation:

• A page that fails to parse is recorded in the document-level errors list ("Page N: <message>") and skipped — the rest of the document still parses.
• A soft failure within a page (e.g. OCR unavailable) keeps the page with whatever was extracted, plus a page-level warnings entry.
• A page that fails to render in render_pages_as_images is logged and skipped, never aborting the batch — in both the sequential and parallel paths.
• Encrypted (password-protected) PDFs raise ValueError with a clear message on every engine — including PyMuPDF, which would otherwise silently open them and read every page as empty.
• Malformed layouts are bounded: corrupt coordinates and pathological column structures degrade to coarser ordering instead of hanging or exhausting memory.

Loading Pre-Parsed Results

Both classes accept a pre-parsed document dict or a .json file path in place of a PDF — parsing is skipped entirely, so generating a DataFrame, Arrow table, or Markdown from previous results is instant:

doc = PDFReader('report.pdf').to_dicts()          # parse once
PDFWriter(doc).write_markdown('report.md')        # instant — no re-parse

writer = PDFWriter('output.json')                 # or from the JSON file
writer.write_markdown('report.md')

Guardrails on this path:

• .json documents are capped at 50 MB (the file is loaded fully into memory); larger files raise ValueError.
• A JSON file whose top level is not an object raises a clear ValueError instead of failing deep in conversion.
• render_pages_as_images raises ValueError on a dict/JSON-backed writer — a parsed document contains no page graphics to rasterize; keep the source PDF for rendering.

Writing Results to Disk

PDFWriter produces JSON, JSONL, Markdown, extracted images, or rasterized page images:

writer = PDFWriter('report.pdf')
writer.write_json('output.json', image_output_dir='extracted_images')
writer.extract_images(output_dir='extracted_images')
writer.render_pages_as_images(output_dir='page_images', dpi=300, image_format='png')

Passing an explicitly empty or whitespace-only filename raises ValueError (that’s almost always a bug); pass None to use the default. image_format accepts png, jpg, or jpeg (case-insensitive); anything else raises ValueError before any file is written. Page and image numbers in filenames are 1-based and zero-padded so directory listings sort in page order. render_pages_as_images honors workers on the PDFium engine (PyMuPDF renders sequentially) and rasterizes whole pages — unlike extract_images, which pulls out the images embedded inside the PDF.

Markdown rendering

write_markdown maps unified elements to Markdown structure:

Element content that begins with a Markdown metacharacter is escaped, so document text can never inject its own headings or links. In native mode, the Markdown is the page text split into paragraphs.

Image Deduplication

When images are written to disk, byte-identical duplicates (common with repeated icons or backgrounds) are collapsed to a single file and all references are repointed to it. Pass dedupe_images=False (or dedupe=False for extract_images) to keep every copy. Deduplication only ever removes files inside the image output directory it manages — it resolves paths defensively and is symlink-safe, so it can never touch files elsewhere on disk.