PDF Parsing

PDFReader and PDFWriter read a PDF into a typed document — headings, paragraphs, key-value pairs, tables, lists, and figures, in reading order — and hand it back as Python objects, DataFrames, Markdown, SQL results, or files on disk. Both require the pdf extra. See the Readers and Writers reference for full signatures.

Datagrunt 5.0 reads every PDF with one engine, PDFium (via pypdfium2), and reads scanned pages with the Tesseract program. Each element carries its page, its bounding box, and the headings it sits under. Text a human reader can’t see is flagged, never dropped.

Installation

uv pip install "datagrunt[pdf]"

The extra installs pypdfium2 (PDFium) and Pillow. import datagrunt — and even from datagrunt import PDFReader — works without it. Constructing a PDFReader or PDFWriter without it raises an ImportError that names the install command:

from datagrunt import PDFReader  # imports without the extra

try:
    reader = PDFReader("report.pdf")
except ImportError as exc:
    print(exc)  # ... Install PDF support with: uv pip install "datagrunt[pdf]"

Scanned pages also need the Tesseract program; see Scanned pages and OCR.

Reading a PDF

from datagrunt import PDFReader

reader = PDFReader("report.pdf")

df = reader.to_dataframe()          # Polars: one row per element, in document order
arrow = reader.to_arrow_table()     # the same rows as a PyArrow table
rows = reader.to_dicts()            # the same rows as dicts
reader.get_sample()                 # the first 20 rows (n_rows=...)
markdown = reader.to_markdown()     # the whole document as Markdown, hidden text left out

# The typed document model
for table in reader.tables:
    print(table.scope_path, table.to_dataframe())
reader.headings, reader.paragraphs, reader.key_values, reader.lists, reader.figures
reader.sections, reader.pages, reader.metadata

# SQL over the element rows (an in-memory DuckDB connection per reader)
result = reader.query_data(f"SELECT kind, count(*) AS n FROM {reader.db_table} GROUP BY kind")
result.pl()                         # also .pa() and .dicts()

• Checked at construction, read on first use. The constructor checks its arguments before any work: a missing path raises FileNotFoundError; a directory without a manifest.json, a file that isn’t a regular file, or a file with no %PDF header in its first 1,025 bytes raises ValueError; a bad option raises ValueError or TypeError (see Options). The PDF itself is read the first time a property or method needs it, then cached for the reader’s life, so mixed calls never extract twice. A failed extraction is not cached: the next call tries again.
• Three kinds of source. PDFReader takes a PDF path (a str or os.PathLike), a bundle directory or its manifest.json, or a dict of the document JSON that PDFWriter.write_json writes.
• Nothing to close. A reader holds no file, PDFium handle, or process between calls. Its DuckDB connection opens on the first query_data call and is released automatically when the reader goes out of scope; close() or with PDFReader(...) as reader: releases it early, and the reader stays usable.

The document model

reader.document is the whole extraction as a PDFDocument: immutable, typed objects you can import from datagrunt (PDFDocument, Page, Section, Heading, Paragraph, KeyValue, Table, BulletList, Figure, PageImage, BBox, DocumentMetadata, PageQuality). The reader exposes its collections directly:

Every element has an element_id (p0001_e003: page 1, position 3 in its reading order), page_number, reading_order (0-based), bbox, scope_path, and hidden_reasons, plus to_markdown():

Coordinates are PDF points (1/72 inch) with the origin at the top-left of the page: bbox is a BBox with x0, y0, x1, y1. scope_path lists the headings an element sits under, outermost first; a heading’s own scope_path ends with its text.

from datagrunt import PDFReader

reader = PDFReader("report.pdf")

for heading in reader.headings:
    print(heading.level, heading.text, heading.scope_path)

for table in reader.tables:
    print(table.element_id, table.extraction_method, table.headers)
    df = table.to_dataframe()           # one String column per table column

for section in reader.sections:
    print(section.scope_path, section.page_span, len(section.elements))

print(reader.metadata.page_count, reader.metadata.sha256)

Figure images are not decoded during extraction: read_bytes() decodes the image when you call it.

Heading levels

A heading’s level is ranked over the whole document, not page by page. Each visible heading’s tier is its font_size_pt and is_bold; the document’s distinct tiers are ranked largest first, and at one size bold first. A heading’s level is 1 plus the number of tiers ranked above its own, at most 6, so a level means the same prominence on every page and a page whose largest heading is a section doesn’t reopen the chapter. Hidden headings take a level from the same ranking, but their tiers aren’t in it, so hidden text never shifts a visible heading’s level.

Tables

A table is either lattice — a ruled grid of at least 2 × 2 cells, drawn with vector lines — or stream — borderless rows that align into at least two columns and are not prose, a masthead, or a list of key-value pairs. Decorative boxes and single rule lines are not tables. A table split across pages keeps both parts; the later part’s continued_from_id names the earlier one. Table.to_dataframe() names its columns after the headers, made unique (a blank header becomes column_<n>); headers keeps the cells exactly as extracted.

The element rows

to_dataframe(), to_arrow_table(), to_dicts(), get_sample(), query_data(), and the row-based writers all share one set of rows: one row per element, in document order (page, then reading order). Every row has thirteen columns:

A row as a dict:

{
  "doc_id": "report__1a2b3c4d5e6f7a8b",
  "element_id": "p0001_e004",
  "page_number": 1,
  "reading_order": 4,
  "kind": "paragraph",
  "scope_path": ["Quarterly Report", "Revenue"],
  "text": "Revenue grew in every region.",
  "markdown": "Revenue grew in every region.",
  "bbox": {"x0": 72.0, "y0": 140.2, "x1": 540.0, "y1": 168.9},
  "route": "digital",
  "integrity_verified": true,
  "hidden_reasons": [],
  "image_only_reasons": []
}

to_arrow_table() always has the same Arrow schema (large_string text, dictionary-encoded kind and route), whatever your Polars version. Each to_dataframe() call returns your own copy, so changing it never changes the reader’s.

Markdown

reader.to_markdown() (and PDFWriter.write_markdown) renders the document page by page, as a reader sees it. Each page opens with a comment naming its route, status, and any image_only marker, followed by its visible elements in reading order:

<!-- page 1 route=digital status=ok -->
<!-- page 2 route=digital status=ok image_only=too_little_text -->

# Quarterly Report

**Prepared by:** Finance

| Region | Revenue |
| --- | --- |
| North | 1,200 |

- Lower costs
- New customers

<!-- figure p0001_e006 sha256=1a2b3c4d -->

Headings become # to ######, key-value pairs **key:** value, tables GitHub-style pipe tables, and lists - or 1. items. A figure is a placeholder comment until it has a file to link to (in a bundle, it is an image link). Element text is escaped, so text in a PDF can never inject its own Markdown or HTML.

Hidden elements are left out by default. Pass include_hidden=True to keep each one, behind a <!-- hidden: reasons --> line:

from datagrunt import PDFReader, PDFWriter

reader = PDFReader("report.pdf")
visible = reader.to_markdown()                         # what a reader sees
everything = reader.to_markdown(include_hidden=True)   # hidden elements behind their markers

PDFWriter(reader.document).write_markdown("out/report.md", include_hidden=True)
page_one = reader.pages[0].to_markdown()               # also on PDFDocument and Section

PDFReader.to_markdown, PDFWriter.write_markdown, and the to_markdown of a PDFDocument, Page, or Section all take include_hidden (False by default); a bundle’s Markdown always leaves hidden elements out. Nothing else does: the JSON, JSONL, Parquet, and CSV outputs, the DataFrame, and the document keep every element, hidden ones included.

Querying with SQL

query_data(sql) runs SQL against the element rows, registered under reader.db_table (tbl_ and the doc_id, with every character outside A–Z, a–z, 0–9, and _ replaced by _). It returns the rows read whole, with .pl() (Polars), .pa() (PyArrow), and .dicts():

from datagrunt import PDFReader

reader = PDFReader("report.pdf")

sql = f"""
SELECT page_number, kind, text
FROM {reader.db_table}
WHERE len(hidden_reasons) = 0 AND kind IN ('heading', 'table')
ORDER BY page_number, reading_order
"""
result = reader.query_data(sql)
result.pl()        # a Polars DataFrame
result.pa()        # a PyArrow Table
result.dicts()     # a list of dicts

Each reader has its own in-memory DuckDB connection. It persists between queries, so a table one query creates is there for the next, until the connection is released.

• No extension installs. DuckDB never downloads an extension on its own: a query that needs one it doesn’t ship with (read_xlsx, for example) raises duckdb.Error until you install it with an explicit INSTALL in your own SQL or in your environment.
• No spilling to disk. A query that needs more memory than DuckDB has raises duckdb.OutOfMemoryException instead of writing temporary files.
• Your SQL runs as written. Datagrunt never rewrites it. Building SQL from untrusted input is an injection risk against that connection, which can read and write files as any DuckDB connection can; validating it is your application’s concern, as it is for the CSV readers.

Writing outputs

from datagrunt import PDFWriter

writer = PDFWriter("report.pdf")

writer.write_json("out/report.json")          # the whole document, lossless
writer.write_jsonl("out/report.jsonl")        # one element per line
writer.write_markdown("out/report.md")
writer.write_parquet("out/report.parquet")    # nested columns kept
writer.write_csv("out/report.csv")            # bbox as four columns, lists as JSON arrays
writer.extract_images("out/images")           # each distinct figure once, as <sha256>.png
writer.render_pages_as_images("out/pages")    # every page as page_0001.png, page_0002.png, ...
writer.write_bundle("out/report_bundle")      # all of the above in one checked directory

PDFWriter takes the same options as PDFReader and checks them, and the PDF path, at construction. It extracts the document on the first call that needs it and caches it for the writer’s life. It holds no file, handle, or process between calls, so there is nothing to close.

Paths and atomic writes

• Every output path is required. There are no default filenames. A path is a str or os.PathLike (bytes are refused). The paths returned are absolute.
• Missing directories are created, for every method but write_bundle, whose parent directory must exist.
• Every file is written atomically: to a new temporary file (.tmp_<hex>) beside the target, then renamed onto it. On any failure, an interrupt included, the temporary file is removed and an existing target is left as it was. A process killed outright can leave its temporary file, never a partial target. A target that exists is replaced. There is no fsync before the rename: a write survives a killed process, not a power loss.
• A file target that is a directory raises IsADirectoryError; an output_dir that exists and is not a directory raises NotADirectoryError. Both are checked before any work.

Writing a document you already have

PDFWriter also accepts a PDFDocument — reader.document, for example — so you can write what you already extracted without extracting it again:

from datagrunt import PDFReader, PDFWriter

reader = PDFReader("report.pdf")
df = reader.to_dataframe()

writer = PDFWriter(reader.document)      # no second extraction
writer.write_markdown("out/report.md")
writer.extract_images("out/images")

Such a writer has no PDF to render, so render_pages_as_images and write_bundle raise ValueError; build the writer from the PDF path for those.

To reload a document you saved with write_json, pass its parsed JSON to PDFReader:

import json
from pathlib import Path

from datagrunt import PDFReader, PDFWriter

PDFWriter("report.pdf").write_json("out/report.json")

reader = PDFReader(json.loads(Path("out/report.json").read_text()))   # no extraction
reader.to_dataframe()

The JSON holds no image bytes, so extract_images writes no figure of a document reloaded from it. To keep figures for later, write a bundle and load that.

Figures

extract_images(output_dir) writes each figure as a PNG. By default (dedupe=True) each distinct picture is written once, named by the SHA-256 of its embedded stream (the figure’s identity), <sha256>.png, so a logo repeated on every page is one file. Every figure is still read, because the same stream drawn under different image settings (a colour space, a palette) is a different picture: that one is written as <sha256>-<its PNG's SHA-256>.png. With dedupe=False, every figure is written, named by its element id (p0001_e003.png). A figure whose bytes can’t be read is skipped with a warning on the datagrunt.pdf.writer logger naming it and why; the others are written. The paths returned are the files written, in document order.

from datagrunt import PDFReader, PDFWriter

reader = PDFReader("report.pdf")
for figure in reader.figures:
    png = figure.read_bytes()            # the image as PNG, decoded now
    raw = figure.read_raw_bytes()        # the embedded stream, verified against figure.sha256

PDFWriter("report.pdf").extract_images("out/figures", dedupe=False)

Filtering small embedded images

Embedded images smaller than 40 px on either side are not figures: rule lines, separators, and icon slivers. Pass min_image_dimension to PDFReader or PDFWriter to change the threshold:

from datagrunt import PDFReader

reader = PDFReader("report.pdf", min_image_dimension=15)    # keep logos and signatures
reader = PDFReader("report.pdf", min_image_dimension=200)   # keep only large figures
reader = PDFReader("report.pdf", min_image_dimension=1)     # keep every image

It must be an int of at least 1.

Page images

render_pages_as_images(output_dir, dpi=None) renders every page as an 8-bit grayscale PNG, page_0001.png, page_0002.png, and so on, at the writer’s dpi (200 by default) or the dpi you pass (at least 36). Every render is capped at max_render_pixels, so a very large page renders at a lower DPI rather than at more pixels. Each page renders in a separate, memory-capped worker process with a time limit (see Concurrency & Environments); a page whose render is refused or fails is logged and skipped, and the others are written. The paths returned are the files written, in page order. The PDF rendered is the file at the source path when you call.

render_pages_as_images rasterizes whole pages; extract_images writes the images embedded inside the PDF.

Bundles

write_bundle(output_dir, *, overwrite=False) writes every output into one directory, with a manifest that records each file’s size and SHA-256:

report_bundle/
├── manifest.json
├── json/
│   ├── document.json                          # the whole document
│   └── elements.jsonl                         # one element row per line, hidden ones included
├── markdown/
│   ├── document.md                            # links figures as ../images/<sha256>.png
│   └── pages/page_0001.md                     # each page's Markdown
├── images/<sha256>.png                        # each distinct figure once
└── page_images/
    ├── page_0001.png                          # every page
    └── bands/page_0001_band_01.png            # a vision page's OCR bands

• A page read by OCR keeps its own image — the render OCR read, at vision_dpi — with the horizontal bands OCR read. Every other page is rendered at dpi.
• Figures are named as extract_images names them: <sha256>.png, or <sha256>-<its PNG's SHA-256>.png for another picture of the same stream.
• The Markdown files leave hidden elements out; document.json and elements.jsonl keep every element.
• manifest.json uses bundle format version "5.3" (which includes image_only_pages list, and holds document JSON version 2). Manifests written in format "5.0" (from Datagrunt 5.0–5.2) still load seamlessly.
• manifest.json records the bundle format version, datagrunt version, source SHA-256, extraction options, integrity_verified, degraded_pages, image_only_pages, and each page image’s DPI. A page image or figure that can’t be produced (a refused render, an unreadable image) is listed under missing, with why; the rest of the bundle is written.
• Opt-out Warnings for Image-Only Pages: If a bundle is written with render_pages=False or include_images=False, or if rendering an image-only page fails, a UserWarning is emitted naming those marked pages. Because their content lives only in their page image and is not written to the bundle, the warning notes that the source PDF is their only full copy. Their missing entries in manifest.json also record this reason.
• Reused when it matches. If output_dir already holds a bundle of the same PDF, written with the same options (workers included) by the same datagrunt version, and its files verify, write_bundle returns that bundle’s document as it is and writes nothing. Any other bundle there raises FileExistsError naming why, unless you pass overwrite=True: the new bundle is then written beside the old one and swapped in, and if anything fails before the swap, the old bundle is left as it was.
• Never overwrites anything else. output_dir must not exist, be an empty directory, or hold a bundle. A directory without a valid bundle manifest (any other non-empty directory), a file, or a symlink raises FileExistsError whatever overwrite says: a bundle is never merged into anything. Its parent must exist (otherwise FileNotFoundError).
• Atomic. The bundle is written into a temporary directory beside output_dir, then renamed onto it. On any failure, nothing is published.
• Bound to its source. The source must still have the SHA-256 it was extracted with (ValueError otherwise), and a source that changes while the bundle is written raises PDFSourceChangedError and publishes nothing.
• It returns the document with its file paths set (or, when it reuses a bundle, that bundle’s document).

Load a bundle by passing its directory, or its manifest.json, to PDFReader:

from datagrunt import PDFReader, PDFWriter

PDFWriter("report.pdf").write_bundle("report_bundle")

reader = PDFReader("report_bundle")      # or PDFReader("report_bundle/manifest.json")
reader.to_dataframe()

The manifest is checked at construction: an invalid one raises ValueError. When the bundle is first used, every file is checked against the manifest’s size and SHA-256, and the document against the manifest; a mismatch raises datagrunt.pdf.BundleCorruptError naming the file. A figure’s or page image’s bytes are checked again every time you read them, so a bundle changed after it was loaded fails loudly, never silently. datagrunt.pdf.load_bundle(path) loads and verifies a bundle the same way and returns its PDFDocument directly.

CSV and formula injection

write_csv writes every value as extracted: a cell that starts with =, +, -, or @ is not escaped. Guarding a spreadsheet that opens the CSV against formula injection is your application’s concern — see Formula Injection Safety.

Hidden text

Text a human reader can’t see is flagged, never dropped. Each element’s hidden_reasons lists why, in a fixed order; an empty list means the element is visible (element.is_hidden is False).

Hidden text never decides a page’s route, and header_context holds visible text only. Markdown leaves hidden elements out unless you pass include_hidden=True, which keeps each behind a <!-- hidden: reasons --> comment; every other output keeps them, flagged. Filter on the column before you pass text on, for example to an LLM:

import polars as pl

from datagrunt import PDFReader

reader = PDFReader("report.pdf")

visible = reader.to_dataframe().filter(pl.col("hidden_reasons").list.len() == 0)
hidden = reader.query_data(f"SELECT element_id, hidden_reasons, text FROM {reader.db_table} WHERE len(hidden_reasons) > 0").pl()

Scanned pages and OCR

Each page is routed digital, hybrid, or vision (page.route), from its visible text:

• vision — the page has too many unreadable characters (under 70% of its visible characters valid), or too little text (fewer than 20 readable visible characters) beside images or vector drawings OCR could read, as a scan does. A page of short, clean text alone — a cover title, a blank page — has nothing for OCR to read, so it stays digital. It is rendered at vision_dpi, converted to grayscale with a 1% autocontrast, turned upright by Tesseract’s orientation detection, and read by Tesseract in overlapping horizontal bands. The page’s own digital text is kept, flagged superseded. Its render is kept as page.page_image, drawn again whenever you read it.
• hybrid — the page has enough readable text, but scanned images cover more than half of it. It is extracted from its text layer, as a digital page is.
• digital — every other page: extracted from its text layer.

The ocr option changes the routing: "auto" (the default) routes as above; "always" sends every page to OCR; "never" sends no page to OCR — a page that would have gone to vision is extracted from its text layer instead, with a warning, and a page that would have been hybrid is routed digital.

Image-Only Pages (image_only_reasons & image_only_pages)

A page whose text cannot stand and that was not read by OCR has its content only in its page image. page.image_only_reasons records why:

• too_little_text — Too few readable characters on a page that draws an image or vector paint.
• garbled_text — Too small a fraction of the page’s visible characters are valid text.
• extraction_failed — An error page where digital extraction failed entirely.

With ocr="never", a scanned page stays route="digital" and status="ok", but is marked with image_only_reasons. document.image_only_pages returns a tuple of 1-based page numbers for all such pages across the document. The marker is preserved in JSON v2, the image_only_reasons column, Markdown headers (<!-- page 2 route=digital status=ok image_only=too_little_text -->), and bundle manifest format 5.3 (image_only_pages).

Downstream readers can inspect these markers to load only the page images that must be processed:

reader = PDFReader("scans.pdf", ocr="never")
for page in reader.pages:
    if page.image_only_reasons and page.page_image is not None:
        png = page.page_image.read_bytes()  # Load image for downstream vision model

Installing Tesseract

Datagrunt runs the tesseract program itself; there is no Python wrapper. Install it with its English (eng) and orientation (osd) data:

• macOS: brew install tesseract
• Debian/Ubuntu: apt-get install tesseract-ocr
• Windows: the UB-Mannheim installer, or a package manager: winget install UB-Mannheim.TesseractOCR, choco install tesseract, or scoop install tesseract

tesseract --list-langs should list eng and osd. Datagrunt runs the tesseract it finds in an absolute directory on your PATH: relative entries (., an empty entry) and the working directory are never searched, so a tesseract planted in your working directory is never run. On Windows, add the Tesseract install directory to PATH. Digital and hybrid pages never need Tesseract.

Page status and error isolation

Extraction is always complete: a bad page is recorded and the rest of the document still extracts. Every page reports how extraction went in page.status:

page.warnings lists the non-fatal problems met on a page, and reader.document.degraded_pages lists every page whose status isn’t ok. Each vision page’s render and OCR share a 600-second budget; a page that runs out is ocr_unavailable, its digital text kept, and its warning says how many bands were read. With ocr="never", a page PDFium can’t read is an error page. A page read by OCR at a mean word confidence under 70 (quality.ocr_mean_confidence) carries a warning that it may be turned the wrong way or unreadable, so its text shouldn’t be trusted as is; its text and status are kept. Tesseract’s orientation detection can turn a page with little text the wrong way, and such a page reads at low confidence and is flagged.

from datagrunt import PDFReader

reader = PDFReader("report.pdf")

for page in reader.pages:
    print(page.page_number, page.route, page.status, page.integrity_verified)
    for warning in page.warnings:
        print("  warning:", warning)
    if page.status == "error":
        print("  not extracted:", page.error)

print(reader.document.degraded_pages)   # pages whose status is not "ok"

page.integrity_verified is True when the page’s elements hold exactly the non-whitespace characters of its text layer — hidden text included — so nothing was dropped or invented. A vision page has no text layer to check against, so it is never verified. page.quality holds the page’s measured signals (valid_char_ratio, char_count, vector_line_count, hidden_spans_flagged, rotation_deg, deskew_angle_deg, ocr_mean_confidence; a signal not measured is None).

Pass strict=True to raise instead: a page that PDFium can’t read, or whose extraction fails, raises PDFCorruptError naming the page rather than becoming an ocr_fallback or error page, and a page whose worker passed its memory cap or time bound raises DatagruntPDFError.

Errors

The constructor raises for bad arguments before any work (TypeError, ValueError, FileNotFoundError, or ImportError without the extra). Problems with the PDF itself surface on first use:

Every one derives from DatagruntPDFError, so one except catches them all. Because PDFEncryptedError is a ValueError and PDFCorruptError a RuntimeError, 4.x handlers keep working. DatagruntPDFError, PDFEncryptedError, and PDFCorruptError import from datagrunt; PDFSourceChangedError and BundleCorruptError from datagrunt.pdf.

from datagrunt import DatagruntPDFError, PDFCorruptError, PDFEncryptedError, PDFReader

reader = PDFReader("report.pdf")             # checks the path and options only
try:
    rows = reader.to_dicts()                 # the PDF is read here
except PDFEncryptedError:
    reader = PDFReader("report.pdf", password="...")
except PDFCorruptError as exc:
    print("unreadable:", exc)
except DatagruntPDFError as exc:
    print("not extracted:", exc)

Options

PDFReader and PDFWriter take the same keyword-only options:

from datagrunt import PDFReader

reader = PDFReader(
    "report.pdf",
    password=None,                  # an encrypted PDF's password
    workers=1,                      # processes to extract pages with
    ocr="auto",                     # "auto", "always" (every page to OCR), or "never"
    dpi=200,                        # page images (render_pages_as_images, bundles)
    vision_dpi=300,                 # pages read by OCR; at least dpi
    max_render_pixels=16_000_000,   # caps every render: a huge page gets a lower DPI
    min_image_dimension=40,         # smaller embedded images aren't figures
    strict=False,                   # raise on a bad page instead of recording it
)

Options are checked at construction. An out-of-range value raises ValueError: workers under 1, dpi under 36, vision_dpi under dpi, max_render_pixels under 262,144 (512 × 512), min_image_dimension under 1, an ocr other than the three modes, or a password containing a NUL character. A wrong type raises TypeError (True is not an int). The password never appears in a repr or an error message.

Every PDF is parsed in memory-capped worker processes, never in your own: one page worker with workers=1, or that many with more. Each is a fresh, isolated python process that never imports your script, so no if __name__ == "__main__": guard is needed — but a frozen application (PyInstaller) can’t start one, so it can’t read PDFs. See Concurrency & Environments.

Zero filesystem writes

PDFReader never writes to the filesystem — not even temporary files. Worker and Tesseract error output is kept in memory, images are encoded in memory, Tesseract reads its image from standard input, and the reader’s DuckDB connection neither spills to disk nor installs extensions. What your own SQL writes (COPY ... TO, INSTALL) is yours. Figures load lazily (figure.read_bytes()); write them with PDFWriter.

Schema-Driven Extraction (rationalize)

Datagrunt 5.1+ introduces rationalize(), a multi-provider structured extraction pipeline over PDFDocument objects. It extracts Pydantic model records with cross-family model consensus, quote and key grounding against visible elements, entity folding across pages, and deterministic Polars/Arrow table conversion.

Requires the [rationalize] extra (uv pip install "datagrunt[pdf,rationalize]").

from decimal import Decimal
from typing import Annotated
from pydantic import BaseModel
from datagrunt import Key, PDFReader, rationalize

class LineItem(BaseModel):
    item_id: Annotated[str, Key]
    description: str
    quantity: int
    unit_price: Decimal

class Invoice(BaseModel):
    invoice_number: Annotated[str, Key("Invoice #")]
    vendor_name: str
    items: list[LineItem] = []

doc = PDFReader("invoice.pdf").document

result = rationalize(
    doc,
    schema=Invoice,
    prompt="Extract invoice header details and all line items.",
    models=("openai:gpt-5.4", "anthropic:claude-sonnet-4-6"),
    min_families=2,         # Requires consensus across >= 2 distinct model families
    cache_dir=".cache/dg",  # Optional atomic on-disk response cache
)

# Export consensus records deterministically to Polars DataFrame
df = result.to_dataframe(flatten="items")

# Inspect items that failed consensus or grounding checks
disputed = result.needs_review

Key Features of rationalize()

• Key & Informational Markers: Key marks entity key fields with optional namespace (Key("part")) or normalizer (Key(normalizer="alnum_upper")) for ASCII punctuation/casing tolerance during consensus and folding. Informational marks descriptive fields whose free-text wording variance does not cause consensus record disputes.

• Cross-Family Consensus (min_families): Ensures high extraction accuracy by requiring consensus across models from at least N distinct provider families (e.g., openai:, anthropic:, gemini:, ollama:). Non-destructive consensus handles child keys, unkeyed pairing, and Anthropic schema_mode.

• Multimodal Wire & Out-of-Prompt OCR Grounding (#473): Supports up to 64 PNG band images (32 MiB total) via ModelRequest.images and render_pages="vision_only". Grounding uses out-of-prompt OCR witness elements to verify source attribution without cluttering LLM prompt envelopes.

• Verbatim Quote Grounding: Cites non-header source elements and verifies verbatim quotes against visible page text (exact match on digital/hybrid pages, token containment $\ge 0.85$ on OCR pages). Hidden elements (hidden_reasons) are automatically excluded before sending prompts.

• Image-Only Page Skipping: Pages marked as image-only (image_only_reasons non-empty) are automatically excluded prior to prompt construction and reported in result.skipped_units with reason "image_only" (or "page_error" for an error page), avoiding unnecessary token consumption on pages with garbled or unextracted text.

• Cross-Page Entity Folding: Automatically merges entity records spanning across page boundaries by entity key with non-destructive folding.

• Section Coalescing (coalesce_sections=True): Greedily packs consecutive text-bearing elements across document section boundaries into unified extraction units, maximizing LLM window utilization and prompt continuity while maintaining exact quote grounding. Enabled by default in rationalize().

• Gemini Thinking Level (thinking_level): Pass thinking_level="HIGH" (or case-insensitive, whitespace-tolerant values such as "low", " Medium ", "HIGH") to GeminiProvider or in model specs (gemini:gemini-2.5-flash?thinking_level=HIGH) to fine-tune Gemini reasoning depth.

• In-Process Zero-Loss Fast Paths: PDF compute routines — character BGR median calculation (pdf_char_bgr_medians), sweep-line box union area (pdf_union_area), character reason run grouping (pdf_char_reason_runs), table lattice segment snapping (pdf_snap_and_merge_segments), and stream LZW decoding (pdf_lzw_decode) — run via optimized in-process zero-loss fast paths without PyO3 boundary overhead.

• Non-Interleaving Stream Line Assembly (#476): Bounded stream layout line assembly (_OpenLine) evaluates neighbor proximity and overlap share to ensure tall glyphs (such as drop caps or large ordinal step numbers) never cause vertically adjacent text lines to collapse into an interleaved line.

• Offline Cloud Batch Mode (prepare_batch & ingest_batch): Prepare preflight requests offline without network calls, export to provider batch JSONL files (50% cost discount on Vertex AI and Anthropic), and ingest batch responses to run identical consensus, grounding, and folding pipelines.

• Deterministic Exports: result.to_dataframe(flatten="..."), result.to_arrow_table(), and result.to_dicts() provide deterministic schema conversions.

Two-Stage Batch Corpus Extraction (extract_corpus)

Datagrunt 5.2 introduces a two-stage batch workflow designed for high-throughput document processing pipelines:

from datagrunt import extract_corpus, load_corpus

# Stage 1: Batch extract PDF files into verified artifact bundles
manifest = extract_corpus(
    "incoming_pdfs/",
    output_dir="corpus_output/",
    overwrite=False,     # Incremental skip: reuse existing valid bundles (reused=True)
    strict=False,        # Fault isolation: record unreadable PDFs in manifest.failures
    workers=8,           # Parallel extraction across worker threads (up to 64)
    include_images=False, # Optional (5.2.2): Skip extracted images for speed and storage savings
    render_pages=True,   # Optional (5.2.2): Render full page images (or set False to skip)
)

print(f"Extracted {len(manifest.entries)} documents successfully.")
if manifest.failures:
    print(f"Failed documents: {len(manifest.failures)}")
    for fail in manifest.failures:
        print(f"  {fail.source_path}: {fail.error_type} - {fail.message}")

# Stage 2: Fast, zero-dependency loading of corpus bundles
corpus = load_corpus("corpus_output/")
for doc_id, document in corpus.items():
    print(doc_id, document.page_count, len(document.sections))

Key Benefits of Batch Corpus Extraction

• Pure-Stdlib Reading (load_corpus, load_bundle): Reading corpus manifests and document bundles uses standard Python libraries and does not require the [pdf] extra (pypdfium2 or Pillow). Lightweight downstream consumers can process corpus bundles without heavy binary dependencies.
• Stage 1 Worker Pool Reuse: extract_corpus() reuses a unified PageWorkerPool process pool across all documents in a batch, eliminating worker spawn overhead per PDF document.
• Graham LPT (Longest Processing Time) Scheduling: When strict=False, multi-worker corpus extraction orders pending documents largest-file-first, minimizing worker tail latency and long-pole stragglers across parallel worker process pools.
• Selective Image Extraction: Set include_images=False or render_pages=False in extract_corpus() or write_bundle() to skip extracting embedded figures or rendering page backdrop PNGs, drastically reducing disk footprint and extraction times.
• Incremental Processing: Setting overwrite=False compares SHA-256 digests of input PDFs against manifest entries. Unchanged PDFs are skipped instantly with reused=True.
• Fault Isolation: With strict=False, bad or corrupted PDFs generate a CorpusFailure record in manifest.failures rather than aborting the entire batch process.

Offline Cloud Batch Rationalization (prepare_batch & ingest_batch)

For large-scale extraction tasks, sending online real-time requests to LLM APIs can face concurrency limits, rate-limit (RPM/TPM) throttling, and higher per-token costs. Datagrunt 5.2.2 introduces Offline Cloud Batch Mode (prepare_batch and ingest_batch), allowing offline batch preparation, 50% API token cost savings, and zero online DSQ contention.

Workflow: 3-Step Offline Cloud Batch

from datagrunt import prepare_batch, ingest_batch
from pydantic import BaseModel, Field

class InvoiceSchema(BaseModel):
    vendor_name: str = Field(description="Name of vendor")
    invoice_number: str = Field(description="Invoice number")
    total_amount: float = Field(description="Total invoice amount")

# Step 1: Prepare batch plan (runs preflight, unit packing, and cache checks without network calls)
plan = prepare_batch(
    "corpus_bundles/",
    schema=InvoiceSchema,
    prompt="Extract invoice details.",
    models=["gemini:gemini-2.5-flash", "anthropic:claude-3-5-sonnet-20241022"],
    min_families=2,
    cache_dir=".cache/rationalize",
)

print(f"Uncached items to execute: {len(plan.items)}")
print(f"Cached hits found: {len(plan.cached_hits)}")

# Step 2: Write cloud provider batch prediction JSONL files
plan.to_vertex_jsonl("batch_requests/vertex_gemini.jsonl")
plan.to_anthropic_jsonl("batch_requests/anthropic_messages.jsonl")

# Submit JSONL files to Vertex AI Batch Prediction or Anthropic Message Batches via cloud CLI/SDK...
# Wait for batch jobs to complete and download response JSONL files.

# Step 3: Ingest completed batch responses to run consensus, quote grounding, and folding
result = ingest_batch(
    plan,
    responses=[
        "batch_responses/vertex_gemini_results.jsonl",
        "batch_responses/anthropic_messages_results.jsonl",
    ],
)

# Access identical RationalizedResult output as rationalize()
df = result.to_dataframe(flatten="items")
print(f"Extracted {len(df)} records across {len(result.documents)} documents.")

Advantages of Offline Batch Rationalization

• 50% Token Cost Discount: Cloud providers (Vertex AI / Gemini and Anthropic Message Batches) offer 50% price reductions for batch jobs.
• Zero Online Contention: Avoid hit limits on active rate limit (RPM/TPM) quotas or online model availability queues.
• Identical Pipeline Parity: ingest_batch() executes the exact same cross-family consensus (min_families), quote grounding, cross-page entity folding, and persistent cache verification as rationalize().