Optimizing PDFium: Resolving Multiprocessing Overhead and Layout Parser Recursion

With the release of Datagrunt 3.1.0, we migrated our default PDF parsing engine to PDFium (pypdfium2). This brought a major licensing win (BSD-3 / Apache-2.0 permissive license) and excellent text extraction parity.

However, during recent real-world benchmarking on both single-page datasheets and large, graphically dense multi-page board game rulebooks (the “Root” rulebooks), we hit two interesting engineering challenges: a significant speed bottleneck on small files, and a recursion crash on complex column layouts.

Here is what went wrong, and how we resolved it.

Challenge 1: The Process Pool Bottleneck (200x Speedup)

The Problem

Because the underlying PDFium C library is not thread-safe within a single process, datagrunt parallelizes page-parsing using Python’s ProcessPoolExecutor (configured with workers=4 by default).

This works exceptionally well for large, multi-page PDFs. However, on single-page PDFs (like typical invoices, receipts, or technical datasheets), the overhead was massive. Spawning/forking child processes on macOS costs between 300ms and 400ms.

For a 1-page PDF, the engine was spending 99% of its time spinning up worker processes and IPC serialization, and less than 1% actually parsing the PDF.

Our initial benchmarks on a suite of 19 single-page PDFs showed:

• pymupdf (using a lightweight ThreadPoolExecutor): 28.4 ms average
• pdfium_native (using ProcessPoolExecutor): 423.7 ms average

PDFium was 15x slower than PyMuPDF on small files because of process-spawning overhead.

The Resolution

If a document only has a single page, there is no work to parallelize. Spawning extra processes is completely wasted.

We updated our routing logic in datagrunt.core.pdf_io.engines to check the total page count before spawning workers:

# Bypass multiprocessing for single-page documents
if self.workers <= 1 or total_pages <= 1 or _is_distributed_env():
    # Run sequentially in the parent process...
else:
    # Spawn ProcessPoolExecutor...

The Result

Bypassing the process pool for single-page files eliminated the process startup overhead entirely. When we re-ran the benchmark on the 19 single-page files, the results were dramatic:

• pdfium_native average time dropped from 423.7 ms to 2.1 ms (a 200x speedup).
• pdfium_structured average time dropped from 454.5 ms to 21.8 ms (a 20x speedup).

PDFium is now significantly faster than PyMuPDF (~26.6 ms) for both single-page and multi-page files.

Challenge 2: Layout Parser Recursion Crash

The Problem

To reconstruct the logical layout of a PDF (e.g. columns and spanning sections), datagrunt’s structured mode uses a recursive XY-Cut algorithm in PageLayoutSorter.partition.

This algorithm detects wide vertical gutters to split the page into columns. During testing on graphically dense rulebooks with complex sidebars and column flows, both PDFium and PyMuPDF crashed with: RecursionError: maximum recursion depth exceeded

The bug occurred during a column split:

# Pure two-column split by midpoint
left = []
right = []
for it in columns:
    x0, _, x1, _ = self.adapter.get_bounds(it)
    mid = (x0 + x1) / 2
    if mid < gutter_x:
        left.append(it)
    else:
        right.append(it)
return self.partition(left) + self.partition(right)

If a page layout had elements clustered such that every element’s midpoint fell on one side of the detected gutter (e.g. all elements ended up in left, and right was empty), self.partition(left) would receive the exact same list of elements. The function would recurse endlessly on the same inputs until it blew the Python call stack.

The Resolution

We added a simple but robust guard check. If either column split is empty, the split is degenerate, meaning the elements cannot be partitioned further:

if not left or not right:
    return [items]
return self.partition(left) + self.partition(right)

This immediately halts the recursion when no clean physical division can be made.

The Result

With the recursion fix in place, both engines now parse 100% of pages successfully with zero crashes. It also unlocked a massive improvement in text completeness: on the Learn to Play rulebook, pdfium_structured successfully extracted 65,853 characters (across all 24 pages) compared to PyMuPDF’s 39,852 characters (which previously skipped pages 1 and 2 due to recursion errors).

Large Document Benchmark Results

To verify the combined effect of our fixes on large files (where multiprocessing does run), we benchmarked the rulebooks (76 pages total, with high-resolution image extraction and saving enabled):

Because PDFium’s CPU-bound page-rendering and image extraction run in a true multi-core process pool, it scales beautifully on large documents, completing the rulebook suite 3x to 8x faster than PyMuPDF’s thread pool, while recovering more text from dense columns.

All fixes are merged into the main branch of both repositories. Happy parsing!