Concurrency & Environments

Datagrunt supports parallel processing and provides mechanisms to safely deploy in environments ranging from multi-core local machines to sandboxed distributed runtimes. This applies to PDFReader and PDFWriter — see the PDF Parsing guide for extraction options.

Parallel Processing & Concurrency

Every PDF is parsed in worker processes, never in your own: your process never even imports PDFium. With the default workers=1, one page worker reads every page in order. Pass a larger workers count to read pages in that many workers at once:

from datagrunt import PDFReader

# No `if __name__ == "__main__":` guard needed
reader = PDFReader("report.pdf", workers=8)
df = reader.to_dataframe()

The pages are split into contiguous runs, one per worker (never more workers than pages). The first worker opens the PDF and reports its page count; the others start as soon as it has. Each worker opens the PDF itself and sends each page back as soon as it is extracted, and the pages come back in page order.

Figure reads and page renders run in workers too: figure.read_bytes(), extract_images, render_pages_as_images, bundle page images, and the renders behind OCR each hand the PDF to a worker process, never to your own.

No __main__ Guard Needed

A worker is a plain subprocess, python -I, not a multiprocessing child: there is no spawn start method involved, and the worker never imports your main module. Your script runs once, however it is written — with or without an if __name__ == "__main__": block, in a notebook, or in a REPL.

-I (isolated mode) keeps your working directory and PYTHON* environment variables (PYTHONPATH, PYTHONSTARTUP, …) out of the worker. It receives your interpreter’s exact sys.path on its standard input instead, so it imports the same datagrunt and PDFium you do.

Frozen applications. An application built with PyInstaller (or any frozen executable) has no Python interpreter to start, so it can’t start workers: reading a PDF there raises DatagruntPDFError saying so. workers=1 doesn’t help, because every PDF is read in a worker.

Memory Caps & Time Bounds

Every worker — page, figure, or render — caps its own memory at 2 GiB above what it holds once started, before PDFium sees the PDF. PDFium decodes compressed data while it merely opens a document or loads a page, before any check can run, so only a capped worker is ever handed one:

  • Linux, macOS, and Windows: a watchdog thread ends the worker once its peak memory passes the cap.
  • Linux and Windows also keep a hard limit above the cap, as a backstop: an address-space limit (RLIMIT_AS) on Linux, a Job Object commit limit on Windows.
  • Other platforms: Datagrunt has no cap there, so no worker reads a PDF.

Each page must come back within a time bound that restarts with every page, so a worker that hangs is killed, never waited on forever. A page whose worker passes its memory cap or its time bound becomes an error page whose page.error says why; the pages the worker already sent are kept, and its remaining pages are read by a new worker. One bad page never costs you another page. With strict=True, such a page raises DatagruntPDFError naming it instead. If the first worker ends before it can even report the page count, there is no page to record the failure on, so reading raises PDFCorruptError.

A failed render or figure read never takes your process down either:

  • A vision page whose render fails becomes ocr_unavailable, its digital text kept.
  • Text the background check couldn’t verify is flagged unchecked_background.
  • render_pages_as_images and extract_images log the page or figure and skip it.
  • A bundle lists the page image or figure under its manifest’s missing.

Tesseract runs as its own process too, killed at its time limit. On Linux and macOS, killing a page worker also ends the render workers and Tesseract runs it started.

PDFWriter.render_pages_as_images and write_bundle render up to workers pages at a time, on threads in your process, each page in its own render worker.

Thread Safety

A PDFReader can be shared between threads: its first use is locked, so two threads never extract the same reader’s document twice, and query_data calls on one reader run one at a time on its connection. The document and its elements are immutable, and each to_dataframe() call returns your own copy.

Distributed Runtimes Fallback

When running inside managed distributed environments (Apache Spark, Apache Beam, Apache Flink, or Celery), the runtime’s executors are the parallelism, and nested process pools are restricted or cause container sandbox permission errors. datagrunt detects these environments — any of the environment variables SPARK_ENV_LOADED, SPARK_HOME, BEAM_WORKER_ID, FLINK_CONF_DIR, or CELERY_BROKER_URL triggers the detection — and reads the pages in one worker, whatever workers says.

That worker, and the render workers and Tesseract runs it starts, are still processes of their own. A runtime that forbids starting processes can’t read PDFs: starting the page worker raises the operating system’s OSError.

Polars to PandasMigrating to 5.0 (PDF)