Concurrency & Environments

Datagrunt supports parallel processing and provides mechanisms to safely deploy in environments ranging from multi-core local machines to sandboxed distributed runtimes. This applies to PDFReader and PDFWriter — see the PDF Parsing guide for engine and extraction options.

Parallel Processing & Concurrency

By default, PDFReader and PDFWriter run sequentially (workers=1). On the default PDFium engine, you can enable parallel processing on multi-core systems by passing a workers count greater than 1:

if __name__ == '__main__':
    # Run with 8 processes to parse pages concurrently
    reader = PDFReader("report.pdf", workers=8)
    document = reader.to_dicts()

Parallelism is a PDFium-only feature. The PyMuPDF engine always runs strictly sequentially because the underlying MuPDF library is not thread-safe; passing workers > 1 with engine='pymupdf' issues a warning and the value is ignored.

Multiprocessing Guard Requirement

Because PDFium is not thread-safe within a single process, datagrunt uses a process pool (ProcessPoolExecutor with the spawn start context on macOS and Windows) to parse pages concurrently.

Under Python’s spawn start context, child processes import the main module to initialize. If you call PDFReader or PDFWriter with workers > 1 outside of a if __name__ == '__main__': block, the child processes will recursively spawn their own process pools, leading to a crash or infinite recursion loop.

For single-page documents, datagrunt automatically bypasses the process pool and executes sequentially to avoid process spawning overhead.

Distributed Runtimes Fallback

When running inside managed distributed environments (Apache Spark, Apache Beam, Apache Flink, or Celery), nested process spawning is restricted or causes container sandbox permission errors. datagrunt automatically detects these environments — any of the environment variables SPARK_ENV_LOADED, SPARK_HOME, BEAM_WORKER_ID, FLINK_CONF_DIR, or CELERY_BROKER_URL triggers the detection — and falls back to sequential, parent-process execution, logging a warning that the workers setting was ignored.