Datagrunt 3.1.40: A Hardening and Performance Release

Most releases add features. This one adds confidence.

Datagrunt 3.1.40 is the product of a focused hardening sweep: 33 pull requests merged in a single day, spanning CSV correctness, PDF robustness, performance, and one genuine security fix. Every fix shipped with a dedicated unit test that was verified red on main — each test was confirmed to fail against the old behavior before the fix turned it green — so every change in this release is pinned down by a regression test. The suite grew from 281 to 408 tests along the way.

Rather than walk through all 33 changes one by one, here is the release organized by theme.

1. CSV Correctness: No More Silent Data Loss

The most important fix in this release is also the easiest to miss. Datagrunt skips leading # comment lines when locating the CSV header — but the Polars engine was applying comment semantics to the entire file. If your data contained #-prefixed values in the first column — hex color codes like #FF5733, hashtags, issue references — those rows were silently dropped.

In 3.1.40, comment handling is now strictly positional: only leading # lines at the top of the file are treated as comments. Once real data begins, #-prefixed rows are data, full stop — preserved on every engine. Two related fixes land alongside it:

• Leading blank lines no longer corrupt header detection or shift data rows.
• Mid-file # lines no longer crash the PyArrow engine, which now matches Polars row-for-row.

One combination remains a known gap: DuckDB’s CSV sniffer can still mis-detect a file that pairs a leading # comment block with #-prefixed data rows. Polars and PyArrow handle that mix correctly, so prefer those engines for such files.

Consistency fixes in the same vein:

• row_count_with_header and row_count_without_header now count logical CSV records, so quoted fields with embedded newlines count as one record instead of several.
• Empty and blank files produce consistent empty results from get_sample() and every writer — no more phantom column0.
• write_parquet now honors lenient=True like every other writer.
• normalize_columns resolves colliding names deterministically (col_a, col_a_1) on all engines; DuckDB previously crashed on collisions.
• The AI schema-report path (csv_string_sample) now respects comment handling and no longer clobbers a user column that happens to be named null_count.

2. Fail Fast, Fail Clearly

Bad inputs now surface immediately at construction time with the obvious exception, instead of failing later somewhere inside an engine:

• An invalid engine name raises a ValueError listing the valid engines.
• A directory path raises a ValueError.
• A missing file raises a FileNotFoundError.
• Non-UTF-8 files no longer crash CSVReader construction during metadata detection.

And in the DuckDB engine, generated SQL now safely escapes filenames, headers, and inferred delimiters — an apostrophe in O'Brien's data.csv can no longer break the query that imports it.

3. CSV Performance: Stop Paying Twice

Two pathways were doing dramatically more work than necessary:

• Repeated queries re-imported the file every time. CSVReader.query_data() now imports the CSV into DuckDB once and reuses the table on subsequent calls. In our benchmarks on a 100 MB file, repeated queries are roughly 500x faster.
• Sampling materialized the whole file. get_sample() previously loaded the entire dataset to show you 20 rows — in our benchmarks, up to 705 MB of memory to sample a 100 MB file. It now streams just the first rows on every engine, keeping memory bounded regardless of file size.

DuckDB connections also open lazily and close deterministically, so constructing a reader is cheap and resources are released as soon as work completes.

4. PDF Robustness: Keep Every Page

Datagrunt’s contract for PDF parsing is 100% extraction — nothing silently dropped. Several fixes in 3.1.40 enforce that contract:

• OCR failures no longer discard pages. When Tesseract fails on a page, the page is kept with whatever content was already extracted, plus a warning. Previously the entire page vanished from the output.
• Encrypted PDFs raise a clear ValueError on every engine, replacing a raw PdfiumError from the depths of the default engine.
• PyMuPDF is now strictly sequential. MuPDF is not thread-safe, so the PyMuPDF engine ignores workers > 1 with a warning. PDFium remains the parallel, process-based engine.

A long list of extraction-fidelity fixes lands with them: rotated pages now produce in-bounds coordinates; native-text bounding boxes sort correctly in y-order; multi-image pages pair bounding boxes by xref instead of by guesswork; CRLF line endings are normalized in native Markdown; Markdown metacharacters are escaped in body text; text-only pages are classified text_only instead of always mixed; PDFWriter guards against empty PDFs; PDFium page handles close deterministically; and the layout sorter is hardened against corrupt, NaN, or absurdly large coordinates that previously caused out-of-memory errors or IndexError.

A Security Fix in Image Dedupe

Image deduplication collapses byte-identical embedded images to a single file. The old implementation derived deletion targets from document content — which meant a malicious PDF could, in principle, direct deletions at arbitrary paths. In 3.1.40, dedupe deletion is contained to the output directory Datagrunt manages and is symlink-safe. It can never touch files elsewhere on disk.

5. PDF Performance

• The PyMuPDF engine opens the document once per parse instead of re-opening it per page — roughly 3x faster on a 100-page benchmark document — and no longer drags in pdfium as a dependency for that engine.
• Text-line clustering is now near-linear. On dense benchmark pages (6,000 items), layout clustering is about 83x faster.

6. Dependencies and Test Coverage

Minimum dependency floors were raised across the board: duckdb>=1.5.3, polars>=1.41.2, pyarrow>=24, google-genai>=2.8, PyMuPDF>=1.27.2, pypdfium2>=5.9, and pytest>=9 for development. And as noted above, the test suite grew from 281 to 408 tests, with every fix in this release locked in by a regression test that was verified to fail before the fix landed.

Upgrade Today

Datagrunt 3.1.40 has been released with no breaking API changes. You can pull the latest update:

uv pip install --upgrade datagrunt