CSV Engines & Rust Acceleration

Datagrunt supports multiple high-performance execution engines for reading and writing CSV files: Polars, DuckDB, and PyArrow. While each engine has distinct internal characteristics, Datagrunt guarantees absolute semantic consistency across all of them.

Comparison Matrix

Whichever engine you choose, results are consistent:

• Leading # comment lines and leading blank lines are skipped.
• Logical record counts are preserved (quoted fields containing newlines count as a single record).
• Column name normalization and collision resolution (e.g. Col A and col_a → col_a, col_a_1) behave identically across all three.
• Mid-file lines starting with # are preserved as regular data.

Note: For PDF parsing, Datagrunt uses the permissively-licensed PDFium engine by default, with PyMuPDF available as an alternative. See Choosing a PDF Engine.

Rust-Accelerated CSV Core

As of version 4.0.0, Datagrunt’s CSV delimiter and dialect inference run in a bundled Rust extension (datagrunt._native), shipped as platform wheels (abi3, Python 3.10+). On a 98 MB CSV, the native engine scans roughly 5x faster than the pure-Python path while producing byte-identical results.

The native engine is the default and is required on supported platforms (it ships as a prebuilt wheel). A byte-for-byte-equivalent pure-Python implementation lives alongside it as the differential-parity oracle — validated against the Rust engine in CI and selectable via a hidden diagnostics toggle — so results are identical no matter which path runs. The acceleration requires no code changes.

Production Build Optimizations & Testing Parity (v4.5.3+)

To maintain a minimal release binary footprint and eliminate compilation overhead, starting in Datagrunt 4.5.3 all test-only eager I/O oracles (such as read_decoded, read_universal_lines, and is_blank) are strictly compiled under test configurations only (#[cfg(test)]).

In production environments, all I/O pathways strictly stream byte data (such as via DecodedReader or streaming line iterators) without loading whole files into memory, keeping the memory and performance profile highly optimized.

In version 4.5.4, we continued refining this high-performance core by modernizing the internal loops of the critical probe_csv_header path. We replaced mapping wrappers with direct, idiomatic boolean predicates (is_some_and) and clean boundary checks, ensuring that byte-level dialect sensing is as lean and robust as possible. The eager, whole-file parsing logic is retained exclusively as differential-parity reference oracles to validate correctness in CI, guaranteeing a lean, exceptionally lightweight production release of the bundled Rust extension (datagrunt._native) without compromising test rigor.

Disabling Rust Acceleration

To force the pure-Python path (for debugging or A/B comparison), set the DATAGRUNT_DISABLE_RUST environment variable before importing Datagrunt:

import os
os.environ["DATAGRUNT_DISABLE_RUST"] = "1"  # use the pure-Python compute path