CSV Engines & Rust Acceleration

Datagrunt supports multiple high-performance execution engines for reading and writing CSV files: Polars, DuckDB, and PyArrow. While each engine has distinct internal characteristics, Datagrunt guarantees absolute semantic consistency across all of them.

Comparison Matrix

Whichever engine you choose, results are consistent:

• Leading # comment lines and leading blank lines are skipped.
• Logical record counts are preserved (quoted fields containing newlines count as a single record).
• Column name normalization and collision resolution (e.g. Col A and col_a → col_a, col_a_1) behave identically across all three.
• Mid-file lines starting with # are preserved as regular data.

Note: For PDF parsing, Datagrunt uses the permissively-licensed PDFium engine by default, with PyMuPDF available as an alternative. See Choosing a PDF Engine.

Rust-Accelerated CSV Core

As of version 4.0.0, Datagrunt’s CSV delimiter and dialect inference run in a bundled Rust extension (datagrunt._native), shipped as platform wheels (abi3, Python 3.10+). On a 98 MB CSV, the native engine scans roughly 5x faster than the pure-Python path while producing byte-identical results.

The native engine is the default and is required on supported platforms (it ships as a prebuilt wheel). A byte-for-byte-equivalent pure-Python implementation lives alongside it as the differential-parity oracle — validated against the Rust engine in CI and selectable via a hidden diagnostics toggle — so results are identical no matter which path runs. The acceleration requires no code changes.

Production Build Optimizations & Testing Parity (v4.5.3+)

To maintain a minimal release binary footprint and eliminate compilation overhead, starting in Datagrunt 4.5.3 all test-only eager I/O oracles (such as read_decoded, read_universal_lines, and is_blank) are strictly compiled under test configurations only (#[cfg(test)]).

In production environments, all I/O pathways strictly stream byte data (such as via DecodedReader or streaming line iterators) without loading whole files into memory, keeping the memory and performance profile highly optimized.

In version 4.5.4, we continued refining this high-performance core by modernizing the internal loops of the critical probe_csv_header path. We replaced mapping wrappers with direct, idiomatic boolean predicates (is_some_and) and clean boundary checks, ensuring that byte-level dialect sensing is as lean and robust as possible. The eager, whole-file parsing logic is retained exclusively as differential-parity reference oracles to validate correctness in CI, guaranteeing a lean, exceptionally lightweight production release of the bundled Rust extension (datagrunt._native) without compromising test rigor.

In version 4.5.5, we introduced a comprehensive sweep of open-source project hardening and compatibility improvements. On the computational side, we resolved a critical compatibility gap where Python versions prior to 3.11 would raise a line contains NUL error when encountering binary or NUL bytes inside the standard library csv.reader. The pure-Python fallback backend now transparently substitutes these characters with the Unicode Replacement Character (\uFFFD) rather than destructively stripping them, ensuring it perfectly mirrors the native Rust engine and achieves 100% differential-parity correctness on older interpreter runtimes. On the governance side, this release adds standardized contribution benchmarks, secure dependency pinning for our automated deployments, and a complete suite of developer guidelines (e.g., Code of Conduct, Security Policy, pull-request blueprints, and automated Dependabot configurations).

In version 4.5.6, we upgraded our underlying Rust dependency footprints and solidified our continuous delivery infrastructure. On the processing side, we bumped our core regular-expression engine fancy-regex to 0.18.0 (from 0.14.0) inside rust/datagrunt-core, gaining critical performance updates and regex parsing robustness within our native CSV-sensing core (datagrunt._native). On the dependency management side, we adopted the increase-if-necessary Dependabot versioning strategy for Datagrunt as a library; this suppresses disruptive, cascading dependency floor-raises that would otherwise force downstream consumers to upgrade their libraries prematurely. On the operational side, we locked down our delivery guarantees: our publishing automation is now strictly gated, executing only after the maintainer-approved release package actually lands successfully on PyPI, completely eliminating the risk of mismatched/empty blog deployments for semi-automated publications. We also fortified our Hugo publishing workflows with automated activate-environment: true guarantees to cleanly integrate with the astral-sh/setup-uv v9 virtual environment standards.

In version 4.5.8, we established rigid, compile-time contract enforcement across our dual-backend architecture. While our core data paths remain separated into a high-performance native Rust extension and a pure-Python reference fallback (oracle), we introduced a formal PEP 544 ComputeBackendProtocol to describe this shared API boundary. To make this contract statically enforceable by mypy, we authored a handwritten _native.pyi type stub mapping our compiled Rust module exports and created a static configuration guard that assigns both active backend references to the protocol type. Concurrently, we locked down this boundary against signature drift using an AST-level runtime test suite and integrated a blocking, scoped mypy static-analysis step into our main CI/CD pipeline, guaranteeing that any future architectural changes are verified at edit-time.

In version 4.5.9, we deployed a dual-layer enhancement spanning critical correctness bug fixes in the native parsing core and a highly sophisticated, property-based differential parity testing workflow.

First, we addressed a subtle byte-versus-character boundary edge case in our universal line-reading engine (UniversalLines inside io.rs). Under errors="ignore" decoding parameters (such as the utf-8-sig signature drop or invalid trailing UTF-8 sequences like b"\x80" or truncated multi-byte sequences), the native engine previously yielded a phantom trailing empty line because some raw bytes remained in the buffer, even though they decoded to nothing. This created a minor output divergence from CPython’s standard string iterator, which simply swallows these empty representations. By adding immediate line emptiness checks on newly decoded character runs, the native engine is now fully aligned with CPython behavior.

Second, we established a complete property-based testing (PBT) harness using Hypothesis that generates randomized, edge-case-dense CSV-shaped data across dozens of “seams” (including CR/LF/CRLF line terminators, embedded delimiters, comment segments, BOM markings, and ragged row distributions). This harness compares execution outcomes (values and exceptions, including PyO3-bridged panic boundaries) across both backends, and is driven by a scheduled, deep-search CI/CD pipeline off the main path to dynamically discover and neutralize any edge-case structural divergence.

In version 4.5.10, we resolved a critical cross-backend dialect/delimiter sniffer divergence (Issue #318) rooted in a subtle Unicode definition mismatch between the fancy-regex crate’s word character (\w) class and CPython’s standard library definition. CPython defines \w using str.isalnum() along with the underscore, which covers the general Unicode categories L* (Letters) and N* (Numbers). However, fancy-regex maps \w to UTS#18 perl_word, which excludes categories like No/Nl (making the superscript two ² U+00B2 a word character to CPython but not to Rust) and includes marks (making U+0301 a word character to Rust but not to CPython). This divergence caused the sniffer to pick different delimiters because it disagreed on what characters were excluded from the delimiter character class [^\w\n"']. By explicitly spelling out CPython’s word definition (\p{L}\p{N}_) as PY_WORD inside our compiled Rust patterns in dialect.rs, we achieved absolute dialect sniffer parity. We also expanded our differential-parity property testing inside test_parity_dialect.py by directly comparing inferred delimiter properties (#319), closed the remaining test-level xfail, and preserved the minimized falsifying sample as a permanent regression case.

Disabling Rust Acceleration

To force the pure-Python path (for debugging or A/B comparison), set the DATAGRUNT_DISABLE_RUST environment variable before importing Datagrunt:

import os
os.environ["DATAGRUNT_DISABLE_RUST"] = "1"  # use the pure-Python compute path