Datagrunt 4.5.5: Enterprise Hardening, Multi-Python CI, and NUL-Safe Dialect Fallbacks

Datagrunt 4.5.5 is a comprehensive security, compatibility, and community-hardening release. By formalizing our open-source governance and expanding our automated test coverage to target the entire advertised Python interpreter spectrum (Python 3.10 through 3.14), this release guarantees enterprise-ready stability. Furthermore, we resolved a critical compatibility gap on older Python runtimes, safeguarding our pure-Python fallback dialect-probing engine against dirty binary or NUL-filled input streams.

The Problem: Python < 3.11 “Line Contains NUL” Error

While our primary native Rust extension (datagrunt._native) seamlessly parses NUL (\0) bytes as regular field contents, Python’s built-in csv library does not share this tolerance on older versions. On Python 3.10 (and earlier), executing standard csv.reader operations over streams containing raw binary or zero-byte characters instantaneously triggers a halting exception:

_csv.Error: line contains NUL

In Datagrunt, this caused direct crashes in row_count_with_header and silent parsing degradations in check_ragged whenever the database engine fell back to the pure-Python differential-parity oracle (e.g., when running with DATAGRUNT_DISABLE_RUST=1 or on uncompiled diagnostic environments).

The Solution: Non-Destructive Replacement

Simply stripping NUL bytes from the stream beforehand is highly destructive. Stripping changes file offsets, removes column separators, and breaks logical comment detection if a comment symbol falls adjacent to stripped data.

To tackle this, Datagrunt 4.5.5 introduces _nul_safe_lines in the Python parser:

def _nul_safe_lines(f):
    """Line stream safe for ``csv.reader`` on every supported Python.

    Python < 3.11's csv module rejects NUL characters ("line contains NUL");
    3.11+ parses them as ordinary field content, as does the Rust backend.
    Substituting U+FFFD (never empty-string stripping) keeps comment
    detection, row emptiness, and column counts identical to 3.11+.
    """
    if sys.version_info >= (3, 11):
        return f
    return (line.replace("\0", "�") for line in f)

By substituting raw \0 bytes with the Unicode Replacement Character \uFFFD (�), Datagrunt preserves physical row boundaries, character indices, and column counts. This guarantees 100% byte-for-byte correctness and absolute parity alignment between our Rust core engine and Python fallback interpreters.

Visualizing the Multi-Python Validation Gates

To enforce this absolute parity, we have completely overhauled our continuous integration (CI) pipelines. Every pull request must now survive parallel build matrices ranging from Python 3.10 up to the experimental Python 3.14 boundaries, evaluated against both active backends:

  graph TD
    PR[Pull Request Submitted] --> LintCmd[Ruff lint & format checklist]
    LintCmd --> RustChk[Cargo Clippy & Cargo Test]
    
    subgraph MultiPythonMatrix [Parallel Python Verification Matrix]
        Py310[Python 3.10 Tests]
        Py311[Python 3.11 Tests]
        Py312[Python 3.12 Tests]
        Py313[Python 3.13 Tests]
        Py314[Python 3.14 Tests]
    end
    
    RustChk --> MultiPythonMatrix
    
    Py310 --> ParityCheck[Differential Parity Suite: Rust Core == Python Oracle]
    Py311 --> ParityCheck
    Py312 --> ParityCheck
    Py313 --> ParityCheck
    Py314 --> ParityCheck
    
    ParityCheck -- All Pass --> Merge[Squash & Auto-publish Site]

This guarantees that any update deployed on PyPI maintains bulletproof stability regardless of which engine or Python interpreter is driving the environment.

Enterprise Governance and Project Hardening

To support Datagrunt’s rapid open-source growth, we established complete professional guidelines and repository protections:

1. Vulnerability Disclosure Policy (SECURITY.md): Privately report vulnerabilities through GitHub Security Advisories. Eager I/O panics across PyO3 barriers are explicitly categorized as DoS bugs in-scope for security fixes, while formula-injection characters remain properly documented application-tier issues.
2. Standardized Contributing Guidelines (CONTRIBUTING.md): Detailed local setup instructions via uv, testing requirements, and technical architectural details (such as our “Python leads, Rust follows” core design principles).
3. Draft-Free Automated Deployments: Pinned our automated documentation-publishing actions (auto-publish-site.yml) to secure, deterministic dependency versions (google-antigravity==0.1.7 and python-dotenv==1.2.2), eliminating supply-chain vulnerabilities.
4. Licensing Compliance: Embedded standard MIT declarations in all Rust crate manifests (Cargo.toml), and updated our global licensing range to cover Copyright (c) 2024-2026.
5. AGPL License Awareness: Added an informative warning in our README and documentation index for the optional PyMuPDF engine, advising enterprise users to evaluate AGPL-3.0 obligations or stick to our default, permissively licensed BSD/Apache-2.0 PDFium engine.

Upgrading to v4.5.5

Get the hardened release with the latest compatibility fixes today:

# Upgrade package via your preferred environment runner (uv recommended)
uv pip install --upgrade datagrunt

# Verify installation metrics
python -c "import datagrunt; print(datagrunt.__version__)"
# Output: 4.5.5

To explore messy CSV parsing nuances, check out the Messy CSV Guide or dive deeper into Rust Acceleration.