Datagrunt 4.5.8: Rigid Dual-Backend Contract Enforcement via PEP 544 Protocols and Scoped MyPy Gates

Datagrunt 4.5.8 delivers superior architecture and contract safety for enterprise data engineering. In previous 4.5.x iterations, we focused heavily on raw parsing performance, memory footprint minimization, and publishing pipeline integrity. Version 4.5.8 takes this rigorous reliability to the engineering codebase itself.

By introducing a formal PEP 544 ComputeBackendProtocol, designing handwritten type stubs (_native.pyi) for our high-speed compiled Rust module, implementing a custom AST runtime parser to verify stub equivalence, and wiring a blocking, scoped MyPy static-analysis gate into our GitHub Actions CI pipeline, we have hard-locked correctness across both execute pathways. This ensures our turbo-charged native Rust engine and pure-Python reference fallback (oracle) can never drift out of sync.

The Challenge: Architectural Drift in Dual-Backend Systems

Datagrunt operates a premium dual-backend model for CSV parsing and dialect sensing:

1. The Native Path (datagrunt._native): A compiled Rust extension written in native code for maximum parsing throughput and 5x execution speeds.
2. The Oracle Path (datagrunt.core.csv_io._compute_python): A pure-Python fallback path behaving exactly as a reference standard to guarantee perfect cross-interpreter compatibility and aid diagnostics.

Maintaining parity across these backends is highly demanding. If a single function’s arguments or return type dynamically change during development, or if any developer changes a parameter name in Python (e.g., from filepath to path), the interface will drift.

Even worse, standard static analysis tools like MyPy treat compiled binaries (.so / .pyd / .dylib) as containing Any types out of the box. This means MyPy statically passes any assignment or signature check against the native _native binary vacuously (without inspecting signatures), leaving contract verification entirely to downstream integration tests.

  graph TD
    subgraph Drift Risk in Dual-Backend Architecture
        direction TB
        Dev[Developer Edits Code] -->|Modifies signature| PythonOracle[_compute_python]
        Dev -->|Modifies signature| RustCore[rust/datagrunt-python]
        PythonOracle -.->|Silently drifted| Drift[Production Crash or Keyword Mismatch]
        RustCore -.->|Silently drifted| Drift
    end

To eliminate this vulnerability completely, Datagrunt 4.5.8 deploys a sophisticated, multi-layered typing and verification contract.

1. Enforcing Structural Parity with PEP 544 Protocols

We have formalised the entire CSV sensing engine interface as a PEP 544 typing.Protocol located at src/datagrunt/core/csv_io/_compute_protocol.py (_compute_protocol.py):

from __future__ import annotations
import os
from typing import Protocol, TypedDict

StrPath = str | os.PathLike[str]

class HeaderProbe(TypedDict):
    """Single-pass header probe result shared by delimiter and dialect inference."""
    empty: bool
    blank: bool
    first_row: str
    sample_rows: list[str]
    sample_lines: list[str]

class SniffedDialect(TypedDict):
    """Sniffed CSV dialect. Constant fields mirror csv.Sniffer."""
    delimiter: str
    quotechar: str
    escapechar: None
    doublequote: bool
    lineterminator: str
    skipinitialspace: bool
    quoting: int

class ComputeBackendProtocol(Protocol):
    """Structural interface every CSV compute backend must satisfy."""

    def is_legacy_mac_newlines(self, path: StrPath, /) -> bool: ...
    def leading_rows(self, path: StrPath, limit: int, /) -> list[str]: ...
    def first_row(self, path: StrPath, /) -> str: ...
    def count_leading_comments(self, path: StrPath, /) -> int: ...
    def count_leading_physical_lines_before_header(self, path: StrPath, /) -> int: ...
    def normalize_columns(self, names: list[str], /) -> list[str]: ...
    def infer_delimiter(self, path: StrPath, /) -> str: ...
    def row_count_with_header(self, path: StrPath, delimiter: str, /) -> int: ...
    def check_ragged(self, path: StrPath, delimiter: str, /) -> bool: ...
    def sniff_dialect(self, path: StrPath, delimiter: str | None = None, /) -> SniffedDialect | None: ...
    def probe_csv_header(self, path: StrPath, /) -> HeaderProbe: ...

The Power of Positional-Only Arguments (/)

You will notice that every function’s arguments end with the positional-only marker /. This is a strict engineering decision rather than a stylistic choice.

Under the hood, the pure-Python fallback originally named its path parameter filepath, whereas the compiled Rust extension named its parameter path (derived from PyO3 native variable mappings). In standard Python, calling either with a keyword parameter (e.g. backend().infer_delimiter(filepath=my_file)) would have crashed on the Rust engine, while backend().infer_delimiter(path=my_file) would have crashed on Python.

By declaring every single method’s parameters as positional-only (/), the Protocol statically forces developers to only call these methods positionally, making signature-naming differences across the backends completely safe and portable.

2. Dynamic Assignment Type-Checking

To force MyPy to statically verify that both backends implement the protocol, we created a static block using Python’s TYPE_CHECKING guard inside src/datagrunt/core/csv_io/_compute.py (_compute.py):

from typing import TYPE_CHECKING
from datagrunt import _native
from datagrunt.core.csv_io import _compute_python
from datagrunt.core.csv_io._compute_protocol import ComputeBackendProtocol

if TYPE_CHECKING:
    # Assigning each backend module to a Protocol-typed name makes mypy verify
    # both satisfy the contract. Drift in either is reported here, at edit time,
    # naming the offending member — rather than as a parity-suite failure later.
    _python_backend: ComputeBackendProtocol = _compute_python
    _rust_backend: ComputeBackendProtocol = _native

During testing and execution, the correct backend is dynamically selected:

def backend() -> ComputeBackendProtocol:
    """Return the active compute backend module (read at call time)."""
    return _compute_python if _DISABLE_RUST else _native

By assigning _compute_python and the native extension _native to variables statically type-declared as ComputeBackendProtocol, MyPy is forced to verify that every single shared method has matching positional-only arguments and mathematically sound return types.

3. Demystifying the Compiled Extension with Handwritten Type Stubs

To ensure MyPy does not treat the compiled native binary reference _native as Any, we implemented a handwritten .pyi type stub file: src/datagrunt/_native.pyi (_native.pyi).

This stub acts as an interface definition file that mirrors the compiled Rust module’s exported #[pyfunction] endpoints, allowing MyPy to statically inspect the types of compiled C/Rust code:

"""Type stub for the compiled Rust extension."""
from datagrunt.core.csv_io._compute_protocol import HeaderProbe, SniffedDialect, StrPath

def is_legacy_mac_newlines(path: StrPath, /) -> bool: ...
def leading_rows(path: StrPath, limit: int, /) -> list[str]: ...
...
def sniff_dialect(path: StrPath, delimiter: str | None = None, /) -> SniffedDialect | None: ...
def probe_csv_header(path: StrPath, /) -> HeaderProbe: ...

4. The Triple-Lock Security Strategy: AST and Runtime Verification

To guarantee that the hand-written _native.pyi type stub file does not drift from the compiled binary itself, version 4.5.8 establishes a triple-lock verification model:

  graph TD
    subgraph Static Analysis Gate (mypy)
        Protocol[ComputeBackendProtocol]
        MypyRun[mypy Type Checker]
        MypyRun -->|Enforces contract suitability| Protocol
        Protocol <-->|Verifies stubs matches| StubFile[_native.pyi]
    end

    subgraph Runtime Verification (pytest)
         AST[AST Parsing Test] -->|Matches callable exports exactly| StubFile
         AST <-->|Introspects callables| CompiledBinary[_native module]
         RuntimeExposure[Backend Exposure Test] -->|Verifies real callables exist| CompiledBinary
         RuntimeExposure -->|Verifies real callables exist| PythonOracle[_compute_python]
    end

Lock 1: Static Type Contract

MyPy validates at build/edit time that the custom Python fallback (_compute_python) and our handwritten stub file (_native.pyi) conform to ComputeBackendProtocol.

Lock 2: AST-Level Stub Verification

To prevent developers from adding a #[pyfunction] to the Rust compiled module without updating the type-stub file, we created an AST-level parsing test inside tests/core_tests/csv_io_tests/test_compute_protocol.py (test_compute_protocol.py):

import ast
from pathlib import Path
from datagrunt import _native

STUB_PATH = Path(datagrunt.__file__).parent / "_native.pyi"

def _stub_function_names() -> set[str]:
    # Parse the hand-written .pyi stub into an Abstract Syntax Tree
    tree = ast.parse(STUB_PATH.read_text())
    return {n.name for n in tree.body if isinstance(n, ast.FunctionDef)}

def test_stub_matches_the_compiled_module():
    """Adding a #[pyfunction] without updating the stub must fail here."""
    compiled = {n for n in dir(_native) if not n.startswith("_") and callable(getattr(_native, n))}
    assert _stub_function_names() == compiled

If a developer adds or removes a function in Rust, the AST parser immediately notices the discrepancy against _native.pyi and fails the test suite!

Lock 3: Full Runtime Backend Validation

To ensure that both backends actually expose every single required protocol callable at runtime without missing attributes, we run runtime presence assertions:

from datagrunt.core.csv_io import _compute_python

def test_both_backends_expose_every_protocol_function():
    """Neither backend may quietly drop a member of the shared surface."""
    for backend in (_native, _compute_python):
        missing = [fn for fn in BACKEND_FUNCTIONS if not callable(getattr(backend, fn, None))]
        assert not missing, f"{backend.__name__} is missing {missing}"

5. Integrating Scoped, Fast MyPy Checks in CI

Running MyPy across a full legacy Python codebase often introduces a wall of third-party library errors due to missing external library stubs.

To solve this, Datagrunt 4.5.8 adopts a scoped, progressive MyPy migration strategy. We configure MyPy specifically over our newly type-safe compute cores in pyproject.toml (pyproject.toml):

[tool.mypy]
python_version = "3.10"
mypy_path = "src"
# Scoped deliberately to the type-checked backend files
files = [
    "src/datagrunt/core/csv_io/_compute.py",
    "src/datagrunt/core/csv_io/_compute_protocol.py",
    "src/datagrunt/_native.pyi",
]
warn_unused_ignores = true
warn_redundant_casts = true

[[tool.mypy.overrides]]
module = [
    "datagrunt._native",
    "datagrunt.core.csv_io._compute",
    "datagrunt.core.csv_io._compute_protocol",
]
disallow_untyped_defs = true

We locked this configuration in CI with a dedicated static-checking gate inside .github/workflows/ci.yml (ci.yml):

      # Scoped to the compute-backend contract (files listed under [tool.mypy]),
      # not the whole tree. #314 tracks widening it.
      - name: "mypy (blocking — scoped, see #314)"
        run: .venv/bin/mypy

Additionally, MyPy minor-version updates periodically introduce new default-on error rules. To ensure CI remains bulletproof, we have pinned MyPy aggressively in pyproject.toml under the dev environment:

    # Upper-bounded the same way as ruff (#312): allow the current minor,
    # exclude the next. mypy adds default-on error codes in 1.x minor
    # releases, and this gate is blocking.
    "mypy>=1.14,<1.21",

This progressive implementation model ensures the most complex data pathways in our ecosystem are checked with maximum mathematical rigor, without delaying shipping velocities.

Installing Datagrunt v4.5.8

Upgrading to version 4.5.8 guarantees type safety, positional-only security, and robust dialect sensing:

# Force upgrade via uv
uv pip install --upgrade datagrunt

# Verify installation details and stub delivery
python -c "import datagrunt; print(f'Version: {datagrunt.__version__}')"

To view the full architectural design or learn how to run custom analytical workloads with our dual-backend engines, visit the Datagrunt Documentation Guide.