Datagrunt 4.5.9: Automated Differential Property Testing with Hypothesis and Zero-Alloc Phantom Line Correction

Datagrunt 4.5.9 is here, adding maximum mathematical certainty and bug correction to our CSV processing core. Continuing from our rigid contract enforcement in version 4.5.8 (which bound our high-speed compiled Rust engine and pure-Python reference fallback under formal PEP 544 Protocols), version 4.5.9 is all about verification.

By introducing a complete, custom property-based testing (PBT) harness powered by Hypothesis, mapping out dozens of “seams” (like CRLF terminators, ragged width lines, invalid UTF-8 traces, and blank strings), and developing a nightly, out-of-band deep search container, we have successfully run tens of thousands of random data tests on our dual-backend engines.

In this cycle, our property-based harness successfully surfaced and helped us resolve a subtle trailing-byte boundary edge case in our universal line-reading engine (UniversalLines in Rust). By resolving this issue, we guarantee absolute, byte-for-byte behavioral replication between the compiled native extension and our fallback standard under every conceivable set of corrupted or incomplete input bytes.

The Challenge: Trailing Edge Cases on Decoded Buffers

Datagrunt implements a custom UniversalLines iterator in io.rs to cleanly process lines with multiple line terminator conventions (LF, CRLF, and legacy Mac CR) over compiled Rust binaries. It uses streaming byte reads for performance and memory optimization.

However, a subtle divergence arises when dealing with invalid trailing UTF-8 sequences. Our decoder is specified with errors="ignore" parameters (mimicking CPython’s behavior when reading damaged text or truncated UTF-8 streams, especially with comments or header fields on Web scrapes).

Under this model, certain invalid byte sequences are completely discarded, resulting in an empty string ("") when decoded to UTF-8.

Why This Caused a Discrepancy

In standard Python, reading a file with an invalid trailing byte (e.g. b"\x80") works by first reading and decoding the byte stream into a Python standard string str. If b"\x80" is ignored, the resulting decoded string becomes "". Iterating over this empty string produces exactly zero lines.

In contrast, our native UniversalLines core worked directly on the raw byte stream. If the stream had non-zero remaining bytes (like our b"\x80" byte), it entered the “remaining tail” condition, performed decode_ignore() on the buffer, and produced Some("") (an empty line).

This meant that:

• Pure Python returned 0 lines.
• Rust returned 1 empty line!

This “phantom trailing empty line” (Issue #317) is a extreme edge case but could ripple into CSV column-count parsing errors, header offset calculations, or comment line checks inside count_leading_physical_lines_before_header.

The Fix: Zero-Alloc Trail Boundary Validation

To align the native compiled Rust engine with CPython, we edited io.rs to perform an explicit character run correctness check on newly decoded trailing lines:

                    let raw = decode_ignore(&self.buf[self.pos..]);
                    let line = take_chars(&raw, MAX_LINE_CHARS);
                    self.pos = self.buf.len();
                    // Bytes remaining is a BYTE-level condition; a line is
                    // DECODED text. A tail whose every byte is dropped by
                    // errors="ignore" is not a line: CPython iterates the
                    // decoded string and yields nothing for it, so emitting
                    // Some("") here produced a phantom final line (#317).
                    // Only the unterminated tail is affected — an explicitly
                    // terminated empty line (b"a\n\n") is emitted by the \n
                    // branch above and still counts, matching CPython.
                    if line.is_empty() {
                        return Ok(None);
                    }
                    return Ok(Some(line));

By verifying that the decoded trailing line contains actual character content, we prevent the creation of phantom lines. This resolves Issue #317 while preserving standard, explicitly-terminated empty lines (e.g. b"a\n\n"), matching CPython exactly.

1. Introducing Property-Based Differential Parity Testing

How did we find this needle in a haystack of bytes? Through property-based differential parity testing, designed using the Hypothesis runtime engine inside test_parity_property.py.

Traditional unit tests are deterministic: someone writes down an input, describes the expected output, and runs it. But in messy real-world scenarios—from missing delimiters to truncated multi-byte files—unit tests can never cover the infinite state space of corrupted or malformed files.

Property-based testing solves this by allowing the test runner to generate thousands of random inputs based on structured strategies, executing both backends, and checking that their results are mathematically identical (differential parity).

  graph TD
    st_csv[Hypothesis csv_bytes Strategy] -->|Generates random CSV-shaped data| file[Temporary CSV File]
    file -->|Read| rust_engine[Rust Extension _native]
    file -->|Read| py_engine[Python core _compute_python]
    rust_engine -->|Returns value or exception| rust_outcome[outcome]
    py_engine -->|Returns value or exception| py_outcome[outcome]
    rust_outcome === py_outcome{Are Outcomes Equal?}
    py_outcome -->|No / Exception mismatch| hyp_shrink[Hypothesis Minimized Repro]
    py_outcome -->|Yes| hyp_next[Next Example]

To prevent testing simple scenarios over and over, our strategy in strategies.py maps and tracks which exact “seams” and boundaries each generated file triggers under SEAM_LABELS:

SEAM_LABELS = frozenset(
    {
        "bom",
        "lf",
        "crlf",
        "legacy-mac",
        "quoted",
        "embedded-delimiter",
        "embedded-newline",
        "comments",
        "blank-lines",
        "ragged",
        "invalid-utf8",
        "c0-controls",
        "no-trailing-newline",
        "tsv-extension",
        "empty",
        "single-column",
        "nul-byte",
    }
)

2. Developing the Hypothesis CSV Generator Strategy

In strategies.py, we assembled a highly comprehensive composite generator strategy (csv_bytes()) that dynamically pieces together valid and invalid structures to trigger edge cases:

@st.composite
def csv_bytes(draw) -> GeneratedCSV:
    """Build a CSV-shaped example plus the set of seams it exercises."""
    seams: set[str] = set()

    delimiter = draw(st.sampled_from(DELIMITERS))
    terminator = draw(st.sampled_from(LINE_TERMINATORS))
    seams.add({"\n": "lf", "\r\n": "crlf", "\r": "legacy-mac"}[terminator])

    suffix = draw(st.sampled_from(EXTENSIONS))
    if suffix.lower() == ".tsv":
        seams.add("tsv-extension")

    n_fields = draw(st.integers(min_value=1, max_value=5))
    if n_fields == 1:
        seams.add("single-column")
    n_rows = draw(st.integers(min_value=0, max_value=6))

    def make_field() -> str:
        text = draw(_TEXT)
        if _rare(draw):
            text += draw(st.sampled_from(C0_WHITESPACE))
            seams.add("c0-controls")
        if _rare(draw):
            text += NUL
            seams.add("nul-byte")
        if _rare(draw):
            text += delimiter
            seams.add("embedded-delimiter")
            text = f'"{text}"'
            seams.add("quoted")
        elif _rare(draw):
            text = f'"{text}\n more"'
            seams.add("embedded-newline")
            seams.add("quoted")
        elif draw(st.booleans()):
            text = f'"{text}"'
            seams.add("quoted")
        return text

    lines: list[str] = []

    for _ in range(draw(st.integers(min_value=0, max_value=3))):
        lines.append("# " + draw(_TEXT))
        seams.add("comments")

    header = [make_field() for _ in range(n_fields)]
    lines.append(delimiter.join(header))

    for _ in range(n_rows):
        if _rare(draw):
            lines.append("")
            seams.add("blank-lines")
        width = n_fields
        if _rare(draw):
            width = draw(st.integers(min_value=1, max_value=n_fields + 2))
            if width != n_fields:
                seams.add("ragged")
        lines.append(delimiter.join(make_field() for _ in range(width)))

    text = terminator.join(lines)
    if lines and draw(st.booleans()):
        text += terminator
    else:
        seams.add("no-trailing-newline")

    data = text.encode("utf-8")

    if _rare(draw):
        data = BOM + data
        seams.add("bom")
    if _rare(draw):
        data += INVALID_UTF8
        seams.add("invalid-utf8")
    if not data:
        seams.add("empty")

    return GeneratedCSV(data=data, suffix=suffix, seams=frozenset(seams))

To compare outcomes cleanly, we created a clever outcome() wrapper. Rust panics crossing PyO3 raise a PanicException. By catching BaseException, this normalizer collapses execution returns and errors into a comparable tuple. It maps standard IO or OS issues to standard family class names, allowing Hypothesis to shrink failures straight down to minimal reproducible configurations:

def outcome(fn, *args):
    """Collapse a call into a comparable ('ok', value) or ('raised', name)."""
    try:
        return ("ok", fn(*args))
    except (KeyboardInterrupt, SystemExit):
        raise
    except OSError:
        return ("raised", "OSError")
    except BaseException as exc:
        return ("raised", type(exc).__name__)

3. High-Depth Out-of-Band Exploration in CI/CD

Because property-based search is CPU-intensive, running thousands of test iterations on every standard pull-request gate would slow down development.

To balance speed and safety, Datagrunt 4.5.9 splits testing into two profiles configured in conftest.py:

1. The PR Gate (ci): Runs derandomized at max_examples=100. Fast, reproducible, and highly deterministic.
2. The Deep Exploration Gate (deep): Configured in .github/workflows/deep-property-search.yml, which executes off the critical path on scheduled cron triggers (Mondays, 06:00 UTC) at max_examples=10,000.

To keep search speed maximized, this overnight job disables coverage tracing, and strictly caches the Hypothesis database (.hypothesis/) across pipeline runs. This means any historically failing edge cases are verified first on subsequent runs:

      - name: Restore the Hypothesis example database
        uses: actions/cache@v4
        with:
          path: .hypothesis
          key: hypothesis-${{ github.run_id }}
          restore-keys: hypothesis-

      - name: Deep property search
        env:
          HYPOTHESIS_PROFILE: deep
        run: .venv/bin/pytest tests/parity/test_parity_property.py -q --no-cov

If a scheduled build discovers a new divergence, the pipeline fails, presenting developers with a minimized repro script and saving the failing binary example to the database, ensuring zero regressions enter our production releases.

Installing Datagrunt v4.5.9

Upgrading to version 4.5.9 guarantees flawless trailing-byte handling and continuous architectural alignment:

# Force upgrade via uv
uv pip install --upgrade datagrunt

# Verify installation details
python -c "import datagrunt; print(f'Version: {datagrunt.__version__}')"

To learn more about engine optimizations, explore the CSV Engines and Rust Acceleration Guide.