Datagrunt 4.6.1: Resolving API Boundary Anomalies and Lock-Free Build Concurrency

Datagrunt 4.6.1 has officially landed. While this is technically a patch release, it represents a major milestone in both code correctness and infrastructure robustness.

This release fixes a subtle boundary defect in our leading_rows CSV scanning API where asking for limit=0 returned one row instead of none. It also revamps our GitHub Actions CI/CD publishing pipeline with native Git rebase-retry logic and robust concurrency group guards, permanently resolving delivery-stage race conditions during batch approvals.

Deep Dive 1: Finding What Differential Testing Missed

In Datagrunt 4.5.9, we introduced differential property testing to assert that our compiled Rust core (datagrunt._native) and our pure-Python fallback/diagnostic oracle (_compute_python) behaved identically across thousands of randomized and CSV-biased byte strings.

However, as we explored in our v4.5.12 write-up, our subsequent investment in absolute boundary-constraint testing—using Rust’s proptest framework—exposed a shared bug that differential testing could never catch.

The Bug: An Identical Boundary Defect

Because both the Rust backend and the Python fallback were written with similar reference logic, they both made the same structural mistake: checking the row cap after appending the row to the result vector.

In Rust, the original implementation of rows.rs checked rows.len() < limit on succeeding lines, but by that point, at least one row had already been read, processed, and added. In Python, the exact same off-by-one check existed in _compute_python.py.

Because both engines suffered from the exact same defect, they returned identical (wrong) results. Since they agreed, our differential parity suite marked the behavior as “green.”

The Solution: Fast-Failing Zero Limits

Datagrunt 4.6.1 repairs this boundary contract on both sides of the foreign function interface (FFI) boundary, returning an empty list immediately when limit is zero or negative.

Crucially, unreadable or non-existent files must still fail fast and throw appropriate exceptions, even when the user asks for zero rows. The fast-path return sits directly after file opening, ensuring directory/read permissions are still validated.

1. Rust Fix in rows.rs

pub fn leading_rows(path: &Path, limit: usize) -> std::io::Result<Vec<String>> {
    let mut rows = Vec::new();
    let lines = universal_lines(path)?;
    
    // A zero limit asks for nothing. The check sits after the open so an
    // unreadable path still errors here, exactly as it does for every other
    // limit (issue #325).
    if limit == 0 {
        return Ok(rows);
    }
    
    for line in lines {
        // ... append and check logic ...
    }
}

2. Python Fix in _compute_python.py

def leading_rows(filepath: StrPath, limit: int) -> list[str]:
    """Up to ``limit`` leading non-blank, non-comment rows, each stripped."""
    rows: list[str] = []
    with open(filepath, "r", encoding=FileProperties(filepath).DEFAULT_ENCODING, errors="ignore") as f:
        # A non-positive limit asks for nothing. The check sits inside the
        # `with` so an unreadable path still raises here, exactly as it does
        # for every other limit (issue #325).
        if limit <= 0:
            return rows
        for line in _capped_lines(f):
            # ... append and check logic ...

Tightening the Test Harness

To secure this fix, we have tightened our testing suite on both ends:

1. We modified the leading_rows_respects_limit property in props_files.rs to enforce rows.len() <= limit unconditionally:

   fn leading_rows_respects_limit(bytes in csvish_bytes(), limit in 0usize..64) {
       let f = file_with(&bytes);
       if let Ok(rows) = datagrunt_core::rows::leading_rows(f.path(), limit) {
           prop_assert!(rows.len() <= limit);
       }
   }
2. We introduced a dedicated regression test file, test_leading_rows_limit.py, which asserts the correct behavior directly against both backends to guarantee long-term parity:

   @backends
   def test_limit_zero_returns_no_rows(backend, sample):
       assert backend.leading_rows(sample, 0) == []

Deep Dive 2: Solving CI Concurrency Race Conditions

When multiple pull requests or releases are approved in rapid succession, the corresponding documentation-publishing workflows run almost simultaneously. Historically, this caused a race condition: multiple pipeline runs attempted to update and push to the datagrunt-site content repository at the same time, leading to push failures and manual recovery.

Datagrunt 4.6.1 introduces three hardening improvements to our auto-publish-site.yml workflow:

1. Smart Concurrency Group Routing

GitHub’s default cancel-in-progress: true is too destructive here—canceling an active generation halfway through would leave the site codebase in an unstable, partially written state, and potentially miss updating documentation for intermediate versions.

Instead, we use a custom concurrency group combined with cancel-in-progress: false:

concurrency:
  group: auto-publish-site-${{ github.event_name == 'workflow_dispatch' && github.run_id || 'chain' }}
  cancel-in-progress: false

• Manual Runs: Workflow dispatches receive a unique run_id group, guaranteeing they are never canceled or queued.
• Automated Pushes: Standard automated events join a shared 'chain' group. GitHub leaves at most one pending run in the queue and automatically cancels any intermediate pending runs. Since each run documents the latest state of the repository, the newest release’s documentation is always guaranteed to build and deploy.

2. Idempotent Post Validation

The generator uses an AI-driven workflow that is content-idempotent but updates timestamps, which used to trigger false diffs and push conflicts. We now preemptively check whether a markdown release post for the current version exists:

VERSION=$(grep -m1 '^current_version = ' datagrunt/pyproject.toml | cut -d'"' -f2)
V_SLUG=$(echo "$VERSION" | tr . -)
if find datagrunt-site/content/blog -type f -name "datagrunt-${V_SLUG}-*.md" -print -quit | grep -q .; then
  echo "Release post for $VERSION already exists; skipping regeneration."
  echo "exists=true" >> "$GITHUB_OUTPUT"
else
  echo "exists=false" >> "$GITHUB_OUTPUT"
fi

If it exists, subsequent steps—dependencies setup, agent execution, and commits—are skipped entirely.

3. Belt-and-Braces Rebase Push

Finally, even if a concurrent change squeezes between our clone and our push, we handle it gracefully with a single-rebase push fallback retry:

git push origin main || { git pull --rebase origin main && git push origin main; }

This ensures that network timing gaps do not break the pipeline.

Summing Up the 4.6.1 Cleanup

With v4.6.1, we have successfully locked in correct bounds for our streaming scanner APIs and built a bulletproof publication engine. Upgrading to Datagrunt v4.6.1 is thoroughly recommended for all codebases processing streaming file records.

uv pip install --upgrade datagrunt