Datagrunt 4.5.10: Aligning Rust's Native Dialect Sniffer with CPython Alphanumeric Unicode Rules and Unleashing Comprehensive Delimiter Parity Testing

Datagrunt 4.5.10 is officially available, delivering a premium bug fix and test coverage expansion for our compiled CSV-handling backend. Focused entirely on absolute mathematical correctness and feature parity across execution environments, this release addresses a highly technical, cross-platform Unicode divergence within our high-speed native CSV dialect sniffer and unlocks a previously hidden testing “blind spot” inside our automated Hypothesis property-based verification suite.

By replacing native regular expression word-boundary character classes with explicit CPython-equivalent Unicode properties in Rust, we guarantee mistake-free delimiter detection and CSV schema inference on complex multi-byte text files.

The Challenge: Regex Character Class Disparities (\w) across Environments

Datagrunt relies on a high-speed compiled native extension (datagrunt._native) built in Rust to process CSV, Excel, Parquet, and PDF files. To ensure seamless operation, we maintain a pure-Python reference fallback that behaves identically byte-for-byte.

To infer the structure of a CSV file automatically (such as identifying candidate delimiters, quoting modes, doublequote handling, and header offsets), both backends run statistical dialect sniffing. Our Rust extension implements these patterns using the fast and robust fancy-regex crate, which supports the backtracking and Python-style lookarounds needed for CPython-compatible parsing.

However, when processing special or non-ASCII multi-byte datasets, we uncovered a subtle but critical sniffer divergence (Issue #318): are superscript numbers, mark characters, and accented codepoints classified as “word characters”?

Why This Mismatch Occurred

In CPython, the \w character class is defined using str.isalnum() and the underscore character. Under Unicode, this maps strictly to characters belonging to general categories L* (Letters) and N* (Numbers), plus _.

In contrast, the fancy-regex crate (and the underlying regex-syntax library in Rust) implements UTS#18 perl_word class for \w. UTS#18 defines a word character as: $$\text{perl_word} = \text{Alphabetic} \cup \text{M (Marks)} \cup \text{Nd (Decimal Numbers)} \cup \text{Pc (Connector Punctuation)} \cup \text{Join_Control}$$

This subtle definitional mismatch caused the two regular-expression engines to diverge in two directions:

1. Excluding Unicode Number Categories (No/Nl) in Rust:
   The superscript representation of two, ² (U+00B2), belongs to Unicode general category No (Other Number). To CPython, this character is alphanumeric (str.isalnum() == True) and is considered a word character (\w), meaning it is excluded from candidate delimiters (which are defined as [^\w\n"']). However, to Rust, ² does not land in UTS#18 perl_word, mapping instead to a non-word character. Consequently, Rust’s sniffer matched ² in the negative class [^\w\n"'] and treated it as a potential column delimiter!

2. Including Unicode Mark Categories (M) in Rust:
   Combining accents—such as the Combining Acute Accent ◌́ (U+0301)—belong to the general Unicode category Mn (Nonspacing Mark). To CPython, combining marks are not alphanumeric, so they are classified as non-word characters. To Rust, they are part of M, classifying them as word characters.

This divergence is visualized in the flowchart below:

  graph TD
    cpython["CPython \\w Definitions (str.isalnum + '_')"]
    rust["Rust fancy-regex \\w Definitions (UTS#18 perl_word)"]

    cpython -->|"Includes Category No (e.g., U+00B2 '²')"| cp_exclusive["CPython Word Characters"]
    rust -->|"Excludes Category No (Treats '²' as candidate delimiter [^\\w])"| cp_exclusive

    rust -->|"Includes Category M (e.g., U+0301 Combining Accent)"| rust_exclusive["Rust Word Characters"]
    cpython -->|"Excludes Category M (Treats accent as candidate delimiter)"| rust_exclusive

    cpython -->|"Includes Letters (L*) & Decimal Digits (Nd)"| common["Common Intersection ('a'..'z', '0'..'9', '_')"]
    rust -->|"Includes Letters (L*) & Decimal Digits (Nd)"| common

Character Classification Matrix

To see the exact discrepancy in action, consider how specific codepoints were evaluated by both engines prior to our fix:

Because our sniffer uses the negative set of word characters ([^\w\n"']) to infer delimiters, and the inverse class (\W) to guess whether doublequoting is active, this definition divergence caused dialect and delimiter mismatches on files containing these edge-case characters.

How It Was Found: Uncovering the Testing Blind Spot

Our continuous integration environment runs differential property-based testing using Hypothesis. The test suite generates thousands of randomized CSV structures dynamically and ensures that both the Rust native engine and pure-Python core produce identical results.

However, we had an architectural “blind spot” in test_parity_dialect.py:

def dialect_properties_from_rust(rust_dict):
    """Apply CSVDialect's property defaults to the raw Rust sniff result."""
    if rust_dict is None:
        return {
            "quotechar": '"',
            "escapechar": None,
            "doublequote": False,
            "skipinitialspace": False,
            "quoting": "quote minimal",
        }
    return {
        "quotechar": rust_dict["quotechar"],
        "escapechar": rust_dict["escapechar"],
        "doublequote": rust_dict["doublequote"],
        "skipinitialspace": rust_dict["skipinitialspace"],
        "quoting": rust_dict["quoting"],
    }

Notice that delimiter is omitted from this comparison catalog! Because we had separate unit tests verifying the standalone infer_delimiter function, we incorrectly assumed that the comprehensive sniff_dialect property suite did not need to verify the delimiter return value.

But sniff_dialect runs on separate logic than infer_delimiter. By leaving delimiter out of the property tests, this cross-backend regex divergence remained completely hidden until we updated the function to explicitly evaluate the delimiter attribute (#319):

 def dialect_properties_from_rust(rust_dict):
     if rust_dict is None:
         return {
+            "delimiter": None,
             "quotechar": '"',
             "escapechar": None,
             "doublequote": False,
@@ -17,4 +25,5 @@
             "quoting": "quote minimal",
         }
     return {
+        "delimiter": rust_dict["delimiter"],
         "quotechar": rust_dict["quotechar"],
         "escapechar": rust_dict["escapechar"],

Once this was committed, our high-depth property-based testing pipelines immediately caught the failure and generated a minimal, elegant regression case. This led us to flag the test with a strict xfail (Issue #318) until the underlying Rust core was aligned.

The Fix: CPython Alphanumeric Equivalence in Rust

To resolve this discrepancy, we edited dialect.rs to bypass the default \w expansion entirely.

First, we defined the exact Unicode categories that represent CPython’s word characters as a shared static regex fragment:

/// CPython's `\w`, spelled out — do NOT write `\w` in a pattern here.
///
/// CPython defines `\w` as "alphanumeric characters (as defined by
/// `str.isalnum()`) as well as the underscore", i.e. general categories L* and
/// N* plus `_`. fancy-regex inherits regex-syntax's UTS#18 `perl_word`, which is
/// `Alphabetic ∪ M ∪ Nd ∪ Pc ∪ Join_Control` — a DIFFERENT set, and different in
/// both directions.
const PY_WORD: &str = r"\p{L}\p{N}_";

We exhaustively verified this definition by running and comparing re.match(r"\w", ch) against category(ch)[0] in "LN" or ch == "_" over all 1,114,112 possible Unicode codepoints, resulting in a 100% mathematical match.

Second, we reconstructed our static sniffer patterns at initialization time using LazyLock string interpolation, bypassing any compiled-level \w ambiguities:

static COMPILED_QUOTE_PATTERNS: LazyLock<[(Regex, bool); 4]> = LazyLock::new(|| {
    let patterns: [(String, bool); 4] = [
        (
            format!(
                r#"(?sm)(?P<delim>[^{PY_WORD}\n"'])(?P<space> ?)(?P<quote>["']).*?\k<quote>\k<delim>"#
            ),
            true,
        ),
        (
            format!(
                r#"(?sm)(?:^|\n)(?P<quote>["']).*?\k<quote>(?P<delim>[^{PY_WORD}\n"'])(?P<space> ?)"#
            ),
            true,
        ),
        (
            format!(
                r#"(?sm)(?P<delim>[^{PY_WORD}\n"'])(?P<space> ?)(?P<quote>["']).*?\k<quote>(?:$|\n)"#
            ),
            true,
        ),
        (
            r#"(?sm)(?:^|\n)(?P<quote>["']).*?\k<quote>(?:$|\n)"#.to_string(),
            false,
        ),
    ];
    patterns.map(|(pattern, has_groups)| {
        (
            Regex::new(&pattern).expect("static sniffer pattern is valid"),
            has_groups,
        )
    })
});

And did the same for doublequote prediction guessing:

    // `[^{PY_WORD}]` rather than `\W`: same reason as QUOTE_PATTERNS above —
    // fancy-regex's `\W` is the complement of a different word set than
    // CPython's, which flipped `doublequote` as well as `delimiter` (#318).
    let dq_pattern = format!(
        r"(?m)(({delim})|^)[^{word}]*{quote}[^{delim}\n]*{quote}[^{delim}\n]*{quote}[^{word}]*(({delim})|$)",
        delim = escaped_delim,
        quote = quotechar,
        word = PY_WORD,
    );

By switching to explicit class fragments, we eliminated the runtime divergence without adding any memory allocations or string formatting overhead during the sniffing execution loop.

Hardened Regression Safeguards

To prevent future discrepancies, we activated several safeguards:

1. Uncovering the xfail: We removed the protective @pytest.mark.xfail(strict=True...) on test_parity_property.py so that any mismatch will immediately break the compilation pipeline.
2. Permanent Regression Case: We checked in the Hypothesis-minimized falsifying specimen (b'"\'"\'\n"\xc2\xb2\'"' containing the ² character) as an explicit, permanent regression scenario inside test_parity_property.py to assert that it passes on every development machine:

@example(
    # #318's minimized falsifying example, kept as a permanent regression case
    # now that it is fixed. U+00B2 is a word character to CPython (category No)
    # but was not to fancy-regex, so the sniffer's [^\w\n"'] delimiter class
    # disagreed about whether it could be a delimiter at all.
    gen=GeneratedCSV(
        data=b'"\'"\'\n"\xc2\xb2\'"',
        suffix=".csv",  # Limit diff size if it is extremely long
    )
)

Both our PR-gated tests (100 examples per run) and our massive, out-of-band scheduled scheduled deep-property pipelines (10,000 runs) are passing flawlessly with zero failures.

Upgrading to Datagrunt v4.5.10

Upgrading to version 4.5.10 is seamless and highly recommended for any environment processing international, accented, or non-ASCII CSV datasets:

# Force upgrade via uv
uv pip install --upgrade datagrunt

# Verify the installed native version details
python -c "import datagrunt; print(f'Version: {datagrunt.__version__}')"

To learn more about Datagrunt’s engine integration and native speed acceleration, explore our CSV Engines and Rust Acceleration Guide.