Datagrunt 3.1.7: Resilient CSV Ingestion with Lenient Loading

CSV files are the lingua franca of data exchange, but they are rarely clean. Leading comments, duplicate headers, inconsistent line endings, and ragged (uneven) rows are common in the wild.

With Datagrunt 3.1.7, we are introducing a highly resilient CSV ingestion pipeline that guarantees data integrity by default, while offering a developer-friendly lenient pathway for loading imperfect data—all optimized for high efficiency on massive datasets.

1. Fail Loudly by Default

In previous versions, parsing ragged rows (where some lines have more columns than defined in the header) could lead to silent data truncation or misaligned rows depending on the engine.

In 3.1.7, we embrace a Pythonic “fail loudly” default across all three parsing engines:

• Polars & PyArrow: Strict schema alignment is enforced. Any extra columns or row formatting issues immediately raise a ComputeError or ArrowInvalid exception.
• DuckDB: Strict mode is enabled (strict_mode=true), catching parsing errors during sniffing or compilation.

This guarantees that your data is never silently dropped or corrupted during ingestion.

2. Lenient Ingestion with Developer Warnings

Sometimes, you just need to get the data loaded, even if a spreadsheet is slightly malformed. For these scenarios, we introduced a new lenient=True parameter:

from datagrunt import CSVReader

# Load messy data leniently without crashing
reader = CSVReader("messy_dataset.csv", engine="polars", lenient=True)
df = reader.to_dataframe()

When lenient=True is enabled:

1. Missing values are automatically padded with null.
2. Extra columns are safely truncated.
3. Developer Warnings are logged using standard Python UserWarning, pinpointing the exact line and mismatch details:

   UserWarning: CSV file contains ragged rows. Row 142 has 5 columns, expected 4. 
   Some fields will be truncated or padded with nulls.
4. Zero-Copy PyArrow Fallback: Since PyArrow does not natively support lenient row padding without skipping records, datagrunt automatically coordinates a Polars-to-Arrow schema conversion under the hood using zero-copy memory mapping (df.to_arrow()) to avoid any serialization overhead.

3. Legacy Mac OS Newline (\r) Support

Legacy Mac OS (version 9 and older) utilizes carriage returns (\r) rather than Unix line feeds (\n) or Windows carriage-return line-feeds (\r\n). Modern engines like Polars and PyArrow do not natively support these legacy endings and can crash or return silently corrupted empty dataframes.

Datagrunt 3.1.7 implements native binary chunk detection for legacy endings:

• Guided Errors: If you attempt to load a legacy \r file using Polars or PyArrow, datagrunt catches this early and raises a helpful, descriptive ValueError:

  ValueError: Polars engine does not support legacy Mac OS carriage return (\r) newlines. 
  Please use engine='duckdb' or convert the file to Unix/Windows newlines.
• DuckDB Fallback: DuckDB natively parses legacy newlines successfully. Simply setting engine="duckdb" allows the file to load cleanly.
• Header & Preview Extraction: Public API metadata features (like .columns, .columns_normalized, and .csv_string_sample) now automatically switch to a python-native fallback when legacy endings are detected, guaranteeing metadata parity across all engines.

4. Efficiency and Scale Optimization

We specifically optimized the metadata scanner and ingestion pathways to ensure that file analysis scales cleanly without memory overhead on gigabyte-sized files:

• Microsecond Metadata Scanning: Operations like comment counting (_count_leading_comments) and legacy newline detection (_is_legacy_mac_newlines) terminate early and use tiny byte buffers (reading just the first 4096 bytes), avoiding reading large files into memory during properties detection.
• Typing Optimizations: Explicit string schema coercion in PyArrow prevents expensive auto-inference parsing passes, protecting against parser crashes on mixed-type columns.
• Proven Scale: We benchmarked and verified 3.1.7 on a 30,000,000-row scale test (~2.8 GB):
  • Polars loaded the dataset in 0.36 seconds.
  • PyArrow loaded the dataset in 0.66 seconds.
  • DuckDB compiled the table in 4.2 seconds.

5. Automatic Header & Comment Parsing

Getting the columns of a CSV often fails when files contain comment headers (e.g. metadata prefix rows starting with #). In 3.1.7, datagrunt automatically counts and skips leading comments to locate the column row cleanly, while protecting actual data headers that contain the # symbol (like Phone#).

Upgrading

Datagrunt 3.1.7 is a minor release with zero breaking changes for well-formed files. To upgrade using UV:

uv pip install --upgrade datagrunt

Or using pip:

pip install --upgrade datagrunt