Datagrunt 4.0: A Rust-Accelerated CSV Core and the AI Feature Removal

Datagrunt 4.0 is the most significant release in the project’s history. It is a major, breaking release built around two headline changes: Datagrunt’s core CSV inference now runs in a bundled Rust extension, and the long-deprecated AI/LLM features have been removed. Along the way, 4.0 also adds a new class for processing large corpora of PDFs in parallel.

Because 4.0.0 is a breaking change, this post walks through everything you need to know to upgrade — what got faster, what was removed, and how to migrate.

1. A Rust-Accelerated CSV Core

Datagrunt’s original promise was simple: hand it a messy CSV and it figures out the delimiter and dialect for you, automatically. That inference — delimiter detection, a port of Python’s csv.Sniffer, comment handling, ragged-row detection, and record-aware row counting — is the work Datagrunt does on every CSV before any engine touches the data.

In 4.0, that work moves into a native Rust extension (datagrunt._native), built with PyO3 and shipped as platform wheels (abi3, Python 3.10+). There is nothing to install or configure — uv pip install datagrunt now resolves a prebuilt wheel for your platform, and the acceleration is on by default.

How much faster? On a 98 MB CSV (the Our World in Data COVID dataset, ~430k rows × 67 columns), inference dropped from 567 ms to 115 ms — roughly 5x faster — while producing byte-identical results.

Identical behavior, guaranteed

A native rewrite of core logic is only worth shipping if it behaves exactly like the code it replaces. Datagrunt 4.0 ships a complete pure-Python reference implementation alongside the Rust engine. The native engine is the default and is required on the platforms we ship wheels for — the Python version is not a runtime fallback — but it serves two purposes that keep the rewrite honest:

1. A differential-parity oracle — the test suite runs the Rust engine and the Python reference against the same edge-case corpus (BOM markers, legacy-Mac \r newlines, invalid UTF-8, quoted embedded newlines, the 64 KiB streaming-decoder chunk boundary, the 10,000-row ragged-scan cap, and more) and asserts the results match. The full test suite also runs end-to-end under both backends in CI.

2. A diagnostics toggle — force the pure-Python path to debug or A/B compare by setting an environment variable before importing Datagrunt:

   import os
   os.environ["DATAGRUNT_DISABLE_RUST"] = "1"  # use the pure-Python compute path

The upshot: you get the speedup by default, and because both paths are proven identical in CI, the results are the same whichever one runs.

2. The AI/LLM Features Have Been Removed (Breaking)

Datagrunt’s experimental AI schema-analysis features — CSVSchemaReportAIGenerated and the datagrunt.core.ai module — were deprecated in 3.3.0 with a clear warning that they would be removed in 4.0.0. That removal has now happened.

What this means for you: if your code imports CSVSchemaReportAIGenerated or anything under datagrunt.core.ai, it will no longer import on 4.0. This also means Datagrunt no longer carries any LLM-client dependencies.

Why remove them? Datagrunt’s value is in being a focused, dependable CSV/PDF utility. LLM schema analysis pulled the library toward a fast-moving, vendor-specific surface area (API keys, model names, provider SDKs) that sat awkwardly next to its deterministic, offline core. Removing it keeps Datagrunt small, predictable, and dependency-light.

Migration: if you still rely on AI schema reports, you have two clean options:

• Pin the old line: uv pip install "datagrunt<4.0" to stay on 3.x.
• Bring your own LLM: pass a representative sample to your own model client. CSVReader(...).get_sample() streams the first rows of any file (memory-bounded even on huge files), which is exactly the input the old feature sent to the model.

3. The 4.0.1 Patch: Packaging and Security

Shortly after 4.0.0, we shipped 4.0.1 to close out two issues that only surface once a native package meets the real world:

• Source distribution packaging. 4.0.0 published platform wheels successfully, but PyPI rejected the source tarball because of how the bundled build tool emitted the license metadata. 4.0.1 fixes this with a standards-based PEP 639 license declaration and a build that produces a valid sdist — so 4.0.1 is complete on PyPI (wheels and source). A twine check step now guards the release pipeline against metadata regressions.
• A dependency security bump. We upgraded PyO3 from 0.23 to 0.29, clearing three advisories flagged against the older version. The upgrade required no changes to Datagrunt’s own code.

Upgrading

uv pip install --upgrade datagrunt          # CSV core
uv pip install --upgrade "datagrunt[pdf]"   # with PDF support

For most users, the upgrade is transparent: same API, same results, faster CSV inference. The one thing to check before upgrading is whether you import any of the removed AI features — if so, follow the migration notes in section 2.

As always, Datagrunt stays focused on doing a few things well: making messy CSV and PDF files easy to work with. 4.0 makes the core faster and leaner without changing what you can count on.