Datagrunt Updates: Context Managers, Class-Level Normalization, and Hardening (v3.2.0 - v3.4.1)

Following our comprehensive 3.1.40 hardening release, we have continued to improve Datagrunt’s robustness, developer experience, and efficiency. Across the v3.2, v3.3, and v3.4 release lines, we have introduced key usability features like the context manager protocol, class-level column normalization, and a clean deprecation strategy for AI tools.

This post consolidates the updates across versions 3.2.0 through 3.4.1 so you can easily catch up on what’s new.

1. Context Manager Protocol: Clean Connections (v3.4.x)

One of Datagrunt’s great strengths is its lazy DuckDB integration. A local DuckDB connection is automatically initialized whenever you execute a SQL query on a CSV file. While convenient, keeping this connection open indefinitely can lead to locked resources.

In v3.4.0, we introduced formal context manager protocol support (__enter__ and __exit__) for CSVReader and DuckDBQueries. You can now wrap your reader in a standard with block to guarantee that the database connection is closed deterministically, even if an exception is raised inside the block:

from datagrunt import CSVReader

# Recommended pattern for DuckDB SQL queries
with CSVReader("electric_vehicle_population_data.csv", engine="duckdb") as dg:
    query = f"SELECT city, COUNT(*) FROM {dg.db_table} GROUP BY 1"
    df = dg.query_data(query).pl()
    print(df)

# The connection is closed automatically here

If you construct CSVReader manually outside of a context manager, you can also call dg.close() directly. The reader remains fully usable after closing; any subsequent query will transparently reopen a fresh connection.

2. Class-Level Column Normalization (v3.2.x)

Prior to version 3.2.0, column name normalization was specified on a per-call basis (e.g. to_dataframe(normalize_columns=True)). This approach proved problematic because different queries and operations had to repeatedly request normalization, risking inconsistent column mappings and query mismatches.

Column normalization has now been moved to the class constructor level of CSVReader and CSVWriter. When you initialize the class with normalize_columns=True, every subsequent operation (including DuckDB SQL table imports) automatically adheres to the same schema:

from datagrunt import CSVReader

# Configure normalization once at construction
dg = CSVReader("data.csv", engine="duckdb", normalize_columns=True)

# The DuckDB table is imported with normalized names,
# allowing you to write queries with normalized names immediately
query = f"SELECT vin_1_10, model_year FROM {dg.db_table} LIMIT 5"
df = dg.query_data(query).pl()

# Outputs automatically use normalized names
df_all = dg.to_dataframe()
sample = dg.get_sample()

3. AI/LLM Feature Deprecation (v3.3.x)

To align Datagrunt’s core focus on highly performant, local, and reliable file parsing, we have made the decision to deprecate all AI/LLM-powered features as of version 3.3.0.

These features—including CSVSchemaReportAIGenerated and the datagrunt.core.ai module—will be completely removed in version 4.0.0. Constructing these classes will now emit a standard python DeprecationWarning pointing to Issue #144.

If you currently rely on CSVSchemaReportAIGenerated for schema inference, we recommend migrating to Datagrunt’s native metadata-based inference engines (CSVReader.columns and CSVReader.columns_normalized), which run entirely locally and are zero-dependency.

4. Key Hardening and Performance Fixes

Alongside these structural changes, a series of important bug fixes and optimizations were rolled out:

• Sniffer & Delimiter Inference (v3.2.x): Improved CSV delimiter sniffer to perform multi-pass inference combined with row-consistency validation. This prevents mis-detecting delimiters in files with mixed structures or messy data.
• DuckDB Table Prefixing (v3.2.x): Fixed an issue where filenames starting with a number (e.g. 2026_sales.csv) generated invalid DuckDB table names. Table names are now automatically prefixed to prevent SQL syntax errors on import.
• Negative-Y Coordinates in PDFs (v3.2.x): Hardened the layout sorter to correctly handle and order elements with negative Y-coordinates, preventing coordinates from throwing index errors or producing wrong reading orders.
• Binary File Classification (v3.2.x): Fixed a bug where BlankFile check misclassified certain binary files as blank instead of treating them as invalid text.
• Efficient MD5 Chunked Streaming (v3.3.x): Embedded image deduplication previously read entire image bytes into memory to compute MD5 hashes. It now streams the files in chunks, resulting in significant memory savings when parsing PDFs with large images.

Upgrade to Datagrunt 3.4.1

Get all the latest optimizations, safety features, and improvements today by upgrading:

uv pip install --upgrade datagrunt