Query Parquet Without Losing the Data Contract

Version scope: DuckDB 2.0. The SQL in this article was run on DuckDB 2.0.0 against small synthetic fixtures.

A query that reads Parquet successfully has proved that the engine can interpret the file. It has not proved that the file represents the dataset your report expects. Direct file analysis becomes dependable when file identity, schema and business meaning travel together.

Consider a daily revenue calculation. The same SQL can produce different answers after an upstream export adds a second currency, republishes yesterday's partition or changes the definition of a completed order. The database may report no error. The analysis still needs to fail its acceptance checks.

Define the input before writing the aggregation

For this example, the dataset contains one row per order, an immutable order identifier, a business date, a currency code and an amount expressed in the currency's minor unit. The report accepts only completed orders in one declared currency. Refunds are outside this deliberately narrow example and require their own accounting policy.

Keep a manifest next to the query with the exact input file list, file digests, export identifier, schema version and report cutoff. A directory wildcard is convenient during exploration, but it can quietly include a backfill or an unfinished export. Freeze the file list for a reviewable run.

DuckDB supports direct Parquet scans and can push column selection and compatible filters into the reader. That makes the file path a useful analytical interface, provided the data contract is independently checked. DuckDB Parquet documentation

Reject ambiguity before computing revenue

The following SQL is an illustrative example, run here against a small synthetic fixture.

CREATE TEMP TABLE report_input AS
SELECT order_id, business_date, currency, amount_minor, status
FROM read_parquet(['orders-2026-10-01.parquet']);

SELECT order_id, count(*) AS copies
FROM report_input
GROUP BY order_id
HAVING count(*) <> 1;

SELECT count(*) AS invalid_rows
FROM report_input
WHERE order_id IS NULL
   OR business_date IS NULL
   OR currency IS NULL
   OR amount_minor IS NULL
   OR status IS NULL;

SELECT currency, sum(amount_minor) AS completed_amount_minor
FROM report_input
WHERE business_date = DATE '2026-10-01'
  AND status = 'completed'
GROUP BY currency
ORDER BY currency;

The harness should stop if the first query returns any row or the second returns a nonzero count. Grouping by currency prevents accidental arithmetic across incompatible units; the harness must additionally assert that the output contains only the expected currency. Do not silently filter unexpected currencies out of an input that was supposed to contain exactly one.

The temporary table gives all checks and the calculation the same captured rows. It does not make a mutable remote export an atomic snapshot. The producer must publish immutable files or provide a snapshot contract that the consumer can verify.

Make failure visible

Test a duplicate order, a missing amount, an unexpected currency and a file that contains no rows for the requested date. Decide whether an empty business day is valid or a missing export. A zero total cannot distinguish those situations.

Keep validation counts and the input manifest with the output. Avoid saving sensitive row values in diagnostic logs. If validation fails, retain an identifiable failed run rather than overwriting yesterday's accepted report.

Direct querying saves an ingestion step. It does not remove ownership of meaning. The useful boundary is an immutable input plus explicit acceptance checks, not simply a file that SQL can open.

The report is released only after checks pass against the same captured input.