DuckDB Concurrency Starts at the Process Boundary

Version scope: DuckDB 2.0 documentation.

Before deciding whether DuckDB fits a service, draw the processes that will access its data. A notebook, an API worker and a scheduled job are three ownership domains even when they run on one machine. Counting threads instead of processes can hide the most important architectural decision.

For native in-process access, the current documentation distinguishes one read-write process from multiple read-only processes with no writer. Multiple threads within the writer process can perform concurrent work subject to transaction conflicts. Current documentation also describes remote and shared-catalog alternatives; therefore, a blanket claim that DuckDB can never support multiple client processes is misleading. DuckDB concurrency documentation

Choose an owner for the file

A straightforward embedded design assigns one service process ownership of a native database file. Requests enter through that service rather than having unrelated worker processes open the file for writes. Within that boundary, the application manages connections, transaction lifetimes, admission control and shutdown.

This is an architectural pattern, not a claim that putting HTTP in front of a database makes it suitable for every transactional workload. Measure the actual mix of updates, reads and request sizes. A service dominated by tiny, latency-sensitive mutations may need a different database design from a batch analytics worker.

If independent processes must write, evaluate a supported remote or catalog-coordinated architecture explicitly. Document its version, deployment components, availability expectations and operational maturity. Do not infer those properties from the behavior of a local embedded connection.

Retry the operation, not an arbitrary statement

Suppose two requests adjust the same account classification. A conflict is a signal that the attempted transaction needs reconsideration. Retrying only the final update may reuse a decision computed from stale reads.

An application-level retry procedure should:

1. Start a fresh transaction and reread the state used by the decision.
2. Recompute the intended change from the request's immutable inputs.
3. Apply the update and commit.
4. Retry only recognized transient conflicts, with a bounded attempt count.
5. Return a useful error when the retry budget is exhausted.

External effects require extra care. A transaction retry must not send a second email, charge a customer again or publish a duplicate event. Keep such effects outside the retry loop or give them a separate idempotency design. A database rollback cannot undo an external API call.

Test the boundary you intend to deploy

Use synthetic data and deterministic barriers in a concurrency test. Have two transactions read the same logical record, release both writers together and record which commits. Then verify the final state and the number of externally visible effects. Repeat with independent records to distinguish genuine conflicts from blanket serialization in your application.

Also test abrupt process termination during a job, graceful shutdown with queued requests and a second process attempting an unsupported access pattern. Record actual outcomes on the pinned release; this article contains no executed concurrency results.

A shared filesystem deserves a separate review. Local success does not establish correct locking or failure behavior on network-mounted storage. The deployment decision should name the storage environment rather than treating all paths as equivalent.

Make ownership legible

Put the owning process, connection strategy and supported clients in the runbook. Explain how an operator stops the owner safely and how maintenance access works. If the architecture changes to a remote service, revise that diagram and the failure tests.

The core question is not how many callers exist. It is which supported component coordinates their writes and what the application does when two valid requests conflict.

In this embedded design, clients do not open the database file directly. Other supported deployment modes require a different diagram.