DuckDB run-record store for local files or MotherDuck databases. Inserts accept structs and derive SQL columns from their fields.
transaction/2 commits successful results and rolls back errors or
exceptions, keeping source rows and their ingest-ledger entry atomic.
Summary
Functions
Close the connection and the database now, rather than whenever the garbage collector reaches them. DuckDB writes its log back into the database file as it closes, so a second instance opened on the same file in the same OS process must not overlap the first one's close. The store cannot be used afterwards.
The ledger key for a source file's contents: the hex sha256 of its bytes.
Whether contents with digest were already ingested, under any file name.
Whether a file named source_file was ingested from source, with any contents.
Record that contents with digest were rejected for reason, so a file
that is left in place is reported once per distinct contents, not on every
run.
Whether contents with digest were already rejected, under any file name.
Types
Functions
@spec close(t()) :: :ok
Close the connection and the database now, rather than whenever the garbage collector reaches them. DuckDB writes its log back into the database file as it closes, so a second instance opened on the same file in the same OS process must not overlap the first one's close. The store cannot be used afterwards.
The ledger key for a source file's contents: the hex sha256 of its bytes.
The ledger is keyed by content, not by name, because a producer can reuse a file name for different contents, and a name match would skip the new contents silently.
Whether contents with digest were already ingested, under any file name.
Whether a file named source_file was ingested from source, with any contents.
@spec insert(t(), String.t(), [struct()]) :: {:ok, non_neg_integer()} | {:error, term()}
@spec record_rejection(t(), String.t(), String.t(), String.t(), String.t()) :: :ok | {:error, term()}
Record that contents with digest were rejected for reason, so a file
that is left in place is reported once per distinct contents, not on every
run.
Whether contents with digest were already rejected, under any file name.