Add queue tables, quote SQL identifiers (v1.3.0)
- isQueue table option adds queue columns and a (status, available_at) index; BunSQLite.queue exposes add, claim, complete, fail and release_stale with atomic claims - Set busy_timeout on connections so workers in separate processes wait instead of failing with SQLITE_BUSY - Quote table and column names with backticks in generated SQL, so reserved words (to, order, group) work and unknown columns error instead of matching as string literals - Tests for queue ops, multi-process claims and reserved-word names Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
@@ -29,6 +29,7 @@ A schema-driven SQLite manager for [Bun](https://bun.sh), featuring automatic sc
|
||||
- [Raw SQL](#raw-sql)
|
||||
- [Query API Reference](#query-api-reference)
|
||||
- [Vector Table Support](#vector-table-support)
|
||||
- [Queue Tables](#queue-tables)
|
||||
- [TypeScript Type Generation](#typescript-type-generation)
|
||||
- [Default Fields](#default-fields)
|
||||
- [Project Structure](#project-structure)
|
||||
@@ -204,6 +205,7 @@ interface BUN_SQLITE_TableSchemaType {
|
||||
tableNameOld?: string; // rename: set this to the old name to trigger ALTER TABLE RENAME
|
||||
isVector?: boolean; // mark this as a sqlite-vec virtual table
|
||||
vectorType?: string; // virtual table type, defaults to "vec0"
|
||||
isQueue?: boolean | BUN_SQLITE_QueueOptions; // make this a job queue (see Queue Tables)
|
||||
}
|
||||
```
|
||||
|
||||
@@ -759,6 +761,75 @@ const res = await BunSQLite.select({
|
||||
|
||||
---
|
||||
|
||||
## Queue Tables
|
||||
|
||||
Set `isQueue` on a table to use it as a job queue. Your fields are the job payload; the schema sync adds the queue columns and an index on `(status, available_at)`.
|
||||
|
||||
```ts
|
||||
{
|
||||
tableName: "email_jobs",
|
||||
isQueue: {
|
||||
maxAttempts: 3, // claims before a failing job is marked "failed" (default 3)
|
||||
lockTimeout: 300_000, // ms a job can stay "processing" before it is released (default 5 min)
|
||||
backoff: 1000, // base retry delay in ms, doubled each attempt (default 1000)
|
||||
deleteOnComplete: false, // delete completed jobs instead of keeping them (default false)
|
||||
},
|
||||
fields: [
|
||||
{ fieldName: "recipient", dataType: "TEXT" },
|
||||
{ fieldName: "subject", dataType: "TEXT" },
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
`isQueue: true` uses the defaults. A table can't be both a queue and a vector table.
|
||||
|
||||
| Field | Type | Description |
|
||||
| -------------- | ---------------------------- | ---------------------------------------------------- |
|
||||
| `status` | `TEXT NOT NULL` | `pending`, `processing`, `completed` or `failed` |
|
||||
| `attempts` | `INTEGER NOT NULL DEFAULT 0` | Times the job has been claimed |
|
||||
| `available_at` | `INTEGER NOT NULL DEFAULT 0` | Unix timestamp (ms) from which the job can be claimed |
|
||||
| `locked_at` | `INTEGER` | Unix timestamp (ms) when the job was claimed |
|
||||
| `locked_by` | `TEXT` | Worker that claimed the job |
|
||||
| `last_error` | `TEXT` | Error from the last failed attempt |
|
||||
|
||||
### Working the queue
|
||||
|
||||
```ts
|
||||
// Enqueue (optionally delayed, in ms)
|
||||
await BunSQLite.queue.add({
|
||||
table: "email_jobs",
|
||||
data: [{ recipient: "[email protected]", subject: "Hi" }],
|
||||
delay: 0,
|
||||
});
|
||||
|
||||
// Worker loop
|
||||
const { payload: jobs } = await BunSQLite.queue.claim({
|
||||
table: "email_jobs",
|
||||
worker: "worker-1",
|
||||
limit: 10,
|
||||
});
|
||||
|
||||
for (const job of jobs || []) {
|
||||
try {
|
||||
await sendEmail(job);
|
||||
await BunSQLite.queue.complete({ table: "email_jobs", id: job.id });
|
||||
} catch (error: any) {
|
||||
await BunSQLite.queue.fail({
|
||||
table: "email_jobs",
|
||||
id: job.id,
|
||||
error: error.message,
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `claim` sets jobs to `processing` in a single `UPDATE ... RETURNING` statement. SQLite only allows one writer at a time, so two workers never get the same job, even across processes.
|
||||
- `fail` puts the job back to `pending` with an exponential backoff, or sets it to `failed` once it has used `maxAttempts`.
|
||||
- Jobs stuck in `processing` longer than `lockTimeout` (for example, the worker crashed) are released by `claim` before it picks jobs. You can also call `BunSQLite.queue.release_stale({ table })` directly.
|
||||
- `complete` and `fail` only touch jobs that are `processing`.
|
||||
|
||||
---
|
||||
|
||||
## TypeScript Type Generation
|
||||
|
||||
Run the `typedef` command (or pass `--typedef` to `schema`) to generate a `.ts` file containing:
|
||||
|
||||
Reference in New Issue
Block a user