Add queue tables
A table with `queue_table` set works as a job queue that many workers can
pull from at once. Claims use SELECT ... FOR UPDATE SKIP LOCKED over the
idx_queue_claim index order, so each job goes to one worker and only the
claimed rows are locked.
- Schema: adds the queue_* columns and the idx_queue_claim /
idx_queue_lease indexes. Declaring these names yourself is an error.
- `queue_table` accepts `true` or { max_attempts, lease_ms, keep_completed }
- BunMariaDB.queue: push, claim, ack, fail, extend, cancel
- Statuses: pending, processing, done, failed, cancelled
- Leases use the database clock. Expired leases are reclaimed (or
dead-lettered on the last attempt) and old lock tokens stop working.
- insert/update can't set queue state fields. update/delete skip claimed
jobs.
Tests: allow the test container's self-signed certificate, and give the
queue test its own client so it doesn't reuse one closed by db.test.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Claude Opus 5.5
@@ -29,6 +29,7 @@ A schema-driven MariaDB manager for [Bun](https://bun.sh), featuring automatic s
|
||||
- [Update](#update)
|
||||
- [Delete](#delete)
|
||||
- [Raw SQL](#raw-sql)
|
||||
- [Queue Tables](#queue-tables)
|
||||
- [Query API Reference](#query-api-reference)
|
||||
- [Vector Support](#vector-support)
|
||||
- [HTML Sanitization](#html-sanitization)
|
||||
@@ -243,6 +244,7 @@ interface BUN_MARIADB_TableSchemaType {
|
||||
parentTableName?: string; // inherit / merge fields from another table
|
||||
tableNameOld?: string; // rename: old name triggers ALTER TABLE RENAME
|
||||
collation?: "utf8mb4_bin" | "utf8mb4_unicode_520_ci";
|
||||
queue_table?: boolean | BUN_MARIADB_QueueTableOptions; // job queue, see Queue Tables
|
||||
}
|
||||
```
|
||||
|
||||
@@ -688,6 +690,126 @@ console.log(res.payload);
|
||||
|
||||
---
|
||||
|
||||
## Queue Tables
|
||||
|
||||
Set `queue_table: true` on a table to use it as a job queue. Any number of workers can pull from it at the same time: claims use `SELECT ... FOR UPDATE SKIP LOCKED` (MariaDB 10.6+), so no job is handed to two workers and workers don't wait on each other.
|
||||
|
||||
```ts
|
||||
{
|
||||
tableName: "email_jobs",
|
||||
queue_table: true,
|
||||
fields: [
|
||||
{ fieldName: "to", dataType: "VARCHAR", dataLength: 255 },
|
||||
{ fieldName: "subject", dataType: "VARCHAR", dataLength: 255 },
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
Pass an options object instead of `true` to set the queue's defaults. Arguments to the `queue` methods override them.
|
||||
|
||||
```ts
|
||||
queue_table: {
|
||||
max_attempts: 5, // attempts before a job is marked `failed` (default 5)
|
||||
lease_ms: 60_000, // how long a claim lasts (default 30s)
|
||||
keep_completed: true, // `ack` marks jobs `done` instead of deleting them (default false)
|
||||
}
|
||||
```
|
||||
|
||||
`bunx bun-mariadb schema` adds these columns, plus the `idx_queue_claim` and `idx_queue_lease` indexes. They are managed by the library: declaring a field or index with one of these names on a queue table is an error.
|
||||
|
||||
| Field | Type | Description |
|
||||
| ------------------------ | --------------------------------------- | --------------------------------------------------- |
|
||||
| `queue_status` | `ENUM(...)` | Job state, see below |
|
||||
| `queue_available_at` | `BIGINT` (ms) | The job can't be claimed before this time |
|
||||
| `queue_attempts` | `INT` | How many times the job has been claimed |
|
||||
| `queue_lease_expires_at` | `BIGINT` (ms) | When the current claim expires |
|
||||
| `queue_lock_token` | `VARCHAR(64)` | Token of the current claim |
|
||||
| `queue_last_error` | `TEXT` | Error from the last failed attempt |
|
||||
|
||||
Statuses:
|
||||
|
||||
| Status | Meaning |
|
||||
| ------------ | ----------------------------------------------------------------- |
|
||||
| `pending` | Waiting to be claimed. The only status `claim` picks up |
|
||||
| `processing` | Claimed by a worker |
|
||||
| `done` | Acknowledged and kept (`keep_completed`). Otherwise jobs are deleted |
|
||||
| `failed` | Used up its attempts (dead letter) |
|
||||
| `cancelled` | Taken out of the queue with `queue.cancel` |
|
||||
|
||||
The statuses are fixed because the queue's SQL depends on them. For your own job outcomes (`sent`, `bounced`, ...), add a field of your own and set it before calling `ack`.
|
||||
|
||||
### Worker loop
|
||||
|
||||
```ts
|
||||
await BunMariaDB.queue.push({
|
||||
table: "email_jobs",
|
||||
data: [{ to: "[email protected]", subject: "Hi" }],
|
||||
delay_ms: 5_000, // optional
|
||||
});
|
||||
|
||||
const { payload: jobs } = await BunMariaDB.queue.claim({
|
||||
table: "email_jobs",
|
||||
limit: 10,
|
||||
lease_ms: 60_000,
|
||||
});
|
||||
|
||||
for (const job of jobs || []) {
|
||||
try {
|
||||
await sendEmail(job);
|
||||
await BunMariaDB.queue.ack({
|
||||
table: "email_jobs",
|
||||
id: job.id,
|
||||
lock_token: job.queue_lock_token,
|
||||
});
|
||||
} catch (e: any) {
|
||||
await BunMariaDB.queue.fail({
|
||||
table: "email_jobs",
|
||||
id: job.id,
|
||||
lock_token: job.queue_lock_token,
|
||||
error: e.message,
|
||||
retry_delay_ms: 30_000,
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Method | Description |
|
||||
| ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| `queue.push({ table, data, delay_ms? })` | Insert jobs as `pending` |
|
||||
| `queue.claim({ table, limit?, lease_ms?, max_attempts? })` | Claim up to `limit` (default 1) jobs, oldest first, for `lease_ms` (default 30s) |
|
||||
| `queue.ack({ table, id, lock_token })` | Job done: deletes it, or marks it `done` with `keep_completed` |
|
||||
| `queue.fail({ table, id, lock_token, error?, retry_delay_ms?, max_attempts? })` | Back to `pending` after `retry_delay_ms`, or `failed` once `max_attempts` (default 5) is reached |
|
||||
| `queue.extend({ table, id, lock_token, lease_ms? })` | Extend the lease for a long-running job |
|
||||
| `queue.cancel({ table, id })` | Take a `pending` job out of the queue. Claimed jobs can't be cancelled |
|
||||
|
||||
Delivery is **at least once**. If a worker crashes or runs past its lease, the next `claim` puts the job back in the queue (or marks it `failed` on its last attempt), and the next claim gets a new `queue_lock_token`. After that, `ack` / `fail` / `extend` with the old token return `success: false`. Make job handlers idempotent. Lease times use the database clock, so worker clock drift doesn't matter.
|
||||
|
||||
How regular CRUD treats queue tables:
|
||||
|
||||
- `insert` / `update` refuse to set `queue_status`, `queue_attempts`, `queue_lease_expires_at`, `queue_lock_token` or `queue_last_error`. Only the queue API changes those.
|
||||
- `update` / `delete` skip jobs that are `processing`, so a worker's claimed job can't change or disappear while it runs.
|
||||
- `select` is unchanged. Filter on `queue_status` to get the rows you want:
|
||||
|
||||
```ts
|
||||
// Dead letters
|
||||
await BunMariaDB.select({
|
||||
table: "email_jobs",
|
||||
query: { query: { queue_status: { value: "failed" } } },
|
||||
});
|
||||
|
||||
// Everything still in the queue
|
||||
await BunMariaDB.select({
|
||||
table: "email_jobs",
|
||||
count: true,
|
||||
query: { query: { queue_status: { value: ["pending", "processing"], equality: "IN" } } },
|
||||
});
|
||||
```
|
||||
|
||||
- To retry a dead-lettered job, delete it and push it again, or reset it with raw `sql`.
|
||||
- With `keep_completed`, `done` rows stay until you delete them, e.g. `delete` with a `queue_status` / `updated_at` filter.
|
||||
|
||||
---
|
||||
|
||||
## Query API Reference
|
||||
|
||||
`query` on `select` / `update` / `delete` accepts `ServerQueryParam<T>`:
|
||||
|
||||
Reference in New Issue
Block a user