Add queue tables
A table with `queue_table` set works as a job queue that many workers can
pull from at once. Claims use SELECT ... FOR UPDATE SKIP LOCKED over the
idx_queue_claim index order, so each job goes to one worker and only the
claimed rows are locked.
- Schema: adds the queue_* columns and the idx_queue_claim /
idx_queue_lease indexes. Declaring these names yourself is an error.
- `queue_table` accepts `true` or { max_attempts, lease_ms, keep_completed }
- BunMariaDB.queue: push, claim, ack, fail, extend, cancel
- Statuses: pending, processing, done, failed, cancelled
- Leases use the database clock. Expired leases are reclaimed (or
dead-lettered on the last attempt) and old lock tokens stop working.
- insert/update can't set queue state fields. update/delete skip claimed
jobs.
Tests: allow the test container's self-signed certificate, and give the
queue test its own client so it doesn't reuse one closed by db.test.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Claude Opus 5.5
@@ -29,6 +29,7 @@ A schema-driven MariaDB manager for [Bun](https://bun.sh), featuring automatic s
|
||||
- [Update](#update)
|
||||
- [Delete](#delete)
|
||||
- [Raw SQL](#raw-sql)
|
||||
- [Queue Tables](#queue-tables)
|
||||
- [Query API Reference](#query-api-reference)
|
||||
- [Vector Support](#vector-support)
|
||||
- [HTML Sanitization](#html-sanitization)
|
||||
@@ -243,6 +244,7 @@ interface BUN_MARIADB_TableSchemaType {
|
||||
parentTableName?: string; // inherit / merge fields from another table
|
||||
tableNameOld?: string; // rename: old name triggers ALTER TABLE RENAME
|
||||
collation?: "utf8mb4_bin" | "utf8mb4_unicode_520_ci";
|
||||
queue_table?: boolean | BUN_MARIADB_QueueTableOptions; // job queue, see Queue Tables
|
||||
}
|
||||
```
|
||||
|
||||
@@ -688,6 +690,126 @@ console.log(res.payload);
|
||||
|
||||
---
|
||||
|
||||
## Queue Tables
|
||||
|
||||
Set `queue_table: true` on a table to use it as a job queue. Any number of workers can pull from it at the same time: claims use `SELECT ... FOR UPDATE SKIP LOCKED` (MariaDB 10.6+), so no job is handed to two workers and workers don't wait on each other.
|
||||
|
||||
```ts
|
||||
{
|
||||
tableName: "email_jobs",
|
||||
queue_table: true,
|
||||
fields: [
|
||||
{ fieldName: "to", dataType: "VARCHAR", dataLength: 255 },
|
||||
{ fieldName: "subject", dataType: "VARCHAR", dataLength: 255 },
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
Pass an options object instead of `true` to set the queue's defaults. Arguments to the `queue` methods override them.
|
||||
|
||||
```ts
|
||||
queue_table: {
|
||||
max_attempts: 5, // attempts before a job is marked `failed` (default 5)
|
||||
lease_ms: 60_000, // how long a claim lasts (default 30s)
|
||||
keep_completed: true, // `ack` marks jobs `done` instead of deleting them (default false)
|
||||
}
|
||||
```
|
||||
|
||||
`bunx bun-mariadb schema` adds these columns, plus the `idx_queue_claim` and `idx_queue_lease` indexes. They are managed by the library: declaring a field or index with one of these names on a queue table is an error.
|
||||
|
||||
| Field | Type | Description |
|
||||
| ------------------------ | --------------------------------------- | --------------------------------------------------- |
|
||||
| `queue_status` | `ENUM(...)` | Job state, see below |
|
||||
| `queue_available_at` | `BIGINT` (ms) | The job can't be claimed before this time |
|
||||
| `queue_attempts` | `INT` | How many times the job has been claimed |
|
||||
| `queue_lease_expires_at` | `BIGINT` (ms) | When the current claim expires |
|
||||
| `queue_lock_token` | `VARCHAR(64)` | Token of the current claim |
|
||||
| `queue_last_error` | `TEXT` | Error from the last failed attempt |
|
||||
|
||||
Statuses:
|
||||
|
||||
| Status | Meaning |
|
||||
| ------------ | ----------------------------------------------------------------- |
|
||||
| `pending` | Waiting to be claimed. The only status `claim` picks up |
|
||||
| `processing` | Claimed by a worker |
|
||||
| `done` | Acknowledged and kept (`keep_completed`). Otherwise jobs are deleted |
|
||||
| `failed` | Used up its attempts (dead letter) |
|
||||
| `cancelled` | Taken out of the queue with `queue.cancel` |
|
||||
|
||||
The statuses are fixed because the queue's SQL depends on them. For your own job outcomes (`sent`, `bounced`, ...), add a field of your own and set it before calling `ack`.
|
||||
|
||||
### Worker loop
|
||||
|
||||
```ts
|
||||
await BunMariaDB.queue.push({
|
||||
table: "email_jobs",
|
||||
data: [{ to: "[email protected]", subject: "Hi" }],
|
||||
delay_ms: 5_000, // optional
|
||||
});
|
||||
|
||||
const { payload: jobs } = await BunMariaDB.queue.claim({
|
||||
table: "email_jobs",
|
||||
limit: 10,
|
||||
lease_ms: 60_000,
|
||||
});
|
||||
|
||||
for (const job of jobs || []) {
|
||||
try {
|
||||
await sendEmail(job);
|
||||
await BunMariaDB.queue.ack({
|
||||
table: "email_jobs",
|
||||
id: job.id,
|
||||
lock_token: job.queue_lock_token,
|
||||
});
|
||||
} catch (e: any) {
|
||||
await BunMariaDB.queue.fail({
|
||||
table: "email_jobs",
|
||||
id: job.id,
|
||||
lock_token: job.queue_lock_token,
|
||||
error: e.message,
|
||||
retry_delay_ms: 30_000,
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Method | Description |
|
||||
| ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| `queue.push({ table, data, delay_ms? })` | Insert jobs as `pending` |
|
||||
| `queue.claim({ table, limit?, lease_ms?, max_attempts? })` | Claim up to `limit` (default 1) jobs, oldest first, for `lease_ms` (default 30s) |
|
||||
| `queue.ack({ table, id, lock_token })` | Job done: deletes it, or marks it `done` with `keep_completed` |
|
||||
| `queue.fail({ table, id, lock_token, error?, retry_delay_ms?, max_attempts? })` | Back to `pending` after `retry_delay_ms`, or `failed` once `max_attempts` (default 5) is reached |
|
||||
| `queue.extend({ table, id, lock_token, lease_ms? })` | Extend the lease for a long-running job |
|
||||
| `queue.cancel({ table, id })` | Take a `pending` job out of the queue. Claimed jobs can't be cancelled |
|
||||
|
||||
Delivery is **at least once**. If a worker crashes or runs past its lease, the next `claim` puts the job back in the queue (or marks it `failed` on its last attempt), and the next claim gets a new `queue_lock_token`. After that, `ack` / `fail` / `extend` with the old token return `success: false`. Make job handlers idempotent. Lease times use the database clock, so worker clock drift doesn't matter.
|
||||
|
||||
How regular CRUD treats queue tables:
|
||||
|
||||
- `insert` / `update` refuse to set `queue_status`, `queue_attempts`, `queue_lease_expires_at`, `queue_lock_token` or `queue_last_error`. Only the queue API changes those.
|
||||
- `update` / `delete` skip jobs that are `processing`, so a worker's claimed job can't change or disappear while it runs.
|
||||
- `select` is unchanged. Filter on `queue_status` to get the rows you want:
|
||||
|
||||
```ts
|
||||
// Dead letters
|
||||
await BunMariaDB.select({
|
||||
table: "email_jobs",
|
||||
query: { query: { queue_status: { value: "failed" } } },
|
||||
});
|
||||
|
||||
// Everything still in the queue
|
||||
await BunMariaDB.select({
|
||||
table: "email_jobs",
|
||||
count: true,
|
||||
query: { query: { queue_status: { value: ["pending", "processing"], equality: "IN" } } },
|
||||
});
|
||||
```
|
||||
|
||||
- To retry a dead-lettered job, delete it and push it again, or reset it with raw `sql`.
|
||||
- With `keep_completed`, `done` rows stay until you delete them, e.g. `delete` with a `queue_status` / `updated_at` filter.
|
||||
|
||||
---
|
||||
|
||||
## Query API Reference
|
||||
|
||||
`query` on `select` / `update` / `delete` accepts `ServerQueryParam<T>`:
|
||||
|
||||
@@ -4,6 +4,12 @@ import DbInsert from "./lib/mariadb/db-insert";
|
||||
import DbSelect from "./lib/mariadb/db-select";
|
||||
import DbSQL from "./lib/mariadb/db-sql";
|
||||
import DbUpdate from "./lib/mariadb/db-update";
|
||||
import QueueAck from "./lib/queue/queue-ack";
|
||||
import QueueCancel from "./lib/queue/queue-cancel";
|
||||
import QueueClaim from "./lib/queue/queue-claim";
|
||||
import QueueExtend from "./lib/queue/queue-extend";
|
||||
import QueueFail from "./lib/queue/queue-fail";
|
||||
import QueuePush from "./lib/queue/queue-push";
|
||||
import grabDbSchema from "./utils/grab-db-schema";
|
||||
import grabJoinFieldsFromQueryObject from "./utils/grab-join-fields-from-query-object";
|
||||
|
||||
@@ -24,6 +30,14 @@ const BunMariaDB = {
|
||||
update: withInit(DbUpdate),
|
||||
delete: withInit(DbDelete),
|
||||
sql: withInit(DbSQL),
|
||||
queue: {
|
||||
push: withInit(QueuePush),
|
||||
claim: withInit(QueueClaim),
|
||||
ack: withInit(QueueAck),
|
||||
fail: withInit(QueueFail),
|
||||
extend: withInit(QueueExtend),
|
||||
cancel: withInit(QueueCancel),
|
||||
},
|
||||
utils: {
|
||||
grab_db_schema: withInit(grabDbSchema),
|
||||
grab_join_fields_from_query_object: grabJoinFieldsFromQueryObject,
|
||||
@@ -50,5 +64,8 @@ export {
|
||||
DockerComposeServices,
|
||||
IndexTypes,
|
||||
DefaultFields,
|
||||
QueueStatuses,
|
||||
QueueFields,
|
||||
QueueProtectedFields,
|
||||
RequiredENVs,
|
||||
} from "./types";
|
||||
|
||||
@@ -7,6 +7,7 @@ import type {
|
||||
} from "../../types";
|
||||
import sqlGenerator from "../../utils/sql-generator";
|
||||
import { quoteIdentifier } from "../../utils/sql-identifier";
|
||||
import { isQueueTable, QUEUE_NOT_CLAIMED_SQL } from "../queue/queue-utils";
|
||||
|
||||
type Params<
|
||||
Schema extends { [k: string]: any } = { [k: string]: any },
|
||||
@@ -38,7 +39,7 @@ export default async function DbDelete<
|
||||
targetId,
|
||||
});
|
||||
|
||||
const whereClause = sqlObj.where;
|
||||
let whereClause = sqlObj.where;
|
||||
|
||||
if (!whereClause) {
|
||||
return {
|
||||
@@ -50,6 +51,10 @@ export default async function DbDelete<
|
||||
};
|
||||
}
|
||||
|
||||
if (isQueueTable(table)) {
|
||||
whereClause += ` AND ${QUEUE_NOT_CLAIMED_SQL}`;
|
||||
}
|
||||
|
||||
let sql = `DELETE FROM ${quoteIdentifier(table)} ${whereClause}`;
|
||||
sql += ` RETURNING *`;
|
||||
sqlObj.string = sql;
|
||||
|
||||
@@ -7,6 +7,10 @@ import type {
|
||||
import sqlInsertGenerator from "../../utils/sql-insert-generator";
|
||||
import { sanitizeHtmlFieldsBatch } from "../../utils/sanitize-html-fields";
|
||||
import grabDuplicateSafeInsertSql from "../grab-duplicate-safe-insert-sql";
|
||||
import {
|
||||
assertNoProtectedQueueFields,
|
||||
isQueueTable,
|
||||
} from "../queue/queue-utils";
|
||||
|
||||
type Params<
|
||||
Schema extends { [k: string]: any } = { [k: string]: any },
|
||||
@@ -30,6 +34,10 @@ export default async function DbInsert<
|
||||
let sqlObj: SQLInsertGenReturn | null = null;
|
||||
|
||||
try {
|
||||
if (isQueueTable(table)) {
|
||||
assertNoProtectedQueueFields(table, data);
|
||||
}
|
||||
|
||||
const sanitizedData = sanitizeHtmlFieldsBatch({
|
||||
table,
|
||||
data,
|
||||
|
||||
@@ -10,6 +10,11 @@ import sanitizeHtmlFields from "../../utils/sanitize-html-fields";
|
||||
import { quoteIdentifier } from "../../utils/sql-identifier";
|
||||
import sqlValuePlaceholder from "../../utils/sql-value-placeholder";
|
||||
import grabTableFields from "../../utils/grab-table-fields";
|
||||
import {
|
||||
assertNoProtectedQueueFields,
|
||||
isQueueTable,
|
||||
QUEUE_NOT_CLAIMED_SQL,
|
||||
} from "../queue/queue-utils";
|
||||
|
||||
type Params<
|
||||
Schema extends { [k: string]: any } = { [k: string]: any },
|
||||
@@ -55,7 +60,7 @@ export default async function DbUpdate<
|
||||
targetId,
|
||||
});
|
||||
|
||||
const whereClause = sqlQueryObj.where;
|
||||
let whereClause = sqlQueryObj.where;
|
||||
|
||||
if (!whereClause) {
|
||||
return {
|
||||
@@ -64,6 +69,11 @@ export default async function DbUpdate<
|
||||
};
|
||||
}
|
||||
|
||||
if (isQueueTable(table)) {
|
||||
assertNoProtectedQueueFields(table, [data]);
|
||||
whereClause += ` AND ${QUEUE_NOT_CLAIMED_SQL}`;
|
||||
}
|
||||
|
||||
const sanitizedData = sanitizeHtmlFields({
|
||||
table,
|
||||
data,
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
import dbHandler from "../db-handler";
|
||||
import type { BunMariaDBConfig, DBResponseObject } from "../../types";
|
||||
import { quoteIdentifier } from "../../utils/sql-identifier";
|
||||
import { QUEUE_NOW_SQL, requireQueueTable } from "./queue-utils";
|
||||
|
||||
type Params<Table extends string = string> = {
|
||||
table: Table;
|
||||
id: number | string;
|
||||
/** `queue_lock_token` from the claimed row */
|
||||
lock_token: string;
|
||||
config?: BunMariaDBConfig;
|
||||
};
|
||||
|
||||
/**
|
||||
* Mark a claimed job as done: deletes it, or sets it to `done` when the
|
||||
* table has `keep_completed`.
|
||||
*
|
||||
* Fails when the lock token no longer matches, i.e. the lease expired and
|
||||
* another worker reclaimed the job.
|
||||
*/
|
||||
export default async function QueueAck<Table extends string = string>({
|
||||
table,
|
||||
id,
|
||||
lock_token,
|
||||
config,
|
||||
}: Params<Table>): Promise<DBResponseObject> {
|
||||
try {
|
||||
const options = requireQueueTable(table);
|
||||
const tableSql = quoteIdentifier(table);
|
||||
const claimedWhere = `WHERE id = ? AND queue_status = 'processing' AND queue_lock_token = ?`;
|
||||
|
||||
if (!options.keep_completed) {
|
||||
const res = await dbHandler({
|
||||
query: `DELETE FROM ${tableSql} ${claimedWhere} RETURNING *`,
|
||||
values: [id, lock_token],
|
||||
config,
|
||||
});
|
||||
|
||||
if (!res.success) {
|
||||
return { ...res, success: false };
|
||||
}
|
||||
|
||||
if (!res.payload?.length) {
|
||||
return {
|
||||
...res,
|
||||
success: false,
|
||||
msg: "Job is not claimed with this lock token",
|
||||
};
|
||||
}
|
||||
|
||||
return res;
|
||||
}
|
||||
|
||||
const res = await dbHandler({
|
||||
query: `UPDATE ${tableSql} SET queue_status = 'done', queue_lease_expires_at = NULL, queue_lock_token = NULL, updated_at = ${QUEUE_NOW_SQL} ${claimedWhere}`,
|
||||
values: [id, lock_token],
|
||||
config,
|
||||
});
|
||||
|
||||
if (!res.success) {
|
||||
return { ...res, success: false };
|
||||
}
|
||||
|
||||
if (!res.db_res?.affectedRows) {
|
||||
return {
|
||||
...res,
|
||||
success: false,
|
||||
msg: "Job is not claimed with this lock token",
|
||||
};
|
||||
}
|
||||
|
||||
return await dbHandler({
|
||||
query: `SELECT * FROM ${tableSql} WHERE id = ?`,
|
||||
values: [id],
|
||||
config,
|
||||
});
|
||||
} catch (error: any) {
|
||||
return {
|
||||
success: false,
|
||||
error: error.message,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
import dbHandler from "../db-handler";
|
||||
import type { BunMariaDBConfig, DBResponseObject } from "../../types";
|
||||
import { quoteIdentifier } from "../../utils/sql-identifier";
|
||||
import { QUEUE_NOW_SQL, requireQueueTable } from "./queue-utils";
|
||||
|
||||
type Params<Table extends string = string> = {
|
||||
table: Table;
|
||||
id: number | string;
|
||||
config?: BunMariaDBConfig;
|
||||
};
|
||||
|
||||
/**
|
||||
* Take a `pending` job out of the queue without deleting it. Jobs a worker
|
||||
* has already claimed can't be cancelled.
|
||||
*/
|
||||
export default async function QueueCancel<Table extends string = string>({
|
||||
table,
|
||||
id,
|
||||
config,
|
||||
}: Params<Table>): Promise<DBResponseObject> {
|
||||
try {
|
||||
requireQueueTable(table);
|
||||
const tableSql = quoteIdentifier(table);
|
||||
|
||||
const res = await dbHandler({
|
||||
query: `UPDATE ${tableSql} SET queue_status = 'cancelled', updated_at = ${QUEUE_NOW_SQL} WHERE id = ? AND queue_status = 'pending'`,
|
||||
values: [id],
|
||||
config,
|
||||
});
|
||||
|
||||
if (!res.success) {
|
||||
return { ...res, success: false };
|
||||
}
|
||||
|
||||
if (!res.db_res?.affectedRows) {
|
||||
return {
|
||||
...res,
|
||||
success: false,
|
||||
msg: "Job is not pending",
|
||||
};
|
||||
}
|
||||
|
||||
return await dbHandler({
|
||||
query: `SELECT * FROM ${tableSql} WHERE id = ?`,
|
||||
values: [id],
|
||||
config,
|
||||
});
|
||||
} catch (error: any) {
|
||||
return {
|
||||
success: false,
|
||||
error: error.message,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
import dbHandler, { holdEventLoop, resolveClient } from "../db-handler";
|
||||
import type { BunMariaDBConfig, DBResponseObject } from "../../types";
|
||||
import { quoteIdentifier } from "../../utils/sql-identifier";
|
||||
import { QUEUE_NOW_SQL, requireQueueTable } from "./queue-utils";
|
||||
|
||||
type Params<Table extends string = string> = {
|
||||
table: Table;
|
||||
/** Max jobs to claim. Defaults to 1 */
|
||||
limit?: number;
|
||||
/** How long the claim lasts before another worker may take the job. Defaults to the table's `lease_ms` */
|
||||
lease_ms?: number;
|
||||
/** Jobs whose lease expired after this many attempts are marked `failed`. Defaults to the table's `max_attempts` */
|
||||
max_attempts?: number;
|
||||
config?: BunMariaDBConfig;
|
||||
};
|
||||
|
||||
/**
|
||||
* Claim the next available jobs, oldest `queue_available_at` first. Each
|
||||
* returned row carries a `queue_lock_token` that must be passed to `ack`,
|
||||
* `fail` or `extend`.
|
||||
*
|
||||
* `FOR UPDATE SKIP LOCKED` lets concurrent workers claim different jobs
|
||||
* without blocking on each other or claiming the same job twice.
|
||||
*/
|
||||
export default async function QueueClaim<
|
||||
Schema extends { [k: string]: any } = { [k: string]: any },
|
||||
Table extends string = string,
|
||||
>({
|
||||
table,
|
||||
limit = 1,
|
||||
lease_ms,
|
||||
max_attempts,
|
||||
config,
|
||||
}: Params<Table>): Promise<DBResponseObject<Schema>> {
|
||||
try {
|
||||
const options = requireQueueTable(table);
|
||||
|
||||
const tableSql = quoteIdentifier(table);
|
||||
const finalLimit = Math.max(1, Math.floor(Number(limit) || 1));
|
||||
const leaseMs = Math.max(
|
||||
0,
|
||||
Math.floor(Number(lease_ms ?? options.lease_ms) || 0),
|
||||
);
|
||||
const maxAttempts = Math.max(
|
||||
1,
|
||||
Math.floor(Number(max_attempts ?? options.max_attempts) || 1),
|
||||
);
|
||||
|
||||
/**
|
||||
* Release jobs whose worker died or stalled past its lease: back to
|
||||
* `pending`, or `failed` if that was the last attempt.
|
||||
*/
|
||||
const expired_res = await dbHandler({
|
||||
query: `UPDATE ${tableSql} SET queue_status = IF(queue_attempts >= ?, 'failed', 'pending'), queue_lease_expires_at = NULL, queue_lock_token = NULL, queue_last_error = 'Lease expired', updated_at = ${QUEUE_NOW_SQL} WHERE queue_status = 'processing' AND queue_lease_expires_at <= ${QUEUE_NOW_SQL}`,
|
||||
values: [maxAttempts],
|
||||
config,
|
||||
});
|
||||
|
||||
if (!expired_res.success) {
|
||||
return { ...expired_res, success: false } as DBResponseObject<Schema>;
|
||||
}
|
||||
|
||||
const CLIENT = resolveClient(config);
|
||||
|
||||
if (!CLIENT) {
|
||||
throw new Error(`Couldn't grab MariaDB Client.`);
|
||||
}
|
||||
|
||||
const lock_token = crypto.randomUUID();
|
||||
|
||||
return await holdEventLoop(() =>
|
||||
CLIENT.begin(async (tx) => {
|
||||
/**
|
||||
* The ORDER BY must match `idx_queue_claim`. A filesort would
|
||||
* lock every pending row before LIMIT applies, and SKIP LOCKED
|
||||
* would then hide the whole queue from other workers.
|
||||
*/
|
||||
const matched_res = await dbHandler<{ id: number }>({
|
||||
query: `SELECT id FROM ${tableSql} FORCE INDEX (idx_queue_claim) WHERE queue_status = 'pending' AND queue_available_at <= ${QUEUE_NOW_SQL} ORDER BY queue_status, queue_available_at, id LIMIT ? FOR UPDATE SKIP LOCKED`,
|
||||
values: [finalLimit],
|
||||
client: tx,
|
||||
});
|
||||
|
||||
if (!matched_res.success) {
|
||||
return {
|
||||
...matched_res,
|
||||
success: false,
|
||||
} as DBResponseObject<Schema>;
|
||||
}
|
||||
|
||||
const ids = (matched_res.payload || []).map((row) => row.id);
|
||||
|
||||
if (ids.length === 0) {
|
||||
return { success: true, payload: [] };
|
||||
}
|
||||
|
||||
const idPlaceholders = ids.map(() => "?").join(",");
|
||||
|
||||
const update_res = await dbHandler({
|
||||
query: `UPDATE ${tableSql} SET queue_status = 'processing', queue_attempts = queue_attempts + 1, queue_lease_expires_at = ${QUEUE_NOW_SQL} + ?, queue_lock_token = ?, updated_at = ${QUEUE_NOW_SQL} WHERE id IN (${idPlaceholders})`,
|
||||
values: [leaseMs, lock_token, ...ids],
|
||||
client: tx,
|
||||
});
|
||||
|
||||
if (!update_res.success) {
|
||||
throw new Error(update_res.error || "Queue claim failed");
|
||||
}
|
||||
|
||||
return await dbHandler<Schema>({
|
||||
query: `SELECT * FROM ${tableSql} WHERE id IN (${idPlaceholders}) ORDER BY queue_available_at, id`,
|
||||
values: ids,
|
||||
client: tx,
|
||||
});
|
||||
}),
|
||||
);
|
||||
} catch (error: any) {
|
||||
return {
|
||||
success: false,
|
||||
error: error.message,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
import dbHandler from "../db-handler";
|
||||
import type { BunMariaDBConfig, DBResponseObject } from "../../types";
|
||||
import { quoteIdentifier } from "../../utils/sql-identifier";
|
||||
import { QUEUE_NOW_SQL, requireQueueTable } from "./queue-utils";
|
||||
|
||||
type Params<Table extends string = string> = {
|
||||
table: Table;
|
||||
id: number | string;
|
||||
/** `queue_lock_token` from the claimed row */
|
||||
lock_token: string;
|
||||
/** New lease length, counted from now. Defaults to the table's `lease_ms` */
|
||||
lease_ms?: number;
|
||||
config?: BunMariaDBConfig;
|
||||
};
|
||||
|
||||
/**
|
||||
* Extend the lease on a claimed job, for jobs that run longer than the
|
||||
* lease given to `claim`.
|
||||
*/
|
||||
export default async function QueueExtend<Table extends string = string>({
|
||||
table,
|
||||
id,
|
||||
lock_token,
|
||||
lease_ms,
|
||||
config,
|
||||
}: Params<Table>): Promise<DBResponseObject> {
|
||||
try {
|
||||
const options = requireQueueTable(table);
|
||||
|
||||
const leaseMs = Math.max(
|
||||
0,
|
||||
Math.floor(Number(lease_ms ?? options.lease_ms) || 0),
|
||||
);
|
||||
|
||||
const res = await dbHandler({
|
||||
query: `UPDATE ${quoteIdentifier(table)} SET queue_lease_expires_at = ${QUEUE_NOW_SQL} + ?, updated_at = ${QUEUE_NOW_SQL} WHERE id = ? AND queue_status = 'processing' AND queue_lock_token = ?`,
|
||||
values: [leaseMs, id, lock_token],
|
||||
config,
|
||||
});
|
||||
|
||||
if (!res.success) {
|
||||
return { ...res, success: false };
|
||||
}
|
||||
|
||||
if (!res.db_res?.affectedRows) {
|
||||
return {
|
||||
...res,
|
||||
success: false,
|
||||
msg: "Job is not claimed with this lock token",
|
||||
};
|
||||
}
|
||||
|
||||
return await dbHandler({
|
||||
query: `SELECT * FROM ${quoteIdentifier(table)} WHERE id = ?`,
|
||||
values: [id],
|
||||
config,
|
||||
});
|
||||
} catch (error: any) {
|
||||
return {
|
||||
success: false,
|
||||
error: error.message,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
import dbHandler from "../db-handler";
|
||||
import type { BunMariaDBConfig, DBResponseObject } from "../../types";
|
||||
import { quoteIdentifier } from "../../utils/sql-identifier";
|
||||
import { QUEUE_NOW_SQL, requireQueueTable } from "./queue-utils";
|
||||
|
||||
type Params<Table extends string = string> = {
|
||||
table: Table;
|
||||
id: number | string;
|
||||
/** `queue_lock_token` from the claimed row */
|
||||
lock_token: string;
|
||||
error?: string;
|
||||
/** Wait this long before the job can be retried. Defaults to 0 */
|
||||
retry_delay_ms?: number;
|
||||
/** Mark the job `failed` instead of retrying once it reaches this many attempts. Defaults to the table's `max_attempts` */
|
||||
max_attempts?: number;
|
||||
config?: BunMariaDBConfig;
|
||||
};
|
||||
|
||||
/**
|
||||
* Release a claimed job after a failed attempt. It goes back to `pending`
|
||||
* (after `retry_delay_ms`) or to `failed` once `max_attempts` is reached.
|
||||
*/
|
||||
export default async function QueueFail<Table extends string = string>({
|
||||
table,
|
||||
id,
|
||||
lock_token,
|
||||
error,
|
||||
retry_delay_ms = 0,
|
||||
max_attempts,
|
||||
config,
|
||||
}: Params<Table>): Promise<DBResponseObject> {
|
||||
try {
|
||||
const options = requireQueueTable(table);
|
||||
|
||||
const tableSql = quoteIdentifier(table);
|
||||
const retryDelayMs = Math.max(0, Math.floor(Number(retry_delay_ms) || 0));
|
||||
const maxAttempts = Math.max(
|
||||
1,
|
||||
Math.floor(Number(max_attempts ?? options.max_attempts) || 1),
|
||||
);
|
||||
|
||||
const res = await dbHandler({
|
||||
query: `UPDATE ${tableSql} SET queue_status = IF(queue_attempts >= ?, 'failed', 'pending'), queue_available_at = ${QUEUE_NOW_SQL} + ?, queue_lease_expires_at = NULL, queue_lock_token = NULL, queue_last_error = ?, updated_at = ${QUEUE_NOW_SQL} WHERE id = ? AND queue_status = 'processing' AND queue_lock_token = ?`,
|
||||
values: [maxAttempts, retryDelayMs, error ?? null, id, lock_token],
|
||||
config,
|
||||
});
|
||||
|
||||
if (!res.success) {
|
||||
return { ...res, success: false };
|
||||
}
|
||||
|
||||
if (!res.db_res?.affectedRows) {
|
||||
return {
|
||||
...res,
|
||||
success: false,
|
||||
msg: "Job is not claimed with this lock token",
|
||||
};
|
||||
}
|
||||
|
||||
return await dbHandler({
|
||||
query: `SELECT * FROM ${quoteIdentifier(table)} WHERE id = ?`,
|
||||
values: [id],
|
||||
config,
|
||||
});
|
||||
} catch (error: any) {
|
||||
return {
|
||||
success: false,
|
||||
error: error.message,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
import type { BunMariaDBConfig, DBResponseObject } from "../../types";
|
||||
import DbInsert from "../mariadb/db-insert";
|
||||
|
||||
type Params<
|
||||
Schema extends { [k: string]: any } = { [k: string]: any },
|
||||
Table extends string = string,
|
||||
> = {
|
||||
table: Table;
|
||||
data: Schema[];
|
||||
/** Wait this long before the jobs can be claimed */
|
||||
delay_ms?: number;
|
||||
config?: BunMariaDBConfig;
|
||||
};
|
||||
|
||||
/**
|
||||
* Add jobs to a queue table.
|
||||
*/
|
||||
export default async function QueuePush<
|
||||
Schema extends { [k: string]: any } = { [k: string]: any },
|
||||
Table extends string = string,
|
||||
>({
|
||||
table,
|
||||
data,
|
||||
delay_ms,
|
||||
config,
|
||||
}: Params<Schema, Table>): Promise<DBResponseObject> {
|
||||
const jobs = data.map((d) => ({
|
||||
...(delay_ms ? { queue_available_at: Date.now() + delay_ms } : {}),
|
||||
...d,
|
||||
}));
|
||||
|
||||
return DbInsert({ table, data: jobs, config });
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
import _ from "lodash";
|
||||
import { readLiveSchema } from "../../functions/live-schema";
|
||||
import {
|
||||
QueueProtectedFields,
|
||||
type BUN_MARIADB_DatabaseSchemaType,
|
||||
type BUN_MARIADB_QueueTableOptions,
|
||||
} from "../../types";
|
||||
|
||||
/**
|
||||
* Current time in ms from the database clock. Leases are set and checked
|
||||
* with this so workers on hosts with drifting clocks agree on expiry.
|
||||
*/
|
||||
export const QUEUE_NOW_SQL = "CAST(UNIX_TIMESTAMP(NOW(3)) * 1000 AS SIGNED)";
|
||||
|
||||
/**
|
||||
* Extra WHERE condition for generic update/delete on queue tables: rows a
|
||||
* worker has claimed are only changed through the queue API.
|
||||
*/
|
||||
export const QUEUE_NOT_CLAIMED_SQL = "`queue_status` <> 'processing'";
|
||||
|
||||
export type ResolvedQueueTableOptions = Required<BUN_MARIADB_QueueTableOptions>;
|
||||
|
||||
const QUEUE_TABLE_DEFAULTS: ResolvedQueueTableOptions = {
|
||||
max_attempts: 5,
|
||||
lease_ms: 30_000,
|
||||
keep_completed: false,
|
||||
};
|
||||
|
||||
/**
|
||||
* A table's `queue_table` options with defaults filled in (looked up
|
||||
* directly or through its parent table), or `undefined` if it is not a
|
||||
* queue table.
|
||||
*/
|
||||
export function grabQueueTableOptions(
|
||||
tableName: string,
|
||||
): ResolvedQueueTableOptions | undefined {
|
||||
const dbSchema: BUN_MARIADB_DatabaseSchemaType | undefined =
|
||||
global.DB_SCHEMA || readLiveSchema();
|
||||
|
||||
const table = dbSchema?.tables?.find((t) => t.tableName === tableName);
|
||||
|
||||
if (!table) return undefined;
|
||||
|
||||
if (table.queue_table === undefined) {
|
||||
return table.parentTableName
|
||||
? grabQueueTableOptions(table.parentTableName)
|
||||
: undefined;
|
||||
}
|
||||
|
||||
if (!table.queue_table) return undefined;
|
||||
|
||||
return {
|
||||
...QUEUE_TABLE_DEFAULTS,
|
||||
...(typeof table.queue_table === "object"
|
||||
? _.omitBy(table.queue_table, _.isUndefined)
|
||||
: {}),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a table is declared with `queue_table` (directly or through its
|
||||
* parent table).
|
||||
*/
|
||||
export function isQueueTable(tableName: string): boolean {
|
||||
return Boolean(grabQueueTableOptions(tableName));
|
||||
}
|
||||
|
||||
/**
|
||||
* Queue options for a table the queue API was called on. Throws if the table
|
||||
* is not a queue table.
|
||||
*/
|
||||
export function requireQueueTable(tableName: string): ResolvedQueueTableOptions {
|
||||
const options = grabQueueTableOptions(tableName);
|
||||
|
||||
if (!options) {
|
||||
throw new Error(`\`${tableName}\` is not a queue table`);
|
||||
}
|
||||
|
||||
return options;
|
||||
}
|
||||
|
||||
/**
|
||||
* Throw if `data` sets queue state the queue API owns.
|
||||
*/
|
||||
export function assertNoProtectedQueueFields(
|
||||
table: string,
|
||||
data: { [k: string]: any }[],
|
||||
): void {
|
||||
const protectedFields = QueueProtectedFields.filter((field) =>
|
||||
data.some((d) => d && d[field] !== undefined),
|
||||
);
|
||||
|
||||
if (protectedFields.length > 0) {
|
||||
throw new Error(
|
||||
`\`${table}\` is a queue table: ${protectedFields.join(", ")} can only be changed through BunMariaDB.queue`,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -96,6 +96,25 @@ export interface BUN_MARIADB_TableSchemaType {
|
||||
*/
|
||||
childTableDbId?: string | number;
|
||||
collation?: (typeof MariaDBCollations)[number];
|
||||
/**
|
||||
* Make this table a job queue. Adds the `QueueFields` columns and
|
||||
* `QueueIndexes`, and routes state changes through `BunMariaDB.queue`.
|
||||
* Pass an options object to set the queue's defaults.
|
||||
*/
|
||||
queue_table?: boolean | BUN_MARIADB_QueueTableOptions;
|
||||
}
|
||||
|
||||
/**
|
||||
* Defaults for a queue table. Per-call arguments to `BunMariaDB.queue`
|
||||
* override them.
|
||||
*/
|
||||
export interface BUN_MARIADB_QueueTableOptions {
|
||||
/** Attempts before a job is marked `failed`. Defaults to 5 */
|
||||
max_attempts?: number;
|
||||
/** How long a claim lasts before another worker may take the job. Defaults to 30s */
|
||||
lease_ms?: number;
|
||||
/** `ack` marks jobs `done` and keeps them instead of deleting them */
|
||||
keep_completed?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1639,6 +1658,96 @@ export const DefaultFields: BUN_MARIADB_FieldSchemaType[] = [
|
||||
},
|
||||
];
|
||||
|
||||
/**
|
||||
* Lifecycle of a queue row. Only `pending` jobs are claimed.
|
||||
* - `done`: acknowledged, kept because of `keep_completed` (otherwise deleted)
|
||||
* - `failed`: used up its attempts (dead letter)
|
||||
* - `cancelled`: removed from the queue with `queue.cancel`
|
||||
*/
|
||||
export const QueueStatuses = [
|
||||
"pending",
|
||||
"processing",
|
||||
"done",
|
||||
"failed",
|
||||
"cancelled",
|
||||
] as const;
|
||||
|
||||
export type QueueStatus = (typeof QueueStatuses)[number];
|
||||
|
||||
/**
|
||||
* Fields added to every table with `queue_table: true`.
|
||||
*/
|
||||
export const QueueFields: BUN_MARIADB_FieldSchemaType[] = [
|
||||
{
|
||||
fieldName: "queue_status",
|
||||
dataType: "ENUM",
|
||||
options: [...QueueStatuses],
|
||||
notNullValue: true,
|
||||
defaultValue: "pending",
|
||||
fieldDescription: "Queue state of the job.",
|
||||
},
|
||||
{
|
||||
fieldName: "queue_available_at",
|
||||
dataType: "BIGINT",
|
||||
notNullValue: true,
|
||||
defaultValue: 0,
|
||||
fieldDescription:
|
||||
"The job cannot be claimed before this time. (Unix Timestamp, ms)",
|
||||
},
|
||||
{
|
||||
fieldName: "queue_attempts",
|
||||
dataType: "INT",
|
||||
notNullValue: true,
|
||||
defaultValue: 0,
|
||||
fieldDescription: "How many times the job has been claimed.",
|
||||
},
|
||||
{
|
||||
fieldName: "queue_lease_expires_at",
|
||||
dataType: "BIGINT",
|
||||
fieldDescription:
|
||||
"When the current claim expires and the job can be reclaimed. (Unix Timestamp, ms)",
|
||||
},
|
||||
{
|
||||
fieldName: "queue_lock_token",
|
||||
dataType: "VARCHAR",
|
||||
dataLength: 64,
|
||||
fieldDescription:
|
||||
"Token of the current claim. Required to ack, fail or extend the job.",
|
||||
},
|
||||
{
|
||||
fieldName: "queue_last_error",
|
||||
dataType: "TEXT",
|
||||
fieldDescription: "Error message from the last failed attempt.",
|
||||
},
|
||||
];
|
||||
|
||||
/**
|
||||
* Queue fields only the queue API may write. `queue_available_at` can be set
|
||||
* on insert and update.
|
||||
*/
|
||||
export const QueueProtectedFields = [
|
||||
"queue_status",
|
||||
"queue_attempts",
|
||||
"queue_lease_expires_at",
|
||||
"queue_lock_token",
|
||||
"queue_last_error",
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* Indexes added to queue tables. Claims scan `idx_queue_claim` in order, so
|
||||
* `LIMIT` stops the scan early and only the claimed rows get locked.
|
||||
*/
|
||||
export const QueueIndexes: BUN_MARIADB_IndexSchemaType[] = [
|
||||
{
|
||||
indexName: "idx_queue_claim",
|
||||
indexTableFields: ["queue_status", "queue_available_at"],
|
||||
},
|
||||
{
|
||||
indexName: "idx_queue_lease",
|
||||
indexTableFields: ["queue_status", "queue_lease_expires_at"],
|
||||
},
|
||||
];
|
||||
|
||||
export type BunMariaDBQueryFieldValues<
|
||||
F extends string = string,
|
||||
T extends string = string,
|
||||
|
||||
@@ -1,10 +1,18 @@
|
||||
import _ from "lodash";
|
||||
import { DefaultFields, type BUN_MARIADB_DatabaseSchemaType } from "../types";
|
||||
import {
|
||||
DefaultFields,
|
||||
QueueFields,
|
||||
QueueIndexes,
|
||||
type BUN_MARIADB_DatabaseSchemaType,
|
||||
} from "../types";
|
||||
|
||||
type Params = {
|
||||
dbSchema: BUN_MARIADB_DatabaseSchemaType;
|
||||
};
|
||||
|
||||
const queueFieldNames = new Set(QueueFields.map((f) => f.fieldName));
|
||||
const queueIndexNames = new Set(QueueIndexes.map((i) => i.indexName));
|
||||
|
||||
export default function ({ dbSchema }: Params): BUN_MARIADB_DatabaseSchemaType {
|
||||
const finaldbSchema = _.cloneDeep(dbSchema);
|
||||
finaldbSchema.tables = finaldbSchema.tables.map((t) => {
|
||||
@@ -13,6 +21,30 @@ export default function ({ dbSchema }: Params): BUN_MARIADB_DatabaseSchemaType {
|
||||
(f) => !f.fieldName?.match(/^(id|created_at|updated_at)$/),
|
||||
);
|
||||
newTable.fields.unshift(...DefaultFields);
|
||||
|
||||
if (newTable.queue_table) {
|
||||
const reserved = [
|
||||
...newTable.fields
|
||||
.map((f) => f.fieldName)
|
||||
.filter((name) => queueFieldNames.has(name)),
|
||||
...(newTable.indexes || [])
|
||||
.map((i) => i.indexName)
|
||||
.filter((name) => queueIndexNames.has(name)),
|
||||
];
|
||||
|
||||
if (reserved.length > 0) {
|
||||
throw new Error(
|
||||
`\`${newTable.tableName}\` is a queue table: ${reserved.join(", ")} ${reserved.length > 1 ? "are" : "is"} managed by bun-mariadb and can't be declared in the schema`,
|
||||
);
|
||||
}
|
||||
|
||||
newTable.fields.push(..._.cloneDeep(QueueFields));
|
||||
newTable.indexes = [
|
||||
...(newTable.indexes || []),
|
||||
..._.cloneDeep(QueueIndexes),
|
||||
];
|
||||
}
|
||||
|
||||
return newTable;
|
||||
});
|
||||
|
||||
|
||||
+2
-1
@@ -36,7 +36,8 @@ describe.skipIf(!TEST_DB)("MariaDB integration", () => {
|
||||
appendDefaults = (await import("../src/utils/append-default-fields-to-db-schema")).default;
|
||||
schemaLib = await import("../src/lib/schema/run-schema-query");
|
||||
|
||||
global.CONFIG = { db_name: TEST_DB! };
|
||||
// The test container uses a self-signed certificate
|
||||
global.CONFIG = { db_name: TEST_DB!, ssl_reject_unauthorized: false };
|
||||
global.DB_SCHEMA = schema;
|
||||
setMariaDBClient({ config: global.CONFIG });
|
||||
|
||||
|
||||
@@ -0,0 +1,258 @@
|
||||
/**
|
||||
* Queue table integration tests against a real MariaDB.
|
||||
*
|
||||
* Skipped unless BUN_MARIADB_TEST_DB is set (see tests/db.test.ts).
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from "bun:test";
|
||||
import type { BUN_MARIADB_DatabaseSchemaType } from "../src/types";
|
||||
|
||||
const TEST_DB = process.env.BUN_MARIADB_TEST_DB;
|
||||
const TABLE = "bmt_jobs";
|
||||
const KEPT_TABLE = "bmt_jobs_kept";
|
||||
|
||||
const schema: BUN_MARIADB_DatabaseSchemaType = {
|
||||
tables: [
|
||||
{
|
||||
tableName: TABLE,
|
||||
queue_table: true,
|
||||
fields: [{ fieldName: "name", dataType: "VARCHAR" }],
|
||||
},
|
||||
{
|
||||
tableName: KEPT_TABLE,
|
||||
queue_table: { keep_completed: true, max_attempts: 2 },
|
||||
fields: [{ fieldName: "name", dataType: "VARCHAR" }],
|
||||
},
|
||||
],
|
||||
} as BUN_MARIADB_DatabaseSchemaType;
|
||||
|
||||
test("queue tables refuse user-declared queue fields and indexes", async () => {
|
||||
const appendDefaults = (await import("../src/utils/append-default-fields-to-db-schema")).default;
|
||||
|
||||
expect(() =>
|
||||
appendDefaults({
|
||||
dbSchema: {
|
||||
tables: [
|
||||
{
|
||||
tableName: "q",
|
||||
queue_table: true,
|
||||
fields: [{ fieldName: "queue_status", dataType: "VARCHAR" }],
|
||||
indexes: [{ indexName: "idx_queue_claim", indexTableFields: ["id"] }],
|
||||
},
|
||||
],
|
||||
},
|
||||
}),
|
||||
).toThrow("queue_status, idx_queue_claim are managed by bun-mariadb");
|
||||
});
|
||||
|
||||
describe.skipIf(!TEST_DB)("Queue tables", () => {
|
||||
let DB: typeof import("../src/index").default;
|
||||
|
||||
beforeAll(async () => {
|
||||
const setMariaDBClient = (await import("../src/functions/set-mariadb-client")).default;
|
||||
const createDBSchema = (await import("../src/lib/schema/create-db-schema")).default;
|
||||
const appendDefaults = (await import("../src/utils/append-default-fields-to-db-schema")).default;
|
||||
|
||||
/**
|
||||
* The test container uses a self-signed certificate. `db_config` gives
|
||||
* this file its own client, since other test files close theirs. The
|
||||
* client is created before `global.CONFIG` changes, so the previous
|
||||
* file's closed global client isn't reused.
|
||||
*/
|
||||
const config = {
|
||||
db_name: TEST_DB!,
|
||||
ssl_reject_unauthorized: false,
|
||||
db_config: { max: 10 },
|
||||
};
|
||||
setMariaDBClient({ config });
|
||||
global.CONFIG = config;
|
||||
global.DB_SCHEMA = schema;
|
||||
|
||||
DB = (await import("../src/index")).default;
|
||||
|
||||
await createDBSchema({
|
||||
db_schema: appendDefaults({ dbSchema: schema }),
|
||||
config: global.CONFIG,
|
||||
});
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await DB.sql({ sql: `DELETE FROM ${TABLE}` });
|
||||
await DB.sql({ sql: `DELETE FROM ${KEPT_TABLE}` });
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await DB?.sql({ sql: `DROP TABLE IF EXISTS ${TABLE}, ${KEPT_TABLE}` });
|
||||
await DB?.sql({
|
||||
sql: `DELETE FROM __db_schema_manager__ WHERE table_name IN ('${TABLE}', '${KEPT_TABLE}')`,
|
||||
});
|
||||
await global.MARIADB_CLIENT?.close();
|
||||
});
|
||||
|
||||
test("schema adds queue columns and indexes", async () => {
|
||||
const cols = await DB.sql<any>({
|
||||
sql: "SELECT COLUMN_NAME FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = ?",
|
||||
values: [TABLE],
|
||||
});
|
||||
const names = cols.payload?.map((c) => c.COLUMN_NAME);
|
||||
expect(names).toContain("queue_status");
|
||||
expect(names).toContain("queue_lock_token");
|
||||
|
||||
const idx = await DB.sql<any>({
|
||||
sql: "SELECT COLUMN_NAME FROM information_schema.STATISTICS WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = ? AND INDEX_NAME = 'idx_queue_claim' ORDER BY SEQ_IN_INDEX",
|
||||
values: [TABLE],
|
||||
});
|
||||
expect(idx.payload?.map((c) => c.COLUMN_NAME)).toEqual(["queue_status", "queue_available_at"]);
|
||||
});
|
||||
|
||||
test("insert and update refuse protected queue fields", async () => {
|
||||
const ins = await DB.insert({ table: TABLE, data: [{ name: "x", queue_status: "processing" }] });
|
||||
expect(ins.success).toBe(false);
|
||||
expect(ins.error).toContain("queue_status");
|
||||
|
||||
await DB.queue.push({ table: TABLE, data: [{ name: "a" }] });
|
||||
const upd = await DB.update({ table: TABLE, query: { query: { name: { value: "a" } } }, data: { queue_attempts: 0 } });
|
||||
expect(upd.success).toBe(false);
|
||||
});
|
||||
|
||||
test("claim, then ack with the lock token deletes the job", async () => {
|
||||
await DB.queue.push({ table: TABLE, data: [{ name: "a" }] });
|
||||
|
||||
const claim = await DB.queue.claim<any>({ table: TABLE });
|
||||
expect(claim.success).toBe(true);
|
||||
const job = claim.payload![0];
|
||||
expect(job.queue_status).toBe("processing");
|
||||
expect(job.queue_attempts).toBe(1);
|
||||
expect(job.queue_lock_token).toBeTruthy();
|
||||
|
||||
expect((await DB.queue.claim({ table: TABLE })).payload).toEqual([]);
|
||||
|
||||
const bad = await DB.queue.ack({ table: TABLE, id: job.id, lock_token: "nope" });
|
||||
expect(bad.success).toBe(false);
|
||||
|
||||
const ack = await DB.queue.ack({ table: TABLE, id: job.id, lock_token: job.queue_lock_token });
|
||||
expect(ack.success).toBe(true);
|
||||
expect((await DB.select({ table: TABLE })).payload).toEqual([]);
|
||||
});
|
||||
|
||||
test("concurrent claims never return the same job twice", async () => {
|
||||
await DB.queue.push({ table: TABLE, data: Array.from({ length: 20 }, (_, i) => ({ name: `j${i}` })) });
|
||||
|
||||
const claims = await Promise.all(
|
||||
Array.from({ length: 8 }, () => DB.queue.claim<any>({ table: TABLE, limit: 3 })),
|
||||
);
|
||||
|
||||
expect(claims.every((c) => c.success)).toBe(true);
|
||||
const ids = claims.flatMap((c) => c.payload!.map((j) => j.id));
|
||||
expect(new Set(ids).size).toBe(ids.length);
|
||||
expect(ids.length).toBe(20);
|
||||
});
|
||||
|
||||
test("update and delete skip claimed jobs", async () => {
|
||||
await DB.queue.push({ table: TABLE, data: [{ name: "a" }] });
|
||||
const job = (await DB.queue.claim<any>({ table: TABLE })).payload![0];
|
||||
|
||||
const upd = await DB.update({ table: TABLE, targetId: job.id, data: { name: "b" } });
|
||||
expect(upd.success).toBe(false);
|
||||
|
||||
await DB.delete({ table: TABLE, targetId: job.id });
|
||||
expect((await DB.select({ table: TABLE })).payload?.length).toBe(1);
|
||||
});
|
||||
|
||||
test("fail retries, then dead-letters after max_attempts", async () => {
|
||||
await DB.queue.push({ table: TABLE, data: [{ name: "a" }] });
|
||||
|
||||
let job = (await DB.queue.claim<any>({ table: TABLE })).payload![0];
|
||||
const retry = await DB.queue.fail({ table: TABLE, id: job.id, lock_token: job.queue_lock_token, error: "boom", max_attempts: 2 });
|
||||
expect(retry.success).toBe(true);
|
||||
expect(retry.single_res?.queue_status).toBe("pending");
|
||||
expect(retry.single_res?.queue_last_error).toBe("boom");
|
||||
|
||||
job = (await DB.queue.claim<any>({ table: TABLE })).payload![0];
|
||||
expect(job.queue_attempts).toBe(2);
|
||||
const dead = await DB.queue.fail({ table: TABLE, id: job.id, lock_token: job.queue_lock_token, error: "boom", max_attempts: 2 });
|
||||
expect(dead.single_res?.queue_status).toBe("failed");
|
||||
|
||||
expect((await DB.queue.claim({ table: TABLE })).payload).toEqual([]);
|
||||
});
|
||||
|
||||
test("an expired lease is reclaimed and the old token stops working", async () => {
|
||||
await DB.queue.push({ table: TABLE, data: [{ name: "a" }] });
|
||||
|
||||
const first = (await DB.queue.claim<any>({ table: TABLE, lease_ms: 0 })).payload![0];
|
||||
const second = (await DB.queue.claim<any>({ table: TABLE })).payload![0];
|
||||
expect(second.id).toBe(first.id);
|
||||
expect(second.queue_attempts).toBe(2);
|
||||
|
||||
expect((await DB.queue.ack({ table: TABLE, id: first.id, lock_token: first.queue_lock_token })).success).toBe(false);
|
||||
expect((await DB.queue.extend({ table: TABLE, id: second.id, lock_token: second.queue_lock_token })).success).toBe(true);
|
||||
});
|
||||
|
||||
test("an expired lease on the last attempt is dead-lettered", async () => {
|
||||
await DB.queue.push({ table: TABLE, data: [{ name: "a" }] });
|
||||
await DB.queue.claim({ table: TABLE, lease_ms: 0 });
|
||||
|
||||
expect((await DB.queue.claim({ table: TABLE, max_attempts: 1 })).payload).toEqual([]);
|
||||
const row = (await DB.select<any>({ table: TABLE })).single_res;
|
||||
expect(row?.queue_status).toBe("failed");
|
||||
});
|
||||
|
||||
test("claims in push order and holds back delayed jobs", async () => {
|
||||
await DB.queue.push({ table: TABLE, data: [{ name: "later" }], delay_ms: 60_000 });
|
||||
await DB.queue.push({ table: TABLE, data: [{ name: "first" }, { name: "second" }] });
|
||||
|
||||
const claim = await DB.queue.claim<any>({ table: TABLE, limit: 5 });
|
||||
expect(claim.payload?.map((j) => j.name)).toEqual(["first", "second"]);
|
||||
});
|
||||
|
||||
test("keep_completed: ack marks the job done and keeps it", async () => {
|
||||
await DB.queue.push({ table: KEPT_TABLE, data: [{ name: "a" }] });
|
||||
const job = (await DB.queue.claim<any>({ table: KEPT_TABLE })).payload![0];
|
||||
|
||||
const ack = await DB.queue.ack({ table: KEPT_TABLE, id: job.id, lock_token: job.queue_lock_token });
|
||||
expect(ack.success).toBe(true);
|
||||
expect(ack.single_res?.queue_status).toBe("done");
|
||||
expect(ack.single_res?.queue_lock_token).toBeNull();
|
||||
|
||||
expect((await DB.queue.claim({ table: KEPT_TABLE })).payload).toEqual([]);
|
||||
expect((await DB.queue.ack({ table: KEPT_TABLE, id: job.id, lock_token: job.queue_lock_token })).success).toBe(false);
|
||||
});
|
||||
|
||||
test("table max_attempts is the default for fail", async () => {
|
||||
await DB.queue.push({ table: KEPT_TABLE, data: [{ name: "a" }] });
|
||||
|
||||
for (const expected of ["pending", "failed"]) {
|
||||
const job = (await DB.queue.claim<any>({ table: KEPT_TABLE })).payload![0];
|
||||
const res = await DB.queue.fail({ table: KEPT_TABLE, id: job.id, lock_token: job.queue_lock_token });
|
||||
expect(res.single_res?.queue_status).toBe(expected);
|
||||
}
|
||||
});
|
||||
|
||||
test("cancel takes pending jobs out of the queue, but not claimed ones", async () => {
|
||||
await DB.queue.push({ table: TABLE, data: [{ name: "a" }, { name: "b" }] });
|
||||
const rows = (await DB.select<any>({ table: TABLE, query: { order: { field: "id", strategy: "ASC" } } })).payload!;
|
||||
|
||||
const cancel = await DB.queue.cancel({ table: TABLE, id: rows[0].id });
|
||||
expect(cancel.single_res?.queue_status).toBe("cancelled");
|
||||
|
||||
const claimed = (await DB.queue.claim<any>({ table: TABLE, limit: 5 })).payload!;
|
||||
expect(claimed.map((j) => j.name)).toEqual(["b"]);
|
||||
|
||||
expect((await DB.queue.cancel({ table: TABLE, id: claimed[0].id })).success).toBe(false);
|
||||
});
|
||||
|
||||
test("select filters queue rows by status", async () => {
|
||||
await DB.queue.push({ table: TABLE, data: [{ name: "a" }, { name: "b" }, { name: "c" }] });
|
||||
const job = (await DB.queue.claim<any>({ table: TABLE })).payload![0];
|
||||
await DB.queue.fail({ table: TABLE, id: job.id, lock_token: job.queue_lock_token, max_attempts: 1 });
|
||||
|
||||
const failed = await DB.select<any>({ table: TABLE, query: { query: { queue_status: { value: "failed" } } } });
|
||||
expect(failed.payload?.map((r) => r.name)).toEqual(["a"]);
|
||||
|
||||
const open = await DB.select<any>({
|
||||
table: TABLE,
|
||||
count: true,
|
||||
query: { query: { queue_status: { value: ["pending", "processing"], equality: "IN" } } },
|
||||
});
|
||||
expect(open.count).toBe(2);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user