Fix data archives and restores failing on their own lock

startDataBackupRun already holds the backup run lock, but the archive
(createDataBackup) and restoreDataBackup took it again, so every run
failed a few seconds in with "Another backup or restore is already
running". The writer and the restore now take no lock of their own.

The run's notice (archiving, restoring, or why it failed) moves to its
own line under the actions row so the buttons don't shift, and archiving
reads in gray rather than the warning colour.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

AGENTS.md

@@ -230,7 +230,7 @@ Live routes under `/admin` (see `AdminSections`): `/admin/bots`, `/admin/workspa
 - `/admin/logs` (live) - Activity, the audit log: super admins only (others are redirected; the popover's "Activity" item shows to super admins only). Rendered in `SettingsShell` with `wide`
 - `/admin/help` (live) - familiar problems and their fixes, one `SettingsGroup` per topic (`(data)/help-topics.ts`, markdown rendered with the chat's `MessageMarkdownComponents`, the topic id as the group's anchor). First topic: signing in to Google and other strict sites with Cookie-Editor over the Desktop view (`HelpSignInTopicId` in `src/dict/help-dict.ts`; `ask_user_to_take_over` points bots to it)
 - Routines, Files, Usage, Activity and Help are the `RailPages` (`src/dict/rail-pages-dict.ts`), opened from the rail's user popover (which lists them, then Settings and Sign out). On them the rail shows its own aside (`(partials)/rail-pages-list.tsx`): a back link to the bots, then those pages, like settings. A page carries `super_admin_only` or a `can_view` rule (Usage is `canManageCrew`), and `canSeeRailItem` (`src/utils/rbac/can-see-rail-item.ts`, tested) decides both the popover and the aside, so a page is never offered in one and hidden in the other
-- `/admin/settings/backups` - super admin only (listed in `SettingsPages`), two groups. Database snapshots: create, list, restore, download and delete in `db/.backups` (`bun-sqlite backup`); "Restore from file" beside "Create backup" opens `upload-restore-modal.tsx` (upload a SQLite database file, which is saved to `.backups` then restored with `bun-sqlite import <file>`, since `restore` is interactive). Whole-data archives: `write-data-backup-archive.ts` tars all of `DATA_DIR` (database, media, computer workspaces, bot and workspace files, MCP installs; an older computer's moved-aside `computers/<id>-home-<ts>` is left out) into `BACKUP_ARCHIVE_DIR` as `cobble-data-<epoch-ms>.tar.gz` (via `tar -czf` into a `.part` name, renamed on success so a cut-off run is never listed, with a `--transform` putting every member under one root folder named after the archive), keeping the newest `AppData.DataBackupsKept` and never touching `db/.backups`; its own "Restore from file" uploads a tar.gz (`upload-restore-data.ts`, gzip header checked first). `restore-data-backup.ts` extracts an archive to a temp dir, requires exactly one root folder holding a `db` folder (`findArchiveRoot`, since an upload is saved under a fresh timestamp and its inner name no longer matches the file name), takes a safety archive before touching anything (`write-data-backup-archive.ts`, not the locked `create-data-backup.ts`), empties `DATA_DIR` (the snapshots included) and copies the archive's in, then `restart-app.ts` restarts the process through the docker socket after the response lands (`process.exit`, the container's restart policy brings it back), because the open SQLite handle points at the replaced file. Row actions are icon-only (`RowIconButton`, `src/components/general/row-icon-button.tsx`): restore, download, delete. `BACKUP_ARCHIVE_DIR` is a `ProcessOnlyEnvKeys` start-up path (the installer mounts `$COBBLE_HOME/backups-archives` there; outside docker it falls back to `~/.cobble/backups`). Every backup and restore, of either kind, runs one at a time via `with-backup-run-lock.ts`. A whole-data archive or restore takes far longer than a request may stay silent (Bun's default `idleTimeout` is 10s, a proxy cuts at about 100s), so its route only starts it (`startDataBackupRun`, `data-backup-run.ts`, which refuses while anything holds the lock) and answers with the run; the page follows it (`use-data-backup-run`, `DataBackupRunContext`) by polling `api/admin/backups/data-status`, reloading once an archive is done, and after a restore once the status comes from a new process (`boot_id`). The upload is still saved within its request. Database snapshots stay synchronous. Helpers in `src/functions/backend/backups/`, API routes in `api/admin/backups/`
+- `/admin/settings/backups` - super admin only (listed in `SettingsPages`), two groups. Database snapshots: create, list, restore, download and delete in `db/.backups` (`bun-sqlite backup`); "Restore from file" beside "Create backup" opens `upload-restore-modal.tsx` (upload a SQLite database file, which is saved to `.backups` then restored with `bun-sqlite import <file>`, since `restore` is interactive). Whole-data archives: `write-data-backup-archive.ts` tars all of `DATA_DIR` (database, media, computer workspaces, bot and workspace files, MCP installs; an older computer's moved-aside `computers/<id>-home-<ts>` is left out) into `BACKUP_ARCHIVE_DIR` as `cobble-data-<epoch-ms>.tar.gz` (via `tar -czf` into a `.part` name, renamed on success so a cut-off run is never listed, with a `--transform` putting every member under one root folder named after the archive), keeping the newest `AppData.DataBackupsKept` and never touching `db/.backups`; its own "Restore from file" uploads a tar.gz (`upload-restore-data.ts`, gzip header checked first). `restore-data-backup.ts` extracts an archive to a temp dir, requires exactly one root folder holding a `db` folder (`findArchiveRoot`, since an upload is saved under a fresh timestamp and its inner name no longer matches the file name), takes a safety archive before touching anything (`write-data-backup-archive.ts`), empties `DATA_DIR` (the snapshots included) and copies the archive's in, then `restart-app.ts` restarts the process through the docker socket after the response lands (`process.exit`, the container's restart policy brings it back), because the open SQLite handle points at the replaced file. Row actions are icon-only (`RowIconButton`, `src/components/general/row-icon-button.tsx`): restore, download, delete. `BACKUP_ARCHIVE_DIR` is a `ProcessOnlyEnvKeys` start-up path (the installer mounts `$COBBLE_HOME/backups-archives` there; outside docker it falls back to `~/.cobble/backups`). Every backup and restore, of either kind, runs one at a time via `with-backup-run-lock.ts`. A whole-data archive or restore takes far longer than a request may stay silent (Bun's default `idleTimeout` is 10s, a proxy cuts at about 100s), so its route only starts it (`startDataBackupRun`, `data-backup-run.ts`, which refuses while anything holds the lock and holds it for the run, so the writer and the restore take no lock of their own) and answers with the run; the page follows it (`use-data-backup-run`, `DataBackupRunContext`) by polling `api/admin/backups/data-status`, reloading once an archive is done, and after a restore once the status comes from a new process (`boot_id`). The upload is still saved within its request. Database snapshots stay synchronous. Helpers in `src/functions/backend/backups/`, API routes in `api/admin/backups/`
 - `/admin/settings/database` - super admin only (listed in `SettingsPages`), a read-only browser of the install's SQLite database, no writes anywhere in it. `grabDatabaseTables` reads the catalog once (every table and its columns, SQLite's own `sqlite_*` tables and sqlite-vec's `bot_file_vectors_*` shadow tables left out) and adds each table's `COUNT(*)` and its size from `dbstat` (index pages folded back onto their table). `grabTableRecords` then returns one page (`AppData.DatabaseBrowserPageSize`, 50) of the open table's records newest key first, every value already cut in SQL at `AppData.DatabaseBrowserCellMaxChars` (400) so a table of long texts never ships a megabyte of JSON, and a `truncated` flag per cell; the search matches over the concatenation of every column's text (`buildSearchClause`: every word, or the whole phrase), which is what also works on a virtual table, where `MATCH` and `LIKE` over a vector blob do not. A record's key is its single primary key column, else its rowid, else nothing (a virtual table with no primary key reads as plain cells); clicking a cell's row opens `grabTableRecord` for that key, which returns the record whole up to `AppData.DatabaseBrowserValueMaxChars` (20000), the answer to a cell cut at three lines. `clampPage` puts a page past the end on the last one. The page's state (table, search, mode, page) lives in the address through `databaseHref`, so a table or a search can be linked to; the open table's first page comes with the page props and anything else loads through the API. Helpers in `src/functions/backend/database/` (`build-table-catalog-sql`, `build-table-sizes-sql`, `build-records-sql`, `build-search-clause`, `resolve-table-key`, `build-record-row`, `quote-sql-identifier`), API `api/admin/database/records` and `record` (GET only, `grabSuperAdminUser`), zod in `src/schemas/database-schema.ts`, page under `src/pages/admin/settings/database/`
 - `/admin/settings` (live) - redirects to `SettingsDefaultPath` (`/admin/settings/general`). Sub pages: `general/` (theme: System/Light/Dark per device via `useThemePreference`, stored in `localStorage[AppData.ThemeStorageKey]`, the same key `ThemeInitScript` and twui's `ColorSchemeSelector` read; applies instantly, no save bar; for super admins the prompt appendix (`prompt-section.tsx`, `system-prompt-append-form.tsx`) saving `api/admin/settings/system-prompt-append` into the `bot_system_prompt_append` app setting, and the run limits (`run-limits-section.tsx`, `run-limits-form.tsx`) saving `api/admin/settings/run-limits` into the `harness_max_tool_steps` and `bot_reply_max_tool_steps` app settings (`grabRunLimitsOverview`, `run_limits` page prop, 0 clearing a limit back to its code default); and, for `canManageCrew`, the bot file quota: `storage-section.tsx`, `storage-quota-form.tsx` saving `api/admin/settings/storage-quota` into `app_settings`; and, for everyone, the running version: `version-section.tsx`, `version-line.tsx` showing `pageProps.running_version` from `runningVersion` (`src/utils/running-version.ts`: `COBBLE_VERSION`, the release tag the image was built with, else `package.json`'s version, in development too, with `(development)` after it and no such suffix in production), `profile/` (current user profile), `notifications/` (this device's Web Push switch, a test send, and every device the user subscribed), `change-password/` (set/change own password) `models/` (super admin only, `super_admin_only` in the dict hides it from the rail: model providers and which model each harness uses) and `environment/` (super admin only: app settings and the envs bots and computers use, per scope), each wrapped in `SettingsShell` (`(partials)/settings-shell.tsx`: 64px pane header like the conversation header, then one centered column that opens with the breadcrumb trail, `src/components/general/admin-breadcrumbs.tsx`: twui's `Breadcrumbs` reading `Home`, the sections and the page off `pageProps.url.pathname`, the last crumb named by the page's own `title` so a record reads `Scout` rather than `3` and `/admin/logs` reads `Activity`; `hasBreadcrumbTrail` drops it only on the crew of bots, which is home, and nothing outside `/admin` gets one). Content is built from `SettingsGroup` blocks (label + description left, controls right, hairline-separated, no cards) and ends with the sticky `SettingsSaveBar` (Save enabled only when dirty, Discard, one-line status). Saves don't reload the page. Keep save notices in component state: `useFormInit` clears `status` whenever the form changes. `bots/` (the crew: chat, edit with `BotForm`, models, delete a worker via `api/admin/bots/delete-bot`), `workspaces/` (open, create through the Add flow, delete), `computers/` (state and resources, live; super admins create, edit with `AddComputerForm`, retry, delete), `mcps/`, `tools/` and `skills/` (see MCPs, custom tools and skills: super admins add, edit, refresh or try, delete; admins only set Access, `extension-access-modal.tsx`; add, edit, access and try open in modals (`AddModalShell`). A skill has no status: nothing connects or runs, so `skill-row.tsx` puts its slug, who gets it (`extension-access-label.ts`) and its description where a server's or tool's status line goes), each MCP server or tool row with a status line under it (`extension-status-line.tsx`: connected or not, last run worked or failed, when, and why), those five for `canManageCrew` only (`grab-crew-settings-server-props.ts`, `can_view` in the dict) and `users/` (for `canListUsers`: super admins and admins; the dict's `can_view` gates it in the rail; add with a password (`canCreateUser`), edit (`canUpdateUser`), both with the one `UserForm` (`admin/settings/users/(partials)/user-form/`, `useFormInit`, photo included, saved in one request), revoke (adds `revoked`, `canRevokeUser`), delete (`canRevokeUser`'s rules, never yourself), pick their user types from `UserTypes` (`user-types-select.tsx`: only the ones `filterAssignableUserTypes` allows, never `UneditableUserTypes`; at least one, since login needs a type; `saveUser` checks each grant and removal with `userTypeAssignment`; the super admin's types are never shown or changed there, and they can't be revoked; the password is entered twice, `SaveUserRequestSchema` checks they match), delete with everything theirs; `api/admin/users/save-user`, `delete-user`). The lists share `SettingsList`, `SettingsListRow`, `SettingsRowAction` and `use-settings-list`. Sub pages are declared once in `src/dict/settings-pages-dict.ts`; on every `/admin/settings/*` page the rail swaps the crew for a back link plus that list. Add a sub page by adding its folder and a dict entry
 

src/functions/backend/backups/create-data-backup.ts

@@ -1,21 +0,0 @@
-import withBackupRunLock from "./with-backup-run-lock";
-import writeDataBackupArchive from "./write-data-backup-archive";
-
-/**
- * Archives the whole `DATA_DIR` (database, media, computer homes, bot and
- * workspace files, MCP installs) into `BACKUP_ARCHIVE_DIR` as
- * `cobble-data-{timestamp}.tar.gz`, then keeps only the newest
- * `AppData.DataBackupsKept`. Every entry sits under one root folder named
- * after the archive (`cobble-data-{timestamp}/`), so extracting it lands
- * one folder, never a heap of files in the current directory. The
- * database's own snapshots in `db/.backups` are inside `DATA_DIR`, so they
- * ride along and are left alone on disk.
- *
- * The archive is a filesystem copy of a live volume, so the SQLite file
- * inside it is only as consistent as the moment tar read it; a consistent
- * database snapshot is what the `db/.backups` list is for. Restoring a
- * whole-data archive is `restoreDataBackup` (`restore-data-backup.ts`).
- */
-export default async function createDataBackup() {
-    return await withBackupRunLock({ task: writeDataBackupArchive });
-}

src/functions/backend/backups/data-backup-run.test.ts

@@ -2,6 +2,7 @@ import { describe, expect, test } from "bun:test";
 import AppError from "@/src/utils/app-error";
 import { BootId, grabDataBackupRun, startDataBackupRun } from "./data-backup-run";
 import { isBackupRunning } from "./with-backup-run-lock";
+import restoreDataBackup from "./restore-data-backup";
 
 function deferred() {
     let resolve!: (name: string) => void;
@@ -80,4 +81,19 @@ describe("startDataBackupRun", () => {
         await settle();
         expect(grabDataBackupRun().status).toBe("done");
     });
+
+    test("a restore runs under the run's lock instead of taking it again", async () => {
+        startDataBackupRun({
+            kind: "restore",
+            task: async () =>
+                (await restoreDataBackup({ backup_name: "missing.tar.gz" }))
+                    .restored_name,
+        });
+        await settle();
+
+        const failed = grabDataBackupRun();
+        expect(failed.status).toBe("failed");
+        expect(failed.error).toBe("Archive not found");
+        expect(isBackupRunning()).toBe(false);
+    });
 });

src/functions/backend/backups/restore-data-backup.ts

@@ -5,7 +5,6 @@ import { AppData } from "@/src/data/app-data";
 import grabDirNames from "@/src/utils/grab-dir-names";
 import writeDataBackupArchive from "./write-data-backup-archive";
 import resolveDataBackupPath from "./resolve-data-backup-path";
-import withBackupRunLock from "./with-backup-run-lock";
 
 export type DataBackupRestoreResult = {
     /**
@@ -61,14 +60,15 @@ function findArchiveRoot({ staging_dir }: RootParams) {
  * is taken as a safety copy, then the data directory is emptied (the
  * database snapshots included) and filled from the archive.
  *
+ * Runs only inside `startDataBackupRun`, which holds the backup run lock
+ * for it (taking it again here would refuse every restore).
+ *
  * The caller restarts the process afterwards: the open SQLite handle still
  * points at the replaced file, so nothing may keep reading it.
  */
 export default async function restoreDataBackup({
     backup_name,
 }: Params): Promise<DataBackupRestoreResult> {
-    return await withBackupRunLock({
-        task: async () => {
     const { DATA_DIR } = grabDirNames();
     const archive_path = resolveDataBackupPath({ backup_name });
     const archive_name = path.basename(archive_path);
@@ -116,8 +116,6 @@ export default async function restoreDataBackup({
 
         // The archive's snapshots would otherwise vanish with the
         // current ones, so the safety copy is taken before them.
-                // The run lock is already held here, hence the unlocked
-                // writer rather than `createDataBackup`.
         const safety = await writeDataBackupArchive();
 
         // Everything goes, the snapshots in `db/.backups` included:
@@ -144,6 +142,4 @@ export default async function restoreDataBackup({
     } finally {
         fs.rmSync(staging_dir, { recursive: true, force: true });
     }
-        },
-    });
 }

src/functions/backend/backups/write-data-backup-archive.ts

@@ -11,9 +11,17 @@ export type CreatedDataBackup = {
 };
 
 /**
- * Writes one archive of the whole `DATA_DIR` and prunes the old ones.
+ * Writes one archive of the whole `DATA_DIR` (database, media, computer
- * Callers already hold the backup run lock (`withBackupRunLock`), so a
+ * workspaces, bot and workspace files, MCP installs) into
- * restore can take its safety copy without deadlocking on it.
+ * `BACKUP_ARCHIVE_DIR` and keeps only the newest `AppData.DataBackupsKept`.
+ * Every entry sits under one root folder named after the archive, so
+ * extracting it lands one folder. The SQLite file inside is only as
+ * consistent as the moment tar read it; `db/.backups` is for consistent
+ * snapshots.
+ *
+ * It takes no lock: callers run inside `startDataBackupRun`, which holds
+ * the backup run lock (`withBackupRunLock`) for the whole run, so the
+ * archive route and a restore's safety copy never take it twice.
  */
 export default async function writeDataBackupArchive(): Promise<CreatedDataBackup> {
     const { DATA_DIR } = grabDirNames();

src/pages/admin/settings/backups/(data)/data-backup-run-context.ts

@@ -14,6 +14,10 @@ export type DataBackupRunState = {
      * Follows a run a request just started until it ends.
      */
     track: (run: DataBackupRun) => void;
+    /**
+     * Shows why a run couldn't start (or clears it with `undefined`).
+     */
+    report: (error?: string) => void;
 };
 
 /**
@@ -23,4 +27,5 @@ export type DataBackupRunState = {
  */
 export const DataBackupRunContext = React.createContext<DataBackupRunState>({
     track: () => {},
+    report: () => {},
 });

src/pages/admin/settings/backups/(hooks)/use-data-backup-run.ts

@@ -94,5 +94,6 @@ export default function useDataBackupRun(): DataBackupRunState {
             setError(undefined);
             setRun(next);
         },
+        report: setError,
     };
 }

src/pages/admin/settings/backups/(partials)/data-backup-list-actions.tsx

@@ -1,6 +1,5 @@
 import { Archive, Upload } from "lucide-react";
 import Button from "@moduletrace/twui/layout/Button";
-import Tag from "@moduletrace/twui/elements/Tag";
 import Row from "@moduletrace/twui/layout/Row";
 import useStatus from "@moduletrace/twui/hooks/useStatus";
 import createDataBackup from "../(functions)/create-data-backup";
@@ -10,18 +9,17 @@ import { useContext, useState } from "react";
 
 /**
  * The data archive list's two ways in: make a fresh archive, or restore
- * one from a file. Also says how the archive or restore running now is
+ * one from a file. How a run is going, and why one failed, is shown under
- * going, and why the last one failed.
+ * the row by `DataBackupRunNotice`, so the buttons never move.
  */
 export default function DataBackupListActions() {
-    const { loading, setLoading, status, setStatus } = useStatus();
+    const { loading, setLoading } = useStatus();
-    const { run, error, track } = useContext(DataBackupRunContext);
+    const { run, track, report } = useContext(DataBackupRunContext);
     const [uploading, setUploading] = useState(false);
     const archiving = run?.kind == "archive";
-    const restoring = run?.kind == "restore";
 
     const create_backup = async () => {
-        setStatus(undefined);
+        report(undefined);
         setLoading(true);
 
         const res = await createDataBackup();
@@ -34,20 +32,12 @@ export default function DataBackupListActions() {
         }
 
         if (res.msg) {
-            setStatus({ error: true, msg: res.msg });
+            report(res.msg);
         }
     };
 
     return (
         <Row className="gap-2! flex-nowrap">
-            {status?.error && <Tag color="error">{status.msg}</Tag>}
-            {!status?.error && error && <Tag color="error">{error}</Tag>}
-            {archiving && (
-                <Tag color="warning">Archiving, this can take a while...</Tag>
-            )}
-            {restoring && (
-                <Tag color="warning">Restoring, the app restarts soon...</Tag>
-            )}
             <Button
                 title="Restore from an archive file"
                 variant="outlined"

src/pages/admin/settings/backups/(partials)/data-backup-run-notice.tsx

@@ -0,0 +1,26 @@
+import { useContext } from "react";
+import Tag from "@moduletrace/twui/elements/Tag";
+import { DataBackupRunContext } from "../(data)/data-backup-run-context";
+
+/**
+ * One line under the data archives' actions: the archive or restore
+ * running now, else why the last one failed. Nothing when neither, so the
+ * actions row keeps its layout either way.
+ */
+export default function DataBackupRunNotice() {
+    const { run, error } = useContext(DataBackupRunContext);
+
+    if (run?.kind == "archive") {
+        return <Tag color="gray">Archiving, this can take a while...</Tag>;
+    }
+
+    if (run?.kind == "restore") {
+        return <Tag color="warning">Restoring, the app restarts soon...</Tag>;
+    }
+
+    if (error) {
+        return <Tag color="error">{error}</Tag>;
+    }
+
+    return null;
+}

src/pages/admin/settings/backups/(sections)/data-backups-section.tsx

@@ -4,6 +4,7 @@ import Span from "@moduletrace/twui/layout/Span";
 import SettingsGroup from "../../(partials)/settings-group";
 import { AppContext } from "@/src/pages/__root";
 import DataBackupListActions from "../(partials)/data-backup-list-actions";
+import DataBackupRunNotice from "../(partials)/data-backup-run-notice";
 import DataBackupsTable from "../(partials)/data-backups-table";
 import { DataBackupRunContext } from "../(data)/data-backup-run-context";
 import useDataBackupRun from "../(hooks)/use-data-backup-run";
@@ -33,6 +34,7 @@ export default function DataBackupsSection() {
                     </Span>
                     <DataBackupListActions />
                 </Row>
+                <DataBackupRunNotice />
                 {backups[0] ? (
                     <DataBackupsTable backups={backups} />
                 ) : (

src/pages/api/admin/backups/create-data.ts

@@ -1,5 +1,5 @@
 import grabSuperAdminUser from "@/src/functions/backend/backups/grab-super-admin-user";
-import createDataBackup from "@/src/functions/backend/backups/create-data-backup";
+import writeDataBackupArchive from "@/src/functions/backend/backups/write-data-backup-archive";
 import { startDataBackupRun } from "@/src/functions/backend/backups/data-backup-run";
 import logEvent, { grabRequestIP, logError } from "@/src/functions/backend/logger";
 import safeErrorMessage from "@/src/utils/safe-error-message";
@@ -12,7 +12,8 @@ import type {
 /**
  * Starts archiving the whole `DATA_DIR` (Settings > Backups > Data
  * archives) and answers at once: the page polls `data-status` until the
- * archive is written.
+ * archive is written. `startDataBackupRun` holds the backup run lock, so
+ * the task calls the unlocked writer.
  */
 export const handler: BunextAPIRouteHandler<
     APIResponseObject<DataBackupRun>
@@ -33,7 +34,7 @@ export const handler: BunextAPIRouteHandler<
             kind: "archive",
             task: async () => {
                 try {
-                    const backup = await createDataBackup();
+                    const backup = await writeDataBackupArchive();
 
                     await logEvent({
                         title: "Data backup created",