Keep every bot's computer work in its own workspace folder

Computers now mount only /home/gem/workspace from DATA_DIR/computers/<id>,
so only the workspace persists and goes into the data archive. An older
computer's whole home is moved aside to <id>-home-<ts> (kept, never
archived) and its container recreated once by sync-computers.

Each run works in bots/<bot>/users/<user> (also its cwd and HOME), with
bots/<bot> for work every user shares and workspaces/<id> shared by a
workspace's bots; the coordinator reaches the whole workspace. Computer
tools are gated by withComputerDirs: bash and code start there, file tool
paths and file: URLs outside the run's folders are refused, and so is sudo.
search_computers, import_file and script tools start there too.

The files browser shows members only the bots' folders they reach, their
own user folder in each and their workspaces', with names on the id folders.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
2026-10-25 08:24:49 +01:00
co-authored by Claude Opus 5.5
parent fefc295d15
commit 86becd50bc
41 changed files with 1012 additions and 105 deletions
+7 -6
View File
@@ -116,7 +116,8 @@ Inspect live data with `bun llm-code-assistant/query-db.ts "<sql>"` - a read-onl
- **Queue handler:** `src/bots-queue/index.ts`, started from `server.ts` after cron and gated the same way. It releases its own cut-off jobs on startup (stable worker `AppData.BotQueueWorker`; polling starts even if that release fails, and a production process without `MAIN_SERVER` says so in the log instead of going quiet), then polls every `BotQueuePollMs`: reports jobs bun-sqlite failed on a lock timeout (`report-lost-bot-jobs`), claims per kind (`claimBotJobs` mirrors bun-sqlite's atomic claim plus `kind`, since `queue.claim` has no filter) up to `BotQueueReplyConcurrency` replies and `BotQueueConcurrency` tasks on separate slots, and runs each with `processBotJob` (heartbeat at a quarter of the lock timeout, abort on cancel/loss, one task per computer and one reply per session via `createKeyLock`, so a bot's other sessions answer side by side; a task waiting for its busy computer shows face `asleep` with `BotComputerBusyStatusLine`; face `working` once it starts, outcome pushed live; a job finishing while others for that user still run puts back the running one's face, `grabRunningBotJobFace`). The actual work is `run-bot-job.ts` (`BotJobRunner`, context in `types.ts`), which dispatches to `run-bot-reply.ts` and `run-bot-task.ts`. Throw `BotJobFatalError` for failures retrying can't fix. When every model is resting (`HarnessError.resting`, nothing was tried) the job is put back with `deferBotJob` until the first comes back, its attempt refunded and the bot's face `asleep`; other retryable harness errors retry no sooner than their `retry_at`. The `prune-bots-queue` cron deletes finished jobs after `BotQueueRetentionDays`. - **Queue handler:** `src/bots-queue/index.ts`, started from `server.ts` after cron and gated the same way. It releases its own cut-off jobs on startup (stable worker `AppData.BotQueueWorker`; polling starts even if that release fails, and a production process without `MAIN_SERVER` says so in the log instead of going quiet), then polls every `BotQueuePollMs`: reports jobs bun-sqlite failed on a lock timeout (`report-lost-bot-jobs`), claims per kind (`claimBotJobs` mirrors bun-sqlite's atomic claim plus `kind`, since `queue.claim` has no filter) up to `BotQueueReplyConcurrency` replies and `BotQueueConcurrency` tasks on separate slots, and runs each with `processBotJob` (heartbeat at a quarter of the lock timeout, abort on cancel/loss, one task per computer and one reply per session via `createKeyLock`, so a bot's other sessions answer side by side; a task waiting for its busy computer shows face `asleep` with `BotComputerBusyStatusLine`; face `working` once it starts, outcome pushed live; a job finishing while others for that user still run puts back the running one's face, `grabRunningBotJobFace`). The actual work is `run-bot-job.ts` (`BotJobRunner`, context in `types.ts`), which dispatches to `run-bot-reply.ts` and `run-bot-task.ts`. Throw `BotJobFatalError` for failures retrying can't fix. When every model is resting (`HarnessError.resting`, nothing was tried) the job is put back with `deferBotJob` until the first comes back, its attempt refunded and the bot's face `asleep`; other retryable harness errors retry no sooner than their `retry_at`. The `prune-bots-queue` cron deletes finished jobs after `BotQueueRetentionDays`.
- **Live updates:** server push only. `sendToUser` (`src/websocket/(functions)/send-to-user.ts`) sends to one user's sockets; `pushBotNotice` sends `server:bot-message` (the message row, user messages too, for other tabs) and `server:bot-updated` (`BotUpdatedData`: face, status line, active task). The bots layout opens one socket (twui `useWebSocket`); components listen with `useServerEvent` (`src/hooks/`), and `useLiveBots` keeps the rail and header current. The conversation dedupes messages by id. Every process also runs `liveRelay` (`src/live-relay/`): every `AppData.LiveRelayPollMs` it reads new `bot_messages`, new `bots_queue_updates`, changed faces and changed bots from the database and pushes them to its own sockets, so work done by another process still shows live; clients drop what they already have. `server:bot-updated` carries the bot's `computer_id` when known, which keeps the header's computer in step across tabs, and its `workspace_ids` after a membership change, so a workspace's crew drops a bot that left it. A bot that joins the crew or a workspace is pushed as `server:bot-created` (`BotCreatedData`: its rail row, the workspaces it's in, and whether it belongs in that user's own chat), from the tool that created it and from the live relay for other processes, so the rail lists it without a reload. A workspace a bot created is pushed the same way as `server:workspace-created` (`WorkspaceCreatedData`: its rail row), from the tool and from the live relay. A deleted bot is pushed as `server:bot-removed` (`BotRemovedData`) to the audience read before the delete, since its assignments go with it, so its rail row goes at once. A deleted workspace is pushed the same way as `server:workspace-removed` (`WorkspaceRemovedData`), read before the delete too. A rail row's subtitle is the bot's status line while it has one, else (coordinator) the crew counts while a worker is busy or needs you, else its last message as plain text (`src/utils/message-preview.ts`). - **Live updates:** server push only. `sendToUser` (`src/websocket/(functions)/send-to-user.ts`) sends to one user's sockets; `pushBotNotice` sends `server:bot-message` (the message row, user messages too, for other tabs) and `server:bot-updated` (`BotUpdatedData`: face, status line, active task). The bots layout opens one socket (twui `useWebSocket`); components listen with `useServerEvent` (`src/hooks/`), and `useLiveBots` keeps the rail and header current. The conversation dedupes messages by id. Every process also runs `liveRelay` (`src/live-relay/`): every `AppData.LiveRelayPollMs` it reads new `bot_messages`, new `bots_queue_updates`, changed faces and changed bots from the database and pushes them to its own sockets, so work done by another process still shows live; clients drop what they already have. `server:bot-updated` carries the bot's `computer_id` when known, which keeps the header's computer in step across tabs, and its `workspace_ids` after a membership change, so a workspace's crew drops a bot that left it. A bot that joins the crew or a workspace is pushed as `server:bot-created` (`BotCreatedData`: its rail row, the workspaces it's in, and whether it belongs in that user's own chat), from the tool that created it and from the live relay for other processes, so the rail lists it without a reload. A workspace a bot created is pushed the same way as `server:workspace-created` (`WorkspaceCreatedData`: its rail row), from the tool and from the live relay. A deleted bot is pushed as `server:bot-removed` (`BotRemovedData`) to the audience read before the delete, since its assignments go with it, so its rail row goes at once. A deleted workspace is pushed the same way as `server:workspace-removed` (`WorkspaceRemovedData`), read before the delete too. A rail row's subtitle is the bot's status line while it has one, else (coordinator) the crew counts while a worker is busy or needs you, else its last message as plain text (`src/utils/message-preview.ts`).
- **Workspaces:** `workspaces` (`name`, `description`, `system_prompt` shared by every bot in it, `computer_id` its default computer (not a foreign key), `archived`), with bots assigned through `workspace_bots` (plus that bot's `system_prompt` and `memory` there; the coordinator is an implicit member) and computers through `workspace_computers`. The first computer linked becomes the default; the user (workspace settings, chat header), any bot there (`set_workspace_computer`) or the coordinator can switch it. A workspace with no computer has none: tasks needing one fail. Workspace models (`workspace_harness_settings`) win over a bot's own, then the defaults. Helpers in `src/functions/backend/db/workspaces/` (`create-workspace`, `update-workspace`, `delete-workspace`, `add/remove-bot-*`, `update-workspace-bot`, `add/remove-computer-*`, `set-workspace-computer`, `grab-user-workspaces`, `grab-workspace-bots`, `grab-workspace-computers`, `grab-workspace-activity`), all access-checked against the session user, the structural ones for super admins and admins only; API in `api/admin/workspaces/`. The coordinator sets a member's instructions there with `set_bot_workspace_instructions`, creates a workspace with `create_workspace`, and changes any workspace's name, description or context with `update_workspace` (chat replies, and its own tasks so findings it was asked to record land in the workspace; a chat-created one is pushed as `server:workspace-created`). - **Workspaces:** `workspaces` (`name`, `description`, `system_prompt` shared by every bot in it, `computer_id` its default computer (not a foreign key), `archived`), with bots assigned through `workspace_bots` (plus that bot's `system_prompt` and `memory` there; the coordinator is an implicit member) and computers through `workspace_computers`. The first computer linked becomes the default; the user (workspace settings, chat header), any bot there (`set_workspace_computer`) or the coordinator can switch it. A workspace with no computer has none: tasks needing one fail. Workspace models (`workspace_harness_settings`) win over a bot's own, then the defaults. Helpers in `src/functions/backend/db/workspaces/` (`create-workspace`, `update-workspace`, `delete-workspace`, `add/remove-bot-*`, `update-workspace-bot`, `add/remove-computer-*`, `set-workspace-computer`, `grab-user-workspaces`, `grab-workspace-bots`, `grab-workspace-computers`, `grab-workspace-activity`), all access-checked against the session user, the structural ones for super admins and admins only; API in `api/admin/workspaces/`. The coordinator sets a member's instructions there with `set_bot_workspace_instructions`, creates a workspace with `create_workspace`, and changes any workspace's name, description or context with `update_workspace` (chat replies, and its own tasks so findings it was asked to record land in the workspace; a chat-created one is pushed as `server:workspace-created`).
- **Computers:** a computer is an AIO sandbox in its own docker container on the `AppData.DockerNetworkName` network. `containers` records every docker container the app knows about except the main app container (`name` doubles as the hostname, plus `docker_id`, `image`, `network`, `port`, `status`, `metadata`). `computers` (the install's, `kind` main|sandbox, `container_id`) and `bot_computers` (direct access). The app creates and manages every computer's container over the docker socket (`src/functions/backend/docker/`; `DOCKER_SOCKET` or `AppData.DockerSocketPath`): `createComputer` records it and `provisionComputer` (in the background, status `creating`, then the container's or `failed` with `metadata.error`, pushed as `server:computer-updated`) pulls `AppData.ComputerImage`, binds the home dir and starts `<ComputerContainerPrefix>-<id>` limited to the computer's `cpus` and `memory_mb` (set at creation, default `AppData.ComputerDefaultCpus`/`ComputerDefaultMemoryMb`: 1 CPU, 2 GB), attached to the network in the same create call, with a docker health check (`AppData.ComputerHealthcheck`: the sandbox API on `ComputerPort` and its browser); the computer stays `creating` until it passes (`waitForHealthyContainer`), then records its IP address on the network (`containers.ip_address`, refreshed by `sync-computers`). `updateComputer` (`api/admin/computers/update-computer`) saves name, description and resources; new resources stop, remove and recreate the container (`provisionComputer`), the home dir kept. A computer's home (`/home/gem`) is `DATA_DIR/computers/<id>` (`AppData.ComputerDataDir`, owned by uid 1000); deleting the computer (`deleteComputer`, never the main one) removes the container and keeps that directory. Run in docker, the app resolves host paths from its own container's mounts and joins the network; outside docker (dev) it can't resolve container hostnames, so it reaches them at their IP address (`pickReachableAddress`: hostname first, then the IP, the pick cached for `AppData.ComputerAddressCacheMs`). Only super admins create, delete or retry computers (`api/admin/computers/`). The `sync-computers` cron records container states every five minutes. The install has one main computer, created at signup (`bun run seed`) and as a fallback by the super admin's `/admin/bots` via `ensureMainComputer` (`src/functions/backend/db/computers/ensure-main-computer.ts`, idempotent), which grants the coordinator access; with `MAIN_COMPUTER_URL` set (dev) it isn't provisioned. That one grant can't be removed (`isLockedComputerAccess`, enforced in `remove-bot-from-computer` and the `bot_computers` CRUD handler); every other grant can. A bot's effective computers (`grab-bot-computers`): the workspace's computers in a workspace, its default first; in its own chat, its direct grants and those of the workspaces it's attached to (the coordinator's are every computer), the one it's on first: `bots.computer_id` (not a foreign key; null or a lost grant means its main one), so `computers[0]` is where its tasks run by default. The user switches it from the chat header (`(partials)/bot-computers-modal.tsx`, a ghost icon beside the model icon; the header names the current computer and the text model and provider under the bot's name) or by clicking a computer the bot named in a message: bots write `[computer:ID]` (`src/utils/computer-ref.ts`, turned into `#computer-ID` links by `link-computer-refs.ts`, rendered by `(partials)/computer-ref-chip.tsx`), both sharing `BotComputersContext` / `use-bot-computers`, saved by `api/admin/bots/set-computer.ts` (`setBotComputer`). Enums in `src/dict/computer-dict.ts`. - **Computers:** a computer is an AIO sandbox in its own docker container on the `AppData.DockerNetworkName` network. `containers` records every docker container the app knows about except the main app container (`name` doubles as the hostname, plus `docker_id`, `image`, `network`, `port`, `status`, `metadata`). `computers` (the install's, `kind` main|sandbox, `container_id`) and `bot_computers` (direct access). The app creates and manages every computer's container over the docker socket (`src/functions/backend/docker/`; `DOCKER_SOCKET` or `AppData.DockerSocketPath`): `createComputer` records it and `provisionComputer` (in the background, status `creating`, then the container's or `failed` with `metadata.error`, pushed as `server:computer-updated`) pulls `AppData.ComputerImage`, binds the computer's workspace dir and starts `<ComputerContainerPrefix>-<id>` limited to the computer's `cpus` and `memory_mb` (set at creation, default `AppData.ComputerDefaultCpus`/`ComputerDefaultMemoryMb`: 1 CPU, 2 GB), attached to the network in the same create call, with a docker health check (`AppData.ComputerHealthcheck`: the sandbox API on `ComputerPort` and its browser); the computer stays `creating` until it passes (`waitForHealthyContainer`), then records its IP address on the network (`containers.ip_address`, refreshed by `sync-computers`). `updateComputer` (`api/admin/computers/update-computer`) saves name, description and resources; new resources stop, remove and recreate the container (`provisionComputer`), the workspace dir kept. Only a computer's workspace (`/home/gem/workspace`, `AppData.ComputerWorkspaceDir`) is mounted, from `DATA_DIR/computers/<id>` (`AppData.ComputerDataDir`, owned by uid 1000, with `bots/`, `workspaces/` and the layout marker `AppData.ComputerWorkspaceMarker`); the rest of `/home/gem` (dotfiles, the browser profile, the env files) lives in the container and goes when it is recreated. `prepareComputerDataDir` moves a directory without the marker (an older computer's whole home) aside to `<id>-home-<ts>` and keeps it on disk, and `sync-computers` recreates a container whose `metadata.layout` isn't `workspace` once (`migrateComputerLayout`). Deleting the computer (`deleteComputer`, never the main one) removes the container and keeps that directory. Run in docker, the app resolves host paths from its own container's mounts and joins the network; outside docker (dev) it can't resolve container hostnames, so it reaches them at their IP address (`pickReachableAddress`: hostname first, then the IP, the pick cached for `AppData.ComputerAddressCacheMs`). Only super admins create, delete or retry computers (`api/admin/computers/`). The `sync-computers` cron records container states every five minutes. The install has one main computer, created at signup (`bun run seed`) and as a fallback by the super admin's `/admin/bots` via `ensureMainComputer` (`src/functions/backend/db/computers/ensure-main-computer.ts`, idempotent), which grants the coordinator access; with `MAIN_COMPUTER_URL` set (dev) it isn't provisioned. That one grant can't be removed (`isLockedComputerAccess`, enforced in `remove-bot-from-computer` and the `bot_computers` CRUD handler); every other grant can. A bot's effective computers (`grab-bot-computers`): the workspace's computers in a workspace, its default first; in its own chat, its direct grants and those of the workspaces it's attached to (the coordinator's are every computer), the one it's on first: `bots.computer_id` (not a foreign key; null or a lost grant means its main one), so `computers[0]` is where its tasks run by default. The user switches it from the chat header (`(partials)/bot-computers-modal.tsx`, a ghost icon beside the model icon; the header names the current computer and the text model and provider under the bot's name) or by clicking a computer the bot named in a message: bots write `[computer:ID]` (`src/utils/computer-ref.ts`, turned into `#computer-ID` links by `link-computer-refs.ts`, rendered by `(partials)/computer-ref-chip.tsx`), both sharing `BotComputersContext` / `use-bot-computers`, saved by `api/admin/bots/set-computer.ts` (`setBotComputer`). Enums in `src/dict/computer-dict.ts`.
- **Computer folders (enforced):** a bot works on a computer only in its own folders there (`computerRunDirs`, `src/utils/computer-work-dirs.ts`): `bots/<bot>/users/<user>` for the run's user (where every shell, code and file tool starts, also `HOME` and `~`), `bots/<bot>` for work every user shares, and in a workspace `workspaces/<workspace>`, shared by its bots; the coordinator reaches the whole workspace. `openComputerTools` creates them and wraps every tool with `withComputerDirs` (`src/harness/tools/computer/`): bash runs with that `cwd`, code `chdir`s there, `sandbox_file_operations`/`sandbox_str_replace_editor` paths and `file:` URLs (`browser_navigate`, `browser_new_tab`, `sandbox_convert_to_markdown`) go through `resolveComputerPath`, which refuses anything outside the run's folders or another user's folder, and `sudo` is refused. `search_computers`, `import_file` (paths relative to the user's folder, folder refs relative to the workspace) and script tools (`computerDirsShell`) start there too; `ComputerFilesNote` states the rule. It is a tool-level gate, not a Linux user per bot: a deliberate `cd /` in bash still works.
- **Opening a computer:** the app serves each computer to whoever sees it (`computersSql`) under `/computers/<id>/` (`ComputerProxyPrefix`): `src/server.ts` hands those paths to `handleComputerProxy` (`src/functions/server/computers/`: `userAuth`, access via `grabUserComputer`, then the sandbox's address from `grabComputerMcpUrl`), which fetches plain requests (app session cookies dropped, `X-Forwarded-Host/Proto/Prefix` set so the sandbox builds its links and CDP address under the prefix, framing headers and redirects fixed) and bridges websockets (`computerProxySocket`, dispatched from `src/websocket/` when `ws.data.proxy` is set). `ComputerView` (`src/components/general/computer-view.tsx`) shows a computer in one of `ComputerViewModes` (browser `/browser-ui`, terminal `/terminal`, code `/code-server/`, desktop: the sandbox's noVNC at `/vnc/vnc.html`, its socket pointed back through the proxy by the `{prefix}` placeholder that `computerViewUrl` fills in; desktop is the only view with Chrome's toolbar (extensions) and a clipboard panel, since the browser view streams pages over CDP and hides `chrome-extension://` targets; more of the sandbox's apps can be added there), keeping visited modes loaded. In the chat, `(partials)/computer-viewer-modal.tsx` shows it, opened from a computer card's open button in the computers modal or a take-over button (`openComputer` on `BotComputersContext`). - **Opening a computer:** the app serves each computer to whoever sees it (`computersSql`) under `/computers/<id>/` (`ComputerProxyPrefix`): `src/server.ts` hands those paths to `handleComputerProxy` (`src/functions/server/computers/`: `userAuth`, access via `grabUserComputer`, then the sandbox's address from `grabComputerMcpUrl`), which fetches plain requests (app session cookies dropped, `X-Forwarded-Host/Proto/Prefix` set so the sandbox builds its links and CDP address under the prefix, framing headers and redirects fixed) and bridges websockets (`computerProxySocket`, dispatched from `src/websocket/` when `ws.data.proxy` is set). `ComputerView` (`src/components/general/computer-view.tsx`) shows a computer in one of `ComputerViewModes` (browser `/browser-ui`, terminal `/terminal`, code `/code-server/`, desktop: the sandbox's noVNC at `/vnc/vnc.html`, its socket pointed back through the proxy by the `{prefix}` placeholder that `computerViewUrl` fills in; desktop is the only view with Chrome's toolbar (extensions) and a clipboard panel, since the browser view streams pages over CDP and hides `chrome-extension://` targets; more of the sandbox's apps can be added there), keeping visited modes loaded. In the chat, `(partials)/computer-viewer-modal.tsx` shows it, opened from a computer card's open button in the computers modal or a take-over button (`openComputer` on `BotComputersContext`).
- **Take-over:** a bot asks the user to take over a computer with `ask_user_to_take_over` (`src/harness/tools/computer/ask-to-take-over-tool.ts`, chat replies and tasks, defaulting to the task's computer). It writes `[take-over:ID:mode]` (`src/utils/take-over-ref.ts`; `withTakeOverRefs` adds it if the bot left it out and flags the message needs-you), linked by `link-computer-refs.ts` and rendered by `(partials)/take-over-chip.tsx`. The viewer then offers "Hand back to <bot>", which sends a chat message so the bot carries on; a task stops and reports instead of waiting. - **Take-over:** a bot asks the user to take over a computer with `ask_user_to_take_over` (`src/harness/tools/computer/ask-to-take-over-tool.ts`, chat replies and tasks, defaulting to the task's computer). It writes `[take-over:ID:mode]` (`src/utils/take-over-ref.ts`; `withTakeOverRefs` adds it if the bot left it out and flags the message needs-you), linked by `link-computer-refs.ts` and rendered by `(partials)/take-over-chip.tsx`. The viewer then offers "Hand back to <bot>", which sends a chat message so the bot carries on; a task stops and reports instead of waiting.
- **Health:** `containers.status` is docker's state with `running` narrowed by the health check (`starting`, `unhealthy`; `inspectDockerContainer`). Only a `running` computer is reached: `grabComputerMcpUrl` returns `ready: false` otherwise (re-reading a stale state from docker first, `refreshComputerContainer`, which `sync-computers` uses too), and callers say why (`computerNotReadyMessage`). - **Health:** `containers.status` is docker's state with `running` narrowed by the health check (`starting`, `unhealthy`; `inspectDockerContainer`). Only a `running` computer is reached: `grabComputerMcpUrl` returns `ready: false` otherwise (re-reading a stale state from docker first, `refreshComputerContainer`, which `sync-computers` uses too), and callers say why (`computerNotReadyMessage`).
@@ -165,7 +166,7 @@ Tools in `src/harness/tools/storage/` (`grabBotFileTools`, every run: chat, task
A task on a computer also gets `import_file` (`src/harness/tools/computer/import-file-tool.ts`): it copies a file or a folder off the computer into the run's files (a folder as a gzipped tar, unpacked into one `bot_files` row per file, source `tool`), ingests it at once, and answers with the `[file:ID]` refs plus the original's path. The bytes come back through `downloadComputerFile` (`download-computer-file.ts`), the sandbox's streaming `GET /v1/file/download` - **never through its shell**: `/v1/shell/exec` caps output at 30,047 characters, so `base64 -w0` silently cut every import at 22,528 bytes and a 1.12 MiB archive was saved as a 22 KB file that would not open, with the tool reporting success. The size is read first (nothing over `AppData.BotFileImportMaxBytes` is transferred), the download asks for `change_policy=abort` (409 if the file changes while it is sent) and the length is checked against the size read, so a cut-short transfer is reported instead of saved. `BotFilesNote` and `ComputerFilesNote` (`src/data/harness-prompts.ts`) tell the bot to save the deliverable before the long tail of a run, so a step limit can't cost the user the report. A task on a computer also gets `import_file` (`src/harness/tools/computer/import-file-tool.ts`): it copies a file or a folder off the computer into the run's files (a folder as a gzipped tar, unpacked into one `bot_files` row per file, source `tool`), ingests it at once, and answers with the `[file:ID]` refs plus the original's path. The bytes come back through `downloadComputerFile` (`download-computer-file.ts`), the sandbox's streaming `GET /v1/file/download` - **never through its shell**: `/v1/shell/exec` caps output at 30,047 characters, so `base64 -w0` silently cut every import at 22,528 bytes and a 1.12 MiB archive was saved as a 22 KB file that would not open, with the tool reporting success. The size is read first (nothing over `AppData.BotFileImportMaxBytes` is transferred), the download asks for `change_policy=abort` (409 if the file changes while it is sent) and the length is checked against the size read, so a cut-short transfer is reported instead of saved. `BotFilesNote` and `ComputerFilesNote` (`src/data/harness-prompts.ts`) tell the bot to save the deliverable before the long tail of a run, so a step limit can't cost the user the report.
Anything a bot leaves on a computer is not in `bot_files`, but the user can still open it: bots name a folder as `[folder:<volume>:<owner>:<path>]` (`src/utils/folder-ref.ts`, e.g. `[folder:computer:3:seo/coderank/raw]`), linked by the same `linkComputerRefs` and rendered by `folder-ref-chip.tsx`, which opens `/admin/files` on that folder and downloads nothing. A `[file:<id>]` ref (`src/utils/file-ref.ts`) is always a saved file, whose chip opens `/media/<id>`; the two are never guessed from a name's extension. Anything a bot leaves on a computer is not in `bot_files`, but the user can still open it: bots name a folder as `[folder:<volume>:<owner>:<path>]` (`src/utils/folder-ref.ts`, e.g. `[folder:computer:3:bots/4/users/1/seo/coderank/raw]`, a computer's path relative to its workspace), linked by the same `linkComputerRefs` and rendered by `folder-ref-chip.tsx`, which opens `/admin/files` on that folder and downloads nothing. A `[file:<id>]` ref (`src/utils/file-ref.ts`) is always a saved file, whose chip opens `/media/<id>`; the two are never guessed from a name's extension.
Ingestion (`bot_file_chunks`, `bot_file_vectors` a vec0 table, `file_ingest_queue`): `ingestBotFile` extracts text (`extractBotFileText`, reusing `fileToText`; images and audio only on request), chunks it (`chunkText`), embeds it through the `embedding` harness and fits every vector to the fixed `AppData.EmbeddingDimensions` (`fitVectorDimensions`, pad or cut then normalize). The `file-ingest` worker (`src/file-ingest/`, started from `server.ts` next to the bots queue) works the queue; `ingest-pending-files` sweeps up anything stranded. With no embedding model the chunks are still stored and `searchBotFiles` falls back to word matching, always saying which it did. Ingestion (`bot_file_chunks`, `bot_file_vectors` a vec0 table, `file_ingest_queue`): `ingestBotFile` extracts text (`extractBotFileText`, reusing `fileToText`; images and audio only on request), chunks it (`chunkText`), embeds it through the `embedding` harness and fits every vector to the fixed `AppData.EmbeddingDimensions` (`fitVectorDimensions`, pad or cut then normalize). The `file-ingest` worker (`src/file-ingest/`, started from `server.ts` next to the bots queue) works the queue; `ingest-pending-files` sweeps up anything stranded. With no embedding model the chunks are still stored and `searchBotFiles` falls back to word matching, always saying which it did.
@@ -177,11 +178,11 @@ A bot's or workspace's folder is removed with it (`purgeStorage`, called from `d
### Files browser (`/admin/files`) ### Files browser (`/admin/files`)
Every folder on the data volume, browsable read-only: a bot's own, a workspace's shared one, a computer's home (`DATA_DIR/computers/<id>`, where everything a bot left in `~/…` actually is), the media library, an MCP server's install folder and the database. A `RailPages` entry beside Routines, Activity and Help. Every folder on the data volume, browsable read-only: a bot's own, a workspace's shared one, a computer's workspace (`DATA_DIR/computers/<id>`, mounted at `/home/gem/workspace`, where everything a bot left on it actually is), the media library, an MCP server's install folder and the database. A `RailPages` entry beside Routines, Activity and Help.
- **Volumes:** `src/utils/storage-volumes.ts` (client-safe): `StorageVolumes` (`bots`, `workspaces`, `computers`, `media`, `mcp`, `db`) with each one's ref name (`bot`, `computer`, …), `parseStorageSpec({ volume, owner_id })` (null for anything unknown, so a bad id never becomes a path), and `storageSpecHref({ spec, path })` → `/admin/files?volume=<v>&owner=<id>&path=<p>`, the one place a link is built. - **Volumes:** `src/utils/storage-volumes.ts` (client-safe): `StorageVolumes` (`bots`, `workspaces`, `computers`, `media`, `mcp`, `db`) with each one's ref name (`bot`, `computer`, …), `parseStorageSpec({ volume, owner_id })` (null for anything unknown, so a bad id never becomes a path), and `storageSpecHref({ spec, path })` → `/admin/files?volume=<v>&owner=<id>&path=<p>`, the one place a link is built.
- **Access:** `resolveStorageSpec` (`src/functions/backend/storage/resolve-storage-spec.ts`) is the only gate, called before any path exists: a bot through `grabUserBot`, a workspace through `grabUserWorkspace`, a computer through `grabUserComputer` (so a member sees only their grants), `media` only their own (super admins and admins anyone's), `mcp` for crew managers, `db` for the super admin alone. `grabStorageSpecs` lists the same folders for the picker, with `reachableBotsSql` (`src/utils/crew-access/reachable-bots-sql.ts`, tested beside the other crew SQL) so the picker never offers one the resolver refuses. - **Access:** `resolveStorageSpec` (`src/functions/backend/storage/resolve-storage-spec.ts`) is the only gate, called before any path exists: a bot through `grabUserBot`, a workspace through `grabUserWorkspace`, a computer through `grabUserComputer` (so a member sees only their grants), `media` only their own (super admins and admins anyone's), `mcp` for crew managers, `db` for the super admin alone. Inside a computer's workspace a member sees only `bots/<bot>` for the bots they reach, within it only their own `users/<them>`, and `workspaces/<id>` for theirs (`grabComputerFolderAccess` on the resolved spec, `computerPathAllowed` in `src/utils/`, applied by `canSeePath` in `listStorageEntries` and `grabStorageFile`); those id folders carry the bot's, user's or workspace's name (`computerFolderLabel`, `StorageEntry.label`). `grabStorageSpecs` lists the same folders for the picker, with `reachableBotsSql` (`src/utils/crew-access/reachable-bots-sql.ts`, tested beside the other crew SQL) so the picker never offers one the resolver refuses.
- **Paths:** `resolveStoragePath` (`src/functions/backend/storage/resolve-storage-path.ts`) applies the `resolveBotPath` rules (no absolute paths, `..`, NUL, too deep, symlink escape) to any volume root. - **Paths:** `resolveStoragePath` (`src/functions/backend/storage/resolve-storage-path.ts`) applies the `resolveBotPath` rules (no absolute paths, `..`, NUL, too deep (`AppData.StorageBrowserMaxDepth`), symlink escape) to any volume root.
- **Reading:** `listStorageEntries` (folders first, `AppData.StorageBrowserPageSize`, a missing folder shows empty, symlinks are skipped, `bot_files` rows joined by `(scope, owner_id, path)` so those files keep opening through `/media/<id>`) and `grabStorageFile` (a regular file under `AppData.StorageBrowserReadMaxBytes`). API in `src/pages/api/admin/storage/` (`list`, `file`, the latter with an ETag like `handleMediaServer`), zod in `src/schemas/storage-schema.ts`. - **Reading:** `listStorageEntries` (folders first, `AppData.StorageBrowserPageSize`, a missing folder shows empty, symlinks are skipped, `bot_files` rows joined by `(scope, owner_id, path)` so those files keep opening through `/media/<id>`) and `grabStorageFile` (a regular file under `AppData.StorageBrowserReadMaxBytes`). API in `src/pages/api/admin/storage/` (`list`, `file`, the latter with an ETag like `handleMediaServer`), zod in `src/schemas/storage-schema.ts`.
- **Page:** `src/pages/admin/files/index.server.ts` + `index.tsx` (in `SettingsShell`, `wide`), route-local code: `(sections)/files-browser.tsx` and `(sections)/files-pane.tsx` (state, `history.pushState`, so a folder link is shareable and the back button walks back up), `(partials)/files-volume-row.tsx`, `files-breadcrumb-row.tsx`, `files-search-input.tsx`, `storage-entry-row.tsx`, `(hooks)/use-storage-folder.ts` (the page's own copy first, the API on navigation). Read-only: deletion stays in the chat's files modal and the bot's tools. - **Page:** `src/pages/admin/files/index.server.ts` + `index.tsx` (in `SettingsShell`, `wide`), route-local code: `(sections)/files-browser.tsx` and `(sections)/files-pane.tsx` (state, `history.pushState`, so a folder link is shareable and the back button walks back up), `(partials)/files-volume-row.tsx`, `files-breadcrumb-row.tsx`, `files-search-input.tsx`, `storage-entry-row.tsx`, `(hooks)/use-storage-folder.ts` (the page's own copy first, the API on navigation). Read-only: deletion stays in the chat's files modal and the bot's tools.
- **Chips:** `[folder:…]` refs open it, and the bot files modal links into it ("Browse this folder in Files"). - **Chips:** `[folder:…]` refs open it, and the bot files modal links into it ("Browse this folder in Files").
@@ -228,7 +229,7 @@ Live routes under `/admin` (see `AdminSections`): `/admin/bots`, `/admin/workspa
- `/admin/logs` (live) - Activity, the audit log: super admins only (others are redirected; the popover's "Activity" item shows to super admins only). Rendered in `SettingsShell` with `wide` - `/admin/logs` (live) - Activity, the audit log: super admins only (others are redirected; the popover's "Activity" item shows to super admins only). Rendered in `SettingsShell` with `wide`
- `/admin/help` (live) - familiar problems and their fixes, one `SettingsGroup` per topic (`(data)/help-topics.ts`, markdown rendered with the chat's `MessageMarkdownComponents`, the topic id as the group's anchor). First topic: signing in to Google and other strict sites with Cookie-Editor over the Desktop view (`HelpSignInTopicId` in `src/dict/help-dict.ts`; `ask_user_to_take_over` points bots to it) - `/admin/help` (live) - familiar problems and their fixes, one `SettingsGroup` per topic (`(data)/help-topics.ts`, markdown rendered with the chat's `MessageMarkdownComponents`, the topic id as the group's anchor). First topic: signing in to Google and other strict sites with Cookie-Editor over the Desktop view (`HelpSignInTopicId` in `src/dict/help-dict.ts`; `ask_user_to_take_over` points bots to it)
- Routines, Files, Usage, Activity and Help are the `RailPages` (`src/dict/rail-pages-dict.ts`), opened from the rail's user popover (which lists them, then Settings and Sign out). On them the rail shows its own aside (`(partials)/rail-pages-list.tsx`): a back link to the bots, then those pages, like settings. A page carries `super_admin_only` or a `can_view` rule (Usage is `canManageCrew`), and `canSeeRailItem` (`src/utils/rbac/can-see-rail-item.ts`, tested) decides both the popover and the aside, so a page is never offered in one and hidden in the other - Routines, Files, Usage, Activity and Help are the `RailPages` (`src/dict/rail-pages-dict.ts`), opened from the rail's user popover (which lists them, then Settings and Sign out). On them the rail shows its own aside (`(partials)/rail-pages-list.tsx`): a back link to the bots, then those pages, like settings. A page carries `super_admin_only` or a `can_view` rule (Usage is `canManageCrew`), and `canSeeRailItem` (`src/utils/rbac/can-see-rail-item.ts`, tested) decides both the popover and the aside, so a page is never offered in one and hidden in the other
- `/admin/settings/backups` - super admin only (listed in `SettingsPages`), two groups. Database snapshots: create, list, restore, download and delete in `db/.backups` (`bun-sqlite backup`); "Restore from file" beside "Create backup" opens `upload-restore-modal.tsx` (upload a SQLite database file, which is saved to `.backups` then restored with `bun-sqlite import <file>`, since `restore` is interactive). Whole-data archives: `write-data-backup-archive.ts` tars all of `DATA_DIR` (database, media, computer homes, bot and workspace files, MCP installs) into `BACKUP_ARCHIVE_DIR` as `cobble-data-<epoch-ms>.tar.gz` (via `tar -czf` into a `.part` name, renamed on success so a cut-off run is never listed, with a `--transform` putting every member under one root folder named after the archive), keeping the newest `AppData.DataBackupsKept` and never touching `db/.backups`; its own "Restore from file" uploads a tar.gz (`upload-restore-data.ts`, gzip header checked first). `restore-data-backup.ts` extracts an archive to a temp dir, requires exactly one root folder holding a `db` folder (`findArchiveRoot`, since an upload is saved under a fresh timestamp and its inner name no longer matches the file name), takes a safety archive before touching anything (`write-data-backup-archive.ts`, not the locked `create-data-backup.ts`), empties `DATA_DIR` (the snapshots included) and copies the archive's in, then `restart-app.ts` restarts the process through the docker socket after the response lands (`process.exit`, the container's restart policy brings it back), because the open SQLite handle points at the replaced file. Row actions are icon-only (`RowIconButton`, `src/components/general/row-icon-button.tsx`): restore, download, delete. `BACKUP_ARCHIVE_DIR` is a `ProcessOnlyEnvKeys` start-up path (the installer mounts `$COBBLE_HOME/backups-archives` there; outside docker it falls back to `~/.cobble/backups`). Every backup and restore, of either kind, runs one at a time via `with-backup-run-lock.ts`. Helpers in `src/functions/backend/backups/`, API routes in `api/admin/backups/` - `/admin/settings/backups` - super admin only (listed in `SettingsPages`), two groups. Database snapshots: create, list, restore, download and delete in `db/.backups` (`bun-sqlite backup`); "Restore from file" beside "Create backup" opens `upload-restore-modal.tsx` (upload a SQLite database file, which is saved to `.backups` then restored with `bun-sqlite import <file>`, since `restore` is interactive). Whole-data archives: `write-data-backup-archive.ts` tars all of `DATA_DIR` (database, media, computer workspaces, bot and workspace files, MCP installs; an older computer's moved-aside `computers/<id>-home-<ts>` is left out) into `BACKUP_ARCHIVE_DIR` as `cobble-data-<epoch-ms>.tar.gz` (via `tar -czf` into a `.part` name, renamed on success so a cut-off run is never listed, with a `--transform` putting every member under one root folder named after the archive), keeping the newest `AppData.DataBackupsKept` and never touching `db/.backups`; its own "Restore from file" uploads a tar.gz (`upload-restore-data.ts`, gzip header checked first). `restore-data-backup.ts` extracts an archive to a temp dir, requires exactly one root folder holding a `db` folder (`findArchiveRoot`, since an upload is saved under a fresh timestamp and its inner name no longer matches the file name), takes a safety archive before touching anything (`write-data-backup-archive.ts`, not the locked `create-data-backup.ts`), empties `DATA_DIR` (the snapshots included) and copies the archive's in, then `restart-app.ts` restarts the process through the docker socket after the response lands (`process.exit`, the container's restart policy brings it back), because the open SQLite handle points at the replaced file. Row actions are icon-only (`RowIconButton`, `src/components/general/row-icon-button.tsx`): restore, download, delete. `BACKUP_ARCHIVE_DIR` is a `ProcessOnlyEnvKeys` start-up path (the installer mounts `$COBBLE_HOME/backups-archives` there; outside docker it falls back to `~/.cobble/backups`). Every backup and restore, of either kind, runs one at a time via `with-backup-run-lock.ts`. Helpers in `src/functions/backend/backups/`, API routes in `api/admin/backups/`
- `/admin/settings/database` - super admin only (listed in `SettingsPages`), a read-only browser of the install's SQLite database, no writes anywhere in it. `grabDatabaseTables` reads the catalog once (every table and its columns, SQLite's own `sqlite_*` tables and sqlite-vec's `bot_file_vectors_*` shadow tables left out) and adds each table's `COUNT(*)` and its size from `dbstat` (index pages folded back onto their table). `grabTableRecords` then returns one page (`AppData.DatabaseBrowserPageSize`, 50) of the open table's records newest key first, every value already cut in SQL at `AppData.DatabaseBrowserCellMaxChars` (400) so a table of long texts never ships a megabyte of JSON, and a `truncated` flag per cell; the search matches over the concatenation of every column's text (`buildSearchClause`: every word, or the whole phrase), which is what also works on a virtual table, where `MATCH` and `LIKE` over a vector blob do not. A record's key is its single primary key column, else its rowid, else nothing (a virtual table with no primary key reads as plain cells); clicking a cell's row opens `grabTableRecord` for that key, which returns the record whole up to `AppData.DatabaseBrowserValueMaxChars` (20000), the answer to a cell cut at three lines. `clampPage` puts a page past the end on the last one. The page's state (table, search, mode, page) lives in the address through `databaseHref`, so a table or a search can be linked to; the open table's first page comes with the page props and anything else loads through the API. Helpers in `src/functions/backend/database/` (`build-table-catalog-sql`, `build-table-sizes-sql`, `build-records-sql`, `build-search-clause`, `resolve-table-key`, `build-record-row`, `quote-sql-identifier`), API `api/admin/database/records` and `record` (GET only, `grabSuperAdminUser`), zod in `src/schemas/database-schema.ts`, page under `src/pages/admin/settings/database/` - `/admin/settings/database` - super admin only (listed in `SettingsPages`), a read-only browser of the install's SQLite database, no writes anywhere in it. `grabDatabaseTables` reads the catalog once (every table and its columns, SQLite's own `sqlite_*` tables and sqlite-vec's `bot_file_vectors_*` shadow tables left out) and adds each table's `COUNT(*)` and its size from `dbstat` (index pages folded back onto their table). `grabTableRecords` then returns one page (`AppData.DatabaseBrowserPageSize`, 50) of the open table's records newest key first, every value already cut in SQL at `AppData.DatabaseBrowserCellMaxChars` (400) so a table of long texts never ships a megabyte of JSON, and a `truncated` flag per cell; the search matches over the concatenation of every column's text (`buildSearchClause`: every word, or the whole phrase), which is what also works on a virtual table, where `MATCH` and `LIKE` over a vector blob do not. A record's key is its single primary key column, else its rowid, else nothing (a virtual table with no primary key reads as plain cells); clicking a cell's row opens `grabTableRecord` for that key, which returns the record whole up to `AppData.DatabaseBrowserValueMaxChars` (20000), the answer to a cell cut at three lines. `clampPage` puts a page past the end on the last one. The page's state (table, search, mode, page) lives in the address through `databaseHref`, so a table or a search can be linked to; the open table's first page comes with the page props and anything else loads through the API. Helpers in `src/functions/backend/database/` (`build-table-catalog-sql`, `build-table-sizes-sql`, `build-records-sql`, `build-search-clause`, `resolve-table-key`, `build-record-row`, `quote-sql-identifier`), API `api/admin/database/records` and `record` (GET only, `grabSuperAdminUser`), zod in `src/schemas/database-schema.ts`, page under `src/pages/admin/settings/database/`
- `/admin/settings` (live) - redirects to `SettingsDefaultPath` (`/admin/settings/general`). Sub pages: `general/` (theme: System/Light/Dark per device via `useThemePreference`, stored in `localStorage[AppData.ThemeStorageKey]`, the same key `ThemeInitScript` and twui's `ColorSchemeSelector` read; applies instantly, no save bar; for super admins the prompt appendix (`prompt-section.tsx`, `system-prompt-append-form.tsx`) saving `api/admin/settings/system-prompt-append` into the `bot_system_prompt_append` app setting, and the run limits (`run-limits-section.tsx`, `run-limits-form.tsx`) saving `api/admin/settings/run-limits` into the `harness_max_tool_steps` and `bot_reply_max_tool_steps` app settings (`grabRunLimitsOverview`, `run_limits` page prop, 0 clearing a limit back to its code default); and, for `canManageCrew`, the bot file quota: `storage-section.tsx`, `storage-quota-form.tsx` saving `api/admin/settings/storage-quota` into `app_settings`; and, for everyone, the running version: `version-section.tsx`, `version-line.tsx` showing `pageProps.running_version` from `runningVersion` (`src/utils/running-version.ts`: `COBBLE_VERSION`, the release tag the image was built with, else `package.json`'s version, in development too, with `(development)` after it and no such suffix in production), `profile/` (current user profile), `notifications/` (this device's Web Push switch, a test send, and every device the user subscribed), `change-password/` (set/change own password) `models/` (super admin only, `super_admin_only` in the dict hides it from the rail: model providers and which model each harness uses) and `environment/` (super admin only: app settings and the envs bots and computers use, per scope), each wrapped in `SettingsShell` (`(partials)/settings-shell.tsx`: 64px pane header like the conversation header, then one centered column that opens with the breadcrumb trail, `src/components/general/admin-breadcrumbs.tsx`: twui's `Breadcrumbs` reading `Home`, the sections and the page off `pageProps.url.pathname`, the last crumb named by the page's own `title` so a record reads `Scout` rather than `3` and `/admin/logs` reads `Activity`; `hasBreadcrumbTrail` drops it only on the crew of bots, which is home, and nothing outside `/admin` gets one). Content is built from `SettingsGroup` blocks (label + description left, controls right, hairline-separated, no cards) and ends with the sticky `SettingsSaveBar` (Save enabled only when dirty, Discard, one-line status). Saves don't reload the page. Keep save notices in component state: `useFormInit` clears `status` whenever the form changes. `bots/` (the crew: chat, edit with `BotForm`, models, delete a worker via `api/admin/bots/delete-bot`), `workspaces/` (open, create through the Add flow, delete), `computers/` (state and resources, live; super admins create, edit with `AddComputerForm`, retry, delete), `mcps/`, `tools/` and `skills/` (see MCPs, custom tools and skills: super admins add, edit, refresh or try, delete; admins only set Access, `extension-access-modal.tsx`; add, edit, access and try open in modals (`AddModalShell`). A skill has no status: nothing connects or runs, so `skill-row.tsx` puts its slug, who gets it (`extension-access-label.ts`) and its description where a server's or tool's status line goes), each MCP server or tool row with a status line under it (`extension-status-line.tsx`: connected or not, last run worked or failed, when, and why), those five for `canManageCrew` only (`grab-crew-settings-server-props.ts`, `can_view` in the dict) and `users/` (for `canListUsers`: super admins and admins; the dict's `can_view` gates it in the rail; add with a password (`canCreateUser`), edit (`canUpdateUser`), both with the one `UserForm` (`admin/settings/users/(partials)/user-form/`, `useFormInit`, photo included, saved in one request), revoke (adds `revoked`, `canRevokeUser`), delete (`canRevokeUser`'s rules, never yourself), pick their user types from `UserTypes` (`user-types-select.tsx`: only the ones `filterAssignableUserTypes` allows, never `UneditableUserTypes`; at least one, since login needs a type; `saveUser` checks each grant and removal with `userTypeAssignment`; the super admin's types are never shown or changed there, and they can't be revoked; the password is entered twice, `SaveUserRequestSchema` checks they match), delete with everything theirs; `api/admin/users/save-user`, `delete-user`). The lists share `SettingsList`, `SettingsListRow`, `SettingsRowAction` and `use-settings-list`. Sub pages are declared once in `src/dict/settings-pages-dict.ts`; on every `/admin/settings/*` page the rail swaps the crew for a back link plus that list. Add a sub page by adding its folder and a dict entry - `/admin/settings` (live) - redirects to `SettingsDefaultPath` (`/admin/settings/general`). Sub pages: `general/` (theme: System/Light/Dark per device via `useThemePreference`, stored in `localStorage[AppData.ThemeStorageKey]`, the same key `ThemeInitScript` and twui's `ColorSchemeSelector` read; applies instantly, no save bar; for super admins the prompt appendix (`prompt-section.tsx`, `system-prompt-append-form.tsx`) saving `api/admin/settings/system-prompt-append` into the `bot_system_prompt_append` app setting, and the run limits (`run-limits-section.tsx`, `run-limits-form.tsx`) saving `api/admin/settings/run-limits` into the `harness_max_tool_steps` and `bot_reply_max_tool_steps` app settings (`grabRunLimitsOverview`, `run_limits` page prop, 0 clearing a limit back to its code default); and, for `canManageCrew`, the bot file quota: `storage-section.tsx`, `storage-quota-form.tsx` saving `api/admin/settings/storage-quota` into `app_settings`; and, for everyone, the running version: `version-section.tsx`, `version-line.tsx` showing `pageProps.running_version` from `runningVersion` (`src/utils/running-version.ts`: `COBBLE_VERSION`, the release tag the image was built with, else `package.json`'s version, in development too, with `(development)` after it and no such suffix in production), `profile/` (current user profile), `notifications/` (this device's Web Push switch, a test send, and every device the user subscribed), `change-password/` (set/change own password) `models/` (super admin only, `super_admin_only` in the dict hides it from the rail: model providers and which model each harness uses) and `environment/` (super admin only: app settings and the envs bots and computers use, per scope), each wrapped in `SettingsShell` (`(partials)/settings-shell.tsx`: 64px pane header like the conversation header, then one centered column that opens with the breadcrumb trail, `src/components/general/admin-breadcrumbs.tsx`: twui's `Breadcrumbs` reading `Home`, the sections and the page off `pageProps.url.pathname`, the last crumb named by the page's own `title` so a record reads `Scout` rather than `3` and `/admin/logs` reads `Activity`; `hasBreadcrumbTrail` drops it only on the crew of bots, which is home, and nothing outside `/admin` gets one). Content is built from `SettingsGroup` blocks (label + description left, controls right, hairline-separated, no cards) and ends with the sticky `SettingsSaveBar` (Save enabled only when dirty, Discard, one-line status). Saves don't reload the page. Keep save notices in component state: `useFormInit` clears `status` whenever the form changes. `bots/` (the crew: chat, edit with `BotForm`, models, delete a worker via `api/admin/bots/delete-bot`), `workspaces/` (open, create through the Add flow, delete), `computers/` (state and resources, live; super admins create, edit with `AddComputerForm`, retry, delete), `mcps/`, `tools/` and `skills/` (see MCPs, custom tools and skills: super admins add, edit, refresh or try, delete; admins only set Access, `extension-access-modal.tsx`; add, edit, access and try open in modals (`AddModalShell`). A skill has no status: nothing connects or runs, so `skill-row.tsx` puts its slug, who gets it (`extension-access-label.ts`) and its description where a server's or tool's status line goes), each MCP server or tool row with a status line under it (`extension-status-line.tsx`: connected or not, last run worked or failed, when, and why), those five for `canManageCrew` only (`grab-crew-settings-server-props.ts`, `can_view` in the dict) and `users/` (for `canListUsers`: super admins and admins; the dict's `can_view` gates it in the rail; add with a password (`canCreateUser`), edit (`canUpdateUser`), both with the one `UserForm` (`admin/settings/users/(partials)/user-form/`, `useFormInit`, photo included, saved in one request), revoke (adds `revoked`, `canRevokeUser`), delete (`canRevokeUser`'s rules, never yourself), pick their user types from `UserTypes` (`user-types-select.tsx`: only the ones `filterAssignableUserTypes` allows, never `UneditableUserTypes`; at least one, since login needs a type; `saveUser` checks each grant and removal with `userTypeAssignment`; the super admin's types are never shown or changed there, and they can't be revoked; the password is entered twice, `SaveUserRequestSchema` checks they match), delete with everything theirs; `api/admin/users/save-user`, `delete-user`). The lists share `SettingsList`, `SettingsListRow`, `SettingsRowAction` and `use-settings-list`. Sub pages are declared once in `src/dict/settings-pages-dict.ts`; on every `/admin/settings/*` page the rail swaps the crew for a back link plus that list. Add a sub page by adding its folder and a dict entry
+8 -1
View File
@@ -35,6 +35,7 @@ import sendMessageTool from "@/src/harness/tools/bot/send-message-tool";
import type { BotMessageWithAttachments } from "@/src/types"; import type { BotMessageWithAttachments } from "@/src/types";
import type { HarnessMessage } from "@/src/harness/types"; import type { HarnessMessage } from "@/src/harness/types";
import type { BotJobContext, BotJobOutcome } from "./types"; import type { BotJobContext, BotJobOutcome } from "./types";
import computerRunDirs from "@/src/utils/computer-work-dirs";
type Params = BotJobContext & { type Params = BotJobContext & {
bot: BUN_SQLITE_COBBLE_BOTS & { id: number }; bot: BUN_SQLITE_COBBLE_BOTS & { id: number };
@@ -169,10 +170,16 @@ export default async function runBotReply({
onPropose: ({ ref }) => proposals.push(ref), onPropose: ({ ref }) => proposals.push(ref),
}); });
/**
* Where the run works on a computer: the user's folder in the bot's.
*/
const dirs = computerRunDirs({ bot_id: bot.id, bot_kind: bot.kind, user_id, workspace_id });
const extensions = await openBotExtensions({ const extensions = await openBotExtensions({
bot_id: bot.id, bot_id: bot.id,
workspace_id, workspace_id,
computer_id: computers[0]?.id || null, computer_id: computers[0]?.id || null,
dirs,
taken: tools.map((tool) => tool.name), taken: tools.map((tool) => tool.name),
signal, signal,
}); });
@@ -202,7 +209,7 @@ export default async function runBotReply({
carry_over, carry_over,
chatReplyNote({ bot_id: bot.id }), chatReplyNote({ bot_id: bot.id }),
botFilesNote({ bot_id: bot.id }), botFilesNote({ bot_id: bot.id }),
ComputerFilesNote({ computer_id: computers[0]?.id || 0, can_import: false }), ComputerFilesNote({ computer_id: computers[0]?.id || 0, dirs, can_import: false }),
TasksNote, TasksNote,
TaskReminderNote, TaskReminderNote,
ActionCheckNote, ActionCheckNote,
+10 -2
View File
@@ -32,6 +32,7 @@ import jobSessionId from "@/src/functions/backend/db/bots-queue/job-session-id";
import withTakeOverRefs from "./with-take-over-refs"; import withTakeOverRefs from "./with-take-over-refs";
import ScheduledRunFailedError from "./scheduled-run-failed-error"; import ScheduledRunFailedError from "./scheduled-run-failed-error";
import type { BotJobContext, BotJobOutcome } from "./types"; import type { BotJobContext, BotJobOutcome } from "./types";
import computerRunDirs from "@/src/utils/computer-work-dirs";
type Params = BotJobContext & { type Params = BotJobContext & {
bot: Pick<BUN_SQLITE_COBBLE_BOTS, "kind"> & { id: number }; bot: Pick<BUN_SQLITE_COBBLE_BOTS, "kind"> & { id: number };
@@ -62,12 +63,17 @@ export default async function runBotTask({
report, report,
}: Params): Promise<BotJobOutcome> { }: Params): Promise<BotJobOutcome> {
const workspace_id = job.workspace_id || null; const workspace_id = job.workspace_id || null;
/**
* Where the run works on a computer: the user's folder in the bot's.
*/
const dirs = computerRunDirs({ bot_id: bot.id, bot_kind: bot.kind, user_id, workspace_id });
const computer = job.computer_id const computer = job.computer_id
? await openComputerTools({ ? await openComputerTools({
computer_id: job.computer_id, computer_id: job.computer_id,
bot_id: bot.id, bot_id: bot.id,
workspace_id, workspace_id,
dirs,
signal, signal,
}) })
: null; : null;
@@ -142,7 +148,7 @@ export default async function runBotTask({
: []), : []),
...(computers.length ...(computers.length
? [ ? [
searchComputersTool({ bot_id: bot.id, computers, workspace_id }), searchComputersTool({ bot_id: bot.id, computers, workspace_id, dirs }),
askToTakeOverTool({ askToTakeOverTool({
computers: take_over_computers, computers: take_over_computers,
in_task: true, in_task: true,
@@ -159,6 +165,7 @@ export default async function runBotTask({
scopes, scopes,
bot_id: bot.id, bot_id: bot.id,
user_id, user_id,
dirs,
}), }),
] ]
: []), : []),
@@ -176,6 +183,7 @@ export default async function runBotTask({
bot_id: bot.id, bot_id: bot.id,
workspace_id, workspace_id,
computer_id: job.computer_id || null, computer_id: job.computer_id || null,
dirs,
taken: tools.map((tool) => tool.name), taken: tools.map((tool) => tool.name),
signal, signal,
}); });
@@ -192,7 +200,7 @@ export default async function runBotTask({
TaskRunNote, TaskRunNote,
scheduled ? ScheduledRunNote : "", scheduled ? ScheduledRunNote : "",
botFilesNote({ bot_id: bot.id }), botFilesNote({ bot_id: bot.id }),
ComputerFilesNote({ computer_id: job.computer_id || 0 }), ComputerFilesNote({ computer_id: job.computer_id || 0, dirs }),
TasksNote, TasksNote,
ActionCheckNote, ActionCheckNote,
workspace_id ? WorkspaceLearningNote : "", workspace_id ? WorkspaceLearningNote : "",
+9 -4
View File
@@ -1,23 +1,26 @@
import bunext from "@moduletrace/bunext"; import bunext from "@moduletrace/bunext";
import BunSQLite from "@moduletrace/bun-sqlite"; import BunSQLite from "@moduletrace/bun-sqlite";
import type { BUN_SQLITE_COBBLE_CONTAINERS } from "@/db/types/db"; import type { BUN_SQLITE_COBBLE_CONTAINERS } from "@/db/types/db";
import migrateComputerLayout from "@/src/functions/backend/db/computers/migrate-computer-layout";
import refreshComputerContainer from "@/src/functions/backend/db/computers/refresh-computer-container"; import refreshComputerContainer from "@/src/functions/backend/db/computers/refresh-computer-container";
type Row = Pick< type Row = Pick<
BUN_SQLITE_COBBLE_CONTAINERS, BUN_SQLITE_COBBLE_CONTAINERS,
"id" | "docker_id" | "status" | "ip_address" "id" | "docker_id" | "status" | "ip_address" | "metadata"
> & { computer_id: number }; > & { computer_id: number };
/** /**
* Reads every computer container's state from docker and records it * Reads every computer container's state from docker and records it
* (`refreshComputerContainer`), telling owners about changes. Containers * (`refreshComputerContainer`), telling owners about changes. Containers
* still being set up, or that failed, are left alone. * still being set up, or that failed, are left alone. A container that
* still mounts its computer's whole home is recreated with only the
* workspace (`migrateComputerLayout`, once per computer).
*/ */
export default async function syncComputers() { export default async function syncComputers() {
const rows = const rows =
( (
await BunSQLite.sql<Row>({ await BunSQLite.sql<Row>({
sql: `SELECT co.id, co.docker_id, co.status, co.ip_address, c.id AS computer_id sql: `SELECT co.id, co.docker_id, co.status, co.ip_address, co.metadata, c.id AS computer_id
FROM containers co JOIN computers c ON c.container_id = co.id FROM containers co JOIN computers c ON c.container_id = co.id
WHERE co.docker_id IS NOT NULL AND co.status NOT IN ('creating', 'failed')`, WHERE co.docker_id IS NOT NULL AND co.status NOT IN ('creating', 'failed')`,
}) })
@@ -25,7 +28,9 @@ export default async function syncComputers() {
let changed = 0; let changed = 0;
for (const { computer_id, ...container } of rows) { for (const { computer_id, metadata, ...container } of rows) {
if (await migrateComputerLayout({ computer_id, metadata })) continue;
const refreshed = await refreshComputerContainer({ const refreshed = await refreshComputerContainer({
computer_id, computer_id,
container, container,
+31 -3
View File
@@ -96,12 +96,34 @@ export const AppData = {
ComputerImage: "ghcr.io/agent-infra/sandbox:latest", ComputerImage: "ghcr.io/agent-infra/sandbox:latest",
ComputerContainerPrefix: `${SiteData["SiteSlug"]}-computer`, ComputerContainerPrefix: `${SiteData["SiteSlug"]}-computer`,
/** /**
* A computer's home directory (`ComputerHomeDir` in its container) is * Only a computer's workspace (`ComputerWorkspaceDir` in its
* kept at `DATA_DIR/<ComputerDataDir>/<id>`, owned by the sandbox's * container) is kept, at `DATA_DIR/<ComputerDataDir>/<id>`, owned by
* user. It stays when the computer is deleted. * the sandbox's user. It stays when the computer is deleted. The rest
* of its home (`ComputerHomeDir`: dotfiles, browser profile, the env
* files) lives in the container and goes with it. Every bot works in
* `bots/<bot id>` there, a user's work in `bots/<bot id>/users/<user
* id>`, and a workspace's bots share `workspaces/<workspace id>`
* (`computerRunDirs`).
*/ */
ComputerDataDir: "computers", ComputerDataDir: "computers",
ComputerHomeDir: "/home/gem", ComputerHomeDir: "/home/gem",
ComputerWorkspaceDir: "/home/gem/workspace",
/**
* Written at the root of a computer's workspace once it uses that
* layout, so an older computer's whole home is told apart and moved
* aside (`prepareComputerDataDir`).
*/
ComputerWorkspaceMarker: ".cobble-workspace",
/**
* An older computer's whole home is moved aside to
* `<id><ComputerOldHomeSuffix><timestamp>` and kept, never archived.
*/
ComputerOldHomeSuffix: "-home-",
/**
* Recorded in a container's `metadata.layout` once it mounts only the
* workspace.
*/
ComputerWorkspaceLayout: "workspace",
ComputerUserUid: 1000, ComputerUserUid: 1000,
ComputerUserGid: 1000, ComputerUserGid: 1000,
/** /**
@@ -488,6 +510,12 @@ export const AppData = {
* Most folders deep a path may go, so nothing buries a file. * Most folders deep a path may go, so nothing buries a file.
*/ */
BotFilePathMaxDepth: 8, BotFilePathMaxDepth: 8,
/**
* Most folders deep the files browser opens: a computer's workspace
* starts four down (`bots/<bot>/users/<user>`) before a bot's own
* folders begin.
*/
StorageBrowserMaxDepth: 16,
/** /**
* The files modal and the bot's file list: one page of them. * The files modal and the bot's file list: one page of them.
*/ */
+10 -2
View File
@@ -6,6 +6,7 @@ import {
coordinatorReplyNote, coordinatorReplyNote,
TasksNote, TasksNote,
} from "./harness-prompts"; } from "./harness-prompts";
import computerRunDirs from "@/src/utils/computer-work-dirs";
/** /**
* Every note a bot reads names it by its own id, so a bot copying an * Every note a bot reads names it by its own id, so a bot copying an
@@ -46,8 +47,15 @@ describe("harness prompts", () => {
expect(botFilesNote({ bot_id: 5 })).toContain("[folder:bot:5:reports]"); expect(botFilesNote({ bot_id: 5 })).toContain("[folder:bot:5:reports]");
}); });
test("the computer files note keeps naming the computer it was given", () => { test("the computer files note names the computer and the run's folders", () => {
expect(ComputerFilesNote({ computer_id: 8 })).toContain("[folder:computer:8:seo/coderank/raw]"); const note = ComputerFilesNote({
computer_id: 8,
dirs: computerRunDirs({ bot_id: 3, user_id: 7, workspace_id: 2 }),
});
expect(note).toContain("[folder:computer:8:bots/3/users/7/seo/coderank/raw]");
expect(note).toContain("/home/gem/workspace/bots/3/users/7");
expect(note).toContain("/home/gem/workspace/workspaces/2");
}); });
test("notes with nothing bot-specific carry no ref example", () => { test("notes with nothing bot-specific carry no ref example", () => {
+16 -2
View File
@@ -1,6 +1,7 @@
import { AppData } from "@/src/data/app-data"; import { AppData } from "@/src/data/app-data";
import botRef from "@/src/utils/bot-ref"; import botRef from "@/src/utils/bot-ref";
import folderRef from "@/src/utils/folder-ref"; import folderRef from "@/src/utils/folder-ref";
import { computerWorkspaceRelative, type ComputerRunDirs } from "@/src/utils/computer-work-dirs";
/** /**
* Prompts the helper harnesses send. The text model gets their output in * Prompts the helper harnesses send. The text model gets their output in
@@ -82,15 +83,28 @@ You have a folder of your own, kept with the app (not on a computer), and it sta
*/ */
export const ComputerFilesNote = ({ export const ComputerFilesNote = ({
computer_id, computer_id,
dirs,
can_import = true, can_import = true,
}: { }: {
computer_id: number | string; computer_id: number | string;
/**
* The run's folders on the computer (`computerRunDirs`).
*/
dirs: ComputerRunDirs;
can_import?: boolean; can_import?: boolean;
}) => `# Files on the computer }) => `# Files on the computer
What you make on the computer (downloads, screenshots, a crawl, raw evidence, logs) lives in its home, not in your files. It survives restarts of the computer, but it is not searchable and it goes if the computer is rebuilt. You work only in your own folders on the computer, never elsewhere in it: every shell, code and file tool starts in ${dirs.cwd} (also ~), and a path outside your folders is refused.
- The user can open any of it: name a folder on the computer as a folder ref, e.g. ${folderRef({ volume: "computer", owner_id: computer_id, path: "seo/coderank/raw" })}. It opens that folder in Files, so it downloads nothing. Point at the folder holding the evidence they may want to see. - ${dirs.cwd}: this user's work. Everything for them goes here.
- ${dirs.bot_dir}: only what is the same for every user you work for (other users' folders in it are not yours).${
dirs.shared_dir ? `\n- ${dirs.shared_dir}: shared with every bot in this workspace.` : ""
}
- The browser downloads to its own folder: move what you download into your folder straight away.
What you make there (downloads, screenshots, a crawl, raw evidence, logs) is kept with the computer, not in your files, and it is not searchable.
- The user can open any of it: name a folder on the computer as a folder ref, its path from the computer's workspace (${AppData.ComputerWorkspaceDir}), e.g. ${folderRef({ volume: "computer", owner_id: computer_id, path: `${computerWorkspaceRelative({ absolute: dirs.cwd })}/seo/coderank/raw` })}. It opens that folder in Files, so it downloads nothing. Point at the folder holding the evidence they may want to see.
${ ${
can_import can_import
? `- import_file copies a file or a folder off the computer into your own files, in one go, and the copy is searchable straight away. Import the finished work (the report, the spreadsheet, the final document), attach it when it is what they asked for, and name it with its [file:ID] ref.\n- The raw evidence can stay on the computer: import it only when the user needs it kept, or when the computer may be rebuilt.\n` ? `- import_file copies a file or a folder off the computer into your own files, in one go, and the copy is searchable straight away. Import the finished work (the report, the spreadsheet, the final document), attach it when it is what they asked for, and name it with its [file:ID] ref.\n- The raw evidence can stay on the computer: import it only when the user needs it kept, or when the computer may be rebuilt.\n`
+3 -2
View File
@@ -1,7 +1,8 @@
/** /**
* A computer is an AIO sandbox running in its own docker container on the * A computer is an AIO sandbox running in its own docker container on the
* cobble network, created by the app (`createComputer`), its home * cobble network, created by the app (`createComputer`), only its
* directory kept under `DATA_DIR/computers/<id>`. * workspace (`AppData.ComputerWorkspaceDir`) kept under
* `DATA_DIR/computers/<id>`.
*/ */
export const ComputerKinds = [ export const ComputerKinds = [
{ {
+1 -1
View File
@@ -68,7 +68,7 @@ export const EnvDict: readonly KnownEnv[] = [
}, },
{ {
key: "MAIN_COMPUTER_DATA_DIR", key: "MAIN_COMPUTER_DATA_DIR",
description: `Where the main computer's home (/home/gem) is on this server's disk, for a computer the app didn't create itself (see MAIN_COMPUTER_URL), e.g. the bind mount your sandbox's compose file uses. Empty reads DATA_DIR/computers/<id>.`, description: `Where the main computer's workspace (/home/gem/workspace, the only folder bots work in) is on this server's disk, for a computer the app didn't create itself (see MAIN_COMPUTER_URL), e.g. the bind mount your sandbox's compose file uses. Empty reads DATA_DIR/computers/<id>.`,
required: false, required: false,
secret: false, secret: false,
}, },
@@ -37,6 +37,13 @@ export default async function writeDataBackupArchive(): Promise<CreatedDataBacku
// after it and an extract drops a single folder. // after it and an extract drops a single folder.
"--transform", "--transform",
`s,^\\.$,${archive_root},;s,^\\./,${archive_root}/,`, `s,^\\.$,${archive_root},;s,^\\./,${archive_root}/,`,
// A computer's old whole home, moved aside when only its
// workspace became the mount (`prepareComputerDataDir`),
// is kept on disk but never archived: only workspaces are.
// `*` stops at `/`, so a file inside a workspace whose name
// happens to hold the suffix is still archived.
"--no-wildcards-match-slash",
`--exclude=./${AppData.ComputerDataDir}/*${AppData.ComputerOldHomeSuffix}*`,
"-C", "-C",
DATA_DIR, DATA_DIR,
".", ".",
@@ -0,0 +1,36 @@
import parseJobJson from "@/src/bots-queue/parse-job-json";
import { AppData } from "@/src/data/app-data";
import logEvent from "@/src/functions/backend/logger";
import provisionComputer from "./provision-computer";
type Params = {
computer_id: number;
/**
* Its container's `metadata`.
*/
metadata?: string | null;
};
/**
* Recreates a computer whose container still mounts its whole home (set up
* before only `AppData.ComputerWorkspaceDir` was mounted), so its bots work
* in the workspace alone. `provisionComputer` moves the old home aside
* (`prepareComputerDataDir`) and records the layout, so this runs once per
* computer. Returns whether it did.
*/
export default async function migrateComputerLayout({ computer_id, metadata }: Params) {
const layout = parseJobJson({ text: metadata || "" })?.layout;
if (layout == AppData.ComputerWorkspaceLayout) return false;
await provisionComputer({ computer_id });
await logEvent({
title: `Computer #${computer_id} now mounts only its workspace`,
category: "system",
plain_text: `Recreated computer #${computer_id} so only ${AppData.ComputerWorkspaceDir} is kept; its old home was moved aside and kept on disk.`,
metadata: { computer_id },
});
return true;
}
@@ -1,4 +1,5 @@
import fs from "fs/promises"; import fs from "fs/promises";
import path from "path";
import { AppData } from "@/src/data/app-data"; import { AppData } from "@/src/data/app-data";
import grabComputerDataDir from "./grab-computer-data-dir"; import grabComputerDataDir from "./grab-computer-data-dir";
@@ -13,28 +14,45 @@ type Params = {
}; };
/** /**
* Creates a computer's home directory, owned by the sandbox's user so it * Creates a computer's workspace directory (mounted at
* can write there, and returns its path. Docker would otherwise create a * `AppData.ComputerWorkspaceDir`), with its `bots` and `workspaces`
* missing mount source owned by root. * folders, owned by the sandbox's user so it can write there, and returns
* its path. Docker would otherwise create a missing mount source owned by
* root.
*
* A directory without the layout marker (`AppData.ComputerWorkspaceMarker`)
* is a computer's whole home from before only the workspace was mounted:
* it is moved aside to `<id>-home-<timestamp>` and kept on disk (left out
* of data backups), so the computer starts with a clean workspace.
*/ */
export default async function prepareComputerDataDir({ computer_id, fresh }: Params) { export default async function prepareComputerDataDir({ computer_id, fresh }: Params) {
const dir = grabComputerDataDir({ computer_id }); const dir = grabComputerDataDir({ computer_id });
const marker = path.join(dir, AppData.ComputerWorkspaceMarker);
if (fresh) {
const existing = await fs.readdir(dir).catch(() => null); const existing = await fs.readdir(dir).catch(() => null);
if (existing?.length) { if (existing?.length) {
if (fresh) {
await fs.rename(dir, `${dir}-${Date.now()}`); await fs.rename(dir, `${dir}-${Date.now()}`);
} else if (!(await fs.stat(marker).catch(() => null))) {
await fs.rename(dir, `${dir}${AppData.ComputerOldHomeSuffix}${Date.now()}`);
} }
} }
await fs.mkdir(dir, { recursive: true }); for (const folder of [dir, path.join(dir, "bots"), path.join(dir, "workspaces")]) {
await fs.mkdir(folder, { recursive: true });
await chownToSandbox({ target: folder });
}
/** await fs.writeFile(marker, "", { flag: "a" });
* Only root can hand a directory to another user; a dev server await chownToSandbox({ target: marker });
* running as that user already owns it.
*/
await fs.chown(dir, AppData.ComputerUserUid, AppData.ComputerUserGid).catch(() => {});
return dir; return dir;
} }
/**
* Only root can hand a file to another user; a dev server running as that
* user already owns it.
*/
async function chownToSandbox({ target }: { target: string }) {
await fs.chown(target, AppData.ComputerUserUid, AppData.ComputerUserGid).catch(() => {});
}
@@ -10,18 +10,20 @@ import pushComputerUpdate from "./push-computer-update";
type Params = { type Params = {
computer_id: number; computer_id: number;
/** /**
* A brand new computer: start from an empty home directory. * A brand new computer: start from an empty workspace.
*/ */
fresh?: boolean; fresh?: boolean;
}; };
/** /**
* Sets up a computer's container: prepares its home directory, pulls the * Sets up a computer's container: prepares its workspace directory, pulls the
* sandbox image and starts the container with the computer's resources * sandbox image and starts the container with the computer's resources
* (`cpus`, `memory_mb`, else the `AppData` defaults), then records it (`containers`: * (`cpus`, `memory_mb`, else the `AppData` defaults), then records it (`containers`:
* docker id, state, and its IP address on the network). A failure is * docker id, state, and its IP address on the network). A failure is
* recorded as `failed` with its reason, which the computer lists show. The * recorded as `failed` with its reason, which the computer lists show. The
* owner's tabs hear about it either way. Never throws. * owner's tabs hear about it either way. The metadata's `layout` says the
* container mounts only the workspace (`syncComputers` recreates one that
* doesn't). Never throws.
*/ */
export default async function provisionComputer({ computer_id, fresh }: Params) { export default async function provisionComputer({ computer_id, fresh }: Params) {
const row = ( const row = (
@@ -65,7 +67,12 @@ export default async function provisionComputer({ computer_id, fresh }: Params)
created.ip_address, created.ip_address,
AppData.ComputerPort, AppData.ComputerPort,
created.status, created.status,
JSON.stringify({ ...kept, data_dir, host_dir: created.host_dir }), JSON.stringify({
...kept,
data_dir,
host_dir: created.host_dir,
layout: AppData.ComputerWorkspaceLayout,
}),
Date.now(), Date.now(),
Date.now(), Date.now(),
row.id, row.id,
@@ -1,16 +1,16 @@
import { describe, expect, test } from "bun:test"; import { describe, expect, test } from "bun:test";
import resolveComputerHomeDir from "./resolve-computer-home-dir"; import resolveComputerWorkspaceDir from "./resolve-computer-workspace-dir";
const data_dir = "/data"; const data_dir = "/data";
describe("resolveComputerHomeDir", () => { describe("resolveComputerWorkspaceDir", () => {
test("uses the directory the app bind-mounted", () => { test("uses the directory the app bind-mounted", () => {
/** /**
* The truth for a container the app provisioned: on a bind mount * The truth for a container the app provisioned: on a bind mount
* that can be anywhere on the docker host, not under DATA_DIR. * that can be anywhere on the docker host, not under DATA_DIR.
*/ */
expect( expect(
resolveComputerHomeDir({ resolveComputerWorkspaceDir({
computer_id: 2, computer_id: 2,
kind: "sandbox", kind: "sandbox",
metadata: JSON.stringify({ data_dir: "/srv/homes/2", host_dir: "/host/2" }), metadata: JSON.stringify({ data_dir: "/srv/homes/2", host_dir: "/host/2" }),
@@ -22,7 +22,7 @@ describe("resolveComputerHomeDir", () => {
test("falls back to MAIN_COMPUTER_DATA_DIR for the main computer", () => { test("falls back to MAIN_COMPUTER_DATA_DIR for the main computer", () => {
expect( expect(
resolveComputerHomeDir({ resolveComputerWorkspaceDir({
computer_id: 1, computer_id: 1,
kind: "main", kind: "main",
metadata: null, metadata: null,
@@ -34,7 +34,7 @@ describe("resolveComputerHomeDir", () => {
test("leaves a sandbox alone whatever the main setting says", () => { test("leaves a sandbox alone whatever the main setting says", () => {
expect( expect(
resolveComputerHomeDir({ resolveComputerWorkspaceDir({
computer_id: 3, computer_id: 3,
kind: "sandbox", kind: "sandbox",
metadata: null, metadata: null,
@@ -45,26 +45,26 @@ describe("resolveComputerHomeDir", () => {
}); });
test("uses the conventional folder when nothing is recorded", () => { test("uses the conventional folder when nothing is recorded", () => {
expect(resolveComputerHomeDir({ computer_id: 4, kind: "main", data_dir })).toBe( expect(resolveComputerWorkspaceDir({ computer_id: 4, kind: "main", data_dir })).toBe(
"/data/computers/4", "/data/computers/4",
); );
}); });
test("ignores an empty or unreadable record", () => { test("ignores an empty or unreadable record", () => {
expect( expect(
resolveComputerHomeDir({ computer_id: 5, metadata: JSON.stringify({ data_dir: " " }), data_dir }), resolveComputerWorkspaceDir({ computer_id: 5, metadata: JSON.stringify({ data_dir: " " }), data_dir }),
).toBe("/data/computers/5"); ).toBe("/data/computers/5");
expect(resolveComputerHomeDir({ computer_id: 5, metadata: "not json", data_dir })).toBe( expect(resolveComputerWorkspaceDir({ computer_id: 5, metadata: "not json", data_dir })).toBe(
"/data/computers/5", "/data/computers/5",
); );
expect(resolveComputerHomeDir({ computer_id: 5, metadata: null, data_dir })).toBe( expect(resolveComputerWorkspaceDir({ computer_id: 5, metadata: null, data_dir })).toBe(
"/data/computers/5", "/data/computers/5",
); );
}); });
test("normalizes a recorded relative path", () => { test("normalizes a recorded relative path", () => {
expect( expect(
resolveComputerHomeDir({ computer_id: 6, metadata: JSON.stringify({ data_dir: "/srv/a/../b" }), data_dir }), resolveComputerWorkspaceDir({ computer_id: 6, metadata: JSON.stringify({ data_dir: "/srv/a/../b" }), data_dir }),
).toBe("/srv/b"); ).toBe("/srv/b");
}); });
}); });
@@ -27,8 +27,9 @@ type Params = {
}; };
/** /**
* Where a computer's home (`AppData.ComputerHomeDir` in its container) * Where a computer's workspace (`AppData.ComputerWorkspaceDir` in its
* actually is on this app's disk, for the files browser and for anything * container, the only directory mounted from the data volume) actually is
* on this app's disk, for the files browser and for anything
* else that has to read a computer's files. * else that has to read a computer's files.
* *
* A container the app provisioned recorded the directory it bind-mounted * A container the app provisioned recorded the directory it bind-mounted
@@ -36,15 +37,16 @@ type Params = {
* that path can be anywhere the docker host allows. A computer the app * that path can be anywhere the docker host allows. A computer the app
* didn't provision has no record. That is the install's main computer on a * didn't provision has no record. That is the install's main computer on a
* server reached at `MAIN_COMPUTER_URL` (a dev server outside docker), * server reached at `MAIN_COMPUTER_URL` (a dev server outside docker),
* whose sandbox keeps its home wherever its compose file says, so * whose sandbox keeps its workspace wherever its compose file says, so
* `MAIN_COMPUTER_DATA_DIR` names that directory for this app. Without * `MAIN_COMPUTER_DATA_DIR` names that directory (the host side of
* `AppData.ComputerWorkspaceDir`) for this app. Without
* either, the conventional place every app-provisioned computer uses: * either, the conventional place every app-provisioned computer uses:
* `DATA_DIR/<AppData.ComputerDataDir>/<id>`. * `DATA_DIR/<AppData.ComputerDataDir>/<id>`.
* *
* Kept apart from the docker and database calls so the precedence is * Kept apart from the docker and database calls so the precedence is
* testable on its own. * testable on its own.
*/ */
export default function resolveComputerHomeDir({ export default function resolveComputerWorkspaceDir({
computer_id, computer_id,
kind, kind,
metadata, metadata,
@@ -19,7 +19,7 @@ type Params = {
*/ */
name: string; name: string;
/** /**
* The computer's home directory as this app sees it. * The computer's workspace directory as this app sees it.
*/ */
data_dir: string; data_dir: string;
/** /**
@@ -36,7 +36,7 @@ type Params = {
* Creates and starts (the API's `docker run`) an AIO sandbox container * Creates and starts (the API's `docker run`) an AIO sandbox container
* for a computer (`AppData.ComputerImage`), attached to the computers' * for a computer (`AppData.ComputerImage`), attached to the computers'
* network (created first when missing), with its home directory * network (created first when missing), with its home directory
* bind-mounted from `data_dir`, limited to `cpus` and `memory_mb`, with a * bind-mounted from `data_dir` (only its workspace, `AppData.ComputerWorkspaceDir`: the rest of its home lives and dies with the container), limited to `cpus` and `memory_mb`, with a
* health check (`AppData.ComputerHealthcheck`), and restarted with * health check (`AppData.ComputerHealthcheck`), and restarted with
* docker. A container left by an earlier try is stopped and replaced. * docker. A container left by an earlier try is stopped and replaced.
* Resolves once the health check passes, with its IP address on the network, which a dev * Resolves once the health check passes, with its IP address on the network, which a dev
@@ -78,7 +78,7 @@ export default async function createComputerContainer({
Retries: AppData.ComputerHealthcheck.Retries, Retries: AppData.ComputerHealthcheck.Retries,
}, },
HostConfig: { HostConfig: {
Binds: [`${host_dir}:${AppData.ComputerHomeDir}`], Binds: [`${host_dir}:${AppData.ComputerWorkspaceDir}`],
NetworkMode: network, NetworkMode: network,
RestartPolicy: { Name: "unless-stopped" }, RestartPolicy: { Name: "unless-stopped" },
SecurityOpt: ["seccomp=unconfined"], SecurityOpt: ["seccomp=unconfined"],
@@ -0,0 +1,87 @@
import BunSQLite from "@moduletrace/bun-sqlite";
import reachableBotsSql from "@/src/utils/crew-access/reachable-bots-sql";
import workspacesSql from "@/src/utils/crew-access/workspaces-sql";
import type { CrewRole } from "@/src/utils/rbac/crew-role";
type Params = {
role: CrewRole;
user_id: number | string;
};
export type ComputerFolderAccess = {
/**
* What `computerPathAllowed` needs about the user.
*/
manages_crew: boolean;
user_id: number;
bot_ids: number[];
workspace_ids: number[];
/**
* Names for the workspace's id folders (`bots/<id>`, `users/<id>`,
* `workspaces/<id>`), the ones the user may see.
*/
names: {
bots: Record<string, string>;
users: Record<string, string>;
workspaces: Record<string, string>;
};
};
/**
* Which of a computer's workspace folders a user may open in the files
* browser, and what they are called: the bots and workspaces they reach,
* by the same crew SQL as the rest of the app. Crew managers see every
* one, and every user's folder in them.
*/
export default async function grabComputerFolderAccess({
role,
user_id,
}: Params): Promise<ComputerFolderAccess> {
const manages_crew = role != "member";
const bot_access = reachableBotsSql({ role, user_id });
const workspace_access = workspacesSql({ role, user_id });
const [bots, workspaces, users] = await Promise.all([
rows<{ id: number; name?: string | null }>({
sql: `SELECT b.id, b.name FROM bots b WHERE ${bot_access.sql}`,
values: bot_access.values,
}),
rows<{ id: number; name?: string | null }>({
sql: `SELECT w.id, w.name FROM workspaces w WHERE ${workspace_access.sql}`,
values: workspace_access.values,
}),
rows<{ id: number; first_name?: string | null; last_name?: string | null; username?: string | null }>({
sql: `SELECT id, first_name, last_name, username FROM users${manages_crew ? "" : " WHERE id = ?"}`,
values: manages_crew ? [] : [Number(user_id)],
}),
]);
return {
manages_crew,
user_id: Number(user_id),
bot_ids: bots.map((bot) => Number(bot.id)),
workspace_ids: workspaces.map((workspace) => Number(workspace.id)),
names: {
bots: Object.fromEntries(bots.map((bot) => [String(bot.id), bot.name || `Bot ${bot.id}`])),
workspaces: Object.fromEntries(
workspaces.map((workspace) => [String(workspace.id), workspace.name || `Workspace ${workspace.id}`]),
),
users: Object.fromEntries(
users.map((user) => [
String(user.id),
[user.first_name, user.last_name].filter(Boolean).join(" ") || user.username || `User ${user.id}`,
]),
),
},
};
}
async function rows<T extends { [k: string]: any }>({
sql,
values,
}: {
sql: string;
values: (string | number)[];
}) {
return (await BunSQLite.sql<T>({ sql, values })).payload || [];
}
@@ -2,6 +2,7 @@ import { statSync } from "node:fs";
import path from "node:path"; import path from "node:path";
import { AppData } from "@/src/data/app-data"; import { AppData } from "@/src/data/app-data";
import { mimeTypeForFile } from "../db/bot-files/media-type-for-file"; import { mimeTypeForFile } from "../db/bot-files/media-type-for-file";
import { canSeePath } from "./list-storage-entries";
import resolveStoragePath from "./resolve-storage-path"; import resolveStoragePath from "./resolve-storage-path";
import type { ResolvedStorageSpec } from "./resolve-storage-spec"; import type { ResolvedStorageSpec } from "./resolve-storage-spec";
@@ -31,7 +32,8 @@ export type StorageFile = {
/** /**
* One file for the browser to open: checked to be inside the volume (the * One file for the browser to open: checked to be inside the volume (the
* same rules a bot's own path gets), to exist, and to be a regular file * same rules a bot's own path gets), to exist, and to be a regular file
* rather than a folder. Null when it isn't one, so a caller can answer * rather than a folder, and one the user may see in a computer's workspace
* (`canSeePath`). Null when it isn't one, so a caller can answer
* "not found" without leaking what is there. * "not found" without leaking what is there.
*/ */
export default function grabStorageFile({ resolved, path: requested }: Params): StorageFile | null { export default function grabStorageFile({ resolved, path: requested }: Params): StorageFile | null {
@@ -39,6 +41,8 @@ export default function grabStorageFile({ resolved, path: requested }: Params):
if (!resolved_path.name) return null; if (!resolved_path.name) return null;
if (!canSeePath({ resolved, path: resolved_path.relative })) return null;
let stats; let stats;
try { try {
@@ -4,6 +4,8 @@ import BunSQLite from "@moduletrace/bun-sqlite";
import type { BUN_SQLITE_COBBLE_BOT_FILES } from "@/db/types/db"; import type { BUN_SQLITE_COBBLE_BOT_FILES } from "@/db/types/db";
import { AppData } from "@/src/data/app-data"; import { AppData } from "@/src/data/app-data";
import grabMediaURL from "../media/grab-media-url"; import grabMediaURL from "../media/grab-media-url";
import computerFolderLabel from "@/src/utils/computer-folder-label";
import computerPathAllowed from "@/src/utils/computer-path-allowed";
import { isInsideRoot } from "./resolve-storage-path"; import { isInsideRoot } from "./resolve-storage-path";
import type { ResolvedStorageSpec } from "./resolve-storage-spec"; import type { ResolvedStorageSpec } from "./resolve-storage-spec";
import { import {
@@ -52,13 +54,20 @@ export type StorageEntry = {
* browser's own streaming route. * browser's own streaming route.
*/ */
url?: string; url?: string;
/**
* What an id folder in a computer's workspace is: the bot's, user's or
* workspace's name.
*/
label?: string;
}; };
/** /**
* One directory on the data volume, for the files browser: its entries * One directory on the data volume, for the files browser: its entries
* (folders first), a page of them, and a `bot_files` row attached to any * (folders first), a page of them, and a `bot_files` row attached to any
* that has one. Entries a symlink leads out of the volume with are left * that has one. Entries a symlink leads out of the volume with are left
* out, so nothing outside the folder is ever listed. * out, so nothing outside the folder is ever listed. In a computer's
* workspace only what the user may see is listed (`computerPathAllowed`),
* and a folder they may not see reads as missing.
*/ */
export default async function listStorageEntries({ export default async function listStorageEntries({
resolved, resolved,
@@ -74,7 +83,9 @@ export default async function listStorageEntries({
* all, which a stale folder link lands on (the page says so, instead of * all, which a stale folder link lands on (the page says so, instead of
* claiming the folder is empty). * claiming the folder is empty).
*/ */
if (!existsSync(absolute)) return { entries: [], total: 0, offset, missing: true }; if (!existsSync(absolute) || !canSeePath({ resolved, path: relative })) {
return { entries: [], total: 0, offset, missing: true };
}
const rows = readdirSync(absolute, { withFileTypes: true }); const rows = readdirSync(absolute, { withFileTypes: true });
const indexed = await botFileRows({ resolved, paths: rows.map((row) => joinPath({ relative, name: row.name })) }); const indexed = await botFileRows({ resolved, paths: rows.map((row) => joinPath({ relative, name: row.name })) });
@@ -86,6 +97,8 @@ export default async function listStorageEntries({
const entry_path = joinPath({ relative, name: row.name }); const entry_path = joinPath({ relative, name: row.name });
if (!canSeePath({ resolved, path: entry_path })) continue;
if (needle && !entry_path.toLowerCase().includes(needle)) continue; if (needle && !entry_path.toLowerCase().includes(needle)) continue;
const entry_absolute = path.join(absolute, row.name); const entry_absolute = path.join(absolute, row.name);
@@ -114,6 +127,9 @@ export default async function listStorageEntries({
size: 0, size: 0,
modified_at: Math.round(stats.mtimeMs), modified_at: Math.round(stats.mtimeMs),
url: storageSpecHref({ spec: resolved.spec, path: entry_path }), url: storageSpecHref({ spec: resolved.spec, path: entry_path }),
label: resolved.computer
? computerFolderLabel({ path: entry_path, names: resolved.computer.names }) || undefined
: undefined,
}); });
continue; continue;
} }
@@ -197,6 +213,14 @@ async function botFileRows({
return rows; return rows;
} }
/**
* Whether the user may see a path in the folder: anything in it, except
* inside a computer's workspace (`computerPathAllowed`).
*/
export function canSeePath({ resolved, path }: { resolved: ResolvedStorageSpec; path: string }) {
return !resolved.computer || computerPathAllowed({ ...resolved.computer, path });
}
function joinPath({ relative, name }: { relative: string; name: string }) { function joinPath({ relative, name }: { relative: string; name: string }) {
return relative ? `${relative}/${name}` : name; return relative ? `${relative}/${name}` : name;
} }
@@ -63,7 +63,7 @@ describe("resolveStoragePath", () => {
test("refuses a path that is too deep", () => { test("refuses a path that is too deep", () => {
expect(() => expect(() =>
resolveStoragePath({ root, path: Array.from({ length: 12 }, (_, i) => `f${i}`).join("/") }), resolveStoragePath({ root, path: Array.from({ length: 20 }, (_, i) => `f${i}`).join("/") }),
).toThrow(); ).toThrow();
}); });
}); });
@@ -78,9 +78,9 @@ export default function resolveStoragePath({
} }
} }
if (segments.length > AppData.BotFilePathMaxDepth) { if (segments.length > AppData.StorageBrowserMaxDepth) {
throw new AppError( throw new AppError(
`That path is too deep: at most ${AppData.BotFilePathMaxDepth} folders.`, `That path is too deep: at most ${AppData.StorageBrowserMaxDepth} folders.`,
); );
} }
@@ -8,9 +8,10 @@ import grabDirNames from "@/src/utils/grab-dir-names";
import type { CrewRole } from "@/src/utils/rbac/crew-role"; import type { CrewRole } from "@/src/utils/rbac/crew-role";
import { storageVolumeLabel, type StorageSpec } from "@/src/utils/storage-volumes"; import { storageVolumeLabel, type StorageSpec } from "@/src/utils/storage-volumes";
import grabCrewRole from "../db/access/grab-crew-role"; import grabCrewRole from "../db/access/grab-crew-role";
import grabComputerFolderAccess from "./grab-computer-folder-access";
import grabUserBot from "../db/bots/grab-user-bot"; import grabUserBot from "../db/bots/grab-user-bot";
import grabUserComputer from "../db/computers/grab-user-computer"; import grabUserComputer from "../db/computers/grab-user-computer";
import resolveComputerHomeDir from "../db/computers/resolve-computer-home-dir"; import resolveComputerWorkspaceDir from "../db/computers/resolve-computer-workspace-dir";
import grabEnv from "../db/envs/grab-env"; import grabEnv from "../db/envs/grab-env";
import grabUserWorkspace from "../db/workspaces/grab-user-workspace"; import grabUserWorkspace from "../db/workspaces/grab-user-workspace";
@@ -29,7 +30,9 @@ type Params = {
* crew uses (`grabUserBot`, `grabUserWorkspace`, `grabUserComputer`, so a * crew uses (`grabUserBot`, `grabUserWorkspace`, `grabUserComputer`, so a
* member sees only their own grants); their own media folder, and only * member sees only their own grants); their own media folder, and only
* their own whatever their role; an MCP server's install folder for crew * their own whatever their role; an MCP server's install folder for crew
* managers; the database for the super admin alone. * managers; the database for the super admin alone. Inside a computer's
* workspace a member sees only their bots' folders, their own user folder
* in each, and their workspaces' (`computerPathAllowed`).
*/ */
export default async function resolveStorageSpec({ user_id, spec }: Params) { export default async function resolveStorageSpec({ user_id, spec }: Params) {
const role = await grabCrewRole({ user_id }); const role = await grabCrewRole({ user_id });
@@ -41,6 +44,11 @@ export default async function resolveStorageSpec({ user_id, spec }: Params) {
spec, spec,
root, root,
label: await storageLabel({ spec, role, user_id }), label: await storageLabel({ spec, role, user_id }),
/**
* A computer's workspace: which of its folders the user may see,
* and their names (`computerPathAllowed`, `computerFolderLabel`).
*/
computer: spec.volume == "computers" ? await grabComputerFolderAccess({ role, user_id }) : null,
}; };
} }
@@ -76,7 +84,7 @@ async function storageRoot({
if (!computer) return null; if (!computer) return null;
return resolveComputerHomeDir({ return resolveComputerWorkspaceDir({
computer_id: spec.owner_id, computer_id: spec.owner_id,
kind: computer.kind, kind: computer.kind,
metadata: await computerContainerMetadata({ container_id: computer.container_id }), metadata: await computerContainerMetadata({ container_id: computer.container_id }),
+7 -1
View File
@@ -22,6 +22,7 @@ import updateWorkspaceTool from "./update-workspace-tool";
import grabUserWorkspaces from "@/src/functions/backend/db/workspaces/grab-user-workspaces"; import grabUserWorkspaces from "@/src/functions/backend/db/workspaces/grab-user-workspaces";
import setWorkspaceInstructionsTool from "./set-workspace-instructions-tool"; import setWorkspaceInstructionsTool from "./set-workspace-instructions-tool";
import startTaskTool from "./start-task-tool"; import startTaskTool from "./start-task-tool";
import computerRunDirs from "@/src/utils/computer-work-dirs";
type Params = { type Params = {
bot: BUN_SQLITE_COBBLE_BOTS; bot: BUN_SQLITE_COBBLE_BOTS;
@@ -109,7 +110,12 @@ export default async function grabBotChatTools({
}), }),
...(computers.length ...(computers.length
? [ ? [
searchComputersTool({ bot_id, computers, workspace_id }), searchComputersTool({
bot_id,
computers,
workspace_id,
dirs: computerRunDirs({ bot_id, bot_kind: bot.kind, user_id, workspace_id }),
}),
askToTakeOverTool({ computers, onAsk: (params) => onTakeOver?.(params) }), askToTakeOverTool({ computers, onAsk: (params) => onTakeOver?.(params) }),
] ]
: []), : []),
@@ -0,0 +1,17 @@
import type { ComputerRunDirs } from "@/src/utils/computer-work-dirs";
import { shellQuote } from "./download-computer-file";
/**
* The shell that puts a command in the run's folder on the computer:
* creates its folders (the user's in the bot's, and the workspace's shared
* one), moves into the user's and makes it `HOME`, so `~` and every
* relative path land there. Ends with `; `, ready to prefix a command.
*/
export default function computerDirsShell({ dirs }: { dirs: ComputerRunDirs }) {
const cwd = shellQuote({ text: dirs.cwd });
const folders = [dirs.cwd, ...(dirs.shared_dir ? [dirs.shared_dir] : [])]
.map((dir) => shellQuote({ text: dir }))
.join(" ");
return `mkdir -p ${folders} && cd ${cwd} && export HOME=${cwd}; `;
}
+35 -28
View File
@@ -7,6 +7,11 @@ import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
import recordBotFile, { grabBotFileRow } from "@/src/functions/backend/db/bot-files/record-bot-file"; import recordBotFile, { grabBotFileRow } from "@/src/functions/backend/db/bot-files/record-bot-file";
import grabComputerMcpUrl from "@/src/functions/backend/db/computers/grab-computer-mcp-url"; import grabComputerMcpUrl from "@/src/functions/backend/db/computers/grab-computer-mcp-url";
import computerNotReadyMessage from "@/src/utils/computer-not-ready-message"; import computerNotReadyMessage from "@/src/utils/computer-not-ready-message";
import {
computerWorkspaceRelative,
resolveComputerPath,
type ComputerRunDirs,
} from "@/src/utils/computer-work-dirs";
import folderRef from "@/src/utils/folder-ref"; import folderRef from "@/src/utils/folder-ref";
import type { StorageScope } from "@/src/utils/storage-access"; import type { StorageScope } from "@/src/utils/storage-access";
import type { HarnessTool } from "../../types"; import type { HarnessTool } from "../../types";
@@ -16,7 +21,7 @@ import walkRegularFiles from "@/src/utils/walk-regular-files";
type Params = { type Params = {
/** /**
* The computer the run is on: what it made is in its home. * The computer the run is on: what it made is in its workspace.
*/ */
computer_id: number | string; computer_id: number | string;
/** /**
@@ -26,6 +31,11 @@ type Params = {
scopes: StorageScope[]; scopes: StorageScope[];
bot_id: number | string; bot_id: number | string;
user_id: number | string; user_id: number | string;
/**
* The run's folders on the computer (`computerRunDirs`): only they are
* imported from, a relative path from the user's.
*/
dirs: ComputerRunDirs;
onAttach?: (params: { media_id: number; name: string }) => void; onAttach?: (params: { media_id: number; name: string }) => void;
}; };
@@ -52,6 +62,7 @@ export default function importFileTool({
scopes, scopes,
bot_id, bot_id,
user_id, user_id,
dirs,
onAttach, onAttach,
}: Params): HarnessTool<Args> { }: Params): HarnessTool<Args> {
const own = scopes[0]; const own = scopes[0];
@@ -59,7 +70,7 @@ export default function importFileTool({
return { return {
name: "import_file", name: "import_file",
description: `Copy a file or folder from this computer into your own files, so it is kept with the app and stays after the computer is rebuilt. Give its path on the computer, e.g. "seo/coderank/raw/crawl.json" or "seo/coderank/raw"; a folder comes in as it is, in one go. Up to ${max_mb} MB. Check with list_files and the computer's own tools what you already imported: importing the same thing twice just writes over the copy, so do it once.${ description: `Copy a file or folder from this computer into your own files, so it is kept with the app and stays after the computer is rebuilt. Give its path on the computer, relative to your folder there (${dirs.cwd}), e.g. "seo/coderank/raw/crawl.json" or "seo/coderank/raw"; a folder comes in as it is, in one go. Up to ${max_mb} MB. Check with list_files and the computer's own tools what you already imported: importing the same thing twice just writes over the copy, so do it once.${
own?.scope == "workspace" ? " The copies land in the workspace's shared files." : "" own?.scope == "workspace" ? " The copies land in the workspace's shared files." : ""
} Imports are searchable at once. Name a copy with its [file:ID] ref. Import the report you wrote before the last long steps of a run, so it is never lost to the step limit.`, } Imports are searchable at once. Name a copy with its [file:ID] ref. Import the report you wrote before the last long steps of a run, so it is never lost to the step limit.`,
schema: z.object({ schema: z.object({
@@ -67,7 +78,7 @@ export default function importFileTool({
.string() .string()
.min(1) .min(1)
.max(AppData.BotFilePathMaxChars) .max(AppData.BotFilePathMaxChars)
.describe(`The file or folder on the computer, relative to its home, e.g. "seo/coderank/raw"`), .describe(`The file or folder on the computer, relative to your folder there, e.g. "seo/coderank/raw"`),
name: z name: z
.string() .string()
.max(AppData.BotFilePathMaxChars) .max(AppData.BotFilePathMaxChars)
@@ -78,9 +89,15 @@ export default function importFileTool({
run: async ({ args, signal }) => { run: async ({ args, signal }) => {
if (!own) return `You have no files folder to import into.`; if (!own) return `You have no files folder to import into.`;
const source = normalizeSourcePath({ path: args.path }); /**
* Workspace-relative from here on, which is what a folder ref
* names.
*/
const source = computerWorkspaceRelative({
absolute: resolveComputerPath({ path: args.path, dirs }),
});
if (!source) return `Give a path inside the computer's home, e.g. "seo/coderank/raw".`; if (!source) return `Give a file or folder inside your folder, e.g. "seo/coderank/raw".`;
const target = await grabComputerMcpUrl({ computer_id }); const target = await grabComputerMcpUrl({ computer_id });
@@ -91,7 +108,7 @@ export default function importFileTool({
const kind = await sourceKind({ base_url: target.base_url, source, signal }); const kind = await sourceKind({ base_url: target.base_url, source, signal });
if (kind == "missing") { if (kind == "missing") {
return `Nothing at ~/${source} on this computer. Check it with search_computers or the computer's own tools before importing.`; return `Nothing at ${onComputer({ source })} on this computer. Check it with search_computers or the computer's own tools before importing.`;
} }
if (kind == "folder") { if (kind == "folder") {
@@ -149,7 +166,7 @@ async function importOneFile({
}) { }) {
const download = await downloadComputerFile({ const download = await downloadComputerFile({
base_url: target.base_url, base_url: target.base_url,
path: `${AppData.ComputerHomeDir}/${target.source}`, path: onComputer({ source: target.source }),
max_bytes: AppData.BotFileImportMaxBytes, max_bytes: AppData.BotFileImportMaxBytes,
signal, signal,
}); });
@@ -182,7 +199,7 @@ async function importOneFile({
return [ return [
previous?.id previous?.id
? `You already had ${saved.path} (same ref, no second copy): its bytes are now this file, ${formatSize({ bytes: download.size })} byte for byte.` ? `You already had ${saved.path} (same ref, no second copy): its bytes are now this file, ${formatSize({ bytes: download.size })} byte for byte.`
: `Copied ~/${target.source} into your files as ${saved.path} [file:${saved.id}], ${formatSize({ bytes: download.size })} byte for byte.`, : `Copied ${onComputer({ source: target.source })} into your files as ${saved.path} [file:${saved.id}], ${formatSize({ bytes: download.size })} byte for byte.`,
`The original is still on the computer, in ${folderRef({ `The original is still on the computer, in ${folderRef({
volume: "computer", volume: "computer",
owner_id: target.computer_id, owner_id: target.computer_id,
@@ -220,10 +237,10 @@ async function importFolder({
const saved = await unpackArchive({ archive: archive.bytes, folder: name.replace(/\/+$/, ""), scope, bot_id, user_id }); const saved = await unpackArchive({ archive: archive.bytes, folder: name.replace(/\/+$/, ""), scope, bot_id, user_id });
if (!saved.length) return `The folder ~/${target.source} is empty, so nothing was imported.`; if (!saved.length) return `The folder ${onComputer({ source: target.source })} is empty, so nothing was imported.`;
return [ return [
`Copied ${saved.length} file(s) from ~/${target.source} into your files under ${name}/:`, `Copied ${saved.length} file(s) from ${onComputer({ source: target.source })} into your files under ${name}/:`,
...saved.slice(0, 20).map((file) => `- ${file.path} [file:${file.id}]`), ...saved.slice(0, 20).map((file) => `- ${file.path} [file:${file.id}]`),
saved.length > 20 ? `(and ${saved.length - 20} more)` : "", saved.length > 20 ? `(and ${saved.length - 20} more)` : "",
`The originals are still on the computer, in ${folderRef({ `The originals are still on the computer, in ${folderRef({
@@ -261,7 +278,7 @@ async function sourceKind({
}): Promise<"file" | "folder" | "missing"> { }): Promise<"file" | "folder" | "missing"> {
const result = await runComputerCommand({ const result = await runComputerCommand({
base_url, base_url,
command: `d="$HOME"/${shellQuote({ text: source })}; if [ -d "$d" ]; then echo folder; elif [ -f "$d" ]; then echo file; else echo missing; fi`, command: `d=${shellQuote({ text: onComputer({ source }) })}; if [ -d "$d" ]; then echo folder; elif [ -f "$d" ]; then echo file; else echo missing; fi`,
timeout_ms: AppData.ComputerSearchTimeoutMs, timeout_ms: AppData.ComputerSearchTimeoutMs,
signal, signal,
}); });
@@ -287,14 +304,14 @@ function importFailure({
const limit = formatSize({ bytes: AppData.BotFileImportMaxBytes }); const limit = formatSize({ bytes: AppData.BotFileImportMaxBytes });
if (reason == "missing") { if (reason == "missing") {
return `Nothing at ~/${source} on this computer. Check it with search_computers or the computer's own tools before importing.`; return `Nothing at ${onComputer({ source })} on this computer. Check it with search_computers or the computer's own tools before importing.`;
} }
if (reason == "too_big") { if (reason == "too_big") {
return `~/${source} is ${formatSize({ bytes: size })}, over the ${limit} limit, so nothing was copied. Zip it on the computer first, or import the files that matter on their own.`; return `${onComputer({ source })} is ${formatSize({ bytes: size })}, over the ${limit} limit, so nothing was copied. Zip it on the computer first, or import the files that matter on their own.`;
} }
return `The copy of ~/${source} came back with ${formatSize({ bytes: size })} of its bytes, so it was thrown away rather than saved. Try again; if it keeps happening, the file is changing while it is read.`; return `The copy of ${onComputer({ source })} came back with ${formatSize({ bytes: size })} of its bytes, so it was thrown away rather than saved. Try again; if it keeps happening, the file is changing while it is read.`;
} }
/** /**
@@ -326,7 +343,7 @@ async function readFolderArchive({
const result = await runComputerCommand({ const result = await runComputerCommand({
base_url, base_url,
command: [ command: [
`d="$HOME"/${shellQuote({ text: source })}`, `d=${shellQuote({ text: onComputer({ source }) })}`,
`t=$(mktemp /tmp/cobble-import-XXXXXX.tgz)`, `t=$(mktemp /tmp/cobble-import-XXXXXX.tgz)`,
`tar -czf "$t" -C "$(dirname "$d")" "$(basename "$d")" || { rm -f "$t"; echo FAILED; exit 0; }`, `tar -czf "$t" -C "$(dirname "$d")" "$(basename "$d")" || { rm -f "$t"; echo FAILED; exit 0; }`,
`s=$(wc -c < "$t")`, `s=$(wc -c < "$t")`,
@@ -437,19 +454,9 @@ function archiveRoot({ dir }: { dir: string }) {
} }
/** /**
* The path the tool was given, made relative to the computer's home and * A workspace-relative path as it is on the computer.
* refused if it leaves it.
*/ */
function normalizeSourcePath({ path: requested }: { path: string }) { function onComputer({ source }: { source: string }) {
const clean = requested return `${AppData.ComputerWorkspaceDir}/${source}`;
.trim()
.replace(/^~\/?/, "")
.replace(/^\/+|\/+$/g, "");
if (!clean) return null;
if (clean.split("/").some((segment) => segment == ".." || segment == ".")) return null;
return clean;
} }
@@ -1,12 +1,17 @@
import { AppData } from "@/src/data/app-data";
import { ComputerMcpExcludedTools } from "@/src/dict/computer-dict"; import { ComputerMcpExcludedTools } from "@/src/dict/computer-dict";
import grabComputerMcpUrl from "@/src/functions/backend/db/computers/grab-computer-mcp-url"; import grabComputerMcpUrl from "@/src/functions/backend/db/computers/grab-computer-mcp-url";
import grabComputerEnvs from "@/src/functions/backend/db/envs/grab-computer-envs"; import grabComputerEnvs from "@/src/functions/backend/db/envs/grab-computer-envs";
import computerNotReadyMessage from "@/src/utils/computer-not-ready-message"; import computerNotReadyMessage from "@/src/utils/computer-not-ready-message";
import type { ComputerRunDirs } from "@/src/utils/computer-work-dirs";
import redactSecrets from "@/src/utils/redact-secrets"; import redactSecrets from "@/src/utils/redact-secrets";
import connectMcpServer from "../../mcp/connect-mcp-server"; import connectMcpServer from "../../mcp/connect-mcp-server";
import grabMcpTools from "../../mcp/grab-mcp-tools"; import grabMcpTools from "../../mcp/grab-mcp-tools";
import readImage from "../../media/read-image"; import readImage from "../../media/read-image";
import type { HarnessTool } from "../../types"; import type { HarnessTool } from "../../types";
import computerDirsShell from "./computer-dirs-shell";
import runComputerCommand from "./run-computer-command";
import withComputerDirs from "./with-computer-dirs";
import withComputerEnv from "./with-computer-env"; import withComputerEnv from "./with-computer-env";
import writeComputerEnv from "./write-computer-env"; import writeComputerEnv from "./write-computer-env";
@@ -21,6 +26,11 @@ type Params = {
* screenshots. * screenshots.
*/ */
workspace_id?: number | string | null; workspace_id?: number | string | null;
/**
* The run's folders on the computer (`computerRunDirs`): every tool is
* kept inside them.
*/
dirs: ComputerRunDirs;
signal?: AbortSignal; signal?: AbortSignal;
}; };
@@ -43,14 +53,16 @@ export type OpenComputer = {
* Connects to a computer (its AIO sandbox, over MCP) and returns its tools: * Connects to a computer (its AIO sandbox, over MCP) and returns its tools:
* shell, files, code and browser. Its envs (`grabComputerEnvs`) are written * shell, files, code and browser. Its envs (`grabComputerEnvs`) are written
* to the computer and loaded by the shell and code tools; secret values are * to the computer and loaded by the shell and code tools; secret values are
* redacted from everything the tools return. Screenshots come back as * redacted from everything the tools return. Every tool is kept in the
* readings from the media reading harness. Call `close` when the run ends. * run's folders (`withComputerDirs`), which are created first. Screenshots
* come back as readings from the media reading harness. Call `close` when the run ends.
* Throws when the computer can't be reached, which the queue retries. * Throws when the computer can't be reached, which the queue retries.
*/ */
export default async function openComputerTools({ export default async function openComputerTools({
computer_id, computer_id,
bot_id, bot_id,
workspace_id, workspace_id,
dirs,
signal, signal,
}: Params): Promise<OpenComputer> { }: Params): Promise<OpenComputer> {
const target = await grabComputerMcpUrl({ computer_id }); const target = await grabComputerMcpUrl({ computer_id });
@@ -65,6 +77,14 @@ export default async function openComputerTools({
const envs = await grabComputerEnvs({ computer_id, bot_id, workspace_id }); const envs = await grabComputerEnvs({ computer_id, bot_id, workspace_id });
const paths = await writeComputerEnv({ base_url: target.base_url, envs, signal }); const paths = await writeComputerEnv({ base_url: target.base_url, envs, signal });
await runComputerCommand({
base_url: target.base_url,
command: `${computerDirsShell({ dirs })}true`,
timeout_ms: AppData.McpConnectTimeoutMs,
signal,
});
const secrets = envs.filter((env) => env.secret); const secrets = envs.filter((env) => env.secret);
const redact = (text: string) => redactSecrets({ text, secrets }); const redact = (text: string) => redactSecrets({ text, secrets });
@@ -87,7 +107,7 @@ export default async function openComputerTools({
return { return {
name: target.name, name: target.name,
tools: tools.map((tool) => withComputerEnv({ tool, ...paths })), tools: tools.map((tool) => withComputerDirs({ tool: withComputerEnv({ tool, ...paths }), dirs })),
envs: envs.map((env) => ({ key: env.key, description: env.description })), envs: envs.map((env) => ({ key: env.key, description: env.description })),
redact, redact,
close: () => client.close().catch(() => {}), close: () => client.close().catch(() => {}),
@@ -5,8 +5,10 @@ import grabComputerMcpUrl from "@/src/functions/backend/db/computers/grab-comput
import grabComputerEnvs from "@/src/functions/backend/db/envs/grab-computer-envs"; import grabComputerEnvs from "@/src/functions/backend/db/envs/grab-computer-envs";
import computerRef from "@/src/utils/computer-ref"; import computerRef from "@/src/utils/computer-ref";
import computerNotReadyMessage from "@/src/utils/computer-not-ready-message"; import computerNotReadyMessage from "@/src/utils/computer-not-ready-message";
import type { ComputerRunDirs } from "@/src/utils/computer-work-dirs";
import redactSecrets from "@/src/utils/redact-secrets"; import redactSecrets from "@/src/utils/redact-secrets";
import type { HarnessTool } from "../../types"; import type { HarnessTool } from "../../types";
import computerDirsShell from "./computer-dirs-shell";
import runComputerCommand from "./run-computer-command"; import runComputerCommand from "./run-computer-command";
type Params = { type Params = {
@@ -22,6 +24,11 @@ type Params = {
* The bot's computers, the current one first (`grabBotComputers`). * The bot's computers, the current one first (`grabBotComputers`).
*/ */
computers: BUN_SQLITE_COBBLE_COMPUTERS[]; computers: BUN_SQLITE_COBBLE_COMPUTERS[];
/**
* The run's folders (`computerRunDirs`): the command starts in the
* user's.
*/
dirs: ComputerRunDirs;
}; };
type Args = { type Args = {
@@ -32,16 +39,18 @@ type Args = {
* Lets a bot look for something across every computer it has: one shell * Lets a bot look for something across every computer it has: one shell
* command runs on each at once (`runComputerCommand`), and the outputs come * command runs on each at once (`runComputerCommand`), and the outputs come
* back labelled with each computer's ref (`computerRef`), secrets redacted, * back labelled with each computer's ref (`computerRef`), secrets redacted,
* so the bot can tell the user exactly which computer holds what. * so the bot can tell the user exactly which computer holds what. It
* starts in the run's own folder on each (`computerDirsShell`).
*/ */
export default function searchComputersTool({ export default function searchComputersTool({
bot_id, bot_id,
workspace_id, workspace_id,
computers, computers,
dirs,
}: Params): HarnessTool<Args> { }: Params): HarnessTool<Args> {
return { return {
name: "search_computers", name: "search_computers",
description: `Run one read-only shell command (bash) on every computer you have at once, to find which one holds something: a file, a running app, a logged-in browser session, a setting. Each computer's output comes back under its ref, e.g. ${computerRef({ id: computers[0]?.id || 1 })}. For a logged-in site, look for its domain in the browser profiles' cookie databases (copy the file first, since the browser locks it) and list only host names, never cookie values. Don't change anything with it; use start_task for that. Output past ${AppData.ComputerSearchMaxOutputChars} characters per computer is cut, so filter in the command.`, description: `Run one read-only shell command (bash) on every computer you have at once, to find which one holds something: a file, a running app, a logged-in browser session, a setting. Each computer's output comes back under its ref, e.g. ${computerRef({ id: computers[0]?.id || 1 })}. For a logged-in site, look for its domain in the browser profiles' cookie databases (copy the file first, since the browser locks it) and list only host names, never cookie values. It starts in your folder on each computer (${dirs.cwd}, also ~). Don't change anything with it; use start_task for that. Output past ${AppData.ComputerSearchMaxOutputChars} characters per computer is cut, so filter in the command.`,
schema: z.object({ schema: z.object({
command: z.string().min(1).max(4000), command: z.string().min(1).max(4000),
}), }),
@@ -60,7 +69,7 @@ export default function searchComputersTool({
const [result, envs] = await Promise.all([ const [result, envs] = await Promise.all([
runComputerCommand({ runComputerCommand({
base_url: target.base_url, base_url: target.base_url,
command: args.command, command: `${computerDirsShell({ dirs })}${args.command}`,
timeout_ms: AppData.ComputerSearchTimeoutMs, timeout_ms: AppData.ComputerSearchTimeoutMs,
signal, signal,
}), }),
@@ -0,0 +1,71 @@
import { describe, expect, test } from "bun:test";
import computerRunDirs from "@/src/utils/computer-work-dirs";
import type { HarnessTool } from "../../types";
import computerDirsShell from "./computer-dirs-shell";
import withComputerDirs from "./with-computer-dirs";
const echoTool = (name: string): HarnessTool => ({
name,
description: "",
parameters: {},
run: async ({ args }) => JSON.stringify(args),
});
const dirs = computerRunDirs({ bot_id: 3, user_id: 7, workspace_id: 2 });
const cwd = "/home/gem/workspace/bots/3/users/7";
const signal = new AbortController().signal;
const call = async (name: string, args: Record<string, unknown>) =>
JSON.parse(await withComputerDirs({ tool: echoTool(name), dirs }).run({ args, signal }));
describe("withComputerDirs", () => {
test("bash runs in the user's folder with it as HOME", async () => {
const sent = await call("sandbox_execute_bash", { cmd: "ls" });
expect(sent.cwd).toBe(cwd);
expect(sent.cmd).toBe(`export HOME="${cwd}"; ls`);
});
test("bash keeps a cwd inside the run's folders and refuses one outside", async () => {
expect((await call("sandbox_execute_bash", { cmd: "ls", cwd: "seo" })).cwd).toBe(`${cwd}/seo`);
await expect(call("sandbox_execute_bash", { cmd: "ls", cwd: "/etc" })).rejects.toThrow();
});
test("code starts in the user's folder", async () => {
const python = await call("sandbox_execute_code", { code: "print(1)", language: "python" });
const js = await call("sandbox_execute_code", { code: "1", language: "javascript" });
expect(python.code).toContain(`_cobble_os.chdir("${cwd}")`);
expect(python.code).toEndWith("\nprint(1)");
expect(js.code).toContain(`process.chdir("${cwd}")`);
});
test("file tools resolve paths and refuse outside ones and sudo", async () => {
expect((await call("sandbox_file_operations", { action: "read", path: "a.txt" })).path).toBe(`${cwd}/a.txt`);
expect((await call("sandbox_str_replace_editor", { command: "view", path: "/home/gem/workspace/workspaces/2/x" })).path).toBe(
"/home/gem/workspace/workspaces/2/x",
);
await expect(call("sandbox_file_operations", { action: "read", path: "/home/gem/.bashrc" })).rejects.toThrow();
await expect(call("sandbox_file_operations", { action: "list", path: "a", sudo: true })).rejects.toThrow();
await expect(call("sandbox_str_replace_editor", { command: "view", path: "../../../4/x" })).rejects.toThrow();
});
test("file URLs must be in the run's folders, web ones pass", async () => {
expect((await call("browser_navigate", { url: "https://example.com" })).url).toBe("https://example.com");
expect((await call("browser_navigate", { url: `file://${cwd}/a.html` })).url).toBe(`file://${cwd}/a.html`);
await expect(call("browser_navigate", { url: "file:///etc/passwd" })).rejects.toThrow();
await expect(call("sandbox_convert_to_markdown", { uri: "file:///home/gem/.config/x" })).rejects.toThrow();
});
test("leaves other tools alone", () => {
const tool = echoTool("browser_click");
expect(withComputerDirs({ tool, dirs })).toBe(tool);
});
});
describe("computerDirsShell", () => {
test("creates the folders and moves in", () => {
expect(computerDirsShell({ dirs })).toBe(
`mkdir -p '${cwd}' '/home/gem/workspace/workspaces/2' && cd '${cwd}' && export HOME='${cwd}'; `,
);
});
});
@@ -0,0 +1,114 @@
import AppError from "@/src/utils/app-error";
import { resolveComputerPath, type ComputerRunDirs } from "@/src/utils/computer-work-dirs";
import type { HarnessTool } from "../../types";
type Params = {
tool: HarnessTool;
dirs: ComputerRunDirs;
};
/**
* Tools whose `path` argument names a file on the computer.
*/
const PathTools = ["sandbox_file_operations", "sandbox_str_replace_editor"];
/**
* Tools that open a URL, which may be a `file:` one.
*/
const UrlTools: Record<string, string> = {
browser_navigate: "url",
browser_new_tab: "url",
sandbox_convert_to_markdown: "uri",
};
/**
* Keeps a computer tool in the run's folders (`computerRunDirs`): bash
* runs in the user's folder with it as `HOME`, code starts there, a file
* tool's path is resolved against it and refused outside the run's folders
* (`resolveComputerPath`, its `sudo` too), and so is a `file:` URL. The
* model gets a refusal back as the tool's error; nothing reaches the
* computer. Other tools pass through.
*/
export default function withComputerDirs({ tool, dirs }: Params): HarnessTool {
if (tool.name == "sandbox_execute_bash") {
return {
...tool,
run: async ({ args, signal }) => {
const cwd = resolveComputerPath({ path: typeof args?.cwd == "string" ? args.cwd : "", dirs });
return tool.run({
args: {
...args,
cwd,
cmd: typeof args?.cmd == "string" ? `export HOME=${JSON.stringify(dirs.cwd)}; ${args.cmd}` : args?.cmd,
},
signal,
});
},
};
}
if (tool.name == "sandbox_execute_code") {
return {
...tool,
run: async ({ args, signal }) =>
tool.run({
args:
typeof args?.code == "string"
? { ...args, code: `${codePrefix({ language: args.language, cwd: dirs.cwd })}${args.code}` }
: args,
signal,
}),
};
}
if (PathTools.includes(tool.name)) {
return {
...tool,
run: async ({ args, signal }) => {
if (args?.sudo) throw new AppError(`sudo isn't available here: work in your own folders.`);
return tool.run({
args: { ...args, path: resolveComputerPath({ path: args?.path, dirs }) },
signal,
});
},
};
}
const url_key = UrlTools[tool.name];
if (url_key) {
return {
...tool,
run: async ({ args, signal }) => {
const url = typeof args?.[url_key] == "string" ? args[url_key].trim() : "";
if (/^file:/i.test(url)) {
const file_path = decodeURIComponent(url.replace(/^file:(\/\/[^/]*)?/i, ""));
resolveComputerPath({ path: file_path.startsWith("/") ? file_path : `/${file_path}`, dirs });
}
return tool.run({ args, signal });
},
};
}
return tool;
}
function codePrefix({ language, cwd }: { language?: unknown; cwd: string }) {
const dir = JSON.stringify(cwd);
const lang = typeof language == "string" ? language.toLowerCase() : "python";
if (lang.startsWith("py")) {
return `import os as _cobble_os; _cobble_os.makedirs(${dir}, exist_ok=True); _cobble_os.chdir(${dir}); _cobble_os.environ["HOME"] = ${dir}\n`;
}
if (lang.startsWith("js") || lang.startsWith("javascript") || lang.startsWith("node")) {
return `try { require("fs").mkdirSync(${dir}, { recursive: true }); process.chdir(${dir}); process.env.HOME = ${dir}; } catch {}\n`;
}
return "";
}
@@ -6,6 +6,7 @@ import checkRequiredArgs from "./check-required-args";
import runHttpTool from "./run-http-tool"; import runHttpTool from "./run-http-tool";
import runPromptTool from "./run-prompt-tool"; import runPromptTool from "./run-prompt-tool";
import runScriptTool from "./run-script-tool"; import runScriptTool from "./run-script-tool";
import type { ComputerRunDirs } from "@/src/utils/computer-work-dirs";
type Params = { type Params = {
tool: CustomToolRecord; tool: CustomToolRecord;
@@ -20,6 +21,11 @@ type Params = {
computer_id?: number | string | null; computer_id?: number | string | null;
bot_id?: number | string; bot_id?: number | string;
workspace_id?: number | string | null; workspace_id?: number | string | null;
/**
* The run's folders on the computer (`computerRunDirs`), where a
* script runs. None for a try from the settings page.
*/
dirs?: ComputerRunDirs | null;
/** /**
* Saves how each run went on the tool. Off for a draft being tried. * Saves how each run went on the tool. Off for a draft being tried.
*/ */
@@ -40,6 +46,7 @@ export default function buildCustomTool({
computer_id, computer_id,
bot_id, bot_id,
workspace_id, workspace_id,
dirs,
record = true, record = true,
}: Params): HarnessTool<Record<string, unknown>> | null { }: Params): HarnessTool<Record<string, unknown>> | null {
if (tool.kind == "script" && !computer_id) return null; if (tool.kind == "script" && !computer_id) return null;
@@ -58,6 +65,7 @@ export default function buildCustomTool({
computer_id: computer_id!, computer_id: computer_id!,
bot_id, bot_id,
workspace_id, workspace_id,
dirs,
signal, signal,
}); });
case "prompt": case "prompt":
+12 -1
View File
@@ -4,7 +4,9 @@ import grabComputerMcpUrl from "@/src/functions/backend/db/computers/grab-comput
import grabComputerEnvs from "@/src/functions/backend/db/envs/grab-computer-envs"; import grabComputerEnvs from "@/src/functions/backend/db/envs/grab-computer-envs";
import type { CustomToolScriptConfig } from "@/src/types"; import type { CustomToolScriptConfig } from "@/src/types";
import computerNotReadyMessage from "@/src/utils/computer-not-ready-message"; import computerNotReadyMessage from "@/src/utils/computer-not-ready-message";
import type { ComputerRunDirs } from "@/src/utils/computer-work-dirs";
import redactSecrets from "@/src/utils/redact-secrets"; import redactSecrets from "@/src/utils/redact-secrets";
import computerDirsShell from "../computer/computer-dirs-shell";
import runComputerCommand from "../computer/run-computer-command"; import runComputerCommand from "../computer/run-computer-command";
import writeComputerEnv from "../computer/write-computer-env"; import writeComputerEnv from "../computer/write-computer-env";
@@ -20,6 +22,12 @@ type Params = {
* The workspace the run is in: its envs apply on the computer. * The workspace the run is in: its envs apply on the computer.
*/ */
workspace_id?: number | string | null; workspace_id?: number | string | null;
/**
* The run's folders on the computer (`computerRunDirs`): the script
* runs in the user's. A try from the settings page has no bot, so it
* runs at the workspace's root.
*/
dirs?: ComputerRunDirs | null;
signal: AbortSignal; signal: AbortSignal;
}; };
@@ -27,7 +35,8 @@ type Params = {
* Runs the tool's code on a computer over the sandbox's REST API, with the * Runs the tool's code on a computer over the sandbox's REST API, with the
* computer's envs set (`writeComputerEnv`) and the arguments as JSON in * computer's envs set (`writeComputerEnv`) and the arguments as JSON in
* `COBBLE_ARGS` and on stdin. Code and arguments travel base64-encoded, * `COBBLE_ARGS` and on stdin. Code and arguments travel base64-encoded,
* so nothing in them needs quoting. Returns what it printed (stdout and * so nothing in them needs quoting. It runs in the run's own folder
* (`computerDirsShell`). Returns what it printed (stdout and
* stderr), secrets redacted; a non-zero exit throws with it. * stderr), secrets redacted; a non-zero exit throws with it.
*/ */
export default async function runScriptTool({ export default async function runScriptTool({
@@ -36,6 +45,7 @@ export default async function runScriptTool({
computer_id, computer_id,
bot_id, bot_id,
workspace_id, workspace_id,
dirs,
signal, signal,
}: Params) { }: Params) {
const target = await grabComputerMcpUrl({ computer_id }); const target = await grabComputerMcpUrl({ computer_id });
@@ -61,6 +71,7 @@ export default async function runScriptTool({
`f=$(mktemp --suffix=.${language.extension})`, `f=$(mktemp --suffix=.${language.extension})`,
`echo ${code} | base64 -d > "$f"`, `echo ${code} | base64 -d > "$f"`,
`[ -f ${shell_path} ] && . ${shell_path}`, `[ -f ${shell_path} ] && . ${shell_path}`,
dirs ? computerDirsShell({ dirs }).replace(/; $/, "") : `cd ${AppData.ComputerWorkspaceDir}`,
`export COBBLE_ARGS="$(echo ${input} | base64 -d)"`, `export COBBLE_ARGS="$(echo ${input} | base64 -d)"`,
`printf '%s' "$COBBLE_ARGS" | ${language.interpreter} "$f" 2>&1`, `printf '%s' "$COBBLE_ARGS" | ${language.interpreter} "$f" 2>&1`,
`status=$?`, `status=$?`,
@@ -8,6 +8,7 @@ import openMcpServer from "../../mcp/open-mcp-server";
import readImage from "../../media/read-image"; import readImage from "../../media/read-image";
import type { HarnessTool } from "../../types"; import type { HarnessTool } from "../../types";
import buildCustomTool from "../custom/build-custom-tool"; import buildCustomTool from "../custom/build-custom-tool";
import type { ComputerRunDirs } from "@/src/utils/computer-work-dirs";
type Params = { type Params = {
bot_id: number | string; bot_id: number | string;
@@ -17,6 +18,11 @@ type Params = {
* current one. None leaves script tools out. * current one. None leaves script tools out.
*/ */
computer_id?: number | string | null; computer_id?: number | string | null;
/**
* The run's folders on that computer (`computerRunDirs`), where its
* scripts run.
*/
dirs: ComputerRunDirs;
/** /**
* Names the run's built-in tools already use: they win. * Names the run's built-in tools already use: they win.
*/ */
@@ -50,6 +56,7 @@ export default async function openBotExtensions({
bot_id, bot_id,
workspace_id, workspace_id,
computer_id, computer_id,
dirs,
taken = [], taken = [],
signal, signal,
}: Params): Promise<OpenBotExtensions> { }: Params): Promise<OpenBotExtensions> {
@@ -74,7 +81,7 @@ export default async function openBotExtensions({
}; };
for (const tool of tools) { for (const tool of tools) {
add(buildCustomTool({ tool, envs, computer_id, bot_id, workspace_id })); add(buildCustomTool({ tool, envs, computer_id, bot_id, workspace_id, dirs }));
} }
for (const server of servers) { for (const server of servers) {
@@ -64,6 +64,7 @@ export default function StorageEntryRow({ entry, selected, onOpenFolder }: Props
<Span className="text-[12px] leading-4 text-ink-secondary"> <Span className="text-[12px] leading-4 text-ink-secondary">
{[ {[
is_folder ? "Folder" : formatBytes({ bytes: entry.size }), is_folder ? "Folder" : formatBytes({ bytes: entry.size }),
entry.label || "",
entry.modified_at ? relativeTime({ at: entry.modified_at }) : "", entry.modified_at ? relativeTime({ at: entry.modified_at }) : "",
entry.file_id ? "Saved by the bot" : "", entry.file_id ? "Saved by the bot" : "",
] ]
+1 -1
View File
@@ -67,7 +67,7 @@ Then tell the bot you're signed in, or click **Hand back** if it asked you to ta
### If it doesn't work ### If it doesn't work
- **The paste in step 7 or 8 is empty.** Use the computer's code editor instead: switch to **Code**, create a file such as \`cookies.json\`, and paste into it (that paste always works). In the computer's Chrome, open \`file:///home/gem/cookies.json\`, press Ctrl+A then Ctrl+C, and import as in step 8. **Delete the file afterwards.** - **The paste in step 7 or 8 is empty.** Use the computer's code editor instead: switch to **Code**, create a file such as \`workspace/cookies.json\`, and paste into it (that paste always works). In the computer's Chrome, open \`file:///home/gem/workspace/cookies.json\`, press Ctrl+A then Ctrl+C, and import as in step 8. **Delete the file afterwards.**
- **You're signed in, then signed out again soon after.** Google sometimes ends a session it sees used from a new place. Sign out and back in on your own browser, export again, and import straight away. Using the same account and keeping the bot's work regular helps the session last. - **You're signed in, then signed out again soon after.** Google sometimes ends a session it sees used from a new place. Sign out and back in on your own browser, export again, and import straight away. Using the same account and keeping the bot's work regular helps the session last.
- **Cookie-Editor's icon doesn't show.** You're in the **Browser** view, which shows pages only. Switch to **Desktop**. - **Cookie-Editor's icon doesn't show.** You're in the **Browser** view, which shows pages only. Switch to **Desktop**.
- **Another site still says no.** Make sure you exported while on that site (Cookie-Editor exports the cookies of the site you're on), and that you opened the same site on the computer before importing.`, - **Another site still says no.** Make sure you exported while on that site (Cookie-Editor exports the cookies of the site you're on), and that you opened the same site on the computer before importing.`,
+5
View File
@@ -879,6 +879,11 @@ export type StorageEntry = {
* streaming route. * streaming route.
*/ */
url?: string; url?: string;
/**
* What an id folder in a computer's workspace is: the bot's, user's or
* workspace's name.
*/
label?: string;
}; };
/** /**
+20
View File
@@ -0,0 +1,20 @@
import { describe, expect, test } from "bun:test";
import computerFolderLabel from "./computer-folder-label";
const names = { bots: { "3": "Scout" }, users: { "7": "Ada Lovelace" }, workspaces: { "2": "Launch" } };
describe("computerFolderLabel", () => {
test("names the id folders", () => {
expect(computerFolderLabel({ path: "bots/3", names })).toBe("Scout");
expect(computerFolderLabel({ path: "bots/3/users/7", names })).toBe("Ada Lovelace");
expect(computerFolderLabel({ path: "workspaces/2", names })).toBe("Launch");
});
test("leaves everything else unnamed", () => {
expect(computerFolderLabel({ path: "bots", names })).toBe("");
expect(computerFolderLabel({ path: "bots/3/users", names })).toBe("");
expect(computerFolderLabel({ path: "bots/3/users/7/seo", names })).toBe("");
expect(computerFolderLabel({ path: "bots/9", names })).toBe("");
expect(computerFolderLabel({ path: "other/3", names })).toBe("");
});
});
+32
View File
@@ -0,0 +1,32 @@
type Params = {
/**
* A path inside a computer's workspace, e.g. `bots/3/users/7`.
*/
path: string;
names: {
bots: Record<string, string>;
users: Record<string, string>;
workspaces: Record<string, string>;
};
};
/**
* What an id folder in a computer's workspace is, for the files browser:
* `bots/3` is the bot's name, `bots/3/users/7` the user's, `workspaces/2`
* the workspace's. Empty for anything else.
*/
export default function computerFolderLabel({ path, names }: Params) {
const [top, owner, sub, sub_owner, ...rest] = path.split("/").filter(Boolean);
if (!owner || rest.length) return "";
if (top == "workspaces" && !sub) return names.workspaces[owner] || "";
if (top != "bots") return "";
if (!sub) return names.bots[owner] || "";
if (sub == "users" && sub_owner) return names.users[sub_owner] || "";
return "";
}
+30
View File
@@ -0,0 +1,30 @@
import { describe, expect, test } from "bun:test";
import computerPathAllowed from "./computer-path-allowed";
const member = { manages_crew: false, user_id: 7, bot_ids: [3], workspace_ids: [2] };
describe("computerPathAllowed", () => {
test("crew managers see everything", () => {
expect(computerPathAllowed({ ...member, manages_crew: true, path: "bots/9/users/1" })).toBe(true);
expect(computerPathAllowed({ ...member, manages_crew: true, path: "legacy" })).toBe(true);
});
test("a member sees their bots and their own user folder", () => {
expect(computerPathAllowed({ ...member, path: "" })).toBe(true);
expect(computerPathAllowed({ ...member, path: "bots" })).toBe(true);
expect(computerPathAllowed({ ...member, path: "bots/3" })).toBe(true);
expect(computerPathAllowed({ ...member, path: "bots/3/notes.md" })).toBe(true);
expect(computerPathAllowed({ ...member, path: "bots/3/users" })).toBe(true);
expect(computerPathAllowed({ ...member, path: "bots/3/users/7/a" })).toBe(true);
expect(computerPathAllowed({ ...member, path: "bots/3/users/8" })).toBe(false);
expect(computerPathAllowed({ ...member, path: "bots/4" })).toBe(false);
expect(computerPathAllowed({ ...member, path: "bots/03" })).toBe(false);
});
test("a member sees the workspaces they reach and nothing else at the root", () => {
expect(computerPathAllowed({ ...member, path: "workspaces/2/x" })).toBe(true);
expect(computerPathAllowed({ ...member, path: "workspaces/5" })).toBe(false);
expect(computerPathAllowed({ ...member, path: ".cobble-workspace" })).toBe(false);
expect(computerPathAllowed({ ...member, path: "other" })).toBe(false);
});
});
+58
View File
@@ -0,0 +1,58 @@
type Params = {
/**
* A path inside a computer's workspace, relative to its root (the files
* browser's), e.g. `bots/3/users/1/reports`. Empty for the root.
*/
path: string;
/**
* Super admins and admins (`canManageCrew`) see the whole workspace.
*/
manages_crew: boolean;
user_id: number | string;
/**
* The bots and workspaces the user reaches (`grabUserBot`,
* `grabUserWorkspace`).
*/
bot_ids: (number | string)[];
workspace_ids: (number | string)[];
};
/**
* Whether the files browser shows a path in a computer's workspace to a
* user. Crew managers see all of it. Anyone else sees a bot's folder only
* when they reach the bot, and within it only their own user folder
* (`bots/<bot>/users/<them>`), a workspace's shared folder only when they
* reach the workspace, and nothing else at the root.
*/
export default function computerPathAllowed({
path,
manages_crew,
user_id,
bot_ids,
workspace_ids,
}: Params) {
if (manages_crew) return true;
const segments = path.split("/").filter(Boolean);
const [top, owner, sub, sub_owner] = segments;
if (!top) return true;
if (top == "bots") {
if (!owner) return true;
if (!bot_ids.some((id) => String(Number(id)) == owner)) return false;
if (sub == "users" && sub_owner) return sub_owner == String(Number(user_id));
return true;
}
if (top == "workspaces") {
if (!owner) return true;
return workspace_ids.some((id) => String(Number(id)) == owner);
}
return false;
}
+75
View File
@@ -0,0 +1,75 @@
import { describe, expect, test } from "bun:test";
import computerRunDirs, { computerWorkspaceRelative, resolveComputerPath } from "./computer-work-dirs";
const W = "/home/gem/workspace";
describe("computerRunDirs", () => {
test("a worker gets its own folder, the user's inside it", () => {
const dirs = computerRunDirs({ bot_id: 3, user_id: 7 });
expect(dirs.cwd).toBe(`${W}/bots/3/users/7`);
expect(dirs.roots).toEqual([`${W}/bots/3`]);
expect(dirs.shared_dir).toBeNull();
});
test("in a workspace the shared folder joins", () => {
const dirs = computerRunDirs({ bot_id: 3, user_id: 7, workspace_id: 2 });
expect(dirs.roots).toEqual([`${W}/bots/3`, `${W}/workspaces/2`]);
});
test("the coordinator reaches the whole workspace", () => {
const dirs = computerRunDirs({ bot_id: 1, bot_kind: "coordinator", user_id: 1 });
expect(dirs.roots).toEqual([W]);
});
});
describe("resolveComputerPath", () => {
const dirs = computerRunDirs({ bot_id: 3, user_id: 7, workspace_id: 2 });
test("relative paths and ~ land in the user's folder", () => {
expect(resolveComputerPath({ path: "seo/raw", dirs })).toBe(`${W}/bots/3/users/7/seo/raw`);
expect(resolveComputerPath({ path: "~/a.txt", dirs })).toBe(`${W}/bots/3/users/7/a.txt`);
expect(resolveComputerPath({ path: "~", dirs })).toBe(`${W}/bots/3/users/7`);
expect(resolveComputerPath({ path: "", dirs })).toBe(`${W}/bots/3/users/7`);
});
test("absolute paths inside the roots pass", () => {
expect(resolveComputerPath({ path: `${W}/bots/3/notes.md`, dirs })).toBe(`${W}/bots/3/notes.md`);
expect(resolveComputerPath({ path: `${W}/workspaces/2/x/`, dirs })).toBe(`${W}/workspaces/2/x`);
});
test("anything outside is refused", () => {
for (const path of [
"/home/gem/x.txt",
"/etc/passwd",
"../../../4/a",
"../8/secret",
`${W}/bots/4`,
`${W}/bots/3/users/8/a`,
`${W}/bots/30`,
`${W}/workspaces/5`,
"~/../../../../x",
"a\0b",
]) {
expect(() => resolveComputerPath({ path, dirs })).toThrow();
}
});
test("the bot-wide folder and its users folder itself pass", () => {
expect(resolveComputerPath({ path: "..", dirs })).toBe(`${W}/bots/3/users`);
expect(resolveComputerPath({ path: "../..", dirs })).toBe(`${W}/bots/3`);
});
test("the coordinator reaches any bot's folder", () => {
const coordinator = computerRunDirs({ bot_id: 1, bot_kind: "coordinator", user_id: 1 });
expect(resolveComputerPath({ path: `${W}/bots/3/report.md`, dirs: coordinator })).toBe(`${W}/bots/3/report.md`);
expect(() => resolveComputerPath({ path: "/home/gem/.config", dirs: coordinator })).toThrow();
});
test("workspace-relative names", () => {
expect(computerWorkspaceRelative({ absolute: `${W}/bots/3/users/7/seo` })).toBe("bots/3/users/7/seo");
});
});
+151
View File
@@ -0,0 +1,151 @@
import path from "node:path";
import { AppData } from "@/src/data/app-data";
import AppError from "@/src/utils/app-error";
/**
* Where one run may work on a computer, every path absolute inside the
* computer's workspace (`AppData.ComputerWorkspaceDir`, the only directory
* mounted from the data volume).
*/
export type ComputerRunDirs = {
/**
* The computer's workspace root.
*/
workspace: string;
/**
* Where the run starts and where a relative path lands: the user's
* folder in the bot's (`bots/<bot>/users/<user>`).
*/
cwd: string;
/**
* The bot's own folder (`bots/<bot>`).
*/
bot_dir: string;
/**
* The workspace's shared folder (`workspaces/<id>`), in a workspace.
*/
shared_dir: string | null;
/**
* The folders the run may touch. The coordinator's is the whole
* workspace.
*/
roots: string[];
/**
* The user the run is for: another user's folder in a bot's is never
* the run's, even inside its roots.
*/
user_id: number;
};
type Params = {
bot_id: number | string;
/**
* `coordinator`: it reaches every bot's and every workspace's folders,
* like its files (`grabStorageScopes`).
*/
bot_kind?: string | null;
user_id: number | string;
workspace_id?: number | string | null;
};
/**
* The folders a run works in on a computer: the user's own inside the
* bot's (`bots/<bot>/users/<user>`, where it starts), the bot's own
* (`bots/<bot>`, for work that is the same for every user) and, in a
* workspace, the folder its bots share (`workspaces/<id>`).
*/
export default function computerRunDirs({
bot_id,
bot_kind,
user_id,
workspace_id,
}: Params): ComputerRunDirs {
const workspace = AppData.ComputerWorkspaceDir;
const bot_dir = path.posix.join(workspace, "bots", String(Number(bot_id)));
const cwd = path.posix.join(bot_dir, "users", String(Number(user_id)));
const shared_dir = Number(workspace_id) > 0
? path.posix.join(workspace, "workspaces", String(Number(workspace_id)))
: null;
return {
workspace,
cwd,
bot_dir,
shared_dir,
roots: bot_kind == "coordinator"
? [workspace]
: [bot_dir, ...(shared_dir ? [shared_dir] : [])],
user_id: Number(user_id),
};
}
/**
* Whether `child` is `parent` or inside it (both absolute, normalized).
*/
export function isInsideDir({ parent, child }: { parent: string; child: string }) {
return child == parent || child.startsWith(`${parent.replace(/\/+$/, "")}/`);
}
/**
* Turns a path a bot gave a computer tool into the absolute one it means,
* only when it is one of the run's: a relative path (or `~`, which the
* run's shells read as its folder) lands in `cwd`, an absolute one must be
* inside `roots`, and another user's folder in a bot's is refused. Throws
* an `AppError` the bot gets back as the tool's error.
*/
export function resolveComputerPath({
path: requested,
dirs,
}: {
path?: string | null;
dirs: ComputerRunDirs;
}) {
const raw = String(requested ?? "").trim().replace(/\\/g, "/");
if (raw.includes("\0") || raw.length > AppData.BotFilePathMaxChars * 2) {
throw new AppError(`That path isn't valid.`);
}
const home_relative = raw == "~" ? "" : raw.startsWith("~/") ? raw.slice(2) : null;
const absolute = path.posix.normalize(
home_relative != null
? path.posix.join(dirs.cwd, home_relative)
: raw.startsWith("/")
? raw
: path.posix.join(dirs.cwd, raw || "."),
).replace(/(.)\/+$/, "$1");
if (!isComputerPathAllowed({ absolute, dirs })) {
throw new AppError(
`${absolute} is outside your folders on this computer. Work in ${dirs.cwd} (this user's), ${dirs.roots.join(" or ")} for work every user shares.`,
);
}
return absolute;
}
/**
* Whether an absolute, normalized path is one the run may touch.
*/
function isComputerPathAllowed({ absolute, dirs }: { absolute: string; dirs: ComputerRunDirs }) {
if (!dirs.roots.some((root) => isInsideDir({ parent: root, child: absolute }))) return false;
const relative = path.posix.relative(dirs.workspace, absolute).split("/");
/**
* `bots/<bot>/users/<user>/...`: only the run's own user.
*/
if (relative[0] == "bots" && relative[2] == "users" && relative[3]) {
return Number(relative[3]) == dirs.user_id && String(dirs.user_id) == relative[3];
}
return true;
}
/**
* A path inside the computer's workspace as a folder ref or the files
* browser names it: relative to the workspace root, no leading slash.
*/
export function computerWorkspaceRelative({ absolute }: { absolute: string }) {
return path.posix.relative(AppData.ComputerWorkspaceDir, absolute);
}