Open computers from the chat, let bots ask for a take-over, show task progress

- Serve each computer to its owner under /computers/<id>/ (HTTP and
  websockets proxied to the AIO sandbox) and show it in a shared
  ComputerView with browser, terminal and code modes, opened from the
  computers modal or a take-over button in the thread.
- ask_user_to_take_over lets a bot hand a computer to the user for a
  login or captcha; the message gets a take-over button and needs-you,
  and the viewer hands it back with a chat message.
- While a task runs the thread shows "<bot> is working..." with the
  task's latest step; steps read as plain words, not tool names.
- A coordinator with no workers no longer claims to delegate, and bots
  describe started tasks in the first person.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

AGENTS.md

@@ -88,15 +88,17 @@ Inspect live data with `bun llm-code-assistant/query-db.ts "<sql>"` - a read-onl
 - **Routes:** `/admin/bots` ensures the coordinator and redirects to its chat (`/admin/bots/1` on a fresh install); `/admin/bots/[bot_id]` renders the conversation (header, thread and composer, the last two held to 760px; there is no right panel); `/admin/bots/[bot_id]/models` is a worker bot's own model settings (the coordinator's redirects to its chat). Both render in the full-bleed `src/layouts/bots/` layout (rail of conversations + user popover), which `__root.tsx` uses for every `/admin/*` page.
 - **Server:** `src/functions/backend/db/bots/` (`grab-bots`, `grab-bot-messages`, `grab-active-bot-task`, `grab-user-bot`, `mark-bot-read`, `insert-user-message`). `api/admin/bots/send-message.ts` posts a user message to a bot owned by the session user (zod: `src/schemas/bot-schema.ts`), queues the bot's reply (`enqueueBotReply`) and wakes the queue (`src/bots-queue/wake-bots-queue.ts`; never import `src/bots-queue/index.ts` from a page or API route, it pulls the bunext runtime into the page bundles). `upload-attachment.ts` stores one file for the next message (private `media` row, paradigm `bot-attachment`, type from `attachmentFileKind`, documents get `text_content`, limits in `AppData.BotAttachment*`) and `send-message` takes their `media_ids` (checked to be the user's own by `grabAttachableMedia`), linked in `bot_message_media`; messages carry `attachments` (`BotMessageWithAttachments`). `mark-read.ts` clears unread for a message that arrives in the open conversation; `save-harness-settings.ts` saves a worker bot's models.
 - **Message rendering:** bot messages are markdown, rendered with `react-markdown` + `remark-gfm` (raw HTML never rendered) through the twui-based map in `[bot_id]/(data)/message-markdown-components.tsx` (`markdown-link`, `markdown-list`, `markdown-code-block` partials). User messages stay plain text.
-- **Attachments and thinking:** the composer uploads files as they're picked, pasted or dropped (`use-composer-attachments`), shows them as chips, and sends once none is still uploading; the thread shows images as thumbnails and other files as chips. Replies read the files of the last `BotReplyAttachmentMessages` messages with any (`load-attachment-parts.ts`: images and audio as parts, documents as their text up to `BotAttachmentMaxTextChars`, videos as a note); older ones are only named. Tasks don't get the files yet. While a reply runs (face `working`, status `BotReplyJobTitle`) or waits for a model (`BotWaitingStatusLine`), `BotThinkingRow` sits under the last message. Live notices come from the process that ran the job at once, and from every other process through the live relay.
+- **Attachments and thinking:** the composer uploads files as they're picked, pasted or dropped (`use-composer-attachments`), shows them as chips, and sends once none is still uploading; the thread shows images as thumbnails and other files as chips. Replies read the files of the last `BotReplyAttachmentMessages` messages with any (`load-attachment-parts.ts`: images and audio as parts, documents as their text up to `BotAttachmentMaxTextChars`, videos as a note); older ones are only named. Tasks don't get the files yet. While a reply runs (face `working`, status `BotReplyJobTitle`) or waits for a model (`BotWaitingStatusLine`), `BotThinkingRow` sits under the last message; while a task runs it reads "<name> is working…" with the task's latest step. A coordinator with no workers gets `CoordinatorSoloNote` instead of `CoordinatorReplyNote`, so it never claims to have handed work to a worker. Live notices come from the process that ran the job at once, and from every other process through the live relay.
 - **Avatars:** `BotStone` (`src/components/general/bot-stone.tsx`) renders the stone inline from `src/data/stone-geometry.ts`, hue as a fill token, face (`idle|working|done|asleep|needs-you`) drawn as eyes. The coordinator is hue-less (ink body, paper eyes; inverts in dark). Never put a stone in a circle/badge or add outline/shadow. Wordmark is vector (`Wordmark`, `src/data/wordmark-paths.ts`), never live text.
 - **Copy helpers:** `src/utils/crew-status.ts` (`3 working · 1 needs you`, "Three bots are working. One needs you."), `src/utils/relative-time.ts` (relative timestamps only).
 - **Prompt and memory (self-learning):** a bot has `system_prompt` (core instructions) and a separate `memory` (markdown, what it must always remember). The bot rewrites both as it adapts to the user (`update-bot-system-prompt`, `update-bot-memory` in `src/functions/backend/db/bots/`; whole-text replace, capped at `AppData.PromptMaxLength`). `grab-bot-prompt` builds the full prompt via `composeBotPrompt` (`src/utils/compose-bot-prompt.ts`): core instructions, then each non-archived workspace's `system_prompt` under `# Workspace: <name>`, then `# Memory`.
-- **Queue:** every bot job goes through `bots_queue`, with `kind` (`BotJobKinds`): `reply` (answers the user in chat, no task row) or `task` (long-running work, typically on a computer). Long-running work goes through `bots_queue`, a bun-sqlite `isQueue` table worked with the native `BunSQLite.queue` API (`status` pending|processing|completed|failed, `attempts`, `available_at`, `locked_at`/`locked_by`, `last_error`; options in `AppData.BotQueue*`). Not every message needs a job: the bot replies right away that it's on it and `enqueueBotJob` queues the brief (`instructions`, `params`, optional `message_id`/`computer_id`, the computer checked against the bot's access; a `bot_tasks` row is created unless `task_id` is given). DB helpers in `src/functions/backend/db/bots-queue/`: `claimBotJobs`, `updateBotJob` (renews the lock, saves `progress`/`checkpoint`; false once lost or cancelled), `completeBotJob` (result message, task done, face done/needs-you), `failBotJob` (backoff retry, or `fatal`; out of attempts posts a needs-you message via `notifyBotJobFailed`), `cancelBotJob` (failed with `BotJobCancelledError`, `src/dict/bot-queue-dict.ts`), `releaseWorkerBotJobs`. Every model turn of a job is a step in `bots_queue_updates` (`kind` tool|note, one-line `text`, JSON `metadata`; tool names, never arguments; `describe-job-step.ts`), written through the job context's `report` (the text loop's `onStep`), which also sets `progress` and pushes `server:bot-job-update` (`BotJobUpdateData`). Steps are deleted with their job when it's pruned. The active task's latest `AppData.BotJobUpdatesShown` steps ride on `BotTaskWithSteps.updates` (`grabBotJobUpdates`); nothing in the UI shows them since the working-on panel was removed.
+- **Queue:** every bot job goes through `bots_queue`, with `kind` (`BotJobKinds`): `reply` (answers the user in chat, no task row) or `task` (long-running work, typically on a computer). Long-running work goes through `bots_queue`, a bun-sqlite `isQueue` table worked with the native `BunSQLite.queue` API (`status` pending|processing|completed|failed, `attempts`, `available_at`, `locked_at`/`locked_by`, `last_error`; options in `AppData.BotQueue*`). Not every message needs a job: the bot replies right away that it's on it and `enqueueBotJob` queues the brief (`instructions`, `params`, optional `message_id`/`computer_id`, the computer checked against the bot's access; a `bot_tasks` row is created unless `task_id` is given). DB helpers in `src/functions/backend/db/bots-queue/`: `claimBotJobs`, `updateBotJob` (renews the lock, saves `progress`/`checkpoint`; false once lost or cancelled), `completeBotJob` (result message, task done, face done/needs-you), `failBotJob` (backoff retry, or `fatal`; out of attempts posts a needs-you message via `notifyBotJobFailed`), `cancelBotJob` (failed with `BotJobCancelledError`, `src/dict/bot-queue-dict.ts`), `releaseWorkerBotJobs`. Every model turn of a job is a step in `bots_queue_updates` (`kind` tool|note, one-line `text`, JSON `metadata`; tool names, never arguments; `describe-job-step.ts`), written through the job context's `report` (the text loop's `onStep`), which also sets `progress` and pushes `server:bot-job-update` (`BotJobUpdateData`). Steps are deleted with their job when it's pruned. The active task's latest `AppData.BotJobUpdatesShown` steps ride on `BotTaskWithSteps.updates` (`grabBotJobUpdates`); the conversation (`bot_task` page prop, `use-task-progress`) shows the latest one under `BotThinkingRow`'s "<name> is working…" while a task runs. Step text is the first line the model wrote, else plain labels for its tools (`JobStepToolLabels` in `src/dict/bot-queue-dict.ts`).
 - **Queue handler:** `src/bots-queue/index.ts`, started from `server.ts` after cron and gated the same way. It releases its own cut-off jobs on startup (stable worker `AppData.BotQueueWorker`), then polls every `BotQueuePollMs`: reports jobs bun-sqlite failed on a lock timeout (`report-lost-bot-jobs`), claims per kind (`claimBotJobs` mirrors bun-sqlite's atomic claim plus `kind`, since `queue.claim` has no filter) up to `BotQueueReplyConcurrency` replies and `BotQueueConcurrency` tasks on separate slots, and runs each with `processBotJob` (heartbeat at a quarter of the lock timeout, abort on cancel/loss, one task per computer and one reply per bot via `createKeyLock`, face `working` once it starts, outcome pushed live). The actual work is `run-bot-job.ts` (`BotJobRunner`, context in `types.ts`), which dispatches to `run-bot-reply.ts` and `run-bot-task.ts`. Throw `BotJobFatalError` for failures retrying can't fix. When every model is resting (`HarnessError.resting`, nothing was tried) the job is put back with `deferBotJob` until the first comes back, its attempt refunded and the bot's face `asleep`; other retryable harness errors retry no sooner than their `retry_at`. The `prune-bots-queue` cron deletes finished jobs after `BotQueueRetentionDays`.
 - **Live updates:** server push only. `sendToUser` (`src/websocket/(functions)/send-to-user.ts`) sends to one user's sockets; `pushBotNotice` sends `server:bot-message` (the message row, user messages too, for other tabs) and `server:bot-updated` (`BotUpdatedData`: face, status line, active task). The bots layout opens one socket (twui `useWebSocket`); components listen with `useServerEvent` (`src/hooks/`), and `useLiveBots` keeps the rail and header current. The conversation dedupes messages by id. Every process also runs `liveRelay` (`src/live-relay/`): every `AppData.LiveRelayPollMs` it reads new `bot_messages`, new `bots_queue_updates` and changed bots from the database and pushes them to its own sockets, so work done by another process still shows live; clients drop what they already have. `server:bot-updated` carries the bot's `computer_id` when known, which keeps the header's computer in step across tabs. A rail row's subtitle is the bot's status line while it has one, else (coordinator) the crew counts while a worker is busy or needs you, else its last message as plain text (`src/utils/message-preview.ts`).
 - **Workspaces:** `workspaces` (per user, `name`, `description`, `system_prompt` shared by every bot in it, `archived`), with bots assigned through `workspace_bots` and computers through `workspace_computers`. Helpers in `src/functions/backend/db/workspaces/` (`create-workspace`, `update-workspace`, `add/remove-bot-*`, `add/remove-computer-*`), all ownership-checked against the session user.
 - **Computers:** a computer is an AIO sandbox in its own docker container on the `AppData.DockerNetworkName` network. `containers` records every docker container the app knows about except the main app container (`name` doubles as the hostname, plus `docker_id`, `image`, `network`, `port`, `status`, `metadata`). `computers` (per user, `kind` main|sandbox, `container_id`) and `bot_computers` (direct access). Each user has one main computer, recorded with the account (signup, `bun run seed`) and as a fallback by `/admin/bots` via `ensureMainComputer` (`src/functions/backend/db/computers/ensure-main-computer.ts`, idempotent), which also records its container (`AppData.MainComputerContainerName`, created by the installer) and grants the coordinator access. That one grant can't be removed (`isLockedComputerAccess`, enforced in `remove-bot-from-computer` and the `bot_computers` CRUD handler); every other grant can. A bot's effective computers (`grab-bot-computers`) are its direct grants plus every computer assigned to a non-archived workspace it's in (the coordinator's are all of the user's computers), the one it's on first: `bots.computer_id` (not a foreign key; null or a lost grant means its main one), so `computers[0]` is where its tasks run by default. The user switches it from the chat header (`(partials)/bot-computers-modal.tsx`, a ghost icon beside the model icon; the header names the current computer and the text model and provider under the bot's name) or by clicking a computer the bot named in a message: bots write `[computer:ID]` (`src/utils/computer-ref.ts`, turned into `#computer-ID` links by `link-computer-refs.ts`, rendered by `(partials)/computer-ref-chip.tsx`), both sharing `BotComputersContext` / `use-bot-computers`, saved by `api/admin/bots/set-computer.ts` (`setBotComputer`). Enums in `src/dict/computer-dict.ts`.
+- **Opening a computer:** the app serves each computer to its owner under `/computers/<id>/` (`ComputerProxyPrefix`): `src/server.ts` hands those paths to `handleComputerProxy` (`src/functions/server/computers/`: `userAuth`, ownership via `grabUserComputer`, then the sandbox's address from `grabComputerMcpUrl`), which fetches plain requests (app session cookies dropped, `X-Forwarded-Host/Proto/Prefix` set so the sandbox builds its links and CDP address under the prefix, framing headers and redirects fixed) and bridges websockets (`computerProxySocket`, dispatched from `src/websocket/` when `ws.data.proxy` is set). `ComputerView` (`src/components/general/computer-view.tsx`) shows a computer in one of `ComputerViewModes` (browser `/browser-ui`, terminal `/terminal`, code `/code-server/`; more of the sandbox's apps can be added there), keeping visited modes loaded. In the chat, `(partials)/computer-viewer-modal.tsx` shows it, opened from a computer card's open button in the computers modal or a take-over button (`openComputer` on `BotComputersContext`).
+- **Take-over:** a bot asks the user to take over a computer with `ask_user_to_take_over` (`src/harness/tools/computer/ask-to-take-over-tool.ts`, chat replies and tasks, defaulting to the task's computer). It writes `[take-over:ID:mode]` (`src/utils/take-over-ref.ts`; `withTakeOverRefs` adds it if the bot left it out and flags the message needs-you), linked by `link-computer-refs.ts` and rendered by `(partials)/take-over-chip.tsx`. The viewer then offers "Hand back to <bot>", which sends a chat message so the bot carries on; a task stops and reports instead of waiting.
 - **Not built yet:** add-bot flow, routines, help, task steps (nothing writes `bot_task_steps` yet).
 
 ### Envs
@@ -117,7 +119,7 @@ Envs are set in Settings, under Environment (super admin only), not in `.env`. T
 - **Automatic picks** (`rank-auto-models`): free models that can do the job (on the setting's provider when it has one; a provider without free models needs a picked model) (`model-serves-harness`: text needs tools), by vendor `AutoModelVendorPreference` (DeepSeek, Qwen, NVIDIA, Liquid), then other vendors that look competent (`AppData.HarnessAutoMin*`), then routers; within a vendor by intelligence index, then full-size before small, newest first. Transcription puts real speech-to-text models first.
 - **Calling:** `resolveHarnessCandidates` (pinned, then automatic) -> `withCandidates` tries them in order with process-wide cooldowns (`resolve/harness-cooldowns.ts`): rate limits and outages rest the model (until `Retry-After` when given), 401/402 rest the whole provider, 403/404 rest the model for an hour. Errors are `HarnessError` (`code` provider|unavailable|aborted, `retryable`), classified in `to-harness-error.ts`. Entry points: `text/generate-text.ts` (one turn), `text/run-text-loop.ts` (tool loop, sticks to one model, wraps up after `HarnessMaxToolSteps`, trims the oldest tool output before each turn to fit the model's context, `fit-transcript-to-context.ts`), `media/read-image.ts`, `media/read-media.ts` (documents locally via `fileToText`, images and audio through their harnesses), `audio/transcribe-audio.ts`, `audio/speak-text.ts`, `image/generate-image.ts`. A text model that can't take an image or audio part gets it read by the vision/transcription harness first (`prepare-messages-for-model.ts`).
 - **Per-bot models:** a worker bot inherits the defaults per harness unless it has a `bot_harness_settings` row (same fields as `harness_settings`, unique per bot and harness). `grabHarnessSetting({ harness, bot_id })` returns the bot's row (`source: "bot"`) or the default; the coordinator always gets the default. Every entry point takes `bot_id` and passes it on (text loop -> media reading/transcription), so a bot's run uses its own models throughout. Saved by `saveBotHarnessSetting` (`inherit` deletes the row); the page data is `grabBotHarnessOverview`. A bot's chat header has a ghost model icon (`(partials)/header-icon-button.tsx`; the text model and provider show under the bot's name) that opens a modal with every harness (`(partials)/bot-models-modal.tsx`); the modal and `/admin/bots/[bot_id]/models` both use `(partials)/bot-harness-form.tsx`. On the coordinator the modal edits the defaults themselves (`api/admin/harness/save-settings`, the same settings as Settings, under Models, without the providers): super admins only, the icon is disabled for everyone else. Every one of these wraps `settings/models/(partials)/harness-settings-form.tsx`: per harness a provider select first (`harness-provider-select.tsx`: any provider, one provider, or on a bot "Same as the default"), then the model select narrowed to it. Removing a provider (`deleteHarnessProvider`) puts default and bot settings that picked it or its models back on any provider (`save` prop; "Same as the default" shows when the overview item has `default`).
-- **Tools:** `HarnessTool` (built per run with its bot bound): own tools validate with a zod `schema`; tools described elsewhere pass a JSON schema as `parameters` and validate themselves. `run-tool-call.ts` reports bad calls and failures back to the model as `Error: ...`. Bot tools in `src/harness/tools/bot/`: `update_memory`, `update_instructions` (`grabBotTools`, every run), `start_task` and, for the coordinator with workers, `delegate_task` (`grabBotChatTools`, chat replies only; the job's `requested_by_bot_id` is the coordinator, and `relayJobToRequester` posts the worker's result or failure in its conversation, pushed as `relay` by `pushBotNotice`), plus for the coordinator `assign_computer` (grant or revoke a worker's direct access, `add/remove-bot-to/from-computer`; access through a workspace stays and the result says so). `search_computers` (`src/harness/tools/computer/search-computers-tool.ts`, chat replies and tasks) runs one read-only command on all of a bot's computers at once over the sandbox REST API (`run-computer-command.ts`), each output under its `[computer:ID]`, secrets redacted, capped by `AppData.ComputerSearch*`.
+- **Tools:** `HarnessTool` (built per run with its bot bound): own tools validate with a zod `schema`; tools described elsewhere pass a JSON schema as `parameters` and validate themselves. `run-tool-call.ts` reports bad calls and failures back to the model as `Error: ...`. Bot tools in `src/harness/tools/bot/`: `update_memory`, `update_instructions` (`grabBotTools`, every run), `start_task` and, for the coordinator with workers, `delegate_task` (`grabBotChatTools`, chat replies only; the job's `requested_by_bot_id` is the coordinator, and `relayJobToRequester` posts the worker's result or failure in its conversation, pushed as `relay` by `pushBotNotice`), `ask_user_to_take_over` (see Take-over), plus for the coordinator `assign_computer` (grant or revoke a worker's direct access, `add/remove-bot-to/from-computer`; access through a workspace stays and the result says so). `search_computers` (`src/harness/tools/computer/search-computers-tool.ts`, chat replies and tasks) runs one read-only command on all of a bot's computers at once over the sandbox REST API (`run-computer-command.ts`), each output under its `[computer:ID]`, secrets redacted, capped by `AppData.ComputerSearch*`.
 - **MCP:** `src/harness/mcp/`: `connectMcpServer` (Streamable HTTP, SSE fallback), `grabMcpTools` (each server tool becomes a `HarnessTool` with its JSON schema), `mcpResultToText` (text as is, images read by the vision harness, `isError` thrown). Computers are reached this way: `openComputerTools` (`src/harness/tools/computer/`) connects to the computer's AIO sandbox at `grabComputerMcpUrl` (container `metadata.mcp_url`, else the `MAIN_COMPUTER_URL` app setting for the main computer, for a dev server outside docker, e.g. `http://localhost:8170`, else `http://<hostname>:<port>/mcp`), leaves out `ComputerMcpExcludedTools`, and sets the computer's envs (see Envs).
 - **Replies and tasks:** `run-bot-reply.ts` answers from the last `BotReplyHistoryMessages` messages (`botMessagesToHarness`) with `ChatReplyNote`, the bot's computers (`compose-computers-note.ts`: what they are and that they're reached through `start_task`) and the chat tools, at most `BotReplyMaxToolSteps` steps, and finishes silently if the bot already had the last word (`enqueueBotReply` reuses a pending reply, so a burst of messages gets one). `run-bot-task.ts` runs the brief (`compose-job-brief.ts`) with `TaskRunNote`, the bot tools and the job's computer tools, closing the MCP connection after. Retryable harness errors go back to the queue's backoff, the rest are `BotJobFatalError`.
 - **API:** `api/admin/harness/` (`save-provider` also syncs, `sync-provider`, `delete-provider`, `save-settings`), super admin only, each returning the fresh `grabHarnessOverview()` page data.
@@ -171,7 +173,7 @@ Server functions use `index.server.ts` alongside `index.tsx` (e.g. `single-docum
 | Directory                    | Description                                                                                                                                                 |
 | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `@moduletrace/twui`          | **The** design system - external package. Contains layout primitives, elements, editors (TinyMCE), form components, hooks, SVGs, and utilities. Import from `@moduletrace/twui` or a deep path such as `@moduletrace/twui/layout/Button` |
-| `src/components/general/`    | Shared page components built on twui: `logo`, `logo-icon`, `wordmark`, `bot-stone` (+ `bot-stone-face`, `stone-eye-*`), `inner-page-hero-section`, `main-hero-section`, `admin-hero`, `login-shell`, `tagline-section`, media/image helpers |
+| `src/components/general/`    | Shared page components built on twui: `logo`, `logo-icon`, `wordmark`, `bot-stone` (+ `bot-stone-face`, `stone-eye-*`), `user-avatar` (round photo or initials, `sm|md|lg`), `computer-view` (+ `computer-view-frame`, `computer-view-mode-button`), `inner-page-hero-section`, `main-hero-section`, `admin-hero`, `login-shell`, `tagline-section`, media/image helpers |
 
 ## UI & Styling Rules (MANDATORY)
 

src/bots-queue/describe-job-step.test.ts

@@ -5,11 +5,11 @@ import describeJobStep from "./describe-job-step";
 const candidate = { model: { slug: "deepseek/deepseek-chat:free" } } as HarnessCandidate;
 
 describe("describeJobStep", () => {
-    test("names each tool once, never its arguments", () => {
+    test("says what each tool does once, never its arguments", () => {
         expect(
             describeJobStep({
                 index: 2,
-                text: "Let me look.",
+                text: null,
                 candidate,
                 tool_calls: [
                     { id: "a", name: "sandbox_execute_bash", arguments: '{"command":"cat secret"}' },
@@ -19,7 +19,7 @@ describe("describeJobStep", () => {
             }),
         ).toEqual({
             kind: "tool",
-            text: "Using sandbox_execute_bash, update_memory",
+            text: "Running a command, Updating memory",
             metadata: {
                 step: 2,
                 model: "deepseek/deepseek-chat:free",
@@ -28,6 +28,32 @@ describe("describeJobStep", () => {
         });
     });
 
+    test("prefers what the model said it's doing", () => {
+        expect(
+            describeJobStep({
+                index: 0,
+                text: "Opening the Gmail tab.\nThen reading it.",
+                candidate,
+                tool_calls: [{ id: "a", name: "browser_navigate", arguments: "{}" }],
+            })?.text,
+        ).toBe("Opening the Gmail tab.");
+    });
+
+    test("labels tool families and falls back to the name", () => {
+        expect(
+            describeJobStep({
+                index: 0,
+                text: "",
+                candidate,
+                tool_calls: [
+                    { id: "a", name: "browser_navigate", arguments: "{}" },
+                    { id: "b", name: "browser_click", arguments: "{}" },
+                    { id: "c", name: "some_mcp_tool", arguments: "{}" },
+                ],
+            })?.text,
+        ).toBe("Using the browser, Using some_mcp_tool");
+    });
+
     test("keeps the first line of a text-only turn, cut to one line", () => {
         const step = describeJobStep({
             index: 0,

src/bots-queue/describe-job-step.ts

@@ -1,15 +1,21 @@
-import type { BotJobUpdateKind } from "@/src/dict/bot-queue-dict";
+import {
+    JobStepToolLabels,
+    JobStepToolPrefixLabels,
+    type BotJobUpdateKind,
+} from "@/src/dict/bot-queue-dict";
 import type { TextLoopStep } from "@/src/harness/text/run-text-loop";
 
 /**
- * Longest step line kept: the panel shows one line per step.
+ * Longest step line kept: the chat shows one line per step.
  */
 const MaxStepChars = 200;
 
 /**
- * One model turn as a `bots_queue_updates` row: the tools it called, or
+ * One model turn as a `bots_queue_updates` row, read by the user while the
- * the first line of what it wrote. Null for an empty turn. Never the tool
+ * bot works: the first line the model wrote (models often say what
- * arguments, which can hold anything the bot typed.
+ * they're about to do), else what its tools do in plain words. Null for an
+ * empty turn. Never the tool arguments, which can hold anything the bot
+ * typed.
  */
 export default function describeJobStep({
     text,
@@ -22,24 +28,34 @@ export default function describeJobStep({
     metadata: Record<string, unknown>;
 } | null {
     const metadata = { step: index, model: candidate.model.slug };
+    const line = firstLine({ text });
 
     if (tool_calls.length) {
         const tools = [...new Set(tool_calls.map((call) => call.name))];
+        const labels = [...new Set(tools.map((name) => toolLabel({ name })))];
 
         return {
             kind: "tool",
-            text: `Using ${tools.join(", ")}`,
+            text: line || labels.join(", "),
             metadata: { ...metadata, tools },
         };
     }
 
+    if (!line) return null;
+
+    return { kind: "note", text: line, metadata };
+}
+
+function firstLine({ text }: { text: string | null }) {
     const line = text?.trim().split("\n").find((l) => l.trim())?.trim();
 
     if (!line) return null;
 
-    return {
+    return line.length > MaxStepChars ? `${line.slice(0, MaxStepChars - 1)}…` : line;
-        kind: "note",
+}
-        text: line.length > MaxStepChars ? `${line.slice(0, MaxStepChars - 1)}…` : line,
+
-        metadata,
+function toolLabel({ name }: { name: string }) {
-    };
+    const prefix = Object.keys(JobStepToolPrefixLabels).find((p) => name.startsWith(p));
+
+    return JobStepToolLabels[name] || (prefix && JobStepToolPrefixLabels[prefix]) || `Using ${name}`;
 }

src/bots-queue/run-bot-reply.ts

@@ -1,6 +1,6 @@
 import type { BUN_SQLITE_COBBLE_BOTS } from "@/db/types/db";
 import { AppData } from "@/src/data/app-data";
-import { ChatReplyNote, CoordinatorReplyNote } from "@/src/data/harness-prompts";
+import { ChatReplyNote, CoordinatorReplyNote, CoordinatorSoloNote } from "@/src/data/harness-prompts";
 import grabBotMessages from "@/src/functions/backend/db/bots/grab-bot-messages";
 import grabBotPrompt from "@/src/functions/backend/db/bots/grab-bot-prompt";
 import grabBotComputers from "@/src/functions/backend/db/computers/grab-bot-computers";
@@ -9,6 +9,7 @@ import grabBotChatTools from "@/src/harness/tools/bot/grab-bot-chat-tools";
 import botMessagesToHarness from "./bot-messages-to-harness";
 import composeComputersNote from "./compose-computers-note";
 import loadAttachmentParts from "./load-attachment-parts";
+import withTakeOverRefs from "./with-take-over-refs";
 import type { BotJobContext, BotJobOutcome } from "./types";
 
 type Params = BotJobContext & {
@@ -40,12 +41,32 @@ export default async function runBotReply({
     const prompt = await grabBotPrompt({ bot_id: bot.id, user_id: bot.user_id });
     const computers = await grabBotComputers({ bot_id: bot.id, user_id: bot.user_id });
 
+    const take_overs: string[] = [];
+
+    const tools = await grabBotChatTools({
+        bot,
+        user_id: bot.user_id,
+        computers,
+        message_id: job.message_id || undefined,
+        onTakeOver: ({ ref }) => take_overs.push(ref),
+    });
+
+    /**
+     * delegate_task is only there while the coordinator has workers.
+     */
+    const crew_note =
+        bot.kind != "coordinator"
+            ? ""
+            : tools.some((tool) => tool.name == "delegate_task")
+              ? CoordinatorReplyNote
+              : CoordinatorSoloNote;
+
     const result = await runTextLoop({
         system: [
             prompt,
             ChatReplyNote,
             composeComputersNote({ computers, is_coordinator: bot.kind == "coordinator" }),
-            bot.kind == "coordinator" ? CoordinatorReplyNote : "",
+            crew_note,
             `Now: ${new Date().toUTCString()}.`,
         ]
             .filter(Boolean)
@@ -54,17 +75,12 @@ export default async function runBotReply({
             messages: history,
             parts: await loadAttachmentParts({ messages: history }),
         }),
-        tools: await grabBotChatTools({
+        tools,
-            bot,
-            user_id: bot.user_id,
-            computers,
-            message_id: job.message_id || undefined,
-        }),
         max_steps: AppData.BotReplyMaxToolSteps,
         bot_id: bot.id,
         signal,
         onStep: report,
     });
 
-    return { result: result.text };
+    return withTakeOverRefs({ text: result.text, refs: take_overs });
 }

src/bots-queue/run-bot-task.ts

@@ -4,9 +4,11 @@ import grabBotPrompt from "@/src/functions/backend/db/bots/grab-bot-prompt";
 import runTextLoop from "@/src/harness/text/run-text-loop";
 import openComputerTools from "@/src/harness/tools/computer/open-computer-tools";
 import grabBotTools from "@/src/harness/tools/bot/grab-bot-tools";
+import askToTakeOverTool from "@/src/harness/tools/computer/ask-to-take-over-tool";
 import searchComputersTool from "@/src/harness/tools/computer/search-computers-tool";
 import computerRef from "@/src/utils/computer-ref";
 import composeJobBrief from "./compose-job-brief";
+import withTakeOverRefs from "./with-take-over-refs";
 import type { BotJobContext, BotJobOutcome } from "./types";
 
 type Params = BotJobContext & {
@@ -17,6 +19,8 @@ type Params = BotJobContext & {
  * Works through a task: the bot follows its brief with its full prompt,
  * its own tools, search_computers across all its computers and, when the
  * job has a computer, that computer's tools (over MCP), until it answers. That answer is the report for the user.
+ * When only the user can go on (a login, a captcha), the bot asks them to
+ * take over (`ask_user_to_take_over`) and the report needs them.
  * Every model turn is reported as a step (`report`).
  */
 export default async function runBotTask({
@@ -33,13 +37,22 @@ export default async function runBotTask({
     try {
         const prompt = await grabBotPrompt({ bot_id: bot.id, user_id: bot.user_id });
         const computers = await grabBotComputers({ bot_id: bot.id, user_id: bot.user_id });
+        const take_overs: string[] = [];
+
+        /**
+         * A take-over defaults to the computer the task runs on.
+         */
+        const take_over_computers = [
+            ...computers.filter((c) => c.id == job.computer_id),
+            ...computers.filter((c) => c.id != job.computer_id),
+        ];
 
         const result = await runTextLoop({
             system: [
                 prompt,
                 TaskRunNote,
                 computer
-                    ? `You're working on the computer "${computer.name}" (a Linux sandbox with a shell, files, code runners and a browser). Use its tools; don't ask the user to run anything.`
+                    ? `You're working on the computer "${computer.name}" (a Linux sandbox with a shell, files, code runners and a browser). Use its tools; don't ask the user to run anything. When only the user can go on (a login, a captcha, a code sent to them), ask them to take over with ask_user_to_take_over.`
                     : `You have no computer for this task.`,
                 computers.length > 1
                     ? `search_computers runs a read-only command on every computer you have. When you name one in the report, write its ref, e.g. ${computerRef({ id: computers[0]!.id || 0 })}: the user can click it to switch you to that computer.`
@@ -64,7 +77,16 @@ export default async function runBotTask({
             ],
             tools: [
                 ...grabBotTools({ user_id: bot.user_id, bot_id: bot.id }),
-                ...(computers.length ? [searchComputersTool({ bot_id: bot.id, computers })] : []),
+                ...(computers.length
+                    ? [
+                          searchComputersTool({ bot_id: bot.id, computers }),
+                          askToTakeOverTool({
+                              computers: take_over_computers,
+                              in_task: true,
+                              onAsk: ({ ref }) => take_overs.push(ref),
+                          }),
+                      ]
+                    : []),
                 ...(computer?.tools || []),
             ],
             bot_id: bot.id,
@@ -73,7 +95,10 @@ export default async function runBotTask({
                 report({ ...step, text: step.text && computer ? computer.redact(step.text) : step.text }),
         });
 
-        return { result: computer ? computer.redact(result.text) : result.text };
+        return withTakeOverRefs({
+            text: computer ? computer.redact(result.text) : result.text,
+            refs: take_overs,
+        });
     } finally {
         await computer?.close();
     }

src/bots-queue/with-take-over-refs.test.ts

@@ -0,0 +1,24 @@
+import { describe, expect, test } from "bun:test";
+import takeOverRef from "@/src/utils/take-over-ref";
+import withTakeOverRefs from "./with-take-over-refs";
+
+const ref = takeOverRef({ id: 1, mode: "browser" });
+
+describe("withTakeOverRefs", () => {
+    test("adds a take-over the bot didn't write, once", () => {
+        expect(withTakeOverRefs({ text: "Please log in to Gmail.", refs: [ref, ref] })).toEqual({
+            result: `Please log in to Gmail.\n\n${ref}`,
+            needs_you: true,
+        });
+    });
+
+    test("keeps one the bot wrote itself", () => {
+        const text = `Log in here: ${ref}`;
+
+        expect(withTakeOverRefs({ text, refs: [ref] })).toEqual({ result: text, needs_you: true });
+    });
+
+    test("leaves a message without a take-over alone", () => {
+        expect(withTakeOverRefs({ text: "Done.", refs: [] })).toEqual({ result: "Done.", needs_you: false });
+    });
+});

src/bots-queue/with-take-over-refs.ts

@@ -0,0 +1,21 @@
+type Params = {
+    text: string;
+    /**
+     * The take-over refs the bot asked for in this run (`askToTakeOverTool`).
+     */
+    refs: string[];
+};
+
+/**
+ * A job's message with every take-over the bot asked for in it: refs the
+ * bot didn't write itself are added at the end, so the user always gets
+ * the button. A message asking for a take-over needs the user.
+ */
+export default function withTakeOverRefs({ text, refs }: Params) {
+    const missing = [...new Set(refs)].filter((ref) => !text.includes(ref));
+
+    return {
+        result: [text.trim(), ...missing].filter(Boolean).join("\n\n"),
+        needs_you: refs.length > 0,
+    };
+}

src/components/general/computer-view-frame.tsx

@@ -0,0 +1,29 @@
+import { ComputerViewModes, type ComputerViewMode } from "@/src/dict/computer-dict";
+import computerViewUrl from "@/src/utils/computer-view-url";
+
+type Props = {
+    computer_id: number;
+    name: string;
+    mode: ComputerViewMode;
+    /**
+     * The mode on show; the others stay loaded but hidden.
+     */
+    active: boolean;
+};
+
+/**
+ * One of a computer's apps in a frame. A raw iframe: twui has no
+ * equivalent.
+ */
+export default function ComputerViewFrame({ computer_id, name, mode, active }: Props) {
+    const title = ComputerViewModes.find((m) => m.value == mode)?.title || mode;
+
+    return (
+        <iframe
+            src={computerViewUrl({ computer_id, mode })}
+            title={`${name}: ${title}`}
+            allow="clipboard-read; clipboard-write; fullscreen"
+            className={`absolute inset-0 w-full h-full border-0 bg-sunken ${active ? "" : "hidden"}`}
+        />
+    );
+}

src/components/general/computer-view-mode-button.tsx

@@ -0,0 +1,31 @@
+import Button from "@moduletrace/twui/layout/Button";
+
+type Props = {
+    title: string;
+    active: boolean;
+    onPick: () => void;
+};
+
+/**
+ * One mode in a computer view's toolbar (Browser, Terminal, Code).
+ */
+export default function ComputerViewModeButton({ title, active, onPick }: Props) {
+    return (
+        <Button
+            type="button"
+            title={title}
+            role="tab"
+            aria-selected={active}
+            variant="ghost"
+            color="text"
+            onClick={onPick}
+            className={`px-3! py-1.5! text-[13px] leading-[18px] rounded-default transition-colors duration-[var(--m-fast)] focus-visible:ring-2 focus-visible:ring-ink ${
+                active
+                    ? "bg-sunken dark:bg-sunken text-ink dark:text-ink font-bold"
+                    : "text-ink-secondary dark:text-ink-secondary hover:text-ink hover:dark:text-ink hover:bg-sunken hover:dark:bg-sunken"
+            }`}
+        >
+            {title}
+        </Button>
+    );
+}

src/components/general/computer-view.tsx

@@ -0,0 +1,77 @@
+import { useState } from "react";
+import Link from "@moduletrace/twui/layout/Link";
+import Row from "@moduletrace/twui/layout/Row";
+import Stack from "@moduletrace/twui/layout/Stack";
+import { ExternalLink } from "lucide-react";
+import { ComputerViewModes, type ComputerViewMode } from "@/src/dict/computer-dict";
+import computerViewUrl from "@/src/utils/computer-view-url";
+import ComputerViewFrame from "./computer-view-frame";
+import ComputerViewModeButton from "./computer-view-mode-button";
+
+type Props = {
+    computer_id: number;
+    /**
+     * The computer's name, for the frame titles.
+     */
+    name: string;
+    /**
+     * The mode it opens in.
+     */
+    default_mode?: ComputerViewMode;
+    className?: string;
+};
+
+/**
+ * A computer the user can work on: its browser, terminal or code editor
+ * (`ComputerViewModes`), served through the app (`/computers/<id>/`).
+ * A mode stays open once visited, so switching back keeps its state.
+ */
+export default function ComputerView({ computer_id, name, default_mode, className }: Props) {
+    const [mode, setMode] = useState<ComputerViewMode>(
+        default_mode || ComputerViewModes[0].value,
+    );
+    const [visited, setVisited] = useState<ComputerViewMode[]>([mode]);
+
+    const pick = ({ value }: { value: ComputerViewMode }) => {
+        setMode(value);
+        setVisited((prev) => (prev.includes(value) ? prev : [...prev, value]));
+    };
+
+    return (
+        <Stack className={`w-full min-h-0 gap-0! items-stretch ${className || ""}`}>
+            <Row className="w-full gap-1! flex-nowrap items-center px-3 py-2 border-0 border-b border-solid border-hairline">
+                <Row role="tablist" aria-label={`${name} modes`} className="gap-1! flex-nowrap">
+                    {ComputerViewModes.map((item) => (
+                        <ComputerViewModeButton
+                            key={item.value}
+                            title={item.title}
+                            active={item.value == mode}
+                            onPick={() => pick({ value: item.value })}
+                        />
+                    ))}
+                </Row>
+                <Link
+                    href={computerViewUrl({ computer_id, mode })}
+                    target="_blank"
+                    rel="noopener noreferrer"
+                    title="Open in a new tab"
+                    aria-label="Open in a new tab"
+                    className="ml-auto p-2 text-ink-secondary hover:text-ink rounded-default hover:bg-sunken"
+                >
+                    <ExternalLink size={16} aria-hidden />
+                </Link>
+            </Row>
+            <Stack className="relative grow min-h-0 gap-0! items-stretch bg-sunken">
+                {visited.map((value) => (
+                    <ComputerViewFrame
+                        key={value}
+                        computer_id={computer_id}
+                        name={name}
+                        mode={value}
+                        active={value == mode}
+                    />
+                ))}
+            </Stack>
+        </Stack>
+    );
+}

src/data/harness-prompts.ts

@@ -24,8 +24,9 @@ export const ChatReplyNote = `# Replying in chat
 You are chatting with the user. Answer their latest messages directly and briefly, in markdown.
 
 - Quick questions and small changes: answer now.
-- Work that takes more than a moment, or anything on a computer (running commands, editing files, browsing): call start_task with a clear brief, then tell the user in a sentence that you're on it. The result is posted here when it's done.
+- Work that takes more than a moment, or anything on a computer (running commands, editing files, browsing): call start_task with a clear brief, then tell the user in one sentence, in the first person, what you're doing, e.g. "I'm going through your inbox now. I'll report back when I'm done." A task you start is your own work: never say you handed it to anyone else. The result is posted here when it's done.
 - When a message mixes both, answer the quick parts in this reply and start a task for the rest.
+- When only the user can do something on a computer (log in, solve a captcha, enter a code), ask them to take over with ask_user_to_take_over and say exactly what to do. They'll tell you when they hand it back; then carry on.
 - When the user tells you something worth keeping, save it with update_memory. When they correct how you work, update your instructions with update_instructions.
 - Never claim you did something you didn't do.`;
 
@@ -34,9 +35,15 @@ You are chatting with the user. Answer their latest messages directly and briefl
  */
 export const CoordinatorReplyNote = `As the coordinator, hand work that fits a crew member's role to them with delegate_task. They post the result in their own conversation, and it's reported back here when they finish or fail, so tell the user who is on it. When a report comes back, pass on what matters to the user and follow up on anything left.`;
 
+/**
+ * Added to the coordinator's chat prompt instead while it has no workers,
+ * so it doesn't talk about a crew that doesn't exist.
+ */
+export const CoordinatorSoloNote = `You have no worker bots yet: you do all the work yourself, with start_task for anything long or on a computer. Never say you assigned, delegated or handed work to a worker or anyone else. If a kind of job keeps coming up, you can suggest a new bot for it.`;
+
 /**
  * Added to a bot's prompt when it works through a queued task.
  */
 export const TaskRunNote = `# Working on a task
 
-You are working through a task on your own; the user isn't watching. Use your tools until the task is done, then answer with the report for the user: what you did, what you found, and anything left for them. Keep the report short and in markdown. If you're blocked on something only the user can give you, stop and ask for it in the report.`;
+You are working through a task on your own; the user isn't watching. Use your tools until the task is done, then answer with the report for the user: what you did, what you found, and anything left for them. Keep the report short and in markdown. If you're blocked on something only the user can give you, stop and ask for it in the report. If they need to do it on the computer themselves (log in, a captcha, a code), call ask_user_to_take_over first.`;

src/dict/bot-queue-dict.ts

@@ -42,3 +42,29 @@ export const BotReplyJobTitle = "Thinking";
  * The bot's status line while its job waits for a rate-limited model.
  */
 export const BotWaitingStatusLine = "Waiting for a free model";
+
+/**
+ * How a job step that called a tool reads in the chat's working row, by
+ * tool name. Tools not listed read "Using <name>".
+ */
+export const JobStepToolLabels: Record<string, string> = {
+    sandbox_execute_bash: "Running a command",
+    sandbox_execute_code: "Running code",
+    sandbox_read_file: "Reading a file",
+    sandbox_write_file: "Writing a file",
+    sandbox_file_operations: "Working with files",
+    search_computers: "Searching the computers",
+    ask_user_to_take_over: "Asking you to take over",
+    start_task: "Starting a task",
+    delegate_task: "Handing work to the crew",
+    assign_computer: "Changing computer access",
+    update_memory: "Updating memory",
+    update_instructions: "Updating instructions",
+};
+
+/**
+ * Labels for whole families of tools, by name prefix.
+ */
+export const JobStepToolPrefixLabels: Record<string, string> = {
+    browser_: "Using the browser",
+};

src/dict/computer-dict.ts

@@ -87,3 +87,34 @@ export const ComputerMcpExcludedTools = [
     "browser_gui_screenshot",
     "browser_gui_execute_action",
 ] as const;
+
+/**
+ * How the user can open a computer, each one of the AIO sandbox's own
+ * apps at `path` (relative to the computer's address). More of them
+ * (VNC, opencode, codex) can join later.
+ */
+export const ComputerViewModes = [
+    {
+        title: "Browser",
+        value: "browser",
+        path: "browser-ui",
+    },
+    {
+        title: "Terminal",
+        value: "terminal",
+        path: "terminal",
+    },
+    {
+        title: "Code",
+        value: "code",
+        path: "code-server/",
+    },
+] as const;
+
+export type ComputerViewMode = (typeof ComputerViewModes)[number]["value"];
+
+/**
+ * Where the app serves a computer to its owner's browser:
+ * `/computers/<id>/<path>`, proxied to the sandbox (`handleComputerProxy`).
+ */
+export const ComputerProxyPrefix = "/computers";

src/functions/server/computers/computer-proxy-socket.ts

@@ -0,0 +1,69 @@
+import type { ServerWebSocket } from "bun";
+import type { WebSocketSession } from "@/src/websocket/types";
+
+type Socket = ServerWebSocket<WebSocketSession>;
+
+/**
+ * Bridges a user's websocket on `/computers/<id>/...` to the same path on
+ * the computer (a terminal shell, the browser's CDP, the code editor).
+ * Messages sent before the computer's side opens are held, and either
+ * side closing closes the other.
+ */
+export const computerProxySocket = {
+    open(ws: Socket) {
+        const proxy = ws.data.proxy;
+
+        if (!proxy) return;
+
+        const upstream = new WebSocket(proxy.url, {
+            headers: proxy.headers,
+            protocols: proxy.protocols,
+        } as any);
+
+        upstream.binaryType = "arraybuffer";
+        proxy.upstream = upstream;
+        proxy.pending = [];
+
+        upstream.onopen = () => {
+            for (const message of proxy.pending || []) upstream.send(message as string | Uint8Array<ArrayBuffer>);
+            proxy.pending = undefined;
+        };
+
+        upstream.onmessage = (event) => {
+            ws.send(typeof event.data == "string" ? event.data : new Uint8Array(event.data));
+        };
+
+        upstream.onclose = (event) => {
+            ws.close(closeCode({ code: event.code }), event.reason);
+        };
+
+        upstream.onerror = () => {
+            ws.close(1011, "The computer isn't answering");
+        };
+    },
+
+    message(ws: Socket, message: string | Buffer) {
+        const proxy = ws.data.proxy;
+
+        if (!proxy?.upstream) return;
+
+        if (proxy.pending) proxy.pending.push(message);
+        else if (proxy.upstream.readyState == WebSocket.OPEN) {
+            proxy.upstream.send(message as string | Uint8Array<ArrayBuffer>);
+        }
+    },
+
+    close(ws: Socket) {
+        const upstream = ws.data.proxy?.upstream;
+
+        if (upstream && upstream.readyState < WebSocket.CLOSING) upstream.close();
+    },
+};
+
+/**
+ * Codes a server may not send itself (no status, abnormal) become a
+ * normal close.
+ */
+function closeCode({ code }: { code: number }) {
+    return code == 1005 || code == 1006 || code == 1015 ? 1000 : code;
+}

src/functions/server/computers/computer-proxy-target.ts

@@ -0,0 +1,46 @@
+import grabComputerMcpUrl from "@/src/functions/backend/db/computers/grab-computer-mcp-url";
+import grabUserComputer from "@/src/functions/backend/db/computers/grab-user-computer";
+import userAuth from "@/src/functions/backend/auth/user-auth";
+import parseComputerProxyPath from "@/src/utils/parse-computer-proxy-path";
+
+type Params = {
+    req: Request;
+};
+
+/**
+ * Where a `/computers/<id>/...` request goes: the computer's own address
+ * for that path, only for the computer's owner. An error response when
+ * the user isn't logged in, the computer isn't theirs, or it has no
+ * container.
+ */
+export default async function computerProxyTarget({ req }: Params) {
+    const url = new URL(req.url);
+    const parsed = parseComputerProxyPath({ pathname: url.pathname });
+
+    if (!parsed) return { error: new Response("Not Found", { status: 404 }) };
+
+    const session = await userAuth({ req });
+
+    if (!session.user?.id) return { error: new Response("Unauthorized", { status: 401 }) };
+
+    const computer = await grabUserComputer({
+        computer_id: parsed.computer_id,
+        user_id: session.user.id,
+    });
+
+    if (!computer?.id) return { error: new Response("Not Found", { status: 404 }) };
+
+    const address = await grabComputerMcpUrl({ computer_id: computer.id });
+
+    if (!address) {
+        return { error: new Response("This computer has no container.", { status: 502 }) };
+    }
+
+    return {
+        session,
+        url,
+        prefix: parsed.prefix,
+        base_url: address.base_url,
+        target: `${address.base_url}/${parsed.rest}${url.search}`,
+    };
+}

src/functions/server/computers/handle-computer-proxy.ts

@@ -0,0 +1,70 @@
+import type { Server } from "bun";
+import computerProxyRequestHeaders from "@/src/utils/computer-proxy-request-headers";
+import computerProxyResponseHeaders from "@/src/utils/computer-proxy-response-headers";
+import type { WebSocketSession } from "@/src/websocket/types";
+import computerProxyTarget from "./computer-proxy-target";
+
+type Params = {
+    req: Request;
+    server: Server<WebSocketSession>;
+};
+
+/**
+ * Serves a computer's own apps (browser, terminal, code) to its owner
+ * under `/computers/<id>/`, so the chat can show them in a frame without
+ * the sandbox being reachable from the user's browser. Plain requests are
+ * fetched and streamed back; websockets are bridged
+ * (`computerProxySocket`).
+ */
+export default async function handleComputerProxy({ req, server }: Params) {
+    const resolved = await computerProxyTarget({ req });
+
+    if (resolved.error) return resolved.error;
+
+    const { session, url, prefix, base_url, target } = resolved;
+    const headers = computerProxyRequestHeaders({ headers: req.headers, url, prefix });
+
+    if (req.headers.get("upgrade")?.toLowerCase() == "websocket") {
+        const protocols = (req.headers.get("sec-websocket-protocol") || "")
+            .split(",")
+            .map((p) => p.trim())
+            .filter(Boolean);
+
+        for (const name of [...headers.keys()]) {
+            if (name.startsWith("sec-websocket-")) headers.delete(name);
+        }
+
+        const upgraded = server.upgrade(req, {
+            headers: protocols[0] ? { "Sec-WebSocket-Protocol": protocols[0] } : undefined,
+            data: {
+                user: session.user!,
+                user_types: session.user_types ?? [],
+                proxy: {
+                    url: target.replace(/^http/, "ws"),
+                    headers: Object.fromEntries(headers.entries()),
+                    protocols,
+                },
+            } satisfies WebSocketSession,
+        });
+
+        return upgraded ? undefined : new Response("Upgrade failed", { status: 400 });
+    }
+
+    const has_body = req.method != "GET" && req.method != "HEAD";
+
+    const res = await fetch(target, {
+        method: req.method,
+        headers,
+        body: has_body ? req.body : undefined,
+        redirect: "manual",
+        signal: req.signal,
+    }).catch(() => null);
+
+    if (!res) return new Response("This computer isn't answering.", { status: 502 });
+
+    return new Response(res.body, {
+        status: res.status,
+        statusText: res.statusText,
+        headers: computerProxyResponseHeaders({ headers: res.headers, base_url, prefix }),
+    });
+}

src/harness/tools/bot/grab-bot-chat-tools.ts

@@ -2,6 +2,7 @@ import BunSQLite from "@moduletrace/bun-sqlite";
 import type { BUN_SQLITE_COBBLE_BOTS, BUN_SQLITE_COBBLE_COMPUTERS } from "@/db/types/db";
 import grabBotComputers from "@/src/functions/backend/db/computers/grab-bot-computers";
 import type { HarnessTool } from "../../types";
+import askToTakeOverTool from "../computer/ask-to-take-over-tool";
 import searchComputersTool from "../computer/search-computers-tool";
 import assignComputerTool from "./assign-computer-tool";
 import delegateTaskTool from "./delegate-task-tool";
@@ -19,12 +20,17 @@ type Params = {
      * The user message being answered.
      */
     message_id?: number | string;
+    /**
+     * Called when the bot asks the user to take over a computer.
+     */
+    onTakeOver?: (params: { ref: string }) => void;
 };
 
 /**
  * The tools a bot replies in chat with: its own (memory, instructions),
  * start_task for longer work, search_computers to find what's on its
- * computers, and for the coordinator delegate_task and assign_computer
+ * computers, ask_user_to_take_over when only the user can do something on
+ * one, and for the coordinator delegate_task and assign_computer
  * (which computers each crew member can use). Anything
  * else on a computer goes through a task, never mid-reply.
  */
@@ -33,6 +39,7 @@ export default async function grabBotChatTools({
     user_id,
     computers,
     message_id,
+    onTakeOver,
 }: Params): Promise<HarnessTool[]> {
     const bot_id = bot.id || 0;
     const tools = [
@@ -43,7 +50,12 @@ export default async function grabBotChatTools({
             computers,
             message_id,
         }),
-        ...(computers.length ? [searchComputersTool({ bot_id, computers })] : []),
+        ...(computers.length
+            ? [
+                  searchComputersTool({ bot_id, computers }),
+                  askToTakeOverTool({ computers, onAsk: (params) => onTakeOver?.(params) }),
+              ]
+            : []),
     ];
 
     if (bot.kind != "coordinator") return tools;

src/harness/tools/bot/start-task-tool.ts

@@ -60,7 +60,7 @@ export default function startTaskTool({
                 computer_id: computer_id || undefined,
             });
 
-            return `Task #${job_id} queued. Tell the user you're on it; the result will be posted here when it's done.`;
+            return `Task #${job_id} queued; you'll work on it yourself in the background. Tell the user in one sentence, in the first person, what you're doing (e.g. "I'm checking your inbox now. I'll report back when I'm done."); the result will be posted here when it's done.`;
         },
     };
 }

src/harness/tools/computer/ask-to-take-over-tool.ts

@@ -0,0 +1,65 @@
+import { z } from "zod";
+import type { BUN_SQLITE_COBBLE_COMPUTERS } from "@/db/types/db";
+import { ComputerViewModes, type ComputerViewMode } from "@/src/dict/computer-dict";
+import takeOverRef from "@/src/utils/take-over-ref";
+import type { HarnessTool } from "../../types";
+
+type Params = {
+    /**
+     * The bot's computers, the one to default to first.
+     */
+    computers: BUN_SQLITE_COBBLE_COMPUTERS[];
+    /**
+     * In a task the bot stops and reports; in chat it says so in its reply.
+     */
+    in_task?: boolean;
+    /**
+     * Called with the take-over ref, which the bot's message must carry
+     * (`withTakeOverRefs`).
+     */
+    onAsk: (params: { ref: string }) => void;
+};
+
+type Args = {
+    computer_id?: number;
+    mode: ComputerViewMode;
+};
+
+const Modes = ComputerViewModes.map((m) => m.value) as [ComputerViewMode, ...ComputerViewMode[]];
+
+/**
+ * Lets a bot ask the user to take over one of its computers, for what
+ * only they can do there (a login, a captcha, a code). Their message
+ * gets a button that opens the computer in that mode, and the bot shows
+ * needs-you until they hand it back in chat.
+ */
+export default function askToTakeOverTool({ computers, in_task, onAsk }: Params): HarnessTool<Args> {
+    const listed = computers
+        .map((computer) => `${computer.id} (${computer.name || computer.kind})`)
+        .join(", ");
+
+    return {
+        name: "ask_user_to_take_over",
+        description: `Ask the user to take over one of your computers when only they can do something on it: log in, solve a captcha, enter a code, approve a prompt, or show you something. They get a button that opens it in the mode you pick: browser (the web browser), terminal (a shell) or code (the code editor). Computers: ${listed}. Defaults to ${computers[0]?.id}.`,
+        schema: z.object({
+            computer_id: z.number().int().optional(),
+            mode: z.enum(Modes),
+        }),
+        run: async ({ args }) => {
+            const computer =
+                computers.find((c) => c.id == args.computer_id) || (args.computer_id ? null : computers[0]);
+
+            if (!computer?.id) {
+                throw new Error(`You have no computer with id ${args.computer_id}. Computers: ${listed}.`);
+            }
+
+            const ref = takeOverRef({ id: computer.id, mode: args.mode });
+
+            onAsk({ ref });
+
+            return in_task
+                ? `Asked; the take-over button ${ref} is added to your report. Stop working now and finish with the report: what you've done so far and exactly what the user needs to do on the computer. When they hand it back they'll tell you in chat, and you carry on from there.`
+                : `Asked; the take-over button ${ref} is added to your reply. Tell the user in a sentence or two exactly what to do on the computer. They'll tell you here when they hand it back; carry on then.`;
+        },
+    };
+}

src/pages/admin/bots/[bot_id]/(data)/bot-computers-context.ts

@@ -2,10 +2,12 @@ import React from "react";
 import type { BotComputers } from "../(hooks)/use-bot-computers";
 
 /**
- * The open bot's computers and the switch, shared by the header's
+ * The open bot's computers, the switch and the viewer, shared by the
- * computer select and the computer buttons in its messages.
+ * header's computers modal and the computer buttons in its messages.
  */
 export const BotComputersContext = React.createContext<BotComputers>({
     computers: [],
     switchComputer: async () => {},
+    openComputer: () => {},
+    closeComputer: () => {},
 });

src/pages/admin/bots/[bot_id]/(hooks)/use-bot-computers.ts

@@ -2,8 +2,19 @@ import { useState } from "react";
 import useServerEvent from "@/src/hooks/use-server-event";
 import type { BotUpdatedData } from "@/src/types";
 import type { BUN_SQLITE_COBBLE_COMPUTERS } from "@/db/types/db";
+import type { ComputerViewMode } from "@/src/dict/computer-dict";
 import setBotComputer from "../(functions)/set-bot-computer";
 
+/**
+ * The computer open in the viewer, in which mode, and whether the bot
+ * asked the user to take it over (the viewer then offers to hand it back).
+ */
+export type ComputerViewing = {
+    computer_id: number;
+    mode: ComputerViewMode;
+    take_over?: boolean;
+};
+
 export type BotComputers = {
     /**
      * The bot's computers, the one it's on first.
@@ -15,6 +26,9 @@ export type BotComputers = {
     switching_id?: number;
     error?: string;
     switchComputer: (params: { computer_id: number }) => Promise<void>;
+    viewing?: ComputerViewing;
+    openComputer: (params: ComputerViewing) => void;
+    closeComputer: () => void;
 };
 
 type Params = {
@@ -25,12 +39,14 @@ type Params = {
 /**
  * Keeps the open bot's computers and switches the one it's on, in place:
  * no reload, the list comes back reordered. A switch made in another tab
- * arrives as `server:bot-updated` with the bot's `computer_id`.
+ * arrives as `server:bot-updated` with the bot's `computer_id`. Also holds
+ * the computer open in the viewer (`ComputerViewerModal`).
  */
 export default function useBotComputers({ bot_id, computers: initial }: Params): BotComputers {
     const [computers, setComputers] = useState(initial);
     const [switching_id, setSwitchingId] = useState<number>();
     const [error, setError] = useState<string>();
+    const [viewing, setViewing] = useState<ComputerViewing>();
 
     useServerEvent({
         event: "server:bot-updated",
@@ -66,5 +82,13 @@ export default function useBotComputers({ bot_id, computers: initial }: Params):
         setSwitchingId(undefined);
     };
 
-    return { computers, switching_id, error, switchComputer };
+    return {
+        computers,
+        switching_id,
+        error,
+        switchComputer,
+        viewing,
+        openComputer: (params) => setViewing(params),
+        closeComputer: () => setViewing(undefined),
+    };
 }

src/pages/admin/bots/[bot_id]/(hooks)/use-task-progress.ts

@@ -0,0 +1,54 @@
+import { useState } from "react";
+import useServerEvent from "@/src/hooks/use-server-event";
+import type { BotJobUpdateData, BotTaskWithSteps, BotUpdatedData } from "@/src/types";
+
+type Params = {
+    bot_id: number;
+    /**
+     * The bot's active task when the page loaded.
+     */
+    task: BotTaskWithSteps | null;
+};
+
+type Progress = {
+    task_id: number | null;
+    text: string | null;
+};
+
+function fromTask({ task }: { task: BotTaskWithSteps | null }): Progress {
+    return {
+        task_id: task?.id || null,
+        text: task?.updates[task.updates.length - 1]?.text || null,
+    };
+}
+
+/**
+ * The latest step of the open bot's active task (a `bots_queue_updates`
+ * row's text), kept current from `server:bot-job-update`. Starts over when
+ * `server:bot-updated` brings a different task, or none.
+ */
+export default function useTaskProgress({ bot_id, task }: Params) {
+    const [progress, setProgress] = useState<Progress>(() => fromTask({ task }));
+
+    useServerEvent({
+        event: "server:bot-job-update",
+        onData: (update: BotJobUpdateData) => {
+            if (update.bot_id != bot_id || !update.task_id || !update.text) return;
+
+            setProgress({ task_id: update.task_id, text: update.text });
+        },
+    });
+
+    useServerEvent({
+        event: "server:bot-updated",
+        onData: (data: BotUpdatedData) => {
+            if (data.id != bot_id) return;
+
+            setProgress((prev) =>
+                (data.task?.id || null) == prev.task_id ? prev : fromTask({ task: data.task }),
+            );
+        },
+    });
+
+    return progress.text;
+}

src/pages/admin/bots/[bot_id]/(partials)/bot-computer-option.tsx

@@ -1,7 +1,8 @@
+import Button from "@moduletrace/twui/layout/Button";
 import Row from "@moduletrace/twui/layout/Row";
 import Span from "@moduletrace/twui/layout/Span";
 import Stack from "@moduletrace/twui/layout/Stack";
-import { Check } from "lucide-react";
+import { AppWindow, Check } from "lucide-react";
 import type { BUN_SQLITE_COBBLE_COMPUTERS } from "@/db/types/db";
 
 type Props = {
@@ -11,14 +12,21 @@ type Props = {
      */
     current: boolean;
     onPick: () => void;
+    /**
+     * Opens the computer in the viewer.
+     */
+    onOpen: () => void;
 };
 
 /**
  * One computer in the header's computers modal, checked when it's the
- * one the bot is on.
+ * one the bot is on, with a button that opens it.
  */
-export default function BotComputerOption({ computer, current, onPick }: Props) {
+export default function BotComputerOption({ computer, current, onPick, onOpen }: Props) {
+    const name = computer.name || `Computer ${computer.id}`;
+
     return (
+        <Row className="w-full gap-1! flex-nowrap items-center pr-2 hover:bg-sunken transition-colors duration-[var(--m-fast)]">
         <Row
             role="radio"
             aria-checked={current}
@@ -30,14 +38,14 @@ export default function BotComputerOption({ computer, current, onPick }: Props)
                     onPick();
                 }
             }}
-            className={`w-full min-h-9 gap-2.5! flex-nowrap items-center px-4 py-1.5 text-ink transition-colors duration-[var(--m-fast)] focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-inset focus-visible:ring-ink ${
+            className={`grow min-w-0 min-h-9 gap-2.5! flex-nowrap items-center px-4 py-1.5 text-ink focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-inset focus-visible:ring-ink ${
-                current ? "cursor-default" : "cursor-pointer hover:bg-sunken"
+                current ? "cursor-default" : "cursor-pointer"
             }`}
         >
             <Check size={16} aria-hidden className={`shrink-0 ${current ? "" : "invisible"}`} />
             <Stack className="min-w-0 gap-0!">
                 <Span className="text-sm leading-5 text-ink truncate w-full">
-                    {computer.name || `Computer ${computer.id}`}
+                    {name}
                 </Span>
                 {computer.description ? (
                     <Span className="text-[13px] leading-[18px] text-ink-secondary truncate w-full">
@@ -46,5 +54,17 @@ export default function BotComputerOption({ computer, current, onPick }: Props)
                 ) : null}
             </Stack>
         </Row>
+        <Button
+            type="button"
+            variant="ghost"
+            color="text"
+            title={`Open ${name}`}
+            aria-label={`Open ${name}`}
+            onClick={onOpen}
+            className="shrink-0 p-2! text-ink-secondary dark:text-ink-secondary hover:text-ink hover:dark:text-ink rounded-default focus-visible:ring-2 focus-visible:ring-ink"
+        >
+            <AppWindow size={16} aria-hidden />
+        </Button>
+        </Row>
     );
 }

src/pages/admin/bots/[bot_id]/(partials)/bot-computers-modal.tsx

@@ -16,10 +16,11 @@ type Props = {
 /**
  * The header's computer icon, beside the model icon: opens what the bot's
  * computers are, the one it's on checked, and with more than one,
- * switches it.
+ * switches it. Each can be opened in the viewer.
  */
 export default function BotComputersModal({ bot }: Props) {
-    const { computers, switching_id, error, switchComputer } = useContext(BotComputersContext);
+    const { computers, switching_id, error, switchComputer, openComputer } =
+        useContext(BotComputersContext);
     const [open, setOpen] = useState(false);
 
     const current = computers[0];
@@ -48,7 +49,7 @@ export default function BotComputersModal({ bot }: Props) {
                     {can_switch ? `${name}'s computers` : `${name}'s computer`}
                 </H2>
                 <Span className="text-[13px] leading-[18px] text-ink-secondary">
-                    {`A computer is a sandbox ${name} works in: a shell, a browser and files of its own. Its tasks run on the checked one.`}
+                    {`A computer is a sandbox ${name} works in: a shell, a browser and files of its own. Its tasks run on the checked one. Open one to use it yourself.`}
                 </Span>
                 <Span className="text-[13px] leading-[18px] text-ink-secondary">
                     {can_switch
@@ -69,6 +70,10 @@ export default function BotComputersModal({ bot }: Props) {
                         onPick={async () => {
                             await switchComputer({ computer_id: computer.id || 0 });
                         }}
+                        onOpen={() => {
+                            setOpen(false);
+                            openComputer({ computer_id: computer.id || 0, mode: "browser" });
+                        }}
                     />
                 ))}
             </Stack>

src/pages/admin/bots/[bot_id]/(partials)/bot-thinking-row.tsx

@@ -1,5 +1,6 @@
 import Row from "@moduletrace/twui/layout/Row";
 import Span from "@moduletrace/twui/layout/Span";
+import Stack from "@moduletrace/twui/layout/Stack";
 import BotStone from "@/src/components/general/bot-stone";
 import type { BotHue } from "@/src/dict/bot-dict";
 import { BotReplyJobTitle, BotWaitingStatusLine } from "@/src/dict/bot-queue-dict";
@@ -7,28 +8,49 @@ import type { BotWithLastMessage } from "@/src/types";
 
 type Props = {
     bot: BotWithLastMessage;
+    /**
+     * The latest step of the bot's active task.
+     */
+    progress?: string | null;
 };
 
 /**
- * Under the last message while the bot is working on a reply (or waiting
+ * Under the last message while the bot is working on a reply or a task
- * for a model to come back), so a slow reply doesn't look like none.
+ * (or waiting for a model to come back), so slow work doesn't look like
+ * none. A task shows its latest step, else its title.
  */
-export default function BotThinkingRow({ bot }: Props) {
+export default function BotThinkingRow({ bot, progress }: Props) {
+    const name = bot.name || "The bot";
     const thinking = bot.face == "working" && bot.status_line == BotReplyJobTitle;
+    const working = bot.face == "working" && !thinking;
     const waiting = bot.face == "asleep" && bot.status_line == BotWaitingStatusLine;
 
-    if (!thinking && !waiting) return null;
+    if (!thinking && !working && !waiting) return null;
+
+    const label = thinking
+        ? `${name} is thinking…`
+        : working
+          ? `${name} is working…`
+          : `${BotWaitingStatusLine}…`;
+    const detail = working ? progress || bot.status_line : null;
 
     return (
         <Row role="status" className="w-full gap-3! flex-nowrap items-center">
             <BotStone
                 hue={(bot.hue || null) as BotHue | null}
-                face={thinking ? "working" : "asleep"}
+                face={waiting ? "asleep" : "working"}
                 size={28}
             />
+            <Stack className="min-w-0 gap-0! items-start">
                 <Span className="text-sm leading-5 text-ink-secondary animate-pulse motion-reduce:animate-none">
-                {thinking ? `${bot.name || "The bot"} is thinking…` : `${BotWaitingStatusLine}…`}
+                    {label}
                 </Span>
+                {detail ? (
+                    <Span className="w-full text-xs leading-4 text-ink-secondary truncate">
+                        {detail}
+                    </Span>
+                ) : null}
+            </Stack>
         </Row>
     );
 }

src/pages/admin/bots/[bot_id]/(partials)/computer-viewer-modal.tsx

@@ -0,0 +1,98 @@
+import { useContext, useState } from "react";
+import Modal from "@moduletrace/twui/elements/Modal";
+import Button from "@moduletrace/twui/layout/Button";
+import H2 from "@moduletrace/twui/layout/H2";
+import Row from "@moduletrace/twui/layout/Row";
+import Span from "@moduletrace/twui/layout/Span";
+import Stack from "@moduletrace/twui/layout/Stack";
+import ComputerView from "@/src/components/general/computer-view";
+import type { BotWithLastMessage } from "@/src/types";
+import { BotComputersContext } from "../(data)/bot-computers-context";
+import sendBotMessage from "../(functions)/send-bot-message";
+
+type Props = {
+    bot: BotWithLastMessage;
+};
+
+/**
+ * The computer the user opened (from the computers modal, or a take-over
+ * the bot asked for), large, in its browser, terminal or code editor.
+ * After a take-over the user hands it back from here, which tells the
+ * bot in chat so it carries on.
+ */
+export default function ComputerViewerModal({ bot }: Props) {
+    const { computers, viewing, closeComputer } = useContext(BotComputersContext);
+    const [handing_back, setHandingBack] = useState(false);
+    const [error, setError] = useState<string>();
+
+    const computer = computers.find((c) => c.id == viewing?.computer_id);
+
+    if (!viewing || !computer?.id) return null;
+
+    const name = computer.name || `Computer ${computer.id}`;
+    const bot_name = bot.name || "the bot";
+
+    const handBack = async () => {
+        setHandingBack(true);
+        setError(undefined);
+
+        const res = await sendBotMessage({
+            bot_id: bot.id || 0,
+            text: `I'm done on ${name}. You can carry on.`,
+            media_ids: [],
+        }).catch(() => null);
+
+        setHandingBack(false);
+
+        if (res?.success) closeComputer();
+        else setError(res?.msg || "Couldn't tell the bot");
+    };
+
+    return (
+        <Modal
+            open
+            setOpen={(open) => {
+                if (!open) closeComputer();
+            }}
+            className="max-w-[1280px] h-[90vh] p-0 bg-paper flex flex-col overflow-hidden"
+        >
+            <Row className="w-full gap-3! flex-nowrap items-center pl-5 pr-14 py-3 border-0 border-b border-solid border-hairline">
+                <Stack className="min-w-0 gap-0! items-start">
+                    <H2 className="text-base! leading-6! tracking-normal! font-bold text-ink truncate w-full">
+                        {name}
+                    </H2>
+                    {viewing.take_over ? (
+                        <Span className="text-[13px] leading-[18px] text-ink-secondary">
+                            {`${bot_name} asked you to take over. Hand it back when you're done.`}
+                        </Span>
+                    ) : null}
+                </Stack>
+                {viewing.take_over ? (
+                    <Stack className="ml-auto shrink-0 gap-1! items-end">
+                        <Button
+                            type="button"
+                            size="small"
+                            loading={handing_back}
+                            onClick={handBack}
+                            title={`Tell ${bot_name} you're done`}
+                        >
+                            {`Hand back to ${bot_name}`}
+                        </Button>
+                        {error ? (
+                            <Span role="alert" className="text-[13px] leading-[18px] text-ink-secondary">
+                                {error}
+                            </Span>
+                        ) : null}
+                    </Stack>
+                ) : null}
+            </Row>
+            <ComputerView
+                key={`${computer.id}-${viewing.mode}`}
+                computer_id={computer.id}
+                name={name}
+                default_mode={viewing.mode}
+                className="grow"
+            />
+        </Modal>
+    );
+}

src/pages/admin/bots/[bot_id]/(partials)/markdown-link.tsx

@@ -1,11 +1,14 @@
 import type { ComponentProps } from "react";
 import Link from "@moduletrace/twui/layout/Link";
-import { ComputerRefHrefPrefix } from "@/src/utils/link-computer-refs";
+import { ComputerViewModes, type ComputerViewMode } from "@/src/dict/computer-dict";
+import { ComputerRefHrefPrefix, TakeOverRefHrefPrefix } from "@/src/utils/link-computer-refs";
 import ComputerRefChip from "./computer-ref-chip";
+import TakeOverChip from "./take-over-chip";
 
 /**
  * A link in a bot message. A computer the bot named (`linkComputerRefs`)
- * becomes its switch button; anything else opens in a new tab.
+ * becomes its switch button, a take-over request a button that opens the
+ * computer; anything else opens in a new tab.
  */
 export default function MarkdownLink({ href, children }: ComponentProps<"a">) {
     if (href?.startsWith(ComputerRefHrefPrefix)) {
@@ -14,6 +17,21 @@ export default function MarkdownLink({ href, children }: ComponentProps<"a">) {
         if (computer_id) return <ComputerRefChip computer_id={computer_id} />;
     }
 
+    if (href?.startsWith(TakeOverRefHrefPrefix)) {
+        const [id, mode] = href.slice(TakeOverRefHrefPrefix.length).split("-");
+        const computer_id = Number(id);
+        const known = ComputerViewModes.some((m) => m.value == mode);
+
+        if (computer_id) {
+            return (
+                <TakeOverChip
+                    computer_id={computer_id}
+                    mode={(known ? mode : ComputerViewModes[0].value) as ComputerViewMode}
+                />
+            );
+        }
+    }
+
     return (
         <Link
             href={href}

src/pages/admin/bots/[bot_id]/(partials)/message-thread.tsx

@@ -7,14 +7,19 @@ import MessageGroupBlock from "./message-group-block";
 type Props = {
     bot: BotWithLastMessage;
     messages: BotMessageWithAttachments[];
+    /**
+     * The latest step of the bot's active task, shown while it works.
+     */
+    progress?: string | null;
 };
 
 /**
  * Bottom-anchored: the stack sits 24px above the composer rule and grows
  * upward. Empty space belongs at the top. New messages announce politely.
- * While the bot works on a reply, a thinking row sits under the last one.
+ * While the bot works on a reply or a task, a status row sits under the
+ * last one.
  */
-export default function MessageThread({ bot, messages }: Props) {
+export default function MessageThread({ bot, messages, progress }: Props) {
     const groups = groupMessages({ messages });
 
     return (
@@ -28,7 +33,7 @@ export default function MessageThread({ bot, messages }: Props) {
                 {groups.map((group) => (
                     <MessageGroupBlock key={group.key} bot={bot} group={group} />
                 ))}
-                <BotThinkingRow bot={bot} />
+                <BotThinkingRow bot={bot} progress={progress} />
             </Stack>
         </Stack>
     );

src/pages/admin/bots/[bot_id]/(partials)/take-over-chip.tsx

@@ -0,0 +1,43 @@
+import { useContext } from "react";
+import Button from "@moduletrace/twui/layout/Button";
+import Span from "@moduletrace/twui/layout/Span";
+import { Hand } from "lucide-react";
+import { ComputerViewModes, type ComputerViewMode } from "@/src/dict/computer-dict";
+import { BotComputersContext } from "../(data)/bot-computers-context";
+
+type Props = {
+    computer_id: number;
+    mode: ComputerViewMode;
+};
+
+/**
+ * The bot asked the user to take over a computer (`takeOverRef`): a
+ * button that opens it in the viewer, in the mode the bot asked for. A
+ * computer the bot can't use (any more) is shown by name only.
+ */
+export default function TakeOverChip({ computer_id, mode }: Props) {
+    const { computers, openComputer } = useContext(BotComputersContext);
+
+    const computer = computers.find((c) => c.id == computer_id);
+    const name = computer?.name || `computer ${computer_id}`;
+    const mode_title = ComputerViewModes.find((m) => m.value == mode)?.title || mode;
+
+    if (!computer) {
+        return <Span className="font-bold">{name}</Span>;
+    }
+
+    return (
+        <Button
+            type="button"
+            variant="outlined"
+            color="gray"
+            size="smaller"
+            title={`Open ${name} (${mode_title}) and take over`}
+            onClick={() => openComputer({ computer_id, mode, take_over: true })}
+            className="inline-flex h-6 gap-1.5! flex-nowrap align-middle px-2! text-[13px] font-bold text-ink whitespace-nowrap"
+        >
+            <Hand size={14} aria-hidden className="shrink-0" />
+            {`Take over ${name} · ${mode_title}`}
+        </Button>
+    );
+}

src/pages/admin/bots/[bot_id]/(sections)/conversation.tsx

@@ -7,15 +7,18 @@ import useLiveBots from "@/src/hooks/use-live-bots";
 import useServerEvent from "@/src/hooks/use-server-event";
 import { BotComputersContext } from "../(data)/bot-computers-context";
 import useBotComputers from "../(hooks)/use-bot-computers";
+import useTaskProgress from "../(hooks)/use-task-progress";
 import ConversationHeader from "../(partials)/conversation-header";
 import MessageThread from "../(partials)/message-thread";
 import Composer from "../(partials)/composer";
+import ComputerViewerModal from "../(partials)/computer-viewer-modal";
 
 /**
  * The conversation column: header, bottom-anchored thread, composer, the
  * thread and composer held to a readable width.
  * The bot's computers are shared with the header's modal and the computer
- * buttons in the thread, so a switch from either shows in both.
+ * buttons in the thread, so a switch from either shows in both; either
+ * can open one in the viewer.
  */
 export default function Conversation() {
     const { pageProps } = useContext(AppContext);
@@ -25,20 +28,32 @@ export default function Conversation() {
         pageProps?.bot_messages || [],
     );
 
-    const bots = useLiveBots({ bots: pageProps?.bots || [], open_bot_id: bot?.id || undefined });
+    const bots = useLiveBots({
+        bots: pageProps?.bots || [],
+        open_bot_id: bot?.id || undefined,
+    });
     const live_bot = bots.find((b) => b.id == bot?.id) || bot;
     const bot_computers = useBotComputers({
         bot_id: bot?.id || 0,
         computers: pageProps?.bot_computers || [],
     });
 
+    const progress = useTaskProgress({
+        bot_id: bot?.id || 0,
+        task: pageProps?.bot_task || null,
+    });
+
+    console.log(`Refreshing ...`);
+
     useServerEvent({
         event: "server:bot-message",
         onData: (message: BotMessageWithAttachments) => {
             if (message.bot_id != bot?.id) return;
 
             setMessages((prev) =>
-                prev.some((m) => m.id == message.id) ? prev : [...prev, message],
+                prev.some((m) => m.id == message.id)
+                    ? prev
+                    : [...prev, message],
             );
 
             if (message.author == "bot") {
@@ -59,16 +74,23 @@ export default function Conversation() {
                     bot={live_bot}
                     harnesses={pageProps?.bot_harness_overview?.harnesses}
                 />
-                <MessageThread bot={live_bot} messages={messages} />
+                <MessageThread
+                    bot={live_bot}
+                    messages={messages}
+                    progress={progress}
+                />
                 <Composer
                     bot={bot}
                     onSent={({ message }) =>
                         setMessages((prev) =>
-                            prev.some((m) => m.id == message.id) ? prev : [...prev, message],
+                            prev.some((m) => m.id == message.id)
+                                ? prev
+                                : [...prev, message],
                         )
                     }
                 />
             </Stack>
+            <ComputerViewerModal bot={live_bot} />
         </BotComputersContext.Provider>
     );
 }

src/pages/admin/bots/[bot_id]/index.server.ts

@@ -1,5 +1,6 @@
 import userAuth from "@/src/functions/backend/auth/user-auth";
 import grabBots from "@/src/functions/backend/db/bots/grab-bots";
+import grabActiveBotTask from "@/src/functions/backend/db/bots/grab-active-bot-task";
 import grabBotMessages from "@/src/functions/backend/db/bots/grab-bot-messages";
 import markBotRead from "@/src/functions/backend/db/bots/mark-bot-read";
 import grabBotComputers from "@/src/functions/backend/db/computers/grab-bot-computers";
@@ -40,13 +41,14 @@ export const server: BunextPageServerFn<PagePropsType> = async (params) => {
      * The header's models modal: a worker's own settings, or for the
      * coordinator the defaults it runs on (without the providers).
      */
-    const [bot_messages, bot_harness_overview, bot_computers] = await Promise.all([
+    const [bot_messages, bot_harness_overview, bot_computers, bot_task] = await Promise.all([
         grabBotMessages({ bot_id: bot.id }),
         (bot.kind == "coordinator"
             ? grabHarnessOverview().then(({ harnesses }) => ({ harnesses }))
             : grabBotHarnessOverview({ bot_id: bot.id })
         ).catch(() => null),
         grabBotComputers({ bot_id: bot.id, user_id: user.id }),
+        grabActiveBotTask({ bot_id: bot.id }),
     ]);
 
     return {
@@ -56,6 +58,7 @@ export const server: BunextPageServerFn<PagePropsType> = async (params) => {
             bot_messages,
             bot_harness_overview,
             bot_computers,
+            bot_task,
         },
     };
 };

src/server.ts

@@ -8,6 +8,8 @@ import { logError, grabRequestIP } from "./functions/backend/logger";
 import botsQueue from "./bots-queue";
 import liveRelay from "./live-relay";
 import enableForeignKeys from "./functions/backend/db/enable-foreign-keys";
+import handleComputerProxy from "./functions/server/computers/handle-computer-proxy";
+import { ComputerProxyPrefix } from "./dict/computer-dict";
 
 declare global {}
 
@@ -32,6 +34,10 @@ const server = Bun.serve({
                 }
             }
 
+            if (url.pathname.startsWith(`${ComputerProxyPrefix}/`)) {
+                return await handleComputerProxy({ req, server });
+            }
+
             if (url.pathname === WEBSOCKET_ENDPOINT) {
                 const session = await userAuth({ req });
 

src/types/index.ts

@@ -277,6 +277,10 @@ export type PagePropsType = {
      * The open bot's computers, the one it's on first.
      */
     bot_computers?: BUN_SQLITE_COBBLE_COMPUTERS[] | null;
+    /**
+     * The open bot's active task and what its job has done so far.
+     */
+    bot_task?: BotTaskWithSteps | null;
     environment?: string;
     has_password?: boolean;
     backups?: DbBackup[] | null;

src/utils/computer-proxy-headers.test.ts

@@ -0,0 +1,68 @@
+import { describe, expect, test } from "bun:test";
+import { AppData } from "@/src/data/app-data";
+import computerProxyRequestHeaders from "./computer-proxy-request-headers";
+import computerProxyResponseHeaders from "./computer-proxy-response-headers";
+
+describe("computerProxyRequestHeaders", () => {
+    test("drops the app's session cookies and forwards where it's served", () => {
+        const headers = computerProxyRequestHeaders({
+            headers: new Headers({
+                host: "localhost:7227",
+                cookie: `${AppData.AuthKeyCookieName}=secret; ${AppData.AuthCSRFCookieName}=csrf; _xsrf=keep`,
+                "user-agent": "test",
+            }),
+            url: new URL("http://localhost:7227/computers/3/terminal"),
+            prefix: "/computers/3",
+        });
+
+        expect(headers.get("cookie")).toBe("_xsrf=keep");
+        expect(headers.get("host")).toBeNull();
+        expect(headers.get("user-agent")).toBe("test");
+        expect(headers.get("x-forwarded-host")).toBe("localhost:7227");
+        expect(headers.get("x-forwarded-proto")).toBe("http");
+        expect(headers.get("x-forwarded-prefix")).toBe("/computers/3");
+    });
+
+    test("sends no cookie header when only the app's were set", () => {
+        const headers = computerProxyRequestHeaders({
+            headers: new Headers({ cookie: `${AppData.AuthKeyCookieName}=secret` }),
+            url: new URL("https://cobble.local/computers/3/"),
+            prefix: "/computers/3",
+        });
+
+        expect(headers.get("cookie")).toBeNull();
+        expect(headers.get("x-forwarded-proto")).toBe("https");
+    });
+});
+
+describe("computerProxyResponseHeaders", () => {
+    const base_url = "http://cobble-computer:8080";
+    const prefix = "/computers/3";
+
+    test("lets the chat frame the computer", () => {
+        const headers = computerProxyResponseHeaders({
+            headers: new Headers({
+                "x-frame-options": "DENY",
+                "content-security-policy": "default-src 'self'; frame-ancestors 'none'",
+            }),
+            base_url,
+            prefix,
+        });
+
+        expect(headers.get("x-frame-options")).toBeNull();
+        expect(headers.get("content-security-policy")).toBe("default-src 'self'");
+    });
+
+    test("points redirects back under the prefix", () => {
+        const location = (value: string) =>
+            computerProxyResponseHeaders({ headers: new Headers({ location: value }), base_url, prefix }).get(
+                "location",
+            );
+
+        expect(location("http://cobble-computer:8080/jupyter/lab")).toBe("/computers/3/jupyter/lab");
+        expect(location("/code-server?folder=/home/gem")).toBe("/computers/3/code-server?folder=/home/gem");
+        expect(location("/computers/3/terminal")).toBe("/computers/3/terminal");
+        expect(location("./?folder=/home/gem")).toBe("./?folder=/home/gem");
+        expect(location("https://example.com/")).toBe("https://example.com/");
+    });
+});

src/utils/computer-proxy-request-headers.ts

@@ -0,0 +1,49 @@
+import { AppData } from "@/src/data/app-data";
+
+type Params = {
+    headers: Headers;
+    /**
+     * The request's own address, as the user's browser sees it.
+     */
+    url: URL;
+    /**
+     * Where the computer is served, e.g. `/computers/3`.
+     */
+    prefix: string;
+};
+
+/**
+ * Headers hop-by-hop or tied to the app's own connection, never sent on.
+ */
+const Dropped = ["host", "connection", "keep-alive", "upgrade", "content-length", "transfer-encoding"];
+
+/**
+ * The headers a request is sent to a computer with: the user's own,
+ * without the app's session cookies, plus the forwarded host, protocol
+ * and prefix, which the sandbox builds its links (and its browser's CDP
+ * address) from.
+ */
+export default function computerProxyRequestHeaders({ headers, url, prefix }: Params) {
+    const out = new Headers(headers);
+
+    for (const name of Dropped) out.delete(name);
+
+    const cookies = (headers.get("cookie") || "")
+        .split(";")
+        .map((c) => c.trim())
+        .filter(Boolean)
+        .filter((c) => {
+            const name = c.split("=")[0];
+
+            return name != AppData.AuthKeyCookieName && name != AppData.AuthCSRFCookieName;
+        });
+
+    if (cookies.length) out.set("cookie", cookies.join("; "));
+    else out.delete("cookie");
+
+    out.set("x-forwarded-host", headers.get("x-forwarded-host") || url.host);
+    out.set("x-forwarded-proto", headers.get("x-forwarded-proto") || url.protocol.replace(":", ""));
+    out.set("x-forwarded-prefix", prefix);
+
+    return out;
+}

src/utils/computer-proxy-response-headers.ts

@@ -0,0 +1,53 @@
+type Params = {
+    headers: Headers;
+    /**
+     * The computer's own address, e.g. `http://cobble-computer:8080`.
+     */
+    base_url: string;
+    /**
+     * Where the computer is served, e.g. `/computers/3`.
+     */
+    prefix: string;
+};
+
+/**
+ * Headers that would stop the chat from showing the computer in a frame,
+ * or that no longer fit the body (fetch has already decoded it).
+ */
+const Dropped = ["x-frame-options", "content-encoding", "content-length", "transfer-encoding", "connection"];
+
+/**
+ * The headers a computer's response goes back with: framing allowed, and
+ * redirects pointed back under the computer's prefix.
+ */
+export default function computerProxyResponseHeaders({ headers, base_url, prefix }: Params) {
+    const out = new Headers(headers);
+
+    for (const name of Dropped) out.delete(name);
+
+    const csp = out.get("content-security-policy");
+
+    if (csp?.includes("frame-ancestors")) {
+        out.set(
+            "content-security-policy",
+            csp
+                .split(";")
+                .filter((d) => !d.trim().startsWith("frame-ancestors"))
+                .join(";"),
+        );
+    }
+
+    const location = out.get("location");
+
+    if (location) {
+        const base = base_url.replace(/\/+$/, "");
+
+        if (location.startsWith(base)) {
+            out.set("location", `${prefix}${location.slice(base.length) || "/"}`);
+        } else if (location.startsWith("/") && !location.startsWith("//") && !location.startsWith(`${prefix}/`)) {
+            out.set("location", `${prefix}${location}`);
+        }
+    }
+
+    return out;
+}

src/utils/computer-view-url.ts

@@ -0,0 +1,20 @@
+import {
+    ComputerProxyPrefix,
+    ComputerViewModes,
+    type ComputerViewMode,
+} from "@/src/dict/computer-dict";
+
+type Params = {
+    computer_id: number | string;
+    mode: ComputerViewMode;
+};
+
+/**
+ * The app address of one of a computer's modes, e.g.
+ * `/computers/3/terminal`.
+ */
+export default function computerViewUrl({ computer_id, mode }: Params) {
+    const path = ComputerViewModes.find((m) => m.value == mode)?.path || "";
+
+    return `${ComputerProxyPrefix}/${computer_id}/${path}`;
+}

src/utils/link-computer-refs.test.ts

@@ -1,6 +1,7 @@
 import { describe, expect, test } from "bun:test";
 import computerRef from "./computer-ref";
 import linkComputerRefs from "./link-computer-refs";
+import takeOverRef from "./take-over-ref";
 
 describe("linkComputerRefs", () => {
     test("links every computer a message names", () => {
@@ -16,4 +17,12 @@ describe("linkComputerRefs", () => {
 
         expect(linkComputerRefs({ text })).toBe(text);
     });
+
+    test("links take-over requests", () => {
+        expect(
+            linkComputerRefs({
+                text: `Please log in to Gmail. ${takeOverRef({ id: 3, mode: "browser" })}`,
+            }),
+        ).toBe("Please log in to Gmail. [take over](#take-over-3-browser)");
+    });
 });

src/utils/link-computer-refs.ts

@@ -6,13 +6,25 @@
 export const ComputerRefHrefPrefix = "#computer-";
 
 /**
- * Turns every computer ref in a message into a markdown link to
+ * The link target a take-over ref (`takeOverRef`) becomes, e.g.
- * `#computer-<id>`, so it survives markdown rendering as a link the chat
+ * `#take-over-3-browser`. The chat renders these as buttons that open the
- * can swap for a switch button.
+ * computer in that mode.
+ */
+export const TakeOverRefHrefPrefix = "#take-over-";
+
+/**
+ * Turns every computer ref (`[computer:3]`) and take-over ref
+ * (`[take-over:3:browser]`) in a message into a markdown link, so it
+ * survives markdown rendering as a link the chat can swap for a button.
  */
 export default function linkComputerRefs({ text }: { text: string }) {
-    return text.replace(
+    return text
+        .replace(
             /\[computer:(\d+)\](?!\()/g,
             (_, id: string) => `[computer ${id}](${ComputerRefHrefPrefix}${id})`,
+        )
+        .replace(
+            /\[take-over:(\d+):([a-z]+)\](?!\()/g,
+            (_, id: string, mode: string) => `[take over](${TakeOverRefHrefPrefix}${id}-${mode})`,
         );
 }

src/utils/parse-computer-proxy-path.test.ts

@@ -0,0 +1,24 @@
+import { describe, expect, test } from "bun:test";
+import parseComputerProxyPath from "./parse-computer-proxy-path";
+
+describe("parseComputerProxyPath", () => {
+    test("splits the computer id from the path on it", () => {
+        expect(parseComputerProxyPath({ pathname: "/computers/3/v1/shell/ws" })).toEqual({
+            computer_id: 3,
+            prefix: "/computers/3",
+            rest: "v1/shell/ws",
+        });
+    });
+
+    test("takes the computer's root with or without a slash", () => {
+        expect(parseComputerProxyPath({ pathname: "/computers/3" })?.rest).toBe("");
+        expect(parseComputerProxyPath({ pathname: "/computers/3/" })?.rest).toBe("");
+    });
+
+    test("ignores anything else", () => {
+        expect(parseComputerProxyPath({ pathname: "/computers" })).toBeNull();
+        expect(parseComputerProxyPath({ pathname: "/computers/x/terminal" })).toBeNull();
+        expect(parseComputerProxyPath({ pathname: "/computers/0/terminal" })).toBeNull();
+        expect(parseComputerProxyPath({ pathname: "/computersx/3" })).toBeNull();
+    });
+});

src/utils/parse-computer-proxy-path.ts

@@ -0,0 +1,22 @@
+import { ComputerProxyPrefix } from "@/src/dict/computer-dict";
+
+type Params = {
+    pathname: string;
+};
+
+/**
+ * Splits `/computers/<id>/<rest>` into the computer id and the path on
+ * the computer (no leading slash). Null for anything else.
+ */
+export default function parseComputerProxyPath({ pathname }: Params) {
+    const match = pathname.match(new RegExp(`^${ComputerProxyPrefix}/(\\d+)(?:/(.*))?$`));
+    const computer_id = Number(match?.[1]);
+
+    if (!match || !computer_id) return null;
+
+    return {
+        computer_id,
+        prefix: `${ComputerProxyPrefix}/${computer_id}`,
+        rest: match[2] || "",
+    };
+}

src/utils/take-over-ref.ts

@@ -0,0 +1,15 @@
+import type { ComputerViewMode } from "@/src/dict/computer-dict";
+
+type Params = {
+    id: number | string;
+    mode: ComputerViewMode;
+};
+
+/**
+ * How a bot asks the user to take over one of its computers in a message:
+ * `[take-over:3:browser]`. The chat shows it as a button that opens the
+ * computer in that mode.
+ */
+export default function takeOverRef({ id, mode }: Params) {
+    return `[take-over:${id}:${mode}]`;
+}

src/websocket/index.ts

@@ -10,6 +10,7 @@ import type { WebSocketDataType } from "@/src/types";
 import { wsSendData } from "./(functions)/ws-send-data";
 import checkSessionAccess from "./(functions)/check-session-access";
 import logEvent from "@/src/functions/backend/logger";
+import { computerProxySocket } from "@/src/functions/server/computers/computer-proxy-socket";
 
 /**
  * The twui `useWebSocket` hook sends a raw keepalive string to keep the
@@ -27,6 +28,8 @@ function parseMessage(message: string | Buffer): WebSocketDataType | undefined {
 
 export const websocket = {
     open(ws: ServerWebSocket<WebSocketSession>) {
+        if (ws.data.proxy) return computerProxySocket.open(ws);
+
         const { user } = ws.data;
 
         joinRoom(ws, "public");
@@ -54,6 +57,8 @@ export const websocket = {
         ws: ServerWebSocket<WebSocketSession>,
         message: string | Buffer,
     ) {
+        if (ws.data.proxy) return computerProxySocket.message(ws, message);
+
         const data = parseMessage(message);
 
         if (!data) return;
@@ -73,6 +78,8 @@ export const websocket = {
     },
 
     close(ws: ServerWebSocket<WebSocketSession>) {
+        if (ws.data.proxy) return computerProxySocket.close(ws);
+
         leaveAllRooms(ws);
     },
 

src/websocket/types.ts

@@ -4,4 +4,20 @@ import type { BUN_SQLITE_COBBLE_USER_TYPES } from "@/db/types/db";
 export type WebSocketSession = {
     user: User;
     user_types: BUN_SQLITE_COBBLE_USER_TYPES[];
+    /**
+     * Set on a socket bridged to a computer, never on the app's own `/ws`.
+     */
+    proxy?: ComputerProxySession;
+};
+/**
+ * A websocket bridged to a computer (`computerProxySocket`): the address
+ * on the computer and the headers to open it with. `upstream` and
+ * `pending` are filled once it opens.
+ */
+export type ComputerProxySession = {
+    url: string;
+    headers: Record<string, string>;
+    protocols: string[];
+    upstream?: WebSocket;
+    pending?: (string | Buffer)[];
 };