Release v0.7.6

Co-Authored-By: Claude Opus 5.5 <[email protected]>

AGENTS.md

@@ -60,7 +60,7 @@ Enables SQLite foreign keys (`enableForeignKeys`), `bunextInit()`, `ensureDocker
 - **Passwords:** bcrypt in `user_passwords` (never in `users`; blocked from CRUD via `src/dict/private-tables.ts`).
 - **User types** (`src/dict/user-types-dict.ts`, the only definition): `super_admin` (user #1), `admin` (only the super admin grants it), `member` (default), `read_only` (non-GET CRUD blocked), `revoked`. A user needs at least one type to sign in.
 - **RBAC:** `src/utils/check-user-access.ts` plus the matrix engine in `src/utils/rbac/` (`userTypeAssignment` over `UserTypeAssignmentMatrix`/`UserTypeDeletionMatrix`).
-- **Audit:** `src/functions/backend/logger.ts` (`logEvent`/`logError` -> `logs`). Auth events, cron runs, every unexpected API error (`safeErrorMessage` -> `logUnexpectedError`, not for `AppError`), every failed or lost bot job. WebSocket connects are not logged.
+- **Audit:** `src/functions/backend/logger.ts` (`logEvent`/`logError` -> `logs`). Auth events, cron runs, every unexpected API error (`safeErrorMessage` -> `logUnexpectedError`, not for `AppError`), every failed or lost bot job. WebSocket connects are not logged. Kept for the app setting `logs_keep_days` (default `AppData.LogsKeepDays`), deleted by the daily `prune-logs`.
 
 ### WebSocket
 
@@ -68,7 +68,7 @@ Enables SQLite foreign keys (`enableForeignKeys`), `bunextInit()`, `ensureDocker
 
 ### Cron
 
-`src/cron/index.ts` registers everything via `registerCronJob` (`src/cron/register-cron-job.ts`); one folder per task. In-process `Bun.cron`, UTC, never overlapping. Notable: `heartbeat`, `sync-harness-models` (hourly), `run-cron-jobs` (every minute, fires schedules), `sync-computers`, `sync-mcp-servers`, `prune-bots-queue`, `prune-chatrooms`.
+`src/cron/index.ts` registers everything via `registerCronJob` (`src/cron/register-cron-job.ts`); one folder per task. In-process `Bun.cron`, UTC, never overlapping. Notable: `heartbeat`, `sync-harness-models` (hourly), `run-cron-jobs` (every minute, fires schedules), `sync-computers`, `sync-mcp-servers`, `clear-job-steps`, `prune-bots-queue`, `prune-chatrooms`, `prune-logs`.
 
 ### Admin CRUD API
 
@@ -88,16 +88,16 @@ Enables SQLite foreign keys (`enableForeignKeys`), `bunextInit()`, `ensureDocker
 ### Bots, sessions, chat
 
 - **Tables:** `bots` (`kind`, `color` `#rrggbb`, `stone`, `role`, `system_prompt`; `hue` is a legacy fallback), `bot_user_faces`, `bot_sessions`, `bot_messages`, `bot_tasks`, `bot_task_steps` (unused yet). Enums in `src/dict/bot-dict.ts`.
-- **Sessions:** one thread per user per bot in its own chat plus one per workspace; split into `bot_sessions` (rollover at the app setting `bot_session_max_messages`, default `AppData.BotSessionMaxMessages`, via `grabBotSessionMaxMessages`; titled and summarised by `name-bot-session.ts` from the session's opening (`grabSessionOpening`; `grabBotMessages` reads the latest), summary carried into the next session only when they're linked by `previous_session_id`, which only a carry-over sets; older implicit links are cleared at boot by `clearImplicitSessionLinks`). New session (`startBotSession`) and a conversation with no active session always start fresh. The rollover carries over like the bot does (`rollOverBotSession` -> `continueBotSession`, falling back to the stored summary; kept for the next message while a job is open in the full session or there's nothing to carry), inside send-message, so the user's message lands under the summary. Helpers in `src/functions/backend/db/bot-sessions/`. Job results land in the job's own session, even after it closes, except a reply that called `continue_in_new_session` (`carryOverBotSession`; the menu's and sessions modal's Continue in new session does the same through `continueBotSession`, the model writing title and summary, refused while a job is open in the session): its summary is kept on the closed session and opens the new one as a bot message (`carried_from_session_id`), the job's `session_id` moves so the answer follows, the new session is titled by the bot then (it has no first exchange to be named from), and the old one can never be reopened (`reopenBotSession` refuses; a thread showing it swaps its composer for a link to the new one). Session lists (rail, sessions modal) reopen a session on click, as before; a carried-over one can't be, so it opens to read (`?session_id=`, `conversationHref`). The page tells them which session it shows (`announceShownSession` -> `useShownSession`): they reload, mark it Current (greyed out; without `?session_id=` the active one is), list it even when it's older than the rail's latest, and the active row then leads back. From half the session length the reply prompt suggests moving on (`longSessionNote`). Earlier sessions never enter the prompt (only the last one's summary): the thread can open them above a session (`useOlderMessages.loadPrevious`, a divider per session, quote and reply as usual), and replies get `search_past_sessions`/`read_past_session` (scoped to that user, bot and place, never the current session) when the conversation has any.
+- **Sessions:** one thread per user per bot in its own chat plus one per workspace; split into `bot_sessions` (rollover at the app setting `bot_session_max_messages`, default `AppData.BotSessionMaxMessages`, via `grabBotSessionMaxMessages`; titled and summarised by `name-bot-session.ts` from the session's opening after a reply or a task (a session with no user message, work handed to the bot, is named from the bot's own messages; `grabSessionOpening`; `grabBotMessages` reads the latest), summary carried into the next session only when they're linked by `previous_session_id`, which only a carry-over sets; older implicit links are cleared at boot by `clearImplicitSessionLinks`). New session (`startBotSession`) and a conversation with no active session always start fresh. The rollover carries over like the bot does (`rollOverBotSession` -> `continueBotSession`, falling back to the stored summary; kept for the next message while a job is open in the full session or there's nothing to carry), inside send-message, so the user's message lands under the summary. Helpers in `src/functions/backend/db/bot-sessions/`. Job results land in the job's own session, even after it closes, except a reply that called `continue_in_new_session` (`carryOverBotSession`; the menu's and sessions modal's Continue in new session does the same through `continueBotSession`, the model writing title and summary, refused while a job is open in the session): its summary is kept on the closed session and opens the new one as a bot message (`carried_from_session_id`), the job's `session_id` moves so the answer follows, the new session is titled by the bot then (it has no first exchange to be named from), and the old one can never be reopened (`reopenBotSession` refuses; a thread showing it swaps its composer for a link to the new one). Session lists (rail, sessions modal) reopen a session on click, as before; a carried-over one can't be, so it opens to read (`?session_id=`, `conversationHref`). The page tells them which session it shows (`announceShownSession` -> `useShownSession`): they reload, mark it Current (greyed out; without `?session_id=` the active one is), list it even when it's older than the rail's latest, and the active row then leads back. From half the session length the reply prompt suggests moving on (`longSessionNote`). Earlier sessions never enter the prompt (only the last one's summary): the thread can open them above a session (`useOlderMessages.loadPrevious`, a divider per session, quote and reply as usual), and replies get `search_past_sessions`/`read_past_session` (scoped to that user, bot and place, never the current session) when the conversation has any.
 - **Coordinator:** exactly one, the super admin's alone, created idempotently by `ensureCoordinatorBot`. Default prompt `src/data/coordinator-system-prompt.ts`, backfilled only when empty.
 - **Routes:** `/admin/bots` reopens the last visited bot (cookie `cobble_last_visited_bot_id`, never a workspace conversation) if still reachable, else coordinator (super admin) / latest bot / first workspace / empty. `/admin/bots/[bot_id]` conversation (header, thread and composer at 760px, no right panel; `?session_id=` opens another of the user's sessions there: a closed one is reopened, except a carried-over one, which shows with `SessionMovedBar` instead of the composer, pointing where the conversation went on); `/admin/bots/[bot_id]/models`; `/admin/workspaces/[id]` dashboard; `/admin/workspaces/[id]/chatrooms` that workspace's chatrooms; `/admin/workspaces/[id]/bots/[bot_id]` same conversation scoped to the workspace.
-- **Conversation loading:** pages render everything server-side (`grabConversationProps`) **except the history**, which the client loads (`api/admin/bots/conversation-history`, merged with what's already shown, refetched on bfcache restore) and pages older messages on scroll. Rail lists go through `railBots`/`railWorkspaces` (no prompts/memory, previews cut). While a reply is due the page polls `session-messages` (`use-reply-catch-up`) in case a push was missed.
+- **Conversation loading:** pages render everything server-side (`grabConversationProps`) **except the history**, which the client loads (`api/admin/bots/conversation-history`, merged with what's already shown, refetched on bfcache restore) and pages older messages on scroll. Rail lists go through `railBots`/`railWorkspaces` (no prompts/memory, previews cut). While a reply is due (read with the live face, never the page's) the page polls `session-messages` (`use-reply-catch-up`, also at once when the page comes back into view or online) in case a push was missed, replaying the messages and the bot's face and task as their pushes.
 - **Sending:** `api/admin/bots/send-message.ts` -> `enqueueBotReply` -> `wakeBotsQueue`. **Never import `src/bots-queue/index.ts` from a page or API route** (pulls the bunext runtime into page bundles). Optimistic bubble in `pending_messages` until the saved message replaces it.
 - **Composer:** Tiptap editor reading/writing markdown, drafts in `localStorage`, `@` bots / `#` workspaces written as `[bot:ID]`/`[workspace:ID]`. Enter sends (Mod+Enter in lists/code). `@tiptap/react` needs `use-sync-external-store`; the app ships an ESM stand-in in `packages/use-sync-external-store-esm` (file dependency + override; the Dockerfile copies `packages/` before `bun install`).
 - **Rendering:** markdown via `react-markdown` + `remark-gfm`, raw HTML never rendered, component map in `[bot_id]/(data)/message-markdown-components.tsx`. Inline refs, all linked by `link-computer-refs.ts` and rendered by chips in `(partials)/`: `[bot:ID]`, `[workspace:ID]`, `[computer:ID]`, `[file:ID]` (a saved file), `[folder:<volume>:<owner>:<path>]` (opens `/admin/files`), `[take-over:ID:mode]`, `[proposal:ID]`, `[hook-proposal:ID]`, `[secret:ID]` (masked env, Reveal/Copy), `[secret-request:ID]` (masked input card), `[graphic:ID]` (a bot graphic, framed inline), `[room:ID]` (a chatroom). Parsers in `src/utils/*-ref.ts`. A ref quoted alone in a code span still links; one inside longer code stays literal (`linkComputerRefs` splits out code first). A file chip opens a file from another bot's folder for a user who reaches that folder (`userReachesFileScope`, the same rule as its bytes), so answers passed on from chatrooms keep working.
 - **Copy chat** (header menu): the active session (the one on screen) as one markdown document (`api/admin/bots/conversation-export` -> `conversationMarkdown`, refs as names, message headings demoted). The clipboard write starts on the click with a pending `ClipboardItem` (`copyPendingText`), since Safari drops the gesture after a fetch.
 - **Replies/reactions:** `reply_to_message_id` (not an FK); reactions from the fixed `BotMessageReactions`, toggled by `react-message.ts` and the `react_to_message` tool, pushed as `server:bot-reaction`.
-- **Attachments:** uploaded on pick (`upload-attachment.ts`, private `media`, text extracted when possible), linked via `bot_message_media`. Replies read files of the last `BotReplyAttachmentMessages` messages. Tasks don't get files yet.
+- **Attachments:** uploaded on pick (`upload-attachment.ts`, private `media`, no text kept: a reply reads a document's text from the file), linked via `bot_message_media`. Replies read files of the last `BotReplyAttachmentMessages` messages. Tasks don't get files yet.
 - **Thinking row / stop:** `bot-busy-state.ts` decides the row's line and whether Stop shows. Stop (`api/admin/bots/stop.ts`, `stop-bot-jobs`) fails every open job of that bot for the user, aborts the running handler (`job-abort-registry`) and drops late face updates.
 - **Voice:** mic -> `api/admin/bots/transcribe.ts`; play -> `GET speak-message` streamed into `<audio>`. Nothing recorded is stored. Both disabled with a note when the harness has no model.
 - **Pictures:** `image_media_id` (private media, `bot-image`/`workspace-image`), via `set-profile-image.ts`, the `set_picture` tool or `update_workspace`. Bytes are **copied**, never moved; only picture-paradigm rows are deleted on replace. Format kept by `imageOutputFormat`.
@@ -113,7 +113,7 @@ Enables SQLite foreign keys (`enableForeignKeys`), `bunextInit()`, `ensureDocker
 
 - Kinds (`BotJobKinds`): `reply` (chat answer, no task row) and `task` (long work, usually on a computer). bun-sqlite `isQueue` table. Helpers in `src/functions/backend/db/bots-queue/`: `enqueueBotJob`, `claimBotJobs` (kind-filtered atomic claim), `updateBotJob` (false once lost/cancelled), `completeBotJob`, `failBotJob` (backoff or fatal), `deferBotJob` (all models resting: attempt refunded, face `asleep`), `cancelBotJob`, `stop-bot-jobs`.
 - Handler `src/bots-queue/index.ts`: separate reply/task slots, heartbeat, per-attempt deadline (`AppData.BotJobAttemptTimeoutMs`), one task per computer and one reply per session (`createKeyLock`). Work in `run-bot-job.ts` -> `run-bot-reply.ts` / `run-bot-task.ts`. Throw `BotJobFatalError` for failures retrying can't fix.
-- Every model turn is a step in `bots_queue_updates` (tool names, never arguments; `describe-job-step.ts`, labels in `JobStepToolLabels`), pushed as `server:bot-job-update`.
+- Every model turn is a step in `bots_queue_updates` (tool names, never arguments; `describe-job-step.ts`, labels in `JobStepToolLabels`), pushed as `server:bot-job-update` by the live relay only. Steps feed the live working row alone: the `clear-job-steps` cron deletes them `AppData.BotJobStepsKeepMs` after the job ends, however it ended, keeping their count on `bots_queue.steps`. Never delete them the moment a job ends (the relay would miss the last ones).
 - **A bot can't vouch for itself:** a run's tool calls (`tool_uses`, names and ok, never arguments) go to the wrap-up turn when steps run out and onto the result message as `bot_messages.tool_log`; history follows each recorded bot message with an `[App record: ...]` line (`botMessagesToHarness`), and copies of that line in an answer are stripped (`stripAppRecords`).
 - **`start_task`/`delegate_task` jobs are runs, not tasks:** their results say `Run #<job id>` and that `list_tasks` doesn't show them (bots sent users to the tracker for them). `start_task` refuses a second open run in its session on the same computer unless `separate` (it would only wait behind it) and says how many runs are ahead on that computer.
 - **Replies take in messages sent mid-run** (`incoming`, `IncomingMessagesNote`, redraft); separate requests are acknowledged with `send_message` (interim). A queued reply finishes silently only when a finished reply already read every user message (`hasUnansweredMessage`), never by who wrote last.
@@ -137,7 +137,8 @@ A worker reaches another worker **only through the coordinator**: `ask_crew` (ev
 - **Browser profile:** `DATA_DIR/computer-browsers/<id>` mounted at `~/.cobble-browser`, which Chrome is pointed at (`BROWSER_EXTRA_ARGS=--user-data-dir=... --password-store=basic`), so logins, extensions and site permissions survive restarts and recreates. Never use the image's `~/.config/browser`: every boot overwrites its `Default/Preferences` with session-only cookies. Seed prefs (`computer-browser-preferences.ts`) are written only when missing; stale `Singleton*` links are removed before each provision. Bumping `AppData.ComputerWorkspaceLayout` recreates every computer once (`migrateComputerLayout`, skipped while a job runs on it). Default extensions (`ComputerBrowserExtensions`, `normal_installed`: can be turned off, not removed) come from a managed policy written as root into the running container (`syncComputerBrowserPolicy`, on provision and every `sync-computers` run), not a mount, so existing computers get new ones without a recreate; Chrome installs them live. Deleting a computer keeps its directory. Outside docker (dev) containers are reached by IP (`pickReachableAddress`).
 - One main computer (`ensureMainComputer`), whose coordinator grant is locked (`isLockedComputerAccess`). With `MAIN_COMPUTER_URL` set (dev) it isn't provisioned. Only super admins create/delete/retry computers.
 - A bot's effective computers (`grab-bot-computers`): the workspace's in a workspace; otherwise direct grants plus its workspaces' (coordinator: all); `bots.computer_id` first. Only a `running` (healthy) computer is reached (`grabComputerMcpUrl` `ready`, `computerNotReadyMessage`).
-- **Folders (enforced in tools, not a Linux user):** a run works only in `bots/<bot>/users/<user>` (cwd, `HOME`), `bots/<bot>` and `workspaces/<workspace>` (`computerRunDirs`, `withComputerDirs`, `resolveComputerPath`; `sudo` refused).
+- **Folders (enforced in tools, not a Linux user):** a run works only in `bots/<bot>/users/<user>` (cwd, `HOME`), `bots/<bot>` and `workspaces/<workspace>` (`computerRunDirs`, `withComputerDirs`, `resolveComputerPath`; the file tools' `sudo` refused; bash runs as `gem`, who has sudo, so packages can be installed, and they're lost on a restart).
+- **Bash on a computer:** the sandbox ignores `sandbox_execute_bash`'s `timeout` and runs every call in one shared shell session, so a command that never ends blocked every later one on that computer, from any run, and Stop never killed it. `withCommandTimeout` runs each in its own `bash -c` under coreutils `timeout` (at most `AppData.ComputerCommandMaxSeconds`, below `McpToolTimeoutMs`), stdin from `/dev/null`. Never `exit` in a wrapper (it would kill the session shell).
 - **Viewing:** `/computers/<id>/` proxy (`handleComputerProxy`, strips app cookies, bridges websockets). `ComputerView` modes: browser, terminal, code, desktop (noVNC; the only one with Chrome's toolbar). **Take-over:** `ask_user_to_take_over` writes `[take-over:ID:mode]`; "Hand back" sends a chat message; tasks stop and report rather than wait.
 
 ### Schedules (routines in the UI)
@@ -152,7 +153,7 @@ A secret URL, `POST /api/hooks/<token>` (`src/pages/api/hooks/[token].ts`, publi
 
 ### Tasks (the to-do tracker)
 
-`tasks` (a bot's own list is `bot_id` alone; a workspace's task has `workspace_id` plus the `bot_id` that has it there, null while shared; a bot reaches its own list and its own plus shared tasks in its workspaces, a workspace admin all of that workspace's, `botTaskScopesSql`; a shared task becomes the bot's when it starts it; removing or deleting a bot makes its workspace tasks shared again), `task_notes`, `task_media` (deliverable|reference). Not the same as `bot_tasks`/`bots_queue` (those record jobs). Visibility SQL in `src/utils/task-access/`. Helpers in `src/functions/backend/db/tasks/`; tools `list_tasks`/`create_task`/`update_task`; API `api/admin/tasks/` (`userAuth` -> `isReadOnlyUser` -> `assertUserReachesTask` -> `safeErrorMessage`). UI in the workspace dashboard and the header's Tasks modal (shared components in `src/components/general/`). **Hand-over:** `handOverTask` (bot: `delegate_task` with `task_id`; UI: Hand over, crew managers) moves an open task to the new bot (a workspace's stays in the workspace, now the new bot's, and is worked there), sets it doing and queues a job with `params.tracker_task_id`; `noteTrackerTask` notes the job's end on it and blocks it on failure. **Not built yet:** task steps, `server:task-changed` pushes, a `/admin/tasks` page.
+`tasks` (**every task belongs to a bot**, the coordinator included, never shared: a bot's own list is `bot_id` alone, a workspace's task has `workspace_id` plus the `bot_id` that has it there, enforced by `taskScopeValues`; a worker reaches only its own, a workspace admin all of that workspace's, the coordinator all, `botTaskScopesSql`; a bot removed from a workspace or deleted passes its tasks there to the workspace's admin bot, else the coordinator (`passOnBotTasks`, also run at boot for rows without a bot)), `task_notes`, `task_media` (deliverable|reference). Not the same as `bot_tasks`/`bots_queue` (those record jobs). Visibility SQL in `src/utils/task-access/`. Helpers in `src/functions/backend/db/tasks/`; tools `list_tasks`/`create_task`/`update_task`; API `api/admin/tasks/` (`userAuth` -> `isReadOnlyUser` -> `assertUserReachesTask` -> `safeErrorMessage`). UI in the workspace dashboard and the header's Tasks modal (shared components in `src/components/general/`); a card's title opens `TaskDetailModal` with everything about the task and its status moves (a modal over the list, never swapped in place). **Hand-over:** `handOverTask` (bot: `delegate_task` with `task_id`; UI: Hand over, crew managers) moves an open task to the new bot (a workspace's stays in the workspace, now the new bot's, and is worked there), sets it doing and queues a job with `params.tracker_task_id`; `noteTrackerTask` notes the job's end on it and blocks it on failure. **Not built yet:** task steps, `server:task-changed` pushes, a `/admin/tasks` page.
 
 ### Envs
 
@@ -176,7 +177,7 @@ Set in Settings > Environment (super admin), not `.env`. `envs` (private; encryp
 - Files live on the data volume, not on computers: `DATA_DIR/bots/<id>` and `DATA_DIR/workspaces/<id>`. Catalog `bot_files`, bytes in private `media` (`bot-file`) written by `recordBotFile`. Reach via `grabStorageScopes`; paths via `resolveBotPath` (no absolute, `..`, NUL, symlink escape); quota checked by `assertStorageRoom`.
 - Tools in `src/harness/tools/storage/` (every run). `attach` puts a file on the reply. `grabConversationFiles` must list the **whole** folder tree, or nested `[file:ID]` refs render dead. A workspace run saves to the bot's own folder unless it picks the workspace's, so a chip in a workspace conversation looks up both (`api/admin/bots/files?file_id=` with `bot_id` and `workspace_id`).
 - `import_file` copies from a computer through the sandbox's streaming `GET /v1/file/download`, **never through its shell** (`/v1/shell/exec` caps output at ~30K chars and silently truncated imports). Size read first, `change_policy=abort`, length checked.
-- Ingestion: chunks + embeddings into `bot_file_chunks` / `bot_file_vectors` (vec0), worked by `src/file-ingest/`; without an embedding model search falls back to words and says so. sqlite-vec limits: metadata columns are NOT NULL; side-car columns can't be filtered with `MATCH` (filter in code). **Never pass `--vector` to `bunx bun-sqlite schema` in `entrypoint.sh`** (wipes vectors on boot). Changing `AppData.EmbeddingDimensions` needs a manual `--vector` run and a re-ingest.
+- **Only knowledgebases store text.** Bot files and uploads are never indexed and keep no extracted text (`media.text_content` only for knowledgebase documents); bots find files by name or path (`list_files` `search`) and read them with `read_file`, and the Files searches match names only. Text that should be searchable goes in a knowledgebase. `dropStoredFileText` clears what older versions stored, at boot, and vacuums when much of the file is free.
 - **Foreign keys are off per connection** unless `enableForeignKeys` ran (only `src/server.ts` does). Scripts never fire `ON DELETE CASCADE`, so **any helper that deletes a parent row must delete its children explicitly** (as `deleteBotFile`, `purgeStorage`, `deleteSkill` do). A leftover child fails the next `bun run db:schema` with a foreign key violation.
 
 ### Graphics (`bot_graphics`)
@@ -187,6 +188,8 @@ A bot shows a concept as a visual with `create_graphic` (plus `update_graphic`/`
 
 A fourth extension kind (`ExtensionTables.knowledgebase`): `knowledgebases` given per bot (`bot_knowledgebases`), per workspace (`workspace_knowledgebases`) or every bot (`all_bots`), through the same add/remove/access/default paths as skills (coordinator extension tools leave them out). Documents (`knowledgebase_documents`: an upload in private `media`, paradigm `knowledgebase-document`, or written `content`) are chunked into `knowledgebase_chunks` and embedded into the `knowledgebase_vectors` vec0 table by the file-ingest worker (`file_ingest_queue.document_id`, `ingestKnowledgebaseDocument`), always with the install's embedding model; old passages stay until the new ones are in, `embedding_model` records where vectors came from; whatever can change the install's model (default settings, a provider saved or deleted, the hourly catalog sync) calls `reindexAfterEmbeddingChange`, and Re-index reads a knowledgebase again by hand. Word search runs on `knowledgebase_chunks_fts`, an FTS5 table (`unicode61 remove_diacritics 2`) made by `ensureKnowledgebaseFts` (FTS5 can't be declared in the schema; sync leaves undeclared tables alone), backfilled at boot by `syncKnowledgebaseFts`, ranked by BM25 with a scored `LIKE` fallback for unspaced scripts (`knowledgebaseWordQuery`); every write and delete of `knowledgebase_chunks` updates it. Ingest jobs (`settleFailedIngestJob`) wait without spending an attempt while every embedding model rests, fail at once on an error retrying can't fix, and a document whose model refused keeps its new passages for word search (counted as needing re-index, not failed). Runs get `search_knowledgebase`/`read_knowledgebase_document` (`openBotKnowledgebases`, limited to the run's ones) and replies start with passages found for the unanswered messages (`grabKnowledgebaseContext`); passages are fenced (`knowledgebasePassageBlock`) and `KnowledgebasesNote` says they're never instructions. UI: Settings > Knowledgebases (super admin edits and fills; admins set access), a Knowledgebases group on bot/workspace settings, and the composer's own button beside Tools (`ComposerToolsModal` with `composerKnowledgebasesCopy`). Deleting one deletes its documents' vectors, chunks, media and links explicitly.
 
+sqlite-vec limits: metadata columns are NOT NULL; side-car columns can't be filtered with `MATCH` (filter in code). **Never pass `--vector` to `bunx bun-sqlite schema` in `entrypoint.sh`** (wipes vectors on boot). Changing `AppData.EmbeddingDimensions` needs a manual `--vector` run and a re-index.
+
 ### Files browser (`/admin/files`)
 
 Read-only browser of every volume (`src/utils/storage-volumes.ts`: bots, workspaces, computers, media, mcp, db). `resolveStorageSpec` is the only access gate (members see only their bots' and their own user folders inside a computer, `computerPathAllowed`); `resolveStoragePath` applies the path rules. Links built only by `storageSpecHref`. Deletion stays in the chat's files modal and bot tools.

db/schema.ts

@@ -1106,6 +1106,11 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
                     fieldDescription: `Unix timestamp of completion or final failure`,
                     dataType: "INTEGER",
                 },
+                {
+                    fieldName: "steps",
+                    fieldDescription: `How many steps it recorded, written when its \`bots_queue_updates\` are cleared after it ends (null while they're still there)`,
+                    dataType: "INTEGER",
+                },
             ],
             indexes: [
                 {
@@ -1116,7 +1121,7 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
         },
         {
             tableName: "bots_queue_updates",
-            tableDescription: `What a bot did while working through a queued job, one row per step, oldest first. Live in the working-on panel while the job runs; deleted with the job when it's pruned`,
+            tableDescription: `What a bot did while working through a queued job, one row per step, oldest first. Live in the working-on panel while the job runs; cleared shortly after the job ends (clearFinishedJobSteps), its count kept on the job`,
             fields: [
                 {
                     fieldName: "bots_queue_id",
@@ -3527,6 +3532,10 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
                     indexName: "idx_logs_user_id",
                     indexTableFields: ["user_id"],
                 },
+                {
+                    indexName: "idx_logs_created_at",
+                    indexTableFields: ["created_at"],
+                },
             ],
         },
         {
@@ -3787,16 +3796,6 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
                     options: BotFileSources.map((s) => s.value),
                     defaultValue: "bot-write",
                 },
-                {
-                    fieldName: "ingested_at",
-                    fieldDescription: `When its text was chunked and embedded. Null: not ingested (yet)`,
-                    dataType: "INTEGER",
-                },
-                {
-                    fieldName: "ingest_error",
-                    fieldDescription: `Why the last ingestion failed, when it did`,
-                    dataType: "TEXT",
-                },
             ],
             uniqueConstraints: [
                 {
@@ -3820,56 +3819,6 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
                 },
             ],
         },
-        {
-            tableName: "bot_file_chunks",
-            tableDescription: `A piece of a bot file's text, the unit a search returns. Replaced whole when the file is ingested again`,
-            fields: [
-                {
-                    fieldName: "file_id",
-                    dataType: "INTEGER",
-                    foreignKey: {
-                        cascadeDelete: true,
-                        destinationTableName: "bot_files",
-                        destinationTableColumnName: "id",
-                        destinationTableColumnType: "INTEGER",
-                        foreignKeyName: "bot_file_chunks_file_fk",
-                    },
-                },
-                {
-                    fieldName: "scope",
-                    dataType: "TEXT",
-                    options: BotFileScopes.map((s) => s.value),
-                },
-                {
-                    fieldName: "owner_id",
-                    fieldDescription: `The bot or workspace the file belongs to, so a search filters without a join`,
-                    dataType: "INTEGER",
-                },
-                {
-                    fieldName: "chunk_index",
-                    dataType: "INTEGER",
-                },
-                {
-                    fieldName: "text",
-                    dataType: "TEXT",
-                },
-                {
-                    fieldName: "token_estimate",
-                    fieldDescription: `A rough count (characters / 4), for trimming results`,
-                    dataType: "INTEGER",
-                },
-            ],
-            indexes: [
-                {
-                    indexName: "idx_bot_file_chunks_file_id",
-                    indexTableFields: ["file_id", "chunk_index"],
-                },
-                {
-                    indexName: "idx_bot_file_chunks_owner",
-                    indexTableFields: ["scope", "owner_id"],
-                },
-            ],
-        },
         {
             tableName: "token_usages",
             tableDescription: `One model call that reported its usage: the tokens it spent, on which model and provider, for whom and for what. A bot message's total is the sum of the calls it took; the usage page sums them by model, provider and period`,
@@ -4018,44 +3967,9 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
                 },
             ],
         },
-        {
-            tableName: "bot_file_vectors",
-            tableDescription: `sqlite-vec vectors of a file's chunks, at \`AppData.EmbeddingDimensions\` wide whatever the model returns. A vec0 table is only recreated with \`bunx bun-sqlite schema --vector\`, so the entrypoint never passes it`,
-            isVector: true,
-            vectorType: "vec0",
-            fields: [
-                {
-                    fieldName: "embedding",
-                    fieldDescription: `Normalized, padded or cut to \`AppData.EmbeddingDimensions\``,
-                    dataType: "BLOB",
-                    isVector: true,
-                    vectorSize: AppData.EmbeddingDimensions,
-                },
-                {
-                    fieldName: "chunk_id",
-                    dataType: "INTEGER",
-                    sideCar: true,
-                },
-                {
-                    fieldName: "file_id",
-                    dataType: "INTEGER",
-                    sideCar: true,
-                },
-                {
-                    fieldName: "scope",
-                    dataType: "TEXT",
-                    sideCar: true,
-                },
-                {
-                    fieldName: "owner_id",
-                    dataType: "INTEGER",
-                    sideCar: true,
-                },
-            ],
-        },
         {
             tableName: "file_ingest_queue",
-            tableDescription: `Files (and knowledgebase documents) waiting to have their text chunked and embedded. A queue, so a restart never strands a file and an upload never waits on a model. A finished job is deleted: \`bot_files.ingested_at\` and \`ingest_error\` keep the outcome`,
+            tableDescription: `Knowledgebase documents waiting to have their text chunked and embedded (bot files are never indexed). A queue, so a restart never strands a document and an upload never waits on a model. A finished job is deleted: \`knowledgebase_documents.ingested_at\` and \`ingest_error\` keep the outcome`,
             isQueue: {
                 maxAttempts: 3,
                 lockTimeout: 10 * 60 * 1000,
@@ -4063,22 +3977,13 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
                 deleteOnComplete: true,
             },
             fields: [
-                {
-                    fieldName: "file_id",
-                    fieldDescription: `The \`bot_files\` row to (re)ingest`,
-                    dataType: "INTEGER",
-                },
                 {
                     fieldName: "document_id",
-                    fieldDescription: `The \`knowledgebase_documents\` row to (re)ingest, instead of a file`,
+                    fieldDescription: `The \`knowledgebase_documents\` row to (re)ingest`,
                     dataType: "INTEGER",
                 },
             ],
             indexes: [
-                {
-                    indexName: "idx_file_ingest_queue_file_id",
-                    indexTableFields: ["file_id"],
-                },
                 {
                     indexName: "idx_file_ingest_queue_document_id",
                     indexTableFields: ["document_id"],

db/types/db.ts

@@ -61,9 +61,7 @@ export const BunSQLiteTables = [
     "push_subscriptions",
     "envs",
     "bot_files",
-    "bot_file_chunks",
     "token_usages",
-    "bot_file_vectors",
     "file_ingest_queue",
     "app_settings",
 ] as const
@@ -505,11 +503,11 @@ export type BUN_SQLITE_COBBLE_TASKS = {
      */
     updated_at?: number | "";
     /** 
-     * The bot the task belongs to; null for a workspace's task
+     * The bot that has the task: its own list's, or with `workspace_id` its task in that workspace; null for a workspace's shared task nobody has yet
      */
     bot_id?: number | "";
     /** 
-     * The workspace the task belongs to (its bots all see it); null for a bot's own
+     * The workspace the task is in; null for a bot's own list
      */
     workspace_id?: number | "";
     title?: string;
@@ -679,6 +677,10 @@ export type BUN_SQLITE_COBBLE_BOTS_QUEUE = {
      * Unix timestamp of completion or final failure
      */
     finished_at?: number | "";
+    /** 
+     * How many steps it recorded, written when its `bots_queue_updates` are cleared after it ends (null while they're still there)
+     */
+    steps?: number | "";
     /** 
      * The state of the job. (pending | processing | completed | failed)
      */
@@ -2331,41 +2333,6 @@ export type BUN_SQLITE_COBBLE_BOT_FILES = {
      * How it got here: the user attached it to a message, the bot wrote it, or a tool made it
      */
     source?: "chat-upload" | "bot-write" | "tool" | "";
-    /** 
-     * When its text was chunked and embedded. Null: not ingested (yet)
-     */
-    ingested_at?: number | "";
-    /** 
-     * Why the last ingestion failed, when it did
-     */
-    ingest_error?: string;
-}
-
-export type BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS = {
-    /** 
-     * The unique identifier of the record.
-     */
-    id?: number | "";
-    /** 
-     * The time when the record was created. (Unix Timestamp)
-     */
-    created_at?: number | "";
-    /** 
-     * The time when the record was updated. (Unix Timestamp)
-     */
-    updated_at?: number | "";
-    file_id?: number | "";
-    scope?: "bot" | "workspace" | "";
-    /** 
-     * The bot or workspace the file belongs to, so a search filters without a join
-     */
-    owner_id?: number | "";
-    chunk_index?: number | "";
-    text?: string;
-    /** 
-     * A rough count (characters / 4), for trimming results
-     */
-    token_estimate?: number | "";
 }
 
 export type BUN_SQLITE_COBBLE_TOKEN_USAGES = {
@@ -2443,29 +2410,6 @@ export type BUN_SQLITE_COBBLE_TOKEN_USAGES = {
     settings?: string;
 }
 
-export type BUN_SQLITE_COBBLE_BOT_FILE_VECTORS = {
-    /** 
-     * The unique identifier of the record.
-     */
-    id?: number | "";
-    /** 
-     * The time when the record was created. (Unix Timestamp)
-     */
-    created_at?: number | "";
-    /** 
-     * The time when the record was updated. (Unix Timestamp)
-     */
-    updated_at?: number | "";
-    /** 
-     * Normalized, padded or cut to `AppData.EmbeddingDimensions`
-     */
-    embedding?: Float32Array<ArrayBuffer> | Buffer<ArrayBuffer> | null;
-    chunk_id?: number | "";
-    file_id?: number | "";
-    scope?: string;
-    owner_id?: number | "";
-}
-
 export type BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE = {
     /** 
      * The unique identifier of the record.
@@ -2480,11 +2424,7 @@ export type BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE = {
      */
     updated_at?: number | "";
     /** 
-     * The `bot_files` row to (re)ingest
-     */
-    file_id?: number | "";
-    /** 
-     * The `knowledgebase_documents` row to (re)ingest, instead of a file
+     * The `knowledgebase_documents` row to (re)ingest
      */
     document_id?: number | "";
     /** 
@@ -2530,4 +2470,4 @@ export type BUN_SQLITE_COBBLE_APP_SETTINGS = {
     value?: string;
 }
 
-export type BUN_SQLITE_COBBLE_ALL_TYPEDEFS = BUN_SQLITE_COBBLE_USERS & BUN_SQLITE_COBBLE_USER_TYPES & BUN_SQLITE_COBBLE_SSO_LOGIN_CODES & BUN_SQLITE_COBBLE_USER_PASSWORDS & BUN_SQLITE_COBBLE_MEDIA & BUN_SQLITE_COBBLE_MEDIA_PARADIGMS & BUN_SQLITE_COBBLE_BOTS & BUN_SQLITE_COBBLE_CHATROOMS & BUN_SQLITE_COBBLE_BOT_SESSIONS & BUN_SQLITE_COBBLE_BOT_MESSAGES & BUN_SQLITE_COBBLE_BOT_MESSAGE_REACTIONS & BUN_SQLITE_COBBLE_BOT_TASKS & BUN_SQLITE_COBBLE_BOT_TASK_STEPS & BUN_SQLITE_COBBLE_TASKS & BUN_SQLITE_COBBLE_TASK_MEDIA & BUN_SQLITE_COBBLE_TASK_NOTES & BUN_SQLITE_COBBLE_BOTS_QUEUE & BUN_SQLITE_COBBLE_BOTS_QUEUE_UPDATES & BUN_SQLITE_COBBLE_CRON_JOBS & BUN_SQLITE_COBBLE_BOT_HOOKS & BUN_SQLITE_COBBLE_BOT_HOOK_EVENTS & BUN_SQLITE_COBBLE_HOOK_PROPOSALS & BUN_SQLITE_COBBLE_SECRET_REQUESTS & BUN_SQLITE_COBBLE_BOT_GRAPHICS & BUN_SQLITE_COBBLE_WORKSPACES & BUN_SQLITE_COBBLE_WORKSPACE_BOTS & BUN_SQLITE_COBBLE_BOT_TYPES & BUN_SQLITE_COBBLE_WORKSPACE_USERS & BUN_SQLITE_COBBLE_BOT_USERS & BUN_SQLITE_COBBLE_BOT_USER_MEMORIES & BUN_SQLITE_COBBLE_BOT_USER_FACES & BUN_SQLITE_COBBLE_BOT_MESSAGE_MEDIA & BUN_SQLITE_COBBLE_CONTAINERS & BUN_SQLITE_COBBLE_COMPUTERS & BUN_SQLITE_COBBLE_BOT_COMPUTERS & BUN_SQLITE_COBBLE_WORKSPACE_COMPUTERS & BUN_SQLITE_COBBLE_HARNESS_PROVIDERS & BUN_SQLITE_COBBLE_HARNESS_MODELS & BUN_SQLITE_COBBLE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_BOT_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_WORKSPACE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_MCP_SERVERS & BUN_SQLITE_COBBLE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_SKILLS & BUN_SQLITE_COBBLE_EXTENSION_PROPOSALS & BUN_SQLITE_COBBLE_BOT_MCP_SERVERS & BUN_SQLITE_COBBLE_WORKSPACE_MCP_SERVERS & BUN_SQLITE_COBBLE_BOT_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_WORKSPACE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_BOT_SKILLS & BUN_SQLITE_COBBLE_WORKSPACE_SKILLS & BUN_SQLITE_COBBLE_KNOWLEDGEBASES & BUN_SQLITE_COBBLE_KNOWLEDGEBASE_DOCUMENTS & BUN_SQLITE_COBBLE_KNOWLEDGEBASE_CHUNKS & BUN_SQLITE_COBBLE_KNOWLEDGEBASE_VECTORS & BUN_SQLITE_COBBLE_BOT_KNOWLEDGEBASES & BUN_SQLITE_COBBLE_WORKSPACE_KNOWLEDGEBASES & BUN_SQLITE_COBBLE_LOGS & BUN_SQLITE_COBBLE_USER_PRESENCE & BUN_SQLITE_COBBLE_PUSH_SUBSCRIPTIONS & BUN_SQLITE_COBBLE_ENVS & BUN_SQLITE_COBBLE_BOT_FILES & BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS & BUN_SQLITE_COBBLE_TOKEN_USAGES & BUN_SQLITE_COBBLE_BOT_FILE_VECTORS & BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE & BUN_SQLITE_COBBLE_APP_SETTINGS
+export type BUN_SQLITE_COBBLE_ALL_TYPEDEFS = BUN_SQLITE_COBBLE_USERS & BUN_SQLITE_COBBLE_USER_TYPES & BUN_SQLITE_COBBLE_SSO_LOGIN_CODES & BUN_SQLITE_COBBLE_USER_PASSWORDS & BUN_SQLITE_COBBLE_MEDIA & BUN_SQLITE_COBBLE_MEDIA_PARADIGMS & BUN_SQLITE_COBBLE_BOTS & BUN_SQLITE_COBBLE_CHATROOMS & BUN_SQLITE_COBBLE_BOT_SESSIONS & BUN_SQLITE_COBBLE_BOT_MESSAGES & BUN_SQLITE_COBBLE_BOT_MESSAGE_REACTIONS & BUN_SQLITE_COBBLE_BOT_TASKS & BUN_SQLITE_COBBLE_BOT_TASK_STEPS & BUN_SQLITE_COBBLE_TASKS & BUN_SQLITE_COBBLE_TASK_MEDIA & BUN_SQLITE_COBBLE_TASK_NOTES & BUN_SQLITE_COBBLE_BOTS_QUEUE & BUN_SQLITE_COBBLE_BOTS_QUEUE_UPDATES & BUN_SQLITE_COBBLE_CRON_JOBS & BUN_SQLITE_COBBLE_BOT_HOOKS & BUN_SQLITE_COBBLE_BOT_HOOK_EVENTS & BUN_SQLITE_COBBLE_HOOK_PROPOSALS & BUN_SQLITE_COBBLE_SECRET_REQUESTS & BUN_SQLITE_COBBLE_BOT_GRAPHICS & BUN_SQLITE_COBBLE_WORKSPACES & BUN_SQLITE_COBBLE_WORKSPACE_BOTS & BUN_SQLITE_COBBLE_BOT_TYPES & BUN_SQLITE_COBBLE_WORKSPACE_USERS & BUN_SQLITE_COBBLE_BOT_USERS & BUN_SQLITE_COBBLE_BOT_USER_MEMORIES & BUN_SQLITE_COBBLE_BOT_USER_FACES & BUN_SQLITE_COBBLE_BOT_MESSAGE_MEDIA & BUN_SQLITE_COBBLE_CONTAINERS & BUN_SQLITE_COBBLE_COMPUTERS & BUN_SQLITE_COBBLE_BOT_COMPUTERS & BUN_SQLITE_COBBLE_WORKSPACE_COMPUTERS & BUN_SQLITE_COBBLE_HARNESS_PROVIDERS & BUN_SQLITE_COBBLE_HARNESS_MODELS & BUN_SQLITE_COBBLE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_BOT_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_WORKSPACE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_MCP_SERVERS & BUN_SQLITE_COBBLE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_SKILLS & BUN_SQLITE_COBBLE_EXTENSION_PROPOSALS & BUN_SQLITE_COBBLE_BOT_MCP_SERVERS & BUN_SQLITE_COBBLE_WORKSPACE_MCP_SERVERS & BUN_SQLITE_COBBLE_BOT_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_WORKSPACE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_BOT_SKILLS & BUN_SQLITE_COBBLE_WORKSPACE_SKILLS & BUN_SQLITE_COBBLE_KNOWLEDGEBASES & BUN_SQLITE_COBBLE_KNOWLEDGEBASE_DOCUMENTS & BUN_SQLITE_COBBLE_KNOWLEDGEBASE_CHUNKS & BUN_SQLITE_COBBLE_KNOWLEDGEBASE_VECTORS & BUN_SQLITE_COBBLE_BOT_KNOWLEDGEBASES & BUN_SQLITE_COBBLE_WORKSPACE_KNOWLEDGEBASES & BUN_SQLITE_COBBLE_LOGS & BUN_SQLITE_COBBLE_USER_PRESENCE & BUN_SQLITE_COBBLE_PUSH_SUBSCRIPTIONS & BUN_SQLITE_COBBLE_ENVS & BUN_SQLITE_COBBLE_BOT_FILES & BUN_SQLITE_COBBLE_TOKEN_USAGES & BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE & BUN_SQLITE_COBBLE_APP_SETTINGS

package.json

@@ -1,6 +1,6 @@
 {
     "name": "cobble",
-    "version": "0.6.1",
+    "version": "0.7.6",
     "module": "index.ts",
     "type": "module",
     "private": true,
@@ -8,12 +8,11 @@
         "dev": "bun src/server.ts",
         "gen:favicon": "bun src/scripts/generate-brand-assets.ts",
         "build": "bunx bunext build",
-        "db:schema": "bunx bun-sqlite schema -t",
+        "db:schema": "bunx bun-sqlite schema -t -y",
         "db:admin": "bunx bun-sqlite admin",
         "db:backup": "bunx bun-sqlite backup",
         "seed": "bun src/scripts/seed-super-admin.tsx",
         "seed:bots": "bun src/scripts/seed-bots.ts",
-        "media:prune-text": "bun src/scripts/prune-media-text-content.ts",
         "start": "MAIN_SERVER=1 NODE_ENV=production bun src/server.ts",
         "typecheck": "tsc --noEmit",
         "test": "bun test",

src/bots-queue/compose-envs-note.test.ts

@@ -16,6 +16,18 @@ describe("composeEnvsNote", () => {
         expect(note).toContain("Don't ask the user for them");
     });
 
+    test("says where each env comes from", () => {
+        const note = composeEnvsNote({
+            on_computer: [{ key: "DB_PASSWORD", description: "Production database", paradigm: "workspace" }],
+            tools_only: [{ key: "TOKEN", description: "", paradigm: "user" }],
+            has_computer: true,
+        });
+
+        expect(note).toContain("- DB_PASSWORD (this workspace's, every bot in it has it): Production database");
+        expect(note).toContain("- TOKEN (this user's own, for you)");
+        expect(note).toContain("never needs setting again per bot");
+    });
+
     test("never mentions values", () => {
         const note = composeEnvsNote({ on_computer: [{ key: "K", description: "d" }], tools_only: [], has_computer: true });
 

src/bots-queue/compose-envs-note.ts

@@ -1,4 +1,6 @@
-type Env = { key: string; description: string };
+import type { EnvParadigm } from "@/src/dict/env-dict";
+
+type Env = { key: string; description: string; paradigm?: EnvParadigm };
 
 type Params = {
     /**
@@ -16,21 +18,39 @@ type Params = {
 };
 
 /**
- * Added to a chat reply so the bot knows which envs it already has (keys
- * and descriptions, never values). Without it a reply can't tell, asks the
- * user for keys that are set (`request_secret`), or "checks" them with
- * `show_secret` or a shell where they aren't loaded.
+ * Where an env comes from, as the bot is told: who else has it.
+ */
+const EnvSources: Record<EnvParadigm, string> = {
+    general: "install-wide, every bot has it",
+    workspace: "this workspace's, every bot in it has it",
+    computer: "this computer's, every bot on it has it",
+    bot: "yours, for every user",
+    user: "this user's own, for you",
+};
+
+/**
+ * Added to a chat reply so the bot knows which envs it already has (keys,
+ * descriptions and where each comes from, never values). Without it a
+ * reply can't tell, asks the user for keys that are set
+ * (`request_secret`), "checks" them with `show_secret` or a shell where
+ * they aren't loaded, or tells the user a workspace's keys must be set
+ * again for every bot.
  */
 export default function composeEnvsNote({ on_computer, tools_only, has_computer }: Params) {
     const list = (envs: Env[]) =>
-        envs.map((env) => `- ${env.key}${env.description ? `: ${env.description}` : ""}`).join("\n");
+        envs
+            .map(
+                (env) =>
+                    `- ${env.key}${env.paradigm ? ` (${EnvSources[env.paradigm]})` : ""}${env.description ? `: ${env.description}` : ""}`,
+            )
+            .join("\n");
 
     if (!on_computer.length && !tools_only.length) {
         return `# Your envs\n\nNo envs are set for you here. If you need a secret, ask for it with request_secret.`;
     }
 
     return [
-        `# Your envs\n\nThese are already set for you here (keys and descriptions; you never see values). Don't ask the user for them, and don't call show_secret or run printenv to check them: this list is the check.`,
+        `# Your envs\n\nThese are already set for you here (keys and descriptions; you never see values). Don't ask the user for them, and don't call show_secret or run printenv to check them: this list is the check. A workspace, computer or install-wide env reaches every bot there the same way, so it never needs setting again per bot.`,
         on_computer.length
             ? `On your computer, as $KEY in a task's shell and code (start_task; search_computers doesn't load them):\n${list(on_computer)}`
             : "",

src/bots-queue/load-attachment-parts.ts

@@ -5,6 +5,7 @@ import grabMessagesMedia, {
 } from "@/src/functions/backend/db/bots/grab-messages-media";
 import type { HarnessContentPart } from "@/src/harness/types";
 import type { BotMessageWithAttachments } from "@/src/types";
+import extractBotFileText from "@/src/functions/backend/db/bot-files/extract-bot-file-text";
 import describeAttachment from "./describe-attachment";
 
 type Params = {
@@ -27,10 +28,7 @@ export default async function loadAttachmentParts({ messages }: Params) {
         .slice(-AppData.BotReplyAttachmentMessages)
         .map((message) => Number(message.id));
 
-    const media = await grabMessagesMedia({
-        message_ids: recent,
-        with_text_content: true,
-    });
+    const media = await grabMessagesMedia({ message_ids: recent });
     const parts = new Map<number, HarnessContentPart[]>();
 
     for (const [message_id, files] of media) {
@@ -53,7 +51,17 @@ async function fileToParts({ file }: { file: MessageMediaRow }): Promise<Harness
     const label = describeAttachment({ name, id: file.id! });
 
     if (file.media_type == "document") {
-        const text = file.text_content?.trim();
+        /**
+         * Read from the file each time: no text is kept outside
+         * knowledgebases.
+         */
+        const text = (
+            await extractBotFileText({
+                write_path: file.media_write_path,
+                name: file.title,
+                mime_type: file.mime_type,
+            }).catch(() => null)
+        )?.trim();
 
         if (!text) {
             return [{ type: "text", text: `[Attached file ${label}: its text couldn't be read.]` }];

src/bots-queue/name-bot-session.ts

@@ -7,6 +7,7 @@ import generateText from "@/src/harness/text/generate-text";
 import { runWithHarnessUsageScope } from "@/src/harness/usage/harness-usage-scope";
 import parseSessionName from "./parse-session-name";
 import sessionNameTranscript from "./session-name-transcript";
+import sessionReadyToName from "./session-ready-to-name";
 
 type Params = {
     session_id: number | string;
@@ -16,7 +17,8 @@ type Params = {
 };
 
 /**
- * Names a session once its first exchange is in: the bot writes the title
+ * Names a session once its first exchange is in (or, with no user message,
+ * once the bot has written in it: work handed to it): the bot writes the title
  * the rail and sessions list show and a short summary of it, from the
  * session's first messages (however late the first reply came). The summary
  * says how many messages it covers (`summary_message_count`), so
@@ -51,12 +53,7 @@ export default async function nameBotSession({
         limit: AppData.BotSessionNameMessages,
     });
 
-    if (
-        !messages.some((message) => message.author == "user") ||
-        !messages.some((message) => message.author == "bot" && !message.carried_from_session_id)
-    ) {
-        return false;
-    }
+    if (!sessionReadyToName({ messages })) return false;
 
     const transcript = sessionNameTranscript({ messages });
 

src/bots-queue/process-bot-job.ts

@@ -235,9 +235,10 @@ export default async function processBotJob({
          * Once the first exchange is sent, the bot names the session and
          * previews it. After the reply is pushed, so it never waits on it;
          * a model that can't answer now leaves the session for the next
-         * reply to name.
+         * job to name. Tasks name too: a session the coordinator's
+         * delegation or a routine opened has no reply to name it.
          */
-        if (job.kind == "reply" && !in_room && completion.message?.session_id) {
+        if (!in_room && completion.message?.session_id) {
             /**
              * The message's session, not the job's as it was claimed: a
              * reply that moved the conversation on answered in the new one.

src/bots-queue/session-ready-to-name.test.ts

@@ -0,0 +1,43 @@
+import { describe, expect, test } from "bun:test";
+import sessionReadyToName from "./session-ready-to-name";
+
+describe("sessionReadyToName", () => {
+    test("waits for the bot to answer the user", () => {
+        expect(sessionReadyToName({ messages: [{ id: 1, author: "user", text: "Hi" }] })).toBe(false);
+    });
+
+    test("names a first exchange", () => {
+        expect(
+            sessionReadyToName({
+                messages: [
+                    { id: 1, author: "user", text: "Hi" },
+                    { id: 2, author: "bot", text: "Hello" },
+                ],
+            }),
+        ).toBe(true);
+    });
+
+    test("names work handed to the bot, with no user message", () => {
+        expect(
+            sessionReadyToName({
+                messages: [{ id: 1, author: "bot", text: "Coordinator asked me to: **SEO lookup**" }],
+            }),
+        ).toBe(true);
+    });
+
+    test("a carried summary alone is not the bot's answer", () => {
+        expect(
+            sessionReadyToName({
+                messages: [
+                    { id: 1, author: "bot", text: "- Crew set up", carried_from_session_id: 9 },
+                    { id: 2, author: "user", text: "Continue here" },
+                ],
+            }),
+        ).toBe(false);
+        expect(
+            sessionReadyToName({
+                messages: [{ id: 1, author: "bot", text: "- Crew set up", carried_from_session_id: 9 }],
+            }),
+        ).toBe(false);
+    });
+});

src/bots-queue/session-ready-to-name.ts

@@ -0,0 +1,28 @@
+import type { SessionOpeningMessage } from "@/src/functions/backend/db/bot-sessions/grab-session-opening";
+
+type Params = {
+    messages: SessionOpeningMessage[];
+};
+
+/**
+ * Whether a session's opening says enough to name it: the bot has said
+ * something of its own (a carried summary doesn't count), and, when the
+ * user wrote, after the user's first message. A session with no user
+ * message at all is work handed to the bot (the coordinator's delegations,
+ * routines, hooks), named from the bot's own messages.
+ */
+export default function sessionReadyToName({ messages }: Params) {
+    const has_bot_message = messages.some(
+        (message) => message.author == "bot" && !message.carried_from_session_id,
+    );
+
+    if (!has_bot_message) return false;
+
+    const first_user = messages.findIndex((message) => message.author == "user");
+
+    if (first_user < 0) return true;
+
+    return messages
+        .slice(first_user + 1)
+        .some((message) => message.author == "bot" && !message.carried_from_session_id);
+}

src/components/general/chatroom-stone-cluster.tsx

@@ -0,0 +1,48 @@
+import Row from "@moduletrace/twui/layout/Row";
+import Stack from "@moduletrace/twui/layout/Stack";
+import type { ChatroomBot } from "@/src/types";
+import ChatroomStone from "./chatroom-stone";
+
+type Props = {
+    bots: ChatroomBot[];
+};
+
+/**
+ * The square the cluster fills (`w-7 h-7`), the size of a rail avatar.
+ */
+const ClusterSize = 28;
+
+/**
+ * A chatroom's bots packed into one avatar-sized square: the asking bot
+ * on top, the two below it in a triangle (four make a square, the rest
+ * are left out). Stones stay small and apart: never overlapping, in a
+ * badge or outlined. One bot fills the square alone.
+ */
+export default function ChatroomStoneCluster({ bots }: Props) {
+    const shown = bots.slice(0, 4);
+    const stone = Math.floor((ClusterSize - 2) / 2);
+
+    if (shown.length <= 1) {
+        return shown[0] ? <ChatroomStone bot={shown[0]} size={ClusterSize} /> : null;
+    }
+
+    const top = shown.length == 3 ? shown.slice(0, 1) : shown.slice(0, 2);
+    const bottom = shown.length == 2 ? [] : shown.slice(top.length);
+
+    return (
+        <Stack className="shrink-0 gap-0.5! items-center justify-center w-7 h-7">
+            <Row className="gap-0.5! flex-nowrap justify-center">
+                {top.map((bot) => (
+                    <ChatroomStone key={bot.id} bot={bot} size={stone} />
+                ))}
+            </Row>
+            {bottom.length ? (
+                <Row className="gap-0.5! flex-nowrap justify-center">
+                    {bottom.map((bot) => (
+                        <ChatroomStone key={bot.id} bot={bot} size={stone} />
+                    ))}
+                </Row>
+            ) : null}
+        </Stack>
+    );
+}

src/components/general/task-detail-modal.tsx

@@ -0,0 +1,92 @@
+import { useEffect, useState } from "react";
+import Span from "@moduletrace/twui/layout/Span";
+import Stack from "@moduletrace/twui/layout/Stack";
+import fetchApi from "@moduletrace/twui/utils/fetch/fetchApi";
+import type { APIResponseObject } from "@moduletrace/bunext/types";
+import type { TaskDetail, TaskWithNames } from "@/src/types";
+import AddModalShell from "@/src/layouts/bots/(partials)/add-modal-shell";
+import taskPlaceLabel from "@/src/utils/task-place-label";
+import TaskDetailPanel from "./task-detail-panel";
+
+type Props = {
+    /**
+     * The list row clicked; open while set. Its notes and files load when
+     * it opens, so the list stays small.
+     */
+    task?: TaskWithNames | null;
+    /**
+     * A crew manager can change it, file onto it and delete it.
+     */
+    can_manage?: boolean;
+    onClose: () => void;
+    onChanged: (params: { task: TaskDetail; msg?: string }) => void;
+    onDeleted: (params: { task_id: number }) => void;
+};
+
+/**
+ * A task opened from a list in a modal of its own: everything about it, and
+ * for a crew manager its edit, hand-over, files and delete.
+ */
+export default function TaskDetailModal({ task, can_manage, onClose, onChanged, onDeleted }: Props) {
+    const [detail, set_detail] = useState<TaskDetail | null>(null);
+    const [error, set_error] = useState<string>();
+    const task_id = task?.id ? Number(task.id) : 0;
+
+    useEffect(() => {
+        set_detail(null);
+        set_error(undefined);
+
+        if (!task_id) return;
+
+        let live = true;
+
+        fetchApi<{}, APIResponseObject<TaskDetail>>(`/api/admin/tasks/get-task?task_id=${task_id}`, {
+            method: "GET",
+        })
+            .catch(() => null)
+            .then((res) => {
+                if (!live) return;
+
+                if (!res?.success || !res.singleRes) {
+                    set_error(res?.msg || "Couldn't open the task.");
+                    return;
+                }
+
+                set_detail(res.singleRes);
+            });
+
+        return () => {
+            live = false;
+        };
+    }, [task_id]);
+
+    const shown = detail || task;
+
+    return (
+        <AddModalShell
+            open={Boolean(task_id)}
+            setOpen={(open) => !open && onClose()}
+            title={shown?.title || (task_id ? `Task ${task_id}` : "Task")}
+            description={shown ? taskPlaceLabel({ task: shown }) : undefined}
+        >
+            <Stack className="w-full mt-4 gap-0! items-stretch">
+                {detail ? (
+                    <TaskDetailPanel
+                        key={detail.id}
+                        task={detail}
+                        can_manage={can_manage}
+                        onChanged={(saved, msg) => {
+                            set_detail(saved);
+                            onChanged({ task: saved, msg });
+                        }}
+                        onDeleted={() => onDeleted({ task_id })}
+                    />
+                ) : (
+                    <Span className="py-6 text-[13px] leading-[18px] text-ink-secondary">
+                        {error || "Loading the task..."}
+                    </Span>
+                )}
+            </Stack>
+        </AddModalShell>
+    );
+}

src/components/general/task-detail-panel.tsx

@@ -1,4 +1,4 @@
-import { useRef, useState } from "react";
+import { useContext, useRef, useState } from "react";
 import Button from "@moduletrace/twui/layout/Button";
 import Row from "@moduletrace/twui/layout/Row";
 import Span from "@moduletrace/twui/layout/Span";
@@ -10,6 +10,14 @@ import TaskFileRow from "@/src/components/general/task-file-row";
 import HandOverForm from "@/src/components/general/hand-over-form";
 import TaskForm from "@/src/components/general/task-form";
 import TaskStatusChip from "@/src/components/general/task-status-chip";
+import TaskStatusMoves from "@/src/components/general/task-status-moves";
+import BotAvatar from "@/src/components/general/bot-avatar";
+import TaskFactRow from "@/src/components/general/task-fact-row";
+import TaskNoteRow from "@/src/components/general/task-note-row";
+import type { BotFace } from "@/src/dict/bot-dict";
+import { AppContext } from "@/src/pages/__root";
+import botStoneProps from "@/src/utils/bot-stone-props";
+import relativeTime from "@/src/utils/relative-time";
 import fetchApi from "@moduletrace/twui/utils/fetch/fetchApi";
 import type { APIResponseObject } from "@moduletrace/bunext/types";
 
@@ -24,12 +32,14 @@ type Props = {
 };
 
 /**
- * One task opened: what it is, its notes oldest first, its files (each
- * opening its bytes), a box to add a note, and edit or delete. A crew
+ * One task opened: its status (a crew manager moves it along), who has it and where, who added it and
+ * when it last moved, what it is, its files (each opening its bytes), its
+ * notes oldest first, a box to add a note, and edit or delete. A crew
  * manager can file a new deliverable straight onto it, or hand an open
  * one to another bot, which starts on it.
  */
 export default function TaskDetailPanel({ task, can_manage, onChanged, onDeleted }: Props) {
+    const { pageProps } = useContext(AppContext);
     const [editing, setEditing] = useState(false);
     const [handing_over, setHandingOver] = useState(false);
     const [note, setNote] = useState("");
@@ -135,13 +145,21 @@ export default function TaskDetailPanel({ task, can_manage, onChanged, onDeleted
         <Stack className="w-full gap-4! items-stretch">
             <Row className="w-full gap-2! flex-nowrap items-center">
                 <TaskStatusChip status={task.status} />
-                <Span className="grow min-w-0 text-[13px] leading-[18px] text-ink-secondary truncate">
-                    {task.workspace_id
-                        ? task.workspace_name || `Workspace ${task.workspace_id}`
-                        : task.bot_name || `Bot ${task.bot_id}`}
-                </Span>
                 {can_manage ? (
-                    <Row className="shrink-0 gap-1! flex-nowrap">
+                    <TaskStatusMoves
+                        status={task.status}
+                        disabled={busy}
+                        onMove={(next) =>
+                            run({
+                                path: "tasks/save-task",
+                                body: { task_id: Number(task.id), status: next },
+                                done: "Status changed.",
+                            })
+                        }
+                    />
+                ) : null}
+                {can_manage ? (
+                    <Row className="ml-auto shrink-0 gap-1! flex-nowrap">
                         {(OpenTaskStatuses as readonly string[]).includes(task.status || "") ? (
                             <Button
                                 title="Hand the task to another bot"
@@ -200,6 +218,59 @@ export default function TaskDetailPanel({ task, can_manage, onChanged, onDeleted
                 ) : null}
             </Row>
 
+            <Stack className="w-full gap-1.5! items-stretch">
+                <TaskFactRow label="Who has it">
+                    {task.bot_id ? (
+                        <>
+                            <BotAvatar
+                                image_media_id={task.bot_image_media_id}
+                                {...botStoneProps({ color: task.bot_color, stone: task.bot_stone, hue: task.bot_hue })}
+                                face={(task.bot_face || "idle") as BotFace}
+                                size={18}
+                            />
+                            <Span className="min-w-0 truncate">{task.bot_name || `Bot ${task.bot_id}`}</Span>
+                        </>
+                    ) : (
+                        <Span className="text-ink-secondary">No bot</Span>
+                    )}
+                </TaskFactRow>
+                <TaskFactRow label="Where">
+                    <Span className="min-w-0 truncate">
+                        {task.workspace_id
+                            ? task.workspace_name || `Workspace ${task.workspace_id}`
+                            : `${task.bot_name || "The bot"}'s own list`}
+                    </Span>
+                </TaskFactRow>
+                <TaskFactRow label="Added">
+                    <Span className="min-w-0 truncate">
+                        {[
+                            task.created_by_bot_id
+                                ? `By ${task.created_by_bot_name || `bot ${task.created_by_bot_id}`}`
+                                : task.created_by_user_id
+                                  ? task.created_by_user_id == pageProps?.user?.id
+                                      ? "By you"
+                                      : `By ${task.created_by_user_name || "a user"}`
+                                  : task.origin == "bot"
+                                    ? "By a bot"
+                                    : "By hand",
+                            task.created_at ? relativeTime({ at: Number(task.created_at) }) : null,
+                        ]
+                            .filter(Boolean)
+                            .join(" · ")}
+                    </Span>
+                </TaskFactRow>
+                {task.updated_at ? (
+                    <TaskFactRow label="Last moved">
+                        <Span>{relativeTime({ at: Number(task.updated_at) })}</Span>
+                    </TaskFactRow>
+                ) : null}
+                {task.completed_at ? (
+                    <TaskFactRow label="Finished">
+                        <Span>{relativeTime({ at: Number(task.completed_at) })}</Span>
+                    </TaskFactRow>
+                ) : null}
+            </Stack>
+
             {task.status == "blocked" && task.blocked_reason ? (
                 <Span className="text-[13px] leading-[18px] text-ink">{`Waiting on you: ${task.blocked_reason}`}</Span>
             ) : null}
@@ -209,7 +280,9 @@ export default function TaskDetailPanel({ task, can_manage, onChanged, onDeleted
             ) : null}
 
             <Stack className="w-full gap-1! items-stretch">
-                <Span className="text-[12px] font-bold leading-4 text-ink-secondary">Files</Span>
+                <Span className="text-[12px] font-bold leading-4 text-ink-secondary">
+                    {task.files.length ? `Files (${task.files.length})` : "Files"}
+                </Span>
                 {deliverables.length || references.length ? (
                     <Stack className="w-full gap-0! items-stretch">
                         {[...deliverables, ...references].map((file) => (
@@ -253,18 +326,15 @@ export default function TaskDetailPanel({ task, can_manage, onChanged, onDeleted
             </Stack>
 
             <Stack className="w-full gap-1! items-stretch">
-                <Span className="text-[12px] font-bold leading-4 text-ink-secondary">Notes</Span>
+                <Span className="text-[12px] font-bold leading-4 text-ink-secondary">
+                    {task.notes.length ? `Notes (${task.notes.length})` : "Notes"}
+                </Span>
                 {task.notes.length ? (
-                    task.notes.map((entry) => (
-                        <Row key={entry.id} className="w-full gap-2! flex-nowrap items-start">
-                            <Span className="shrink-0 text-[11px] leading-[18px] text-ink-secondary">
-                                {entry.author == "user" ? "You" : "Bot"}
-                            </Span>
-                            <Span className="grow min-w-0 whitespace-pre-wrap text-[13px] leading-[18px] text-ink">
-                                {entry.text}
-                            </Span>
-                        </Row>
-                    ))
+                    <Stack className="w-full gap-0! items-stretch">
+                        {task.notes.map((entry) => (
+                            <TaskNoteRow key={entry.id} note={entry} />
+                        ))}
+                    </Stack>
                 ) : (
                     <Span className="text-[13px] leading-[18px] text-ink-secondary">No notes yet.</Span>
                 )}

src/components/general/task-fact-row.tsx

@@ -0,0 +1,22 @@
+import type { ReactNode } from "react";
+import Row from "@moduletrace/twui/layout/Row";
+import Span from "@moduletrace/twui/layout/Span";
+
+type Props = {
+    label: string;
+    children: ReactNode;
+};
+
+/**
+ * One fact about an opened task: a label, then its value.
+ */
+export default function TaskFactRow({ label, children }: Props) {
+    return (
+        <Row className="w-full gap-3! flex-nowrap items-center">
+            <Span className="w-24 shrink-0 text-[12px] leading-[18px] text-ink-secondary">{label}</Span>
+            <Row className="grow min-w-0 gap-2! flex-nowrap items-center text-[13px] leading-[18px] text-ink">
+                {children}
+            </Row>
+        </Row>
+    );
+}

src/components/general/task-form.tsx

@@ -6,6 +6,7 @@ import Input from "@moduletrace/twui/form/Input";
 import Select from "@moduletrace/twui/form/Select";
 import Textarea from "@moduletrace/twui/form/Textarea";
 import Stack from "@moduletrace/twui/layout/Stack";
+import CodeEditor from "@/src/components/general/code-editor";
 import { AppData } from "@/src/data/app-data";
 import { TaskStatuses, type TaskStatus } from "@/src/dict/task-dict";
 import AddFormActions from "@/src/layouts/bots/(partials)/add-form-actions";
@@ -17,6 +18,11 @@ type Props = {
      */
     bot_id?: number | null;
     workspace_id?: number | null;
+    /**
+     * Adding to a workspace without a `bot_id`: the bots there to pick who
+     * has it, since every task belongs to a bot.
+     */
+    bots?: { id?: number | string | null; name?: string | null }[];
     /**
      * Edits this task; adds one when not given.
      */
@@ -26,15 +32,30 @@ type Props = {
 };
 
 /**
- * The one task form: a title, the brief, the status and, while it's
+ * The one task form: a title, the brief (in the code editor), the status and, while it's
  * blocked, what the user has to do. The workspace's tasks group and the
  * conversation's tasks modal both add and edit with it.
  */
-export default function TaskForm({ bot_id, workspace_id, task, onSaved, onCancel }: Props) {
+export default function TaskForm({
+    bot_id,
+    workspace_id,
+    bots = [],
+    task,
+    onSaved,
+    onCancel,
+}: Props) {
     const [title, setTitle] = useState(task?.title || "");
     const [description, setDescription] = useState(task?.description || "");
-    const [status, setStatus] = useState<TaskStatus>((task?.status as TaskStatus) || "todo");
-    const [blocked_reason, setBlockedReason] = useState(task?.blocked_reason || "");
+    const [status, setStatus] = useState<TaskStatus>(
+        (task?.status as TaskStatus) || "todo",
+    );
+    const [blocked_reason, setBlockedReason] = useState(
+        task?.blocked_reason || "",
+    );
+    const pick_bot = !task && !bot_id && Boolean(workspace_id);
+    const [picked_bot_id, setPickedBotId] = useState<number | undefined>(
+        bots[0]?.id ? Number(bots[0].id) : undefined,
+    );
     const [loading, setLoading] = useState(false);
     const [error, setError] = useState<string>();
 
@@ -46,24 +67,30 @@ export default function TaskForm({ bot_id, workspace_id, task, onSaved, onCancel
             return;
         }
 
+        if (pick_bot && !picked_bot_id) {
+            setError("Pick the bot that has it.");
+            return;
+        }
+
         setLoading(true);
         setError(undefined);
 
-        const res = await fetchApi<Record<string, unknown>, APIResponseObject<TaskDetail>>(
-            "/api/admin/tasks/save-task",
-            {
-                method: "POST",
-                body: {
-                    task_id: task?.id ? Number(task.id) : undefined,
-                    bot_id: task ? undefined : bot_id || undefined,
-                    workspace_id: task ? undefined : workspace_id || undefined,
-                    title,
-                    description,
-                    status,
-                    blocked_reason: status == "blocked" ? blocked_reason || null : null,
-                },
+        const res = await fetchApi<
+            Record<string, unknown>,
+            APIResponseObject<TaskDetail>
+        >("/api/admin/tasks/save-task", {
+            method: "POST",
+            body: {
+                task_id: task?.id ? Number(task.id) : undefined,
+                bot_id: task ? undefined : bot_id || picked_bot_id || undefined,
+                workspace_id: task ? undefined : workspace_id || undefined,
+                title,
+                description,
+                status,
+                blocked_reason:
+                    status == "blocked" ? blocked_reason || null : null,
             },
-        );
+        });
 
         setLoading(false);
 
@@ -77,25 +104,41 @@ export default function TaskForm({ bot_id, workspace_id, task, onSaved, onCancel
 
     return (
         <Form onSubmit={submit} className="gap-3!">
-            <Stack className="gap-3! items-stretch">
+            <Stack className="gap-6! items-stretch">
                 <Input
+                    title="Title"
                     label="Title"
+                    showLabel
                     value={title}
                     changeHandler={setTitle}
                     placeholder="What needs doing?"
                     maxLength={AppData.TaskTitleMaxLength}
                     disabled={loading}
                 />
-                <Textarea
-                    label="Brief"
-                    value={description}
-                    changeHandler={setDescription}
+                {pick_bot ? (
+                    <Select
+                        title="Who has it"
+                        label="Who has it"
+                        showLabel
+                        options={bots.map((bot) => ({
+                            title: bot.name || `Bot ${bot.id}`,
+                            value: String(bot.id),
+                            default: Number(bot.id) == picked_bot_id,
+                        }))}
+                        changeHandler={(value) => setPickedBotId(Number(value) || undefined)}
+                    />
+                ) : null}
+                <CodeEditor
+                    mode="markdown"
+                    className="h-[500px]"
+                    title="Brief"
                     placeholder="What to do, where, and how you'll know it's done."
-                    maxLength={AppData.TaskDescriptionMaxLength}
-                    rows={5}
-                    disabled={loading}
+                    content={description}
+                    info={`Up to ${AppData.TaskDescriptionMaxLength.toLocaleString()} characters.`}
+                    onChange={setDescription}
                 />
                 <Select
+                    title="Status"
                     label="Status"
                     showLabel
                     options={TaskStatuses.map((it) => ({
@@ -103,11 +146,15 @@ export default function TaskForm({ bot_id, workspace_id, task, onSaved, onCancel
                         value: it.value,
                         default: it.value == status,
                     }))}
-                    changeHandler={(value) => setStatus(String(value) as TaskStatus)}
+                    changeHandler={(value) =>
+                        setStatus(String(value) as TaskStatus)
+                    }
                 />
                 {status == "blocked" ? (
                     <Textarea
+                        title="Waiting on the user for"
                         label="Waiting on the user for"
+                        showLabel
                         value={blocked_reason}
                         changeHandler={setBlockedReason}
                         placeholder="What the user has to do before this can go on."

src/components/general/task-list-item.tsx

@@ -1,4 +1,5 @@
 import { useContext } from "react";
+import { twMerge } from "tailwind-merge";
 import Row from "@moduletrace/twui/layout/Row";
 import Span from "@moduletrace/twui/layout/Span";
 import Stack from "@moduletrace/twui/layout/Stack";
@@ -8,35 +9,41 @@ import { AppContext } from "@/src/pages/__root";
 import BotAvatar from "@/src/components/general/bot-avatar";
 import type { BotFace } from "@/src/dict/bot-dict";
 import botStoneProps from "@/src/utils/bot-stone-props";
-import { TaskStatuses } from "@/src/dict/task-dict";
 import canManageCrew from "@/src/utils/rbac/can-manage-crew";
 import relativeTime from "@/src/utils/relative-time";
 import taskPlaceLabel from "@/src/utils/task-place-label";
+import { Trash2 } from "lucide-react";
+import RowIconButton from "./row-icon-button";
 import TaskStatusChip from "./task-status-chip";
+import TaskStatusMoves from "./task-status-moves";
 
 type Props = {
     task: TaskWithNames;
     /**
-     * Opens the task to read its notes and files, and to edit it.
+     * Opens the task in its modal: everything about it, and for a crew
+     * manager its edit, hand-over, files and delete. The title opens it.
      */
     onOpen?: () => void;
     /**
      * Moves it to one status; only crew managers get the controls.
      */
     onStatusChange?: (status: string) => void;
+    /**
+     * Deletes it after asking; only crew managers get the button.
+     */
+    onDelete?: () => void;
     busy?: boolean;
 };
 
 /**
  * One task: whose it is (a bot's stone), its title and status, where it
  * is and who has it, what it's blocked on, its latest note and how long ago it
- * moved.
+ * moved. Its title opens it in a modal.
  */
-export default function TaskListItem({ task, onOpen, onStatusChange, busy }: Props) {
+export default function TaskListItem({ task, onOpen, onStatusChange, onDelete, busy }: Props) {
     const { pageProps } = useContext(AppContext);
     const can_manage = canManageCrew({ user_types: pageProps?.user_types });
     const done = task.status == "done" || task.status == "cancelled";
-    const moves = TaskStatuses.filter((it) => ["todo", "doing", "done"].includes(it.value));
 
     return (
         <Row className="w-full gap-3! flex-nowrap items-start px-3 py-3 border-0 border-b border-solid border-hairline last:border-b-0">
@@ -49,18 +56,27 @@ export default function TaskListItem({ task, onOpen, onStatusChange, busy }: Pro
             <Stack className="grow min-w-0 gap-0.5! items-stretch">
                 <Row className="w-full gap-2! flex-nowrap items-center">
                     <Button
-                        title={task.title || "Open the task"}
+                        title={`Open ${task.title || `task ${task.id}`}`}
                         type="button"
                         variant="ghost"
                         color="text"
                         onClick={onOpen}
-                        className={`grow min-w-0 justify-start px-0! h-auto! py-0! text-sm! font-bold! leading-5! truncate text-left hover:bg-transparent! hover:underline ${
-                            done ? "text-ink-secondary! line-through" : "text-ink!"
-                        }`}
+                        className={twMerge(
+                            "grow min-w-0 justify-start px-0! h-auto! py-0! text-sm! font-bold! leading-5! truncate text-left hover:bg-transparent! hover:underline",
+                            done ? "text-ink-secondary! line-through" : "text-ink!",
+                        )}
                     >
                         <Span className="truncate">{task.title || `Task ${task.id}`}</Span>
                     </Button>
                     <TaskStatusChip status={task.status} />
+                    {can_manage && onDelete ? (
+                        <RowIconButton
+                            icon={Trash2}
+                            title={`Delete ${task.title || `task ${task.id}`}`}
+                            disabled={busy}
+                            onClick={onDelete}
+                        />
+                    ) : null}
                 </Row>
                 <Row className="w-full gap-2! flex-nowrap items-center">
                     <Span className="shrink-0 text-xs leading-4 text-ink-secondary truncate">
@@ -81,25 +97,7 @@ export default function TaskListItem({ task, onOpen, onStatusChange, busy }: Pro
                     <Span className="text-xs leading-4 text-ink-secondary line-clamp-2">{task.latest_note}</Span>
                 ) : null}
                 {can_manage && onStatusChange ? (
-                    <Row className="gap-1! flex-nowrap items-center pt-0.5">
-                        {moves
-                            .filter((it) => it.value != task.status)
-                            .map((it) => (
-                                <Button
-                                    key={it.value}
-                                    title={`Mark it ${it.title.toLowerCase()}`}
-                                    type="button"
-                                    variant="ghost"
-                                    color="gray"
-                                    size="small"
-                                    disabled={busy}
-                                    onClick={() => onStatusChange(it.value)}
-                                    className="px-1.5! text-[12px]"
-                                >
-                                    {it.title}
-                                </Button>
-                            ))}
-                    </Row>
+                    <TaskStatusMoves status={task.status} disabled={busy} onMove={onStatusChange} className="pt-0.5" />
                 ) : null}
             </Stack>
         </Row>

src/components/general/task-note-row.tsx

@@ -0,0 +1,31 @@
+import Row from "@moduletrace/twui/layout/Row";
+import Span from "@moduletrace/twui/layout/Span";
+import Stack from "@moduletrace/twui/layout/Stack";
+import type { TaskNote } from "@/src/types";
+import relativeTime from "@/src/utils/relative-time";
+
+type Props = {
+    note: TaskNote;
+};
+
+/**
+ * One note on an opened task: who wrote it (the user, or the bot by name),
+ * how long ago, and what it says.
+ */
+export default function TaskNoteRow({ note }: Props) {
+    const author = note.author == "user" ? "You" : note.bot_name || "Bot";
+
+    return (
+        <Stack className="w-full gap-0.5! items-stretch py-1.5 border-0 border-b border-solid border-hairline last:border-b-0">
+            <Row className="w-full gap-2! flex-nowrap items-center">
+                <Span className="text-[12px] font-bold leading-4 text-ink">{author}</Span>
+                {note.created_at ? (
+                    <Span className="text-[11px] leading-4 text-ink-secondary">
+                        {relativeTime({ at: Number(note.created_at) })}
+                    </Span>
+                ) : null}
+            </Row>
+            <Span className="whitespace-pre-wrap text-[13px] leading-[18px] text-ink">{note.text}</Span>
+        </Stack>
+    );
+}

src/components/general/task-status-chip.tsx

@@ -1,3 +1,4 @@
+import { twMerge } from "tailwind-merge";
 import Tag from "@moduletrace/twui/elements/Tag";
 import { TaskStatuses, type TaskStatus } from "@/src/dict/task-dict";
 
@@ -14,15 +15,20 @@ export default function TaskStatusChip({ status = "todo" }: Props) {
     const value = (status || "todo") as TaskStatus;
     const title = TaskStatuses.find((it) => it.value == value)?.title || "To do";
 
-    const tone =
-        value == "blocked"
-            ? "text-ink border-ink"
-            : value == "doing"
-              ? "text-ink border-hairline-strong"
-              : "text-ink-secondary border-hairline";
-
+    /**
+     * A `Tag` is a filled primary button by default (ink in light mode), so
+     * the chip clears its fill and sets its own text: only the outline speaks.
+     */
     return (
-        <Tag className={`shrink-0 px-2! py-0.5! text-[11px]! font-bold! leading-4! border border-solid ${tone}`}>
+        <Tag
+            className={twMerge(
+                "shrink-0 px-2! py-0.5! text-[11px]! font-bold! leading-4! border border-solid",
+                "bg-transparent! dark:bg-transparent! hover:bg-transparent!",
+                value == "blocked" && "text-ink! dark:text-ink! border-ink",
+                value == "doing" && "text-ink! dark:text-ink! border-hairline-strong",
+                value != "blocked" && value != "doing" && "text-ink-secondary! dark:text-ink-secondary! border-hairline",
+            )}
+        >
             {title}
         </Tag>
     );

src/components/general/task-status-move-button.tsx

@@ -0,0 +1,28 @@
+import Button from "@moduletrace/twui/layout/Button";
+
+type Props = {
+    title: string;
+    value: string;
+    disabled?: boolean;
+    onMove: (status: string) => void;
+};
+
+/**
+ * Moves a task to one status: To do, Doing or Done.
+ */
+export default function TaskStatusMoveButton({ title, value, disabled, onMove }: Props) {
+    return (
+        <Button
+            title={`Mark it ${title.toLowerCase()}`}
+            type="button"
+            variant="ghost"
+            color="gray"
+            size="small"
+            disabled={disabled}
+            onClick={() => onMove(value)}
+            className="px-1.5! text-[12px]"
+        >
+            {title}
+        </Button>
+    );
+}

src/components/general/task-status-moves.tsx

@@ -0,0 +1,36 @@
+import Row from "@moduletrace/twui/layout/Row";
+import { twMerge } from "tailwind-merge";
+import { TaskStatuses } from "@/src/dict/task-dict";
+import TaskStatusMoveButton from "./task-status-move-button";
+
+type Props = {
+    /**
+     * The task's status now; its own button is left out.
+     */
+    status?: string | null;
+    disabled?: boolean;
+    onMove: (status: string) => void;
+    className?: string;
+};
+
+const MoveStatuses = TaskStatuses.filter((it) => ["todo", "doing", "done"].includes(it.value));
+
+/**
+ * The buttons that move a task along (To do, Doing, Done), the one it's
+ * at left out. Shown to crew managers on a list row and in the task modal.
+ */
+export default function TaskStatusMoves({ status, disabled, onMove, className }: Props) {
+    return (
+        <Row className={twMerge("gap-1! flex-nowrap items-center", className)}>
+            {MoveStatuses.filter((it) => it.value != status).map((it) => (
+                <TaskStatusMoveButton
+                    key={it.value}
+                    title={it.title}
+                    value={it.value}
+                    disabled={disabled}
+                    onMove={onMove}
+                />
+            ))}
+        </Row>
+    );
+}

src/cron/clear-job-steps/index.ts

@@ -0,0 +1,8 @@
+import clearFinishedJobSteps from "@/src/functions/backend/db/bots-queue/clear-finished-job-steps";
+
+/**
+ * Clears the steps of jobs that have ended (`clearFinishedJobSteps`).
+ */
+export default async function cronClearJobSteps() {
+    return await clearFinishedJobSteps();
+}

src/cron/index.ts

@@ -1,8 +1,10 @@
 import bunext from "@moduletrace/bunext";
 import cronHeartbeat from "./heartbeat";
 import pruneBotsQueue from "./prune-bots-queue";
+import cronClearJobSteps from "./clear-job-steps";
 import pruneBotHookEvents from "./prune-bot-hook-events";
 import cronPruneChatrooms from "./prune-chatrooms";
+import pruneLogs from "./prune-logs";
 import syncNotifyWatchers from "./sync-notify-watchers";
 import cronSyncHarnessModels from "./sync-harness-models";
 import syncComputers from "./sync-computers";
@@ -31,6 +33,13 @@ export default function cron() {
         task: pruneBotsQueue,
     });
 
+    registerCronJob({
+        title: "clear-job-steps",
+        schedule: "*/5 * * * *",
+        task: cronClearJobSteps,
+        quiet: true,
+    });
+
     registerCronJob({
         title: "prune-bot-hook-events",
         schedule: "@daily",
@@ -43,6 +52,12 @@ export default function cron() {
         task: cronPruneChatrooms,
     });
 
+    registerCronJob({
+        title: "prune-logs",
+        schedule: "@daily",
+        task: pruneLogs,
+    });
+
     registerCronJob({
         title: "sync-notify-watchers",
         schedule: "*/5 * * * *",

src/cron/ingest-pending-files/index.ts

@@ -1,31 +1,12 @@
 import BunSQLite from "@moduletrace/bun-sqlite";
-import type { BUN_SQLITE_COBBLE_BOT_FILES } from "@/db/types/db";
 import enqueueDocumentIngest from "@/src/file-ingest/enqueue-document-ingest";
-import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
 
 /**
- * Queues any bot file or knowledgebase document that never got ingested (an upload that happened
- * while the worker was down, a restart mid-ingest), so nothing is left
- * unsearchable. Runs every ten minutes.
+ * Queues any knowledgebase document that never got ingested (an upload
+ * that happened while the worker was down, a restart mid-ingest), so
+ * nothing is left unsearchable. Runs every ten minutes.
  */
 export default async function ingestPendingFiles() {
-    const res = await BunSQLite.sql<BUN_SQLITE_COBBLE_BOT_FILES>({
-        sql: `SELECT f.id FROM bot_files f
-            WHERE f.ingested_at IS NULL
-              AND f.ingest_error IS NULL
-              AND f.created_at > ?
-              AND NOT EXISTS (
-                  SELECT 1 FROM file_ingest_queue q
-                  WHERE q.file_id = f.id AND q.status IN ('pending', 'processing')
-              )
-            LIMIT 25`,
-        values: [Date.now() - 24 * 60 * 60 * 1000],
-    });
-
-    for (const file of res.payload || []) {
-        if (file.id) await enqueueFileIngest({ file_id: file.id });
-    }
-
     const documents = await BunSQLite.sql<{ id: number }>({
         sql: `SELECT d.id FROM knowledgebase_documents d
             WHERE d.ingested_at IS NULL
@@ -41,5 +22,5 @@ export default async function ingestPendingFiles() {
         if (document.id) await enqueueDocumentIngest({ document_id: document.id });
     }
 
-    return (res.payload?.length || 0) + (documents.payload?.length || 0);
+    return documents.payload?.length || 0;
 }

src/cron/prune-logs/index.ts

@@ -0,0 +1,15 @@
+import BunSQLite from "@moduletrace/bun-sqlite";
+import { grabLogsKeepDays } from "@/src/functions/backend/db/app-settings/grab-logs-keep-days";
+
+/**
+ * Deletes logs older than the install's retention (`grabLogsKeepDays`).
+ * Nothing points at a log, so no children go with them.
+ */
+export default async function pruneLogs() {
+    const cutoff = Date.now() - (await grabLogsKeepDays()) * 24 * 60 * 60 * 1000;
+
+    return await BunSQLite.sql({
+        sql: `DELETE FROM logs WHERE created_at < ? RETURNING id`,
+        values: [cutoff],
+    });
+}

src/data/app-data.ts

@@ -278,6 +278,13 @@ export const AppData = {
     BotHookMaxEventsPerRun: 20,
     BotHookEventsKeepDays: 30,
     BotHookEventsKeepDaysMax: 3650,
+    /**
+     * Logs (Activity) are deleted after this many days by the daily
+     * `prune-logs` cron: the default, with the most an install may set
+     * (`logs_keep_days`, Settings > General).
+     */
+    LogsKeepDays: 90,
+    LogsKeepDaysMax: 3650,
     /**
      * How many hook calls a list loads at once, and how much of each call's
      * payload and response a list shows (the whole of both opens on one).
@@ -354,6 +361,13 @@ export const AppData = {
      * One MCP tool call gives up after this long without progress.
      */
     McpToolTimeoutMs: 5 * 60 * 1000,
+    /**
+     * A bash command on a computer is killed there after at most this long
+     * (`withCommandTimeout`), below `McpToolTimeoutMs`: the sandbox ignores
+     * its own `timeout` and runs every call in one shell session, so a
+     * command left running would hold up every later one on that computer.
+     */
+    ComputerCommandMaxSeconds: 270,
     /**
      * MCP servers set up in Settings, under MCPs: a stdio server runs in
      * `DATA_DIR/<McpDataDir>/<id>` (its setup installs there, so it
@@ -414,6 +428,12 @@ export const AppData = {
      * working-on panel shows.
      */
     BotJobUpdatesShown: 12,
+    /**
+     * How long a finished job's steps stay before they're cleared
+     * (`clear-job-steps` cron). Long enough for every process's live relay
+     * to have pushed the last ones (`LiveRelayPollMs`).
+     */
+    BotJobStepsKeepMs: 60 * 1000,
     /**
      * How often each server process reads new bot messages, job steps and
      * bot changes from the database for its own sockets (`liveRelay`).
@@ -648,8 +668,9 @@ export const AppData = {
      */
     BotFileImportMaxBytes: 20 * 1024 * 1024,
     /**
-     * Ingesting a bot file: how much text one chunk holds, how much of the
-     * previous chunk it repeats, and the largest file it reads at all.
+     * Ingesting a knowledgebase document: how much text one chunk holds,
+     * how much of the previous chunk it repeats, and the largest file it
+     * reads at all (also the largest attachment a reply reads).
      */
     ChunkTargetChars: 1200,
     ChunkOverlapChars: 200,
@@ -668,12 +689,24 @@ export const AppData = {
     EmbeddingBatchSize: 32,
     EmbeddingMaxInputChars: 8000,
     /**
-     * How many chunks a file search returns, and how many more it asks
-     * the vector index for before filtering to the run's scopes.
+     * How many chunks a knowledgebase search returns, and how many more it
+     * asks the vector index for before filtering to the run's ones.
      */
     RagTopChunks: 8,
     RagOverFetch: 4,
     RagMaxK: 60,
+    /**
+     * How long a search box waits after the last keystroke before it
+     * searches.
+     */
+    SearchDebounceMs: 250,
+    /**
+     * The database is vacuumed at boot once this share of its pages, and
+     * at least this many bytes, sit free (after a big delete, such as the
+     * bot file index being dropped).
+     */
+    DbVacuumMinFreeRatio: 0.2,
+    DbVacuumMinFreeBytes: 4 * 1024 * 1024,
     /**
      * Knowledgebases: the most one document holds (an upload, or text
      * written in), how much of a pasted text a form takes, how many

src/data/harness-prompts.ts

@@ -60,18 +60,18 @@ export const botFilesNote = ({ bot_id }: { bot_id: number | string }) => `# Your
 You have a folder of your own, kept with the app (not on a computer), and it stays between runs and conversations.
 
 - list_files shows what's in it; read_file opens one (by its [file:ID] ref or path); write_file saves a text file, make_pdf a PDF, generate_image an image. Move and delete with move_file and delete_file when something is out of date, so the folder stays useful.
-- Text files are chunked and searchable as soon as they're saved. search_files finds the passages that answer a question, by meaning when an embedding model is set and by words otherwise. Search your files before saying you don't know something they may hold, and when the user asks about a document they gave you.
-- Uploaded files land in the same folder, under uploads/ (uploads/2026-10/logo.png), so what the user sent you earlier is searchable later too.
+- Files are found by name or path, never by what's in them: list_files with "search" matches names and paths, then read_file reads the one you want. Look for a file and read it before saying you don't know something it may hold, and when the user asks about a document they gave you. Searching by content is what knowledgebases are for; your files aren't indexed.
+- Uploaded files land in the same folder, under uploads/ (uploads/2026-10/logo.png), so what the user sent you earlier is there to list and read later too.
 - A picture can be set from a file: set_picture makes a file your own avatar, and update_workspace sets a workspace's. Both take the [file:ID] ref, so a logo the user sent you is usually ready to go: list_files for it, then set it. An SVG or another kind of file is turned into a PNG first. Its picture is free to change and never removes your file.
 - A file and a folder are two different refs, and you never guess which one from the name:
-    - A file you saved is a [file:ID] ref (from write_file, list_files, search_files or import_file). The user sees a chip that opens the file.
+    - A file you saved is a [file:ID] ref (from write_file, list_files or import_file). The user sees a chip that opens the file.
     - A folder is a folder ref, e.g. ${ownFolderRef({ bot_id })}: the user sees a chip that opens that folder in Files, and nothing is downloaded. Use the volume of the folder you mean ("bot" for your own, "workspace" for a workspace's shared one).
     - Never write a path as plain text and expect the user to find it; never write a [file:ID] ref for something that is a folder.
     - Write refs as they are, never in backticks or a code block: inside code they show as plain text, not a chip.
 - A file you only described is not saved. Save it first, then say it's saved.
 - Save what the user will want before the long tail of a run. A step limit ends a run where it stands: write the report first with write_file (or copy it off the computer with import_file), then carry on verifying.
-- One copy of a thing is enough. Before you write or import something, list_files (or search_files) for it: if it's already there, update it (write_file over the same path) instead of saving a second copy under a new name. Never split a file into parts unless the user asked for parts.
-- Your files don't disappear. If you can't find something, list_files or search_files before saying anything is missing, and never claim the folder is empty without having just listed it.
+- One copy of a thing is enough. Before you write or import something, list_files for it: if it's already there, update it (write_file over the same path) instead of saving a second copy under a new name. Never split a file into parts unless the user asked for parts.
+- Your files don't disappear. If you can't find something, list_files (with and without "search") before saying anything is missing, and never claim the folder is empty without having just listed it.
 - Report what the tools actually returned. Don't invent limits, sizes or caps you haven't seen in a tool result: if an import or a write failed, say what the tool said.`;
 
 /**
@@ -102,6 +102,7 @@ You work only in your own folders on the computer, never elsewhere in it: every
     dirs.shared_dir ? `\n- ${dirs.shared_dir}: shared with every bot in this workspace.` : ""
 }
 - The browser downloads to its own folder: move what you download into your folder straight away.
+- A computer can be restarted or rebuilt at any time, and only ${AppData.ComputerWorkspaceDir} and the browser's logins survive it. Programs you install (sudo apt-get install -y, pip, npm -g) may be gone afterwards: check for one first (command -v) and install it again when it's missing.
 
 What you make there (downloads, screenshots, a crawl, raw evidence, logs) is kept with the computer, not in your files, and it is not searchable.
 
@@ -219,7 +220,7 @@ You are working through a task on your own; the user isn't watching. Use your to
  */
 export const TasksNote = `# Tasks
 
-Tasks are the crew's shared to-do list, one per bot and one per workspace. A workspace's tasks are seen by every bot working there, and the coordinator sees everything.
+Tasks are the crew's to-do list. Every task belongs to one bot, on its own list or in a workspace. You see your own; a workspace's admin sees every task in that workspace, and the coordinator sees everything.
 
 - list_tasks shows what's open: yours, a workspace's, or (as the coordinator) all of it. Check it before you plan work, so you don't start something already tracked.
 - create_task adds one, with a brief: what to do, where, and what done looks like. Only add a task for work that will take more than this reply; never one for something you're doing now.
@@ -278,8 +279,8 @@ update_memory and update_instructions save to this workspace by default: what yo
 export const SessionNamePrompt = `Name and summarize this conversation from its first exchange. Answer with a JSON object only, no markdown fence and no other text:
 {"title": "...", "summary": "..."}
 
-- title: what the conversation is about, at most 60 characters, no quotes and no trailing period. Say what the user wanted, not that it is a conversation. When it opens with a summary of an earlier session, name what this one carries on with, not every topic the summary recaps.
-- summary: what the user wanted and what was done so far. Keep names, numbers, paths and links exactly. Plain markdown bullets, no preamble.`;
+- title: what the conversation is about, at most 60 characters, no quotes and no trailing period. Say what the user wanted, not that it is a conversation. With no user message, it is work handed to you (by the coordinator, a schedule or a hook): name that work. When it opens with a summary of an earlier session, name what this one carries on with, not every topic the summary recaps.
+- summary: what the user (or whoever handed you the work) wanted and what was done so far. Keep names, numbers, paths and links exactly. Plain markdown bullets, no preamble.`;
 
 /**
  * Asks the text model to carry a conversation into a new session when the

src/data/pwa-data.ts

@@ -160,6 +160,20 @@ export const PwaDefaultInstallIcon: PwaInstallIcon = {
 export const PwaCornerMarkInset = 0.2;
 export const PwaCornerMarkScale = 0.22;
 
+/**
+ * The corner mark on the regular (rounded-square) icons, which aren't cut
+ * to a circle: larger and nearer the corner, clear of its rounding.
+ */
+export const PwaSquareCornerMarkInset = 0.16;
+export const PwaSquareCornerMarkScale = 0.3;
+
+/**
+ * The corner mark on a round favicon: larger, to stay readable at 32px,
+ * and still inside the circle.
+ */
+export const PwaFaviconCornerMarkInset = 0.15;
+export const PwaFaviconCornerMarkScale = 0.32;
+
 /**
  * Uploads must be at least this many pixels on their shorter side, and are
  * kept as a square of the source size.
@@ -167,12 +181,18 @@ export const PwaCornerMarkScale = 0.22;
 export const PwaInstallIconMinSide = 512;
 export const PwaInstallIconSourceSide = 1024;
 
+/**
+ * Bump when `renderInstallIcons` draws differently: it's part of a set's
+ * version, so a re-saved icon gets a new URL past the `immutable` cache.
+ */
+export const PwaInstallIconRenderRevision = 4;
+
 export type PwaInstallIconFile = {
     name: string;
     size: number;
     /**
      * `plain` is the image without the mark (the settings preview);
-     * `favicon` never gets the mark (unreadable at 32px).
+     * `favicon` is cut to a circle, the mark kept inside it.
      */
     kind: "any" | "maskable" | "favicon" | "plain";
 };

src/dict/log-categories.ts

@@ -23,4 +23,10 @@ export const LogCategories = [
         title: "Error",
         value: "error",
     },
-] as const;
+] as const;
+
+/**
+ * The app setting (Settings > General) for how many days logs are kept,
+ * else `AppData.LogsKeepDays`.
+ */
+export const LogsKeepDaysSettingKey = "logs_keep_days";

src/dict/task-dict.ts

@@ -1,6 +1,6 @@
 /**
- * A task's status. Tasks are the crew's shared to-do list, one per bot and
- * one per workspace, and every move through this list is a step in the
+ * A task's status. Tasks are the crew's to-do list, every task one bot's
+ * (on its own list or in a workspace), and every move through this list is a step in the
  * work: `blocked` is what a bot (or the user) sets when only the user can
  * unblock it, and `done` is only ever set once the work was verified.
  */

src/file-ingest/enqueue-file-ingest.ts

@@ -1,37 +0,0 @@
-import BunSQLite from "@moduletrace/bun-sqlite";
-import type { BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE } from "@/db/types/db";
-import wakeFileIngest from "./wake-file-ingest";
-
-type Params = {
-    file_id: number | string;
-};
-
-/**
- * Queues a file to be chunked and embedded, unless it's already waiting.
- * Never fails the caller: a file that can't be queued is picked up by the
- * `ingest-pending-files` cron instead.
- */
-export default async function enqueueFileIngest({ file_id }: Params) {
-    try {
-        const open = await BunSQLite.sql<BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE>({
-            sql: `SELECT id FROM file_ingest_queue WHERE file_id = ? AND status IN ('pending', 'processing') LIMIT 1`,
-            values: [file_id],
-        });
-
-        if (open.payload?.[0]?.id) {
-            wakeFileIngest();
-            return;
-        }
-
-        await BunSQLite.queue.add({
-            table: "file_ingest_queue",
-            data: [{ file_id: Number(file_id) }],
-        });
-
-        wakeFileIngest();
-    } catch {
-        /**
-         * The cron sweeps it up.
-         */
-    }
-}

src/file-ingest/index.ts

@@ -14,9 +14,9 @@ let timer: ReturnType<typeof setTimeout> | undefined;
 /**
  * Starts the file ingestion worker: polls every `AppData.FileIngestPollMs`
  * for `file_ingest_queue` jobs and runs up to
- * `AppData.FileIngestConcurrency` at once (chunking and embedding a bot's
- * file). Gated in production to the `MAIN_SERVER` container, like cron and
- * the bots queue.
+ * `AppData.FileIngestConcurrency` at once (chunking and embedding a
+ * knowledgebase document). Gated in production to the `MAIN_SERVER`
+ * container, like cron and the bots queue.
  */
 export default async function fileIngest() {
     if (process.env.NODE_ENV == "production" && !process.env.MAIN_SERVER) {

src/file-ingest/process-ingest-job.ts

@@ -1,6 +1,5 @@
 import BunSQLite from "@moduletrace/bun-sqlite";
 import type { BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE } from "@/db/types/db";
-import ingestBotFile from "@/src/functions/backend/db/bot-files/ingest-bot-file";
 import ingestKnowledgebaseDocument from "@/src/functions/backend/db/knowledgebases/ingest-knowledgebase-document";
 import { logError } from "@/src/functions/backend/logger";
 import settleFailedIngestJob from "./settle-failed-ingest-job";
@@ -10,56 +9,30 @@ type Params = {
 };
 
 /**
- * Ingests one queued file (or knowledgebase document) and completes (or
- * fails) its job. One that was deleted meanwhile is simply done.
+ * Ingests one queued knowledgebase document and completes (or fails) its
+ * job. One that was deleted meanwhile is simply done, and so is a job
+ * without a document (bot files are no longer indexed).
  */
 export default async function processIngestJob({ job }: Params) {
     if (!job.id) return;
 
-    if (job.document_id) return await processDocumentJob({ job_id: job.id, document_id: job.document_id });
-
-    if (!job.file_id) {
+    if (!job.document_id) {
         await BunSQLite.queue.complete({ table: "file_ingest_queue", id: job.id }).catch(() => {});
         return;
     }
 
     try {
-        const result = await ingestBotFile({ file_id: job.file_id });
-
+        await ingestKnowledgebaseDocument({ document_id: job.document_id });
         await BunSQLite.queue.complete({ table: "file_ingest_queue", id: job.id });
-
-        if (result.status == "ingested") {
-            BunSQLite.sql({
-                sql: `UPDATE bot_files SET ingest_error = NULL WHERE id = ?`,
-                values: [job.file_id],
-            }).catch(() => {});
-        }
     } catch (error) {
         const { deferred } = await settleFailedIngestJob({ job_id: job.id, error });
 
         if (deferred) return;
 
         await logError({
-            title: `Bot file #${job.file_id} couldn't be ingested`,
+            title: `Knowledgebase document #${job.document_id} couldn't be ingested`,
             error,
-            metadata: { file_id: job.file_id },
-        }).catch(() => {});
-    }
-}
-
-async function processDocumentJob({ job_id, document_id }: { job_id: number; document_id: number }) {
-    try {
-        await ingestKnowledgebaseDocument({ document_id });
-        await BunSQLite.queue.complete({ table: "file_ingest_queue", id: job_id });
-    } catch (error) {
-        const { deferred } = await settleFailedIngestJob({ job_id, error });
-
-        if (deferred) return;
-
-        await logError({
-            title: `Knowledgebase document #${document_id} couldn't be ingested`,
-            error,
-            metadata: { document_id },
+            metadata: { document_id: job.document_id },
         }).catch(() => {});
     }
 }

src/functions/backend/database/build-table-catalog-sql.ts

@@ -5,7 +5,7 @@
  * table name as an argument the way a table-valued function does.
  *
  * Left out: SQLite's own `sqlite_*` tables, and the shadow tables a
- * virtual table keeps its internals in (the `bot_file_vectors_*` tables
+ * virtual table keeps its internals in (the `knowledgebase_vectors_*` tables
  * behind the sqlite-vec index), which are storage, not rows anyone
  * browses.
  */

src/functions/backend/db/app-settings/grab-logs-keep-days.ts

@@ -0,0 +1,30 @@
+import { AppData } from "@/src/data/app-data";
+import { LogsKeepDaysSettingKey } from "@/src/dict/log-categories";
+import type { LogRetentionOverview } from "@/src/types";
+import readStepLimit from "@/src/utils/read-step-limit";
+import grabAppSetting from "./grab-app-setting";
+
+/**
+ * How many days logs are kept: the install's own number (`logs_keep_days`,
+ * Settings > General), else `AppData.LogsKeepDays`.
+ */
+export async function grabLogsKeepDays() {
+    return readStepLimit({
+        saved: await grabAppSetting({ key: LogsKeepDaysSettingKey }),
+        fallback: AppData.LogsKeepDays,
+        max: AppData.LogsKeepDaysMax,
+    });
+}
+
+/**
+ * The number with its default and ceiling, for the settings page.
+ */
+export default async function grabLogRetentionOverview() {
+    const overview: LogRetentionOverview = {
+        keep_days: await grabLogsKeepDays(),
+        default_days: AppData.LogsKeepDays,
+        max: AppData.LogsKeepDaysMax,
+    };
+
+    return overview;
+}

src/functions/backend/db/bot-files/delete-bot-file.ts

@@ -15,7 +15,7 @@ type Params = {
 };
 
 /**
- * Deletes a bot file: the catalog row (its chunks go with it), its bytes
+ * Deletes a bot file: the catalog row, its bytes
  * and its `media` row. Refuses anything outside the run's directories.
  */
 export default async function deleteBotFile({ scopes, file_id }: Params) {
@@ -32,22 +32,7 @@ export default async function deleteBotFile({ scopes, file_id }: Params) {
         rmSync(absolute, { force: true });
     }
 
-    await BunSQLite.sql({ sql: `DELETE FROM bot_file_vectors WHERE file_id = ?`, values: [file.id] });
-    /**
-     * Deleted here rather than left to the schema's cascade: SQLite has
-     * foreign keys off per connection unless `enableForeignKeys` ran, and
-     * a script or CLI process that deletes a file would otherwise leave
-     * its chunks behind, which then fail the next schema sync.
-     */
-    await BunSQLite.sql({ sql: `DELETE FROM bot_file_chunks WHERE file_id = ?`, values: [file.id] });
     await BunSQLite.sql({ sql: `DELETE FROM bot_files WHERE id = ?`, values: [file.id] });
-    /**
-     * Its pending ingestion would only find nothing left.
-     */
-    await BunSQLite.sql({
-        sql: `DELETE FROM file_ingest_queue WHERE file_id = ? AND status IN ('pending', 'processing')`,
-        values: [file.id],
-    }).catch(() => {});
 
     if (file.media_id) {
         const media = await BunSQLite.sql<{ media_write_path?: string; media_thumbnail_write_path?: string }>({

src/functions/backend/db/bot-files/grab-conversation-files.ts

@@ -53,8 +53,6 @@ export default async function grabConversationFiles({
                 purpose: String(file.purpose || "writing"),
                 scope: String(file.scope || "bot"),
                 owner_id: Number(file.owner_id) || 0,
-                ingested_at: file.ingested_at ? Number(file.ingested_at) : undefined,
-                ingest_error: file.ingest_error || undefined,
                 created_at: Number(file.created_at) || 0,
                 url: media_url || "",
             };

src/functions/backend/db/bot-files/ingest-bot-file.ts

@@ -1,329 +0,0 @@
-import BunSQLite from "@moduletrace/bun-sqlite";
-import type {
-    BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS,
-    BUN_SQLITE_COBBLE_BOT_FILES,
-    BunSQLiteTables,
-} from "@/db/types/db";
-import { AppData } from "@/src/data/app-data";
-import embedText from "@/src/harness/embeddings/embed-text";
-import HarnessError from "@/src/harness/harness-error";
-import { grabHarnessUsageScope } from "@/src/harness/usage/harness-usage-scope";
-import type { HarnessUsageScope } from "@/src/harness/usage/types";
-import runWithFallbackScope from "@/src/harness/usage/run-with-fallback-scope";
-import grabUsageOwnerUserId from "@/src/functions/backend/db/usage/grab-usage-owner";
-import chunkText from "./chunk-text";
-import extractBotFileText from "./extract-bot-file-text";
-
-type Params = {
-    file_id: number | string;
-    /**
-     * Read images and audio too (a bot asking for it).
-     */
-    read_media?: boolean;
-    signal?: AbortSignal;
-};
-
-export type IngestResult = {
-    /**
-     * `ingested`, `empty` (no text to read), `skipped` (too large) or
-     * `unavailable` (no embedding model), which keeps the file's text but
-     * leaves it searchable by words only.
-     */
-    status: "ingested" | "empty" | "skipped" | "unavailable";
-    chunks: number;
-};
-
-/**
- * Reads a bot file's text, splits it into chunks and embeds them into
- * `bot_file_vectors`, replacing whatever was stored before. The chunks
- * stay in `bot_file_chunks` even when there's no embedding model, so a
- * search can still match words in them.
- *
- * Failures are recorded on the file (`ingest_error`) and rethrown, so the
- * queue can retry.
- */
-export default async function ingestBotFile({
-    file_id,
-    read_media,
-    signal,
-}: Params): Promise<IngestResult> {
-    const file = await BunSQLite.sql<BUN_SQLITE_COBBLE_BOT_FILES>({
-        sql: `SELECT * FROM bot_files WHERE id = ? LIMIT 1`,
-        values: [file_id],
-    }).then((res) => res.payload?.[0]);
-
-    if (!file?.id) return { status: "empty", chunks: 0 };
-
-    const media = file.media_id
-        ? await BunSQLite.sql<{ media_write_path?: string; text_content?: string }>({
-              sql: `SELECT media_write_path, text_content FROM media WHERE id = ? LIMIT 1`,
-              values: [file.media_id],
-          }).then((res) => res.payload?.[0])
-        : undefined;
-
-    /**
-     * Decided on the file's size alone. A cached `text_content` is not a
-     * reason to read an oversized file: it was only ever a slice of it,
-     * and taking it as the whole document made a file too big to unpack
-     * look already read.
-     */
-    if (Number(file.size) > AppData.IngestMaxBytes) {
-        await BunSQLite.sql({
-            sql: `UPDATE bot_files SET ingest_error = ?, updated_at = ? WHERE id = ?`,
-            values: [`Too large to read (over ${AppData.IngestMaxBytes / (1024 * 1024)} MB).`, Date.now(), file.id],
-        });
-
-        return { status: "skipped", chunks: 0 };
-    }
-
-    /**
-     * A bot's own run already carries its user. The ingest queue has no run
-     * behind it, so everything this file costs - reading an image or an
-     * audio file, embedding its chunks - is entered under the owner's scope
-     * instead: it spends real tokens and must show on the usage page.
-     */
-    const owner_scope: HarnessUsageScope | null = grabHarnessUsageScope()?.user_id
-        ? null
-        : await resolveOwnerScope({ file });
-
-    try {
-        const text = await runWithFallbackScope({
-            scope: owner_scope,
-            task: () =>
-                extractBotFileText({
-                    write_path: media?.media_write_path,
-                    text_content: media?.text_content,
-                    name: file.name,
-                    mime_type: file.mime_type,
-                    read_media,
-                }),
-        });
-
-        await clearChunks({ file_id: file.id });
-
-        if (!text?.trim()) {
-            await markIngested({ file_id: file.id, chunks: 0 });
-
-            return { status: "empty", chunks: 0 };
-        }
-
-        const chunks = chunkText({ text });
-
-        if (!chunks.length) {
-            await markIngested({ file_id: file.id, chunks: 0 });
-
-            return { status: "empty", chunks: 0 };
-        }
-
-        const chunk_ids = await insertChunks({ file, chunks });
-        let embedded = 0;
-
-        try {
-            /**
-             * A big file's chunks go in batches, so one provider refusal
-             * doesn't throw away the whole file.
-             */
-            for (let start = 0; start < chunks.length; start += AppData.EmbeddingBatchSize) {
-                const batch = chunks.slice(start, start + AppData.EmbeddingBatchSize);
-                const { vectors } = await runEmbed({
-                    scope: owner_scope,
-                    signal,
-                    file,
-                    batch,
-                });
-
-                for (let i = 0; i < vectors.length; i++) {
-                    await insertVector({
-                        vector: vectors[i]!,
-                        chunk_id: chunk_ids[start + i]!,
-                        file_id: file.id,
-                        scope: String(file.scope),
-                        owner_id: Number(file.owner_id),
-                    });
-
-                    embedded++;
-                }
-            }
-        } catch (error) {
-            if (error instanceof HarnessError && error.code == "unavailable") {
-                /**
-                 * No embedding model: the chunks still let a search match
-                 * words, so the file isn't a failure.
-                 */
-                await markIngested({
-                    file_id: file.id,
-                    chunks: chunks.length,
-                    error: "No embedding model is set, so this file is searchable by words only.",
-                });
-
-                return { status: "unavailable", chunks: chunks.length };
-            }
-
-            throw error;
-        }
-
-        await markIngested({ file_id: file.id, chunks: chunks.length });
-
-        return { status: "ingested", chunks: embedded };
-    } catch (error) {
-        /**
-         * Every model resting means nothing was tried: the job waits for
-         * them, and the file keeps whatever it said before.
-         */
-        if (error instanceof HarnessError && error.resting) throw error;
-
-        const message = error instanceof Error ? error.message : String(error);
-
-        await BunSQLite.sql({
-            sql: `UPDATE bot_files SET ingest_error = ?, updated_at = ? WHERE id = ?`,
-            values: [message.slice(0, 500), Date.now(), file.id],
-        }).catch(() => {});
-
-        throw error;
-    }
-}
-
-async function clearChunks({ file_id }: { file_id: number }) {
-    await BunSQLite.sql({ sql: `DELETE FROM bot_file_vectors WHERE file_id = ?`, values: [file_id] });
-    await BunSQLite.sql({ sql: `DELETE FROM bot_file_chunks WHERE file_id = ?`, values: [file_id] });
-}
-
-async function insertChunks({
-    file,
-    chunks,
-}: {
-    file: BUN_SQLITE_COBBLE_BOT_FILES;
-    chunks: string[];
-}) {
-    const ids: number[] = [];
-
-    for (let index = 0; index < chunks.length; index++) {
-        const text = chunks[index]!;
-
-        const res = await BunSQLite.insert<
-            BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS,
-            (typeof BunSQLiteTables)[number]
-        >({
-            table: "bot_file_chunks",
-            data: [
-                {
-                    file_id: file.id,
-                    scope: file.scope as any,
-                    owner_id: Number(file.owner_id),
-                    chunk_index: index,
-                    text,
-                    token_estimate: Math.ceil(text.length / 4),
-                },
-            ],
-        });
-
-        const id = res.postInsertReturn?.insertId;
-
-        if (id) ids.push(id);
-    }
-
-    return ids;
-}
-
-async function insertVector({
-    vector,
-    chunk_id,
-    file_id,
-    scope,
-    owner_id,
-}: {
-    vector: Float32Array;
-    chunk_id: number;
-    file_id: number;
-    scope: string;
-    owner_id: number;
-}) {
-    /**
-     * vec0's metadata columns (its id and timestamps) are NOT NULL, so a
-     * raw insert has to fill them; `BunSQLite.insert` would add them but
-     * the vector needs `vec_f32`.
-     */
-    const now = Date.now();
-
-    const res = await BunSQLite.sql({
-        sql: `INSERT INTO bot_file_vectors (created_at, updated_at, embedding, chunk_id, file_id, scope, owner_id) VALUES (?, ?, vec_f32(?), ?, ?, ?, ?)`,
-        values: [now, now, vector as any, chunk_id, file_id, scope, owner_id],
-    });
-
-    if (!res.success) {
-        throw new Error(`Couldn't store an embedding: ${res.error || res.msg}`);
-    }
-}
-
-async function markIngested({
-    file_id,
-    chunks,
-    error,
-}: {
-    file_id: number;
-    chunks: number;
-    error?: string;
-}) {
-    const now = Date.now();
-
-    await BunSQLite.sql({
-        sql: `UPDATE bot_files SET ingested_at = ?, ingest_error = ?, updated_at = ? WHERE id = ?`,
-        values: [now, error || null, now, file_id],
-    });
-
-    await BunSQLite.sql({
-        sql: `UPDATE media SET is_rag_ingested = ?, updated_at = ? WHERE id = (SELECT media_id FROM bot_files WHERE id = ?)`,
-        values: [chunks > 0 ? 1 : 0, now, file_id],
-    }).catch(() => {});
-}
-
-/**
- * One batch of a file's chunks, embedded under whatever attribution owns
- * it: the run that asked for the ingest, else the file's owner.
- */
-async function runEmbed({
-    scope,
-    signal,
-    file,
-    batch,
-}: {
-    scope: HarnessUsageScope | null;
-    signal?: AbortSignal;
-    file: BUN_SQLITE_COBBLE_BOT_FILES;
-    batch: string[];
-}) {
-    return await runWithFallbackScope({
-        scope,
-        task: () =>
-            embedText({
-                input: batch,
-                bot_id: file.bot_id!,
-                workspace_id: file.scope == "workspace" ? file.owner_id : null,
-                signal,
-            }),
-    });
-}
-
-/**
- * The attribution a file the queue ingests reports under: its workspace's
- * creator when it shares one, else its bot's.
- */
-async function resolveOwnerScope({
-    file,
-}: {
-    file: BUN_SQLITE_COBBLE_BOT_FILES;
-}): Promise<HarnessUsageScope | null> {
-    const workspace_id = file.scope == "workspace" ? file.owner_id : null;
-    const user_id = await grabUsageOwnerUserId({ bot_id: file.bot_id, workspace_id });
-
-    if (!user_id) return null;
-
-    return {
-        user_id,
-        bot_id: file.scope == "bot" ? file.owner_id : file.bot_id,
-        workspace_id,
-        session_id: null,
-        message_id: null,
-        job_id: null,
-    };
-}

src/functions/backend/db/bot-files/purge-storage.ts

@@ -54,22 +54,6 @@ export default async function purgeStorage({ scope, owner_id, media_ids }: Param
             }).catch(() => {});
         }
 
-        await BunSQLite.sql({
-            sql: `DELETE FROM bot_file_vectors WHERE scope = ? AND owner_id = ?`,
-            values: [scope, owner_id],
-        }).catch(() => {});
-
-        /**
-         * Deleted here rather than left to the cascade, since SQLite has
-         * foreign keys off per connection unless `enableForeignKeys` ran (a
-         * script or CLI process never does), and a chunk left behind fails
-         * the next schema sync.
-         */
-        await BunSQLite.sql({
-            sql: `DELETE FROM bot_file_chunks WHERE scope = ? AND owner_id = ?`,
-            values: [scope, owner_id],
-        }).catch(() => {});
-
         /**
          * Usually gone already, with the bot or workspace (the cascade).
          */

src/functions/backend/db/bot-files/record-bot-file.ts

@@ -104,7 +104,7 @@ export default async function recordBotFile({
     if (existing?.id) {
         await BunSQLite.sql({
             sql: `UPDATE bot_files SET media_id = ?, name = ?, folder = ?, mime_type = ?, media_type = ?, size = ?,
-                purpose = ?, source = ?, ingested_at = NULL, ingest_error = NULL, updated_at = ? WHERE id = ?`,
+                purpose = ?, source = ?, updated_at = ? WHERE id = ?`,
             values: [
                 media_id,
                 resolved.name,
@@ -129,8 +129,6 @@ export default async function recordBotFile({
             size: bytes.byteLength,
             purpose,
             source,
-            ingested_at: undefined,
-            ingest_error: undefined,
             updated_at: now,
         };
     }

src/functions/backend/db/bot-files/search-bot-files.ts

@@ -1,271 +0,0 @@
-import BunSQLite from "@moduletrace/bun-sqlite";
-import { AppData } from "@/src/data/app-data";
-import { embedQuery } from "@/src/harness/embeddings/embed-text";
-import HarnessError from "@/src/harness/harness-error";
-import type { StorageScope } from "@/src/utils/storage-access";
-
-type Params = {
-    scopes: StorageScope[];
-    query: string;
-    /**
-     * Only this file's chunks.
-     */
-    file_id?: number | string | null;
-    limit?: number;
-    signal?: AbortSignal;
-    /**
-     * Whose embedding model to use.
-     */
-    bot_id?: number | string;
-    workspace_id?: number | string | null;
-};
-
-export type BotFileSearchHit = {
-    file_id: number;
-    chunk_id: number;
-    chunk_index: number;
-    name: string;
-    path: string;
-    scope: string;
-    /**
-     * Cosine distance when the vector index answered, null for a word
-     * match.
-     */
-    distance: number | null;
-    text: string;
-};
-
-export type BotFileSearchResult = {
-    hits: BotFileSearchHit[];
-    /**
-     * How the search ran: `vector`, `words` (no embedding model, or nothing
-     * matched) or `none` (no files at all).
-     */
-    mode: "vector" | "words" | "none";
-    /**
-     * Said plainly for the bot's own reading of what it got back.
-     */
-    note: string;
-};
-
-/**
- * Searches a run's files: by meaning through the embedding vectors when a
- * model is set, else (or when nothing matches) by words over the stored
- * chunks. Never throws for an unavailable model: the bot always gets
- * something it can act on.
- */
-export default async function searchBotFiles({
-    scopes,
-    query,
-    file_id,
-    limit = AppData.RagTopChunks,
-    signal,
-    bot_id,
-    workspace_id,
-}: Params): Promise<BotFileSearchResult> {
-    const text = query.trim();
-
-    if (!scopes.length || !text) {
-        return { hits: [], mode: "none", note: "No files to search." };
-    }
-
-    try {
-        const vector = await embedQuery({
-            text,
-            signal,
-            bot_id,
-            workspace_id,
-        });
-
-        const hits = await vectorSearch({
-            scopes,
-            query_vector: vector,
-            file_id,
-            limit,
-            seek: Math.min(limit * AppData.RagOverFetch, AppData.RagMaxK),
-        });
-
-        if (hits.length) {
-            return {
-                hits,
-                mode: "vector",
-                note: `${hits.length} passage(s), found by meaning.`,
-            };
-        }
-    } catch (error) {
-        if (!(error instanceof HarnessError)) throw error;
-    }
-
-    const hits = await wordSearch({ scopes, query: text, file_id, limit });
-
-    return {
-        hits,
-        mode: hits.length ? "words" : "none",
-        note: hits.length
-            ? `${hits.length} passage(s), found by matching words (no embedding model, or nothing matched by meaning).`
-            : `Nothing in your files matched "${text}".`,
-    };
-}
-
-/**
- * The nearest chunks by vector. sqlite-vec refuses a filter on a vec0
- * side-car column in the same query as the match, so the index is asked
- * for its nearest `seek` and the run's scopes are applied to that list
- * here, before the chunk text is read.
- */
-async function vectorSearch({
-    scopes,
-    query_vector,
-    file_id,
-    limit,
-    seek,
-}: {
-    scopes: StorageScope[];
-    query_vector: Float32Array;
-    file_id?: number | string | null;
-    limit: number;
-    seek: number;
-}): Promise<BotFileSearchHit[]> {
-    const nearest = await BunSQLite.sql<{
-        chunk_id: number;
-        file_id: number;
-        scope: string;
-        owner_id: number;
-        distance: number;
-    }>({
-        sql: `SELECT chunk_id, file_id, scope, owner_id, distance FROM bot_file_vectors WHERE embedding MATCH vec_f32(?) AND k = ? ORDER BY distance`,
-        values: [query_vector as any, seek],
-    });
-
-    /**
-     * `BunSQLite.sql` resolves with `success: false` instead of throwing, so
-     * a broken vector query would otherwise read as "nothing matched" and
-     * silently fall back to words. A failure here is a real error.
-     */
-    if (!nearest.success) {
-        throw new Error(`The file search index couldn't be read: ${nearest.error || nearest.msg}`);
-    }
-
-    const allowed = nearest.payload?.filter((row) =>
-        scopes.some((scope) => scope.scope == row.scope && Number(scope.owner_id) == Number(row.owner_id)),
-    );
-
-    const picked = (allowed || [])
-        .filter((row) => !file_id || Number(row.file_id) == Number(file_id))
-        .slice(0, limit);
-
-    if (!picked.length) return [];
-
-    const chunk_ids = picked.map((row) => Number(row.chunk_id));
-    const placeholders = chunk_ids.map(() => "?").join(", ");
-
-    const chunks = await BunSQLite.sql<{
-        id: number;
-        text: string;
-        chunk_index: number;
-        name: string;
-        path: string;
-    }>({
-        sql: `SELECT c.id, c.text, c.chunk_index, f.name, f.path FROM bot_file_chunks c
-            JOIN bot_files f ON f.id = c.file_id
-            WHERE c.id IN (${placeholders})`,
-        values: chunk_ids,
-    });
-
-    const by_id = new Map((chunks.payload || []).map((row) => [Number(row.id), row]));
-
-    return picked.flatMap((row) => {
-        const chunk = by_id.get(Number(row.chunk_id));
-
-        if (!chunk) return [];
-
-        return [
-            toHit({
-                chunk_id: row.chunk_id,
-                file_id: row.file_id,
-                scope: row.scope,
-                distance: Number(row.distance),
-                text: chunk.text,
-                chunk_index: chunk.chunk_index,
-                name: chunk.name,
-                path: chunk.path,
-            }),
-        ];
-    });
-}
-
-/**
- * Chunks whose text has the query's words, newest files first. Used when
- * there's no embedding model, and as the fallback when the vectors
- * answered nothing.
- */
-async function wordSearch({
-    scopes,
-    query,
-    file_id,
-    limit,
-}: {
-    scopes: StorageScope[];
-    query: string;
-    file_id?: number | string | null;
-    limit: number;
-}): Promise<BotFileSearchHit[]> {
-    const words = [
-        ...new Set(
-            query
-                .toLowerCase()
-                .split(/[^a-z0-9]+/)
-                .filter((word) => word.length > 2),
-        ),
-    ].slice(0, 6);
-
-    if (!words.length) return [];
-
-    const scope_sql = scopes.map(() => `(c.scope = ? AND c.owner_id = ?)`).join(" OR ");
-    const scope_values = scopes.flatMap((s) => [s.scope, s.owner_id]);
-    const word_sql = words.map(() => `lower(c.text) LIKE ?`).join(" OR ");
-    const word_values = words.map((word) => `%${word}%`);
-    const file_clause = file_id ? "AND c.file_id = ?" : "";
-
-    const res = await BunSQLite.sql<{
-        chunk_id: number;
-        file_id: number;
-        text: string;
-        chunk_index: number;
-        name: string;
-        path: string;
-        scope: string;
-    }>({
-        sql: `SELECT c.id AS chunk_id, c.file_id, c.text, c.chunk_index, c.scope, f.name, f.path
-            FROM bot_file_chunks c
-            JOIN bot_files f ON f.id = c.file_id
-            WHERE (${scope_sql}) ${file_clause} AND (${word_sql})
-            ORDER BY f.updated_at DESC, c.chunk_index
-            LIMIT ?`,
-        values: [...scope_values, ...(file_id ? [file_id] : []), ...word_values, limit],
-    });
-
-    return (res.payload || []).map((row) => toHit({ ...row, distance: null }));
-}
-
-function toHit(row: {
-    chunk_id: number;
-    file_id: number;
-    text: string;
-    chunk_index: number;
-    name: string;
-    path: string;
-    scope: string;
-    distance: number | null;
-}): BotFileSearchHit {
-    return {
-        file_id: Number(row.file_id),
-        chunk_id: Number(row.chunk_id),
-        chunk_index: Number(row.chunk_index),
-        name: row.name || "",
-        path: row.path || "",
-        scope: row.scope || "bot",
-        distance: row.distance == null ? null : Number(row.distance),
-        text: row.text || "",
-    };
-}

src/functions/backend/db/bot-files/store-chat-attachments.test.ts

@@ -13,10 +13,8 @@ async function source() {
 describe("storeChatAttachments", () => {
     /**
      * The same text used to be copied onto the new `bot-file` media row,
-     * so a chat upload kept a second copy of its extracted text (and left
-     * it behind when the upload was pruned). The text belongs only on the
-     * message's own `bot-attachment` row, where a reply reads it;
-     * ingestion re-reads the bytes through `extractBotFileText`.
+     * so a chat upload kept a second copy of its extracted text. Bot files
+     * keep no text at all: a bot reads the bytes when it needs them.
      */
     test("never writes text_content onto the file's media row", async () => {
         const code = await source();
@@ -25,10 +23,10 @@ describe("storeChatAttachments", () => {
         expect(code).not.toContain("text_content = ?");
     });
 
-    test("still saves the bytes and queues the ingest", async () => {
+    test("saves the bytes and indexes nothing", async () => {
         const code = await source();
 
         expect(code).toContain("recordBotFile(");
-        expect(code).toContain("enqueueFileIngest(");
+        expect(code).not.toContain("enqueueFileIngest(");
     });
 });

src/functions/backend/db/bot-files/store-chat-attachments.ts

@@ -1,7 +1,6 @@
 import mediaFilePath from "@/src/utils/media-file-path";
 import type { BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db";
 import { BotUploadFolder } from "@/src/dict/bot-file-dict";
-import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
 import { logError } from "@/src/functions/backend/logger";
 import recordBotFile from "./record-bot-file";
 
@@ -14,7 +13,7 @@ type Params = {
 
 /**
  * Copies the files the user attached to a message into the conversation's
- * folder (`uploads/<yyyy-mm>/<name>`), so they stay searchable after the
+ * folder (`uploads/<yyyy-mm>/<name>`), so they stay findable after the
  * message scrolls away and a later run can find them. Best effort: a
  * failure is logged, never shown, and the message still goes out.
  */
@@ -48,8 +47,8 @@ export default async function storeChatAttachments({
              * The text stays on the message's own `bot-attachment` row,
              * where a reply reads it. Copying it onto the file's row kept
              * the same text twice and left the copy behind when the
-             * upload was pruned; ingestion re-reads the bytes once
-             * through `extractBotFileText` instead.
+             * upload was pruned; a bot reads the bytes with `read_file`
+             * instead.
              */
             const saved = await recordBotFile({
                 bot_id,
@@ -63,8 +62,6 @@ export default async function storeChatAttachments({
                 mime_type: file.mime_type || undefined,
                 user_id,
             });
-
-            await enqueueFileIngest({ file_id: saved.id! });
         } catch (error) {
             await logError({
                 title: `A chat attachment couldn't be saved to the bot's files`,

src/functions/backend/db/bot-files/vector-search.test.ts

@@ -1,132 +0,0 @@
-import { afterAll, describe, expect, test } from "bun:test";
-import { Database } from "bun:sqlite";
-import * as sqliteVec from "sqlite-vec";
-import fitVectorDimensions from "@/src/harness/embeddings/fit-vector-dimensions";
-
-/**
- * The exact insert and nearest-neighbour shape `ingestBotFile` and
- * `searchBotFiles` use, on a scratch database: vec0's metadata columns are
- * NOT NULL, and a side-car filter can't share the match query.
- */
-function createDb() {
-    const db = new Database(":memory:");
-    sqliteVec.load(db);
-    db.run(`CREATE VIRTUAL TABLE bot_file_vectors USING vec0(
-        id INTEGER PRIMARY KEY AUTOINCREMENT,
-        created_at INTEGER,
-        updated_at INTEGER,
-        embedding float[8],
-        +chunk_id INTEGER,
-        +file_id INTEGER,
-        +scope TEXT,
-        +owner_id INTEGER
-    )`);
-
-    return db;
-}
-
-const vector = (values: number[]) => fitVectorDimensions({ vector: values, dimensions: 8 });
-
-const insert = (db: Database, row: { chunk_id: number; file_id: number; scope: string; owner_id: number; v: number[] }) => {
-    const now = Date.now();
-
-    db.run(
-        `INSERT INTO bot_file_vectors (created_at, updated_at, embedding, chunk_id, file_id, scope, owner_id) VALUES (?, ?, vec_f32(?), ?, ?, ?, ?)`,
-        [now, now, vector(row.v), row.chunk_id, row.file_id, row.scope, row.owner_id],
-    );
-};
-
-describe("bot file vectors", () => {
-    test("stores a vector and matches the nearest one", () => {
-        const db = createDb();
-
-        insert(db, { chunk_id: 1, file_id: 10, scope: "bot", owner_id: 1, v: [1, 0, 0, 0] });
-        insert(db, { chunk_id: 2, file_id: 11, scope: "bot", owner_id: 1, v: [0, 1, 0, 0] });
-
-        const rows = db
-            .query(
-                `SELECT chunk_id, file_id, distance FROM bot_file_vectors WHERE embedding MATCH vec_f32(?1) AND k = 2 ORDER BY distance`,
-            )
-            .all(vector([0.9, 0.1, 0, 0])) as { chunk_id: number; distance: number }[];
-
-        expect(rows.length).toBe(2);
-        expect(rows[0]!.chunk_id).toBe(1);
-        expect(rows[0]!.distance).toBeLessThan(rows[1]!.distance);
-    });
-
-    test("filters scopes in the app, not in the match query", () => {
-        const db = createDb();
-
-        insert(db, { chunk_id: 1, file_id: 10, scope: "bot", owner_id: 1, v: [1, 0] });
-        insert(db, { chunk_id: 2, file_id: 20, scope: "bot", owner_id: 2, v: [1, 0.05] });
-        insert(db, { chunk_id: 3, file_id: 30, scope: "workspace", owner_id: 3, v: [1, 0.1] });
-
-        const nearest = db
-            .query(`SELECT chunk_id, scope, owner_id, distance FROM bot_file_vectors WHERE embedding MATCH vec_f32(?1) AND k = 10 ORDER BY distance`)
-            .all(vector([1, 0])) as { chunk_id: number; scope: string; owner_id: number }[];
-
-        expect(nearest.length).toBe(3);
-
-        const mine = nearest.filter((row) => row.scope == "bot" && row.owner_id == 1);
-
-        expect(mine.map((row) => row.chunk_id)).toEqual([1]);
-    });
-
-    test("a side-car filter in the match query is refused", () => {
-        const db = createDb();
-
-        insert(db, { chunk_id: 1, file_id: 10, scope: "bot", owner_id: 1, v: [1, 0] });
-
-        expect(() =>
-            db
-                .query(
-                    `SELECT chunk_id FROM bot_file_vectors WHERE embedding MATCH vec_f32(?1) AND k = 2 AND scope = 'bot'`,
-                )
-                .all(vector([1, 0])),
-        ).toThrow(/illegal WHERE constraint/);
-    });
-
-    test("an insert without vec0's metadata columns is refused", () => {
-        const db = createDb();
-
-        expect(() =>
-            db.run(
-                `INSERT INTO bot_file_vectors (embedding, chunk_id, file_id, scope, owner_id) VALUES (vec_f32(?1), 1, 10, 'bot', 1)`,
-                [vector([1, 0])],
-            ),
-        ).toThrow();
-    });
-
-    test("deletes a file's vectors by its side-car column", () => {
-        const db = createDb();
-
-        insert(db, { chunk_id: 1, file_id: 10, scope: "bot", owner_id: 1, v: [1, 0] });
-
-        db.run(`DELETE FROM bot_file_vectors WHERE file_id = ?`, [10]);
-
-        const count = db.query(`SELECT COUNT(*) AS n FROM bot_file_vectors`).get() as { n: number };
-
-        expect(count.n).toBe(0);
-    });
-});
-
-describe("fitVectorDimensions", () => {
-    test("pads a short vector and normalizes it", () => {
-        const out = fitVectorDimensions({ vector: [3, 4], dimensions: 4 });
-        const norm = Math.sqrt([...out].reduce((sum, value) => sum + value * value, 0));
-
-        expect(out.length).toBe(4);
-        expect(Math.round(norm * 1000) / 1000).toBe(1);
-        expect(Math.round(out[0]! * 1000) / 1000).toBe(0.6);
-    });
-
-    test("cuts a long vector", () => {
-        expect(fitVectorDimensions({ vector: [1, 1, 1, 1, 1, 1], dimensions: 2 }).length).toBe(2);
-    });
-
-    test("leaves a zero vector as zeros", () => {
-        expect([...fitVectorDimensions({ vector: [0, 0], dimensions: 2 })]).toEqual([0, 0]);
-    });
-});
-
-afterAll(() => {});

src/functions/backend/db/bots-queue/clear-finished-job-steps-sql.test.ts

@@ -0,0 +1,99 @@
+import { Database } from "bun:sqlite";
+import { describe, expect, test } from "bun:test";
+import clearFinishedJobStepsSql from "./clear-finished-job-steps-sql";
+
+/**
+ * Clearing a finished job's steps must keep their count on the job (the
+ * message details show it) and never touch a job that can still run: its
+ * working row reads them.
+ */
+function setup() {
+    const db = new Database(":memory:");
+
+    db.run(
+        `CREATE TABLE bots_queue (id INTEGER PRIMARY KEY, status TEXT, finished_at INTEGER, updated_at INTEGER, steps INTEGER)`,
+    );
+    db.run(`CREATE TABLE bots_queue_updates (id INTEGER PRIMARY KEY, bots_queue_id INTEGER)`);
+    db.run(`INSERT INTO bots_queue VALUES
+        (1, 'completed', 100, 100, NULL),
+        (2, 'failed', 100, 100, NULL),
+        (3, 'processing', NULL, 100, NULL),
+        (4, 'pending', NULL, 100, NULL),
+        (5, 'completed', 900, 900, NULL),
+        (6, 'failed', NULL, 100, NULL),
+        (7, 'completed', 100, 100, NULL)`);
+    db.run(`INSERT INTO bots_queue_updates (bots_queue_id) VALUES (1), (1), (1), (2), (3), (3), (4), (5), (6)`);
+
+    return db;
+}
+
+function run({ db, before }: { db: Database; before: number }) {
+    const { count, clear } = clearFinishedJobStepsSql({ before });
+
+    db.query(count.sql).run(...count.values);
+    db.query(clear.sql).run(...clear.values);
+}
+
+function stepsLeft({ db }: { db: Database }) {
+    return db
+        .query(`SELECT bots_queue_id AS id FROM bots_queue_updates ORDER BY id`)
+        .all()
+        .map((row) => (row as { id: number }).id);
+}
+
+function counts({ db }: { db: Database }) {
+    return Object.fromEntries(
+        db
+            .query(`SELECT id, steps FROM bots_queue ORDER BY id`)
+            .all()
+            .map((row) => [(row as { id: number }).id, (row as { steps: number | null }).steps]),
+    );
+}
+
+describe("clearFinishedJobStepsSql", () => {
+    test("clears the steps of jobs that ended before the cutoff, completed or failed", () => {
+        const db = setup();
+        run({ db, before: 500 });
+        expect(stepsLeft({ db })).not.toContain(1);
+        expect(stepsLeft({ db })).not.toContain(2);
+    });
+
+    test("keeps how many steps each job recorded", () => {
+        const db = setup();
+        run({ db, before: 500 });
+        expect(counts({ db })[1]).toBe(3);
+        expect(counts({ db })[2]).toBe(1);
+    });
+
+    test("never touches a job that can still run", () => {
+        const db = setup();
+        run({ db, before: 500 });
+        expect(stepsLeft({ db })).toEqual([3, 3, 4, 5]);
+        expect(counts({ db })[3]).toBeNull();
+        expect(counts({ db })[4]).toBeNull();
+    });
+
+    test("leaves a job that ended after the cutoff for the next sweep", () => {
+        const db = setup();
+        run({ db, before: 500 });
+        expect(counts({ db })[5]).toBeNull();
+        run({ db, before: 1000 });
+        expect(stepsLeft({ db })).not.toContain(5);
+        expect(counts({ db })[5]).toBe(1);
+    });
+
+    test("a job lost without a finish time is still cleared by when it last changed", () => {
+        const db = setup();
+        run({ db, before: 500 });
+        expect(stepsLeft({ db })).not.toContain(6);
+        expect(counts({ db })[6]).toBe(1);
+    });
+
+    test("a job with no steps keeps a null count, and a second sweep changes nothing", () => {
+        const db = setup();
+        run({ db, before: 500 });
+        run({ db, before: 500 });
+        expect(counts({ db })[7]).toBeNull();
+        expect(counts({ db })[1]).toBe(3);
+    });
+});

src/functions/backend/db/bots-queue/clear-finished-job-steps-sql.ts

@@ -0,0 +1,29 @@
+type Params = {
+    /**
+     * Jobs that ended before this (unix ms) have their steps cleared.
+     */
+    before: number;
+};
+
+/**
+ * The two statements that clear the steps (`bots_queue_updates`) of jobs
+ * that ended before `before`: first each job keeps how many it recorded
+ * (`bots_queue.steps`, for the message details), then the steps go. A job
+ * that's still open, or one retrying after a failed attempt, keeps its
+ * steps. Run in this order: the count reads the rows the delete removes.
+ */
+export default function clearFinishedJobStepsSql({ before }: Params) {
+    const finished = `SELECT id FROM bots_queue WHERE status IN ('completed', 'failed') AND COALESCE(finished_at, updated_at) < ?`;
+
+    return {
+        count: {
+            sql: `UPDATE bots_queue SET steps = COALESCE(steps, 0) + (SELECT COUNT(*) FROM bots_queue_updates u WHERE u.bots_queue_id = bots_queue.id)
+                WHERE id IN (${finished}) AND EXISTS (SELECT 1 FROM bots_queue_updates u WHERE u.bots_queue_id = bots_queue.id)`,
+            values: [before],
+        },
+        clear: {
+            sql: `DELETE FROM bots_queue_updates WHERE bots_queue_id IN (${finished})`,
+            values: [before],
+        },
+    };
+}

src/functions/backend/db/bots-queue/clear-finished-job-steps.ts

@@ -0,0 +1,21 @@
+import BunSQLite from "@moduletrace/bun-sqlite";
+import { AppData } from "@/src/data/app-data";
+import clearFinishedJobStepsSql from "./clear-finished-job-steps-sql";
+
+/**
+ * Clears the steps of jobs that ended more than `AppData.BotJobStepsKeepMs`
+ * ago, however they ended (completed, failed, cancelled, stopped, lost).
+ * Steps only feed the live working row; once the job's message is in they
+ * have no reader, and the job keeps their count. Not done the moment a job
+ * ends: steps reach the page only through the live relay, which would
+ * miss the last ones.
+ */
+export default async function clearFinishedJobSteps() {
+    const { count, clear } = clearFinishedJobStepsSql({
+        before: Date.now() - AppData.BotJobStepsKeepMs,
+    });
+
+    await BunSQLite.sql(count);
+
+    return await BunSQLite.sql(clear);
+}

src/functions/backend/db/bots/delete-bot.ts

@@ -2,6 +2,7 @@ import BunSQLite from "@moduletrace/bun-sqlite";
 import deleteMedia from "@/src/functions/backend/db/media/delete-media";
 import purgeStorage, { grabStorageMediaIds } from "@/src/functions/backend/db/bot-files/purge-storage";
 import { grabTaskMediaIds } from "@/src/functions/backend/db/tasks/purge-task-media";
+import passOnBotTasks from "@/src/functions/backend/db/tasks/pass-on-bot-tasks";
 import AppError from "@/src/utils/app-error";
 import { clearEnvCache } from "../envs/grab-all-envs";
 import grabUserBot from "./grab-user-bot";
@@ -40,19 +41,18 @@ export default async function deleteBot({ user_id, bot_id }: Params) {
     );
     const task_media_ids = await grabTaskMediaIds({ bot_id: bot.id! }).catch(() => []);
 
+    /**
+     * Its tasks in workspaces belong to the workspace: they go to each one's
+     * admin bot (else the coordinator) instead of cascading away with its
+     * own list.
+     */
+    await passOnBotTasks({ bot_id: bot.id, bot_name: bot.name });
+
     /**
      * Explicitly, since foreign keys are off outside the server process.
      */
     await BunSQLite.sql({ sql: `DELETE FROM bot_types WHERE bot_id = ?`, values: [bot.id] });
 
-    /**
-     * Its tasks in workspaces belong to the workspace: they stay, shared
-     * again, instead of cascading away with its own list.
-     */
-    await BunSQLite.sql({
-        sql: `UPDATE tasks SET bot_id = NULL, updated_at = ? WHERE bot_id = ? AND workspace_id IS NOT NULL`,
-        values: [Date.now(), bot.id],
-    });
     await BunSQLite.sql({ sql: `DELETE FROM envs WHERE bot_id = ?`, values: [bot.id] });
     await BunSQLite.sql({ sql: `DELETE FROM secret_requests WHERE bot_id = ?`, values: [bot.id] });
     await BunSQLite.sql({ sql: `DELETE FROM bot_graphics WHERE bot_id = ?`, values: [bot.id] });

src/functions/backend/db/bots/grab-messages-media.test.ts

@@ -30,18 +30,16 @@ db.run(
 db.run(`INSERT INTO bot_message_media (id, message_id, media_id) VALUES (1, 10, 1)`);
 
 /**
- * The query `grabMessagesMedia` builds, with the text column included or
- * left out. It is the largest column in the table, so only the reply path
- * that hands a document to a model asks for it.
+ * The query `grabMessagesMedia` builds. The text column is never read: a
+ * reply reads an attachment's text from the file itself.
  */
-function read({ with_text_content }: { with_text_content: boolean }) {
+function read() {
     const columns = [
         "m.id",
         "m.title",
         "m.media_type",
         "m.media_write_path",
         "m.mime_type",
-        ...(with_text_content ? ["m.text_content"] : []),
     ].join(", ");
 
     return db
@@ -55,15 +53,11 @@ function read({ with_text_content }: { with_text_content: boolean }) {
 
 describe("grabMessagesMedia", () => {
     test("a plain read never drags the extracted text along", () => {
-        expect(read({ with_text_content: false })).not.toHaveProperty("text_content");
-    });
-
-    test("the reply path can ask for it", () => {
-        expect(read({ with_text_content: true }).text_content).toBe("Total: 12");
+        expect(read()).not.toHaveProperty("text_content");
     });
 
     test("the row's name and path are always there", () => {
-        expect(read({ with_text_content: false })).toMatchObject({
+        expect(read()).toMatchObject({
             title: "invoice.pdf",
             media_write_path: "media/1.pdf",
             message_id: 10,

src/functions/backend/db/bots/grab-messages-media.ts

@@ -4,12 +4,6 @@ import mediaColumns from "@/src/functions/backend/db/media/media-columns";
 
 type Params = {
     message_ids: number[];
-    /**
-     * Also read `text_content`, the extracted text a reply hands a
-     * document to the model as. Off by default: it is the largest column
-     * in the table and nothing else here uses it.
-     */
-    with_text_content?: boolean;
 };
 
 export type MessageMediaRow = BUN_SQLITE_COBBLE_MEDIA & { message_id: number };
@@ -20,14 +14,13 @@ export type MessageMediaRow = BUN_SQLITE_COBBLE_MEDIA & { message_id: number };
  */
 export default async function grabMessagesMedia({
     message_ids,
-    with_text_content,
 }: Params) {
     const by_message = new Map<number, MessageMediaRow[]>();
     const ids = [...new Set(message_ids.filter(Boolean))];
 
     if (!ids.length) return by_message;
 
-    const columns = mediaColumns({ alias: "m", with_text_content });
+    const columns = mediaColumns({ alias: "m" });
 
     const rows =
         (

src/functions/backend/db/media/media-columns.ts

@@ -5,9 +5,8 @@
  * `AppData.BotAttachmentTextStoreMaxChars` of extracted document text, and
  * every caller that only needs a file's name, type or path (`SELECT m.*`
  * did) pulled it into memory, into page props and into API responses.
- * Read it explicitly where it is really used:
- * `src/bots-queue/load-attachment-parts.ts` and
- * `src/functions/backend/db/bot-files/ingest-bot-file.ts`.
+ * Only knowledgebase documents keep it now; read it explicitly where it
+ * is really used.
  */
 export default function mediaColumns({
     alias = "m",
@@ -31,8 +30,7 @@ export default function mediaColumns({
     ];
 
     /**
-     * Only the two paths that really use it ask for it: the reply that
-     * hands a document to a model, and ingestion.
+     * Only a caller that really uses it asks for it.
      */
     if (with_text_content) columns.push("text_content");
 

src/functions/backend/db/media/plan-text-content-prune.test.ts

@@ -1,65 +0,0 @@
-import { describe, expect, test } from "bun:test";
-import { AppData } from "@/src/data/app-data";
-import planTextContentPrune from "./plan-text-content-prune";
-
-describe("planTextContentPrune", () => {
-    test("clears the raw bytes stored for a type the extractor can't read", () => {
-        const changes = planTextContentPrune({
-            rows: [
-                { id: 1, title: "logo.svg", mime_type: "image/svg+xml", text_content: "<svg><path/></svg>" },
-                { id: 2, title: "bundle.zip", mime_type: "application/zip", text_content: "PK\u0003\u0004" },
-                { id: 3, title: "sheet.xlsx", mime_type: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", text_content: "PK\u0003\u0004" },
-            ],
-        });
-
-        expect(changes.map((change) => change.media_id)).toEqual([1, 2, 3]);
-        expect(changes.every((change) => change.text_content === null)).toBe(true);
-        expect(changes.every((change) => change.after_chars === 0)).toBe(true);
-    });
-
-    test("leaves a document's text alone", () => {
-        const changes = planTextContentPrune({
-            rows: [
-                { id: 4, title: "invoice.pdf", mime_type: "application/pdf", text_content: "Total: 12" },
-                { id: 5, title: "notes.md", mime_type: "text/markdown", text_content: "# Hi" },
-                { id: 6, title: "rows.csv", mime_type: "text/csv", text_content: "a,b\n1,2" },
-            ],
-        });
-
-        expect(changes).toEqual([]);
-    });
-
-    test("cuts a document over the store ceiling", () => {
-        const text = "a".repeat(AppData.BotAttachmentTextStoreMaxChars + 100);
-        const changes = planTextContentPrune({
-            rows: [{ id: 7, title: "long.txt", mime_type: "text/plain", text_content: text }],
-        });
-
-        expect(changes).toHaveLength(1);
-        expect(changes[0]!.after_chars).toBeLessThanOrEqual(AppData.BotAttachmentTextStoreMaxChars);
-        expect(changes[0]!.text_content).toContain("Truncated");
-        expect(changes[0]!.before_chars).toBe(text.length);
-    });
-
-    test("nothing to say about a row with no text or no id", () => {
-        expect(
-            planTextContentPrune({
-                rows: [
-                    { id: 8, title: "empty.pdf", text_content: "" },
-                    { id: 9, title: "blank.pdf", text_content: "   \n " },
-                    { id: 10, title: "none.pdf" },
-                    { title: "no id.pdf", text_content: "text" },
-                ],
-            }),
-        ).toEqual([]);
-    });
-
-    test("counting what each change reclaims is left to the caller", () => {
-        const changes = planTextContentPrune({
-            rows: [{ id: 11, title: "icon.svg", mime_type: "image/svg+xml", text_content: "12345" }],
-        });
-
-        expect(changes).toHaveLength(1);
-        expect(changes[0]!.before_chars - changes[0]!.after_chars).toBe(5);
-    });
-});

src/functions/backend/db/media/plan-text-content-prune.ts

@@ -1,55 +0,0 @@
-import type { BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db";
-import decideTextContent from "@/src/utils/decide-text-content";
-
-export type TextContentChange = {
-    media_id: number;
-    name: string;
-    before_chars: number;
-    after_chars: number;
-    /**
-     * What the row's `text_content` should become: `null` clears it.
-     */
-    text_content: string | null;
-};
-
-type Params = {
-    /**
-     * The `media` rows that have any `text_content`.
-     */
-    rows: Partial<BUN_SQLITE_COBBLE_MEDIA>[];
-};
-
-/**
- * What `pruneMediaTextContent` would change, worked out without touching
- * the database: one entry per row whose text belongs nowhere or is over
- * the store ceiling, and nothing at all for a row that is already right.
- */
-export default function planTextContentPrune({ rows }: Params): TextContentChange[] {
-    const changes: TextContentChange[] = [];
-
-    for (const row of rows) {
-        if (!row.id) continue;
-
-        const before = row.text_content || "";
-
-        if (!before.trim()) continue;
-
-        const text_content = decideTextContent({
-            name: row.title,
-            mime_type: row.mime_type,
-            text: before,
-        });
-
-        if (text_content === before) continue;
-
-        changes.push({
-            media_id: Number(row.id),
-            name: row.title || "no name",
-            before_chars: before.length,
-            after_chars: text_content?.length || 0,
-            text_content,
-        });
-    }
-
-    return changes;
-}

src/functions/backend/db/media/prune-media-text-content.test.ts

@@ -1,120 +0,0 @@
-import { describe, expect, test } from "bun:test";
-import { Database } from "bun:sqlite";
-import { AppData } from "@/src/data/app-data";
-import pruneMediaTextContent, { type SqlRunner } from "./prune-media-text-content";
-
-type Row = { id: number; title: string; mime_type: string; media_type: string; text_content: string | null; updated_at?: number };
-
-function makeDb({ rows }: { rows: Row[] }) {
-    const db = new Database(":memory:");
-
-    db.run(
-        `CREATE TABLE media (
-            id INTEGER PRIMARY KEY,
-            title TEXT,
-            mime_type TEXT,
-            media_type TEXT,
-            text_content TEXT,
-            updated_at INTEGER
-        )`,
-    );
-
-    const insert = db.prepare(
-        `INSERT INTO media (id, title, mime_type, media_type, text_content) VALUES (?, ?, ?, ?, ?)`,
-    );
-
-    for (const row of rows) insert.run(row.id, row.title, row.mime_type, row.media_type, row.text_content);
-
-    const run: SqlRunner = async ({ sql, values }) => {
-        const statement = db.prepare(sql);
-
-        if (sql.trim().toUpperCase().startsWith("SELECT")) {
-            return { payload: statement.all(...((values || []) as never[])) as unknown[] };
-        }
-
-        statement.run(...((values || []) as never[]));
-
-        return { payload: [] };
-    };
-
-    const read = (id: number) =>
-        (db.query(`SELECT text_content FROM media WHERE id = ?`).get(id) as { text_content: string | null }).text_content;
-
-    return { db, run, read };
-}
-
-describe("pruneMediaTextContent", () => {
-    test("clears raw bytes from a type the extractor can't read, and keeps a document's text", async () => {
-        const { run, read } = makeDb({
-            rows: [
-                { id: 1, title: "logo.svg", mime_type: "image/svg+xml", media_type: "document", text_content: "<svg><path d='M0 0'/></svg>" },
-                { id: 2, title: "bundle.zip", mime_type: "application/zip", media_type: "document", text_content: "PK\u0003\u0004binary" },
-                { id: 3, title: "invoice.pdf", mime_type: "application/pdf", media_type: "document", text_content: "Total: 12" },
-            ],
-        });
-
-        const result = await pruneMediaTextContent({ run });
-
-        expect(result.checked).toBe(3);
-        expect(result.cleared).toBe(2);
-        expect(result.truncated).toBe(0);
-        expect(read(1)).toBeNull();
-        expect(read(2)).toBeNull();
-        expect(read(3)).toBe("Total: 12");
-    });
-
-    test("cuts an oversized document to the store ceiling", async () => {
-        const text = "a".repeat(AppData.BotAttachmentTextStoreMaxChars + 1000);
-        const { run, read } = makeDb({
-            rows: [{ id: 4, title: "huge.txt", mime_type: "text/plain", media_type: "document", text_content: text }],
-        });
-
-        const result = await pruneMediaTextContent({ run });
-        const after = read(4)!;
-
-        expect(result.truncated).toBe(1);
-        expect(result.cleared).toBe(0);
-        expect(after.length).toBeLessThanOrEqual(AppData.BotAttachmentTextStoreMaxChars);
-        expect(after).toContain("Truncated");
-        expect(result.reclaimed_bytes).toBeGreaterThan(0);
-    });
-
-    test("is safe to rerun: a second sweep changes nothing", async () => {
-        const { run, read } = makeDb({
-            rows: [{ id: 5, title: "icon.svg", mime_type: "image/svg+xml", media_type: "document", text_content: "raw" }],
-        });
-
-        await pruneMediaTextContent({ run });
-        const second = await pruneMediaTextContent({ run });
-
-        expect(second.checked).toBe(0);
-        expect(second.changes).toEqual([]);
-        expect(second.reclaimed_bytes).toBe(0);
-        expect(read(5)).toBeNull();
-    });
-
-    test("stamps updated_at on the rows it rewrites", async () => {
-        const now = 1_700_000_000_000;
-        const { run, db } = makeDb({
-            rows: [{ id: 6, title: "x.zip", mime_type: "application/zip", media_type: "document", text_content: "raw" }],
-        });
-
-        await pruneMediaTextContent({ run, now });
-
-        expect(
-            (db.query(`SELECT updated_at FROM media WHERE id = 6`).get() as { updated_at: number }).updated_at,
-        ).toBe(now);
-    });
-
-    test("reports every change with its row, name and sizes", async () => {
-        const { run } = makeDb({
-            rows: [{ id: 7, title: "sheet.xlsx", mime_type: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", media_type: "document", text_content: "PK\u0003\u0004" }],
-        });
-
-        const result = await pruneMediaTextContent({ run });
-
-        expect(result.changes).toEqual([
-            { media_id: 7, name: "sheet.xlsx", before_chars: 4, after_chars: 0, text_content: null },
-        ]);
-    });
-});

src/functions/backend/db/media/prune-media-text-content.ts

@@ -1,75 +0,0 @@
-import BunSQLite from "@moduletrace/bun-sqlite";
-import type { BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db";
-import planTextContentPrune, { type TextContentChange } from "./plan-text-content-prune";
-
-/**
- * One statement against the database. `BunSQLite.sql` is the real one; a
- * test passes its own, so the whole sweep can be run without the app's
- * database.
- */
-export type SqlRunner = (params: {
-    sql: string;
-    values?: (string | number | null)[];
-}) => Promise<{ payload?: unknown[] | null }>;
-
-export type PruneMediaTextResult = {
-    checked: number;
-    cleared: number;
-    truncated: number;
-    reclaimed_bytes: number;
-    changes: TextContentChange[];
-};
-
-type Params = {
-    run?: SqlRunner;
-    /**
-     * When the sweep runs, for the rows it writes.
-     */
-    now?: number;
-};
-
-/**
- * Brings every `media.text_content` in line with what may be stored there
- * (`planTextContentPrune`): no text on a type the extractor can't read,
- * and no text longer than the store ceiling.
- *
- * Run once per install after the allowlist landed; safe to rerun, since a
- * row that is already right is not written.
- */
-export default async function pruneMediaTextContent({
-    run,
-    now = Date.now(),
-}: Params = {}): Promise<PruneMediaTextResult> {
-    const sql: SqlRunner =
-        run || (({ sql, values }) => BunSQLite.sql<BUN_SQLITE_COBBLE_MEDIA>({ sql, values }));
-
-    const res = await sql({
-        sql: `SELECT id, title, mime_type, media_type, text_content
-            FROM media
-            WHERE text_content IS NOT NULL AND TRIM(text_content) != ''`,
-    });
-
-    const rows = (res.payload || []) as BUN_SQLITE_COBBLE_MEDIA[];
-    const changes = planTextContentPrune({ rows });
-    const result: PruneMediaTextResult = {
-        checked: rows.length,
-        cleared: 0,
-        truncated: 0,
-        reclaimed_bytes: 0,
-        changes,
-    };
-
-    for (const change of changes) {
-        await sql({
-            sql: `UPDATE media SET text_content = ?, updated_at = ? WHERE id = ?`,
-            values: [change.text_content, now, change.media_id],
-        });
-
-        result.reclaimed_bytes += change.before_chars - change.after_chars;
-
-        if (change.text_content === null) result.cleared += 1;
-        else result.truncated += 1;
-    }
-
-    return result;
-}

src/functions/backend/db/stored-text/drop-stored-file-text-sql.test.ts

@@ -0,0 +1,65 @@
+import { Database } from "bun:sqlite";
+import { describe, expect, test } from "bun:test";
+import dropStoredFileTextSql from "./drop-stored-file-text-sql";
+
+function scratch() {
+    const db = new Database(":memory:");
+
+    db.run(`CREATE TABLE media (id INTEGER PRIMARY KEY, text_content TEXT)`);
+    db.run(`CREATE TABLE knowledgebase_documents (id INTEGER PRIMARY KEY, media_id INTEGER)`);
+    db.run(`CREATE TABLE file_ingest_queue (id INTEGER PRIMARY KEY, document_id INTEGER)`);
+    db.run(`CREATE TABLE bot_file_chunks (id INTEGER PRIMARY KEY, text TEXT)`);
+
+    db.run(`INSERT INTO media (id, text_content) VALUES (1, 'an attachment'), (2, 'a knowledgebase upload'), (3, NULL)`);
+    db.run(`INSERT INTO knowledgebase_documents (id, media_id) VALUES (1, 2), (2, NULL)`);
+    db.run(`INSERT INTO file_ingest_queue (id, document_id) VALUES (1, NULL), (2, 1)`);
+    db.run(`INSERT INTO bot_file_chunks (id, text) VALUES (1, 'old passage')`);
+
+    return db;
+}
+
+function run(db: Database) {
+    for (const sql of dropStoredFileTextSql()) db.run(sql);
+}
+
+describe("dropStoredFileTextSql", () => {
+    test("drops the bot file index", () => {
+        const db = scratch();
+
+        run(db);
+
+        const tables = db.query(`SELECT name FROM sqlite_master WHERE name LIKE 'bot_file_%'`).all();
+
+        expect(tables).toEqual([]);
+    });
+
+    test("clears upload text but keeps a knowledgebase document's", () => {
+        const db = scratch();
+
+        run(db);
+
+        const rows = db.query(`SELECT id, text_content FROM media ORDER BY id`).all();
+
+        expect(rows).toEqual([
+            { id: 1, text_content: null },
+            { id: 2, text_content: "a knowledgebase upload" },
+            { id: 3, text_content: null },
+        ]);
+    });
+
+    test("keeps only document jobs on the ingest queue", () => {
+        const db = scratch();
+
+        run(db);
+
+        expect(db.query(`SELECT id FROM file_ingest_queue`).all()).toEqual([{ id: 2 }]);
+    });
+
+    test("runs again on a clean database without failing", () => {
+        const db = scratch();
+
+        run(db);
+
+        expect(() => run(db)).not.toThrow();
+    });
+});

src/functions/backend/db/stored-text/drop-stored-file-text-sql.ts

@@ -0,0 +1,17 @@
+/**
+ * The statements that remove text stored outside knowledgebases: the old
+ * bot file index (chunks and their vectors; the vec0 drop takes its shadow
+ * tables with it), file jobs left on the ingest queue, and the extracted
+ * text uploads used to keep. A knowledgebase document's media keeps its
+ * text. Every statement is a no-op once it has run.
+ */
+export default function dropStoredFileTextSql(): string[] {
+    return [
+        `DROP TABLE IF EXISTS bot_file_vectors`,
+        `DROP TABLE IF EXISTS bot_file_chunks`,
+        `DELETE FROM file_ingest_queue WHERE document_id IS NULL`,
+        `UPDATE media SET text_content = NULL
+            WHERE text_content IS NOT NULL
+              AND id NOT IN (SELECT media_id FROM knowledgebase_documents WHERE media_id IS NOT NULL)`,
+    ];
+}

src/functions/backend/db/stored-text/drop-stored-file-text.ts

@@ -0,0 +1,46 @@
+import BunSQLite from "@moduletrace/bun-sqlite";
+import logEvent, { logError } from "@/src/functions/backend/logger";
+import dropStoredFileTextSql from "./drop-stored-file-text-sql";
+import shouldVacuum from "./should-vacuum";
+
+/**
+ * Knowledgebases are the only place text is kept for search. Removes what
+ * bot files and uploads stored before that (`dropStoredFileTextSql`), then
+ * vacuums once if the database has a lot of free pages, so the space comes
+ * back on disk. Safe on every start: once clean, it only reads two pragmas.
+ */
+export default async function dropStoredFileText() {
+    for (const sql of dropStoredFileTextSql()) {
+        const res = await BunSQLite.sql({ sql });
+
+        if (!res.success) throw new Error(`${sql.split("\n")[0]}: ${res.error || res.msg}`);
+    }
+
+    const [page_count, freelist_count, page_size] = await Promise.all(
+        ["page_count", "freelist_count", "page_size"].map((pragma) =>
+            BunSQLite.sql<Record<string, number>>({ sql: `PRAGMA ${pragma}` }).then((res) =>
+                Number(res.payload?.[0]?.[pragma]) || 0,
+            ),
+        ),
+    );
+
+    if (!shouldVacuum({ page_count: page_count!, freelist_count: freelist_count!, page_size: page_size! })) return;
+
+    const res = await BunSQLite.sql({ sql: `VACUUM` });
+
+    if (!res.success) {
+        await logError({
+            title: "Could not vacuum the database",
+            error: new Error(`${res.error || res.msg}. It needs about as much free disk as the database takes.`),
+            metadata: { free_bytes: freelist_count! * page_size! },
+        }).catch(() => {});
+        return;
+    }
+
+    await logEvent({
+        title: "Vacuumed the database",
+        category: "system",
+        plain_text: `Gave back ${Math.round((freelist_count! * page_size!) / (1024 * 1024))} MB of free pages to the disk.`,
+        metadata: { freed_bytes: freelist_count! * page_size! },
+    }).catch(() => {});
+}

src/functions/backend/db/stored-text/should-vacuum.test.ts

@@ -0,0 +1,24 @@
+import { describe, expect, test } from "bun:test";
+import { AppData } from "@/src/data/app-data";
+import shouldVacuum from "./should-vacuum";
+
+const page_size = 4096;
+const min_pages = Math.ceil(AppData.DbVacuumMinFreeBytes / page_size);
+
+describe("shouldVacuum", () => {
+    test("vacuums when a large share of a big file is free", () => {
+        expect(shouldVacuum({ page_count: min_pages * 2, freelist_count: min_pages, page_size })).toBe(true);
+    });
+
+    test("leaves a database with little free space alone", () => {
+        expect(shouldVacuum({ page_count: min_pages * 100, freelist_count: min_pages, page_size })).toBe(false);
+    });
+
+    test("leaves a small database alone even when mostly free", () => {
+        expect(shouldVacuum({ page_count: 20, freelist_count: 15, page_size })).toBe(false);
+    });
+
+    test("never vacuums on missing numbers", () => {
+        expect(shouldVacuum({ page_count: 0, freelist_count: 0, page_size })).toBe(false);
+    });
+});

src/functions/backend/db/stored-text/should-vacuum.ts

@@ -0,0 +1,22 @@
+import { AppData } from "@/src/data/app-data";
+
+type Params = {
+    page_count: number;
+    freelist_count: number;
+    page_size: number;
+};
+
+/**
+ * Whether enough of the database file sits free to be worth a `VACUUM`
+ * (`AppData.DbVacuumMinFreeRatio` of its pages and at least
+ * `AppData.DbVacuumMinFreeBytes`). Deleted rows only free pages inside the
+ * file; it shrinks on disk only when vacuumed.
+ */
+export default function shouldVacuum({ page_count, freelist_count, page_size }: Params) {
+    if (!(page_count > 0) || !(freelist_count > 0)) return false;
+
+    return (
+        freelist_count / page_count >= AppData.DbVacuumMinFreeRatio &&
+        freelist_count * page_size >= AppData.DbVacuumMinFreeBytes
+    );
+}

src/functions/backend/db/tasks/assign-ownerless-tasks.ts

@@ -0,0 +1,10 @@
+import passOnBotTasks from "./pass-on-bot-tasks";
+
+/**
+ * At boot: every task belongs to a bot. Workspace tasks left without one
+ * by earlier versions ("shared") go to the workspace's admin bot, else the
+ * coordinator.
+ */
+export default async function assignOwnerlessTasks() {
+    return await passOnBotTasks({});
+}

src/functions/backend/db/tasks/create-task.ts

@@ -27,7 +27,7 @@ type Params = {
 
 /**
  * Adds a task to a bot's own list or a workspace's, there for one of its
- * bots (`bot_id` too) or shared. A user adds one to a bot they chat with or
+ * bots (`bot_id` too): every task belongs to a bot. A user adds one to a bot they chat with or
  * a workspace they're in; a bot's run adds one to its own list or to a
  * workspace it works in (its tool checks that). A workspace's task can
  * only go to a bot working there. Caps a list at `TaskMaxPerScope` open and

src/functions/backend/db/tasks/grab-task-heir.ts

@@ -0,0 +1,29 @@
+import BunSQLite from "@moduletrace/bun-sqlite";
+import type { BUN_SQLITE_COBBLE_BOTS } from "@/db/types/db";
+import grabWorkspaceAdmins from "../bot-types/grab-workspace-admins";
+
+type Params = {
+    workspace_id: number | string;
+    /**
+     * The bot leaving, never its own heir.
+     */
+    except_bot_id?: number | string | null;
+};
+
+/**
+ * Who takes over a workspace task whose bot is gone: the workspace's admin
+ * bot, else the coordinator. Every task belongs to a bot, so none is left
+ * without one.
+ */
+export default async function grabTaskHeir({ workspace_id, except_bot_id }: Params) {
+    const admins = await grabWorkspaceAdmins({ workspace_id });
+    const admin = admins.find((bot) => Number(bot.id) != Number(except_bot_id || 0));
+
+    if (admin?.id) return admin;
+
+    const res = await BunSQLite.sql<BUN_SQLITE_COBBLE_BOTS>({
+        sql: `SELECT * FROM bots WHERE kind = 'coordinator' ORDER BY id ASC LIMIT 1`,
+    });
+
+    return res.payload?.[0] || null;
+}

src/functions/backend/db/tasks/grab-task.ts

@@ -1,6 +1,5 @@
 import BunSQLite from "@moduletrace/bun-sqlite";
-import type { BUN_SQLITE_COBBLE_TASK_NOTES } from "@/db/types/db";
-import type { TaskDetail } from "@/src/types";
+import type { TaskDetail, TaskNote } from "@/src/types";
 import AppError from "@/src/utils/app-error";
 import botFaceSql from "@/src/utils/bot-face-sql";
 import taskScopesSql from "@/src/utils/task-access/task-scopes-sql";
@@ -16,7 +15,8 @@ type Params = {
 };
 
 /**
- * One task with its notes (oldest first) and files, only if the user
+ * One task with its notes (oldest first, each with the bot that wrote it),
+ * its files and who added it, only if the user
  * reaches the bot or workspace it belongs to (`taskScopesSql`, the same
  * condition the lists use).
  */
@@ -26,10 +26,14 @@ export default async function grabTask({ user_id, task_id }: Params): Promise<Ta
 
     const row_res = await BunSQLite.sql<TaskDetail>({
         sql: `SELECT t.*, b.name AS bot_name, b.kind AS bot_kind, b.hue AS bot_hue, b.color AS bot_color, b.stone AS bot_stone,
-                b.image_media_id AS bot_image_media_id, ${face.sql}, w.name AS workspace_name
+                b.image_media_id AS bot_image_media_id, ${face.sql}, w.name AS workspace_name,
+                cb.name AS created_by_bot_name,
+                NULLIF(TRIM(COALESCE(cu.first_name, '') || ' ' || COALESCE(cu.last_name, '')), '') AS created_by_user_name
             FROM tasks t
             LEFT JOIN bots b ON b.id = t.bot_id
             LEFT JOIN workspaces w ON w.id = t.workspace_id
+            LEFT JOIN bots cb ON cb.id = t.created_by_bot_id
+            LEFT JOIN users cu ON cu.id = t.created_by_user_id
             WHERE t.id = ? AND ${access.sql}
             LIMIT 1`,
         values: [...face.values, Number(task_id), ...access.values],
@@ -42,8 +46,10 @@ export default async function grabTask({ user_id, task_id }: Params): Promise<Ta
     }
 
     const [notes_res, files] = await Promise.all([
-        BunSQLite.sql<BUN_SQLITE_COBBLE_TASK_NOTES>({
-            sql: `SELECT * FROM task_notes WHERE task_id = ? ORDER BY id ASC`,
+        BunSQLite.sql<TaskNote>({
+            sql: `SELECT n.*, nb.name AS bot_name FROM task_notes n
+                LEFT JOIN bots nb ON nb.id = n.bot_id
+                WHERE n.task_id = ? ORDER BY n.id ASC`,
             values: [Number(task_id)],
         }),
         grabTaskMedia({ task_id }),

src/functions/backend/db/tasks/grab-tasks.ts

@@ -27,10 +27,6 @@ type Params = {
      * Only the tasks this bot has (anywhere, unless narrowed).
      */
     only_bot_id?: number | string | null;
-    /**
-     * With `only_bot_id`: also the shared ones nobody has yet.
-     */
-    with_unassigned?: boolean;
     /**
      * Only one workspace's tasks.
      */
@@ -61,7 +57,6 @@ export default async function grabTasks({
     is_coordinator,
     admin_workspace_ids,
     only_bot_id,
-    with_unassigned,
     only_workspace_id,
     own_list,
     statuses,
@@ -82,7 +77,7 @@ export default async function grabTasks({
     }
 
     if (only_bot_id) {
-        where.push(with_unassigned ? `(t.bot_id = ? OR t.bot_id IS NULL)` : `t.bot_id = ?`);
+        where.push(`t.bot_id = ?`);
         values.push(Number(only_bot_id));
     }
 

src/functions/backend/db/tasks/move-task.ts

@@ -10,7 +10,7 @@ type Params = {
     task_id: number | string;
     /**
      * Where it goes: a bot's own list (`bot_id` alone), or a workspace,
-     * with the bot that has it there (none: unassigned).
+     * with the bot that has it there (always one).
      */
     bot_id?: number | string | null;
     workspace_id?: number | string | null;

src/functions/backend/db/tasks/pass-on-bot-tasks.ts

@@ -0,0 +1,65 @@
+import BunSQLite from "@moduletrace/bun-sqlite";
+import type { BUN_SQLITE_COBBLE_TASKS } from "@/db/types/db";
+import addTaskNote from "./add-task-note";
+import grabTaskHeir from "./grab-task-heir";
+
+type Params = {
+    /**
+     * The bot leaving; none for tasks that have no bot at all.
+     */
+    bot_id?: number | string | null;
+    /**
+     * Only its tasks in this workspace; none for every workspace.
+     */
+    workspace_id?: number | string | null;
+    /**
+     * Who is leaving, for the note on each task.
+     */
+    bot_name?: string | null;
+};
+
+/**
+ * Gives a leaving bot's workspace tasks (or ones with no bot) to the
+ * workspace's admin bot, else the coordinator, with a note on each. Its own
+ * list is left alone. Returns how many moved.
+ */
+export default async function passOnBotTasks({ bot_id, workspace_id, bot_name }: Params) {
+    const where = [`workspace_id IS NOT NULL`, bot_id ? `bot_id = ?` : `bot_id IS NULL`];
+    const values: number[] = bot_id ? [Number(bot_id)] : [];
+
+    if (workspace_id) {
+        where.push(`workspace_id = ?`);
+        values.push(Number(workspace_id));
+    }
+
+    const tasks =
+        (
+            await BunSQLite.sql<BUN_SQLITE_COBBLE_TASKS>({
+                sql: `SELECT * FROM tasks WHERE ${where.join(" AND ")}`,
+                values,
+            })
+        ).payload || [];
+
+    let moved = 0;
+
+    for (const task of tasks) {
+        const heir = await grabTaskHeir({ workspace_id: task.workspace_id!, except_bot_id: bot_id });
+
+        if (!heir?.id) continue;
+
+        await BunSQLite.sql({
+            sql: `UPDATE tasks SET bot_id = ?, updated_at = ? WHERE id = ?`,
+            values: [Number(heir.id), Date.now(), Number(task.id)],
+        });
+
+        await addTaskNote({
+            task_id: task.id!,
+            text: bot_name ? `${bot_name} left; this is now ${heir.name}'s.` : `This is now ${heir.name}'s.`,
+            author: "user",
+        }).catch(() => {});
+
+        moved++;
+    }
+
+    return moved;
+}

src/functions/backend/db/tasks/task-scope-values.test.ts

@@ -0,0 +1,14 @@
+import { describe, expect, test } from "bun:test";
+import taskScopeValues from "./task-scope-values";
+
+describe("taskScopeValues", () => {
+    test("a task on a bot's own list or in a workspace has its bot", () => {
+        expect(taskScopeValues({ bot_id: 3 })).toEqual({ bot_id: 3, workspace_id: null });
+        expect(taskScopeValues({ bot_id: "3", workspace_id: "11" })).toEqual({ bot_id: 3, workspace_id: 11 });
+    });
+
+    test("a workspace task without a bot is refused", () => {
+        expect(() => taskScopeValues({ workspace_id: 11 })).toThrow();
+        expect(() => taskScopeValues({})).toThrow();
+    });
+});

src/functions/backend/db/tasks/task-scope-values.ts

@@ -6,17 +6,17 @@ type Params = {
 };
 
 /**
- * Where a task lives and who has it. A bot's own list is `bot_id` alone; a
- * workspace's task has `workspace_id`, plus `bot_id` when a bot there has
- * it (none: shared, not yet anyone's). Checked in the helpers rather than
- * the schema.
+ * Where a task lives and who has it. Every task belongs to a bot: a bot's
+ * own list is `bot_id` alone, a workspace's task has `workspace_id` plus
+ * the `bot_id` that has it there. Checked in the helpers rather than the
+ * schema.
  */
 export default function taskScopeValues({ bot_id, workspace_id }: Params) {
     const bot = bot_id ? Number(bot_id) : null;
     const workspace = workspace_id ? Number(workspace_id) : null;
 
-    if (!bot && !workspace) {
-        throw new AppError(`Say which bot or workspace the task is for.`);
+    if (!bot) {
+        throw new AppError(`Say which bot has the task.`);
     }
 
     return { bot_id: bot, workspace_id: workspace };

src/functions/backend/db/usage/grab-message-details.ts

@@ -105,7 +105,7 @@ async function grabJob({ job_id }: { job_id: number }): Promise<MessageDetailsJo
     const row = (
         await BunSQLite.sql<JobRow>({
             sql: `SELECT q.id, q.kind, q.title, q.status, q.attempts, q.started_at, q.finished_at,
-                    (SELECT COUNT(*) FROM bots_queue_updates u WHERE u.bots_queue_id = q.id) AS steps
+                    COALESCE(q.steps, 0) + (SELECT COUNT(*) FROM bots_queue_updates u WHERE u.bots_queue_id = q.id) AS steps
                 FROM bots_queue q WHERE q.id = ? LIMIT 1`,
             values: [job_id],
         })

src/functions/backend/db/workspaces/grab-workspace-activity.ts

@@ -10,8 +10,8 @@ type Params = {
 
 /**
  * The user's latest conversations in a workspace, most recent first, each
- * with the bot it's with (its face as the user sees it), its message count and latest message. Empty
- * sessions are left out unless active.
+ * with the bot it's with (its face as the user sees it), its message count, first and latest message. Sessions
+ * with no messages yet are left out, active or not.
  */
 export default async function grabWorkspaceActivity({
     user_id,
@@ -28,11 +28,12 @@ export default async function grabWorkspaceActivity({
                 b.stone AS bot_stone,
                 ${face.sql},
                 (SELECT COUNT(*) FROM bot_messages m WHERE m.session_id = s.id) AS message_count,
-                (SELECT m.text FROM bot_messages m WHERE m.session_id = s.id ORDER BY m.created_at DESC, m.id DESC LIMIT 1) AS last_message_text
+                (SELECT m.text FROM bot_messages m WHERE m.session_id = s.id ORDER BY m.created_at DESC, m.id DESC LIMIT 1) AS last_message_text,
+                (SELECT m.text FROM bot_messages m WHERE m.session_id = s.id ORDER BY m.created_at ASC, m.id ASC LIMIT 1) AS first_message_text
             FROM bot_sessions s
             JOIN bots b ON b.id = s.bot_id
             WHERE s.workspace_id = ? AND s.user_id = ? AND s.chatroom_id IS NULL
-            AND (s.status = 'active' OR EXISTS (SELECT 1 FROM bot_messages m WHERE m.session_id = s.id))
+            AND EXISTS (SELECT 1 FROM bot_messages m WHERE m.session_id = s.id)
             AND (b.kind = 'coordinator' OR EXISTS (
                 SELECT 1 FROM workspace_bots wb WHERE wb.workspace_id = s.workspace_id AND wb.bot_id = s.bot_id
             ))

src/functions/backend/db/workspaces/remove-bot-from-workspace.ts

@@ -1,6 +1,8 @@
 import BunSQLite from "@moduletrace/bun-sqlite";
 import AppError from "@/src/utils/app-error";
 import grabUserWorkspace from "./grab-user-workspace";
+import grabBotById from "../bots/grab-bot-by-id";
+import passOnBotTasks from "../tasks/pass-on-bot-tasks";
 import assertCrewManager from "@/src/functions/backend/db/access/assert-crew-manager";
 
 type Params = {
@@ -31,12 +33,11 @@ export default async function removeBotFromWorkspace({
     });
 
     /**
-     * The tasks it had there stay in the workspace, shared again.
+     * The tasks it had there stay in the workspace, passed to its admin bot
+     * (else the coordinator), read before its admin type goes below.
      */
-    await BunSQLite.sql({
-        sql: `UPDATE tasks SET bot_id = NULL, updated_at = ? WHERE workspace_id = ? AND bot_id = ?`,
-        values: [Date.now(), workspace.id, bot_id],
-    });
+    const bot = await grabBotById({ bot_id });
+    await passOnBotTasks({ bot_id, workspace_id: workspace.id, bot_name: bot?.name });
 
     /**
      * Its types there (workspace admin) go with its place.

src/functions/backend/install-icon/save-install-icon.ts

@@ -3,6 +3,7 @@ import { existsSync } from "node:fs";
 import { mkdir, readdir, rm, writeFile } from "node:fs/promises";
 import path from "node:path";
 import {
+    PwaInstallIconRenderRevision,
     PwaInstallIconSettingKey,
     type PwaInstallIcon,
     type PwaInstallIconMode,
@@ -57,7 +58,7 @@ export default async function saveInstallIcon({ mode, stone, mark, image }: Para
     const version = is_stock
         ? null
         : createHash("sha256")
-              .update(JSON.stringify({ mode, stone, mark }))
+              .update(JSON.stringify({ mode, stone, mark, render: PwaInstallIconRenderRevision }))
               .update(source || "")
               .digest("hex")
               .slice(0, 12);

src/functions/server/pwa/render-install-icons.ts

@@ -1,10 +1,14 @@
-import sharp from "sharp";
+import sharp, { type OverlayOptions } from "sharp";
 import {
     PwaAppIconMarkScale,
     PwaCornerMarkInset,
     PwaCornerMarkScale,
+    PwaFaviconCornerMarkInset,
+    PwaFaviconCornerMarkScale,
     PwaInstallIconFiles,
     PwaMaskableMarkScale,
+    PwaSquareCornerMarkInset,
+    PwaSquareCornerMarkScale,
     type PwaInstallIcon,
     type PwaInstallIconFile,
 } from "@/src/data/pwa-data";
@@ -26,7 +30,8 @@ type Params = {
 /**
  * Every file of an install's icon set (`PwaInstallIconFiles`), by name.
  * A stone icon is the stone on paper, like the stock one. An image fills
- * the icon, with the stone in the top-left corner when `mark` is on.
+ * the icon, with the stone in the top-left corner when `mark` is on; its
+ * favicon is cut to a circle, the mark kept inside it.
  */
 export default async function renderInstallIcons({ icon, stone_svg, image, background }: Params) {
     const files: Record<string, Buffer> = {};
@@ -79,19 +84,41 @@ async function renderImageFile({
         .resize(file.size, file.size, { fit: "cover" })
         .flatten({ background });
 
-    if (!mark || file.kind == "favicon" || file.kind == "plain") {
-        return base.png().toBuffer();
+    const layers: OverlayOptions[] = [];
+
+    if (mark && file.kind != "plain") {
+        const { scale, inset_scale } = markPlacement({ kind: file.kind });
+        const mark_size = Math.max(1, Math.round(file.size * scale));
+        const inset = Math.round(file.size * inset_scale);
+        const stone = await sharp(stone_svg)
+            .resize(mark_size, mark_size, { fit: "contain" })
+            .png()
+            .toBuffer();
+
+        layers.push({ input: stone, left: inset, top: inset });
     }
 
-    const mark_size = Math.max(1, Math.round(file.size * PwaCornerMarkScale));
-    const inset = Math.round(file.size * PwaCornerMarkInset);
-    const stone = await sharp(stone_svg)
-        .resize(mark_size, mark_size, { fit: "contain" })
-        .png()
-        .toBuffer();
+    if (file.kind == "favicon") {
+        const half = file.size / 2;
+        const circle = Buffer.from(
+            `<svg xmlns="http://www.w3.org/2000/svg" width="${file.size}" height="${file.size}"><circle cx="${half}" cy="${half}" r="${half}"/></svg>`,
+        );
 
-    return base
-        .composite([{ input: stone, left: inset, top: inset }])
-        .png()
-        .toBuffer();
+        layers.push({ input: circle, blend: "dest-in" });
+    }
+
+    // Layers apply in order, so the circle cuts the mark too.
+    return base.ensureAlpha().composite(layers).png().toBuffer();
+}
+
+function markPlacement({ kind }: { kind: PwaInstallIconFile["kind"] }) {
+    if (kind == "favicon") {
+        return { scale: PwaFaviconCornerMarkScale, inset_scale: PwaFaviconCornerMarkInset };
+    }
+
+    if (kind == "any") {
+        return { scale: PwaSquareCornerMarkScale, inset_scale: PwaSquareCornerMarkInset };
+    }
+
+    return { scale: PwaCornerMarkScale, inset_scale: PwaCornerMarkInset };
 }

src/harness/embeddings/fit-vector-dimensions.ts

@@ -4,7 +4,7 @@ type Params = {
     vector: number[];
     /**
      * Defaults to `AppData.EmbeddingDimensions`, the width of the fixed
-     * `bot_file_vectors` table.
+     * `knowledgebase_vectors` table.
      */
     dimensions?: number;
 };

src/harness/tools/bot/tasks/create-task-tool.ts

@@ -39,19 +39,18 @@ type Args = {
     description: string;
     bot_id?: number;
     workspace_id?: number;
-    shared?: boolean;
     status?: TaskStatus;
 };
 
-type Place = { bot_id?: number; workspace_id?: number };
+type Place = { bot_id: number; workspace_id?: number };
 
 /**
  * Worked out from the arguments and who is asking: where the task goes and
- * who has it, or why it can't. Outside a workspace a worker adds only to
- * its own list. In one the task stays in that workspace: its own by
- * default, a member's as the workspace's admin, or shared (nobody's yet).
- * The coordinator adds to any worker's own list or any workspace, for one
- * of its bots or shared.
+ * who has it, or why it can't. Every task belongs to a bot. Outside a
+ * workspace a worker adds only to its own list. In one the task stays in
+ * that workspace: its own by default, or a member's as the workspace's
+ * admin. The coordinator adds to any worker's own list or any workspace,
+ * for one of its bots or itself.
  */
 function resolvePlace({
     args,
@@ -69,22 +68,18 @@ function resolvePlace({
     const target_bot = args.bot_id ? Number(args.bot_id) : null;
     const target_workspace = args.workspace_id ? Number(args.workspace_id) : null;
 
-    if (args.shared && target_bot) {
-        return `A shared task is nobody's yet: leave bot_id out, or drop shared to give it to that bot.`;
-    }
-
     if (is_coordinator) {
         const place_workspace = target_workspace || (workspace_id ? Number(workspace_id) : null);
 
         if (place_workspace) {
-            return target_bot ? { workspace_id: place_workspace, bot_id: target_bot } : { workspace_id: place_workspace };
+            return { workspace_id: place_workspace, bot_id: target_bot || Number(bot_id) };
         }
 
         return { bot_id: target_bot || Number(bot_id) };
     }
 
     if (!workspace_id) {
-        if (target_workspace || args.shared) {
+        if (target_workspace) {
             return `You're not in a workspace here, so a task can only go on your own list (bot_id ${bot_id}).`;
         }
 
@@ -99,14 +94,12 @@ function resolvePlace({
         return `You can only add tasks in this workspace (#${workspace_id}).`;
     }
 
-    if (args.shared) return { workspace_id: Number(workspace_id) };
-
     const is_member = members.some((member) => member.id == target_bot);
 
     if (target_bot && target_bot != Number(bot_id) && !is_member) {
         return members.length
-            ? `You can only give a task here to yourself (#${bot_id}) or a member (${members.map((m) => m.id).join(", ")}), or leave it shared.`
-            : `You can only give a task here to yourself (#${bot_id}), or leave it shared.`;
+            ? `You can only give a task here to yourself (#${bot_id}) or a member (${members.map((m) => m.id).join(", ")}).`
+            : `You can only give a task here to yourself (#${bot_id}).`;
     }
 
     return { workspace_id: Number(workspace_id), bot_id: target_bot || Number(bot_id) };
@@ -114,8 +107,8 @@ function resolvePlace({
 
 /**
  * Lets a bot add a task to a list it works on: its own, or in a workspace
- * its own there (a member's as its admin) or a shared one. The coordinator
- * can add to any worker's list and any workspace.
+ * its own there (a member's as its admin). The coordinator can add to any
+ * worker's list and any workspace. Every task belongs to a bot.
  */
 export default function createTaskTool({
     user_id,
@@ -134,8 +127,8 @@ export default function createTaskTool({
     return {
         name: "create_task",
         description: is_coordinator
-            ? `Add a task: to a worker's own list (bot_id: ${crew || "no workers yet"}), to a workspace (workspace_id: ${places || "no workspaces yet"}) for one of its bots (with bot_id) or shared (without), or to your own list (leave both ids out${workspace_id ? "; here, leaving them out makes it a shared task in this workspace" : ""}). Write the description as a brief: what to do, where, and how you'll know it's done.`
-            : `Add a task: ${workspace_id ? `in this workspace, yours by default${team ? `, a member's as this workspace's admin (bot_id: ${team})` : ""}, or shared (shared: true) for whichever bot here takes it up` : `on your own list (bot_id ${bot_id}, the default)`}. Write the description as a brief: what to do, where, and how you'll know it's done.`,
+            ? `Add a task: to a worker's own list (bot_id: ${crew || "no workers yet"}), to a workspace (workspace_id: ${places || "no workspaces yet"}) for one of its bots (with bot_id; without, it's yours there), or to your own list (leave both ids out${workspace_id ? "; here, leaving them out makes it yours in this workspace" : ""}). Every task belongs to one bot. Write the description as a brief: what to do, where, and how you'll know it's done.`
+            : `Add a task: ${workspace_id ? `in this workspace, yours by default${team ? `, or a member's as this workspace's admin (bot_id: ${team})` : ""}` : `on your own list (bot_id ${bot_id}, the default)`}. Write the description as a brief: what to do, where, and how you'll know it's done.`,
         schema: z.object({
             title: z.string().trim().min(1).max(AppData.TaskTitleMaxLength).describe(`A short label`),
             description: z
@@ -144,7 +137,7 @@ export default function createTaskTool({
                 .describe(`The brief, in markdown: what to do, where, and what done looks like`),
             bot_id: z.number().int().optional().describe(
                 is_coordinator
-                    ? `Who has it: a worker's own list, or with workspace_id a bot working there. Leave out for your own list or a shared workspace task.`
+                    ? `Who has it: a worker's own list, or with workspace_id a bot working there. Leave out to have it yourself.`
                     : team
                       ? `Who has it here: you (${bot_id}, the default) or a member (${team}).`
                       : `Your own id (${bot_id}); leaving it out has the same effect.`,
@@ -158,10 +151,6 @@ export default function createTaskTool({
                         ? `The workspace it's in, when it isn't a bot's own list.`
                         : `This workspace (${workspace_id || "none"}); leaving it out has the same effect.`,
                 ),
-            shared: z
-                .boolean()
-                .optional()
-                .describe(`In a workspace: nobody's yet, for whichever bot there takes it up. Defaults to false.`),
             status: z.enum(TaskStatusValues).optional().describe(`Defaults to to do`),
         }),
         run: async ({ args }) => {
@@ -201,7 +190,7 @@ export default function createTaskTool({
             });
 
             const where = place.workspace_id
-                ? `in workspace ${place.workspace_id}, ${!place.bot_id ? "shared" : place.bot_id == Number(bot_id) ? "yours" : `bot ${place.bot_id}'s`}`
+                ? `in workspace ${place.workspace_id}, ${place.bot_id == Number(bot_id) ? "yours" : `bot ${place.bot_id}'s`}`
                 : place.bot_id == Number(bot_id)
                   ? "to your list"
                   : `to bot ${place.bot_id}'s list`;

src/harness/tools/bot/tasks/grab-bot-task-tools.ts

@@ -25,8 +25,8 @@ type Params = {
 
 /**
  * The task tools every run gets (`src/harness/tools/bot/tasks/`): the
- * to-do list (a bot's own, and in each workspace its tasks there and the
- * shared ones), so a chat reply, a task and a scheduled run all read it,
+ * to-do list (every task belongs to one bot: its own list and its tasks in
+ * each workspace; an admin's whole workspace), so a chat reply, a task and a scheduled run all read it,
  * move it along and file what they produced. The coordinator's create_task
  * may target any worker or workspace; its update_task may link any file it
  * can reach. A workspace admin's may give tasks there to the other members.

src/harness/tools/bot/tasks/list-tasks-tool.ts

@@ -27,14 +27,15 @@ type Args = {
 };
 
 /**
- * Lets a bot read the tasks it can work on: its own list and every
- * workspace it works in, or for the coordinator the whole install. Used
+ * Lets a bot read the tasks it can work on: its own (its list and what it
+ * has in workspaces), every task of a workspace it's admin of, or for the
+ * coordinator the whole install. Used
  * both to plan work and to remind the user of what's pending.
  */
 export default function listTasksTool({ bot_id, user_id, is_coordinator, admin_workspace_ids, workspace_id }: Params): HarnessTool<Args> {
     return {
         name: "list_tasks",
-        description: `List tasks: yours (your own list and the ones you have in workspaces), the ones in ${workspace_id ? "this workspace" : "a workspace"}, or ${is_coordinator ? "every task in the install" : "every task you can reach (yours, and in your workspaces the shared ones nobody has yet)"}. Each line is its id, status, title, whose it is, its files and its latest note. Open ones come first (blocked, then doing, then to do). Use it to plan work, and to check what's still pending when the user says hello or asks what's next.`,
+        description: `List tasks: yours (your own list and the ones you have in workspaces), the ones in ${workspace_id ? "this workspace" : "a workspace"}, or ${is_coordinator ? "every task in the install" : "every task you can reach (yours, and as a workspace's admin every task there)"}. Each line is its id, status, title, whose it is, its files and its latest note. Open ones come first (blocked, then doing, then to do). Use it to plan work, and to check what's still pending when the user says hello or asks what's next.`,
         schema: z.object({
             scope: z
                 .enum(["mine", "workspace", "all"])
@@ -64,7 +65,7 @@ export default function listTasksTool({ bot_id, user_id, is_coordinator, admin_w
 
             if (!tasks.length) {
                 return args.open_only
-                    ? `You have no open tasks${args.status ? ` with that status` : ""}.`
+                    ? `No open tasks${args.status ? ` with that status` : ""} here. Finished and cancelled ones aren't listed with open_only: list without it before saying a task is gone.`
                     : `No tasks${args.status ? ` with that status` : ""}.`;
             }
 
@@ -74,7 +75,7 @@ export default function listTasksTool({ bot_id, user_id, is_coordinator, admin_w
 
             return tasks
                 .map((task) => {
-                    const who = task.bot_id ? task.bot_name || `bot ${task.bot_id}` : "shared, nobody's yet";
+                    const who = task.bot_name || `bot ${task.bot_id}`;
                     const where = task.workspace_id
                         ? `in ${names.get(Number(task.workspace_id)) || `workspace ${task.workspace_id}`}, ${who}`
                         : `on ${who}'s list`;

src/harness/tools/bot/tasks/update-task-tool.ts

@@ -4,7 +4,6 @@ import { type TaskStatus, TaskStatusValues } from "@/src/dict/task-dict";
 import addTaskNote from "@/src/functions/backend/db/tasks/add-task-note";
 import attachTaskFile from "@/src/functions/backend/db/tasks/attach-task-file";
 import grabRunTask from "@/src/functions/backend/db/tasks/grab-run-task";
-import moveTask from "@/src/functions/backend/db/tasks/move-task";
 import grabTaskMedia from "@/src/functions/backend/db/tasks/grab-task-media";
 import updateTask from "@/src/functions/backend/db/tasks/update-task";
 import type { StorageScope } from "@/src/utils/storage-access";
@@ -41,7 +40,7 @@ type Args = {
 export default function updateTaskTool({ user_id, bot_id, is_coordinator, admin_workspace_ids, scopes = [] }: Params): HarnessTool<Args> {
     return {
         name: "update_task",
-        description: `Move a task along. Set its status (blocked needs a reason: what the user has to do), log a note of what you did or found, and file the files it produced. Setting a shared workspace task to doing makes it yours. Call it as the work advances, not only at the end; every note is what the user sees on the task. Mark done only once you've verified the work (see the check note): starting a task isn't finishing it. To file a deliverable, save it first with write_file, make_pdf or generate_image, then pass its [file:ID] number.`,
+        description: `Move a task along. Set its status (blocked needs a reason: what the user has to do), log a note of what you did or found, and file the files it produced. Call it as the work advances, not only at the end; every note is what the user sees on the task. Mark done only once you've verified the work (see the check note): starting a task isn't finishing it. To file a deliverable, save it first with write_file, make_pdf or generate_image, then pass its [file:ID] number.`,
         schema: z.object({
             task_id: z.number().int(),
             status: z.enum(TaskStatusValues).optional(),
@@ -68,15 +67,6 @@ export default function updateTaskTool({ user_id, bot_id, is_coordinator, admin_
             const task = await grabRunTask({ task_id: args.task_id, bot_id, is_coordinator, admin_workspace_ids });
             const said: string[] = [];
 
-            /**
-             * A shared task in a workspace is nobody's until a bot starts on
-             * it: then it's that bot's.
-             */
-            if (args.status == "doing" && task.workspace_id && !task.bot_id && !is_coordinator) {
-                await moveTask({ task_id: task.id!, bot_id, workspace_id: task.workspace_id });
-                said.push(`it's yours now`);
-            }
-
             if (args.status || args.blocked_reason !== undefined) {
                 if (args.status == "blocked" && !args.blocked_reason?.trim()) {
                     return `Blocked needs a reason: say in blocked_reason what the user has to do.`;
@@ -90,7 +80,9 @@ export default function updateTaskTool({ user_id, bot_id, is_coordinator, admin_
                     updated_by_bot_id: bot_id,
                 });
 
-                if (args.status) said.push(`status is now ${args.status}`);
+                if (args.status) {
+                    said.push(task.status == args.status ? `status was already ${args.status}` : `status is now ${args.status}`);
+                }
             }
 
             if (args.note?.trim()) {

src/harness/tools/computer/import-file-tool.ts

@@ -3,7 +3,6 @@ import { tmpdir } from "node:os";
 import path from "node:path";
 import { z } from "zod";
 import { AppData } from "@/src/data/app-data";
-import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
 import recordBotFile, { grabBotFileRow } from "@/src/functions/backend/db/bot-files/record-bot-file";
 import grabComputerMcpUrl from "@/src/functions/backend/db/computers/grab-computer-mcp-url";
 import computerNotReadyMessage from "@/src/utils/computer-not-ready-message";
@@ -49,7 +48,7 @@ type Args = {
  * Copies what a bot made on its computer into its own files: one file, or
  * a whole folder (taken as a gzipped tar and unpacked into one row per
  * file). The copies are kept with the app, stay after the computer is
- * rebuilt, and are searchable at once; the originals stay on the computer,
+ * rebuilt, and are readable at once; the originals stay on the computer,
  * where the message's folder ref points at the folder holding them.
  *
  * The bytes come back over the sandbox's streaming download endpoint
@@ -72,7 +71,7 @@ export default function importFileTool({
         name: "import_file",
         description: `Copy a file or folder from this computer into your own files, so it is kept with the app and stays after the computer is rebuilt. Give its path on the computer, relative to your folder there (${dirs.cwd}), e.g. "seo/coderank/raw/crawl.json" or "seo/coderank/raw"; a folder comes in as it is, in one go. Up to ${max_mb} MB. Check with list_files and the computer's own tools what you already imported: importing the same thing twice just writes over the copy, so do it once.${
             own?.scope == "workspace" ? " The copies land in the workspace's shared files." : ""
-        } Imports are searchable at once. Name a copy with its [file:ID] ref. Import the report you wrote before the last long steps of a run, so it is never lost to the step limit.`,
+        } Imports are readable at once. Name a copy with its [file:ID] ref. Import the report you wrote before the last long steps of a run, so it is never lost to the step limit.`,
         schema: z.object({
             path: z
                 .string()
@@ -190,8 +189,6 @@ async function importOneFile({
         user_id: target.user_id,
     });
 
-    if (saved.id) await enqueueFileIngest({ file_id: saved.id });
-
     if (attach && saved.media_id) {
         onAttach?.({ media_id: Number(saved.media_id), name: saved.name || name });
     }
@@ -427,8 +424,6 @@ async function unpackArchive({
 
             if (!row?.id) continue;
 
-            await enqueueFileIngest({ file_id: row.id }).catch(() => {});
-
             saved.push({ id: Number(row.id), path: row.path || `${folder}/${relative}` });
         }
     } finally {

src/harness/tools/computer/open-computer-tools.ts

@@ -12,6 +12,7 @@ import type { HarnessTool } from "../../types";
 import computerDirsShell from "./computer-dirs-shell";
 import captureSecretTool from "./capture-secret-tool";
 import runComputerCommand from "./run-computer-command";
+import withCommandTimeout from "./with-command-timeout";
 import withComputerDirs from "./with-computer-dirs";
 import withComputerEnv from "./with-computer-env";
 import writeComputerEnv from "./write-computer-env";
@@ -65,7 +66,8 @@ export type OpenComputer = {
  * shell, files, code and browser. Its envs (`grabComputerEnvs`) are written
  * to the computer and loaded by the shell and code tools; secret values are
  * redacted from everything the tools return. Every tool is kept in the
- * run's folders (`withComputerDirs`), which are created first. Screenshots
+ * run's folders (`withComputerDirs`), which are created first, and every
+ * bash command is stopped on the computer (`withCommandTimeout`). Screenshots
  * come back as readings from the media reading harness. Call `close` when the run ends.
  * Throws when the computer can't be reached, which the queue retries.
  */
@@ -125,7 +127,7 @@ export default async function openComputerTools({
         return {
             name: target.name,
             tools: [
-                ...tools.map((tool) => withComputerDirs({ tool: withComputerEnv({ tool, ...paths }), dirs })),
+                ...tools.map((tool) => withComputerDirs({ tool: withComputerEnv({ tool: withCommandTimeout({ tool }), ...paths }), dirs })),
                 ...(user_id && bot_id && onSecretRef
                     ? [
                           captureSecretTool({

src/harness/tools/computer/with-command-timeout.test.ts

@@ -0,0 +1,44 @@
+import { describe, expect, test } from "bun:test";
+import { AppData } from "@/src/data/app-data";
+import type { HarnessTool } from "../../types";
+import withCommandTimeout from "./with-command-timeout";
+
+const echoTool = (name: string): HarnessTool => ({
+    name,
+    description: "Run a command.",
+    parameters: {},
+    run: async ({ args }) => JSON.stringify(args),
+});
+
+const signal = new AbortController().signal;
+const call = async (args: Record<string, unknown>) =>
+    JSON.parse(await withCommandTimeout({ tool: echoTool("sandbox_execute_bash") }).run({ args, signal }));
+const max = AppData.ComputerCommandMaxSeconds;
+
+describe("withCommandTimeout", () => {
+    test("runs the command under timeout, reading no input, in its own bash", async () => {
+        const sent = await call({ cmd: "echo 'hi'", timeout: 20 });
+
+        expect(sent.cmd).toStartWith(`timeout -k 5 20 bash -c 'echo '\\''hi'\\''' < /dev/null; `);
+        expect(sent.cmd).toContain("[Stopped after 20s");
+        expect(sent.cmd).toEndWith("(exit $_cobble_status)");
+        expect(sent.cmd).not.toMatch(/(^|; )exit /);
+    });
+
+    test("keeps the timeout between 1 second and the cap, the cap without one", async () => {
+        expect((await call({ cmd: "ls" })).cmd).toStartWith(`timeout -k 5 ${max} `);
+        expect((await call({ cmd: "ls", timeout: 0 })).cmd).toStartWith(`timeout -k 5 ${max} `);
+        expect((await call({ cmd: "ls", timeout: 100000 })).cmd).toStartWith(`timeout -k 5 ${max} `);
+        expect((await call({ cmd: "ls", timeout: 2.5 })).cmd).toStartWith(`timeout -k 5 3 `);
+        expect(max * 1000).toBeLessThan(AppData.McpToolTimeoutMs);
+    });
+
+    test("tells the model how commands run", () => {
+        expect(withCommandTimeout({ tool: echoTool("sandbox_execute_bash") }).description).toContain("reads no input");
+    });
+
+    test("leaves other tools alone", () => {
+        const tool = echoTool("sandbox_execute_code");
+        expect(withCommandTimeout({ tool })).toBe(tool);
+    });
+});

src/harness/tools/computer/with-command-timeout.ts

@@ -0,0 +1,64 @@
+import { AppData } from "@/src/data/app-data";
+import type { HarnessTool } from "../../types";
+import { shellQuote } from "./download-computer-file";
+
+type Params = {
+    tool: HarnessTool;
+};
+
+/**
+ * Makes the computer's bash tool stop its commands on the computer. The
+ * sandbox ignores the tool's `timeout` and runs every call in one shared
+ * shell session, so a command that never ends (a network read without a
+ * timeout, a password prompt) used to keep running after the call gave up,
+ * and every later command on that computer, from any run, waited behind
+ * it. Each command now runs under `timeout` (its own, at most
+ * `ComputerCommandMaxSeconds`), reading no input, in its own bash, and
+ * says so when it was stopped. Other tools pass through.
+ */
+export default function withCommandTimeout({ tool }: Params): HarnessTool {
+    if (tool.name != "sandbox_execute_bash") return tool;
+
+    const max = AppData.ComputerCommandMaxSeconds;
+
+    return {
+        ...tool,
+        description: `${tool.description}\n\nEvery command runs in a fresh shell (cd and export don't carry over to the next call), reads no input (anything waiting for a prompt fails at once: pass -y, a password through its env, and so on) and is stopped after timeout seconds (at most ${max}, ${max} when not given). Start longer work in the background (nohup ... > log 2>&1 &) and check its log in later calls.`,
+        run: async ({ args, signal }) => {
+            if (typeof args?.cmd != "string") return tool.run({ args, signal });
+
+            const seconds = commandSeconds({ timeout: args.timeout });
+
+            return tool.run({
+                args: { ...args, timeout: seconds + 10, cmd: timedCommand({ cmd: args.cmd, seconds }) },
+                signal,
+            });
+        },
+    };
+}
+
+/**
+ * The command's own timeout in whole seconds, kept to 1..`ComputerCommandMaxSeconds`.
+ */
+function commandSeconds({ timeout }: { timeout?: unknown }) {
+    const max = AppData.ComputerCommandMaxSeconds;
+    const asked = typeof timeout == "number" ? timeout : Number(timeout);
+
+    if (!Number.isFinite(asked) || asked <= 0) return max;
+
+    return Math.min(max, Math.max(1, Math.ceil(asked)));
+}
+
+/**
+ * `cmd` under `timeout` (TERM, then KILL 5 seconds later, to its whole
+ * process group), with its exit code kept. Never `exit`s: the sandbox's
+ * session shell would go with it.
+ */
+function timedCommand({ cmd, seconds }: { cmd: string; seconds: number }) {
+    return [
+        `timeout -k 5 ${seconds} bash -c ${shellQuote({ text: cmd })} < /dev/null`,
+        `_cobble_status=$?`,
+        `if [ $_cobble_status -eq 124 ] || [ $_cobble_status -eq 137 ]; then echo "[Stopped after ${seconds}s: the command timed out. Run long work in the background and check its log.]"; fi`,
+        `(exit $_cobble_status)`,
+    ].join("; ");
+}

src/harness/tools/storage/generate-image-tool.ts

@@ -1,5 +1,4 @@
 import { z } from "zod";
-import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
 import recordBotFile from "@/src/functions/backend/db/bot-files/record-bot-file";
 import generateImage from "@/src/harness/image/generate-image";
 import type { HarnessTool } from "../../types";
@@ -98,8 +97,6 @@ export default function generateImageTool(context: StorageToolContext): HarnessT
                     user_id: context.user_id,
                 });
 
-                await enqueueFileIngest({ file_id: file.id! });
-
                 if (args.attach && file.media_id) {
                     context.onAttach?.({ media_id: Number(file.media_id), name: file.name || "" });
                 }

src/harness/tools/storage/grab-storage-tools.ts

@@ -1,11 +1,9 @@
 import type { HarnessTool } from "../../types";
 import copyFileTool from "./copy-file-tool";
 import deleteFileTool from "./delete-file-tool";
-import ingestFileTool from "./ingest-file-tool";
 import listFilesTool from "./list-files-tool";
 import moveFileTool from "./move-file-tool";
 import readFileTool from "./read-file-tool";
-import searchFilesTool from "./search-files-tool";
 import type { StorageToolContext } from "./storage-tool-context";
 import writeFileTool from "./write-file-tool";
 
@@ -19,10 +17,8 @@ export default function grabStorageTools(context: StorageToolContext): HarnessTo
         listFilesTool(context),
         readFileTool(context),
         writeFileTool(context),
-        searchFilesTool(context),
         moveFileTool(context),
         deleteFileTool(context),
-        ingestFileTool(context),
         ...(context.scopes.length > 1 ? [copyFileTool(context)] : []),
     ];
 }

src/harness/tools/storage/ingest-file-tool.ts

@@ -1,44 +0,0 @@
-import { z } from "zod";
-import { AppData } from "@/src/data/app-data";
-import ingestBotFile from "@/src/functions/backend/db/bot-files/ingest-bot-file";
-import type { HarnessTool } from "../../types";
-import type { StorageToolContext } from "./storage-tool-context";
-
-type Args = {
-    file_id: number;
-    read_media?: boolean;
-};
-
-/**
- * Ingests a file again: reads its text, chunks it and embeds it, so a
- * changed or previously failed file is searchable. Images and audio are
- * only read when asked for, since that costs a model call.
- */
-export default function ingestFileTool(_context: StorageToolContext): HarnessTool<Args> {
-    return {
-        name: "ingest_file",
-        description: `Make one of your files searchable again (a course correction when search misses something you know is in it), by its ref id. Images and scans are only read when "read_media" is true.`,
-        schema: z.object({
-            file_id: z.number().int().positive(),
-            read_media: z.boolean().optional(),
-        }),
-        run: async ({ args, signal }) => {
-            const result = await ingestBotFile({
-                file_id: args.file_id,
-                read_media: args.read_media,
-                signal,
-            });
-
-            switch (result.status) {
-                case "ingested":
-                    return `Ingested: ${result.chunks} passage(s), searchable now.`;
-                case "unavailable":
-                    return `Its text is stored and searchable by words, but no embedding model is set, so meaning search is off.`;
-                case "skipped":
-                    return `It's too large to read (over ${Math.round(AppData.IngestMaxBytes / (1024 * 1024))} MB).`;
-                default:
-                    return `It has no text to read.`;
-            }
-        },
-    };
-}

src/harness/tools/storage/list-files-tool.ts

@@ -50,9 +50,8 @@ export default function listFilesTool(context: StorageToolContext): HarnessTool<
 
             const lines = files.map((file) => {
                 const where = file.scope == "workspace" ? ` (workspace ${file.owner_id})` : "";
-                const ingested = file.ingested_at ? "" : " - not searchable yet";
 
-                return `- ${file.path} - ${formatBytes({ bytes: Number(file.size) || 0 })}${where}${ingested} ${fileRef({ file_id: file.id! })}`;
+                return `- ${file.path} - ${formatBytes({ bytes: Number(file.size) || 0 })}${where} ${fileRef({ file_id: file.id! })}`;
             });
 
             const more =

src/harness/tools/storage/make-pdf-tool.ts

@@ -1,6 +1,5 @@
 import { z } from "zod";
 import { PDFDocument, StandardFonts, rgb } from "pdf-lib";
-import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
 import recordBotFile from "@/src/functions/backend/db/bot-files/record-bot-file";
 import type { HarnessTool } from "../../types";
 import {
@@ -69,8 +68,6 @@ export default function makePdfTool(context: StorageToolContext): HarnessTool<Ar
                 user_id: context.user_id,
             });
 
-            await enqueueFileIngest({ file_id: file.id! });
-
             if (args.attach && file.media_id) {
                 context.onAttach?.({ media_id: Number(file.media_id), name: file.name || "" });
             }

src/harness/tools/storage/search-files-tool.ts

@@ -1,49 +0,0 @@
-import { z } from "zod";
-import { AppData } from "@/src/data/app-data";
-import searchBotFiles from "@/src/functions/backend/db/bot-files/search-bot-files";
-import type { HarnessTool } from "../../types";
-import { fileRef, type StorageToolContext } from "./storage-tool-context";
-
-type Args = {
-    query: string;
-    file_id?: number;
-    limit?: number;
-};
-
-/**
- * Searches the bot's saved files by meaning (through the embedding
- * vectors), falling back to words, and returns the passages with their
- * file's ref so the bot can read more.
- */
-export default function searchFilesTool(context: StorageToolContext): HarnessTool<Args> {
-    return {
-        name: "search_files",
-        description: `Search the files you have saved (documents the user gave you and what you wrote) and get the passages that answer "query", with their file's ref. Search before saying you don't know something that may be in a file, and before writing a document from memory.`,
-        schema: z.object({
-            query: z.string().min(1).max(500),
-            file_id: z.number().int().positive().optional().describe(`Only this file's passages`),
-            limit: z.number().int().min(1).max(20).optional(),
-        }),
-        run: async ({ args, signal }) => {
-            const result = await searchBotFiles({
-                scopes: context.scopes,
-                query: args.query,
-                file_id: args.file_id,
-                limit: args.limit || AppData.RagTopChunks,
-                signal,
-                bot_id: context.bot_id,
-                workspace_id: context.workspace_id,
-            });
-
-            if (!result.hits.length) return result.note;
-
-            const passages = result.hits.map((hit, index) => {
-                const where = hit.scope == "workspace" ? ` (workspace ${hit.scope})` : "";
-
-                return `## ${index + 1}. ${hit.name}${where} ${fileRef({ file_id: hit.file_id })}\n${hit.text.trim()}`;
-            });
-
-            return `${result.note}\n\n${passages.join("\n\n")}`;
-        },
-    };
-}

src/harness/tools/storage/write-file-tool.ts

@@ -1,7 +1,6 @@
 import { z } from "zod";
 import { AppData } from "@/src/data/app-data";
 import type { BotFilePurpose } from "@/src/dict/bot-file-dict";
-import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
 import recordBotFile from "@/src/functions/backend/db/bot-files/record-bot-file";
 import type { HarnessTool } from "../../types";
 import {
@@ -23,7 +22,7 @@ type Args = {
 /**
  * Writes a text file (markdown, CSV, JSON, code, plain text) into one of
  * the bot's directories, making folders as needed. The file stays for
- * later runs and becomes searchable.
+ * later runs.
  */
 export default function writeFileTool(context: StorageToolContext): HarnessTool<Args> {
     const own = context.scopes[0];
@@ -32,7 +31,7 @@ export default function writeFileTool(context: StorageToolContext): HarnessTool<
         name: "write_file",
         description: `Save a text file for later: markdown, CSV, JSON, code or plain text. Give it a path like "reports/q3.md"; folders are made as needed. Files belong to ${
             own?.scope == "workspace" ? "the workspace you're working in" : "you"
-        } and stay between runs: write a file before telling the user something is saved, and search_files to find one you wrote before. It is readable and searchable at once.${
+        } and stay between runs: write a file before telling the user something is saved, and list_files (by name or path) to find one you wrote before. It is readable at once.${
             context.scopes.length > 1
                 ? ` "scope" picks a directory: ${context.scopes.map((s) => `${s.scope}:${s.owner_id}`).join(", ")}.`
                 : ""
@@ -74,8 +73,6 @@ export default function writeFileTool(context: StorageToolContext): HarnessTool<
                 user_id: context.user_id,
             });
 
-            await enqueueFileIngest({ file_id: file.id! });
-
             if (args.attach && file.media_id) {
                 context.onAttach?.({ media_id: Number(file.media_id), name: file.name || "" });
             }

src/hooks/use-debounced-value.ts

@@ -0,0 +1,30 @@
+import { useEffect, useState } from "react";
+import { AppData } from "@/src/data/app-data";
+
+type Params<T> = {
+    value: T;
+    delay_ms?: number;
+};
+
+/**
+ * `value` once it has stopped changing for `delay_ms`
+ * (`AppData.SearchDebounceMs` by default), so a search box searches as the
+ * user types without a request per keystroke. An emptied value comes
+ * through at once.
+ */
+export default function useDebouncedValue<T>({ value, delay_ms = AppData.SearchDebounceMs }: Params<T>) {
+    const [debounced, setDebounced] = useState(value);
+
+    useEffect(() => {
+        if (!value) {
+            setDebounced(value);
+            return;
+        }
+
+        const timer = setTimeout(() => setDebounced(value), delay_ms);
+
+        return () => clearTimeout(timer);
+    }, [value, delay_ms]);
+
+    return debounced;
+}

src/layouts/bots/(partials)/rail-chatroom-row.tsx

@@ -2,7 +2,7 @@ import Link from "@moduletrace/twui/layout/Link";
 import Span from "@moduletrace/twui/layout/Span";
 import Stack from "@moduletrace/twui/layout/Stack";
 import { twMerge } from "tailwind-merge";
-import ChatroomStone from "@/src/components/general/chatroom-stone";
+import ChatroomStoneCluster from "@/src/components/general/chatroom-stone-cluster";
 import type { ChatroomSummary } from "@/src/types";
 import chatroomStatusLine from "@/src/utils/chatroom-status-line";
 
@@ -13,12 +13,11 @@ type Props = {
 };
 
 /**
- * A chatroom in the rail, the same 48px row as a bot: the asking bot's
- * stone, the request and where it stands. Unread is said in words.
+ * A chatroom in the rail, the same 48px row as a bot: its bots' stones
+ * in one avatar-sized cluster, the request and where it stands. Unread
+ * is said in words.
  */
 export default function RailChatroomRow({ chatroom, href, selected }: Props) {
-    const requester = chatroom.bots[0];
-
     return (
         <Link
             href={href}
@@ -32,7 +31,7 @@ export default function RailChatroomRow({ chatroom, href, selected }: Props) {
             )}
         >
             <Span className="mx-auto @rail:mx-0 inline-flex shrink-0">
-                {requester ? <ChatroomStone bot={requester} size={28} /> : null}
+                <ChatroomStoneCluster bots={chatroom.bots} />
             </Span>
             <Stack className="hidden @rail:flex grow min-w-0 gap-0!">
                 <Span className="text-sm font-bold leading-5 text-ink truncate w-full">

src/pages/admin/bots/[bot_id]/(data)/message-markdown-components.tsx

@@ -42,7 +42,7 @@ export const MessageMarkdownComponents: Components = {
         </MarkdownList>
     ),
     code: ({ children }) => (
-        <Span className="font-mono text-[13px] px-1 py-px rounded-default-xs bg-sunken text-ink break-words">
+        <Span className="font-mono text-[13px] px-1 py-px rounded-default-xs bg-sunken text-ink wrap-anywhere">
             {children}
         </Span>
     ),
@@ -58,7 +58,7 @@ export const MessageMarkdownComponents: Components = {
     ),
     table: ({ children }) => (
         <Stack className="w-full max-w-full overflow-x-auto items-start">
-            <table className="text-[13px] leading-5 text-ink border-collapse">{children}</table>
+            <table className="text-[13px] leading-5 text-ink border-collapse wrap-break-word">{children}</table>
         </Stack>
     ),
     th: ({ children }) => (

src/pages/admin/bots/[bot_id]/(data)/user-markdown-components.tsx

@@ -39,7 +39,7 @@ export const UserMarkdownComponents: Components = {
         </MarkdownList>
     ),
     code: ({ children }) => (
-        <Span className="font-mono text-[13px] px-1 py-px rounded-default-xs bg-on-ink/15 text-on-ink break-words">
+        <Span className="font-mono text-[13px] px-1 py-px rounded-default-xs bg-on-ink/15 text-on-ink wrap-anywhere">
             {children}
         </Span>
     ),

src/pages/admin/bots/[bot_id]/(hooks)/use-bot-files.ts

@@ -1,6 +1,6 @@
 import fetchApi from "@moduletrace/twui/utils/fetch/fetchApi";
 import type { APIResponseObject } from "@moduletrace/bunext/types";
-import { useCallback, useContext, useEffect, useState } from "react";
+import { useCallback, useContext, useEffect, useRef, useState } from "react";
 import { AppContext } from "@/src/pages/__root";
 import type { BotFileScopesPayload } from "@/src/types";
 
@@ -12,22 +12,30 @@ type Params = {
      * the first page of files).
      */
     enabled?: boolean;
+    /**
+     * Only files whose name or path has this.
+     */
+    search?: string;
 };
 
 /**
  * A conversation's files: the page's own copy first, then the folder and
- * search the modal asks for. Saves and deletes refresh it.
+ * search the modal asks for (reloaded when the search changes; an answer
+ * that a newer request overtook is dropped). Saves and deletes refresh it.
  */
-export default function useBotFiles({ bot_id, workspace_id, enabled = true }: Params) {
+export default function useBotFiles({ bot_id, workspace_id, enabled = true, search }: Params) {
     const { pageProps } = useContext(AppContext);
     const [data, setData] = useState<BotFileScopesPayload | undefined>(
         pageProps?.bot_files || undefined,
     );
     const [loading, setLoading] = useState(false);
     const [error, setError] = useState<string>();
+    const latest = useRef(0);
 
     const load = useCallback(
         async (params?: { folder?: string; search?: string }) => {
+            const request = ++latest.current;
+
             setLoading(true);
             setError(undefined);
 
@@ -44,6 +52,8 @@ export default function useBotFiles({ bot_id, workspace_id, enabled = true }: Pa
                 { method: "GET" },
             ).catch(() => null);
 
+            if (request != latest.current) return;
+
             setLoading(false);
 
             if (!res?.success || !res.singleRes) {
@@ -57,8 +67,8 @@ export default function useBotFiles({ bot_id, workspace_id, enabled = true }: Pa
     );
 
     useEffect(() => {
-        if (enabled) load().catch(() => {});
-    }, [enabled, load]);
+        if (enabled) load({ search }).catch(() => {});
+    }, [enabled, load, search]);
 
     return { data, loading, error, reload: load, setData };
 }

src/pages/admin/bots/[bot_id]/(hooks)/use-bot-tasks.ts

@@ -15,7 +15,7 @@ type Params = {
 
 /**
  * A conversation's tasks: a bot's own list, or in a workspace the ones it
- * has there plus the shared ones nobody has yet. Loaded on open and refreshed after every change,
+ * has there. Loaded on open and refreshed after every change,
  * never part of the page props.
  */
 export default function useBotTasks({ bot_id, workspace_id, enabled = true }: Params) {

src/pages/admin/bots/[bot_id]/(hooks)/use-reply-catch-up.ts

@@ -1,6 +1,10 @@
 import { useEffect, useRef } from "react";
 import { AppData } from "@/src/data/app-data";
-import type { BotMessageWithAttachments, WebSocketDataType } from "@/src/types";
+import type {
+    BotMessageWithAttachments,
+    SessionMessagesResponse,
+    WebSocketDataType,
+} from "@/src/types";
 import postSessionApi from "@/src/functions/frontend/post-session-api";
 
 type Params = {
@@ -19,9 +23,13 @@ type Params = {
  * While a reply is due, checks the session's latest messages every
  * `AppData.BotReplyCatchUpMs`. Replies arrive live; this only catches one
  * whose push was missed (the socket was down at the time), so the
- * thinking row can't stay up until a reload. A missed message is replayed
+ * thinking row can't stay up until a reload. It also checks at once when
+ * the page comes back into view or online. A missed message is replayed
  * as the `server:bot-message` it would have been, so the thread, the rail
- * and the unread flag all take it the usual way.
+ * and the unread flag all take it the usual way. The bot's face, status
+ * line and active task are replayed too, as `server:bot-updated`, so a
+ * reply that ended without a final message (its face push missed) can't
+ * leave the bot working on screen.
  */
 export default function useReplyCatchUp({ session_id, awaiting_reply, last_message_id }: Params) {
     const last_message = useRef(Number(last_message_id) || 0);
@@ -33,28 +41,52 @@ export default function useReplyCatchUp({ session_id, awaiting_reply, last_messa
 
         let stale = false;
 
-        const timer = setInterval(async () => {
-            const res = await postSessionApi<BotMessageWithAttachments>({
+        const replay = ({ data }: { data: WebSocketDataType }) =>
+            window.dispatchEvent(new CustomEvent("wsDataEvent", { detail: { data, message: data } }));
+
+        const check = async () => {
+            const res = (await postSessionApi<BotMessageWithAttachments>({
                 path: "session-messages",
                 body: { session_id },
-            }).catch(() => null);
+            }).catch(() => null)) as SessionMessagesResponse | null;
 
             if (stale || !res?.success || !Array.isArray(res.payload)) return;
 
-            const missed = (res.payload as BotMessageWithAttachments[]).filter(
+            const missed = res.payload.filter(
                 (message) => Number(message.id) > last_message.current,
             );
 
             for (const message of missed) {
-                const data: WebSocketDataType = { event: "server:bot-message", data: message };
-
-                window.dispatchEvent(new CustomEvent("wsDataEvent", { detail: { data, message: data } }));
+                replay({ data: { event: "server:bot-message", data: message } });
             }
-        }, AppData.BotReplyCatchUpMs);
+
+            /**
+             * After the messages, so a reply that ended silently (its last
+             * word an interim one) drops the row once its face moves on.
+             */
+            if (res.bot?.id) replay({ data: { event: "server:bot-updated", data: res.bot } });
+        };
+
+        /**
+         * Coming back to the app (a notification tapped, the PWA brought
+         * forward) checks at once: the socket was likely down meanwhile.
+         */
+        const onVisible = () => {
+            if (document.visibilityState == "visible") check();
+        };
+
+        const timer = setInterval(check, AppData.BotReplyCatchUpMs);
+
+        window.addEventListener("online", check);
+        window.addEventListener("pageshow", check);
+        document.addEventListener("visibilitychange", onVisible);
 
         return () => {
             stale = true;
             clearInterval(timer);
+            window.removeEventListener("online", check);
+            window.removeEventListener("pageshow", check);
+            document.removeEventListener("visibilitychange", onVisible);
         };
     }, [awaiting_reply, session_id]);
 }

src/pages/admin/bots/[bot_id]/(partials)/bot-files-modal.tsx

@@ -12,6 +12,7 @@ import type { BUN_SQLITE_COBBLE_WORKSPACES } from "@/db/types/db";
 import type { BotWithLastMessage } from "@/src/types";
 import HeaderIconButton from "@/src/components/general/header-icon-button";
 import { AppContext } from "@/src/pages/__root";
+import useDebouncedValue from "@/src/hooks/use-debounced-value";
 import canManageCrew from "@/src/utils/rbac/can-manage-crew";
 import useBotFiles from "../(hooks)/use-bot-files";
 import FileRow from "./file-row";
@@ -29,21 +30,23 @@ type Props = {
 
 /**
  * The files a conversation works with: a bot's own folder, or the
- * workspace's shared one. Each can be opened, read again so it can be
- * searched, or deleted (a crew manager only). The list loads when the
- * modal opens, and search goes through the same endpoint, so the page
- * props stay small.
+ * workspace's shared one. Each can be opened or deleted (a crew manager
+ * only). The list loads when the modal opens, and search (by name or path,
+ * as the user types) goes through the same endpoint, so the page props
+ * stay small.
  */
 export default function BotFilesModal({ bot, workspace, menu, on_open }: Props) {
     const { pageProps } = useContext(AppContext);
     const [open, setOpen] = useState(false);
     const [search, setSearch] = useState("");
+    const query = useDebouncedValue({ value: search.trim() });
     const [busy_id, setBusyId] = useState<number>();
     const [status, setStatus] = useState<string>();
     const { data, loading, error, reload } = useBotFiles({
         bot_id: Number(bot.id),
         workspace_id: workspace?.id ? Number(workspace.id) : null,
         enabled: open,
+        search: query,
     });
 
     const can_manage = canManageCrew({ user_types: pageProps?.user_types });
@@ -76,7 +79,7 @@ export default function BotFilesModal({ bot, workspace, menu, on_open }: Props)
         setBusyId(undefined);
         setStatus(res?.success ? done : res?.msg || `That didn't work.`);
 
-        if (res?.success) await reload({ search });
+        if (res?.success) await reload({ search: query });
     };
 
     const files = data?.files || [];
@@ -105,7 +108,7 @@ export default function BotFilesModal({ bot, workspace, menu, on_open }: Props)
                 <H2 className="text-lg! leading-6! tracking-normal! font-bold text-ink">{title}</H2>
                 <Span className="text-[13px] leading-[18px] text-ink-secondary">
                     {workspace?.id
-                        ? `Files every bot here can read and search. They're kept with the app, not on a computer.`
+                        ? `Files every bot here can find and read. They're kept with the app, not on a computer.`
                         : `What ${name} has saved: documents you sent, files it wrote and images it made. They're kept with the app, not on a computer.${can_manage ? " Deleting one removes it for good." : ""}`}
                 </Span>
                 {usage ? (
@@ -128,13 +131,10 @@ export default function BotFilesModal({ bot, workspace, menu, on_open }: Props)
 
             <Stack className="w-full mt-4 gap-2! items-stretch">
                 <Input
-                    placeholder="Search files"
+                    placeholder="Search by name or path"
                     value={search}
                     onChange={(event) => setSearch(event.target.value)}
-                    onKeyDown={(event) => {
-                        if (event.key == "Enter") reload({ search }).catch(() => {});
-                    }}
-                    clearInputProps={{ onClick: () => reload({}).catch(() => {}) }}
+                    clearInputProps={{ onClick: () => setSearch("") }}
                 />
             </Stack>
 
@@ -146,14 +146,6 @@ export default function BotFilesModal({ bot, workspace, menu, on_open }: Props)
                             file={file}
                             can_manage={can_manage}
                             busy={busy_id == file.id}
-                            onIngest={() =>
-                                run({
-                                    url: `/api/admin/bots/ingest-file`,
-                                    payload: { file_id: file.id },
-                                    file_id: file.id,
-                                    done: `${file.name} will be read again shortly.`,
-                                })
-                            }
                             onDelete={() =>
                                 run({
                                     url: `/api/admin/bots/delete-file`,
@@ -168,7 +160,7 @@ export default function BotFilesModal({ bot, workspace, menu, on_open }: Props)
                     <Span className="py-6 text-[13px] leading-[18px] text-ink-secondary">
                         {loading
                             ? "Loading files..."
-                            : error || (search ? `Nothing matches "${search}".` : "No files yet.")}
+                            : error || (query ? `Nothing matches "${query}".` : "No files yet.")}
                     </Span>
                 )}
             </Stack>

src/pages/admin/bots/[bot_id]/(partials)/bot-message-line.tsx

@@ -65,7 +65,7 @@ function BotMessageLine({ message, bot_name, at }: Props) {
             ) : null}
             <Stack
                 {...{ [QuotableAttribute]: "" }}
-                className="w-full gap-2.5! items-start break-words"
+                className="w-full gap-2.5! items-start wrap-anywhere [&>*]:max-w-full"
             >
                 <Markdown
                     remarkPlugins={RemarkPlugins}

src/pages/admin/bots/[bot_id]/(partials)/bot-ref-chip.tsx

@@ -35,7 +35,7 @@ export default function BotRefChip({ bot_id }: Props) {
                     : `/admin/bots/${bot.id}`
             }
             title={`Open ${bot.name}`}
-            className="inline-flex h-6 gap-1.5 items-center px-1.5 align-middle rounded-default-xs border-0 bg-sunken text-[13px] font-bold leading-4 text-ink whitespace-nowrap hover:opacity-80"
+            className="inline-flex h-6 max-w-full gap-1.5 items-center px-1.5 align-middle rounded-default-xs border-0 bg-sunken text-[13px] font-bold leading-4 text-ink whitespace-nowrap hover:opacity-80"
         >
             <BotAvatar
                 image_media_id={bot.image_media_id}
@@ -43,9 +43,10 @@ export default function BotRefChip({ bot_id }: Props) {
                 face={(bot.face || "idle") as BotFace}
                 label={bot.name || ""}
                 size={18}
+                className="shrink-0"
             />
-            {bot.name}
-            {workspace_id && bot.is_admin ? <AdminShield size={13} /> : null}
+            <Span className="min-w-0 truncate">{bot.name}</Span>
+            {workspace_id && bot.is_admin ? <AdminShield size={13} className="shrink-0" /> : null}
         </Link>
     );
 }