Commit Graph
72 Commits
Author SHA1 Message Date
tbenandClaude Opus 5.5 e7b23b7454 Add knowledgebases
Knowledgebases give bots RAG over documents, scoped like extensions: to
particular bots, to a workspace, or to every bot, with their own button
beside Tools in the composer. Documents are chunked and embedded by the
file-ingest worker with the install's embedding model, which is now an
install-wide setting rather than a per-bot one.

- Word search runs on an FTS5 index ranked by BM25, with a scored LIKE
  fallback for scripts written without spaces.
- Changing the embedding model re-indexes documents automatically.
- Ingest jobs wait without spending attempts while every embedding model
  rests, fail at once on errors retrying can't fix, and keep passages
  searchable by words when the model refuses.
- Settings > Knowledgebases shows each knowledgebase's and document's
  index state: spinner, warning, grey check (words only) or green check.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-26 16:41:01 +01:00
tbenandClaude Opus 5.5 cc109bc361 Carry conversations into new sessions with a visible summary
- Bots get continue_in_new_session: the bot writes a title and summary,
  the session closes for good (read-only) and the new one opens with the
  summary as a "Carried over from the last session" message that links
  back to the old session.
- Continue in new session (conversation menu, sessions modal) and the
  automatic rollover at the session length do the same; New session now
  starts a fresh, unlinked session.
- Session length is an app setting in Settings > General.
- ?session_id= opens a session; a carried-over one shows a bar linking to
  where the conversation went on instead of the composer.
- The thread can open the previous session above the current one (shown
  only, never in the prompt), and replies get search_past_sessions and
  read_past_session.
- Sessions are named from their opening messages, not their latest, and
  a carried summary is labelled as the earlier session's.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-26 05:35:52 +01:00
tbenandClaude Opus 5.5 0e59559ea9 Keep every hook call's response, searchable by the user and the bot
A hook call used to record only how it was taken; its run's answer went
to the chat alone and was lost to the call once the job was pruned. Now
`bot_hook_events` keeps `response`, `response_message_id` and
`responded_at`, written on every call a run took in when that run ends
(completed, failed, lost, stopped or cancelled) by
`recordHookCallResponse` in the job helpers, and calls turn done or
failed. The Calls view searches what calls sent and what runs answered,
filters by status and pages; Show all loads one call whole. Bots get
`search_hook_calls` and `read_hook_call` for their own hooks (never in a
hook call's own run), with payloads always marked as outside data.
Calls are kept for the app setting `hook_events_keep_days` (Settings >
General, default now 30 days). The caller still only gets 202 Queued.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 17:47:28 +01:00
tbenandClaude Opus 5.5 fb4c89fca1 Tie each hook to a session, picked in the hook form
A hook's calls used to run in whatever session was active at the time.
Now `bot_hooks.session_id` pins them to one session, open or closed
(null still follows the active one). The hook form picks where it runs
(the bot's own chat or one of its workspaces) and a session there, or a
new one of its own started beside the active one so the chat on screen
stays. A bot's proposed hook gets the session it was proposed in, and a
missing session is replaced on the next call. Changing place or session
is refused while a run is open; the computer stays only if reachable.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 17:08:12 +01:00
tbenandClaude Opus 5.5 69988e08fc Move secrets between users and computers without the model seeing them
A new personal env scope (paradigm user: one user's secret for one bot)
beats every other scope, reaches only that user's runs of that bot, and
is redacted like any secret. Bots take secrets in with request_secret,
which puts a [secret-request:ID] card in the chat: the user types the
value into a masked input there, never into a message, and the bot is
told only the key. Bots show secrets with capture_secret (a task's
computer tool that saves a file or a command's output straight to a
personal env, then redacts it from the rest of the run) and show_secret
(an env by key): both give a [secret:ID] chip whose Reveal and Copy fetch
the value on click, for its own user (personal) or a super admin
(anything else), logged by key. Values never go into messages, page
props, history, Copy chat or logs. Users manage their own secrets on
Settings > Secrets; the Environment page lists everyone's, remove only.
Deleting a bot or user removes its personal secrets, and SecretsNote
tells bots never to ask for or print secrets in chat.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 16:51:32 +01:00
tbenandClaude Opus 5.5 380d3ebb2f Let bots manage their hooks, with the user confirming what widens them
Bots get list_hooks, pause_hook and delete_hook, which act directly since
they only take away. A new hook, or a new name, brief or filter, or
switching a paused one back on, goes through propose_hook: it stores a
hook_proposals row and puts a [hook-proposal:ID] card in the reply, which
the user it was proposed to confirms or rejects. The card is read from
what Confirm saves, with the bot's reason labelled as its own; confirming
checks everything again and saves through the Hooks modal's helpers. A
new hook's URL is shown on the card once, to the user only, never in the
outcome the bot is told. What calls a hook, its computers and its signing
are never proposed. Only chat replies propose, and a hook call's own run
gets no hook tools, so a caller's payload can't reach them.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 15:18:54 +01:00
tbenandClaude Opus 5.5 bd9744043e Let outside services and computers' browser notifications give bots work
A bot hook (bot_hooks) is a secret URL, POST /api/hooks/<token>, whose
calls queue the hook's brief plus the raw body as a task, per user and
bot like a routine. Only the token's hash is kept (shown once, rotatable);
an optional HMAC-SHA256 signing secret is stored encrypted. Calls are
rate limited per address and per hook, recorded in bot_hook_events
(pruned daily), deduplicated by delivery id, and a call while the last
run still waits joins its brief instead of queueing another. The payload
is fenced data and HookRunNote tells the bot never to take instructions
from it; failed runs retry like routines.

A hook can instead watch a computer's browser notifications: the app
installs a small watcher that owns org.freedesktop.Notifications on the
sandbox's session bus, so Chrome hands it every notification (it only
looks for that service at start-up, so Chrome is restarted once, never
mid-task), and it posts each one to its hooks, optionally filtered.
Computers reach the app at COMPUTER_APP_URL or the app container's name.

Users make hooks in the conversation's Hooks modal and manage them all
on /admin/routines; bots never create or change hooks.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 14:06:59 +01:00
tbenandClaude Opus 5.5 2d85177c59 Let crew managers edit a bot's workspace instructions where they work with it
A bot's instructions and memory for one workspace were only reachable
from the workspace's settings. The editor is now shared and also opens
from the conversation menu in that workspace ("Instructions here",
loaded on open through api/admin/workspaces/workspace-bot) and from the
workspace rows on the bot's settings page.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 13:17:32 +01:00
tbenandClaude Opus 5.5 84f8c0cac8 Let users copy the current chat session as markdown
The conversation header menu gets a Copy chat row that puts the active
session on the clipboard as one markdown document: a summary, then each
message under its author and time, with quotes, attachments and
reactions. Bot and workspace refs read as names, message headings drop
below the message headings and code blocks stay untouched. The clipboard
write starts on the click with a pending ClipboardItem so Safari allows
it after the fetch.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 11:49:46 +01:00
tbenandClaude Opus 5.5 93896646c7 Let the coordinator and workspace admins hand over tasks and routines
delegate_task takes a task_id to hand over a task already on a list: it
moves to the worker (a workspace's task stays on its list), goes to
doing, and the job is linked back so its end is noted on the task.
hand_over_schedule moves a bot's own routine to another bot, keeping its
workspace. Users get Hand over in the Tasks UI and both routine lists.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 11:23:26 +01:00
tbenandClaude Opus 5.5 e462806438 Let each install pick its app icon
Settings > General gets an App icon (super admin): one of the seven
stones, or an uploaded image with the picked stone as a top-left corner
mark (inside a round launcher's crop, clear of the browser's badge),
previewed in both launcher crops before saving.

Uploads are decoded and cropped to a stored square (no SVG); the set is
drawn into DATA_DIR/install-icon/<version>/ and served publicly and
immutable. The manifest, head icon links and push notifications point at
it; the ember stone keeps the stock icons. The generator shares the icon
renderer.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 10:38:25 +01:00
tbenandClaude Opus 5.5 3b7c05bfa0 Name each install, fix the notification badge, switch to Archivo
- Settings > General: an App name (super admin) for the installed app.
  The manifest's name/short_name, the iOS head tags and the loading
  page's title use it, so two installs on one phone can be told apart.
- Notifications get a real badge (white stone silhouette): Android masks
  the badge by alpha, so the opaque app icon showed as a square.
- The loading and offline pages load the app's font.
- The font is now Archivo (variable, one request for every weight),
  matching the marketing site.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 10:17:40 +01:00
tbenandClaude Opus 5.5 373210c2a5 Make the skill runbook limits a general setting
The longest runbook a skill may save and how much of it load_skill hands
back are now set on Settings > General (skill_body_max_chars,
skill_body_shown_chars), defaulting to AppData.SkillBodyMaxChars and
SkillBodyShownChars. Saving, confirming and proposing a skill check the
install's limit; a skill saved before it was lowered can still be toggled.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 09:43:00 +01:00
tbenandClaude Opus 5.5 577523de3c Only ignore the root logs folder
The bare "logs" pattern ignored every folder named logs, so
api/admin/logs/clear-logs.ts never reached the repository or the
production build. Anchor it to the root and add the route.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 09:24:55 +01:00
tbenandClaude Opus 5.5 b9a49ee09c Fix data archives and restores failing on their own lock
startDataBackupRun already holds the backup run lock, but the archive
(createDataBackup) and restoreDataBackup took it again, so every run
failed a few seconds in with "Another backup or restore is already
running". The writer and the restore now take no lock of their own.

The run's notice (archiving, restoring, or why it failed) moves to its
own line under the actions row so the buttons don't shift, and archiving
reads in gray rather than the warning colour.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 09:24:55 +01:00
tbenandClaude Opus 5.5 ea62dd11f0 Add bot types with a permission matrix, and workspace admin bots
A bot's powers came from bots.kind alone, so only the coordinator could
hand work to other bots. bot_types now works like user_types: a row with
no workspace holds everywhere (the coordinator), one with a workspace
only there (a workspace admin), and no row is a worker. bots.kind stays
the legacy marker of the one coordinator for who sees it.

The matrices in src/dict/bot-types-dict.ts say what each type may do,
whom it may direct and who grants it, read by one generic interpreter
(rbac/type-assignment.ts) that userTypeAssignment now wraps too. Every
run reads its powers once (grabBotRunPowers) and the chat tools, task
runs and task tools gate on them instead of kind; a run's powers are its
type's and its user's together.

In the workspace it leads, an admin hands tasks to the workers there
(results relayed back to it), sets their instructions, adds tasks to
their lists and keeps the workspace's context when its user may. Super
admins and admins grant it from the workspace's settings or the chat
header's menu, and the coordinator with set_bot_type. Admins are marked
with a filled green shield beside their name across the workspace, kept
live by pushBotUpdated's admin_workspace_ids.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 09:04:14 +01:00
tbenandClaude Opus 5.5 4161863587 Run whole-data archives and restores in the background
A data archive or restore ran inside its request, which sent nothing
until tar finished: Bun closes a silent socket after 10s (its default
idleTimeout in production) and a proxy cuts it at about 100s, so the
page showed a network error while the archive kept going, and a retry
said another backup was already running.

The routes now start the run (startDataBackupRun, refused while the
backup lock is held) and answer at once. The page follows it by polling
api/admin/backups/data-status: an archive reloads once written, a
restore once the status comes from a new process after its restart,
and a failure shows its error. A run in progress survives a reload.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 08:34:29 +01:00
tbenandClaude Opus 5.5 fefc295d15 Let the coordinator manage skills, MCP servers and tools, behind a confirm card
- Coordinator tools in its own chat (super admin only): list_extensions,
  set_extension_access, delete_extension, and update_skill /
  update_mcp_server / update_custom_tool for what can't change where
  anything is sent (on/off, names, arguments, left-out tools)
- Adding a skill, MCP server or tool, or changing a skill's words, a
  server's URL or headers or a tool's kind or config, goes through
  propose_skill / propose_mcp_server / propose_custom_tool: stored in
  extension_proposals with the bot's background check, shown as a
  [proposal:ID] chip that opens a card with Confirm and Reject
- The card's facts are worked out by the app from what Confirm saves: the
  host, every {{env.KEY}} and whether it's saved, the config; for a skill
  its links, hosts, tool names, mentions of secrets and a diff against
  the saved runbook
- Confirming decides once and checks again; a server or tool missing a key
  is saved switched off, and the outcome is posted to the bot in chat
- Never from a bot: stdio servers, a URL whose host a template decides,
  envs in a prompt tool, a key's value

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 06:08:09 +01:00
tbenandClaude Opus 5.5 e6c278198b Quiet the logs and capture the errors that were missing
- Activity page: "Delete all" empties the open tab (api/admin/logs/clear-logs,
  super admin only, logged under system)
- Stop logging every WebSocket connect
- Log unexpected API errors (safeErrorMessage -> logUnexpectedError), every
  failed bot job attempt and every job the queue lost
- Drop the cron jobs' console output; a failed model sync goes to the logs

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 05:11:31 +01:00
tbenandClaude Opus 5.5 c13b9ea826 Show how each bot message was generated
Every bot message gets an info button that opens its details: the
provider and model behind it, tokens and cost, the job that ran it, each
model used and every call in order. Each model call now records how it
was made in token_usages.settings (generation settings sent, whose
settings they were, pinned or automatic model, fallbacks), shown per
model in the modal.

Live replies now carry their token usage, from the job's push and the
live relay alike, so it shows without a reload.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 04:33:47 +01:00
tbenandClaude Opus 5.5 d542b4e9ad Load a conversation's history on the client
The conversation pages rendered their messages on the server, so a page
the back button restored from the back-forward cache showed an old
thread. The server now renders everything else (rail, header, composer)
and the history (active session, latest messages, active task) loads
from api/admin/bots/conversation-history once the page is on screen,
with a loading line under the header, and again on a pageshow restore.
Messages sent or pushed while it loads are merged with it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 03:55:38 +01:00
tbenandClaude Opus 5.5 cdb93ac1bc Fix file access, XSS and access-control bugs from a security scan
- Media rows can no longer be created through the CRUD API, updates only
  take description fields on the caller's own row, and every media read
  or delete goes through mediaFilePath, which refuses paths outside
  DATA_DIR or inside the database folder.
- Uploads strip the file name and extension to safe characters and keep
  only title, description and summary from the request's media object.
- Folder imports no longer follow symlinks (walkRegularFiles), and the
  source path is quoted in the computer's shell.
- User and bot files are served with nosniff, and HTML, SVG, XML and
  scripts are sandboxed by CSP.
- Sandbox cookies are kept to the computer's prefix, and websocket
  upgrades on /ws and the computer proxy must come from the app's own
  address (README and Help note nginx's Host header).
- user_types CRUD writes go through the assignment matrix; members only
  read their own user record; read-only users can read CRUD again;
  grabUsers builds valid SQL for array search terms and bad pages; an
  admin media DELETE needs an id; read-only users can't delete media.
- Rate limits and the audit log use the connection's address, trusting
  forwarded headers only from a proxy, and login limits are split per
  address, per account from that address and per account overall.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-25 03:38:07 +01:00
tben 024d847b9a Let the install set how many steps a run may take
The two limits were compiled in: a task run stopped at
AppData.HarnessMaxToolSteps (200) and a chat reply at
AppData.BotReplyMaxToolSteps (8), so an install that needed a longer task
had to rebuild.

Both are now `app_settings` rows (`harness_max_tool_steps` and
`bot_reply_max_tool_steps`), read at the two loop call sites through
grabHarnessMaxToolSteps and grabReplyMaxToolSteps, and edited by the super
admin on Settings > General beside the prompt appendix. Zero clears a
limit back to the code default, like the bot file quota.

readStepLimit keeps the saved number only when it is a whole number of at
least 1 and no more than AppData.BotStepLimitMax (1000), so a blank, a
zero or a typo falls back instead of stopping every run before its first
turn, and a huge number still ends in the wrap-up report.

runTextLoop keeps AppData.HarnessMaxToolSteps as its default, so it stays
free of database reads and its tests run as they did.
2026-10-23 16:20:38 +01:00
tben d9aea8e9c8 Record what every model call spends, and show it
Nothing anywhere recorded what a model call cost: the harness used a
provider's usage numbers only to decide whether a turn had finished, then
dropped them. So a reply could not say what it took to make, one model
could not be compared with another on price, and there was no way to ask
what a day, a week or a month of the crew's work came to.

- `token_usages` records one row per call: user, bot, workspace, session,
  job and message, the harness, the provider and model that actually
  answered, the prompt and completion counts, and the cost when the
  provider reports one.
- `harness-usage-scope.ts` carries those ids in an AsyncLocalStorage
  scope that `run-bot-job.ts` enters once, so a run's turns, its session
  summary, its transcriptions, images and embeddings are all attributed
  without threading ids through every harness function. Nested runs merge
  rather than replace, and a null genuinely clears a field (a bot's own
  chat has no workspace).
- Work with no run behind it is charged to its owner instead:
  `run-with-fallback-scope.ts` reaches the workspace's creator else the
  bot's, and deliberately does not override a real run, so an ingest a
  bot asked for is not misattributed to the bot's owner.
- All six harnesses record through `withCandidates`'s `on_success`.
  Recording never throws: a model that answered must not lose its answer
  to a failed insert.
- A job's calls land before its message exists, so `complete-bot-job.ts`
  back-fills `message_id` once the message is inserted.
- The message's actions row shows what it cost, beside the timestamp and
  the play button, with the breakdown in its title.
- `/admin/usage` totals the install's spending for 7, 30 or 90 days: the
  total against the previous period, a day-by-day chart, a split per kind
  of call, and a table per model with the price per thousand tokens
  worked out from cost and tokens, which is what makes two models
  comparable. Crew managers only, since it is the whole install's work
  rather than one user's conversation. It sits with Routines and Activity
  in the user popover rather than under Settings, because it is a record
  of what happened, not something to configure.

Days are grouped in local time (`date(created_at / 1000, 'unixepoch',
'localtime')`), matching the local midnights the window builds and the
`YYYY-MM-DD` buckets it looks up. Grouping by the bare UTC day put a late
evening's calls under a name no bucket carried, so they vanished and the
chart's total silently disagreed with the totals card; a test pins it.

`canSeeRailItem` now decides who sees a popover or rail entry. The two
each filtered with their own inline rule and only one understood
`can_view`, so a gated page could be offered in the popover to everyone.
2026-10-23 15:41:41 +01:00
tben fcad4b49d6 Let a user stop a bot, and say the app is still alpha
The thinking row and the header menu both carry Stop while a bot has work
open for the user anywhere: every pending or processing job of that bot in
their conversations is failed as stopped, its task closed, a Stopped step
recorded and one closing message posted per session, so the thread stops
waiting and the bot goes idle. The handler running the job is aborted from
this process through a registry, so a job stuck waiting on a model stops at
once instead of on its next heartbeat, and a late face change after the
abort is dropped.

The release stage is now said in the app too: Settings > General reports
"Cobble v0.1.5 (alpha)" next to development, and the README and RELEASING.md
say what alpha means for installs.
2026-10-23 10:19:41 +01:00
tben 5beb5f3de4 Let a conversation's menu show the routines that run in it
The header menu had Tasks but nothing for the schedules whose results
land in that conversation. It now has Schedules, right under Tasks,
opened in a modal that lists only what runs there: a bot's own chat holds
the routines that run nowhere, a workspace only that bot's routines in
it (`routineScopeSql`, tested against SQLite).

The list loads when the modal opens, through
`api/admin/bots/list-routines`, which checks the user chats in that
conversation first, so the page props stay small. `grabCronJobs` gained
an optional `workspace_id`: left out it still means everywhere.

The routines page's row, form and edit modal moved to
`src/components/general/`, and its subtitle to `src/utils/`, since both
the page and the modal use them.
2026-10-23 07:23:15 +01:00
tben 41bb4aaac2 Let a super admin read the database from the app
A new super admin settings page, /admin/settings/database, lists every
table with its record count and size, and one page of the open table's
records newest key first. Values are cut inside SQL (400 characters, with
a flag saying so) and a record opens whole in a modal, so a table of long
texts reads without shipping it all. A search matches every column's
text, word by word or as a phrase, which is the one thing that also works
on a virtual table. Nothing in it writes: the catalog, the records and
the single record are all reads, GET only.
2026-10-23 06:03:42 +01:00
tben f2802c1bf3 Let bots set a picture from a file they can see
A coordinator asked to set an uploaded logo as a workspace picture could
not: the picture tools take a media id, and nothing ever gave the bot one.
Attachments reached the model as "[Attached image \"logo.png\"]", every
file tool answers with a [file:ID] ref, and an SVG is deliberately stored
as a document rather than an image.

- copyMediaAsPicture copies an approved image into a new picture row
  (bot-image/workspace-image). The source keeps its row and bytes, so a
  message that attached it still loads and changing a picture can never
  take a file away from a conversation.
- grabPictureSource resolves a picture by [file:ID] (the same scope gate
  every file tool uses) or by the user's own upload; parseFileRef reads
  the [file:12] form.
- The new set_picture tool ships on every run: a bot sets its own face,
  the coordinator can also dress a worker. update_workspace gained
  picture_file_id, and setBotPicture/setProfileImage share the copy and
  the delete.
- removeReplacedPicture deletes a replaced picture only when it really is
  one, never a chat upload.
- uploadAndRecordMedia writes images in their own format: PNG stays PNG
  with its transparency, WebP stays WebP, GIF becomes PNG, everything
  else JPEG, and the row records the real mime type (it used to say jpeg
  for every image).
- Pictures change live: BotUpdatedData carries image_media_id and a new
  server:workspace-updated pushes workspace pictures, relayed between
  processes.
- Attachments are labelled with their media id, and BotFilesNote tells a
  bot that a [file:ID] can become a picture.
- grabRunScopes is now the single source of a run's reachable folders,
  shared by the file tools, imports and the picture tools.
2026-10-23 05:24:55 +01:00
tben d3357cb7df Stop storing raw file bytes in media.text_content
`text_content` is an extracted-text cache, but nothing stopped it from
holding a dump of bytes:

- `fileToText` ended its switch with `default: return buffer.toString("utf-8")`
  and defaulted a missing extension to "txt", so any binary was decoded.
- Both extraction callers gated on `media_type == "document"`, and
  `attachmentFileKind` classes everything non-image/audio/video as a
  document: an SVG, a zip, a spreadsheet, a .bin.
- Nothing capped the length, so a 20 MB attachment could leave 20 MB of
  text in the database.
- `storeChatAttachments` copied the text onto a second `media` row, and
  `SELECT m.*` pulled the column into page props and API responses.

On a real install an SVG held 6852 characters of markup and a
`.designcraft` file held 142 KB of raw bytes.

Changes:

- `file-to-text.ts` lists the text extensions explicitly and returns
  undefined for anything else; no extension is no text.
- `can-extract-text.ts` is the one allowlist (reusing
  `BotTextFileExtensions`), used by both upload routes and by
  `uploadAndRecordMedia`; `extension-of.ts` is shared with the storage
  catalog instead of duplicated.
- `BotAttachmentTextStoreMaxChars` (200k) caps the column via
  `cap-text-content.ts`.
- `store-chat-attachments.ts` no longer copies the text; ingestion
  re-reads the bytes once.
- `media-columns.ts` selects every media column but `text_content`, which
  now has to be asked for (`grab-messages-media`'s reply path and
  ingestion do); the media CRUD routes sanitize rows they write.
- `ingest-bot-file.ts` decides on size alone, since a cached slice made
  an oversized file look already read.
- `bun run media:prune-text` cleans rows that were already stored
  (rerunnable, with a VACUUM hint).

Verified on the dev database: both bad rows cleared (146 KB), rerun
reports nothing to do.
2026-10-23 05:22:04 +01:00
tben 5475241371 Move the notification settings onto the general settings page 2026-10-23 04:11:21 +01:00
tben 08d02e6033 Give bots skills: runbooks they load when a job matches
A skill is a runbook: a name, a unique slug, a one-line description of when
it applies, and the markdown body. A run only ever sees the slug and
description (one load_skill tool carries them), and reads the whole body
only when it calls the tool, so a long runbook costs nothing until needed.

Skills are the third extension kind beside MCP servers and custom tools, on
the same botExtensionsSql path, so per-bot, per-workspace and all-bots
grants work the same way, and attachDefaultExtensions hands an every-bot
skill to new bots. load_skill is a reserved tool name, so a custom tool
cannot shadow it, and a built-in wins a clash.

Managed on Settings > Skills; ExtensionRow links each kind to its own page
and a skill never reports a failure, since nothing connects or runs.
2026-10-22 12:31:42 +01:00
tben 1a6249afd3 Let a bot take any colour
A bot's identity was one of seven hues, and the hue picked both its colour
and its stone shape. Now its colour is free and its stone is its own:

- `bots.color` holds any `#rrggbb`, `bots.stone` the flat-cut shape it's
  drawn on. `bots.hue` stays as the palette column they replaced, read only
  as a fallback; `src/scripts/move-bot-colour-to-owners.ts` backfills an
  install from it (safe to rerun).
- `src/utils/bot-color.ts` is the only place a colour is validated or
  resolved (`resolveBotColor`/`resolveBotStone` carry the old hue forward,
  `botColorNeedsPaperFace` flips the eyes to paper on a very dark body), and
  `src/utils/bot-stone-props.ts` gives every row its props, so a row saved
  before this still renders as it did.
- `BotStone` takes the colour as the body's SVG `fill`, the one documented
  exception to the token rule: an arbitrary hex has no token to point at.
  `UnreadDot` follows it.
- The picker keeps the palette tiles (still shape and colour together) and
  adds a native swatch and a hex field, so a custom colour keeps its stone.
  Same place as before: the header avatar's modal and Settings > Bots, for
  crew managers. The coordinator stays hue-less.
- A recolour reaches other open tabs through `pushBotUpdated`.
2026-10-22 11:55:36 +01:00
tben a8d8b680ad Append install-wide instructions to every bot's prompt
Settings > General gains a prompt appendix the super admin sets once:
markdown every bot runs with, right after its own core instructions.

- The text lives in app_settings (bot_system_prompt_append, read through
  grabSystemPromptAppend, cached like every app setting), so it changes
  without a deploy and applies to every bot on the next run.
- composeBotPrompt takes an appendix and pushes it as
  # Install-wide instructions directly after system_prompt, before the
  workspace block and memory, so the bot's own more specific rules still
  win. grabBotPrompt feeds it in, so chat replies, tasks, scheduled runs
  and the coordinator all get it with no call site touched.
- api/admin/settings/system-prompt-append saves it (super admin only,
  capped at AppData.PromptMaxLength, empty clears it) and is audit
  logged under system.
- The section and its form are hidden from everyone but the super admin,
  save in place with a character count and no reload, and say what an
  empty text does.
2026-10-22 11:12:38 +01:00
tben a88898d169 Archive the whole data volume in Backups, not just the database
Settings > Backups gains a whole-data archive beside the database
snapshots, and both lists now fully work from the page.

- Data archives: `write-data-backup-archive.ts` tars all of DATA_DIR into
  BACKUP_ARCHIVE_DIR as cobble-data-<epoch-ms>.tar.gz, every entry under one
  root folder named after the archive, writing a .part name and renaming on
  success so a cut-off run is never listed. The newest
  AppData.DataBackupsKept are kept; db/.backups is left as it is.
- Restore: `restore-data-backup.ts` extracts to a temp dir, requires one
  root folder holding a db folder, takes a safety archive first, then
  replaces DATA_DIR (the database snapshots included) and `restart-app.ts`
  restarts the process through the docker socket, since the open SQLite
  handle points at the replaced file.
- Upload and restore: a modal over each list (database file or data
  archive), replacing the old /admin/settings/backups/restore page.
- Delete for both kinds, icon-only row actions shared through
  RowIconButton, and restore on every row.
- One backup or restore of either kind at a time (with-backup-run-lock).
- BACKUP_ARCHIVE_DIR is a start-up path (ProcessOnlyEnvKeys); the installer
  mounts $COBBLE_HOME/backups-archives at /backups-archives and recreates a
  container that lacks it. The image installs tar and gzip.
2026-10-22 11:00:37 +01:00
tben fd75111bb1 Open a folder chip on its folder, in the folder that really holds it
A bot's folder chip went to bare /admin/files instead of the folder it
named, and even once there the computer volume pointed at the wrong
directory, so the folder a bot linked to looked empty.

The chip lost its folder because folderRefUrl only accepted a ref's
volume name (computer) while the chip hands it the browser's
(computers), so storageVolumeFromRef found nothing and the whole query
was dropped. Both spellings resolve now, through parseStorageSpec, so a
bad owner falls back cleanly instead of emitting owner=0.

The deeper bug: a computer's home was assumed to be
DATA_DIR/computers/<id>, but on a bind mount it can be anywhere on the
docker host, and a computer the app didn't provision (the main one on a
server reached at MAIN_COMPUTER_URL, i.e. a dev server outside docker)
records nothing at all. resolveComputerHomeDir holds the precedence:
the directory provisionComputer actually bound in
(containers.metadata.data_dir), else the new MAIN_COMPUTER_DATA_DIR app
setting for the main computer, else the conventional folder.

A path that isn't on disk also no longer reads as an empty folder:
listStorageEntries reports it as missing and the page says so, so a
stale folder link can't look like a folder nothing has written yet.
2026-10-22 10:57:14 +01:00
tben 841149921b Refresh the session cookie after a profile save 2026-10-22 10:21:58 +01:00
tben aebd6b8ccc Serve every media file privately at /media/<id>
There is no public media any more. `/media/public/<id>` served a media
row with no session at all, so anyone with the id could read a file.
The public/private split is gone: the path is `/media/<id>` (and
`/media/<id>/thumbnail`), every request is signed in first, and
`grabUserMedia` decides whether the user may see it: their own upload,
a bot's or workspace's picture or file they reach, or a file on a task
they reach. Anything else is a 404 before a path is built.

The `media.is_private` column and the two per-user directories went
with it: uploads live in `media/<user_id>/` and there is nothing to
pick between. The files browser's media volume is now the viewer's
own folder whatever their role.

Bots also say which of the two things they mean. A `[file:ID]` ref is a
saved file and its chip opens the bytes; a folder is a new
`[folder:volume:owner:path]` ref (`src/utils/folder-ref.ts`) whose chip
opens the files browser there and downloads nothing. The old `[path:...]`
ref and the extension sniffing that guessed file from folder are gone,
so `import_file` names the folder holding a file it copied rather than
the file itself.

The prompts say what the Assistant got wrong: list before importing or
saving, one copy per artifact, never split a file into parts unless
asked, never claim the store is empty without a fresh list, and never
invent a size cap. A re-import now says the path already existed
instead of silently overwriting and reporting a fresh copy.
2026-10-22 10:10:53 +01:00
tben 96735de48f Let a bot and a workspace set their own envs
Envs had three scopes, general, workspace and computer. A bot had no
scope of its own, so the only way to give one worker a key was to set it
on a computer it happened to share.

The envs table gets a bot_id, and resolution now layers bot over
computer over workspace over general, so a bot's own key wins wherever it
runs. Every computer call site passes the run's bot_id, and grabRunEnvs
does the same for the {{env.KEY}} a custom tool or an MCP server's headers
read.

A bot's and a workspace's settings page can now edit their own envs, in
place, through the same save-env and delete-env routes: sending `scoped`
makes them answer with just that scope instead of the whole install's
overview. Super admins only, like the Environment page.
2026-10-22 08:30:40 +01:00
tben 956942bba9 Add a files browser for every folder on the data volume
A bot's deliverables and the raw material behind them only existed as
paths in a message, so they couldn't be opened from the app at all. This
browses them read-only at /admin/files, one volume at a time: a bot's own
folder, a workspace's shared one, a computer's home (where a bot's ~/...
actually is), the media library, an MCP server's install folder and the
database.

Access is the whole gate: resolveStorageSpec checks the viewer against
the volume first (a bot through grabUserBot, a workspace through
grabUserWorkspace, a computer through grabUserComputer, media only their
own unless they manage the crew) and no path is built before it passes.
resolveStoragePath then applies the resolveBotPath rules to any root.
The picker lists folders through reachableBotsSql, so it never offers one
the resolver would refuse.

Reading is listStorageEntries (folders first, paginated, symlinks
skipped, catalog rows joined so a bot file still opens through its media
row) and grabStorageFile for bytes under a cap, both behind an ETag.
2026-10-22 07:37:16 +01:00
tben 7cb26c4e36 Add tasks: the crew's shared to-do list
Bots and workspaces each keep a list of work tracked until it's done, and
the bots move it along themselves as they work.

- tasks (one collection per bot or per workspace), task_notes (the
  progress log) and task_media (a deliverable or a reference file, its
  bytes a private media row served at /media/private/<id>)
- statuses todo/doing/blocked/done/cancelled, with blocked_reason while
  only the user can unblock it
- visibility as SQL in src/utils/task-access: a user's own bots and
  workspaces, a run's own list plus its workspaces, everything for the
  coordinator; every tool and endpoint resolves through it
- list_tasks, create_task and update_task on every run, so a chat reply,
  a task and a routine all read the list, log progress and file what they
  produced
- TasksNote on every run and TaskReminderNote in chat, so a greeting or a
  proposed next step brings pending work up in one measured line
- tasks group on the workspace dashboard and a tasks modal in the
  conversation header, sharing one form, row and detail panel
- a bot's or workspace's own chat uploads are read before the delete so
  none are stranded; bot files keep their folder

Also corrects the README's RAM guidance and table alignment.
2026-10-22 05:57:50 +01:00
tben c2553a4746 Add per-model generation settings to the model selector
Each text and vision harness setting now carries the generation settings
its model takes (temperature, sampling limits, penalties, seed, reasoning
effort), edited in a modal under the model pick so the harness block stays
short. Nothing is set by default, so models keep their provider's own
values until a super admin or admin changes them.

A setting is only offered and only sent when the catalog says the model
takes it: OpenRouter's supported_parameters is now stored on
harness_models and refreshed by the hourly sync. Other providers list no
parameters, so their models get them all.
2026-10-22 04:52:55 +01:00
tben d17a4a7bdd Add bots files handling logic 2026-10-21 17:56:09 +01:00
tbenandClaude Opus 5.5 f13fa863c5 Trim page props and page the conversation's messages
Bot and workspace lists in page props leave out instructions, memory and
workspace context, and carry the last message as a short preview. The
header avatar's edit form loads the bot's instructions when it opens.
The conversation renders its latest 30 messages and loads older ones as
the thread scrolls to its top.

Also: the composer shows a placeholder until its editor loads, and
`bun start` no longer builds first.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-20 20:09:01 +01:00
tben 4f7e0a6bf0 Finish the mobile performance pass
- Cache the model catalog once per user and each conversation's setting
  separately; clear both on logout and after model saves
- Compress hashed assets asynchronously with a byte budget and set Vary
- Memoize message parsing and throttle selection reads
- Debounce draft writes to localStorage
- Keep the composer editor on the page and focused, as before
- Add a bundle budget check and run it in CI
- Show a loading block while sessions load and a workspace avatar in the
  mobile conversation header
2026-10-20 14:37:11 +01:00
tbenandClaude Opus 5.5 c1a94fc9a5 Make pages load fast on mobile
- Compress responses: page scripts and styles once (brotli or gzip, in
  memory) and cached immutable; pages and API JSON gzipped as they stream
- Service worker serves content-named assets cache-first
- Load the models modal's data when it opens (cached in localStorage)
  instead of shipping the model catalog in every conversation page
- Lazy-load the Tiptap composer and memoize thread messages
- Import lucide icons directly; twui 0.1.4 drops by-name icons from Button
- bun run start builds the pages minified before serving

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-20 13:41:45 +01:00
tben e2093b0df6 Update frontend UI/UX 2026-10-20 04:59:08 +01:00
tbenandClaude Opus 5.5 100ef0eb89 Only offer reply and reactions on bot messages
The user's own messages no longer get the reply and react row. A bot's
reaction to one still shows, read-only, and react-message refuses a
reaction on the user's own message.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-19 22:04:29 +01:00
tbenandClaude Opus 5.5 d1b5b6d68a Add replies and emoji reactions to chat messages
Every message has a row under it to reply or react. Replying quotes the
message above the composer, and the sent message stores
reply_to_message_id (a plain id, not a foreign key, so deleting the
quoted message only drops the quote). send-message accepts a quote from
any session of the same conversation, so a rollover or a thread that
moved on still sends; the conversation drops a picked quote when its
session changes. The bot reads the quote as "[Replying to ...]" before
the user's turn, file-only replies included, and relayed messages carry
it too.

Reactions are a fixed set (BotMessageReactions) in
bot_message_reactions, one per message, emoji and user or bot. Users
toggle their own through api/admin/bots/react-message, and every bot
gets react_to_message for the session it answers (the user's latest
message by default). Both push the fresh list as server:bot-reaction,
with mine worked out for the session's user.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-19 21:56:31 +01:00
tbenandClaude Opus 5.5 bc499a7466 Let admins hear a voice sample before picking it
The Speech section of every model settings form now has "Hear a
sample", which streams a short sentence (AppData.VoicePreviewText) in
the form's model and voice before they're saved, through
GET api/admin/harness/preview-voice (super admins and admins, that
model only via grabModelCandidate). pickVoice is shared with
speakText, and explainAudioError reads why a streamed player failed
for both read-aloud and the sample.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-19 21:32:49 +01:00
tbenandClaude Opus 5.5 163d9a434c Add dictation and streamed read-aloud to the chat
- A mic in the composer records, transcribes with the bot's
  transcription model (api/admin/bots/transcribe) and writes the text
  into the input at the cursor to edit before sending. Escape cancels;
  recordings aren't stored.
- Every bot message has a read-aloud button that plays
  GET api/admin/bots/speak-message, streamed from the provider as the
  audio is made, so it starts in a couple of seconds. One message plays
  at a time; only the user's own sessions can be read.
- Both buttons stay visible but disabled while the bot has no model for
  them (bot_voice page prop).
- Transcription and speech no longer pick free models automatically:
  they're not available until a model is picked, and Settings says so.
  Transcription lists speech-to-text models only, so the chat-model
  transcript path and its missing-audio check are gone.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-19 21:21:04 +01:00