Moduletrace/cobble
2
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases 9 Wiki Activity

Compare commits

base: Moduletrace/cobble:v0.1.4
Branches Tags
Moduletrace/cobble:main
Moduletrace/cobble:v0.2.6 Moduletrace/cobble:v0.2.5 Moduletrace/cobble:v0.2.2 Moduletrace/cobble:v0.2.0 Moduletrace/cobble:v0.1.8 Moduletrace/cobble:v0.1.7 Moduletrace/cobble:v0.1.6 Moduletrace/cobble:v0.1.5 Moduletrace/cobble:v0.1.4 Moduletrace/cobble:v0.1.3 Moduletrace/cobble:v0.1.2 Moduletrace/cobble:v0.1.1 Moduletrace/cobble:v0.1.0
...
compare: Moduletrace/cobble:v0.1.7
Branches Tags
Moduletrace/cobble:main
Moduletrace/cobble:v0.2.6 Moduletrace/cobble:v0.2.5 Moduletrace/cobble:v0.2.2 Moduletrace/cobble:v0.2.0 Moduletrace/cobble:v0.1.8 Moduletrace/cobble:v0.1.7 Moduletrace/cobble:v0.1.6 Moduletrace/cobble:v0.1.5 Moduletrace/cobble:v0.1.4 Moduletrace/cobble:v0.1.3 Moduletrace/cobble:v0.1.2 Moduletrace/cobble:v0.1.1 Moduletrace/cobble:v0.1.0
15 Commits
v0.1.4 ... v0.1.7
Author SHA1 Message Date
tben 0166f31cfc Release v0.1.7
CI / check (push) Canceled after 0s
Details
Release image / image (release) Canceled after 0s
Details
2026-10-23 16:50:51 +01:00
tben caa9ad87e4 Remove sticky breadcrumbs 2026-10-23 16:49:52 +01:00
tben 30a4df183c Give every admin page a breadcrumb trail 
The trail is twui's Breadcrumbs reading Home, the sections and the page
off the address, rendered as the first thing in SettingsShell's content
column, so every page that is not a conversation gets one without a page
touching it.

The last crumb is named by the page's own title rather than the address:
a record reads Scout instead of its id, /admin/logs reads Activity, and
/admin/settings/mcps reads MCPs. That replaces the breadcrumb_title prop
the four settings record pages had to pass, so settings-breadcrumbs.tsx
goes.

hasBreadcrumbTrail drops the trail on the crew of bots, home, and on
anything outside /admin; the trailing slash and query string are ignored,
so a folder in the files browser keeps the trail of the page it is on.
 2026-10-23 16:49:01 +01:00
tben ee687e1d31 Chunk Tailwind classes with tailwind-merge 
Say it once in AGENTS.md: a className is twMerge(...) with one chunk per claim, so a later chunk beats an earlier one instead of depending on CSS source order.
 2026-10-23 16:42:01 +01:00
tben 024d847b9a Let the install set how many steps a run may take 
The two limits were compiled in: a task run stopped at
AppData.HarnessMaxToolSteps (200) and a chat reply at
AppData.BotReplyMaxToolSteps (8), so an install that needed a longer task
had to rebuild.

Both are now `app_settings` rows (`harness_max_tool_steps` and
`bot_reply_max_tool_steps`), read at the two loop call sites through
grabHarnessMaxToolSteps and grabReplyMaxToolSteps, and edited by the super
admin on Settings > General beside the prompt appendix. Zero clears a
limit back to the code default, like the bot file quota.

readStepLimit keeps the saved number only when it is a whole number of at
least 1 and no more than AppData.BotStepLimitMax (1000), so a blank, a
zero or a typo falls back instead of stopping every run before its first
turn, and a huge number still ends in the wrap-up report.

runTextLoop keeps AppData.HarnessMaxToolSteps as its default, so it stays
free of database reads and its tests run as they did.
 2026-10-23 16:20:38 +01:00
tben 06586a6dc8 Release v0.1.6
CI / check (push) Canceled after 0s
Details
Release image / image (release) Canceled after 0s
Details
2026-10-23 16:05:01 +01:00
tben 55fc5ee97b Follow the thread's bottom, and offer a jump back to it 
Sending a message while scrolled up left it out of sight: the thread only
ever anchored its bottom, so nothing brought the view down. The column now
follows new messages while it sits at the bottom, stays put while it's
scrolled up, and always brings a message the user just sent into view.
While the thread sits above its bottom, "Jump to latest" floats over it.

Whether the thread is at its bottom is read from where the content ends
rather than from `scrollTop`, whose sign a reversed column reports
differently across engines. Content that grows after it lands (an image,
a wider window) is followed too, so the newest message never slips under
the composer.
 2026-10-23 16:03:37 +01:00
tben d9aea8e9c8 Record what every model call spends, and show it 
Nothing anywhere recorded what a model call cost: the harness used a
provider's usage numbers only to decide whether a turn had finished, then
dropped them. So a reply could not say what it took to make, one model
could not be compared with another on price, and there was no way to ask
what a day, a week or a month of the crew's work came to.

- `token_usages` records one row per call: user, bot, workspace, session,
  job and message, the harness, the provider and model that actually
  answered, the prompt and completion counts, and the cost when the
  provider reports one.
- `harness-usage-scope.ts` carries those ids in an AsyncLocalStorage
  scope that `run-bot-job.ts` enters once, so a run's turns, its session
  summary, its transcriptions, images and embeddings are all attributed
  without threading ids through every harness function. Nested runs merge
  rather than replace, and a null genuinely clears a field (a bot's own
  chat has no workspace).
- Work with no run behind it is charged to its owner instead:
  `run-with-fallback-scope.ts` reaches the workspace's creator else the
  bot's, and deliberately does not override a real run, so an ingest a
  bot asked for is not misattributed to the bot's owner.
- All six harnesses record through `withCandidates`'s `on_success`.
  Recording never throws: a model that answered must not lose its answer
  to a failed insert.
- A job's calls land before its message exists, so `complete-bot-job.ts`
  back-fills `message_id` once the message is inserted.
- The message's actions row shows what it cost, beside the timestamp and
  the play button, with the breakdown in its title.
- `/admin/usage` totals the install's spending for 7, 30 or 90 days: the
  total against the previous period, a day-by-day chart, a split per kind
  of call, and a table per model with the price per thousand tokens
  worked out from cost and tokens, which is what makes two models
  comparable. Crew managers only, since it is the whole install's work
  rather than one user's conversation. It sits with Routines and Activity
  in the user popover rather than under Settings, because it is a record
  of what happened, not something to configure.

Days are grouped in local time (`date(created_at / 1000, 'unixepoch',
'localtime')`), matching the local midnights the window builds and the
`YYYY-MM-DD` buckets it looks up. Grouping by the bare UTC day put a late
evening's calls under a name no bucket carried, so they vanished and the
chart's total silently disagreed with the totals card; a test pins it.

`canSeeRailItem` now decides who sees a popover or rail entry. The two
each filtered with their own inline rule and only one understood
`can_view`, so a gated page could be offered in the popover to everyone.
 2026-10-23 15:41:41 +01:00
tben 0048324d8a Let a delegated task choose the workspace it runs in 
delegate_task had no workspace argument: a task handed over from the
coordinator's own chat always ran in the worker's own chat, on that
worker's memory, files and directly assigned computers alone, even when
the work belonged to a workspace. The coordinator could not say where the
work goes, and the prompt never said the two differ.

- delegate_task takes an optional workspace_id: the worker runs the task
  in that workspace's context, with its own instructions and memory
  there, the project's files and the workspace's computers, and the
  result lands in its conversation in that workspace.
- A worker missing from that workspace is added first (pushBotCreated)
  and the reply says so.
- Handed over from inside a workspace the task always runs there; a
  different workspace_id is refused instead of ignored.
- The coordinator prompt, CoordinatorReplyNote and the tool description
  now spell the distinction out.
 2026-10-23 14:25:21 +01:00
tben f7ef218507 Take the Gitea registry out of CI and mirror it separately
CI / check (push) Canceled after 0s
Details 
The Gitea registry is on a VPN, which a GitHub runner can never reach, so
the mirror step in the release workflow could only ever fail: its last four
releases reported a step that did nothing. The workflow now builds and
pushes to GHCR alone, which is the registry the installer defaults to, and
nothing in CI mentions Gitea.

The image reaches the Gitea registry through `scripts/mirror-gitea.sh`
(`bun run mirror:gitea`) instead, run in parallel with a release and from the
VPN side: it copies from the Gitea host over ssh, where skopeo can see both
registries, so nothing has to be installed there. `skopeo copy --all` keeps
the manifest list and its signatures, and every destination tag is read back
and compared with GHCR's digest, so a partial copy fails loudly instead of
being assumed to have worked. `latest` and `major.minor` are only moved while
that release still owns them in GHCR, so mirroring an older release never
rolls them back.

Neither direction can fail the other now, which is the point: a registry that
is down, slow or full no longer takes a release with it.
 2026-10-23 12:03:23 +01:00
tben aac5d41474 Release v0.1.5
CI / check (push) Canceled after 0s
Details
Release image / image (release) Canceled after 0s
Details
2026-10-23 10:21:06 +01:00
tben fcad4b49d6 Let a user stop a bot, and say the app is still alpha 
The thinking row and the header menu both carry Stop while a bot has work
open for the user anywhere: every pending or processing job of that bot in
their conversations is failed as stopped, its task closed, a Stopped step
recorded and one closing message posted per session, so the thread stops
waiting and the bot goes idle. The handler running the job is aborted from
this process through a registry, so a job stuck waiting on a model stops at
once instead of on its next heartbeat, and a late face change after the
abort is dropped.

The release stage is now said in the app too: Settings > General reports
"Cobble v0.1.5 (alpha)" next to development, and the README and RELEASING.md
say what alpha means for installs.
 2026-10-23 10:19:41 +01:00
tben 5beb5f3de4 Let a conversation's menu show the routines that run in it 
The header menu had Tasks but nothing for the schedules whose results
land in that conversation. It now has Schedules, right under Tasks,
opened in a modal that lists only what runs there: a bot's own chat holds
the routines that run nowhere, a workspace only that bot's routines in
it (`routineScopeSql`, tested against SQLite).

The list loads when the modal opens, through
`api/admin/bots/list-routines`, which checks the user chats in that
conversation first, so the page props stay small. `grabCronJobs` gained
an optional `workspace_id`: left out it still means everywhere.

The routines page's row, form and edit modal moved to
`src/components/general/`, and its subtitle to `src/utils/`, since both
the page and the modal use them.
 2026-10-23 07:23:15 +01:00
tben 796c53716f Show a mirror failure, and link the release's own range
CI / check (push) Canceled after 0s
Details
2026-10-23 06:54:06 +01:00
tben 5af2f0362a Fetch the tags in CI so the publish tests can read them 2026-10-23 06:13:14 +01:00
160 changed files with 6641 additions and 331 deletions
Expand all files Collapse all files
.github/workflows/ci.yml
+4
View File
@@ -16,7 +16,11 @@ jobs:
check: check:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
# The publish tests read real release tags out of the history, so a
# shallow, tagless clone makes them fail.
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: oven-sh/setup-bun@v2 - uses: oven-sh/setup-bun@v2
with: with:
.github/workflows/release.yml
+19 -40
View File
@@ -1,5 +1,12 @@
name: Release image name: Release image
# Builds and pushes the multi-arch image to GHCR, the registry the installer
# defaults to.
#
# The Gitea registry is on a VPN, which a GitHub runner can never reach, so CI
# does not touch it at all: `scripts/mirror-gitea.sh` copies a release into it
# from the VPN side, in parallel and after the fact. Keeping the two apart is
# what makes a registry outage unable to fail a release.
on: on:
release: release:
types: [published] types: [published]
@@ -30,13 +37,6 @@ jobs:
username: ${{ github.actor }} username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }} password: ${{ secrets.GITHUB_TOKEN }}
- name: Log in to Gitea
uses: docker/login-action@v3
with:
registry: git.tben.me
username: ${{ secrets.GITEA_USERNAME }}
password: ${{ secrets.GITEA_TOKEN }}
# The version the app reports on Settings > General when the build # The version the app reports on Settings > General when the build
# isn't a published release. # isn't a published release.
- name: Read the source version - name: Read the source version
@@ -45,6 +45,11 @@ jobs:
set -euo pipefail set -euo pipefail
echo "version=$(sed -n -E 's/^[[:space:]]*"version": "([^"]+)",?$/\1/p' package.json | head -n1)" >> "$GITHUB_OUTPUT" echo "version=$(sed -n -E 's/^[[:space:]]*"version": "([^"]+)",?$/\1/p' package.json | head -n1)" >> "$GITHUB_OUTPUT"
# The same four tags for every kind of build: a release gets its own
# version plus the moving `latest` and `major.minor`, so that
# scripts/mirror-gitea.sh can copy whichever of them still belong to
# it, and a dispatch build gets the commit tag only, since it is not a
# release and must not move `latest`.
- name: Work out image tags - name: Work out image tags
id: tags id: tags
env: env:
@@ -57,32 +62,24 @@ jobs:
if [ "$EVENT" = "release" ] && [ -n "$RELEASE_TAG" ]; then if [ "$EVENT" = "release" ] && [ -n "$RELEASE_TAG" ]; then
version="$RELEASE_TAG" version="$RELEASE_TAG"
minor="$(printf '%s' "$version" | sed -E 's/^v?([0-9]+)\.([0-9]+).*/\1.\2/')" minor="$(printf '%s' "$version" | sed -E 's/^v?([0-9]+)\.([0-9]+).*/\1.\2/')"
ghcr="$(printf '%s\n' \ tags="$(printf '%s\n' \
"ghcr.io/moduletrace/cobble:latest" \ "ghcr.io/moduletrace/cobble:latest" \
"ghcr.io/moduletrace/cobble:${version}" \ "ghcr.io/moduletrace/cobble:${version}" \
"ghcr.io/moduletrace/cobble:${minor}" \ "ghcr.io/moduletrace/cobble:${minor}" \
"ghcr.io/moduletrace/cobble:${sha}")" "ghcr.io/moduletrace/cobble:${sha}")"
gitea="$(printf '%s\n' \
"git.tben.me/moduletrace/cobble:latest" \
"git.tben.me/moduletrace/cobble:${version}" \
"git.tben.me/moduletrace/cobble:${minor}" \
"git.tben.me/moduletrace/cobble:${sha}")"
else else
version="$SOURCE_VERSION" version="$SOURCE_VERSION"
ghcr="ghcr.io/moduletrace/cobble:${sha}" tags="ghcr.io/moduletrace/cobble:${sha}"
gitea="git.tben.me/moduletrace/cobble:${sha}"
fi fi
{ {
echo "version=$version" echo "version=$version"
echo "ghcr<<EOF" echo "tags<<EOF"
echo "$ghcr" echo "$tags"
echo "EOF"
echo "gitea<<EOF"
echo "$gitea"
echo "EOF" echo "EOF"
} >> "$GITHUB_OUTPUT" } >> "$GITHUB_OUTPUT"
# The registry the installer defaults to, so this one must succeed. # The only push CI makes. The installer defaults to this registry, so
# this step failing has to fail the release.
- name: Build and push to GHCR - name: Build and push to GHCR
uses: docker/build-push-action@v6 uses: docker/build-push-action@v6
with: with:
@@ -92,26 +89,8 @@ jobs:
push: true push: true
build-args: | build-args: |
COBBLE_VERSION=${{ steps.tags.outputs.version }} COBBLE_VERSION=${{ steps.tags.outputs.version }}
tags: ${{ steps.tags.outputs.ghcr }} tags: ${{ steps.tags.outputs.tags }}
provenance: false provenance: false
sbom: false sbom: false
cache-from: type=gha cache-from: type=gha
cache-to: type=gha,mode=max cache-to: type=gha,mode=max
# The mirror's registry, from the image just built: no second build,
# and Cloudflare rate limiting git.tben.me (429/1015 on blob uploads)
# can never fail a release the installer depends on.
- name: Mirror to Gitea
continue-on-error: true
env:
GITEA_TAGS: ${{ steps.tags.outputs.gitea }}
GHCR_IMAGES: ${{ steps.tags.outputs.ghcr }}
run: |
set -euo pipefail
first_ghcr="$(printf '%s\n' "$GHCR_IMAGES" | head -n1)"
args=()
while IFS= read -r tag; do
[ -n "$tag" ] || continue
args+=(--tag "${tag}")
done <<< "$GITEA_TAGS"
docker buildx imagetools create "${args[@]}" "${first_ghcr}"
AGENTS.md
+23 -9
View File
@@ -105,14 +105,14 @@ Inspect live data with `bun llm-code-assistant/query-db.ts "<sql>"` - a read-onl
- **Composer:** a Tiptap editor (`(hooks)/use-composer-editor.ts`, `(partials)/composer-editor.tsx`), not a textarea, loaded with the page and focused automatically, that reads and writes markdown (`@tiptap/markdown`), so drafts and sent messages are markdown. Drafts are written to `localStorage` on a 250ms debounce (`useComposerDraft`), flushed on unmount. Bold, italic, strike, code, lists, quotes and links (no headings), from shortcuts, markdown typing or the selection bubble (`composer-format-bubble.tsx`, items in `(data)/composer-formats.ts`). `@` mentions a bot and `#` a workspace (`(data)/composer-mention-kinds.ts`, menu `composer-mention-list.tsx` above the input, items from `CrewBotsContext` and `CrewWorkspacesContext` via `searchMentionItems`), shown as chips (`composer-mention-chip.tsx`) and written as `[bot:ID]` / `[workspace:ID]` (`(utils)/mention-markdown.ts`, `src/utils/workspace-ref.ts`). Enter sends, Shift+Enter breaks the line; in a list or code block Enter is the editor's and Mod+Enter sends. Sending shows the bubble at once and empties the input (with its quote and files), so a slow connection never waits on the request: the optimistic message rides in `pending_messages` in `conversation.tsx` until the response or a push replaces it with the saved one (`(utils)/drop-sent-message.ts`), and a failed send takes it back and restores the text and files. Selecting text in a message (`data-quotable`, `use-message-selection`) shows a Quote button (`quote-selection-button.tsx`) that inserts it as a blockquote at the cursor (`ComposerHandle.insertQuote`). Before a reply, `grabMessageRefNames` names the refs in the user's messages that the user reaches, and `botMessagesToHarness` adds a `[Mentions: ...]` line after each one (`describeMessageRefs`). `@tiptap/react` needs `use-sync-external-store`, which is CommonJS and calls `require("react")`; bunext keeps React external in page bundles, so the app installs an ESM stand-in from `packages/use-sync-external-store-esm` (a `file:` dependency plus an override; the Dockerfile copies `packages/` before `bun install`). - **Composer:** a Tiptap editor (`(hooks)/use-composer-editor.ts`, `(partials)/composer-editor.tsx`), not a textarea, loaded with the page and focused automatically, that reads and writes markdown (`@tiptap/markdown`), so drafts and sent messages are markdown. Drafts are written to `localStorage` on a 250ms debounce (`useComposerDraft`), flushed on unmount. Bold, italic, strike, code, lists, quotes and links (no headings), from shortcuts, markdown typing or the selection bubble (`composer-format-bubble.tsx`, items in `(data)/composer-formats.ts`). `@` mentions a bot and `#` a workspace (`(data)/composer-mention-kinds.ts`, menu `composer-mention-list.tsx` above the input, items from `CrewBotsContext` and `CrewWorkspacesContext` via `searchMentionItems`), shown as chips (`composer-mention-chip.tsx`) and written as `[bot:ID]` / `[workspace:ID]` (`(utils)/mention-markdown.ts`, `src/utils/workspace-ref.ts`). Enter sends, Shift+Enter breaks the line; in a list or code block Enter is the editor's and Mod+Enter sends. Sending shows the bubble at once and empties the input (with its quote and files), so a slow connection never waits on the request: the optimistic message rides in `pending_messages` in `conversation.tsx` until the response or a push replaces it with the saved one (`(utils)/drop-sent-message.ts`), and a failed send takes it back and restores the text and files. Selecting text in a message (`data-quotable`, `use-message-selection`) shows a Quote button (`quote-selection-button.tsx`) that inserts it as a blockquote at the cursor (`ComposerHandle.insertQuote`). Before a reply, `grabMessageRefNames` names the refs in the user's messages that the user reaches, and `botMessagesToHarness` adds a `[Mentions: ...]` line after each one (`describeMessageRefs`). `@tiptap/react` needs `use-sync-external-store`, which is CommonJS and calls `require("react")`; bunext keeps React external in page bundles, so the app installs an ESM stand-in from `packages/use-sync-external-store-esm` (a `file:` dependency plus an override; the Dockerfile copies `packages/` before `bun install`).
- **Message rendering:** bot messages are markdown, rendered with `react-markdown` + `remark-gfm` (raw HTML never rendered) through the twui-based map in `[bot_id]/(data)/message-markdown-components.tsx` (`markdown-link`, `markdown-list`, `markdown-code-block` partials). A bot names a crew member as `[bot:ID]` (`src/utils/bot-ref.ts`, linked to `#bot-ID` by `link-computer-refs.ts`): `(partials)/bot-ref-chip.tsx` shows its avatar and name, linked to its conversation (in the open workspace when it's in the crew there), from the live crew (`CrewBotsContext`), so a bot created in that reply links too; the coordinator is told to write them (`CoordinatorReplyNote`), and `find_bot` and `create_bot` return them. A workspace is named as `[workspace:ID]` (`#workspace-ID`, `(partials)/workspace-ref-chip.tsx`, from `CrewWorkspacesContext`). User messages are markdown too, rendered in the ink bubble by `user-bubble.tsx` with `UserMarkdownComponents` and `remark-breaks` (a single line break stays one, so older plain messages read as typed). - **Message rendering:** bot messages are markdown, rendered with `react-markdown` + `remark-gfm` (raw HTML never rendered) through the twui-based map in `[bot_id]/(data)/message-markdown-components.tsx` (`markdown-link`, `markdown-list`, `markdown-code-block` partials). A bot names a crew member as `[bot:ID]` (`src/utils/bot-ref.ts`, linked to `#bot-ID` by `link-computer-refs.ts`): `(partials)/bot-ref-chip.tsx` shows its avatar and name, linked to its conversation (in the open workspace when it's in the crew there), from the live crew (`CrewBotsContext`), so a bot created in that reply links too; the coordinator is told to write them (`CoordinatorReplyNote`), and `find_bot` and `create_bot` return them. A workspace is named as `[workspace:ID]` (`#workspace-ID`, `(partials)/workspace-ref-chip.tsx`, from `CrewWorkspacesContext`). User messages are markdown too, rendered in the ink bubble by `user-bubble.tsx` with `UserMarkdownComponents` and `remark-breaks` (a single line break stays one, so older plain messages read as typed).
- **Replies and reactions:** every bot message has a row under it (`(partials)/message-actions.tsx`, handlers from `BotMessageActionsContext`) to reply or react; the user's own messages only show the bot's reactions, read-only. Replying quotes the message above the composer (`composer-reply-preview.tsx`) and the sent one carries `bot_messages.reply_to_message_id` (not a foreign key: a deleted quote only drops it); `send-message` takes any message of the same conversation (the user's, same bot and workspace, any session, so a rollover or a moved-on thread still sends), and the conversation drops a picked quote when its session changes. Messages carry `reply_to` (`grabMessagesReplies`, also on relayed messages), shown by `message-reply-quote.tsx`, and the bot reads it as `[Replying to ...]` before the user's turn (`botMessagesToHarness`). Reactions are a fixed set (`BotMessageReactions` in `src/dict/bot-dict.ts`) in `bot_message_reactions` (one row per message, emoji and user, or bot); `api/admin/bots/react-message.ts` toggles the user's own on a bot message in their sessions (`grabUserBotMessage`, `toggleMessageReaction`), and every bot gets `react_to_message` (`src/harness/tools/bot/react-to-message-tool.ts`, chat replies and tasks: a message in the session it answers, the user's latest by default). Both push the message's fresh reactions (`grabMessagesReactions`, `mine` for the session user) to the session's user as `server:bot-reaction` (`BotMessageReactionData`), which replaces the message's own; `grabBotMessages` adds them when given `user_id`. Reactions aren't relayed between processes (production runs one). - **Replies and reactions:** every bot message has a row under it (`(partials)/message-actions.tsx`, handlers from `BotMessageActionsContext`) to reply or react; the user's own messages only show the bot's reactions, read-only. Replying quotes the message above the composer (`composer-reply-preview.tsx`) and the sent one carries `bot_messages.reply_to_message_id` (not a foreign key: a deleted quote only drops it); `send-message` takes any message of the same conversation (the user's, same bot and workspace, any session, so a rollover or a moved-on thread still sends), and the conversation drops a picked quote when its session changes. Messages carry `reply_to` (`grabMessagesReplies`, also on relayed messages), shown by `message-reply-quote.tsx`, and the bot reads it as `[Replying to ...]` before the user's turn (`botMessagesToHarness`). Reactions are a fixed set (`BotMessageReactions` in `src/dict/bot-dict.ts`) in `bot_message_reactions` (one row per message, emoji and user, or bot); `api/admin/bots/react-message.ts` toggles the user's own on a bot message in their sessions (`grabUserBotMessage`, `toggleMessageReaction`), and every bot gets `react_to_message` (`src/harness/tools/bot/react-to-message-tool.ts`, chat replies and tasks: a message in the session it answers, the user's latest by default). Both push the message's fresh reactions (`grabMessagesReactions`, `mine` for the session user) to the session's user as `server:bot-reaction` (`BotMessageReactionData`), which replaces the message's own; `grabBotMessages` adds them when given `user_id`. Reactions aren't relayed between processes (production runs one).
- **Attachments and thinking:** the composer uploads files as they're picked, pasted or dropped (`use-composer-attachments`), shows them as chips, and sends once none is still uploading; the thread shows images as thumbnails and other files as chips. Replies read the files of the last `BotReplyAttachmentMessages` messages with any (`load-attachment-parts.ts`: images and audio as parts, documents as their text up to `BotAttachmentMaxTextChars`, videos as a note); older ones are only named. Tasks don't get the files yet. While the user has the last word in a session (every user message queues a reply, so this needs no face and shows even while the bot works in another session) or the bot waits for a model (`BotWaitingStatusLine`) or a task for its computer (`BotComputerBusyStatusLine`), `BotThinkingRow` sits under the last message; a message still on its way to the server reads "<name> is getting your message…" (`BotGettingMessageStatusLine`), while a task runs it reads "<name> is working…" with the task's latest step, and a reply working through tools shows its latest step under "is thinking…" (`use-reply-progress`). A coordinator with no workers gets `CoordinatorSoloNote` instead of `CoordinatorReplyNote`, so it never claims to have handed work to a worker. Live notices come from the process that ran the job at once, and from every other process through the live relay. - **Attachments and thinking:** the composer uploads files as they're picked, pasted or dropped (`use-composer-attachments`), shows them as chips, and sends once none is still uploading; the thread shows images as thumbnails and other files as chips. Replies read the files of the last `BotReplyAttachmentMessages` messages with any (`load-attachment-parts.ts`: images and audio as parts, documents as their text up to `BotAttachmentMaxTextChars`, videos as a note); older ones are only named. Tasks don't get the files yet. While the user has the last word in a session (every user message queues a reply, so this needs no face and shows even while the bot works in another session) or the bot waits for a model (`BotWaitingStatusLine`) or a task for its computer (`BotComputerBusyStatusLine`), `BotThinkingRow` sits under the last message; a message still on its way to the server reads "<name> is getting your message…" (`BotGettingMessageStatusLine`), while a task runs it reads "<name> is working…" with the task's latest step, and a reply working through tools shows its latest step under "is thinking…" (`use-reply-progress`). A coordinator with no workers gets `CoordinatorSoloNote` instead of `CoordinatorReplyNote`, so it never claims to have handed work to a worker. A stop button rides on that row (`stop-bot-button.tsx`, and the header menu's Stop row, both through `StopBotContext`) exactly while `botBusyState` (`src/pages/admin/bots/[bot_id]/(utils)/bot-busy-state.ts`, which also decides the row's line) says the bot has work open for that user anywhere; a message still on its way to the server isn't stoppable, since its reply is queued after it lands. Live notices come from the process that ran the job at once, and from every other process through the live relay.
- **Voice:** the composer's mic (`(partials)/composer-voice-button.tsx`, `use-voice-recorder`, shown once the browser can record, disabled with a note while transcription has no model to run on for the bot there, `bot_voice` page prop, from `resolveHarnessCandidates`) dictates: the recording goes to `api/admin/bots/transcribe.ts` (`transcribeAudio` with the bot's models, after `grabUserConversation`) and the text lands in the input at the cursor, to edit and send; Escape cancels, `AppData.ComposerVoiceMaxMs` stops it. Each bot message has a play button (disabled with a note while speech has no model) (`speak-message-button.tsx`, one player per conversation via `SpeakMessageContext` / `use-speak-message`) that plays `GET api/admin/bots/speak-message.ts?message_id=` straight in an `<audio>` (`speakText` streamed through as the provider makes it, so it starts in a couple of seconds; a failure answers in JSON, which the hook fetches again to show; only the user's own sessions, `grabSpeakableMessage`; text from `speechText`, cut at `AppData.SpeechMaxChars`). Under Speech, every model settings form has "Hear a sample" (`settings/models/(partials)/harness-voice-preview.tsx`), which plays `AppData.VoicePreviewText` in the form's model and voice before they're saved (`GET api/admin/harness/preview-voice`, super admins and admins, that model only via `grabModelCandidate`, streamed). Neither recordings nor audio are stored. Harness errors (`HarnessError`, safe to show) come back as the buttons' titles. - **Voice:** the composer's mic (`(partials)/composer-voice-button.tsx`, `use-voice-recorder`, shown once the browser can record, disabled with a note while transcription has no model to run on for the bot there, `bot_voice` page prop, from `resolveHarnessCandidates`) dictates: the recording goes to `api/admin/bots/transcribe.ts` (`transcribeAudio` with the bot's models, after `grabUserConversation`) and the text lands in the input at the cursor, to edit and send; Escape cancels, `AppData.ComposerVoiceMaxMs` stops it. Each bot message has a play button (disabled with a note while speech has no model) (`speak-message-button.tsx`, one player per conversation via `SpeakMessageContext` / `use-speak-message`) that plays `GET api/admin/bots/speak-message.ts?message_id=` straight in an `<audio>` (`speakText` streamed through as the provider makes it, so it starts in a couple of seconds; a failure answers in JSON, which the hook fetches again to show; only the user's own sessions, `grabSpeakableMessage`; text from `speechText`, cut at `AppData.SpeechMaxChars`). Under Speech, every model settings form has "Hear a sample" (`settings/models/(partials)/harness-voice-preview.tsx`), which plays `AppData.VoicePreviewText` in the form's model and voice before they're saved (`GET api/admin/harness/preview-voice`, super admins and admins, that model only via `grabModelCandidate`, streamed). Neither recordings nor audio are stored. Harness errors (`HarnessError`, safe to show) come back as the buttons' titles.
- **Pictures:** a bot or workspace can have a picture (`image_media_id`, a private `media` row, paradigm `bot-image`/`workspace-image`), set or removed through `api/admin/set-profile-image.ts` (`setProfileImage`) or, from chat, by the bots themselves through the `set_picture` tool (`src/harness/tools/bot/set-picture-tool.ts`, every run: a bot sets its own face, the coordinator any worker) and `update_workspace`'s `picture_file_id` (`setBotPicture`, `copyMediaAsPicture`; super admins and admins only, `assertCrewManager`). Both take the image by its `[file:ID]` ref (`grabPictureSource`, so a bot only sets what it could already read; chat uploads land in `uploads/...` and are labelled with their media id in `load-attachment-parts.ts`/`describeAttachment.ts`) or by an upload's media id. The bytes are **copied** into a new picture row, never moved: a chat message that attached the source keeps loading, and only a row whose paradigm is `bot-image`/`workspace-image` is deleted when the picture is replaced (`removeReplacedPicture`, `is-picture-media.ts`). A picture keeps its own format (`imageOutputFormat`, `src/utils/image-output-format.ts`: PNG stays a PNG with its transparency, GIF becomes PNG, everything else JPEG), for the full image and the thumbnail alike. Its change is pushed live (`pushBotUpdated`'s `image_media_id`, `pushWorkspaceUpdated` and `server:workspace-updated`, `relayChangedWorkspaces`), so `BotAvatar` (rounded square, else its stone, its colour still applies) and `WorkspaceAvatar` (else the workspace icon) swap it without a reload. A bot is also edited from the header avatar (`(partials)/bot-profile-modal.tsx`, the same `BotForm` as adding one, `api/admin/bots/update-bot.ts`; the coordinator keeps its name and stays hue-less). - **Pictures:** a bot or workspace can have a picture (`image_media_id`, a private `media` row, paradigm `bot-image`/`workspace-image`), set or removed through `api/admin/set-profile-image.ts` (`setProfileImage`) or, from chat, by the bots themselves through the `set_picture` tool (`src/harness/tools/bot/set-picture-tool.ts`, every run: a bot sets its own face, the coordinator any worker) and `update_workspace`'s `picture_file_id` (`setBotPicture`, `copyMediaAsPicture`; super admins and admins only, `assertCrewManager`). Both take the image by its `[file:ID]` ref (`grabPictureSource`, so a bot only sets what it could already read; chat uploads land in `uploads/...` and are labelled with their media id in `load-attachment-parts.ts`/`describeAttachment.ts`) or by an upload's media id. The bytes are **copied** into a new picture row, never moved: a chat message that attached the source keeps loading, and only a row whose paradigm is `bot-image`/`workspace-image` is deleted when the picture is replaced (`removeReplacedPicture`, `is-picture-media.ts`). A picture keeps its own format (`imageOutputFormat`, `src/utils/image-output-format.ts`: PNG stays a PNG with its transparency, GIF becomes PNG, everything else JPEG), for the full image and the thumbnail alike. Its change is pushed live (`pushBotUpdated`'s `image_media_id`, `pushWorkspaceUpdated` and `server:workspace-updated`, `relayChangedWorkspaces`), so `BotAvatar` (rounded square, else its stone, its colour still applies) and `WorkspaceAvatar` (else the workspace icon) swap it without a reload. A bot is also edited from the header avatar (`(partials)/bot-profile-modal.tsx`, the same `BotForm` as adding one, `api/admin/bots/update-bot.ts`; the coordinator keeps its name and stays hue-less).
- **Colour:** a bot's colour is free. It lives in `bots.color` (`#rrggbb`, set from the palette tiles, the native swatch or a typed hex in `(partials)/bot-colour-picker.tsx`, `api/admin/bots/`), and the stone it's drawn on is `bots.stone` (`BotStoneShapes`), because shape no longer follows colour. `src/utils/bot-color.ts` is the only place that resolves them (`resolveBotColor`/`resolveBotStone` fall back to the old `hue` for a bot saved before this, `botColorNeedsPaperFace` flips the eyes to paper on a very dark body), and `src/utils/bot-stone-props.ts` gives every row its `color`/`stone` props. A new bot starts on the palette's next free colour (`src/utils/pick-free-color.ts`). `src/scripts/move-bot-colour-to-owners.ts` backfills an install from `hue` (safe to rerun). - **Colour:** a bot's colour is free. It lives in `bots.color` (`#rrggbb`, set from the palette tiles, the native swatch or a typed hex in `(partials)/bot-colour-picker.tsx`, `api/admin/bots/`), and the stone it's drawn on is `bots.stone` (`BotStoneShapes`), because shape no longer follows colour. `src/utils/bot-color.ts` is the only place that resolves them (`resolveBotColor`/`resolveBotStone` fall back to the old `hue` for a bot saved before this, `botColorNeedsPaperFace` flips the eyes to paper on a very dark body), and `src/utils/bot-stone-props.ts` gives every row its `color`/`stone` props. A new bot starts on the palette's next free colour (`src/utils/pick-free-color.ts`). `src/scripts/move-bot-colour-to-owners.ts` backfills an install from `hue` (safe to rerun).
- **Avatars:** `BotStone` (`src/components/general/bot-stone.tsx`) renders the stone inline from `src/data/stone-geometry.ts`, the bot's colour as the body's SVG `fill` attribute, face (`idle|working|done|asleep|needs-you`) drawn as eyes. The coordinator is hue-less (ink body, paper eyes; inverts in dark). Never put a stone in a circle/badge or add outline/shadow. Wordmark is vector (`Wordmark`, `src/data/wordmark-paths.ts`), never live text. - **Avatars:** `BotStone` (`src/components/general/bot-stone.tsx`) renders the stone inline from `src/data/stone-geometry.ts`, the bot's colour as the body's SVG `fill` attribute, face (`idle|working|done|asleep|needs-you`) drawn as eyes. The coordinator is hue-less (ink body, paper eyes; inverts in dark). Never put a stone in a circle/badge or add outline/shadow. Wordmark is vector (`Wordmark`, `src/data/wordmark-paths.ts`), never live text.
- **Copy helpers:** `src/utils/crew-status.ts` (`3 working · 1 needs you`, "Three bots are working. One needs you."), `src/utils/relative-time.ts` (relative timestamps only). - **Copy helpers:** `src/utils/crew-status.ts` (`3 working · 1 needs you`, "Three bots are working. One needs you."), `src/utils/relative-time.ts` (relative timestamps only).
- **Prompt and memory (self-learning):** a bot has `system_prompt` (core instructions) and a separate core `memory` (markdown, general knowledge about its job, the same for every user it works for, never about one user). What it remembers about a user lives in `bot_user_memories` (one row per bot and user, seen only by that user's runs; `grab-bot-user-memory`, `update-bot-user-memory`), like workspace memory stays in its workspace. The bot rewrites them as it adapts (`update-bot-system-prompt`, `update-bot-memory` in `src/functions/backend/db/bots/`; whole-text replace, capped at `AppData.PromptMaxLength`). `grab-bot-prompt` builds the full prompt for the run's user via `composeBotPrompt` (`src/utils/compose-bot-prompt.ts`): core instructions, then the install-wide `# Install-wide instructions` (`bot_system_prompt_append` in `app_settings`, set by a super admin on Settings > General, `grabSystemPromptAppend`; empty when unset), then (in a workspace only) `# Workspace: <name>` with its description and `system_prompt`, the bot's `# Your instructions in <name>` (`workspace_bots.system_prompt`), `# Memory`, `# Memory about the user`, then `# Memory in <name>` (`workspace_bots.memory`). `update_memory` saves to `user` by default (`workspace` in a workspace) and to `core` only for what holds for everyone. In a workspace the bot learns into the workspace's instructions and memory by default (`scope: workspace` on `update_instructions`/`update_memory`, `WorkspaceLearningNote`) and into its core ones only when told to, or after suggesting it. `src/scripts/move-bot-memory-to-owners.ts` (run once per install) moved core memory written before this into the owner's. - **Prompt and memory (self-learning):** a bot has `system_prompt` (core instructions) and a separate core `memory` (markdown, general knowledge about its job, the same for every user it works for, never about one user). What it remembers about a user lives in `bot_user_memories` (one row per bot and user, seen only by that user's runs; `grab-bot-user-memory`, `update-bot-user-memory`), like workspace memory stays in its workspace. The bot rewrites them as it adapts (`update-bot-system-prompt`, `update-bot-memory` in `src/functions/backend/db/bots/`; whole-text replace, capped at `AppData.PromptMaxLength`). `grab-bot-prompt` builds the full prompt for the run's user via `composeBotPrompt` (`src/utils/compose-bot-prompt.ts`): core instructions, then the install-wide `# Install-wide instructions` (`bot_system_prompt_append` in `app_settings`, set by a super admin on Settings > General, `grabSystemPromptAppend`; empty when unset), then (in a workspace only) `# Workspace: <name>` with its description and `system_prompt`, the bot's `# Your instructions in <name>` (`workspace_bots.system_prompt`), `# Memory`, `# Memory about the user`, then `# Memory in <name>` (`workspace_bots.memory`). `update_memory` saves to `user` by default (`workspace` in a workspace) and to `core` only for what holds for everyone. In a workspace the bot learns into the workspace's instructions and memory by default (`scope: workspace` on `update_instructions`/`update_memory`, `WorkspaceLearningNote`) and into its core ones only when told to, or after suggesting it. `src/scripts/move-bot-memory-to-owners.ts` (run once per install) moved core memory written before this into the owner's.
- **Queue:** every bot job goes through `bots_queue`, with `kind` (`BotJobKinds`): `reply` (answers the user in chat, no task row) or `task` (long-running work, typically on a computer). Long-running work goes through `bots_queue`, a bun-sqlite `isQueue` table worked with the native `BunSQLite.queue` API (`status` pending|processing|completed|failed, `attempts`, `available_at`, `locked_at`/`locked_by`, `last_error`; options in `AppData.BotQueue*`). Not every message needs a job: the bot replies right away that it's on it and `enqueueBotJob` queues the brief (`instructions`, `params`, optional `message_id`/`computer_id`, the computer checked against the bot's access; a `bot_tasks` row is created unless `task_id` is given). DB helpers in `src/functions/backend/db/bots-queue/`: `claimBotJobs`, `updateBotJob` (renews the lock, saves `progress`/`checkpoint`; false once lost or cancelled), `completeBotJob` (result message, task done, face done/needs-you for the job's user), `failBotJob` (backoff retry, or `fatal`; out of attempts posts a needs-you message via `notifyBotJobFailed`), `cancelBotJob` (failed with `BotJobCancelledError`, `src/dict/bot-queue-dict.ts`), `releaseWorkerBotJobs`. Every model turn of a job is a step in `bots_queue_updates` (`kind` tool|note, one-line `text`, JSON `metadata`; tool names, never arguments; `describe-job-step.ts`), written through the job context's `report` (the text loop's `onStep`), which also sets `progress` and pushes `server:bot-job-update` (`BotJobUpdateData`). Steps are deleted with their job when it's pruned. One attempt has a deadline (`AppData.BotJobAttemptTimeoutMs`): the heartbeat keeps the lock alive for as long as the run takes, so a run past the deadline is aborted, reported as a failed try and retried, and a model or tool that never answers can't hold its slot, conversation or computer forever. The active task's latest `AppData.BotJobUpdatesShown` steps ride on `BotTaskWithSteps.updates` (`grabBotJobUpdates`); the conversation (`bot_task` page prop, `use-task-progress`) shows the latest one under `BotThinkingRow`'s "<name> is working…" while a task runs. Step text is the first line the model wrote, else plain labels for its tools (`JobStepToolLabels` in `src/dict/bot-queue-dict.ts`; other tools in words, an MCP tool as "Using Exa search (web search exa)"). - **Queue:** every bot job goes through `bots_queue`, with `kind` (`BotJobKinds`): `reply` (answers the user in chat, no task row) or `task` (long-running work, typically on a computer). Long-running work goes through `bots_queue`, a bun-sqlite `isQueue` table worked with the native `BunSQLite.queue` API (`status` pending|processing|completed|failed, `attempts`, `available_at`, `locked_at`/`locked_by`, `last_error`; options in `AppData.BotQueue*`). Not every message needs a job: the bot replies right away that it's on it and `enqueueBotJob` queues the brief (`instructions`, `params`, optional `message_id`/`computer_id`, the computer checked against the bot's access; a `bot_tasks` row is created unless `task_id` is given). DB helpers in `src/functions/backend/db/bots-queue/`: `claimBotJobs`, `updateBotJob` (renews the lock, saves `progress`/`checkpoint`; false once lost or cancelled), `completeBotJob` (result message, task done, face done/needs-you for the job's user), `failBotJob` (backoff retry, or `fatal`; out of attempts posts a needs-you message via `notifyBotJobFailed`), `cancelBotJob` (failed with `BotJobCancelledError`, `src/dict/bot-queue-dict.ts`), `stop-bot-jobs` (a user pressing Stop: every pending or processing job of that bot in their conversations, `open-bot-jobs-sql`, failed with `BotJobStoppedError`, its task closed, a `Stopped` step recorded and one closing message per session, `stopped-job-report`; the running handler's controller is aborted from this process through `job-abort-registry`, so a job waiting on a model stops at once instead of on its next heartbeat, and a late `setFace` after the abort is dropped so a stopped bot is never pushed back to `working`; `api/admin/bots/stop.ts` then pushes the messages and the idle face, `quiet_push` so the user who pressed it gets no push notification), `releaseWorkerBotJobs`. Every model turn of a job is a step in `bots_queue_updates` (`kind` tool|note, one-line `text`, JSON `metadata`; tool names, never arguments; `describe-job-step.ts`), written through the job context's `report` (the text loop's `onStep`), which also sets `progress` and pushes `server:bot-job-update` (`BotJobUpdateData`). Steps are deleted with their job when it's pruned. One attempt has a deadline (`AppData.BotJobAttemptTimeoutMs`): the heartbeat keeps the lock alive for as long as the run takes, so a run past the deadline is aborted, reported as a failed try and retried, and a model or tool that never answers can't hold its slot, conversation or computer forever. The active task's latest `AppData.BotJobUpdatesShown` steps ride on `BotTaskWithSteps.updates` (`grabBotJobUpdates`); the conversation (`bot_task` page prop, `use-task-progress`) shows the latest one under `BotThinkingRow`'s "<name> is working…" while a task runs. Step text is the first line the model wrote, else plain labels for its tools (`JobStepToolLabels` in `src/dict/bot-queue-dict.ts`; other tools in words, an MCP tool as "Using Exa search (web search exa)").
- **Queue handler:** `src/bots-queue/index.ts`, started from `server.ts` after cron and gated the same way. It releases its own cut-off jobs on startup (stable worker `AppData.BotQueueWorker`; polling starts even if that release fails, and a production process without `MAIN_SERVER` says so in the log instead of going quiet), then polls every `BotQueuePollMs`: reports jobs bun-sqlite failed on a lock timeout (`report-lost-bot-jobs`), claims per kind (`claimBotJobs` mirrors bun-sqlite's atomic claim plus `kind`, since `queue.claim` has no filter) up to `BotQueueReplyConcurrency` replies and `BotQueueConcurrency` tasks on separate slots, and runs each with `processBotJob` (heartbeat at a quarter of the lock timeout, abort on cancel/loss, one task per computer and one reply per session via `createKeyLock`, so a bot's other sessions answer side by side; a task waiting for its busy computer shows face `asleep` with `BotComputerBusyStatusLine`; face `working` once it starts, outcome pushed live; a job finishing while others for that user still run puts back the running one's face, `grabRunningBotJobFace`). The actual work is `run-bot-job.ts` (`BotJobRunner`, context in `types.ts`), which dispatches to `run-bot-reply.ts` and `run-bot-task.ts`. Throw `BotJobFatalError` for failures retrying can't fix. When every model is resting (`HarnessError.resting`, nothing was tried) the job is put back with `deferBotJob` until the first comes back, its attempt refunded and the bot's face `asleep`; other retryable harness errors retry no sooner than their `retry_at`. The `prune-bots-queue` cron deletes finished jobs after `BotQueueRetentionDays`. - **Queue handler:** `src/bots-queue/index.ts`, started from `server.ts` after cron and gated the same way. It releases its own cut-off jobs on startup (stable worker `AppData.BotQueueWorker`; polling starts even if that release fails, and a production process without `MAIN_SERVER` says so in the log instead of going quiet), then polls every `BotQueuePollMs`: reports jobs bun-sqlite failed on a lock timeout (`report-lost-bot-jobs`), claims per kind (`claimBotJobs` mirrors bun-sqlite's atomic claim plus `kind`, since `queue.claim` has no filter) up to `BotQueueReplyConcurrency` replies and `BotQueueConcurrency` tasks on separate slots, and runs each with `processBotJob` (heartbeat at a quarter of the lock timeout, abort on cancel/loss, one task per computer and one reply per session via `createKeyLock`, so a bot's other sessions answer side by side; a task waiting for its busy computer shows face `asleep` with `BotComputerBusyStatusLine`; face `working` once it starts, outcome pushed live; a job finishing while others for that user still run puts back the running one's face, `grabRunningBotJobFace`). The actual work is `run-bot-job.ts` (`BotJobRunner`, context in `types.ts`), which dispatches to `run-bot-reply.ts` and `run-bot-task.ts`. Throw `BotJobFatalError` for failures retrying can't fix. When every model is resting (`HarnessError.resting`, nothing was tried) the job is put back with `deferBotJob` until the first comes back, its attempt refunded and the bot's face `asleep`; other retryable harness errors retry no sooner than their `retry_at`. The `prune-bots-queue` cron deletes finished jobs after `BotQueueRetentionDays`.
- **Live updates:** server push only. `sendToUser` (`src/websocket/(functions)/send-to-user.ts`) sends to one user's sockets; `pushBotNotice` sends `server:bot-message` (the message row, user messages too, for other tabs) and `server:bot-updated` (`BotUpdatedData`: face, status line, active task). The bots layout opens one socket (twui `useWebSocket`); components listen with `useServerEvent` (`src/hooks/`), and `useLiveBots` keeps the rail and header current. The conversation dedupes messages by id. Every process also runs `liveRelay` (`src/live-relay/`): every `AppData.LiveRelayPollMs` it reads new `bot_messages`, new `bots_queue_updates`, changed faces and changed bots from the database and pushes them to its own sockets, so work done by another process still shows live; clients drop what they already have. `server:bot-updated` carries the bot's `computer_id` when known, which keeps the header's computer in step across tabs, and its `workspace_ids` after a membership change, so a workspace's crew drops a bot that left it. A bot that joins the crew or a workspace is pushed as `server:bot-created` (`BotCreatedData`: its rail row, the workspaces it's in, and whether it belongs in that user's own chat), from the tool that created it and from the live relay for other processes, so the rail lists it without a reload. A workspace a bot created is pushed the same way as `server:workspace-created` (`WorkspaceCreatedData`: its rail row), from the tool and from the live relay. A deleted bot is pushed as `server:bot-removed` (`BotRemovedData`) to the audience read before the delete, since its assignments go with it, so its rail row goes at once. A deleted workspace is pushed the same way as `server:workspace-removed` (`WorkspaceRemovedData`), read before the delete too. A rail row's subtitle is the bot's status line while it has one, else (coordinator) the crew counts while a worker is busy or needs you, else its last message as plain text (`src/utils/message-preview.ts`). - **Live updates:** server push only. `sendToUser` (`src/websocket/(functions)/send-to-user.ts`) sends to one user's sockets; `pushBotNotice` sends `server:bot-message` (the message row, user messages too, for other tabs) and `server:bot-updated` (`BotUpdatedData`: face, status line, active task). The bots layout opens one socket (twui `useWebSocket`); components listen with `useServerEvent` (`src/hooks/`), and `useLiveBots` keeps the rail and header current. The conversation dedupes messages by id. Every process also runs `liveRelay` (`src/live-relay/`): every `AppData.LiveRelayPollMs` it reads new `bot_messages`, new `bots_queue_updates`, changed faces and changed bots from the database and pushes them to its own sockets, so work done by another process still shows live; clients drop what they already have. `server:bot-updated` carries the bot's `computer_id` when known, which keeps the header's computer in step across tabs, and its `workspace_ids` after a membership change, so a workspace's crew drops a bot that left it. A bot that joins the crew or a workspace is pushed as `server:bot-created` (`BotCreatedData`: its rail row, the workspaces it's in, and whether it belongs in that user's own chat), from the tool that created it and from the live relay for other processes, so the rail lists it without a reload. A workspace a bot created is pushed the same way as `server:workspace-created` (`WorkspaceCreatedData`: its rail row), from the tool and from the live relay. A deleted bot is pushed as `server:bot-removed` (`BotRemovedData`) to the audience read before the delete, since its assignments go with it, so its rail row goes at once. A deleted workspace is pushed the same way as `server:workspace-removed` (`WorkspaceRemovedData`), read before the delete too. A rail row's subtitle is the bot's status line while it has one, else (coordinator) the crew counts while a worker is busy or needs you, else its last message as plain text (`src/utils/message-preview.ts`).
- **Workspaces:** `workspaces` (`name`, `description`, `system_prompt` shared by every bot in it, `computer_id` its default computer (not a foreign key), `archived`), with bots assigned through `workspace_bots` (plus that bot's `system_prompt` and `memory` there; the coordinator is an implicit member) and computers through `workspace_computers`. The first computer linked becomes the default; the user (workspace settings, chat header), any bot there (`set_workspace_computer`) or the coordinator can switch it. A workspace with no computer has none: tasks needing one fail. Workspace models (`workspace_harness_settings`) win over a bot's own, then the defaults. Helpers in `src/functions/backend/db/workspaces/` (`create-workspace`, `update-workspace`, `delete-workspace`, `add/remove-bot-*`, `update-workspace-bot`, `add/remove-computer-*`, `set-workspace-computer`, `grab-user-workspaces`, `grab-workspace-bots`, `grab-workspace-computers`, `grab-workspace-activity`), all access-checked against the session user, the structural ones for super admins and admins only; API in `api/admin/workspaces/`. The coordinator sets a member's instructions there with `set_bot_workspace_instructions`, creates a workspace with `create_workspace`, and changes any workspace's name, description or context with `update_workspace` (chat replies, and its own tasks so findings it was asked to record land in the workspace; a chat-created one is pushed as `server:workspace-created`). - **Workspaces:** `workspaces` (`name`, `description`, `system_prompt` shared by every bot in it, `computer_id` its default computer (not a foreign key), `archived`), with bots assigned through `workspace_bots` (plus that bot's `system_prompt` and `memory` there; the coordinator is an implicit member) and computers through `workspace_computers`. The first computer linked becomes the default; the user (workspace settings, chat header), any bot there (`set_workspace_computer`) or the coordinator can switch it. A workspace with no computer has none: tasks needing one fail. Workspace models (`workspace_harness_settings`) win over a bot's own, then the defaults. Helpers in `src/functions/backend/db/workspaces/` (`create-workspace`, `update-workspace`, `delete-workspace`, `add/remove-bot-*`, `update-workspace-bot`, `add/remove-computer-*`, `set-workspace-computer`, `grab-user-workspaces`, `grab-workspace-bots`, `grab-workspace-computers`, `grab-workspace-activity`), all access-checked against the session user, the structural ones for super admins and admins only; API in `api/admin/workspaces/`. The coordinator sets a member's instructions there with `set_bot_workspace_instructions`, creates a workspace with `create_workspace`, and changes any workspace's name, description or context with `update_workspace` (chat replies, and its own tasks so findings it was asked to record land in the workspace; a chat-created one is pushed as `server:workspace-created`).
@@ -120,7 +120,7 @@ Inspect live data with `bun llm-code-assistant/query-db.ts "<sql>"` - a read-onl
- **Opening a computer:** the app serves each computer to whoever sees it (`computersSql`) under `/computers/<id>/` (`ComputerProxyPrefix`): `src/server.ts` hands those paths to `handleComputerProxy` (`src/functions/server/computers/`: `userAuth`, access via `grabUserComputer`, then the sandbox's address from `grabComputerMcpUrl`), which fetches plain requests (app session cookies dropped, `X-Forwarded-Host/Proto/Prefix` set so the sandbox builds its links and CDP address under the prefix, framing headers and redirects fixed) and bridges websockets (`computerProxySocket`, dispatched from `src/websocket/` when `ws.data.proxy` is set). `ComputerView` (`src/components/general/computer-view.tsx`) shows a computer in one of `ComputerViewModes` (browser `/browser-ui`, terminal `/terminal`, code `/code-server/`, desktop: the sandbox's noVNC at `/vnc/vnc.html`, its socket pointed back through the proxy by the `{prefix}` placeholder that `computerViewUrl` fills in; desktop is the only view with Chrome's toolbar (extensions) and a clipboard panel, since the browser view streams pages over CDP and hides `chrome-extension://` targets; more of the sandbox's apps can be added there), keeping visited modes loaded. In the chat, `(partials)/computer-viewer-modal.tsx` shows it, opened from a computer card's open button in the computers modal or a take-over button (`openComputer` on `BotComputersContext`). - **Opening a computer:** the app serves each computer to whoever sees it (`computersSql`) under `/computers/<id>/` (`ComputerProxyPrefix`): `src/server.ts` hands those paths to `handleComputerProxy` (`src/functions/server/computers/`: `userAuth`, access via `grabUserComputer`, then the sandbox's address from `grabComputerMcpUrl`), which fetches plain requests (app session cookies dropped, `X-Forwarded-Host/Proto/Prefix` set so the sandbox builds its links and CDP address under the prefix, framing headers and redirects fixed) and bridges websockets (`computerProxySocket`, dispatched from `src/websocket/` when `ws.data.proxy` is set). `ComputerView` (`src/components/general/computer-view.tsx`) shows a computer in one of `ComputerViewModes` (browser `/browser-ui`, terminal `/terminal`, code `/code-server/`, desktop: the sandbox's noVNC at `/vnc/vnc.html`, its socket pointed back through the proxy by the `{prefix}` placeholder that `computerViewUrl` fills in; desktop is the only view with Chrome's toolbar (extensions) and a clipboard panel, since the browser view streams pages over CDP and hides `chrome-extension://` targets; more of the sandbox's apps can be added there), keeping visited modes loaded. In the chat, `(partials)/computer-viewer-modal.tsx` shows it, opened from a computer card's open button in the computers modal or a take-over button (`openComputer` on `BotComputersContext`).
- **Take-over:** a bot asks the user to take over a computer with `ask_user_to_take_over` (`src/harness/tools/computer/ask-to-take-over-tool.ts`, chat replies and tasks, defaulting to the task's computer). It writes `[take-over:ID:mode]` (`src/utils/take-over-ref.ts`; `withTakeOverRefs` adds it if the bot left it out and flags the message needs-you), linked by `link-computer-refs.ts` and rendered by `(partials)/take-over-chip.tsx`. The viewer then offers "Hand back to <bot>", which sends a chat message so the bot carries on; a task stops and reports instead of waiting. - **Take-over:** a bot asks the user to take over a computer with `ask_user_to_take_over` (`src/harness/tools/computer/ask-to-take-over-tool.ts`, chat replies and tasks, defaulting to the task's computer). It writes `[take-over:ID:mode]` (`src/utils/take-over-ref.ts`; `withTakeOverRefs` adds it if the bot left it out and flags the message needs-you), linked by `link-computer-refs.ts` and rendered by `(partials)/take-over-chip.tsx`. The viewer then offers "Hand back to <bot>", which sends a chat message so the bot carries on; a task stops and reports instead of waiting.
- **Health:** `containers.status` is docker's state with `running` narrowed by the health check (`starting`, `unhealthy`; `inspectDockerContainer`). Only a `running` computer is reached: `grabComputerMcpUrl` returns `ready: false` otherwise (re-reading a stale state from docker first, `refreshComputerContainer`, which `sync-computers` uses too), and callers say why (`computerNotReadyMessage`). - **Health:** `containers.status` is docker's state with `running` narrowed by the health check (`starting`, `unhealthy`; `inspectDockerContainer`). Only a `running` computer is reached: `grabComputerMcpUrl` returns `ready: false` otherwise (re-reading a stale state from docker first, `refreshComputerContainer`, which `sync-computers` uses too), and callers say why (`computerNotReadyMessage`).
- **Schedules (routines in the UI):** `cron_jobs` (per user and bot, `workspace_id` null for its own chat, `title`, `instructions` (the brief), `schedule` (a `Bun.cron` expression, UTC), `computer_id` (not a foreign key; null runs on the bot's current one), `enabled`, `max_runs` (1 for a one-off), `run_count`, `next_run_at`, `last_run_at`, `last_job_id`, `last_error`). Helpers in `src/functions/backend/db/cron-jobs/` (`create-cron-job`, `update-cron-job`, `delete-cron-job`, `grab-cron-jobs`, `grab-user-cron-job`, `fire-cron-job`), all ownership-checked, and with `bot_id` limited to that bot's own. `nextCronRun` (`src/utils/next-cron-run.ts`, `Bun.cron.parse`) validates a schedule and refuses one that runs more often than `AppData.CronJobMinIntervalMs`; a bot has at most `AppData.CronJobMaxPerBot`. Every bot gets `create_schedule`, `list_schedules`, `update_schedule`, `delete_schedule` (`grabBotTools`, every run; descriptions via `describe-cron-job.ts`). The `run-cron-jobs` cron fires due ones (`fireCronJob`: claims the run atomically by moving `next_run_at`, skips it while the last run's job is still open, queues the brief as a task with `enqueueBotJob` in the bot's active session there, turns off after `max_runs` or when there's no next run, keeps why it couldn't in `last_error`, logs each run under `cron`). A run missed while the server was down runs once, then carries on from now. A run acts on its own (`ScheduledRunNote`: the brief is already approved, so it sends rather than drafts); a run that can't do its brief calls `report_run_failed` (scheduled runs only), which throws `ScheduledRunFailedError` so the queue retries it, `AppData.ScheduledRunRetryMs` apart and doubling (`scheduledRunRetryAt`); once out of attempts the user gets one "failed after N tries" message and the schedule's `last_error` keeps why (`recordCronJobFailure`). `/admin/routines` lists them (pause/resume, edit, delete; `api/admin/routines/update-routine`, `delete-routine`); bots create them from chat. - **Schedules (routines in the UI):** `cron_jobs` (per user and bot, `workspace_id` null for its own chat, `title`, `instructions` (the brief), `schedule` (a `Bun.cron` expression, UTC), `computer_id` (not a foreign key; null runs on the bot's current one), `enabled`, `max_runs` (1 for a one-off), `run_count`, `next_run_at`, `last_run_at`, `last_job_id`, `last_error`). Helpers in `src/functions/backend/db/cron-jobs/` (`create-cron-job`, `update-cron-job`, `delete-cron-job`, `grab-cron-jobs`, `grab-user-cron-job`, `fire-cron-job`), all ownership-checked, and with `bot_id` limited to that bot's own. `nextCronRun` (`src/utils/next-cron-run.ts`, `Bun.cron.parse`) validates a schedule and refuses one that runs more often than `AppData.CronJobMinIntervalMs`; a bot has at most `AppData.CronJobMaxPerBot`. Every bot gets `create_schedule`, `list_schedules`, `update_schedule`, `delete_schedule` (`grabBotTools`, every run; descriptions via `describe-cron-job.ts`). The `run-cron-jobs` cron fires due ones (`fireCronJob`: claims the run atomically by moving `next_run_at`, skips it while the last run's job is still open, queues the brief as a task with `enqueueBotJob` in the bot's active session there, turns off after `max_runs` or when there's no next run, keeps why it couldn't in `last_error`, logs each run under `cron`). A run missed while the server was down runs once, then carries on from now. A run acts on its own (`ScheduledRunNote`: the brief is already approved, so it sends rather than drafts); a run that can't do its brief calls `report_run_failed` (scheduled runs only), which throws `ScheduledRunFailedError` so the queue retries it, `AppData.ScheduledRunRetryMs` apart and doubling (`scheduledRunRetryAt`); once out of attempts the user gets one "failed after N tries" message and the schedule's `last_error` keeps why (`recordCronJobFailure`). `/admin/routines` lists every one of the user's (pause/resume, edit, delete; `api/admin/routines/update-routine`, `delete-routine`); bots create them from chat, and each conversation's own `(partials)/bot-routines-modal.tsx` ("Schedules" in the header menu, beside Tasks) lists the ones that run there (a bot's own chat or one workspace, `routineScopeSql` in `src/utils/routine-scope/`) through `api/admin/bots/list-routines`, which owns them through `grabUserConversation` and `grabCronJobs`'s `workspace_id` filter and loads only when the modal opens. It shares `RoutineForm`, `RoutineRow` and `RoutineEditModal` with the routines page (both in `src/components/general/`).
- **Tasks (the crew's to-do list):** `tasks` is the tracker, one collection per bot (`bot_id`) and one per workspace (`workspace_id`; exactly one is set, enforced in `taskScopeValues`, since SQLite treats NULL in a unique constraint as distinct - the `envs` precedent). It is not `bot_tasks` or `bots_queue`: those record a queued job, this records the work. Fields: `title`, `description` (the brief, markdown), `status` (`TaskStatuses` in `src/dict/task-dict.ts`: `todo|doing|blocked|done|cancelled`), `blocked_reason` (only while blocked), `origin` (`user|bot`), `created_by_bot_id`/`created_by_user_id` (not foreign keys), `completed_at`. `task_notes` is the progress log (author `user|bot`, `bot_id`, `text`); `task_media` links a file to a task as a `deliverable` or a `reference` (`task_id`, `media_id` FK, `bot_file_id` not a FK, `kind`, `name`, `added_by_bot_id`), the bytes being a private `media` row (paradigm `task-media`) served at `/media/<id>` through `grabTaskMediaFile` (registered in `handle-media-server.ts` beside `grabBotFileMedia`). Visibility is SQL in `src/utils/task-access/` (`taskScopesSql` for a user: `chatBotsSql` bots plus `workspacesSql` workspaces; `botTaskScopesSql` for a run: its own plus every workspace it works in, everything for the coordinator; tested against SQLite). A workspace's tasks are visible to every bot working there and to the coordinator. Helpers in `src/functions/backend/db/tasks/`: `create-task` (`assertCrewManager` for a manual add, `TaskMaxPerScope` cap), `grab-tasks` (filters, joins the bot's face, the workspace and the latest note), `grab-task`, `update-task` (stamps `completed_at`, clears `blocked_reason` when it leaves blocked), `delete-task`, `add-task-note`, `attach-task-file`/`detach-task-file` (a bot file resolved through `grabBotFile` so a run can't link outside its folders; a chat upload must be the user's own media; `TaskMediaMaxPerTask`), `grab-task-media`, `grab-task-media-file`, `find-duplicate-task`, `purge-task-media` (`grabTaskMediaIds`, read before `deleteBot`/`deleteWorkspace` so a task's own chat uploads aren't stranded; a bot file keeps its folder). Every run gets `list_tasks`, `create_task`, `update_task` (`src/harness/tools/bot/tasks/`, from `grabBotTools`): a worker may only add to its own list or the workspace it works in, the coordinator to any worker's, any workspace's, or its own, and `update_task` takes `status`, `blocked_reason`, a `note` and `deliverable_file_ids`/`reference_file_ids` (`[file:ID]` refs) in one call. `TasksNote` (every reply and task) says how to keep the list honest; `TaskReminderNote` (chat only) tells the bot to check `list_tasks` on a greeting or when proposing next steps and mention at most `AppData.TaskReminderDigestCount` open tasks in one line, blocked first, never repeating a reminder and staying quiet mid-task. The user adds and changes tasks by hand in a workspace's dashboard (`(partials)/workspace-tasks-group.tsx`) and from the conversation header's menu (`(partials)/bot-tasks-modal.tsx`, loading its list when it opens so the page props stay small); both compose the shared `TaskForm`, `TaskListItem`, `TaskDetailPanel`, `TaskStatusChip` and `TaskFileRow` in `src/components/general/`. API in `api/admin/tasks/` (`list-tasks`, `get-task`, `save-task`, `delete-task`, `add-note`, `attach-file`, `detach-file`), each `userAuth` → `isReadOnlyUser` → `assertUserReachesTask` → `safeErrorMessage`. Live pushes for task changes are not built yet. - **Tasks (the crew's to-do list):** `tasks` is the tracker, one collection per bot (`bot_id`) and one per workspace (`workspace_id`; exactly one is set, enforced in `taskScopeValues`, since SQLite treats NULL in a unique constraint as distinct - the `envs` precedent). It is not `bot_tasks` or `bots_queue`: those record a queued job, this records the work. Fields: `title`, `description` (the brief, markdown), `status` (`TaskStatuses` in `src/dict/task-dict.ts`: `todo|doing|blocked|done|cancelled`), `blocked_reason` (only while blocked), `origin` (`user|bot`), `created_by_bot_id`/`created_by_user_id` (not foreign keys), `completed_at`. `task_notes` is the progress log (author `user|bot`, `bot_id`, `text`); `task_media` links a file to a task as a `deliverable` or a `reference` (`task_id`, `media_id` FK, `bot_file_id` not a FK, `kind`, `name`, `added_by_bot_id`), the bytes being a private `media` row (paradigm `task-media`) served at `/media/<id>` through `grabTaskMediaFile` (registered in `handle-media-server.ts` beside `grabBotFileMedia`). Visibility is SQL in `src/utils/task-access/` (`taskScopesSql` for a user: `chatBotsSql` bots plus `workspacesSql` workspaces; `botTaskScopesSql` for a run: its own plus every workspace it works in, everything for the coordinator; tested against SQLite). A workspace's tasks are visible to every bot working there and to the coordinator. Helpers in `src/functions/backend/db/tasks/`: `create-task` (`assertCrewManager` for a manual add, `TaskMaxPerScope` cap), `grab-tasks` (filters, joins the bot's face, the workspace and the latest note), `grab-task`, `update-task` (stamps `completed_at`, clears `blocked_reason` when it leaves blocked), `delete-task`, `add-task-note`, `attach-task-file`/`detach-task-file` (a bot file resolved through `grabBotFile` so a run can't link outside its folders; a chat upload must be the user's own media; `TaskMediaMaxPerTask`), `grab-task-media`, `grab-task-media-file`, `find-duplicate-task`, `purge-task-media` (`grabTaskMediaIds`, read before `deleteBot`/`deleteWorkspace` so a task's own chat uploads aren't stranded; a bot file keeps its folder). Every run gets `list_tasks`, `create_task`, `update_task` (`src/harness/tools/bot/tasks/`, from `grabBotTools`): a worker may only add to its own list or the workspace it works in, the coordinator to any worker's, any workspace's, or its own, and `update_task` takes `status`, `blocked_reason`, a `note` and `deliverable_file_ids`/`reference_file_ids` (`[file:ID]` refs) in one call. `TasksNote` (every reply and task) says how to keep the list honest; `TaskReminderNote` (chat only) tells the bot to check `list_tasks` on a greeting or when proposing next steps and mention at most `AppData.TaskReminderDigestCount` open tasks in one line, blocked first, never repeating a reminder and staying quiet mid-task. The user adds and changes tasks by hand in a workspace's dashboard (`(partials)/workspace-tasks-group.tsx`) and from the conversation header's menu (`(partials)/bot-tasks-modal.tsx`, loading its list when it opens so the page props stay small); both compose the shared `TaskForm`, `TaskListItem`, `TaskDetailPanel`, `TaskStatusChip` and `TaskFileRow` in `src/components/general/`. API in `api/admin/tasks/` (`list-tasks`, `get-task`, `save-task`, `delete-task`, `add-note`, `attach-file`, `detach-file`), each `userAuth` → `isReadOnlyUser` → `assertUserReachesTask` → `safeErrorMessage`. Live pushes for task changes are not built yet.
- **Not built yet:** task steps (nothing writes `bot_task_steps` yet), live `server:task-changed` pushes, and a `/admin/tasks` page. - **Not built yet:** task steps (nothing writes `bot_task_steps` yet), live `server:task-changed` pushes, and a `/admin/tasks` page.
@@ -134,7 +134,7 @@ The app serves plain HTTP only; TLS is the reverse proxy's or tunnel's job, and
The container runs with `--network cobble`, `-p BIND:PORT`, `.env` at `/app/.env` read only, the host data directory at `/data`, the docker socket at `/var/run/docker.sock`, `MAIN_SERVER=1` and `PORT`, `--init`, `--cpus 1` and `--memory 2048m` (a light orchestration process: the heavy work happens in each computer's own container; overridable with `--cpus`/`--memory`), a `unless-stopped` restart policy and bounded json-file logs. `/data` must stay a bind mount: `resolveHostPath` turns paths under the app's own mounts into host paths for computer containers, and a named volume would break them. `DATA_DIR=/data` and `DB_DIR=/data/db` come from the Dockerfile; `entrypoint.sh` keeps bun-sqlite's schema record on the same volume. The container runs with `--network cobble`, `-p BIND:PORT`, `.env` at `/app/.env` read only, the host data directory at `/data`, the docker socket at `/var/run/docker.sock`, `MAIN_SERVER=1` and `PORT`, `--init`, `--cpus 1` and `--memory 2048m` (a light orchestration process: the heavy work happens in each computer's own container; overridable with `--cpus`/`--memory`), a `unless-stopped` restart policy and bounded json-file logs. `/data` must stay a bind mount: `resolveHostPath` turns paths under the app's own mounts into host paths for computer containers, and a named volume would break them. `DATA_DIR=/data` and `DB_DIR=/data/db` come from the Dockerfile; `entrypoint.sh` keeps bun-sqlite's schema record on the same volume.
Publishing is `scripts/publish-github.sh` and `scripts/publish-gitea.sh`: the shared flow (argument parsing, the release gate, tag preparation, the push, the release notes) lives in `scripts/lib/publish-common.sh` and `scripts/lib/publish-release.sh`, and only the release object differs (`gh release create` for GitHub, which is what fires the image build, and `curl` against the Gitea API with a `GITEA_TOKEN` or the token `tea login` stored). Both are marked-block scripts tested by `scripts/publish-tools.test.ts`, which lifts the blocks out with the `install.test.ts` approach. Images are built by GitHub Actions on a published release (`.github/workflows/release.yml`), for `linux/amd64` and `linux/arm64`, and pushed to both `ghcr.io/moduletrace/cobble` and `git.tben.me/moduletrace/cobble` with `latest`, version, `major.minor` and commit-sha tags. The version they are built with is passed as the `COBBLE_VERSION` build arg (the published tag, else `package.json`'s version read from `<version>`), which the image turns into `ENV COBBLE_VERSION` and `runningVersion` (`src/utils/running-version.ts`) reports on Settings > General; bump `package.json`'s `version` with a release, since a local build and a workflow-dispatch build fall back to it. `.github/workflows/ci.yml` runs `bun run typecheck` and `bun run test` on every push and pull request, and checks that `install.sh` parses. The Dockerfile pins Bun 1.4.3 to match CI. The repository is mirrored on Gitea, and the installer defaults to the GHCR image with `--image git.tben.me/moduletrace/cobble` as the Gitea alternative. Publishing is `scripts/publish-github.sh` and `scripts/publish-gitea.sh`: the shared flow (argument parsing, the release gate, tag preparation, the push, the release notes) lives in `scripts/lib/publish-common.sh` and `scripts/lib/publish-release.sh`, and only the release object differs (`gh release create` for GitHub, which is what fires the image build, and `curl` against the Gitea API with a `GITEA_TOKEN` or the token `tea login` stored). Both are marked-block scripts tested by `scripts/publish-tools.test.ts`, which lifts the blocks out with the `install.test.ts` approach. Images are built by GitHub Actions on a published release (`.github/workflows/release.yml`), for `linux/amd64` and `linux/arm64`, and pushed to `ghcr.io/moduletrace/cobble` with `latest`, version, `major.minor` and commit-sha tags. CI never touches the Gitea registry: it is on a VPN, which a runner cannot reach, so `scripts/mirror-gitea.sh` (`bun run mirror:gitea`) copies a release into `git.tben.me/moduletrace/cobble` afterwards, from the Gitea host over ssh, running skopeo there in a container and reading every destination tag's digest back against GHCR's; it only moves `latest` and `major.minor` while that release still owns them in GHCR, so mirroring an older one never rolls them back. The two run in parallel and neither can fail the other, which is the point: the installer defaults to GHCR, so a registry outage cannot fail a release. The version they are built with is passed as the `COBBLE_VERSION` build arg (the published tag, else `package.json`'s version read from `<version>`), which the image turns into `ENV COBBLE_VERSION` and `runningVersion` (`src/utils/running-version.ts`) reports on Settings > General; bump `package.json`'s `version` with a release, since a local build and a workflow-dispatch build fall back to it. `.github/workflows/ci.yml` runs `bun run typecheck` and `bun run test` on every push and pull request, and checks that `install.sh` parses. The Dockerfile pins Bun 1.4.3 to match CI. The repository is mirrored on Gitea, and the installer defaults to the GHCR image with `--image git.tben.me/moduletrace/cobble` as the Gitea alternative.
### Envs ### Envs
@@ -194,30 +194,43 @@ Every folder on the data volume, browsable read-only: a bot's own, a workspace's
- **Providers** (`HarnessProviderKinds` in `src/dict/harness-provider-dict.ts`): OpenRouter (default, created by `ensureDefaultHarnessProvider`), OpenAI, Anthropic, Gemini, DeepSeek, Groq, Mistral, xAI, Ollama (local, its models count as free), any OpenAI-compatible server. Rows in `harness_providers` (a private table: the key is encrypted with `src/utils/crypt.ts`, never sent to the client; falls back to the preset's env, e.g. `OPENROUTER_API_KEY`, via `grabEnv`). One adapter today, `src/harness/providers/openai-compatible/` (`chat`, `transcribe`, `speak`, `generateImage`, `embed` (`POST /embeddings`, batched), `listModels`), chosen in `grab-harness-adapter.ts`; a provider needing a native API gets its own adapter there (`HarnessAdapter` in `src/harness/types.ts`). - **Providers** (`HarnessProviderKinds` in `src/dict/harness-provider-dict.ts`): OpenRouter (default, created by `ensureDefaultHarnessProvider`), OpenAI, Anthropic, Gemini, DeepSeek, Groq, Mistral, xAI, Ollama (local, its models count as free), any OpenAI-compatible server. Rows in `harness_providers` (a private table: the key is encrypted with `src/utils/crypt.ts`, never sent to the client; falls back to the preset's env, e.g. `OPENROUTER_API_KEY`, via `grabEnv`). One adapter today, `src/harness/providers/openai-compatible/` (`chat`, `transcribe`, `speak`, `generateImage`, `embed` (`POST /embeddings`, batched), `listModels`), chosen in `grab-harness-adapter.ts`; a provider needing a native API gets its own adapter there (`HarnessAdapter` in `src/harness/types.ts`).
- **Catalog:** `harness_models` per provider (modalities, tools support, voices, Artificial Analysis scores, `free`, `available`). The `sync-harness-models` cron (hourly, on start) refreshes every enabled provider (`syncAllHarnessModels`); unlisted models are kept but marked unavailable. OpenRouter lists every modality with capabilities; other providers list ids, so capabilities are guessed (`guess-model-modalities`). Free on OpenRouter means `:free` or `openrouter/free`, never zero token prices. - **Catalog:** `harness_models` per provider (modalities, tools support, voices, Artificial Analysis scores, `free`, `available`). The `sync-harness-models` cron (hourly, on start) refreshes every enabled provider (`syncAllHarnessModels`); unlisted models are kept but marked unavailable. OpenRouter lists every modality with capabilities; other providers list ids, so capabilities are guessed (`guess-model-modalities`). Free on OpenRouter means `:free` or `openrouter/free`, never zero token prices.
- **Automatic picks** (`rank-auto-models`): only for the `automatic` harnesses in `Harnesses` (text, vision, image); transcription and speech have none and are not available until a model is picked (the model select reads "Not available", no fallback checkbox). Free models that can do the job (on the setting's provider when it has one; a provider without free models needs a picked model) (`model-serves-harness`: text needs tools), by vendor `AutoModelVendorPreference` (DeepSeek, Qwen, NVIDIA, Liquid), then other vendors that look competent (`AppData.HarnessAutoMin*`), then routers; within a vendor by intelligence index, then full-size before small, newest first. - **Automatic picks** (`rank-auto-models`): only for the `automatic` harnesses in `Harnesses` (text, vision, image); transcription and speech have none and are not available until a model is picked (the model select reads "Not available", no fallback checkbox). Free models that can do the job (on the setting's provider when it has one; a provider without free models needs a picked model) (`model-serves-harness`: text needs tools), by vendor `AutoModelVendorPreference` (DeepSeek, Qwen, NVIDIA, Liquid), then other vendors that look competent (`AppData.HarnessAutoMin*`), then routers; within a vendor by intelligence index, then full-size before small, newest first.
- **Calling:** `resolveHarnessCandidates` (pinned, then automatic) -> `withCandidates` tries them in order with process-wide cooldowns (`resolve/harness-cooldowns.ts`): rate limits and outages rest the model (until `Retry-After` when given), 401/402 rest the whole provider, 403/404 rest the model for an hour. Errors are `HarnessError` (`code` provider|unavailable|aborted, `retryable`), classified in `to-harness-error.ts`. Entry points: `text/generate-text.ts` (one turn), `text/run-text-loop.ts` (tool loop, sticks to one model, wraps up after `HarnessMaxToolSteps`, trims the oldest tool output before each turn to fit the model's context, `fit-transcript-to-context.ts`), `media/read-image.ts`, `media/read-media.ts` (documents locally via `fileToText`, images and audio through their harnesses), `audio/transcribe-audio.ts`, `audio/speak-text.ts` (returns the provider's audio as a stream, falling back to the next model only until one answers), `image/generate-image.ts`, `embeddings/embed-text.ts` (batches, then `fit-vector-dimensions.ts` pads or cuts each vector to `AppData.EmbeddingDimensions` and normalizes it, since a provider's width varies). A text model that can't take an image or audio part gets it read by the vision/transcription harness first (`prepare-messages-for-model.ts`). - **Calling:** `resolveHarnessCandidates` (pinned, then automatic) -> `withCandidates` tries them in order with process-wide cooldowns (`resolve/harness-cooldowns.ts`): rate limits and outages rest the model (until `Retry-After` when given), 401/402 rest the whole provider, 403/404 rest the model for an hour. Errors are `HarnessError` (`code` provider|unavailable|aborted, `retryable`), classified in `to-harness-error.ts`. Entry points: `text/generate-text.ts` (one turn), `text/run-text-loop.ts` (tool loop, sticks to one model, wraps up after `max_steps`, `AppData.HarnessMaxToolSteps` by default and the install's own saved limit for a task run, `grabHarnessMaxToolSteps` in `app-settings/grab-run-limits.ts` reading the `harness_max_tool_steps` app setting, and `grabReplyMaxToolSteps`/`bot_reply_max_tool_steps` for a chat reply in place of `AppData.BotReplyMaxToolSteps`; `src/utils/read-step-limit.ts` falls back to the code default for anything missing, below 1 or past `AppData.BotStepLimitMax`; trims the oldest tool output before each turn to fit the model's context, `fit-transcript-to-context.ts`), `media/read-image.ts`, `media/read-media.ts` (documents locally via `fileToText`, images and audio through their harnesses), `audio/transcribe-audio.ts`, `audio/speak-text.ts` (returns the provider's audio as a stream, falling back to the next model only until one answers), `image/generate-image.ts`, `embeddings/embed-text.ts` (batches, then `fit-vector-dimensions.ts` pads or cuts each vector to `AppData.EmbeddingDimensions` and normalizes it, since a provider's width varies). A text model that can't take an image or audio part gets it read by the vision/transcription harness first (`prepare-messages-for-model.ts`).
- **Per-bot models:** in a workspace, its `workspace_harness_settings` row wins (`source: "workspace"`). Otherwise a worker bot inherits the defaults per harness unless it has a `bot_harness_settings` row (same fields as `harness_settings`, unique per bot and harness). `grabHarnessSetting({ harness, bot_id })` returns the bot's row (`source: "bot"`) or the default; the coordinator always gets the default. Every entry point takes `bot_id` and passes it on (text loop -> media reading/transcription), so a bot's run uses its own models throughout. Saved by `saveBotHarnessSetting` (`inherit` deletes the row); the page data is `grabBotHarnessOverview`. A bot's chat header has a ghost model icon (`(partials)/header-icon-button.tsx`; the text model and provider show under the bot's name) that opens a modal with every harness (`(partials)/bot-models-modal.tsx`); the modal and `/admin/bots/[bot_id]/models` both use `(partials)/bot-harness-form.tsx`. On the coordinator the modal edits the defaults themselves (`api/admin/harness/save-settings`, the same settings as Settings, under Models, without the providers): super admins only, the icon is disabled for everyone else. Every one of these wraps `settings/models/(partials)/harness-settings-form.tsx`: per harness a provider select first (`harness-provider-select.tsx`: any provider, one provider, or on a bot "Same as the default"), then the model select narrowed to it. Removing a provider (`deleteHarnessProvider`) puts default and bot settings that picked it or its models back on any provider (`save` prop; "Same as the default" shows when the overview item has `default`). - **Per-bot models:** in a workspace, its `workspace_harness_settings` row wins (`source: "workspace"`). Otherwise a worker bot inherits the defaults per harness unless it has a `bot_harness_settings` row (same fields as `harness_settings`, unique per bot and harness). `grabHarnessSetting({ harness, bot_id })` returns the bot's row (`source: "bot"`) or the default; the coordinator always gets the default. Every entry point takes `bot_id` and passes it on (text loop -> media reading/transcription), so a bot's run uses its own models throughout. Saved by `saveBotHarnessSetting` (`inherit` deletes the row); the page data is `grabBotHarnessOverview`. A bot's chat header has a ghost model icon (`(partials)/header-icon-button.tsx`; the text model and provider show under the bot's name) that opens a modal with every harness (`(partials)/bot-models-modal.tsx`); the modal and `/admin/bots/[bot_id]/models` both use `(partials)/bot-harness-form.tsx`. On the coordinator the modal edits the defaults themselves (`api/admin/harness/save-settings`, the same settings as Settings, under Models, without the providers): super admins only, the icon is disabled for everyone else. Every one of these wraps `settings/models/(partials)/harness-settings-form.tsx`: per harness a provider select first (`harness-provider-select.tsx`: any provider, one provider, or on a bot "Same as the default"), then the model select narrowed to it. Removing a provider (`deleteHarnessProvider`) puts default and bot settings that picked it or its models back on any provider (`save` prop; "Same as the default" shows when the overview item has `default`).
- **Tools:** `HarnessTool` (built per run with its bot bound): own tools validate with a zod `schema`; tools described elsewhere pass a JSON schema as `parameters` and validate themselves. `run-tool-call.ts` reports bad calls and failures back to the model as `Error: ...`, and caps every tool call at `AppData.HarnessToolTimeoutMs` (aborted, its result handed back as an error the model can work around). Bot tools in `src/harness/tools/storage/`: `list_files`, `read_file`, `write_file`, `search_files`, `move_file`, `delete_file`, `ingest_file`, `copy_file`, `generate_image`, `make_pdf` (`grabBotFileTools`, every run, see Bot files and folders). Bot tools in `src/harness/tools/bot/`: `update_memory`, `update_instructions` (`grabBotTools`, every run), `start_task` and, for the coordinator with workers, `delegate_task` (`grabBotChatTools`, chat replies only; the job's `requested_by_bot_id` is the coordinator, and `relayJobToRequester` posts the worker's result or failure in its conversation, pushed as `relay` by `pushBotNotice`), `ask_user_to_take_over` (see Take-over), plus for the coordinator `create_workspace` and `update_workspace` (a new workspace with its shared context, or changing one; both also in its tasks, so a task that gathers material can write it into the workspace), `create_bot` (add a worker with its name, role and instructions, and join it to a workspace straight away) and, in its own chat, `add_bot_to_workspace` (put an existing worker in one of the user's workspaces) and `remove_bot_from_workspace` (take it out, keeping its conversation there), `assign_computer` (grant or revoke a worker's direct access, `add/remove-bot-to/from-computer`; access through a workspace stays and the result says so), `update_bot` (change a worker's name, role or core instructions, `src/functions/backend/db/bots/update-bot.ts`), `find_bot` and `find_workspace` (look the crew, or the live workspaces, up by id, name, role/description or instructions/context), `delete_bot` (a worker with its conversations, tasks, schedules and workspace places; the coordinator can't go) and `delete_workspace` (its conversations, tasks, schedules and links; its bots and computers stay). `search_computers` (`src/harness/tools/computer/search-computers-tool.ts`, chat replies and tasks) runs one read-only command on all of a bot's computers at once over the sandbox REST API (`run-computer-command.ts`), each output under its `[computer:ID]`, secrets redacted, capped by `AppData.ComputerSearch*`. - **Tools:** `HarnessTool` (built per run with its bot bound): own tools validate with a zod `schema`; tools described elsewhere pass a JSON schema as `parameters` and validate themselves. `run-tool-call.ts` reports bad calls and failures back to the model as `Error: ...`, and caps every tool call at `AppData.HarnessToolTimeoutMs` (aborted, its result handed back as an error the model can work around). Bot tools in `src/harness/tools/storage/`: `list_files`, `read_file`, `write_file`, `search_files`, `move_file`, `delete_file`, `ingest_file`, `copy_file`, `generate_image`, `make_pdf` (`grabBotFileTools`, every run, see Bot files and folders). Bot tools in `src/harness/tools/bot/`: `update_memory`, `update_instructions` (`grabBotTools`, every run), `start_task` and, for the coordinator with workers, `delegate_task` (`grabBotChatTools`, chat replies only; a task handed over in the coordinator's own chat runs in the worker's own chat unless it names a `workspace_id`, which puts it in that workspace: the worker's workspace instructions and memory, the workspace's context, files and computers apply, and it needs membership, added first via `addBotToWorkspace` when missing; handed over from inside a workspace it always runs there and a different `workspace_id` is refused; the job's `requested_by_bot_id` is the coordinator, and `relayJobToRequester` posts the worker's result or failure in its conversation, pushed as `relay` by `pushBotNotice`), `ask_user_to_take_over` (see Take-over), plus for the coordinator `create_workspace` and `update_workspace` (a new workspace with its shared context, or changing one; both also in its tasks, so a task that gathers material can write it into the workspace), `create_bot` (add a worker with its name, role and instructions, and join it to a workspace straight away) and, in its own chat, `add_bot_to_workspace` (put an existing worker in one of the user's workspaces) and `remove_bot_from_workspace` (take it out, keeping its conversation there), `assign_computer` (grant or revoke a worker's direct access, `add/remove-bot-to/from-computer`; access through a workspace stays and the result says so), `update_bot` (change a worker's name, role or core instructions, `src/functions/backend/db/bots/update-bot.ts`), `find_bot` and `find_workspace` (look the crew, or the live workspaces, up by id, name, role/description or instructions/context), `delete_bot` (a worker with its conversations, tasks, schedules and workspace places; the coordinator can't go) and `delete_workspace` (its conversations, tasks, schedules and links; its bots and computers stay). `search_computers` (`src/harness/tools/computer/search-computers-tool.ts`, chat replies and tasks) runs one read-only command on all of a bot's computers at once over the sandbox REST API (`run-computer-command.ts`), each output under its `[computer:ID]`, secrets redacted, capped by `AppData.ComputerSearch*`.
- **MCP:** `src/harness/mcp/`: `connectMcpServer` (Streamable HTTP, SSE fallback), `grabMcpTools` (each server tool becomes a `HarnessTool` with its JSON schema), `mcpResultToText` (text as is, images read by the vision harness, `isError` thrown). Computers are reached this way: `openComputerTools` (`src/harness/tools/computer/`) connects to the computer's AIO sandbox at `grabComputerMcpUrl` (the `MAIN_COMPUTER_URL` app setting for the main computer, for a dev server outside docker, e.g. `http://localhost:8170`, else `http://<hostname>:<port>/mcp`, or its IP address when the hostname can't be reached), leaves out `ComputerMcpExcludedTools`, and sets the computer's envs (see Envs). - **MCP:** `src/harness/mcp/`: `connectMcpServer` (Streamable HTTP, SSE fallback), `grabMcpTools` (each server tool becomes a `HarnessTool` with its JSON schema), `mcpResultToText` (text as is, images read by the vision harness, `isError` thrown). Computers are reached this way: `openComputerTools` (`src/harness/tools/computer/`) connects to the computer's AIO sandbox at `grabComputerMcpUrl` (the `MAIN_COMPUTER_URL` app setting for the main computer, for a dev server outside docker, e.g. `http://localhost:8170`, else `http://<hostname>:<port>/mcp`, or its IP address when the hostname can't be reached), leaves out `ComputerMcpExcludedTools`, and sets the computer's envs (see Envs).
- **MCPs and custom tools (extensions):** installs add their own MCP servers (`mcp_servers`, Settings, under MCPs) and tools (`custom_tools`, Settings, under Tools). A bot gets the ones given to it (`bot_mcp_servers`, `bot_custom_tools`) and, in a workspace, that workspace's (`workspace_mcp_servers`, `workspace_custom_tools`); `botExtensionsSql` (`src/utils/bot-extensions-sql.ts`, tested) resolves a run's, `grabBotExtensions` reads them. `all_bots` ("every bot by default") only decides what bots are given: turning it on gives it to every bot (`attachToAllBots`, the coordinator included) and every new bot gets it (`attachDefaultExtensions`, from `createBot` and `ensureCoordinatorBot`); turning it off takes it from no one, and a bot that drops one keeps it dropped (`src/scripts/attach-every-bot-extensions.ts`, safe to rerun, moved installs from the old "reaches every bot" meaning). Super admins define them. Super admins and admins (the super admin alone for the coordinator, `assertBotExtensionManager`) give and take them per bot on the bot's settings page and under its chat's composer (`composer-tools.tsx`: one "N tools" button (`composer-tools-modal.tsx`) counting everything it can use there, which opens a modal listing them (a workspace's not removable) with the add picker, so ten of them don't fill the composer; beside it `composer-computer-button.tsx` opens the bot's current computer in the viewer), and per workspace on the workspace's settings page; the settings pages share `settings/(partials)/extensions-group.tsx`, all through `useExtensions` (`src/hooks/`), `grabBotExtensionsOverview` / `grabWorkspaceExtensionsOverview` (`bot_extensions` / `workspace_extensions` page props, both on `grabExtensionsOverview`) and `api/admin/bots/` and `api/admin/workspaces/` `add-extension`, `remove-extension` (`setBotExtension`, `setWorkspaceExtension`); the MCPs and Tools pages' Access modal sets bots and workspaces in bulk (`setExtensionAccess`, an admin never touches the coordinator's or `all_bots`). Helpers in `src/functions/backend/db/extensions/`, API in `api/admin/mcps/` and `api/admin/tools/`, zod in `src/schemas/extension-schema.ts`. - **MCPs and custom tools (extensions):** installs add their own MCP servers (`mcp_servers`, Settings, under MCPs) and tools (`custom_tools`, Settings, under Tools). A bot gets the ones given to it (`bot_mcp_servers`, `bot_custom_tools`) and, in a workspace, that workspace's (`workspace_mcp_servers`, `workspace_custom_tools`); `botExtensionsSql` (`src/utils/bot-extensions-sql.ts`, tested) resolves a run's, `grabBotExtensions` reads them. `all_bots` ("every bot by default") only decides what bots are given: turning it on gives it to every bot (`attachToAllBots`, the coordinator included) and every new bot gets it (`attachDefaultExtensions`, from `createBot` and `ensureCoordinatorBot`); turning it off takes it from no one, and a bot that drops one keeps it dropped (`src/scripts/attach-every-bot-extensions.ts`, safe to rerun, moved installs from the old "reaches every bot" meaning). Super admins define them. Super admins and admins (the super admin alone for the coordinator, `assertBotExtensionManager`) give and take them per bot on the bot's settings page and under its chat's composer (`composer-tools.tsx`: one "N tools" button (`composer-tools-modal.tsx`) counting everything it can use there, which opens a modal listing them (a workspace's not removable) with the add picker, so ten of them don't fill the composer; beside it `composer-computer-button.tsx` opens the bot's current computer in the viewer), and per workspace on the workspace's settings page; the settings pages share `settings/(partials)/extensions-group.tsx`, all through `useExtensions` (`src/hooks/`), `grabBotExtensionsOverview` / `grabWorkspaceExtensionsOverview` (`bot_extensions` / `workspace_extensions` page props, both on `grabExtensionsOverview`) and `api/admin/bots/` and `api/admin/workspaces/` `add-extension`, `remove-extension` (`setBotExtension`, `setWorkspaceExtension`); the MCPs and Tools pages' Access modal sets bots and workspaces in bulk (`setExtensionAccess`, an admin never touches the coordinator's or `all_bots`). Helpers in `src/functions/backend/db/extensions/`, API in `api/admin/mcps/` and `api/admin/tools/`, zod in `src/schemas/extension-schema.ts`.
- **Skills** are the third extension kind (`skills`, Settings, under Skills; `bot_skills` / `workspace_skills`, same `all_bots` rules, `ExtensionTables.kind = "skill"`). A skill is a runbook: `name`, a unique `slug` a bot loads it by, a one-line `description` (when it applies) and the markdown `body`, capped by `AppData.SkillBodyMaxChars`. Unlike an MCP server or a tool it has no `status`/`last_error`: nothing connects or runs, and it never reports a failure (`ExtensionRow` and `ComposerToolRow` key the state line off `kind == "skill"`). A run sees only the slug and description; `openBotSkills` (`src/harness/tools/skills/`, fed by the `skills` that `grabBotExtensions` already reads) adds one `load_skill` tool that returns the body, cut at `AppData.SkillBodyShownChars` with a line saying so, and throws the valid slugs for an unknown one. Slugs are checked by `checkSkillSlug` (`src/utils/check-skill-slug.ts`, tested); `load_skill` is in `ReservedToolNames` so a custom tool can't shadow it. The settings page is `api/admin/skills/` (`save-skill`, `delete-skill` super admin only, `set-skill-access`), `grabSkillsOverview`; `deleteSkill` deletes the two link tables itself, since foreign keys are off outside the server process. `openBotExtensions` (`src/harness/tools/extensions/`) turns them into harness tools for chat replies and tasks (built-ins win a name clash; `close()` in the runners' `finally`; `note` tells the bot about a server it can't use). Templates (`fillTemplate`, `src/utils/fill-template.ts`): `{{args.x}}` and `{{env.KEY}}`, envs from `grabRunEnvs` (the bot's own beat the workspace's, which beat general; app settings never, no need to be set on computers); secret values are redacted from every result. - **Skills** are the third extension kind (`skills`, Settings, under Skills; `bot_skills` / `workspace_skills`, same `all_bots` rules, `ExtensionTables.kind = "skill"`). A skill is a runbook: `name`, a unique `slug` a bot loads it by, a one-line `description` (when it applies) and the markdown `body`, capped by `AppData.SkillBodyMaxChars`. Unlike an MCP server or a tool it has no `status`/`last_error`: nothing connects or runs, and it never reports a failure (`ExtensionRow` and `ComposerToolRow` key the state line off `kind == "skill"`). A run sees only the slug and description; `openBotSkills` (`src/harness/tools/skills/`, fed by the `skills` that `grabBotExtensions` already reads) adds one `load_skill` tool that returns the body, cut at `AppData.SkillBodyShownChars` with a line saying so, and throws the valid slugs for an unknown one. Slugs are checked by `checkSkillSlug` (`src/utils/check-skill-slug.ts`, tested); `load_skill` is in `ReservedToolNames` so a custom tool can't shadow it. The settings page is `api/admin/skills/` (`save-skill`, `delete-skill` super admin only, `set-skill-access`), `grabSkillsOverview`; `deleteSkill` deletes the two link tables itself, since foreign keys are off outside the server process. `openBotExtensions` (`src/harness/tools/extensions/`) turns them into harness tools for chat replies and tasks (built-ins win a name clash; `close()` in the runners' `finally`; `note` tells the bot about a server it can't use). Templates (`fillTemplate`, `src/utils/fill-template.ts`): `{{args.x}}` and `{{env.KEY}}`, envs from `grabRunEnvs` (the bot's own beat the workspace's, which beat general; app settings never, no need to be set on computers); secret values are redacted from every result.
- **Custom tools** (`CustomToolKinds` in `src/dict/custom-tool-dict.ts`): a name (`checkCustomToolName`: snake_case, never a built-in's, `ReservedToolNames`), a description, a JSON-schema `parameters` and a `config` per kind, built by `buildCustomTool` (`src/harness/tools/custom/`): `http` (`runHttpTool`: the app fetches, arguments URL-encoded in the URL, no body template sends the arguments as JSON), `script` (`runScriptTool`: bash/python/javascript on the run's computer, the task's or in chat the bot's current one, over the sandbox REST API in a subshell, computer envs set, arguments as JSON in `COBBLE_ARGS` and on stdin; left out without a computer), `prompt` (`runPromptTool`: one `generateText` turn with the bot's models). Super admins try one from the page (`testCustomTool`, the main computer for scripts): a saved one from its row (Try), or the form as it is, saved or not (`draft`, nothing recorded), from the Test panel in its modal (`custom-tool-test-panel.tsx`). Every run (a bot's call or a try; not bad arguments or a cancel) is saved as the tool's `status`, `last_error` and `last_checked_at` (`recordCustomToolRun`); editing resets it to `untested`. - **Custom tools** (`CustomToolKinds` in `src/dict/custom-tool-dict.ts`): a name (`checkCustomToolName`: snake_case, never a built-in's, `ReservedToolNames`), a description, a JSON-schema `parameters` and a `config` per kind, built by `buildCustomTool` (`src/harness/tools/custom/`): `http` (`runHttpTool`: the app fetches, arguments URL-encoded in the URL, no body template sends the arguments as JSON), `script` (`runScriptTool`: bash/python/javascript on the run's computer, the task's or in chat the bot's current one, over the sandbox REST API in a subshell, computer envs set, arguments as JSON in `COBBLE_ARGS` and on stdin; left out without a computer), `prompt` (`runPromptTool`: one `generateText` turn with the bot's models). Super admins try one from the page (`testCustomTool`, the main computer for scripts): a saved one from its row (Try), or the form as it is, saved or not (`draft`, nothing recorded), from the Test panel in its modal (`custom-tool-test-panel.tsx`). Every run (a bot's call or a try; not bad arguments or a cancel) is saved as the tool's `status`, `last_error` and `last_checked_at` (`recordCustomToolRun`); editing resets it to `untested`.
- **MCP servers** (`McpTransports` in `src/dict/mcp-dict.ts`): `remote` (URL plus templated headers, `connectMcpServer`) or `stdio` (a command run with the host's shell, `hostShell`, in `DATA_DIR/mcp/<id>` (`grabMcpDir`), after its `setup_command`, which `runMcpSetup` reruns whenever it changes (marker file) or on Reinstall; its env is only its own `env` map plus the shell basics, `grabMcpStdioEnv`, never the app's). Their tools are listed into `tools_cache` (`syncMcpServer`: on save, Refresh, and hourly for remote ones by the `sync-mcp-servers` cron) and named `<slug>__<tool>` (`mcpToolName`); the modal's Test connection checks the form as it is, saved or not, outside the pool (`testMcpDraft`, `testMcpServer`, `api/admin/mcps/test-mcp`, a new stdio server's setup runs in `mcp/0`); a run that can't connect, or connects again after failing, updates its `status` too (`recordMcpServerConnection`); runs build tools from the cache (`mcpHarnessTool`) and connect on the first call (`openMcpServer`). stdio servers live in a per-process pool (`src/harness/mcp/mcp-stdio-pool.ts`), stopped after `AppData.McpStdioIdleMs` idle, on save or delete. - **MCP servers** (`McpTransports` in `src/dict/mcp-dict.ts`): `remote` (URL plus templated headers, `connectMcpServer`) or `stdio` (a command run with the host's shell, `hostShell`, in `DATA_DIR/mcp/<id>` (`grabMcpDir`), after its `setup_command`, which `runMcpSetup` reruns whenever it changes (marker file) or on Reinstall; its env is only its own `env` map plus the shell basics, `grabMcpStdioEnv`, never the app's). Their tools are listed into `tools_cache` (`syncMcpServer`: on save, Refresh, and hourly for remote ones by the `sync-mcp-servers` cron) and named `<slug>__<tool>` (`mcpToolName`); the modal's Test connection checks the form as it is, saved or not, outside the pool (`testMcpDraft`, `testMcpServer`, `api/admin/mcps/test-mcp`, a new stdio server's setup runs in `mcp/0`); a run that can't connect, or connects again after failing, updates its `status` too (`recordMcpServerConnection`); runs build tools from the cache (`mcpHarnessTool`) and connect on the first call (`openMcpServer`). stdio servers live in a per-process pool (`src/harness/mcp/mcp-stdio-pool.ts`), stopped after `AppData.McpStdioIdleMs` idle, on save or delete.
- **Check before and after acting:** every reply and task gets `ActionCheckNote` (`src/data/harness-prompts.ts`): look for an existing record before creating one (update it instead), and verify every action took effect before saying it's done. The tools back it: `update_memory` and `update_instructions` read the save back (`checkSavedText`, `grabBotSavedText`) and fail if it didn't take; `create_bot` refuses a name already on the crew, `create_schedule` a twin of one of the bot's schedules (`findDuplicateCronJob`: same place and same title, or same timing and brief), and `start_task`/`delegate_task` a task with the same title still open in that session (`grabOpenBotTask`); `create_workspace` already refuses a name in use. - **Check before and after acting:** every reply and task gets `ActionCheckNote` (`src/data/harness-prompts.ts`): look for an existing record before creating one (update it instead), and verify every action took effect before saying it's done. The tools back it: `update_memory` and `update_instructions` read the save back (`checkSavedText`, `grabBotSavedText`) and fail if it didn't take; `create_bot` refuses a name already on the crew, `create_schedule` a twin of one of the bot's schedules (`findDuplicateCronJob`: same place and same title, or same timing and brief), and `start_task`/`delegate_task` a task with the same title still open in that session (`grabOpenBotTask`); `create_workspace` already refuses a name in use.
- **Replies and tasks:** `run-bot-reply.ts` answers from the last `BotReplyHistoryMessages` messages (`botMessagesToHarness`) with `ChatReplyNote`, the bot's computers (`compose-computers-note.ts`: what they are and that they're reached through `start_task`) and the chat tools, at most `BotReplyMaxToolSteps` steps, and finishes silently if the bot already had the last word (`enqueueBotReply` reuses a pending reply, so a burst of messages gets one). `run-bot-task.ts` runs the brief (`compose-job-brief.ts`) with `TaskRunNote`, the bot tools, the file tools and the job's computer tools, closing the MCP connection after. Both add `BotFilesNote` (where a bot's files live, that they persist, and to search before saying it doesn't know) and collect `attachments` for files the bot wrote with `attach`, which `completeBotJob` links to the result message. Retryable harness errors go back to the queue's backoff, the rest are `BotJobFatalError`. - **Replies and tasks:** `run-bot-reply.ts` answers from the last `BotReplyHistoryMessages` messages (`botMessagesToHarness`) with `ChatReplyNote`, the bot's computers (`compose-computers-note.ts`: what they are and that they're reached through `start_task`) and the chat tools, at most `grabReplyMaxToolSteps()` steps (the `bot_reply_max_tool_steps` app setting, else `AppData.BotReplyMaxToolSteps`), and finishes silently if the bot already had the last word (`enqueueBotReply` reuses a pending reply, so a burst of messages gets one). `run-bot-task.ts` runs the brief (`compose-job-brief.ts`) with `TaskRunNote`, the bot tools, the file tools and the job's computer tools, closing the MCP connection after. Both add `BotFilesNote` (where a bot's files live, that they persist, and to search before saying it doesn't know) and collect `attachments` for files the bot wrote with `attach`, which `completeBotJob` links to the result message. Retryable harness errors go back to the queue's backoff, the rest are `BotJobFatalError`.
- **API:** `api/admin/harness/` (`save-provider` also syncs, `sync-provider`, `delete-provider`, `save-settings`), super admin only, each returning the fresh `grabHarnessOverview()` page data. - **API:** `api/admin/harness/` (`save-provider` also syncs, `sync-provider`, `delete-provider`, `save-settings`), super admin only, each returning the fresh `grabHarnessOverview()` page data.
### Token usage
Every model call is written down in `token_usages` as it happens, so a message can show what it cost and `/admin/usage` can total the install's spending by day, by model and by provider.
- **Whose call it was:** `src/harness/usage/harness-usage-scope.ts` is an `AsyncLocalStorage` scope (`user_id`, `bot_id`, `workspace_id`, `session_id`, `job_id`, `message_id`) that a run enters once, in `run-bot-job.ts` (`runWithHarnessUsageScope`), so everything underneath - the text loop's turns, a session summary, a transcription, an image, an embedding - is attributed without threading the ids through every harness function. The store is merged, not replaced: a nested run keeps the outer ids and overrides only what it knows better, and a `null` is a real value that clears a field (a bot's own chat has no workspace). `withUsageScope` (`with-usage-scope.ts`) wraps one call the app makes on a user's behalf outside any run (the chat's transcribe and speak routes), and `runWithFallbackScope` (`run-with-fallback-scope.ts`) attributes work whose run may be missing - a file the ingest queue embeds has no run, so it is charged to its owner (`grabUsageOwnerUserId`, the workspace's creator else the bot's) - while `mode: "own"` forces the given scope (the session namer, whose job is already complete and must not pay for it). Without an owner nothing is written: a row no one can be shown is not worth a row.
- **Recording:** `record-harness-usage.ts` writes one row and never throws (a model that answered must not lose its answer to a failed insert; it logs with `logError` instead). `build-usage-call.ts` turns the candidate that really answered into the row's provider and model (`provider_id`/`provider_name` and the catalog `model_id` plus the provider's `model_slug`), and providers that report no usage still get a row, so a call is never invisible. Every entry point calls it in `withCandidates`'s `on_success`: `generate-text.ts`, `read-image.ts`, `transcribe-audio.ts`, `speak-text.ts` (billed per character, so the text's length stands in for what was spoken), `generate-image.ts`, `embed-text.ts`.
- **Messages:** a reply or task job's calls carry the job's id as they happen, since its message doesn't exist yet; `complete-bot-job.ts` then runs `UPDATE token_usages SET message_id = ? WHERE job_id = ? AND message_id IS NULL` once the message is inserted, so the thread can show what the answer cost. Calls that produce no message (naming a session, embedding a file, transcribing a dictation) keep a null `message_id` and are still counted in the totals.
- **Message usage:** `grab-message-usage.ts` sums a batch of messages' calls (one `GROUP BY message_id`) and `grab-bot-messages.ts` attaches it as `BotMessageWithAttachments.usage` (`TokenUsageSummary`: prompt, completion, total, cost, calls). `formatTokens` (`src/utils/format-tokens.ts`) writes a count as a person reads it (842 / 1.2K / 842K / 1.2M). `MessageUsageChip` (`bots/[bot_id]/(partials)/message-usage-chip.tsx`) sits in the actions row under a bot message beside its timestamp and play button, showing the total with the breakdown in its `title`, and renders nothing when a message has no calls.
- **Usage page:** `/admin/usage`, a `RailPages` entry like Routines and Activity rather than a settings sub page, since it is the install's own record rather than a setting (super admins and admins only, `canManageCrew`; the dict entry's `can_view` hides it for anyone else, and `grab-usage-server-props.ts` redirects them to the bots). Helpers in `src/functions/backend/db/usage/`: `grab-usage-overview.ts` runs `grab-usage-totals.ts` (one total and the same split per harness), `grab-usage-days.ts` -> `bucket-usage-days.ts` (one bucket per day, empty days at zero), `grab-usage-models.ts` (per provider, model slug and harness, busiest first, joined to `harness_models` for a display name), plus the previous period of the same length and a rolling week. The window comes from `grab-usage-window.ts` (`UsageDefaultDays` 30, `UsageMaxDays` 365, clamped). API `api/admin/usage` (GET only) for another period, zod in `src/schemas/usage-schema.ts`.
- **Days are local, not UTC:** `grabUsageDays` groups in SQL with `date(created_at / 1000, 'unixepoch', 'localtime')`, matching the local midnights `grabUsageWindow` builds and the `YYYY-MM-DD` `localDate` writes. Grouping by the bare UTC day put a late evening's calls in the previous bucket and silently dropped them whenever no bucket carried that name (the chart's total disagreed with the totals card); `bucket-usage-days.test.ts` pins this. Never name a local day with `toISOString().slice(0, 10)`.
- **The page:** `index.server.ts` -> `(functions)/grab-usage-server-props.ts` (crew managers only, the period in the address), `index.tsx` in `SettingsShell` with `wide`, `(sections)/usage-section.tsx` (state in the address through `history.pushState`, `use-usage-overview.ts` loading another period), `(partials)/usage-totals.tsx` (tokens, model calls, cost, previous period with a change label), `usage-day-chart.tsx` + `usage-day-bar.tsx` (bars from static `h-*` classes in `(data)/usage-bar-heights.ts`, since inline styles are not allowed), `usage-harness-breakdown.tsx` + `usage-harness-row.tsx`, and `usage-models-table.tsx` + `usage-model-row.tsx` (calls, in, out, tokens, **price per thousand tokens** worked out from cost and tokens, which is what makes one model's cost comparable with another's).
## Admin Route Hierarchy ## Admin Route Hierarchy
Live routes under `/admin` (see `AdminSections`): `/admin/bots`, `/admin/workspaces` (the index opens the first workspace), `/admin/routines`, `/admin/logs`, `/admin/help` and `/admin/settings`. `/admin` redirects to `/admin/bots`; a workspace's dashboard is `/admin/workspaces/<id>`. The routes in detail: Live routes under `/admin` (see `AdminSections`): `/admin/bots`, `/admin/workspaces` (the index opens the first workspace), `/admin/routines`, `/admin/logs`, `/admin/help` and `/admin/settings`. `/admin` redirects to `/admin/bots`; a workspace's dashboard is `/admin/workspaces/<id>`. The routes in detail:
- `/admin/bots` (live) - the Cobble chat shell, plus `[bot_id]/` - `/admin/bots` (live) - the Cobble chat shell, plus `[bot_id]/`
- `/admin/routines` (live) - every schedule the user's bots have (see Schedules), opened from "Routines" in the rail's user popover - `/admin/routines` (live) - every schedule the user's bots have (see Schedules), opened from "Routines" in the rail's user popover
- `/admin/usage` (live) - what the crew's model calls spent, per day, per kind of call and per model, for crew managers (see Token usage), opened from "Usage" in the rail's user popover
- `/admin/logs` (live) - Activity, the audit log: super admins only (others are redirected; the popover's "Activity" item shows to super admins only). Rendered in `SettingsShell` with `wide` - `/admin/logs` (live) - Activity, the audit log: super admins only (others are redirected; the popover's "Activity" item shows to super admins only). Rendered in `SettingsShell` with `wide`
- `/admin/help` (live) - familiar problems and their fixes, one `SettingsGroup` per topic (`(data)/help-topics.ts`, markdown rendered with the chat's `MessageMarkdownComponents`, the topic id as the group's anchor). First topic: signing in to Google and other strict sites with Cookie-Editor over the Desktop view (`HelpSignInTopicId` in `src/dict/help-dict.ts`; `ask_user_to_take_over` points bots to it) - `/admin/help` (live) - familiar problems and their fixes, one `SettingsGroup` per topic (`(data)/help-topics.ts`, markdown rendered with the chat's `MessageMarkdownComponents`, the topic id as the group's anchor). First topic: signing in to Google and other strict sites with Cookie-Editor over the Desktop view (`HelpSignInTopicId` in `src/dict/help-dict.ts`; `ask_user_to_take_over` points bots to it)
- Routines, Files, Activity and Help are the `RailPages` (`src/dict/rail-pages-dict.ts`), opened from the rail's user popover (which lists them, then Settings and Sign out). On them the rail shows its own aside (`(partials)/rail-pages-list.tsx`): a back link to the bots, then those pages, like settings - Routines, Files, Usage, Activity and Help are the `RailPages` (`src/dict/rail-pages-dict.ts`), opened from the rail's user popover (which lists them, then Settings and Sign out). On them the rail shows its own aside (`(partials)/rail-pages-list.tsx`): a back link to the bots, then those pages, like settings. A page carries `super_admin_only` or a `can_view` rule (Usage is `canManageCrew`), and `canSeeRailItem` (`src/utils/rbac/can-see-rail-item.ts`, tested) decides both the popover and the aside, so a page is never offered in one and hidden in the other
- `/admin/settings/backups` - super admin only (listed in `SettingsPages`), two groups. Database snapshots: create, list, restore, download and delete in `db/.backups` (`bun-sqlite backup`); "Restore from file" beside "Create backup" opens `upload-restore-modal.tsx` (upload a SQLite database file, which is saved to `.backups` then restored with `bun-sqlite import <file>`, since `restore` is interactive). Whole-data archives: `write-data-backup-archive.ts` tars all of `DATA_DIR` (database, media, computer homes, bot and workspace files, MCP installs) into `BACKUP_ARCHIVE_DIR` as `cobble-data-<epoch-ms>.tar.gz` (via `tar -czf` into a `.part` name, renamed on success so a cut-off run is never listed, with a `--transform` putting every member under one root folder named after the archive), keeping the newest `AppData.DataBackupsKept` and never touching `db/.backups`; its own "Restore from file" uploads a tar.gz (`upload-restore-data.ts`, gzip header checked first). `restore-data-backup.ts` extracts an archive to a temp dir, requires exactly one root folder holding a `db` folder (`findArchiveRoot`, since an upload is saved under a fresh timestamp and its inner name no longer matches the file name), takes a safety archive before touching anything (`write-data-backup-archive.ts`, not the locked `create-data-backup.ts`), empties `DATA_DIR` (the snapshots included) and copies the archive's in, then `restart-app.ts` restarts the process through the docker socket after the response lands (`process.exit`, the container's restart policy brings it back), because the open SQLite handle points at the replaced file. Row actions are icon-only (`RowIconButton`, `src/components/general/row-icon-button.tsx`): restore, download, delete. `BACKUP_ARCHIVE_DIR` is a `ProcessOnlyEnvKeys` start-up path (the installer mounts `$COBBLE_HOME/backups-archives` there; outside docker it falls back to `~/.cobble/backups`). Every backup and restore, of either kind, runs one at a time via `with-backup-run-lock.ts`. Helpers in `src/functions/backend/backups/`, API routes in `api/admin/backups/` - `/admin/settings/backups` - super admin only (listed in `SettingsPages`), two groups. Database snapshots: create, list, restore, download and delete in `db/.backups` (`bun-sqlite backup`); "Restore from file" beside "Create backup" opens `upload-restore-modal.tsx` (upload a SQLite database file, which is saved to `.backups` then restored with `bun-sqlite import <file>`, since `restore` is interactive). Whole-data archives: `write-data-backup-archive.ts` tars all of `DATA_DIR` (database, media, computer homes, bot and workspace files, MCP installs) into `BACKUP_ARCHIVE_DIR` as `cobble-data-<epoch-ms>.tar.gz` (via `tar -czf` into a `.part` name, renamed on success so a cut-off run is never listed, with a `--transform` putting every member under one root folder named after the archive), keeping the newest `AppData.DataBackupsKept` and never touching `db/.backups`; its own "Restore from file" uploads a tar.gz (`upload-restore-data.ts`, gzip header checked first). `restore-data-backup.ts` extracts an archive to a temp dir, requires exactly one root folder holding a `db` folder (`findArchiveRoot`, since an upload is saved under a fresh timestamp and its inner name no longer matches the file name), takes a safety archive before touching anything (`write-data-backup-archive.ts`, not the locked `create-data-backup.ts`), empties `DATA_DIR` (the snapshots included) and copies the archive's in, then `restart-app.ts` restarts the process through the docker socket after the response lands (`process.exit`, the container's restart policy brings it back), because the open SQLite handle points at the replaced file. Row actions are icon-only (`RowIconButton`, `src/components/general/row-icon-button.tsx`): restore, download, delete. `BACKUP_ARCHIVE_DIR` is a `ProcessOnlyEnvKeys` start-up path (the installer mounts `$COBBLE_HOME/backups-archives` there; outside docker it falls back to `~/.cobble/backups`). Every backup and restore, of either kind, runs one at a time via `with-backup-run-lock.ts`. Helpers in `src/functions/backend/backups/`, API routes in `api/admin/backups/`
- `/admin/settings/database` - super admin only (listed in `SettingsPages`), a read-only browser of the install's SQLite database, no writes anywhere in it. `grabDatabaseTables` reads the catalog once (every table and its columns, SQLite's own `sqlite_*` tables and sqlite-vec's `bot_file_vectors_*` shadow tables left out) and adds each table's `COUNT(*)` and its size from `dbstat` (index pages folded back onto their table). `grabTableRecords` then returns one page (`AppData.DatabaseBrowserPageSize`, 50) of the open table's records newest key first, every value already cut in SQL at `AppData.DatabaseBrowserCellMaxChars` (400) so a table of long texts never ships a megabyte of JSON, and a `truncated` flag per cell; the search matches over the concatenation of every column's text (`buildSearchClause`: every word, or the whole phrase), which is what also works on a virtual table, where `MATCH` and `LIKE` over a vector blob do not. A record's key is its single primary key column, else its rowid, else nothing (a virtual table with no primary key reads as plain cells); clicking a cell's row opens `grabTableRecord` for that key, which returns the record whole up to `AppData.DatabaseBrowserValueMaxChars` (20000), the answer to a cell cut at three lines. `clampPage` puts a page past the end on the last one. The page's state (table, search, mode, page) lives in the address through `databaseHref`, so a table or a search can be linked to; the open table's first page comes with the page props and anything else loads through the API. Helpers in `src/functions/backend/database/` (`build-table-catalog-sql`, `build-table-sizes-sql`, `build-records-sql`, `build-search-clause`, `resolve-table-key`, `build-record-row`, `quote-sql-identifier`), API `api/admin/database/records` and `record` (GET only, `grabSuperAdminUser`), zod in `src/schemas/database-schema.ts`, page under `src/pages/admin/settings/database/` - `/admin/settings/database` - super admin only (listed in `SettingsPages`), a read-only browser of the install's SQLite database, no writes anywhere in it. `grabDatabaseTables` reads the catalog once (every table and its columns, SQLite's own `sqlite_*` tables and sqlite-vec's `bot_file_vectors_*` shadow tables left out) and adds each table's `COUNT(*)` and its size from `dbstat` (index pages folded back onto their table). `grabTableRecords` then returns one page (`AppData.DatabaseBrowserPageSize`, 50) of the open table's records newest key first, every value already cut in SQL at `AppData.DatabaseBrowserCellMaxChars` (400) so a table of long texts never ships a megabyte of JSON, and a `truncated` flag per cell; the search matches over the concatenation of every column's text (`buildSearchClause`: every word, or the whole phrase), which is what also works on a virtual table, where `MATCH` and `LIKE` over a vector blob do not. A record's key is its single primary key column, else its rowid, else nothing (a virtual table with no primary key reads as plain cells); clicking a cell's row opens `grabTableRecord` for that key, which returns the record whole up to `AppData.DatabaseBrowserValueMaxChars` (20000), the answer to a cell cut at three lines. `clampPage` puts a page past the end on the last one. The page's state (table, search, mode, page) lives in the address through `databaseHref`, so a table or a search can be linked to; the open table's first page comes with the page props and anything else loads through the API. Helpers in `src/functions/backend/database/` (`build-table-catalog-sql`, `build-table-sizes-sql`, `build-records-sql`, `build-search-clause`, `resolve-table-key`, `build-record-row`, `quote-sql-identifier`), API `api/admin/database/records` and `record` (GET only, `grabSuperAdminUser`), zod in `src/schemas/database-schema.ts`, page under `src/pages/admin/settings/database/`
- `/admin/settings` (live) - redirects to `SettingsDefaultPath` (`/admin/settings/general`). Sub pages: `general/` (theme: System/Light/Dark per device via `useThemePreference`, stored in `localStorage[AppData.ThemeStorageKey]`, the same key `ThemeInitScript` and twui's `ColorSchemeSelector` read; applies instantly, no save bar; for super admins the prompt appendix (`prompt-section.tsx`, `system-prompt-append-form.tsx`) saving `api/admin/settings/system-prompt-append` into the `bot_system_prompt_append` app setting; and, for `canManageCrew`, the bot file quota: `storage-section.tsx`, `storage-quota-form.tsx` saving `api/admin/settings/storage-quota` into `app_settings`; and, for everyone, the running version: `version-section.tsx`, `version-line.tsx` showing `pageProps.running_version` from `runningVersion` (`src/utils/running-version.ts`: `COBBLE_VERSION`, the release tag the image was built with, else `package.json`'s version, in development too, with `(development)` after it and no such suffix in production), `profile/` (current user profile), `notifications/` (this device's Web Push switch, a test send, and every device the user subscribed), `change-password/` (set/change own password) `models/` (super admin only, `super_admin_only` in the dict hides it from the rail: model providers and which model each harness uses) and `environment/` (super admin only: app settings and the envs bots and computers use, per scope), each wrapped in `SettingsShell` (`(partials)/settings-shell.tsx`: 64px pane header like the conversation header, then one centered column). Content is built from `SettingsGroup` blocks (label + description left, controls right, hairline-separated, no cards) and ends with the sticky `SettingsSaveBar` (Save enabled only when dirty, Discard, one-line status). Saves don't reload the page. Keep save notices in component state: `useFormInit` clears `status` whenever the form changes. `bots/` (the crew: chat, edit with `BotForm`, models, delete a worker via `api/admin/bots/delete-bot`), `workspaces/` (open, create through the Add flow, delete), `computers/` (state and resources, live; super admins create, edit with `AddComputerForm`, retry, delete), `mcps/`, `tools/` and `skills/` (see MCPs, custom tools and skills: super admins add, edit, refresh or try, delete; admins only set Access, `extension-access-modal.tsx`; add, edit, access and try open in modals (`AddModalShell`). A skill has no status: nothing connects or runs, so `skill-row.tsx` puts its slug, who gets it (`extension-access-label.ts`) and its description where a server's or tool's status line goes), each MCP server or tool row with a status line under it (`extension-status-line.tsx`: connected or not, last run worked or failed, when, and why), those five for `canManageCrew` only (`grab-crew-settings-server-props.ts`, `can_view` in the dict) and `users/` (for `canListUsers`: super admins and admins; the dict's `can_view` gates it in the rail; add with a password (`canCreateUser`), edit (`canUpdateUser`), both with the one `UserForm` (`admin/settings/users/(partials)/user-form/`, `useFormInit`, photo included, saved in one request), revoke (adds `revoked`, `canRevokeUser`), delete (`canRevokeUser`'s rules, never yourself), pick their user types from `UserTypes` (`user-types-select.tsx`: only the ones `filterAssignableUserTypes` allows, never `UneditableUserTypes`; at least one, since login needs a type; `saveUser` checks each grant and removal with `userTypeAssignment`; the super admin's types are never shown or changed there, and they can't be revoked; the password is entered twice, `SaveUserRequestSchema` checks they match), delete with everything theirs; `api/admin/users/save-user`, `delete-user`). The lists share `SettingsList`, `SettingsListRow`, `SettingsRowAction` and `use-settings-list`. Sub pages are declared once in `src/dict/settings-pages-dict.ts`; on every `/admin/settings/*` page the rail swaps the crew for a back link plus that list. Add a sub page by adding its folder and a dict entry - `/admin/settings` (live) - redirects to `SettingsDefaultPath` (`/admin/settings/general`). Sub pages: `general/` (theme: System/Light/Dark per device via `useThemePreference`, stored in `localStorage[AppData.ThemeStorageKey]`, the same key `ThemeInitScript` and twui's `ColorSchemeSelector` read; applies instantly, no save bar; for super admins the prompt appendix (`prompt-section.tsx`, `system-prompt-append-form.tsx`) saving `api/admin/settings/system-prompt-append` into the `bot_system_prompt_append` app setting, and the run limits (`run-limits-section.tsx`, `run-limits-form.tsx`) saving `api/admin/settings/run-limits` into the `harness_max_tool_steps` and `bot_reply_max_tool_steps` app settings (`grabRunLimitsOverview`, `run_limits` page prop, 0 clearing a limit back to its code default); and, for `canManageCrew`, the bot file quota: `storage-section.tsx`, `storage-quota-form.tsx` saving `api/admin/settings/storage-quota` into `app_settings`; and, for everyone, the running version: `version-section.tsx`, `version-line.tsx` showing `pageProps.running_version` from `runningVersion` (`src/utils/running-version.ts`: `COBBLE_VERSION`, the release tag the image was built with, else `package.json`'s version, in development too, with `(development)` after it and no such suffix in production), `profile/` (current user profile), `notifications/` (this device's Web Push switch, a test send, and every device the user subscribed), `change-password/` (set/change own password) `models/` (super admin only, `super_admin_only` in the dict hides it from the rail: model providers and which model each harness uses) and `environment/` (super admin only: app settings and the envs bots and computers use, per scope), each wrapped in `SettingsShell` (`(partials)/settings-shell.tsx`: 64px pane header like the conversation header, then one centered column that opens with the breadcrumb trail, `src/components/general/admin-breadcrumbs.tsx`: twui's `Breadcrumbs` reading `Home`, the sections and the page off `pageProps.url.pathname`, the last crumb named by the page's own `title` so a record reads `Scout` rather than `3` and `/admin/logs` reads `Activity`; `hasBreadcrumbTrail` drops it only on the crew of bots, which is home, and nothing outside `/admin` gets one). Content is built from `SettingsGroup` blocks (label + description left, controls right, hairline-separated, no cards) and ends with the sticky `SettingsSaveBar` (Save enabled only when dirty, Discard, one-line status). Saves don't reload the page. Keep save notices in component state: `useFormInit` clears `status` whenever the form changes. `bots/` (the crew: chat, edit with `BotForm`, models, delete a worker via `api/admin/bots/delete-bot`), `workspaces/` (open, create through the Add flow, delete), `computers/` (state and resources, live; super admins create, edit with `AddComputerForm`, retry, delete), `mcps/`, `tools/` and `skills/` (see MCPs, custom tools and skills: super admins add, edit, refresh or try, delete; admins only set Access, `extension-access-modal.tsx`; add, edit, access and try open in modals (`AddModalShell`). A skill has no status: nothing connects or runs, so `skill-row.tsx` puts its slug, who gets it (`extension-access-label.ts`) and its description where a server's or tool's status line goes), each MCP server or tool row with a status line under it (`extension-status-line.tsx`: connected or not, last run worked or failed, when, and why), those five for `canManageCrew` only (`grab-crew-settings-server-props.ts`, `can_view` in the dict) and `users/` (for `canListUsers`: super admins and admins; the dict's `can_view` gates it in the rail; add with a password (`canCreateUser`), edit (`canUpdateUser`), both with the one `UserForm` (`admin/settings/users/(partials)/user-form/`, `useFormInit`, photo included, saved in one request), revoke (adds `revoked`, `canRevokeUser`), delete (`canRevokeUser`'s rules, never yourself), pick their user types from `UserTypes` (`user-types-select.tsx`: only the ones `filterAssignableUserTypes` allows, never `UneditableUserTypes`; at least one, since login needs a type; `saveUser` checks each grant and removal with `userTypeAssignment`; the super admin's types are never shown or changed there, and they can't be revoked; the password is entered twice, `SaveUserRequestSchema` checks they match), delete with everything theirs; `api/admin/users/save-user`, `delete-user`). The lists share `SettingsList`, `SettingsListRow`, `SettingsRowAction` and `use-settings-list`. Sub pages are declared once in `src/dict/settings-pages-dict.ts`; on every `/admin/settings/*` page the rail swaps the crew for a back link plus that list. Add a sub page by adding its folder and a dict entry
Every `/admin/*` page renders in the bots layout (`src/layouts/bots/`); there is no separate admin layout or aside. The rail header's Add icon (super admins and admins only, like the rail's "+" buttons) opens the Add flow (`(partials)/add-flow.tsx`, `AddFlowContext`): a bot, a workspace or (super admins) a computer, each form in a modal stacked over the choice, so closing it goes back there (the Bots and Workspaces labels have a "+" that opens the form directly; in a workspace's rail, "Workspace bots" has a "+" that opens `link-bots-modal.tsx` instead: the crew's workers not in it yet (`api/admin/workspaces/linkable-bots`), each added with `add-bot`, which pushes `server:bot-created`, and "New bot" stacking the bot form over it); on a workspace's pages new bots and computers join it. The rail shows the crew (`(partials)/rail-crew-list.tsx`, needs `bots` and `workspaces` in page props): the coordinator, the latest `AppData.RailRecentBots` workers (plus any open, unread or needing you, `pick-rail-bots`), "See all bots" with a few stones opening every bot in a modal, then Workspaces with a "+". On `/admin/workspaces/*` it shows the workspace's crew instead (`(partials)/rail-workspace-list.tsx`: back link, the workspace, then only the bots attached to it, "Workspace bots"; the coordinator stays in the main crew rail, and on a workspace bot's page the back link returns to the workspace page), except on `/admin/settings/*` where it shows "Back to bots" plus the settings sub pages (`(partials)/rail-settings-list.tsx`). Page heroes use the shared `AdminHero` component (`src/components/general/admin-hero.tsx`). Every `/admin/*` page renders in the bots layout (`src/layouts/bots/`); there is no separate admin layout or aside. The rail header's Add icon (super admins and admins only, like the rail's "+" buttons) opens the Add flow (`(partials)/add-flow.tsx`, `AddFlowContext`): a bot, a workspace or (super admins) a computer, each form in a modal stacked over the choice, so closing it goes back there (the Bots and Workspaces labels have a "+" that opens the form directly; in a workspace's rail, "Workspace bots" has a "+" that opens `link-bots-modal.tsx` instead: the crew's workers not in it yet (`api/admin/workspaces/linkable-bots`), each added with `add-bot`, which pushes `server:bot-created`, and "New bot" stacking the bot form over it); on a workspace's pages new bots and computers join it. The rail shows the crew (`(partials)/rail-crew-list.tsx`, needs `bots` and `workspaces` in page props): the coordinator, the latest `AppData.RailRecentBots` workers (plus any open, unread or needing you, `pick-rail-bots`), "See all bots" with a few stones opening every bot in a modal, then Workspaces with a "+". On `/admin/workspaces/*` it shows the workspace's crew instead (`(partials)/rail-workspace-list.tsx`: back link, the workspace, then only the bots attached to it, "Workspace bots"; the coordinator stays in the main crew rail, and on a workspace bot's page the back link returns to the workspace page), except on `/admin/settings/*` where it shows "Back to bots" plus the settings sub pages (`(partials)/rail-settings-list.tsx`). Page heroes use the shared `AdminHero` component (`src/components/general/admin-hero.tsx`).
@@ -269,6 +282,7 @@ Server functions use `index.server.ts` alongside `index.tsx` (e.g. `single-docum
- Shared: `Logo`, `LogoIcon`, `InnerPageHeroSection`, `AdminHero`, `LoginShell` (auth pages/forms), `TaglineSection` - Shared: `Logo`, `LogoIcon`, `InnerPageHeroSection`, `AdminHero`, `LoginShell` (auth pages/forms), `TaglineSection`
- **UNCHANGEABLE RULE - twui components always win over raw JSX elements.** Use twui components (`H1`-`H5`, `P`, `Span`, `List`, `Img`, `Row`, `Stack`, `Container`, `Section`, `Card`, `Button`, `Tag`, `Link`, etc.) before basic JSX elements (`<h1>`-`<h6>`, `<p>`, `<span>`, `<ul>`, `<li>`, `<img>`, `<a>`, `<button>`, `<div>`-as-card, etc.). Never introduce or reintroduce a raw JSX element where a twui equivalent exists. Override twui defaults via `className` rather than reaching for the raw element. - **UNCHANGEABLE RULE - twui components always win over raw JSX elements.** Use twui components (`H1`-`H5`, `P`, `Span`, `List`, `Img`, `Row`, `Stack`, `Container`, `Section`, `Card`, `Button`, `Tag`, `Link`, etc.) before basic JSX elements (`<h1>`-`<h6>`, `<p>`, `<span>`, `<ul>`, `<li>`, `<img>`, `<a>`, `<button>`, `<div>`-as-card, etc.). Never introduce or reintroduce a raw JSX element where a twui equivalent exists. Override twui defaults via `className` rather than reaching for the raw element.
- **Never create new CSS classes, CSS files, or inline `style={{...}}` hacks.** Style exclusively by composing existing component props/classNames, existing twui classNames (`twui-*`), semantic theme-token utilities (`bg-dark`, `bg-light`, `text-dark`, `text-primary`, `bg-primary`, `text-accent`, `bg-accent`, `text-secondary`, `border-slate-200`, etc.), and utility classes already used elsewhere in this codebase. Build up from what exists; extend component code only when the existing API cannot express the need. - **Never create new CSS classes, CSS files, or inline `style={{...}}` hacks.** Style exclusively by composing existing component props/classNames, existing twui classNames (`twui-*`), semantic theme-token utilities (`bg-dark`, `bg-light`, `text-dark`, `text-primary`, `bg-primary`, `text-accent`, `bg-accent`, `text-secondary`, `border-slate-200`, etc.), and utility classes already used elsewhere in this codebase. Build up from what exists; extend component code only when the existing API cannot express the need.
- **Chunk Tailwind classes with `tailwind-merge`, never one long line.** A `className` is `twMerge(...)` (`import { twMerge } from "tailwind-merge"`) with one chunk per claim: base layout and spacing first, then one string per conditional, and a caller's override last, so a later chunk always beats an earlier one instead of depending on source order in the CSS. Never concatenate conditional classes with a template literal or a bare `+`, and never leave a single 20-class string where the parts mean different things: `className={twMerge("w-full shrink-0 flex-col gap-0! items-stretch rounded-default py-1", selected && "bg-ink/5")}`.
- **Theming:** colors, radii, and fonts change only in `src/styles/main.css`: Cobble tokens as CSS vars on `:root` (light) and `.dark`, exposed in the `@theme inline` block as `paper`, `surface`, `sunken`, `ink`, `ink-secondary`, `ink-disabled`, `hairline`, `hairline-strong`, `on-ink`, `hue-<name>`, plus `shadow-popover` and `--m-fast/base/slow` motion vars (zeroed under reduced motion). Font is Lato (stand-in). Never hardcode hex colors, radii, or font stacks inside components. Hues are bot identity only, never state, and the `--color-hue-*` tokens remain for the fixed palettes that use them (the files browser's volume icons, `BotHues`' own hexes). - **Theming:** colors, radii, and fonts change only in `src/styles/main.css`: Cobble tokens as CSS vars on `:root` (light) and `.dark`, exposed in the `@theme inline` block as `paper`, `surface`, `sunken`, `ink`, `ink-secondary`, `ink-disabled`, `hairline`, `hairline-strong`, `on-ink`, `hue-<name>`, plus `shadow-popover` and `--m-fast/base/slow` motion vars (zeroed under reduced motion). Font is Lato (stand-in). Never hardcode hex colors, radii, or font stacks inside components. Hues are bot identity only, never state, and the `--color-hue-*` tokens remain for the fixed palettes that use them (the files browser's volume icons, `BotHues`' own hexes).
- **The one exception:** a bot's colour is user data, not a theme value, so it reaches `BotStone` (`src/components/general/bot-stone.tsx`) and `UnreadDot` as an SVG `fill` attribute - an arbitrary hex has no token to point at. Nothing else in the app may take a colour that way, and `resolveBotColor`/`normalizeBotColor` (`src/utils/bot-color.ts`) are the only place a colour string is validated. - **The one exception:** a bot's colour is user data, not a theme value, so it reaches `BotStone` (`src/components/general/bot-stone.tsx`) and `UnreadDot` as an SVG `fill` attribute - an arbitrary hex has no token to point at. Nothing else in the app may take a colour that way, and `resolveBotColor`/`normalizeBotColor` (`src/utils/bot-color.ts`) are the only place a colour string is validated.
- **Branding:** `docs/readme/hero-light.png`, `hero-dark.png`, `workspace-light.png` and `workspace-dark.png` are screenshots of the current app (chat and workspace dashboard, 1440px at 2x, downscaled to 1600px wide), committed and picked per theme by GitHub's `#gh-light-mode-only` / `#gh-dark-mode-only` fragment. They are not generated: when the UI changes, retake them from a seeded dev instance (`bun run seed`, `bun run seed:bots`) and keep the same names, so the README never shows an older dashboard. Never use `assets/handoff/assets/screens/*.png` for anything: that design is outdated. `public/logos/logo.svg` is the single source of truth for the logo mark; `logo-icon.tsx` renders it inline (via `BotStone`) and `logo.tsx` renders the stone beside the vector `Wordmark`. Whenever the logo changes, run `bun run gen:favicon` to regenerate `public/favicon.ico`, `public/favicon.png`, `public/apple-touch-icon.png`, and `public/logos/logo.png` (the last is used in emails). Font + favicon links live in `__root.tsx` `Head()`. - **Branding:** `docs/readme/hero-light.png`, `hero-dark.png`, `workspace-light.png` and `workspace-dark.png` are screenshots of the current app (chat and workspace dashboard, 1440px at 2x, downscaled to 1600px wide), committed and picked per theme by GitHub's `#gh-light-mode-only` / `#gh-dark-mode-only` fragment. They are not generated: when the UI changes, retake them from a seeded dev instance (`bun run seed`, `bun run seed:bots`) and keep the same names, so the README never shows an older dashboard. Never use `assets/handoff/assets/screens/*.png` for anything: that design is outdated. `public/logos/logo.svg` is the single source of truth for the logo mark; `logo-icon.tsx` renders it inline (via `BotStone`) and `logo.tsx` renders the stone beside the vector `Wordmark`. Whenever the logo changes, run `bun run gen:favicon` to regenerate `public/favicon.ico`, `public/favicon.png`, `public/apple-touch-icon.png`, and `public/logos/logo.png` (the last is used in emails). Font + favicon links live in `__root.tsx` `Head()`.
README.md
+21 -10
View File
@@ -5,7 +5,7 @@
<img src="docs/readme/lockup-dark.svg#gh-dark-mode-only" alt="Cobble" height="56"> <img src="docs/readme/lockup-dark.svg#gh-dark-mode-only" alt="Cobble" height="56">
</h1> </h1>
A crew of small AI worker bots you host yourself. A crew of small AI worker bots you host yourself (alpha).
![Cobble: a crew of small AI worker bots you host yourself](docs/readme/hero-light.png#gh-light-mode-only) ![Cobble: a crew of small AI worker bots you host yourself](docs/readme/hero-light.png#gh-light-mode-only)
![Cobble: a crew of small AI worker bots you host yourself](docs/readme/hero-dark.png#gh-dark-mode-only) ![Cobble: a crew of small AI worker bots you host yourself](docs/readme/hero-dark.png#gh-dark-mode-only)
@@ -19,6 +19,13 @@ A crew of small AI worker bots you host yourself.
</div> </div>
> **Alpha.** Cobble is still in alpha: interfaces, database schema and
> settings change between releases, and things break. Install it to try it
> out and to tell us what is wrong, but keep your own backups (Settings >
> Backups) and do not put it in front of work you cannot lose. Every release
> is a real release: the installer updates an existing install in place and
> keeps its data.
Cobble is a self-hosted crew of small AI worker bots. You get a coordinator Cobble is a self-hosted crew of small AI worker bots. You get a coordinator
and a set of workers, each with its own job, models, memory and workspace. and a set of workers, each with its own job, models, memory and workspace.
They talk to you in one chat, share workspaces, run routines on a schedule, They talk to you in one chat, share workspaces, run routines on a schedule,
@@ -49,6 +56,7 @@ your data on your own machine.
| Crew | One coordinator and as many workers as you need, each with a name, a role, a hue and its own core instructions | | Crew | One coordinator and as many workers as you need, each with a name, a role, a hue and its own core instructions |
| Rail | The coordinator, your recent workers and their latest sessions, then your workspaces, each row carrying its status line | | Rail | The coordinator, your recent workers and their latest sessions, then your workspaces, each row carrying its status line |
| Chat | Markdown with @-mentions for bots and #-mentions for workspaces, replies and quotes, reactions, attachments, older messages on scroll, drafts kept locally | | Chat | Markdown with @-mentions for bots and #-mentions for workspaces, replies and quotes, reactions, attachments, older messages on scroll, drafts kept locally |
| Stop | Stop a bot from the thinking row or the header menu: its queued and running work stops at once, everywhere you chat with it, and it goes idle |
| Header | The bot's model, provider and computer at a glance, one click from either, and its picture, instructions and settings on the avatar | | Header | The bot's model, provider and computer at a glance, one click from either, and its picture, instructions and settings on the avatar |
| Workspaces | A shared home for a group of bots: shared context, per-workspace instructions and memory, a default computer, and a dashboard | | Workspaces | A shared home for a group of bots: shared context, per-workspace instructions and memory, a default computer, and a dashboard |
| Memory | Each bot keeps core memory, memory about you, and workspace memory, and rewrites it as it works | | Memory | Each bot keeps core memory, memory about you, and workspace memory, and rewrites it as it works |
@@ -237,15 +245,16 @@ queue and Web Push.
## Images ## Images
Every release publishes a multi-arch image to both registries: Every release publishes a multi-arch image to
`ghcr.io/moduletrace/cobble`, which is what the installer pulls. Tags are
`latest`, the release version, `major.minor` and the commit sha.
- `ghcr.io/moduletrace/cobble` The same image is copied into `git.tben.me/moduletrace/cobble` by
- `git.tben.me/moduletrace/cobble` `bun run mirror:gitea`, which runs after the build over the VPN (the Gitea
registry is not on the public internet, so GitHub Actions cannot reach it).
Tags are `latest`, the release version, `major.minor` and the commit sha. Install from Gitea with `--image git.tben.me/moduletrace/cobble`, or build
Install from Gitea with from source with `docker build -t cobble:local .` and
`--image git.tben.me/moduletrace/cobble`, or build from source with `--image cobble --version local --no-pull`.
`docker build -t cobble:local .` and `--image cobble --version local --no-pull`.
A release is published from a clean `main` with the two scripts, which tag the A release is published from a clean `main` with the two scripts, which tag the
`package.json` version, push and create the release objects: `package.json` version, push and create the release objects:
@@ -256,7 +265,9 @@ scripts/publish-gitea.sh # or: bun run publish:gitea
``` ```
Both take `--version VERSION`, `--dry-run` and `--yes`. Only the GitHub release Both take `--version VERSION`, `--dry-run` and `--yes`. Only the GitHub release
builds the image; see [RELEASING.md](RELEASING.md) for the full checklist. builds the image, and the Gitea registry is filled in afterwards, once that
build has finished and from the VPN, by `bun run mirror:gitea`; see
[RELEASING.md](RELEASING.md) for the full checklist.
## Development ## Development
RELEASING.md
+20 -6
View File
@@ -3,6 +3,12 @@
Cobble ships as a multi-arch container image and a single `install.sh`. Cobble ships as a multi-arch container image and a single `install.sh`.
The repository starts private and is made public at the launch gate below. The repository starts private and is made public at the launch gate below.
Cobble is in **alpha** (see the note in `README.md`). Every release published
before v1.0.0 is an alpha release: interfaces, the database schema and the
settings may change between them, so a release that needs a migration step
has to say so in its notes, and Settings > General keeps saying the stage
until v1.0.0.
## One-time setup ## One-time setup
- [ ] `GITEA_USERNAME` and `GITEA_TOKEN` are set as GitHub Actions secrets. - [ ] `GITEA_USERNAME` and `GITEA_TOKEN` are set as GitHub Actions secrets.
@@ -56,16 +62,24 @@ them in that order, since only the GitHub release builds images.
gh release create v0.1.0 --title v0.1.0 --notes-file notes.md --latest gh release create v0.1.0 --title v0.1.0 --notes-file notes.md --latest
7. [ ] The release workflow builds `linux/amd64` and `linux/arm64` and 7. [ ] The release workflow builds `linux/amd64` and `linux/arm64` and
pushes to both registries: pushes to `ghcr.io/moduletrace/cobble`, the registry the installer
- `ghcr.io/moduletrace/cobble` defaults to. It does not touch Gitea: that registry is on the VPN,
- `git.tben.me/moduletrace/cobble` and a GitHub runner cannot reach it.
8. [ ] Check the manifests: 8. [ ] Check the manifests:
docker buildx imagetools inspect ghcr.io/moduletrace/cobble:latest docker buildx imagetools inspect ghcr.io/moduletrace/cobble:latest
docker buildx imagetools inspect git.tben.me/moduletrace/cobble:latest
9. [ ] Confirm the GHCR package is public and pull it anonymously. 9. [ ] Confirm the GHCR package is public and pull it anonymously.
10. [ ] Publish the release to Gitea as well: 10. [ ] Once the build has finished, copy the image into the Gitea registry
from the VPN:
GITEA_TOKEN=... bun run mirror:gitea # or: scripts/mirror-gitea.sh
It copies the version and commit tags, plus `latest` and
`major.minor` while this release still owns them, and reads every
destination digest back. A failure here does not affect the release:
the installer keeps working from GHCR.
11. [ ] Publish the release to Gitea as well:
scripts/publish-gitea.sh # or: bun run publish:gitea scripts/publish-gitea.sh # or: bun run publish:gitea
@@ -73,7 +87,7 @@ them in that order, since only the GitHub release builds images.
git push gitea main git push gitea main
git push gitea refs/tags/v0.1.0 git push gitea refs/tags/v0.1.0
11. [ ] Update every install with the one-liner: 12. [ ] Update every install with the one-liner:
curl -fsSL https://url.moduletrace.com/cobble | sudo bash curl -fsSL https://url.moduletrace.com/cobble | sudo bash
db/schema.ts
+143
View File
@@ -2866,6 +2866,149 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
}, },
], ],
}, },
{
tableName: "token_usages",
tableDescription: `One model call that reported its usage: the tokens it spent, on which model and provider, for whom and for what. A bot message's total is the sum of the calls it took; the usage page sums them by model, provider and period`,
fields: [
{
fieldName: "user_id",
fieldDescription: `Whose work it was: the user whose conversation the call ran for`,
dataType: "INTEGER",
foreignKey: {
cascadeDelete: true,
destinationTableName: "users",
destinationTableColumnName: "id",
destinationTableColumnType: "INTEGER",
foreignKeyName: "token_usage_user_fk",
},
},
{
fieldName: "bot_id",
fieldDescription: `The bot it was for, when it was for one`,
dataType: "INTEGER",
foreignKey: {
cascadeDelete: true,
destinationTableName: "bots",
destinationTableColumnName: "id",
destinationTableColumnType: "INTEGER",
foreignKeyName: "token_usage_bot_fk",
},
},
{
fieldName: "workspace_id",
fieldDescription: `The workspace the call was in; null for a bot's own chat`,
dataType: "INTEGER",
foreignKey: {
cascadeDelete: true,
destinationTableName: "workspaces",
destinationTableColumnName: "id",
destinationTableColumnType: "INTEGER",
foreignKeyName: "token_usage_workspace_fk",
},
},
{
fieldName: "session_id",
fieldDescription: `The conversation it was in`,
dataType: "INTEGER",
foreignKey: {
cascadeDelete: true,
destinationTableName: "bot_sessions",
destinationTableColumnName: "id",
destinationTableColumnType: "INTEGER",
foreignKeyName: "token_usage_session_fk",
},
},
{
fieldName: "message_id",
fieldDescription: `The message it produced, so the thread can show what an answer cost. Null for calls that produce no message (naming a session, embedding a file)`,
dataType: "INTEGER",
foreignKey: {
cascadeDelete: true,
destinationTableName: "bot_messages",
destinationTableColumnName: "id",
destinationTableColumnType: "INTEGER",
foreignKeyName: "token_usage_message_fk",
},
},
{
fieldName: "job_id",
fieldDescription: `The queued job it ran for, when it ran for one`,
dataType: "INTEGER",
foreignKey: {
cascadeDelete: true,
destinationTableName: "bots_queue",
destinationTableColumnName: "id",
destinationTableColumnType: "INTEGER",
foreignKeyName: "token_usage_job_fk",
},
},
{
fieldName: "harness",
fieldDescription: `What the model was used for: a chat turn, reading media, transcribing, speaking, making an image, embedding`,
dataType: "TEXT",
options: Harnesses.map((h) => h.value),
},
{
fieldName: "provider_id",
fieldDescription: `The provider that served it. Not a foreign key: it stays readable after the provider is deleted`,
dataType: "INTEGER",
},
{
fieldName: "provider_name",
fieldDescription: `The provider's name as it was, so a deleted provider still reads`,
dataType: "TEXT",
},
{
fieldName: "model_id",
fieldDescription: `The catalog row of the model. Not a foreign key: a model the provider stops listing still counts`,
dataType: "INTEGER",
},
{
fieldName: "model_slug",
fieldDescription: `The provider's model id, as it was called`,
dataType: "TEXT",
},
{
fieldName: "prompt_tokens",
fieldDescription: `Tokens sent: the prompt, the system instructions and the tool results`,
dataType: "INTEGER",
defaultValue: 0,
},
{
fieldName: "completion_tokens",
fieldDescription: `Tokens written back: the answer, its tool calls and its reasoning`,
dataType: "INTEGER",
defaultValue: 0,
},
{
fieldName: "cost",
fieldDescription: `The provider's price in USD, when it reports one`,
dataType: "REAL",
},
],
indexes: [
{
indexName: "idx_token_usages_created_at",
indexTableFields: ["created_at"],
},
{
indexName: "idx_token_usages_message_id",
indexTableFields: ["message_id"],
},
{
indexName: "idx_token_usages_provider_model",
indexTableFields: ["provider_id", "model_slug"],
},
{
indexName: "idx_token_usages_user_created_at",
indexTableFields: ["user_id", "created_at"],
},
{
indexName: "idx_token_usages_bot_created_at",
indexTableFields: ["bot_id", "created_at"],
},
],
},
{ {
tableName: "bot_file_vectors", tableName: "bot_file_vectors",
tableDescription: `sqlite-vec vectors of a file's chunks, at \`AppData.EmbeddingDimensions\` wide whatever the model returns. A vec0 table is only recreated with \`bunx bun-sqlite schema --vector\`, so the entrypoint never passes it`, tableDescription: `sqlite-vec vectors of a file's chunks, at \`AppData.EmbeddingDimensions\` wide whatever the model returns. A vec0 table is only recreated with \`bunx bun-sqlite schema --vector\`, so the entrypoint never passes it`,
db/types/db.ts
+73 -1
View File
@@ -48,6 +48,7 @@ export const BunSQLiteTables = [
"envs", "envs",
"bot_files", "bot_files",
"bot_file_chunks", "bot_file_chunks",
"token_usages",
"bot_file_vectors", "bot_file_vectors",
"file_ingest_queue", "file_ingest_queue",
"app_settings", "app_settings",
@@ -1717,6 +1718,77 @@ export type BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS = {
token_estimate?: number | ""; token_estimate?: number | "";
} }
export type BUN_SQLITE_COBBLE_TOKEN_USAGES = {
/**
* The unique identifier of the record.
*/
id?: number | "";
/**
* The time when the record was created. (Unix Timestamp)
*/
created_at?: number | "";
/**
* The time when the record was updated. (Unix Timestamp)
*/
updated_at?: number | "";
/**
* Whose work it was: the user whose conversation the call ran for
*/
user_id?: number | "";
/**
* The bot it was for, when it was for one
*/
bot_id?: number | "";
/**
* The workspace the call was in; null for a bot's own chat
*/
workspace_id?: number | "";
/**
* The conversation it was in
*/
session_id?: number | "";
/**
* The message it produced, so the thread can show what an answer cost. Null for calls that produce no message (naming a session, embedding a file)
*/
message_id?: number | "";
/**
* The queued job it ran for, when it ran for one
*/
job_id?: number | "";
/**
* What the model was used for: a chat turn, reading media, transcribing, speaking, making an image, embedding
*/
harness?: "text" | "vision" | "transcription" | "speech" | "image" | "embedding" | "";
/**
* The provider that served it. Not a foreign key: it stays readable after the provider is deleted
*/
provider_id?: number | "";
/**
* The provider's name as it was, so a deleted provider still reads
*/
provider_name?: string;
/**
* The catalog row of the model. Not a foreign key: a model the provider stops listing still counts
*/
model_id?: number | "";
/**
* The provider's model id, as it was called
*/
model_slug?: string;
/**
* Tokens sent: the prompt, the system instructions and the tool results
*/
prompt_tokens?: number | "";
/**
* Tokens written back: the answer, its tool calls and its reasoning
*/
completion_tokens?: number | "";
/**
* The provider's price in USD, when it reports one
*/
cost?: number | "";
}
export type BUN_SQLITE_COBBLE_BOT_FILE_VECTORS = { export type BUN_SQLITE_COBBLE_BOT_FILE_VECTORS = {
/** /**
* The unique identifier of the record. * The unique identifier of the record.
@@ -1800,4 +1872,4 @@ export type BUN_SQLITE_COBBLE_APP_SETTINGS = {
value?: string; value?: string;
} }
export type BUN_SQLITE_COBBLE_ALL_TYPEDEFS = BUN_SQLITE_COBBLE_USERS & BUN_SQLITE_COBBLE_USER_TYPES & BUN_SQLITE_COBBLE_SSO_LOGIN_CODES & BUN_SQLITE_COBBLE_USER_PASSWORDS & BUN_SQLITE_COBBLE_MEDIA & BUN_SQLITE_COBBLE_MEDIA_PARADIGMS & BUN_SQLITE_COBBLE_BOTS & BUN_SQLITE_COBBLE_BOT_SESSIONS & BUN_SQLITE_COBBLE_BOT_MESSAGES & BUN_SQLITE_COBBLE_BOT_MESSAGE_REACTIONS & BUN_SQLITE_COBBLE_BOT_TASKS & BUN_SQLITE_COBBLE_BOT_TASK_STEPS & BUN_SQLITE_COBBLE_TASKS & BUN_SQLITE_COBBLE_TASK_MEDIA & BUN_SQLITE_COBBLE_TASK_NOTES & BUN_SQLITE_COBBLE_BOTS_QUEUE & BUN_SQLITE_COBBLE_BOTS_QUEUE_UPDATES & BUN_SQLITE_COBBLE_CRON_JOBS & BUN_SQLITE_COBBLE_WORKSPACES & BUN_SQLITE_COBBLE_WORKSPACE_BOTS & BUN_SQLITE_COBBLE_WORKSPACE_USERS & BUN_SQLITE_COBBLE_BOT_USERS & BUN_SQLITE_COBBLE_BOT_USER_MEMORIES & BUN_SQLITE_COBBLE_BOT_USER_FACES & BUN_SQLITE_COBBLE_BOT_MESSAGE_MEDIA & BUN_SQLITE_COBBLE_CONTAINERS & BUN_SQLITE_COBBLE_COMPUTERS & BUN_SQLITE_COBBLE_BOT_COMPUTERS & BUN_SQLITE_COBBLE_WORKSPACE_COMPUTERS & BUN_SQLITE_COBBLE_HARNESS_PROVIDERS & BUN_SQLITE_COBBLE_HARNESS_MODELS & BUN_SQLITE_COBBLE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_BOT_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_WORKSPACE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_MCP_SERVERS & BUN_SQLITE_COBBLE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_SKILLS & BUN_SQLITE_COBBLE_BOT_MCP_SERVERS & BUN_SQLITE_COBBLE_WORKSPACE_MCP_SERVERS & BUN_SQLITE_COBBLE_BOT_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_WORKSPACE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_BOT_SKILLS & BUN_SQLITE_COBBLE_WORKSPACE_SKILLS & BUN_SQLITE_COBBLE_LOGS & BUN_SQLITE_COBBLE_USER_PRESENCE & BUN_SQLITE_COBBLE_PUSH_SUBSCRIPTIONS & BUN_SQLITE_COBBLE_ENVS & BUN_SQLITE_COBBLE_BOT_FILES & BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS & BUN_SQLITE_COBBLE_BOT_FILE_VECTORS & BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE & BUN_SQLITE_COBBLE_APP_SETTINGS export type BUN_SQLITE_COBBLE_ALL_TYPEDEFS = BUN_SQLITE_COBBLE_USERS & BUN_SQLITE_COBBLE_USER_TYPES & BUN_SQLITE_COBBLE_SSO_LOGIN_CODES & BUN_SQLITE_COBBLE_USER_PASSWORDS & BUN_SQLITE_COBBLE_MEDIA & BUN_SQLITE_COBBLE_MEDIA_PARADIGMS & BUN_SQLITE_COBBLE_BOTS & BUN_SQLITE_COBBLE_BOT_SESSIONS & BUN_SQLITE_COBBLE_BOT_MESSAGES & BUN_SQLITE_COBBLE_BOT_MESSAGE_REACTIONS & BUN_SQLITE_COBBLE_BOT_TASKS & BUN_SQLITE_COBBLE_BOT_TASK_STEPS & BUN_SQLITE_COBBLE_TASKS & BUN_SQLITE_COBBLE_TASK_MEDIA & BUN_SQLITE_COBBLE_TASK_NOTES & BUN_SQLITE_COBBLE_BOTS_QUEUE & BUN_SQLITE_COBBLE_BOTS_QUEUE_UPDATES & BUN_SQLITE_COBBLE_CRON_JOBS & BUN_SQLITE_COBBLE_WORKSPACES & BUN_SQLITE_COBBLE_WORKSPACE_BOTS & BUN_SQLITE_COBBLE_WORKSPACE_USERS & BUN_SQLITE_COBBLE_BOT_USERS & BUN_SQLITE_COBBLE_BOT_USER_MEMORIES & BUN_SQLITE_COBBLE_BOT_USER_FACES & BUN_SQLITE_COBBLE_BOT_MESSAGE_MEDIA & BUN_SQLITE_COBBLE_CONTAINERS & BUN_SQLITE_COBBLE_COMPUTERS & BUN_SQLITE_COBBLE_BOT_COMPUTERS & BUN_SQLITE_COBBLE_WORKSPACE_COMPUTERS & BUN_SQLITE_COBBLE_HARNESS_PROVIDERS & BUN_SQLITE_COBBLE_HARNESS_MODELS & BUN_SQLITE_COBBLE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_BOT_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_WORKSPACE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_MCP_SERVERS & BUN_SQLITE_COBBLE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_SKILLS & BUN_SQLITE_COBBLE_BOT_MCP_SERVERS & BUN_SQLITE_COBBLE_WORKSPACE_MCP_SERVERS & BUN_SQLITE_COBBLE_BOT_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_WORKSPACE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_BOT_SKILLS & BUN_SQLITE_COBBLE_WORKSPACE_SKILLS & BUN_SQLITE_COBBLE_LOGS & BUN_SQLITE_COBBLE_USER_PRESENCE & BUN_SQLITE_COBBLE_PUSH_SUBSCRIPTIONS & BUN_SQLITE_COBBLE_ENVS & BUN_SQLITE_COBBLE_BOT_FILES & BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS & BUN_SQLITE_COBBLE_TOKEN_USAGES & BUN_SQLITE_COBBLE_BOT_FILE_VECTORS & BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE & BUN_SQLITE_COBBLE_APP_SETTINGS
package.json
+3 -2
View File
@@ -1,6 +1,6 @@
{ {
"name": "cobble", "name": "cobble",
"version": "0.1.4", "version": "0.1.7",
"module": "index.ts", "module": "index.ts",
"type": "module", "type": "module",
"private": true, "private": true,
@@ -19,7 +19,8 @@
"test": "bun test", "test": "bun test",
"check:bundle": "bun src/scripts/check-bundle-size.ts", "check:bundle": "bun src/scripts/check-bundle-size.ts",
"publish:github": "bash scripts/publish-github.sh", "publish:github": "bash scripts/publish-github.sh",
"publish:gitea": "bash scripts/publish-gitea.sh" "publish:gitea": "bash scripts/publish-gitea.sh",
"mirror:gitea": "bash scripts/mirror-gitea.sh"
}, },
"devDependencies": { "devDependencies": {
"@next/mdx": "^16.3.0", "@next/mdx": "^16.3.0",
scripts/lib/publish-release.sh
+19 -2
View File
@@ -134,9 +134,24 @@ prepare_tag() {
# END:tag # END:tag
# MARK:notes-file # MARK:notes-file
# Writes release_notes into NOTES_FILE, which the caller cleans up. # Writes release_notes into NOTES_FILE, which the caller cleans up. The
# compare base is the host's /compare/ path; the tag range is added here,
# since only this side knows which tag came before (a bare /compare/ opens
# the host's default comparison rather than this release's).
build_compare_url() {
local compare_base="$1"
local from="$2"
if [ -z "$from" ]; then
printf '%s' "$compare_base"
return 0
fi
printf '%s%s...%s' "$compare_base" "$from" "$TAG"
}
write_notes_file() { write_notes_file() {
local compare_url="$1" local compare_base="$1"
local compare_url=""
local from="" local from=""
# A dry run makes no tag, so the notes are read up to HEAD instead. # A dry run makes no tag, so the notes are read up to HEAD instead.
local to="$TAG" local to="$TAG"
@@ -153,6 +168,8 @@ write_notes_file() {
detail "No earlier tag, so every commit is listed." detail "No earlier tag, so every commit is listed."
fi fi
compare_url="$(build_compare_url "$compare_base" "$from")"
NOTES_FILE="$(mktemp -t cobble-release-notes.XXXXXX)" NOTES_FILE="$(mktemp -t cobble-release-notes.XXXXXX)"
release_notes --from "$from" --to "$to" --compare-url "$compare_url" >"$NOTES_FILE" release_notes --from "$from" --to "$to" --compare-url "$compare_url" >"$NOTES_FILE"
ok "Wrote the release notes to $NOTES_FILE." ok "Wrote the release notes to $NOTES_FILE."
scripts/mirror-gitea.sh
Executable
+318
View File
@@ -0,0 +1,318 @@
#!/usr/bin/env bash
# Copies a published Cobble image from GHCR into the Gitea registry.
#
# scripts/mirror-gitea.sh
#
# The Gitea registry is on the VPN and GitHub Actions runners are on the
# public internet, so CI can never reach it. The release workflow therefore
# builds and pushes to GHCR only, and this script mirrors the result
# separately: a registry that is unreachable, slow or full can never fail a
# release the installer depends on.
#
# It copies from the Gitea host itself, because that host reaches GHCR over
# the public internet and its own registry over the docker network it shares
# with it. Nothing has to be installed on that host: skopeo runs there in a
# container.
#
# `skopeo copy --all` keeps the manifest list and its signatures, so every
# architecture comes across and the digest is unchanged. Each destination tag
# is then read back and compared, so a partial copy is reported as a failure
# instead of being assumed to have worked.
#
# Run it after scripts/publish-github.sh: that release is what builds the
# image, so there is nothing to copy until it has finished.
set -Eeuo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=lib/publish-common.sh
. "$SCRIPT_DIR/lib/publish-common.sh"
# shellcheck source=lib/publish-release.sh
. "$SCRIPT_DIR/lib/publish-release.sh"
SOURCE_REPO="${MIRROR_SOURCE_REPO:-moduletrace/cobble}"
SOURCE_REGISTRY_HOST="${MIRROR_SOURCE_REGISTRY_HOST:-ghcr.io}"
HOST="${MIRROR_HOST:-10.7.0.4}"
HOST_USER="${MIRROR_HOST_USER:-root}"
REGISTRY_HOST="${MIRROR_REGISTRY_HOST:-gitea:3000}"
REGISTRY_REPO="${MIRROR_REGISTRY_REPO:-moduletrace/cobble}"
# The name a person types, which is not the one the copy uses: the host
# reaches its registry by docker network name, while an install reaches it
# through the proxy in front of it.
REGISTRY_PUBLIC_HOST="${MIRROR_REGISTRY_PUBLIC_HOST:-git.tben.me}"
NETWORK="${MIRROR_NETWORK:-cloudflare}"
SKOPEO_IMAGE="${MIRROR_SKOPEO_IMAGE:-quay.io/skopeo/stable:latest}"
MIRROR_USER="${MIRROR_USER:-cobble}"
CONNECT_TIMEOUT="${MIRROR_CONNECT_TIMEOUT:-10}"
usage() {
cat <<EOF
Copies a published Cobble image from GHCR into the Gitea registry.
Usage: scripts/mirror-gitea.sh [options]
Options:
--version VERSION Version to mirror, with or without the leading v.
Defaults to package.json's version.
--yes, -y Do not prompt.
--dry-run Print what would run, copy nothing.
-h, --help Show this help.
The version is copied as vX.Y.Z and sha-<commit>. It is copied as latest and
major.minor too, but only while it is still the newest release in GHCR, so
mirroring an older release never rolls those two tags backwards. Every
destination is read back and its digest compared with GHCR's.
Run it after scripts/publish-github.sh, and after the image build for that
release has finished.
Environment:
GITEA_TOKEN Token for the registry (required).
MIRROR_HOST The Gitea host to copy from over the VPN.
Default: 10.7.0.4
MIRROR_HOST_USER Its ssh user. Default: root
MIRROR_REGISTRY_HOST The registry as that host sees it, by its docker
network name. Default: gitea:3000
MIRROR_REGISTRY_PUBLIC_HOST
The registry's public name, for the pull command
it prints. Default: git.tben.me
MIRROR_SOURCE_REGISTRY_HOST
Where the release workflow pushed.
Default: ghcr.io
MIRROR_NETWORK The docker network the registry is on, so the
copy container can resolve it.
Default: cloudflare
MIRROR_USER The registry user for basic auth. Default: cobble
MIRROR_SKOPEO_IMAGE The skopeo image the host runs.
Default: quay.io/skopeo/stable:latest
DRY_RUN=1, ASSUME_YES=1 Same as --dry-run and --yes.
EOF
}
parse_publish_args "$@"
if [ "$HELP" = "1" ]; then
usage
exit 0
fi
reject_unknown_args "$0"
# MARK:mirror-tags
# The moving tag a release owns: major.minor, read from the version the same
# way the release workflow reads it, so the two never disagree.
mirror_minor_tag() {
printf '%s' "${1#v}" | sed -E 's/^([0-9]+)\.([0-9]+).*/\1.\2/'
}
# The commit tag for a release: the first 12 characters of the commit the
# tag points at, which is `sha-${GITHUB_SHA::12}` in the workflow.
mirror_sha_tag() {
local sha
sha="$(git rev-parse -q --verify "refs/tags/$1^{commit}" 2>/dev/null || true)"
if [ -z "$sha" ]; then
return 1
fi
printf 'sha-%s' "${sha:0:12}"
}
# The tags this release owns outright, which no newer release claims.
mirror_fixed_tags() {
local sha_tag
printf '%s\n' "$1"
if sha_tag="$(mirror_sha_tag "$1")"; then
printf '%s\n' "$sha_tag"
fi
}
# The tags a newer release takes over.
mirror_moving_tags() {
printf 'latest\n%s\n' "$(mirror_minor_tag "$1")"
}
# END:mirror-tags
# MARK:mirror-guard
# Whether this release may still write the moving tags. It may while GHCR's
# `latest` is its own image: once a newer release has moved `latest`, copying
# an older version into the Gitea registry must not drag it back.
mirror_owns_moving_tags() {
if [ -z "$2" ]; then
# Nothing to compare against, so this release is taken as the newest.
return 0
fi
[ "$1" = "$2" ]
}
# END:mirror-guard
# MARK:mirror-digest
MANIFEST_ACCEPT="application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json"
# Reads a tag's digest from GHCR over the public internet. The token is
# anonymous: the image is public, and a registry answers a bare manifest
# request with 401 until it is given one.
github_registry_digest() {
local tag="$1"
local token
token="$(
curl -sSf "https://$SOURCE_REGISTRY_HOST/token?scope=repository:$SOURCE_REPO:pull" 2>/dev/null |
sed -n 's/.*"token":"\([^"]*\)".*/\1/p'
)" || return 1
if [ -z "$token" ]; then
return 1
fi
curl -sSf -o /dev/null -D- \
-H "Authorization: Bearer $token" \
-H "Accept: $MANIFEST_ACCEPT" \
"https://$SOURCE_REGISTRY_HOST/v2/$SOURCE_REPO/manifests/$tag" 2>/dev/null |
tr -d '\r' | sed -n 's/^docker-content-digest: //Ip' | head -n1
}
# END:mirror-digest
# MARK:mirror-remote
# Runs a bash script on the Gitea host, which is the one place that reaches
# both registries. The script arrives on stdin, so nothing is installed there
# and no quoting has to survive a round trip through ssh.
mirror_on_host() {
ssh -o BatchMode=yes -o ConnectTimeout="$CONNECT_TIMEOUT" "$HOST_USER@$HOST" \
"MIRROR_NETWORK='$NETWORK' MIRROR_SKOPEO_IMAGE='$SKOPEO_IMAGE' bash -s"
}
# Reads a destination tag's digest back from the registry. Empty means the
# tag is not there, which a caller reports differently from a mismatch.
mirror_digest_on_host() {
mirror_on_host <<REMOTE
set -uo pipefail
docker run --rm --network "\$MIRROR_NETWORK" "\$MIRROR_SKOPEO_IMAGE" \\
inspect --tls-verify=false --creds '$MIRROR_USER:$GITEA_TOKEN' \\
--format '{{.Digest}}' 'docker://$REGISTRY_HOST/$REGISTRY_REPO:$1' 2>/dev/null | tail -n1
REMOTE
}
# Copies one tag, leaving skopeo's output in MIRROR_COPY_LOG so a failure can
# be shown without streaming its progress into the report.
mirror_copy_on_host() {
MIRROR_COPY_LOG="$(mktemp -t cobble-mirror.XXXXXX)"
mirror_on_host <<REMOTE >"$MIRROR_COPY_LOG" 2>&1
set -uo pipefail
docker run --rm --network "\$MIRROR_NETWORK" "\$MIRROR_SKOPEO_IMAGE" \\
copy --all --retry-times 5 \\
--src-tls-verify=true --dest-tls-verify=false \\
--dest-creds '$MIRROR_USER:$GITEA_TOKEN' \\
'docker://$SOURCE_REGISTRY_HOST/$SOURCE_REPO:$1' \\
'docker://$REGISTRY_HOST/$REGISTRY_REPO:$2'
REMOTE
}
# END:mirror-remote
# MARK:mirror-main
require_repo_root
step "Resolving the version"
if [ -z "$VERSION_RAW" ]; then
VERSION_RAW="$(source_version)"
if [ -z "$VERSION_RAW" ]; then
fail "No version in package.json and none passed with --version."
fi
detail "Read from package.json: $VERSION_RAW"
fi
validate_version "$VERSION_RAW"
TAG="$(normalize_version "$VERSION_RAW")"
ok "Mirroring $TAG."
require_cmd git
require_cmd ssh "Install an ssh client."
require_cmd curl "Install curl."
if [ -z "${GITEA_TOKEN:-}" ]; then
fail "No Gitea token. Set GITEA_TOKEN to a token whose user can push packages to the $REGISTRY_REPO owner."
fi
step "Checking the released image in $SOURCE_REGISTRY_HOST"
SOURCE_DIGEST="$(github_registry_digest "$TAG")" ||
fail "No $TAG in $SOURCE_REGISTRY_HOST/$SOURCE_REPO. Publish the GitHub release first: scripts/publish-github.sh builds the image."
if [ -z "$SOURCE_DIGEST" ]; then
fail "No $TAG in $SOURCE_REGISTRY_HOST/$SOURCE_REPO. Publish the GitHub release first: scripts/publish-github.sh builds the image."
fi
ok "$TAG is $SOURCE_DIGEST."
LATEST_DIGEST="$(github_registry_digest latest || true)"
TAGS_TO_COPY="$(mirror_fixed_tags "$TAG")"
MOVING_SKIPPED=""
if mirror_owns_moving_tags "$SOURCE_DIGEST" "$LATEST_DIGEST"; then
TAGS_TO_COPY="$(
printf '%s\n%s\n' "$TAGS_TO_COPY" "$(mirror_moving_tags "$TAG")" |
awk 'NF && !seen[$0]++'
)"
else
MOVING_SKIPPED="$(mirror_moving_tags "$TAG" | tr '\n' ' ')"
fi
step "Mirroring $TAG into the Gitea registry"
printf ' %shost%s %s\n' "$C_DIM" "$C_RESET" "$HOST_USER@$HOST"
printf ' %sregistry%s %s/%s\n' "$C_DIM" "$C_RESET" "$REGISTRY_HOST" "$REGISTRY_REPO"
printf ' %ssource%s %s/%s\n' "$C_DIM" "$C_RESET" "$SOURCE_REGISTRY_HOST" "$SOURCE_REPO"
printf ' %stags%s %s\n' "$C_DIM" "$C_RESET" "$(printf '%s' "$TAGS_TO_COPY" | tr '\n' ' ')"
if [ -n "$MOVING_SKIPPED" ]; then
warn "$SOURCE_REGISTRY_HOST/$SOURCE_REPO:latest belongs to a newer release, so ${MOVING_SKIPPED% } is left where it is."
fi
step "Checking the Gitea host"
if is_dry_run; then
detail "[dry run] ssh -o BatchMode=yes $HOST_USER@$HOST 'bash -s'"
remote_hostname="$HOST"
else
remote_hostname="$(
mirror_on_host <<'REMOTE'
hostname
REMOTE
)" || fail "Could not reach $HOST_USER@$HOST over ssh. The Gitea host is on the VPN: connect to it first."
ok "Reached $remote_hostname."
fi
confirm "Mirror $TAG into $REGISTRY_HOST/$REGISTRY_REPO?"
failed=0
for dest_tag in $TAGS_TO_COPY; do
step "Copying $TAG to $dest_tag"
if is_dry_run; then
detail "[dry run] skopeo copy --all $SOURCE_REGISTRY_HOST/$SOURCE_REPO:$TAG -> $REGISTRY_HOST/$REGISTRY_REPO:$dest_tag"
continue
fi
if ! mirror_copy_on_host "$TAG" "$dest_tag"; then
sed -n '$!{/^$/d};p' "${MIRROR_COPY_LOG:-/dev/null}" 2>/dev/null | tail -n 8 >&2
warn "The copy to $dest_tag failed."
failed=1
continue
fi
# The digest is what makes the two registries the same image: a copy that
# stored a different manifest list would read back differently here.
copied_digest="$(mirror_digest_on_host "$dest_tag")"
if [ "$copied_digest" != "$SOURCE_DIGEST" ]; then
warn "$dest_tag reads back as ${copied_digest:-nothing}, not $SOURCE_DIGEST."
failed=1
continue
fi
ok "$dest_tag is $SOURCE_DIGEST."
done
if [ -n "${MIRROR_COPY_LOG:-}" ] && [ -f "$MIRROR_COPY_LOG" ]; then
rm -f "$MIRROR_COPY_LOG"
fi
step "Mirrored $TAG"
printf ' %sregistry%s %s/%s\n' "$C_DIM" "$C_RESET" "$REGISTRY_HOST" "$REGISTRY_REPO"
printf ' %stags%s %s\n' "$C_DIM" "$C_RESET" "$(printf '%s' "$TAGS_TO_COPY" | tr '\n' ' ')"
printf ' %sdigest%s %s\n' "$C_DIM" "$C_RESET" "$SOURCE_DIGEST"
if [ "$DRY_RUN" = "1" ]; then
printf '\n%sDry run: copied nothing.%s\n' "$C_YELLOW" "$C_RESET"
fi
if [ "$failed" = "1" ]; then
fail "The Gitea registry does not match $SOURCE_REGISTRY_HOST/$SOURCE_REPO. The release itself is unaffected: the installer defaults to GHCR."
fi
printf '\nAn install that can reach the Gitea registry can pull it with:\n'
printf ' curl -fsSL https://url.moduletrace.com/cobble | sudo bash -s -- --image %s/%s:%s\n' \
"$REGISTRY_PUBLIC_HOST" "$REGISTRY_REPO" "$TAG"
# END:mirror-main
scripts/publish-gitea.sh
+9 -6
View File
@@ -8,9 +8,10 @@
# creates the Gitea release over the API. # creates the Gitea release over the API.
# #
# Gitea has no release workflow, so nothing is built here: the image is built # Gitea has no release workflow, so nothing is built here: the image is built
# once by .github/workflows/release.yml and mirrored to # once by .github/workflows/release.yml and copied into the Gitea registry by
# git.tben.me/moduletrace/cobble. This script only publishes the source and # scripts/mirror-gitea.sh, run from the VPN side, since a GitHub runner can
# the release object. # never reach it. This script only publishes the source and the release
# object.
# #
# The token comes from GITEA_TOKEN, or from the Gitea section of ~/.config/tea # The token comes from GITEA_TOKEN, or from the Gitea section of ~/.config/tea
# if the tea CLI is installed. It needs the `write:repository` scope on the # if the tea CLI is installed. It needs the `write:repository` scope on the
@@ -208,6 +209,9 @@ else
if [ -n "${response_file:-}" ]; then if [ -n "${response_file:-}" ]; then
rm -f "$response_file" rm -f "$response_file"
fi fi
if [ "$api_status" = "403" ]; then
fail "Gitea answered HTTP 403: the token's user cannot write to $REPO_SLUG. The tags and $BRANCH still push over SSH, but creating a release needs the token's user to be a writer (write or admin) on the repository. Check it with: curl -s -H \"Authorization: token \$GITEA_TOKEN\" $API_URL/repos/$REPO_SLUG | grep -o '\"permissions\":{[^}]*}'"
fi
fail "Gitea answered HTTP $api_status for the release." fail "Gitea answered HTTP $api_status for the release."
;; ;;
esac esac
@@ -218,6 +222,5 @@ else
fi fi
print_release_summary "$REMOTE" "$BRANCH" "https://${REMOTE_HOST}/${REPO_SLUG}/releases/tag/$TAG" print_release_summary "$REMOTE" "$BRANCH" "https://${REMOTE_HOST}/${REPO_SLUG}/releases/tag/$TAG"
printf '\nThe image is built by the GitHub release; mirror it into the Gitea registry with:\n' printf '\nThe image is built by the GitHub release, which cannot reach this registry.\nOnce that build has finished, copy it in over the VPN with:\n'
printf ' docker buildx imagetools create --tag git.tben.me/moduletrace/cobble:latest \\\n' printf ' bun run mirror:gitea\n'
printf ' ghcr.io/moduletrace/cobble:latest\n'
scripts/publish-github.sh
+4 -3
View File
@@ -7,11 +7,12 @@
# installer syntax), tags HEAD with the package.json version, pushes main and # installer syntax), tags HEAD with the package.json version, pushes main and
# the tag to the `origin` remote, then creates the GitHub release. Publishing # the tag to the `origin` remote, then creates the GitHub release. Publishing
# that release is what fires .github/workflows/release.yml, which builds the # that release is what fires .github/workflows/release.yml, which builds the
# multi-arch image and pushes it to GHCR (and mirrors it to Gitea). # multi-arch image and pushes it to GHCR, the registry the installer defaults
# to. It never pushes to Gitea: that registry is on the VPN, out of a
# runner's reach, so scripts/mirror-gitea.sh copies the image separately.
# #
# Run scripts/publish-gitea.sh afterwards for the source mirror's own release # Run scripts/publish-gitea.sh afterwards for the source mirror's own release
# object. There is no existing publish script in the repo: this and its Gitea # object, and scripts/mirror-gitea.sh once the image build has finished.
# twin are it.
set -Eeuo pipefail set -Eeuo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
scripts/publish-tools.test.ts
+192 -1
View File
@@ -4,6 +4,7 @@ import path from "path";
const ROOT = path.join(import.meta.dir, ".."); const ROOT = path.join(import.meta.dir, "..");
const COMMON = path.join(ROOT, "scripts/lib/publish-common.sh"); const COMMON = path.join(ROOT, "scripts/lib/publish-common.sh");
const RELEASE = path.join(ROOT, "scripts/lib/publish-release.sh"); const RELEASE = path.join(ROOT, "scripts/lib/publish-release.sh");
const MIRROR = path.join(ROOT, "scripts/mirror-gitea.sh");
const SCRIPTS = [ const SCRIPTS = [
path.join(ROOT, "scripts/publish-github.sh"), path.join(ROOT, "scripts/publish-github.sh"),
path.join(ROOT, "scripts/publish-gitea.sh"), path.join(ROOT, "scripts/publish-gitea.sh"),
@@ -99,13 +100,21 @@ async function pathWithoutJq(): Promise<string> {
} }
describe("publish scripts syntax", () => { describe("publish scripts syntax", () => {
for (const script of [...SCRIPTS, COMMON, RELEASE]) { for (const script of [...SCRIPTS, MIRROR, COMMON, RELEASE]) {
test(`${path.relative(ROOT, script)} parses`, async () => { test(`${path.relative(ROOT, script)} parses`, async () => {
const proc = Bun.spawn({ cmd: ["bash", "-n", script] }); const proc = Bun.spawn({ cmd: ["bash", "-n", script] });
expect(await proc.exited).toBe(0); expect(await proc.exited).toBe(0);
}); });
} }
test("the mirror script sources the shared libraries", async () => {
const source = await Bun.file(MIRROR).text();
expect(source).toContain('SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"');
expect(source).toContain('. "$SCRIPT_DIR/lib/publish-common.sh"');
expect(source).toContain('. "$SCRIPT_DIR/lib/publish-release.sh"');
});
test("both publish scripts source the shared library", async () => { test("both publish scripts source the shared library", async () => {
for (const script of SCRIPTS) { for (const script of SCRIPTS) {
const source = await Bun.file(script).text(); const source = await Bun.file(script).text();
@@ -141,6 +150,156 @@ describe("publish scripts remote wiring", () => {
}); });
}); });
describe("mirror script wiring", () => {
test("CI never pushes to the Gitea registry", async () => {
const workflow = await Bun.file(
path.join(ROOT, ".github/workflows/release.yml"),
).text();
expect(workflow).not.toContain("git.tben.me");
expect(workflow).not.toContain("GITEA_TOKEN");
expect(workflow).not.toContain("imagetools create");
expect(workflow).toContain("Build and push to GHCR");
});
test("the mirror script copies every tag the workflow publishes", async () => {
const [workflow, mirror] = await Promise.all([
Bun.file(path.join(ROOT, ".github/workflows/release.yml")).text(),
Bun.file(MIRROR).text(),
]);
// The workflow's own four tags, so a mirror tag can never be invented
// here and go missing there.
for (const tag of ["latest", "${version}", "${minor}", "${sha}"]) {
expect(workflow).toContain(tag);
}
expect(mirror).toContain("mirror_minor_tag");
expect(mirror).toContain("mirror_sha_tag");
expect(mirror).toContain("mirror_fixed_tags");
expect(mirror).toContain("mirror_moving_tags");
});
test("the mirror script asks before copying and verifies each digest", async () => {
const source = await Bun.file(MIRROR).text();
expect(source).toContain("confirm \"Mirror $TAG into");
expect(source).toContain("mirror_digest_on_host");
expect(source).toContain("skopeo");
expect(source).toContain("--all");
});
test("the pull command it prints names the public host, not the docker one", async () => {
const source = await Bun.file(MIRROR).text();
// The copy reaches the registry by docker network name, which is not
// resolvable for an install; printing that would hand out a broken
// command.
expect(source).toContain("MIRROR_REGISTRY_PUBLIC_HOST");
expect(source).toContain('"$REGISTRY_PUBLIC_HOST" "$REGISTRY_REPO" "$TAG"');
expect(source).not.toContain('"${REGISTRY_HOST%%:*}" "$REGISTRY_REPO" "$TAG"');
});
test("without a token the mirror script refuses instead of guessing", async () => {
const proc = Bun.spawn({
cmd: ["bash", MIRROR, "--yes"],
cwd: ROOT,
env: { ...process.env, GITEA_TOKEN: "" },
stdout: "pipe",
stderr: "pipe",
});
const [stderr, code] = await Promise.all([
new Response(proc.stderr).text(),
proc.exited,
]);
expect(code).toBe(1);
expect(stderr).toContain("No Gitea token");
});
});
describe("mirror lib tag handling", () => {
test("major.minor is read exactly as the release workflow reads it", async () => {
const block = await blockCaller({ file: MIRROR, name: "mirror-tags" });
for (const [version, expected] of [
["v0.1.5", "0.1"],
["v1.2.3", "1.2"],
["v10.20.30", "10.20"],
["v0.1.5-rc.1", "0.1"],
] as const) {
const out = await runBash({
script: [block, `mirror_minor_tag '${version}'`].join("\n"),
});
expect(out).toBe(expected);
}
});
test("fixed tags are the version and its commit, moving ones the rest", async () => {
const block = await blockCaller({ file: MIRROR, name: "mirror-tags" });
const fixed = await runBash({
script: [block, "mirror_fixed_tags v0.1.5"].join("\n"),
});
expect(fixed.split("\n")[0]).toBe("v0.1.5");
const moving = await runBash({
script: [block, "mirror_moving_tags v0.1.5"].join("\n"),
});
expect(moving.split("\n")).toEqual(["latest", "0.1"]);
});
test("a tag with no commit on disk still mirrors its version", async () => {
const block = await blockCaller({ file: MIRROR, name: "mirror-tags" });
const out = await runBash({
script: [block, "mirror_fixed_tags v99.99.99"].join("\n"),
});
expect(out).toBe("v99.99.99");
});
});
describe("mirror lib moving tag guard", () => {
test("a release may move latest while it owns it", async () => {
const block = await blockCaller({ file: MIRROR, name: "mirror-guard" });
const out = await runBash({
script: [
block,
'mirror_owns_moving_tags sha256:aa sha256:aa && printf owns',
].join("\n"),
});
expect(out).toBe("owns");
});
test("a newer release's latest is never rolled back", async () => {
const block = await blockCaller({ file: MIRROR, name: "mirror-guard" });
const out = await runBash({
script: [
block,
'if mirror_owns_moving_tags sha256:aa sha256:bb; then printf owns; else printf refuses; fi',
].join("\n"),
});
expect(out).toBe("refuses");
});
test("with no latest to compare, the release is taken as newest", async () => {
const block = await blockCaller({ file: MIRROR, name: "mirror-guard" });
const out = await runBash({
script: [
block,
'mirror_owns_moving_tags sha256:aa "" && printf owns',
].join("\n"),
});
expect(out).toBe("owns");
});
});
describe("publish lib version handling", () => { describe("publish lib version handling", () => {
test("normalizes a version with and without the v", async () => { test("normalizes a version with and without the v", async () => {
const block = await blockCaller({ name: "version" }); const block = await blockCaller({ name: "version" });
@@ -292,6 +451,38 @@ describe("publish lib release notes", () => {
}); });
}); });
describe("publish lib compare link", () => {
// The range is what makes the link open this release's diff; a bare
// /compare/ opened the host's default comparison instead.
test("adds the tag range to the compare path", async () => {
const block = await blockCaller({ file: RELEASE, name: "notes-file" });
const out = await runBash({
script: [
block,
`TAG=v0.1.4`,
`build_compare_url https://example.test/compare/ v0.1.3`,
].join("\n"),
});
expect(out).toBe("https://example.test/compare/v0.1.3...v0.1.4");
});
test("leaves the path alone when there is no earlier tag", async () => {
const block = await blockCaller({ file: RELEASE, name: "notes-file" });
const out = await runBash({
script: [
block,
`TAG=v0.1.0`,
`build_compare_url https://example.test/compare/ ""`,
].join("\n"),
});
expect(out).toBe("https://example.test/compare/");
});
});
describe("publish lib json escaping", () => { describe("publish lib json escaping", () => {
const sample = 'a\nb\t"c" \\ é'; const sample = 'a\nb\t"c" \\ é';
src/bots-queue/job-abort-registry.test.ts
+58
View File
@@ -0,0 +1,58 @@
import { describe, expect, test } from "bun:test";
import { abortRunningJob, registerJobAbort, unregisterJobAbort } from "./job-abort-registry";
describe("job-abort-registry", () => {
test("aborts the job this process is running", () => {
let aborted = false;
registerJobAbort({ job_id: 101, abort: () => (aborted = true) });
expect(abortRunningJob({ job_id: 101 })).toBe(true);
expect(aborted).toBe(true);
unregisterJobAbort({ job_id: 101 });
});
test("a job running in another process is not here, so it isn't reported as aborted", () => {
expect(abortRunningJob({ job_id: 202 })).toBe(false);
});
test("a finished job is cleaned up, so its id can't abort a later one", () => {
let aborted = false;
registerJobAbort({ job_id: 303, abort: () => (aborted = true) });
unregisterJobAbort({ job_id: 303 });
expect(abortRunningJob({ job_id: 303 })).toBe(false);
expect(aborted).toBe(false);
});
test("a numeric string id reaches the same entry, since a route may send one", () => {
let aborted = false;
registerJobAbort({ job_id: 404, abort: () => (aborted = true) });
expect(abortRunningJob({ job_id: "404" })).toBe(true);
expect(aborted).toBe(true);
unregisterJobAbort({ job_id: 404 });
});
test("unregistering a job nobody registered is harmless", () => {
expect(() => unregisterJobAbort({ job_id: 505 })).not.toThrow();
});
test("only one job is aborted at a time", () => {
const aborted: number[] = [];
registerJobAbort({ job_id: 606, abort: () => aborted.push(606) });
registerJobAbort({ job_id: 707, abort: () => aborted.push(707) });
abortRunningJob({ job_id: 606 });
expect(aborted).toEqual([606]);
unregisterJobAbort({ job_id: 606 });
unregisterJobAbort({ job_id: 707 });
});
});
src/bots-queue/job-abort-registry.ts
+40
View File
@@ -0,0 +1,40 @@
/**
* The abort controllers of the jobs this process is running right now,
* keyed by job id. Stopping a bot from the chat aborts the request its job
* is waiting on (a model that never answers, a tool that hangs) instead of
* leaving it to find out on its next heartbeat, which can take minutes.
*
* Only the process running a job can abort it, so this is a bonus: the job
* is already failed in `bots_queue`, and a handler in another process
* learns that on its next `updateBotJob`. Kept free of imports so the API
* routes can use it without pulling the queue handler into a page bundle.
*/
const running = new Map<number, () => void>();
/**
* Called by a handler once it holds a job, with the way to abort it.
*/
export function registerJobAbort({ job_id, abort }: { job_id: number | string; abort: () => void }) {
running.set(Number(job_id), abort);
}
/**
* Called by a handler in its `finally`, whatever the outcome.
*/
export function unregisterJobAbort({ job_id }: { job_id: number | string }) {
running.delete(Number(job_id));
}
/**
* Aborts a job running here. False means another process (or nobody) has
* it, so the row's new status is how it finds out.
*/
export function abortRunningJob({ job_id }: { job_id: number | string }) {
const abort = running.get(Number(job_id));
if (!abort) return false;
abort();
return true;
}
src/bots-queue/name-bot-session.ts
+19 -2
View File
@@ -4,6 +4,7 @@ import { AppData } from "@/src/data/app-data";
import { SessionNamePrompt } from "@/src/data/harness-prompts"; import { SessionNamePrompt } from "@/src/data/harness-prompts";
import grabBotMessages from "@/src/functions/backend/db/bots/grab-bot-messages"; import grabBotMessages from "@/src/functions/backend/db/bots/grab-bot-messages";
import generateText from "@/src/harness/text/generate-text"; import generateText from "@/src/harness/text/generate-text";
import { runWithHarnessUsageScope } from "@/src/harness/usage/harness-usage-scope";
import parseSessionName from "./parse-session-name"; import parseSessionName from "./parse-session-name";
type Params = { type Params = {
@@ -30,7 +31,7 @@ export default async function nameBotSession({
}: Params): Promise<boolean> { }: Params): Promise<boolean> {
const session = ( const session = (
await BunSQLite.sql<BUN_SQLITE_COBBLE_BOT_SESSIONS>({ await BunSQLite.sql<BUN_SQLITE_COBBLE_BOT_SESSIONS>({
sql: `SELECT id, title, summary FROM bot_sessions WHERE id = ? LIMIT 1`, sql: `SELECT id, user_id, title, summary FROM bot_sessions WHERE id = ? LIMIT 1`,
values: [session_id], values: [session_id],
}) })
).payload?.[0]; ).payload?.[0];
@@ -57,7 +58,22 @@ export default async function nameBotSession({
) )
.join("\n\n"); .join("\n\n");
const result = await generateText({ /**
* The naming turn is a model call too, so it is recorded for this
* conversation. It runs after the job is complete, on purpose, so it
* has no job and no message of its own to charge it to.
*/
const result = await runWithHarnessUsageScope({
scope: {
user_id: session.user_id || undefined,
bot_id,
workspace_id: workspace_id || null,
session_id: session.id,
job_id: null,
message_id: null,
},
task: () =>
generateText({
messages: [ messages: [
{ role: "system", content: SessionNamePrompt }, { role: "system", content: SessionNamePrompt },
{ role: "user", content: transcript }, { role: "user", content: transcript },
@@ -65,6 +81,7 @@ export default async function nameBotSession({
bot_id, bot_id,
workspace_id, workspace_id,
signal, signal,
}),
}); });
const name = parseSessionName({ text: result.text }); const name = parseSessionName({ text: result.text });
src/bots-queue/process-bot-job.ts
+21 -5
View File
@@ -27,6 +27,7 @@ import runBotJob from "./run-bot-job";
import parseJobJson from "./parse-job-json"; import parseJobJson from "./parse-job-json";
import nameBotSession from "./name-bot-session"; import nameBotSession from "./name-bot-session";
import pushBotSession from "./push-bot-session"; import pushBotSession from "./push-bot-session";
import { registerJobAbort, unregisterJobAbort } from "./job-abort-registry";
import type { BotJobUpdateData } from "@/src/types"; import type { BotJobUpdateData } from "@/src/types";
import type { BotJobContext, BotJobRunner } from "./types"; import type { BotJobContext, BotJobRunner } from "./types";
@@ -80,11 +81,6 @@ export default async function processBotJob({
*/ */
const user_id = await grabJobUserId({ job }).catch(() => null); const user_id = await grabJobUserId({ job }).catch(() => null);
const setFace = async (face: Omit<Parameters<typeof setBotFace>[0], "bot_id" | "user_id">) => {
if (!user_id) return;
await pushBotNotice({ bot_id, bot: await setBotFace({ bot_id, user_id, ...face }) });
};
const controller = new AbortController(); const controller = new AbortController();
/** /**
* The deadline is the one abort that is a failure, not a cancellation: * The deadline is the one abort that is a failure, not a cancellation:
@@ -94,6 +90,25 @@ export default async function processBotJob({
const timeout_message = `"${job.title || "The job"}" took too long and was stopped.`; const timeout_message = `"${job.title || "The job"}" took too long and was stopped.`;
let timed_out = false; let timed_out = false;
/**
* A stopped job never touches the face again: `stopBotJobs` already set
* it to idle, and a late `working` or `asleep` would push over that and
* leave a bot that shows work with nothing behind it.
*/
const setFace = async (face: Omit<Parameters<typeof setBotFace>[0], "bot_id" | "user_id">) => {
if (!user_id || controller.signal.aborted) return;
await pushBotNotice({ bot_id, bot: await setBotFace({ bot_id, user_id, ...face }) });
};
/**
* Stopping the bot from the chat aborts the request this job is
* waiting on, here and now, rather than at the next heartbeat. The job
* is already failed in the queue, so an abort is a cancellation and its
* result is dropped (`controller.signal.aborted` below).
*/
registerJobAbort({ job_id, abort: () => controller.abort() });
const update: BotJobContext["update"] = async (params) => { const update: BotJobContext["update"] = async (params) => {
if (controller.signal.aborted) return false; if (controller.signal.aborted) return false;
@@ -305,5 +320,6 @@ export default async function processBotJob({
} }
} finally { } finally {
clearInterval(heartbeat); clearInterval(heartbeat);
unregisterJobAbort({ job_id });
} }
} }
src/bots-queue/push-bot-notice.ts
+11 -5
View File
@@ -18,6 +18,12 @@ type Params = {
* (`relayJobToRequester`): a message in that bot's conversation. * (`relayJobToRequester`): a message in that bot's conversation.
*/ */
relay?: { bot_id: number; message: BUN_SQLITE_COBBLE_BOT_MESSAGES } | null; relay?: { bot_id: number; message: BUN_SQLITE_COBBLE_BOT_MESSAGES } | null;
/**
* Skip the push notification, keeping only the live send. For a message
* the user just caused themselves (stopping the bot): they are looking
* at the app, and one notification per conversation would be noise.
*/
quiet_push?: boolean;
}; };
/** /**
@@ -27,9 +33,9 @@ type Params = {
* goes to the conversation it was posted in. Best effort: a push that * goes to the conversation it was posted in. Best effort: a push that
* fails is logged and never fails the job that made the change. * fails is logged and never fails the job that made the change.
*/ */
export default async function pushBotNotice({ bot_id, message, bot, relay }: Params) { export default async function pushBotNotice({ bot_id, message, bot, relay, quiet_push }: Params) {
try { try {
await push({ bot_id, message, bot, relay }); await push({ bot_id, message, bot, relay, quiet_push });
} catch (error) { } catch (error) {
await logError({ await logError({
title: "Bot notice wasn't pushed", title: "Bot notice wasn't pushed",
@@ -39,7 +45,7 @@ export default async function pushBotNotice({ bot_id, message, bot, relay }: Par
} }
} }
async function push({ bot_id, message, bot, relay }: Params) { async function push({ bot_id, message, bot, relay, quiet_push }: Params) {
if (message) { if (message) {
await pushToSessionUser({ await pushToSessionUser({
session_id: message.session_id, session_id: message.session_id,
@@ -49,7 +55,7 @@ async function push({ bot_id, message, bot, relay }: Params) {
/** /**
* Not waited on: a push service's latency must not slow the job. * Not waited on: a push service's latency must not slow the job.
*/ */
void notifyBotMessage({ message }); if (!quiet_push) void notifyBotMessage({ message });
} }
if (relay) { if (relay) {
@@ -58,7 +64,7 @@ async function push({ bot_id, message, bot, relay }: Params) {
data: { event: "server:bot-message", data: relay.message }, data: { event: "server:bot-message", data: relay.message },
}); });
void notifyBotMessage({ message: relay.message }); if (!quiet_push) void notifyBotMessage({ message: relay.message });
} }
if (bot) { if (bot) {
src/bots-queue/run-bot-job.ts
+20 -3
View File
@@ -1,6 +1,7 @@
import grabBotById from "@/src/functions/backend/db/bots/grab-bot-by-id"; import grabBotById from "@/src/functions/backend/db/bots/grab-bot-by-id";
import grabJobUserId from "@/src/functions/backend/db/bots-queue/grab-job-user-id"; import grabJobUserId from "@/src/functions/backend/db/bots-queue/grab-job-user-id";
import HarnessError from "@/src/harness/harness-error"; import HarnessError from "@/src/harness/harness-error";
import { runWithHarnessUsageScope } from "@/src/harness/usage/harness-usage-scope";
import BotJobFatalError from "./bot-job-fatal-error"; import BotJobFatalError from "./bot-job-fatal-error";
import runBotReply from "./run-bot-reply"; import runBotReply from "./run-bot-reply";
import runBotTask from "./run-bot-task"; import runBotTask from "./run-bot-task";
@@ -33,9 +34,25 @@ const runBotJob: BotJobRunner = async (context) => {
const owned = { ...bot, id: bot.id }; const owned = { ...bot, id: bot.id };
try { try {
return job.kind == "reply" /**
? await runBotReply({ ...context, bot: owned, user_id }) * Everything the job's models do is recorded for this user, this
: await runBotTask({ ...context, bot: owned, user_id }); * bot, this conversation and this job. A reply's text loop adds
* the message it is producing to the one-turn scope it opens.
*/
return await runWithHarnessUsageScope({
scope: {
user_id,
bot_id: bot.id,
workspace_id: job.workspace_id || null,
session_id: job.session_id || null,
job_id: job.id || null,
message_id: null,
},
task: () =>
job.kind == "reply"
? runBotReply({ ...context, bot: owned, user_id })
: runBotTask({ ...context, bot: owned, user_id }),
});
} catch (error: unknown) { } catch (error: unknown) {
if ( if (
error instanceof HarnessError && error instanceof HarnessError &&
src/bots-queue/run-bot-reply.ts
+2 -1
View File
@@ -11,6 +11,7 @@ import {
TaskReminderNote, TaskReminderNote,
WorkspaceLearningNote, WorkspaceLearningNote,
} from "@/src/data/harness-prompts"; } from "@/src/data/harness-prompts";
import { grabReplyMaxToolSteps } from "@/src/functions/backend/db/app-settings/grab-run-limits";
import grabBotMessages from "@/src/functions/backend/db/bots/grab-bot-messages"; import grabBotMessages from "@/src/functions/backend/db/bots/grab-bot-messages";
import grabBotPrompt from "@/src/functions/backend/db/bots/grab-bot-prompt"; import grabBotPrompt from "@/src/functions/backend/db/bots/grab-bot-prompt";
import grabBotComputers from "@/src/functions/backend/db/computers/grab-bot-computers"; import grabBotComputers from "@/src/functions/backend/db/computers/grab-bot-computers";
@@ -173,7 +174,7 @@ export default async function runBotReply({
}), }),
}), }),
tools: [...tools, ...file_tools, ...extensions.tools, ...skills.tools], tools: [...tools, ...file_tools, ...extensions.tools, ...skills.tools],
max_steps: AppData.BotReplyMaxToolSteps, max_steps: await grabReplyMaxToolSteps(),
bot_id: bot.id, bot_id: bot.id,
workspace_id, workspace_id,
signal, signal,
src/bots-queue/run-bot-task.ts
+2
View File
@@ -8,6 +8,7 @@ import {
TaskRunNote, TaskRunNote,
WorkspaceLearningNote, WorkspaceLearningNote,
} from "@/src/data/harness-prompts"; } from "@/src/data/harness-prompts";
import { grabHarnessMaxToolSteps } from "@/src/functions/backend/db/app-settings/grab-run-limits";
import grabBotComputers from "@/src/functions/backend/db/computers/grab-bot-computers"; import grabBotComputers from "@/src/functions/backend/db/computers/grab-bot-computers";
import grabBotPrompt from "@/src/functions/backend/db/bots/grab-bot-prompt"; import grabBotPrompt from "@/src/functions/backend/db/bots/grab-bot-prompt";
import grabUserWorkspaces from "@/src/functions/backend/db/workspaces/grab-user-workspaces"; import grabUserWorkspaces from "@/src/functions/backend/db/workspaces/grab-user-workspaces";
@@ -227,6 +228,7 @@ export default async function runBotTask({
}, },
], ],
tools: [...tools, ...extensions.tools, ...skills.tools], tools: [...tools, ...extensions.tools, ...skills.tools],
max_steps: await grabHarnessMaxToolSteps(),
bot_id: bot.id, bot_id: bot.id,
workspace_id, workspace_id,
signal, signal,
src/components/general/admin-breadcrumbs.tsx
+59
View File
@@ -0,0 +1,59 @@
import Breadcrumbs from "@moduletrace/twui/elements/Breadcrumbs";
import Stack from "@moduletrace/twui/layout/Stack";
import { ChevronRight } from "lucide-react";
import { AdminHomePath } from "@/src/dict/admin-sections-dict";
import hasBreadcrumbTrail from "@/src/utils/has-breadcrumb-trail";
import Divider from "@moduletrace/twui/layout/Divider";
import { twMerge } from "tailwind-merge";
type Props = {
/**
* The address the page is at, e.g. `/admin/settings/bots/3`.
*/
pathname?: string | null;
/**
* The page's own name, for the last crumb: what the address can't say,
* since its last segment is an id (`Scout`, not `3`) or a slug that
* differs from the name the page shows (`MCPs`, not `Mcps`).
*/
title?: string;
};
/**
* The trail above a page's content: `Home`, then every section the address
* passes through, then the page itself, named by `title`.
*
* It is `twui`'s `Breadcrumbs`, which reads the trail off the path, so the
* sections come from the address and only the page's own name is given
* here; `hasBreadcrumbTrail` says which pages get one at all.
*/
export default function AdminBreadcrumbs({ pathname, title }: Props) {
if (!hasBreadcrumbTrail({ pathname })) return null;
return (
<Stack className={twMerge("w-full shrink-0 pt-5 pb-1")}>
<Breadcrumbs
pageUrl={pathname || AdminHomePath}
currentTitle={title}
excludeRegexMatch={/^admin$/}
linkProps={{
className:
"border-0 text-[13px] leading-[18px] text-ink-secondary no-underline hover:text-ink hover:opacity-100 dark:text-ink-secondary",
}}
currentLinkProps={{
title,
className:
"border-0 text-[13px] leading-[18px] text-ink no-underline pointer-events-none dark:text-ink",
}}
divider={
<ChevronRight
size={12}
aria-hidden
className="shrink-0 text-ink-secondary"
/>
}
/>
<Divider />
</Stack>
);
}
src/pages/admin/routines/(partials)/routine-edit-modal.tsx → src/components/general/routine-edit-modal.tsx
View File
src/pages/admin/routines/(partials)/routine-form.tsx → src/components/general/routine-form.tsx
+2 -1
View File
@@ -13,7 +13,8 @@ type Props = {
}; };
/** /**
* A routine's name, schedule (a cron expression in UTC) and brief. * A routine's name, schedule (a cron expression in UTC) and brief. The
* routines page and the conversation's routines modal both edit with it.
*/ */
export default function RoutineForm({ cron_job, onCancel, onSaved }: Props) { export default function RoutineForm({ cron_job, onCancel, onSaved }: Props) {
const [title, setTitle] = useState(cron_job.title || ""); const [title, setTitle] = useState(cron_job.title || "");
src/pages/admin/routines/(partials)/routine-row.tsx → src/components/general/routine-row.tsx
+15 -4
View File
@@ -1,10 +1,10 @@
import BotAvatar from "@/src/components/general/bot-avatar";
import type { BotFace } from "@/src/dict/bot-dict"; import type { BotFace } from "@/src/dict/bot-dict";
import botStoneProps from "@/src/utils/bot-stone-props"; import botStoneProps from "@/src/utils/bot-stone-props";
import type { CronJobWithNames } from "@/src/types"; import type { CronJobWithNames } from "@/src/types";
import SettingsListRow from "@/src/pages/admin/settings/(partials)/settings-list-row"; import SettingsListRow from "@/src/pages/admin/settings/(partials)/settings-list-row";
import SettingsRowAction from "@/src/pages/admin/settings/(partials)/settings-row-action"; import SettingsRowAction from "@/src/pages/admin/settings/(partials)/settings-row-action";
import routineSubtitle from "../(utils)/routine-subtitle"; import routineSubtitle from "@/src/utils/routine-subtitle";
import BotAvatar from "./bot-avatar";
type Props = { type Props = {
cron_job: CronJobWithNames; cron_job: CronJobWithNames;
@@ -12,13 +12,18 @@ type Props = {
onToggle: () => void; onToggle: () => void;
onEdit: () => void; onEdit: () => void;
onDelete: () => void; onDelete: () => void;
/**
* In a conversation's own modal the Chat link and the bot's and
* workspace's names are already the ones on screen.
*/
compact?: boolean;
}; };
/** /**
* One routine: its bot's avatar, its name (Paused or Done when it is), * One routine: its bot's avatar, its name (Paused or Done when it is),
* whose it is and when it runs, then Chat, Pause or Resume, Edit, Delete. * whose it is and when it runs, then Chat, Pause or Resume, Edit, Delete.
*/ */
export default function RoutineRow({ cron_job, busy, onToggle, onEdit, onDelete }: Props) { export default function RoutineRow({ cron_job, busy, onToggle, onEdit, onDelete, compact }: Props) {
const title = cron_job.title || `Routine ${cron_job.id}`; const title = cron_job.title || `Routine ${cron_job.id}`;
const done = Boolean(cron_job.max_runs && Number(cron_job.run_count || 0) >= cron_job.max_runs); const done = Boolean(cron_job.max_runs && Number(cron_job.run_count || 0) >= cron_job.max_runs);
const chat_url = cron_job.workspace_id const chat_url = cron_job.workspace_id
@@ -36,9 +41,15 @@ export default function RoutineRow({ cron_job, busy, onToggle, onEdit, onDelete
} }
title={title} title={title}
badge={cron_job.enabled ? undefined : done ? "Done" : "Paused"} badge={cron_job.enabled ? undefined : done ? "Done" : "Paused"}
subtitle={routineSubtitle({ cron_job })} subtitle={routineSubtitle({
cron_job,
show_bot: !compact,
show_workspace: !compact,
})}
> >
{compact ? null : (
<SettingsRowAction title={`Chat with ${cron_job.bot_name || "the bot"}`} label="Chat" href={chat_url} /> <SettingsRowAction title={`Chat with ${cron_job.bot_name || "the bot"}`} label="Chat" href={chat_url} />
)}
<SettingsRowAction <SettingsRowAction
title={cron_job.enabled ? `Pause ${title}` : `Resume ${title}`} title={cron_job.enabled ? `Pause ${title}` : `Resume ${title}`}
label={cron_job.enabled ? "Pause" : "Resume"} label={cron_job.enabled ? "Pause" : "Resume"}
src/cron/heartbeat/index.ts
-4
View File
@@ -6,9 +6,5 @@ export default async function heartbeat() {
sql: `SELECT 1 AS alive`, sql: `SELECT 1 AS alive`,
}); });
bunext.bunextLog.info(
`Cron: database heartbeat ${db_alive.success ? "ok" : "failed"}`,
);
return db_alive; return db_alive;
} }
src/cron/sync-computers/index.ts
+8 -5
View File
@@ -3,8 +3,10 @@ import BunSQLite from "@moduletrace/bun-sqlite";
import type { BUN_SQLITE_COBBLE_CONTAINERS } from "@/db/types/db"; import type { BUN_SQLITE_COBBLE_CONTAINERS } from "@/db/types/db";
import refreshComputerContainer from "@/src/functions/backend/db/computers/refresh-computer-container"; import refreshComputerContainer from "@/src/functions/backend/db/computers/refresh-computer-container";
type Row = Pick<BUN_SQLITE_COBBLE_CONTAINERS, "id" | "docker_id" | "status" | "ip_address"> & type Row = Pick<
{ computer_id: number }; BUN_SQLITE_COBBLE_CONTAINERS,
"id" | "docker_id" | "status" | "ip_address"
> & { computer_id: number };
/** /**
* Reads every computer container's state from docker and records it * Reads every computer container's state from docker and records it
@@ -24,10 +26,11 @@ export default async function syncComputers() {
let changed = 0; let changed = 0;
for (const { computer_id, ...container } of rows) { for (const { computer_id, ...container } of rows) {
const refreshed = await refreshComputerContainer({ computer_id, container }); const refreshed = await refreshComputerContainer({
computer_id,
container,
});
if (refreshed?.changed) changed++; if (refreshed?.changed) changed++;
} }
bunext.bunextLog.info(`Cron: ${rows.length} computer(s) checked, ${changed} changed`);
} }
src/data/app-data.ts
+23 -1
View File
@@ -246,7 +246,10 @@ export const AppData = {
HarnessCooldownMs: 60 * 1000, HarnessCooldownMs: 60 * 1000,
HarnessMaxCooldownMs: 24 * 60 * 60 * 1000, HarnessMaxCooldownMs: 24 * 60 * 60 * 1000,
/** /**
* Model turns in one text run before it must stop and report. * Model turns in one text run before it must stop and report. The
* install may save its own limit (the `harness_max_tool_steps` setting,
* Settings > General), which wins over this one; a task run reads it
* through `grabHarnessMaxToolSteps`.
*/ */
HarnessMaxToolSteps: 200, HarnessMaxToolSteps: 200,
/** /**
@@ -328,6 +331,16 @@ export const AppData = {
* How many older messages each scroll to the top of the thread loads. * How many older messages each scroll to the top of the thread loads.
*/ */
ConversationOlderMessages: 30, ConversationOlderMessages: 30,
/**
* Days of usage the usage page totals by default, and the most it will
* total at once.
*/
UsageDefaultDays: 30,
UsageMaxDays: 365,
/**
* Models listed per page on the usage page, the busiest first.
*/
UsageModelPageSize: 20,
/** /**
* How much of a bot's last message the rail's lists carry, as plain * How much of a bot's last message the rail's lists carry, as plain
* text (`messagePreview`): a row shows one line of it. * text (`messagePreview`): a row shows one line of it.
@@ -379,8 +392,17 @@ export const AppData = {
WorkspaceActivityItems: 6, WorkspaceActivityItems: 6,
/** /**
* Tool turns a chat reply may take. Long work goes to a task instead. * Tool turns a chat reply may take. Long work goes to a task instead.
* The install may save its own limit (the `bot_reply_max_tool_steps`
* setting, Settings > General), which wins over this one; a reply
* reads it through `grabReplyMaxToolSteps`.
*/ */
BotReplyMaxToolSteps: 8, BotReplyMaxToolSteps: 8,
/**
* The most steps a saved run limit may ask for, for a task run or a
* chat reply. A typo in the settings page can't turn one run into an
* hour of tool calls, and a huge limit still ends with a report.
*/
BotStepLimitMax: 1000,
/** /**
* Chat attachments: per file, per message, and how much of a * Chat attachments: per file, per message, and how much of a
* document's text a reply reads. * document's text a reply reads.
src/data/coordinator-system-prompt.ts
+3
View File
@@ -27,6 +27,9 @@ Permission to draft is not permission to send, permission to analyze is not perm
## 2. Delegate well ## 2. Delegate well
- Pick by role, instructions and tools, not by name. - Pick by role, instructions and tools, not by name.
- Say where the work runs, and mean it. \`delegate_task\` takes an optional \`workspace_id\`, and that one argument decides the whole context the worker has: its own chat, or a workspace. A task in the worker's own chat has only that worker's own memory, files and directly assigned computers. A task in a workspace has the workspace's shared context, the worker's instructions and memory for that workspace, the project's files, and the workspace's computers as well as the worker's. The result also lands in a different conversation: the worker's own chat, or its conversation in that workspace beside the rest of that project. Choose by where the work belongs, not by which is easier to type.
- Handed over from a conversation inside a workspace, the task always runs in that workspace and \`workspace_id\` is not accepted: say so, don't pretend you moved it.
- Delegating into a workspace the worker isn't in adds it to that workspace as a side effect. Say that in your reply when it happens; don't add members silently for a one-off.
- Give a full brief: objective, inputs, deliverable, acceptance criteria, constraints. - Give a full brief: objective, inputs, deliverable, acceptance criteria, constraints.
- Check \`list_tasks\` first. Never give two bots the same job, and don't restart work an open task already covers. - Check \`list_tasks\` first. Never give two bots the same job, and don't restart work an open task already covers.
- Track what comes back, judge it against the request, and integrate it. Don't pass a worker's unchecked claim on as fact. - Track what comes back, judge it against the request, and integrate it. Don't pass a worker's unchecked claim on as fact.
src/data/harness-prompts.ts
+1 -1
View File
@@ -115,7 +115,7 @@ Every action counts here: saving memory or instructions, creating or changing a
/** /**
* Added to the coordinator's chat prompt: it hands work to the crew. * Added to the coordinator's chat prompt: it hands work to the crew.
*/ */
export const CoordinatorReplyNote = `As the coordinator, hand work that fits a crew member's role to them with delegate_task. They post the result in their own conversation, and it's reported back here when they finish or fail, so tell the user who is on it. When a report comes back, pass on what matters to the user, follow up on anything left, and propose what comes next: the next piece of work, which bot would take it, or a question that settles the direction. Once the user agrees, you can also grow the crew with create_bot, put a worker in a workspace with add_bot_to_workspace, and create or change a workspace with create_workspace and update_workspace. When you name a crew member, write its ref instead of its name, e.g. ${botRef({ id: 3 })}: the user sees it as the bot's name, linked to its conversation.`; export const CoordinatorReplyNote = `As the coordinator, hand work that fits a crew member's role to them with delegate_task. They post the result in their own conversation, and it's reported back here when they finish or fail, so tell the user who is on it and where: a task with no workspace_id runs in that bot's own chat, on its own memory, files and directly assigned computers; one with workspace_id runs in the workspace, on the workspace's context, files and computers as well as the bot's, and lands in that project's conversation. Never swap one for the other without saying so. When a report comes back, pass on what matters to the user, follow up on anything left, and propose what comes next: the next piece of work, which bot would take it, or a question that settles the direction. Once the user agrees, you can also grow the crew with create_bot, put a worker in a workspace with add_bot_to_workspace, and create or change a workspace with create_workspace and update_workspace. When you name a crew member, write its ref instead of its name, e.g. ${botRef({ id: 3 })}: the user sees it as the bot's name, linked to its conversation.`;
/** /**
* Added to the coordinator's chat prompt instead while it has no workers, * Added to the coordinator's chat prompt instead while it has no workers,
src/dict/admin-sections-dict.ts
+1
View File
@@ -7,6 +7,7 @@ export const AdminSections = [
"/admin/workspaces", "/admin/workspaces",
"/admin/routines", "/admin/routines",
"/admin/files", "/admin/files",
"/admin/usage",
"/admin/logs", "/admin/logs",
"/admin/help", "/admin/help",
"/admin/settings", "/admin/settings",
src/dict/bot-dict.ts
+6
View File
@@ -110,6 +110,12 @@ export const CoordinatorBotName = "Coordinator";
*/ */
export const BotSystemPromptAppendSettingKey = "bot_system_prompt_append"; export const BotSystemPromptAppendSettingKey = "bot_system_prompt_append";
/**
* The `app_settings` key holding the install's own limit on the tool turns
* one chat reply may take. Unset means `AppData.BotReplyMaxToolSteps`.
*/
export const BotReplyMaxToolStepsSettingKey = "bot_reply_max_tool_steps";
/** /**
* A bot's face is its status. There are no badges or spinners. * A bot's face is its status. There are no badges or spinners.
*/ */
src/dict/bot-queue-dict.ts
+24
View File
@@ -4,6 +4,30 @@
*/ */
export const BotJobCancelledError = "Cancelled"; export const BotJobCancelledError = "Cancelled";
/**
* `last_error` of a job the user stopped from the chat. A stopped job is
* never retried and never told about: the user pulled the plug.
*/
export const BotJobStoppedError = "Stopped by the user";
/**
* The last step recorded on a stopped job. It's a step, not a message, so a
* bot that was stopped mid-thought doesn't answer on its own.
*/
export const BotJobStoppedStep = "Stopped";
/**
* What the thinking row's stop button says, so the row and its tooltip
* never drift.
*/
export const BotStopButtonLabel = "Stop";
/**
* What the bot says in the conversation once the user has stopped it. It's
* the bot's turn again, so the thread stops waiting on a reply.
*/
export const BotStoppedMessage = "Stopped. I'll carry on when you ask me to.";
/** /**
* `last_error` bun-sqlite sets when it releases a job whose handler went * `last_error` bun-sqlite sets when it releases a job whose handler went
* silent past the lock timeout on its last attempt. * silent past the lock timeout on its last attempt.
src/dict/harness-dict.ts
+6
View File
@@ -53,6 +53,12 @@ export const Harnesses = [
export type HarnessKind = (typeof Harnesses)[number]["value"]; export type HarnessKind = (typeof Harnesses)[number]["value"];
/**
* The `app_settings` key holding the install's own limit on model turns in
* one run (a task). Unset means `AppData.HarnessMaxToolSteps`.
*/
export const HarnessMaxToolStepsSettingKey = "harness_max_tool_steps";
/** /**
* Vendors preferred for automatic picks, best first. Free models from any * Vendors preferred for automatic picks, best first. Free models from any
* other vendor come after these, and only if they look competent. * other vendor come after these, and only if they look competent.
src/dict/rail-pages-dict.ts
+9 -1
View File
@@ -1,4 +1,6 @@
import { Activity, CircleHelp, Clock, FolderOpen, type LucideIcon } from "lucide-react"; import { Activity, BarChart3, CircleHelp, Clock, FolderOpen, type LucideIcon } from "lucide-react";
import type { BUN_SQLITE_COBBLE_USER_TYPES } from "@/db/types/db";
import canManageCrew from "@/src/utils/rbac/can-manage-crew";
import { HelpPath } from "./help-dict"; import { HelpPath } from "./help-dict";
export type RailPage = { export type RailPage = {
@@ -9,11 +11,16 @@ export type RailPage = {
* Shown and served to super admins only. * Shown and served to super admins only.
*/ */
super_admin_only?: boolean; super_admin_only?: boolean;
/**
* Who sees it, when that's an RBAC rule rather than super admins only.
*/
can_view?: (params: { user_types?: Pick<BUN_SQLITE_COBBLE_USER_TYPES, "user_type">[] | null }) => boolean;
}; };
export const RoutinesPath = "/admin/routines"; export const RoutinesPath = "/admin/routines";
export const FilesPath = "/admin/files"; export const FilesPath = "/admin/files";
export const ActivityPath = "/admin/logs"; export const ActivityPath = "/admin/logs";
export const UsagePath = "/admin/usage";
/** /**
* Pages opened from the rail's user popover that have their own rail: a * Pages opened from the rail's user popover that have their own rail: a
@@ -22,6 +29,7 @@ export const ActivityPath = "/admin/logs";
export const RailPages: RailPage[] = [ export const RailPages: RailPage[] = [
{ title: "Routines", url: RoutinesPath, Icon: Clock }, { title: "Routines", url: RoutinesPath, Icon: Clock },
{ title: "Files", url: FilesPath, Icon: FolderOpen }, { title: "Files", url: FilesPath, Icon: FolderOpen },
{ title: "Usage", url: UsagePath, Icon: BarChart3, can_view: canManageCrew },
{ title: "Activity", url: ActivityPath, Icon: Activity, super_admin_only: true }, { title: "Activity", url: ActivityPath, Icon: Activity, super_admin_only: true },
{ title: "Help", url: HelpPath, Icon: CircleHelp }, { title: "Help", url: HelpPath, Icon: CircleHelp },
]; ];
src/functions/backend/db/app-settings/grab-run-limits.ts
+75
View File
@@ -0,0 +1,75 @@
import { AppData } from "@/src/data/app-data";
import { BotReplyMaxToolStepsSettingKey } from "@/src/dict/bot-dict";
import { HarnessMaxToolStepsSettingKey } from "@/src/dict/harness-dict";
import readStepLimit from "@/src/utils/read-step-limit";
import grabAppSetting from "./grab-app-setting";
/**
* Model turns a task run may take: the install's own limit
* (`harness_max_tool_steps`, Settings > General), else
* `AppData.HarnessMaxToolSteps`. `runTextLoop`'s default is the code's one,
* so a task run passes this in.
*/
export async function grabHarnessMaxToolSteps() {
return readStepLimit({
saved: await grabAppSetting({ key: HarnessMaxToolStepsSettingKey }),
fallback: AppData.HarnessMaxToolSteps,
max: AppData.BotStepLimitMax,
});
}
/**
* Model turns a chat reply may take: the install's own limit
* (`bot_reply_max_tool_steps`, Settings > General), else
* `AppData.BotReplyMaxToolSteps`.
*/
export async function grabReplyMaxToolSteps() {
return readStepLimit({
saved: await grabAppSetting({ key: BotReplyMaxToolStepsSettingKey }),
fallback: AppData.BotReplyMaxToolSteps,
max: AppData.BotStepLimitMax,
});
}
/**
* Both limits with their defaults, for the settings page.
*/
export default async function grabRunLimitsOverview() {
const [task_steps, reply_steps] = await Promise.all([
grabHarnessMaxToolSteps(),
grabReplyMaxToolSteps(),
]);
const overview: RunLimitsOverview = {
task_steps,
reply_steps,
task_default: AppData.HarnessMaxToolSteps,
reply_default: AppData.BotReplyMaxToolSteps,
max: AppData.BotStepLimitMax,
};
return overview;
}
export type RunLimitsOverview = {
/**
* The limit a task run uses right now.
*/
task_steps: number;
/**
* The limit a chat reply uses right now.
*/
reply_steps: number;
/**
* What a task run uses when the install saved no limit.
*/
task_default: number;
/**
* What a chat reply uses when the install saved no limit.
*/
reply_default: number;
/**
* The most the settings page accepts.
*/
max: number;
};
src/functions/backend/db/bot-files/ingest-bot-file.ts
+74 -5
View File
@@ -7,6 +7,10 @@ import type {
import { AppData } from "@/src/data/app-data"; import { AppData } from "@/src/data/app-data";
import embedText from "@/src/harness/embeddings/embed-text"; import embedText from "@/src/harness/embeddings/embed-text";
import HarnessError from "@/src/harness/harness-error"; import HarnessError from "@/src/harness/harness-error";
import { grabHarnessUsageScope } from "@/src/harness/usage/harness-usage-scope";
import type { HarnessUsageScope } from "@/src/harness/usage/types";
import runWithFallbackScope from "@/src/harness/usage/run-with-fallback-scope";
import grabUsageOwnerUserId from "@/src/functions/backend/db/usage/grab-usage-owner";
import chunkText from "./chunk-text"; import chunkText from "./chunk-text";
import extractBotFileText from "./extract-bot-file-text"; import extractBotFileText from "./extract-bot-file-text";
@@ -72,13 +76,27 @@ export default async function ingestBotFile({
return { status: "skipped", chunks: 0 }; return { status: "skipped", chunks: 0 };
} }
/**
* A bot's own run already carries its user. The ingest queue has no run
* behind it, so everything this file costs - reading an image or an
* audio file, embedding its chunks - is entered under the owner's scope
* instead: it spends real tokens and must show on the usage page.
*/
const owner_scope: HarnessUsageScope | null = grabHarnessUsageScope()?.user_id
? null
: await resolveOwnerScope({ file });
try { try {
const text = await extractBotFileText({ const text = await runWithFallbackScope({
scope: owner_scope,
task: () =>
extractBotFileText({
write_path: media?.media_write_path, write_path: media?.media_write_path,
text_content: media?.text_content, text_content: media?.text_content,
name: file.name, name: file.name,
mime_type: file.mime_type, mime_type: file.mime_type,
read_media, read_media,
}),
}); });
await clearChunks({ file_id: file.id }); await clearChunks({ file_id: file.id });
@@ -107,11 +125,11 @@ export default async function ingestBotFile({
*/ */
for (let start = 0; start < chunks.length; start += AppData.EmbeddingBatchSize) { for (let start = 0; start < chunks.length; start += AppData.EmbeddingBatchSize) {
const batch = chunks.slice(start, start + AppData.EmbeddingBatchSize); const batch = chunks.slice(start, start + AppData.EmbeddingBatchSize);
const { vectors } = await embedText({ const { vectors } = await runEmbed({
input: batch, scope: owner_scope,
bot_id: file.bot_id!,
workspace_id: file.scope == "workspace" ? file.owner_id : null,
signal, signal,
file,
batch,
}); });
for (let i = 0; i < vectors.length; i++) { for (let i = 0; i < vectors.length; i++) {
@@ -252,3 +270,54 @@ async function markIngested({
values: [chunks > 0 ? 1 : 0, now, file_id], values: [chunks > 0 ? 1 : 0, now, file_id],
}).catch(() => {}); }).catch(() => {});
} }
/**
* One batch of a file's chunks, embedded under whatever attribution owns
* it: the run that asked for the ingest, else the file's owner.
*/
async function runEmbed({
scope,
signal,
file,
batch,
}: {
scope: HarnessUsageScope | null;
signal?: AbortSignal;
file: BUN_SQLITE_COBBLE_BOT_FILES;
batch: string[];
}) {
return await runWithFallbackScope({
scope,
task: () =>
embedText({
input: batch,
bot_id: file.bot_id!,
workspace_id: file.scope == "workspace" ? file.owner_id : null,
signal,
}),
});
}
/**
* The attribution a file the queue ingests reports under: its workspace's
* creator when it shares one, else its bot's.
*/
async function resolveOwnerScope({
file,
}: {
file: BUN_SQLITE_COBBLE_BOT_FILES;
}): Promise<HarnessUsageScope | null> {
const workspace_id = file.scope == "workspace" ? file.owner_id : null;
const user_id = await grabUsageOwnerUserId({ bot_id: file.bot_id, workspace_id });
if (!user_id) return null;
return {
user_id,
bot_id: file.scope == "bot" ? file.owner_id : file.bot_id,
workspace_id,
session_id: null,
message_id: null,
job_id: null,
};
}
src/functions/backend/db/bots-queue/complete-bot-job.ts
+12
View File
@@ -89,6 +89,18 @@ export default async function completeBotJob({
media_ids: attachments, media_ids: attachments,
}); });
if (job.id && message?.id) {
/**
* The job's model calls were written down as they happened, before
* its message existed. Now that it does, they belong to it, so the
* thread can show what the answer cost.
*/
await BunSQLite.sql({
sql: `UPDATE token_usages SET message_id = ? WHERE job_id = ? AND message_id IS NULL`,
values: [message.id, job.id],
}).catch(() => {});
}
const user_id = await grabJobUserId({ job }); const user_id = await grabJobUserId({ job });
const bot = !user_id const bot = !user_id
src/functions/backend/db/bots-queue/open-bot-jobs-sql.test.ts
+69
View File
@@ -0,0 +1,69 @@
import { Database } from "bun:sqlite";
import { describe, expect, test } from "bun:test";
import openBotJobsSql from "./open-bot-jobs-sql";
import type { SqlCondition } from "@/src/utils/crew-access/types";
/**
* What "stop the bot" pulls the plug on: a bot's open jobs for one user,
* across every one of their conversations with it. The test runs the
* condition against real SQLite, since a wrong join would either leave a
* job running (a stuck bot stays stuck) or stop another user's work.
*/
const db = new Database(":memory:");
db.run(`CREATE TABLE bot_sessions (id INTEGER PRIMARY KEY, user_id INTEGER, bot_id INTEGER)`);
db.run(
`CREATE TABLE bots_queue (id INTEGER PRIMARY KEY, bot_id INTEGER, session_id INTEGER, status TEXT, kind TEXT, title TEXT)`,
);
db.run(`INSERT INTO bot_sessions VALUES (100, 7, 2), (101, 7, 2), (102, 8, 2), (103, 9, 3)`);
db.run(`INSERT INTO bots_queue VALUES
(1, 2, 100, 'pending', 'reply', 'Thinking'),
(2, 2, 101, 'processing', 'task', 'The audit'),
(3, 2, 102, 'processing', 'task', 'Another user''s job'),
(4, 2, 100, 'completed', 'reply', 'Thinking'),
(5, 2, 100, 'failed', 'task', 'An old failure'),
(6, 3, 103, 'pending', 'reply', 'Thinking'),
(7, 3, 103, 'processing', 'task', 'A finished bot''s job')`);
function ids(condition: SqlCondition) {
return db
.query(`SELECT q.id FROM bots_queue q WHERE ${condition.sql} ORDER BY q.id`)
.all(...condition.values)
.map((row) => (row as { id: number }).id);
}
describe("openBotJobsSql", () => {
test("takes every open job of the bot across the user's conversations, tasks and replies alike", () => {
expect(ids(openBotJobsSql({ bot_id: 2, user_id: 7 }))).toEqual([1, 2]);
});
test("never takes another user's job, even on the same bot", () => {
expect(ids(openBotJobsSql({ bot_id: 2, user_id: 7 }))).not.toContain(3);
});
test("leaves finished jobs alone, so the work they did still counts", () => {
expect(ids(openBotJobsSql({ bot_id: 2, user_id: 7 }))).not.toContain(4);
expect(ids(openBotJobsSql({ bot_id: 2, user_id: 7 }))).not.toContain(5);
});
test("another bot's jobs are never in it", () => {
expect(ids(openBotJobsSql({ bot_id: 2, user_id: 7 }))).not.toContain(6);
expect(ids(openBotJobsSql({ bot_id: 3, user_id: 9 }))).toEqual([6, 7]);
});
test("a user with nothing open stops nothing", () => {
expect(ids(openBotJobsSql({ bot_id: 2, user_id: 8 }))).toEqual([3]);
expect(ids(openBotJobsSql({ bot_id: 2, user_id: 99 }))).toEqual([]);
});
test("reads a join alias when one is given", () => {
const condition = openBotJobsSql({ bot_id: 2, user_id: 7, alias: "j" });
expect(
db
.query(`SELECT j.id FROM bots_queue j WHERE ${condition.sql} ORDER BY j.id`)
.all(...condition.values)
.map((row) => (row as { id: number }).id),
).toEqual([1, 2]);
});
});
src/functions/backend/db/bots-queue/open-bot-jobs-sql.ts
+29
View File
@@ -0,0 +1,29 @@
import type { SqlCondition } from "@/src/utils/crew-access/types";
type Params = {
bot_id: number | string;
/**
* Only jobs in this user's conversations, never another user's.
*/
user_id: number | string;
/**
* The `bots_queue` alias the condition reads.
*/
alias?: string;
};
/**
* The SQL for a bot's jobs for one user that can still run or are running:
* pending (including one held back with `available_at` until a model comes
* back) and processing (including one waiting for its computer). A job's
* session is what makes it the user's, so another user's job is never in
* it. This is what "stop the bot" pulls the plug on: with none of these
* left, nothing of the bot's starts again until the user asks.
*/
export default function openBotJobsSql({ bot_id, user_id, alias = "q" }: Params): SqlCondition {
return {
sql: `(${alias}.bot_id = ? AND ${alias}.status IN ('pending', 'processing')
AND ${alias}.session_id IN (SELECT id FROM bot_sessions WHERE user_id = ?))`,
values: [Number(bot_id), Number(user_id)],
};
}
src/functions/backend/db/bots-queue/stop-bot-jobs.test.ts
+31
View File
@@ -0,0 +1,31 @@
import { Database } from "bun:sqlite";
import { describe, expect, test } from "bun:test";
import stopBotJobs from "./stop-bot-jobs";
/**
* Stopping the bot from the chat stops every open job of it in the user's
* conversations and nothing else: another user's work, another bot's work
* and finished jobs all survive, and the bot's own handler must find out
* (its lock is gone) so it drops what it was producing.
*
* The helper talks to bun-sqlite's process-wide client, so this suite has to
* drive a real one. It's skipped where the client isn't set up rather than
* failing the other suites; `open-bot-jobs-sql.test.ts` covers the
* selection itself against real SQLite unconditionally.
*/
const sqlite_ready = (() => {
try {
new Database(":memory:").run(`SELECT 1`);
return true;
} catch {
return false;
}
})();
describe.skipIf(!sqlite_ready)("stopBotJobs", () => {
test("a user with nothing open is a no-op", async () => {
const stopped = await stopBotJobs({ bot_id: 999_999, user_id: 999_999 });
expect(stopped).toBeNull();
});
});
src/functions/backend/db/bots-queue/stop-bot-jobs.ts
+130
View File
@@ -0,0 +1,130 @@
import BunSQLite from "@moduletrace/bun-sqlite";
import type { BUN_SQLITE_COBBLE_BOT_MESSAGES, BUN_SQLITE_COBBLE_BOTS_QUEUE } from "@/db/types/db";
import { BotJobStoppedError, BotJobStoppedStep } from "@/src/dict/bot-queue-dict";
import insertBotMessage from "../bots/insert-bot-message";
import setBotFace, { type BotFaceChange } from "../bots/set-bot-face";
import { abortRunningJob } from "@/src/bots-queue/job-abort-registry";
import insertBotJobUpdate from "./insert-bot-job-update";
import openBotJobsSql from "./open-bot-jobs-sql";
import stoppedJobReport, { pickReportedJobs } from "./stopped-job-report";
type Params = {
bot_id: number | string;
/**
* Whose conversations stop. Never another user's jobs.
*/
user_id: number | string;
};
export type StoppedBotJobs = {
/**
* The jobs that were open and are now failed as stopped.
*/
jobs: BUN_SQLITE_COBBLE_BOTS_QUEUE[];
/**
* One closing message per session the bot worked in, for the live
* push. It's what ends the thread's "still thinking" state: the last
* word is the bot's again.
*/
messages: BUN_SQLITE_COBBLE_BOT_MESSAGES[];
/**
* The user's face change (idle), for the live push.
*/
bot: BotFaceChange;
};
/**
* Pulls the plug on a bot for one user: every job of it that could still
* run or is running in their conversations (replies it still owes, tasks
* it's on, across all of them) is failed as stopped, its task closed. A
* running handler finds out on its next `updateBotJob` (which returns
* false) and drops whatever it was producing. Each job gets a closing
* step, so a thread's working row says why it went quiet even after a
* reload, one message per session ends it out loud, and the bot goes
* idle. It's not a bus: nothing restarts by itself, so a stopped bot
* works again only when the user asks. A schedule the user set up still
* comes round, since that is work they asked for.
*/
export default async function stopBotJobs({ bot_id, user_id }: Params): Promise<StoppedBotJobs | null> {
const condition = openBotJobsSql({ bot_id, user_id });
const open = (
await BunSQLite.sql<BUN_SQLITE_COBBLE_BOTS_QUEUE>({
sql: `SELECT * FROM bots_queue q WHERE ${condition.sql} ORDER BY q.id ASC`,
values: condition.values,
})
).payload || [];
if (!open.length) return null;
const now = Date.now();
const jobs: BUN_SQLITE_COBBLE_BOTS_QUEUE[] = [];
for (const job of open) {
/**
* One by one, and only while it's still open: a job that finished
* on its own between the read and here keeps its result.
*/
const res = await BunSQLite.sql<BUN_SQLITE_COBBLE_BOTS_QUEUE>({
sql: `UPDATE bots_queue SET status = 'failed', last_error = ?, locked_at = NULL, locked_by = NULL, finished_at = ?, updated_at = ?
WHERE id = ? AND status IN ('pending', 'processing')
RETURNING *`,
values: [BotJobStoppedError, now, now, Number(job.id)],
});
const stopped = res.payload?.[0];
if (!stopped?.id) continue;
jobs.push(stopped);
/**
* If this process is the one running it, stop what it's waiting on
* now. Anywhere else, the handler finds out on its next heartbeat.
*/
abortRunningJob({ job_id: Number(stopped.id) });
if (stopped.task_id) {
await BunSQLite.sql({
sql: `UPDATE bot_tasks SET status = 'done', updated_at = ? WHERE id = ?`,
values: [now, Number(stopped.task_id)],
});
}
/**
* Recorded as the job's last step, so the working row says why it
* went quiet.
*/
try {
await insertBotJobUpdate({
job_id: Number(stopped.id),
bot_id: Number(stopped.bot_id) || Number(bot_id),
kind: "note",
text: BotJobStoppedStep,
});
} catch {
/**
* The step is a courtesy: the job is stopped either way.
*/
}
}
const messages: BUN_SQLITE_COBBLE_BOT_MESSAGES[] = [];
for (const [session_id, job] of pickReportedJobs({ jobs })) {
const message = await insertBotMessage({
bot_id: Number(bot_id),
session_id,
workspace_id: job.workspace_id || null,
text: stoppedJobReport({ job }),
}).catch(() => null);
if (message?.id) messages.push(message);
}
return {
jobs,
messages,
bot: await setBotFace({ bot_id, user_id, face: "idle" }),
};
}
src/functions/backend/db/bots-queue/stopped-job-report.test.ts
+79
View File
@@ -0,0 +1,79 @@
import { describe, expect, test } from "bun:test";
import { BotStoppedMessage } from "@/src/dict/bot-queue-dict";
import stoppedJobReport, { pickReportedJobs } from "./stopped-job-report";
describe("stoppedJobReport", () => {
test("a stopped task names the work that was cut off", () => {
expect(stoppedJobReport({ job: { session_id: 1, kind: "task", title: "The audit" } })).toBe(
`${BotStoppedMessage} I was on **The audit**.`,
);
});
test("a stopped reply just says so, since it has no title worth quoting", () => {
expect(stoppedJobReport({ job: { session_id: 1, kind: "reply", title: "Thinking" } })).toBe(
BotStoppedMessage,
);
});
test("a task without a title doesn't leave a dangling bold sentence", () => {
expect(stoppedJobReport({ job: { session_id: 1, kind: "task", title: " " } })).toBe(
BotStoppedMessage,
);
});
test("the message is a sentence the user can read, and it says the bot waits for them", () => {
expect(BotStoppedMessage).toContain("Stopped");
expect(BotStoppedMessage).toContain("when you ask");
});
});
describe("pickReportedJobs", () => {
test("one job speaks per session, so a reply and the task it started don't both report", () => {
const picked = pickReportedJobs({
jobs: [
{ session_id: 7, kind: "reply", id: 1 },
{ session_id: 7, kind: "task", id: 2 },
],
});
expect([...picked.keys()]).toEqual([7]);
});
test("the task wins, since it's the one that can name the work", () => {
const picked = pickReportedJobs({
jobs: [
{ session_id: 7, kind: "reply", id: 1 },
{ session_id: 7, kind: "task", id: 2 },
],
});
expect(picked.get(7)?.id).toBe(2);
});
test("a task first isn't replaced by a later reply", () => {
const picked = pickReportedJobs({
jobs: [
{ session_id: 7, kind: "task", id: 2 },
{ session_id: 7, kind: "reply", id: 1 },
],
});
expect(picked.get(7)?.id).toBe(2);
});
test("every session gets its own report", () => {
const picked = pickReportedJobs({
jobs: [
{ session_id: 7, kind: "reply", id: 1 },
{ session_id: 8, kind: "task", id: 2 },
{ session_id: 9, kind: "task", id: 3 },
],
});
expect([...picked.keys()].sort()).toEqual([7, 8, 9]);
});
test("nothing open picks nothing", () => {
expect(pickReportedJobs({ jobs: [] }).size).toBe(0);
});
});
src/functions/backend/db/bots-queue/stopped-job-report.ts
+45
View File
@@ -0,0 +1,45 @@
import { BotStoppedMessage } from "@/src/dict/bot-queue-dict";
type Job = {
session_id?: number | string | null;
kind?: string | null;
title?: string | null;
};
/**
* What the bot says in a conversation the user stopped it in. A task names
* the work that was cut off, so a long job that quietly went nowhere reads
* as stopped rather than as finished; a reply has no title worth quoting.
*/
export default function stoppedJobReport({ job }: { job: Job }): string {
const title = (job.title || "").trim();
return job.kind == "task" && title
? `${BotStoppedMessage} I was on **${title}**.`
: BotStoppedMessage;
}
/**
* The job that speaks for each session: one per session, since a reply and
* the task it started are the same conversation and two notes there read as
* a stutter. A task wins, because it's the one that can name the work.
*/
export function pickReportedJobs<T extends Job>({ jobs }: { jobs: T[] }): Map<number, T> {
const picked = new Map<number, T>();
for (const job of jobs) {
/**
* A job with no session is stopped all the same; it just has
* nowhere to report it.
*/
if (job.session_id === null || job.session_id === undefined) continue;
const known = picked.get(Number(job.session_id));
if (!known || (known.kind != "task" && job.kind == "task")) {
picked.set(Number(job.session_id), job);
}
}
return picked;
}
src/functions/backend/db/bots/grab-bot-messages.ts
+6 -1
View File
@@ -1,6 +1,7 @@
import type { BUN_SQLITE_COBBLE_BOT_MESSAGES } from "@/db/types/db"; import type { BUN_SQLITE_COBBLE_BOT_MESSAGES } from "@/db/types/db";
import type { BotMessageWithAttachments } from "@/src/types"; import type { BotMessageWithAttachments } from "@/src/types";
import BunSQLite from "@moduletrace/bun-sqlite"; import BunSQLite from "@moduletrace/bun-sqlite";
import grabMessageUsage from "../usage/grab-message-usage";
import grabMessagesMedia from "./grab-messages-media"; import grabMessagesMedia from "./grab-messages-media";
import grabMessagesReactions from "./grab-messages-reactions"; import grabMessagesReactions from "./grab-messages-reactions";
import grabMessagesReplies from "./grab-messages-replies"; import grabMessagesReplies from "./grab-messages-replies";
@@ -53,12 +54,13 @@ export default async function grabBotMessages({
const messages = res.payload || []; const messages = res.payload || [];
const message_ids = messages.map((message) => Number(message.id)); const message_ids = messages.map((message) => Number(message.id));
const [media, replies, reactions] = await Promise.all([ const [media, replies, reactions, usage] = await Promise.all([
grabMessagesMedia({ message_ids }), grabMessagesMedia({ message_ids }),
grabMessagesReplies({ message_ids }), grabMessagesReplies({ message_ids }),
user_id user_id
? grabMessagesReactions({ message_ids, user_id }) ? grabMessagesReactions({ message_ids, user_id })
: Promise.resolve(new Map()), : Promise.resolve(new Map()),
grabMessageUsage({ message_ids }),
]); ]);
/** /**
@@ -74,6 +76,8 @@ export default async function grabBotMessages({
const files = media.get(id); const files = media.get(id);
const reply_to = replies.get(id); const reply_to = replies.get(id);
const spent = usage.get(id);
return { return {
...message, ...message,
...(files ...(files
@@ -88,6 +92,7 @@ export default async function grabBotMessages({
: {}), : {}),
...(reply_to ? { reply_to } : {}), ...(reply_to ? { reply_to } : {}),
...(user_id ? { reactions: reactions.get(id) || [] } : {}), ...(user_id ? { reactions: reactions.get(id) || [] } : {}),
...(spent ? { usage: spent } : {}),
}; };
}); });
} }
src/functions/backend/db/cron-jobs/grab-cron-jobs.ts
+23 -3
View File
@@ -1,6 +1,7 @@
import BunSQLite from "@moduletrace/bun-sqlite"; import BunSQLite from "@moduletrace/bun-sqlite";
import type { CronJobWithNames } from "@/src/types"; import type { CronJobWithNames } from "@/src/types";
import botFaceSql from "@/src/utils/bot-face-sql"; import botFaceSql from "@/src/utils/bot-face-sql";
import routineScopeSql from "@/src/utils/routine-scope/routine-scope-sql";
type Params = { type Params = {
user_id: number | string; user_id: number | string;
@@ -8,6 +9,12 @@ type Params = {
* Only this bot's. * Only this bot's.
*/ */
bot_id?: number | string; bot_id?: number | string;
/**
* With `bot_id`: only the routines of that conversation, here
* (`routineScopeSql`). `null` is the bot's own chat, so it has to be
* given rather than left out to mean "everywhere".
*/
workspace_id?: number | string | null;
/** /**
* Only this one. * Only this one.
*/ */
@@ -16,17 +23,30 @@ type Params = {
/** /**
* The user's schedules with their bot (its face as the user sees it) and * The user's schedules with their bot (its face as the user sees it) and
* workspace names, the next to * workspace names, the next to run first and paused ones last. Given a
* run first and paused ones last. * `bot_id` and a `workspace_id` (null for its own chat) it is one
* conversation's: the routines that run right there.
*/ */
export default async function grabCronJobs({ user_id, bot_id, cron_job_id }: Params): Promise<CronJobWithNames[]> { export default async function grabCronJobs({
user_id,
bot_id,
workspace_id,
cron_job_id,
}: Params): Promise<CronJobWithNames[]> {
const face = botFaceSql({ user_id, prefix: "bot_" }); const face = botFaceSql({ user_id, prefix: "bot_" });
const where = [`c.user_id = ?`]; const where = [`c.user_id = ?`];
const values: (number | string)[] = [user_id]; const values: (number | string)[] = [user_id];
if (bot_id) { if (bot_id) {
if (workspace_id === undefined) {
where.push(`c.bot_id = ?`); where.push(`c.bot_id = ?`);
values.push(bot_id); values.push(bot_id);
} else {
const scope = routineScopeSql({ bot_id, workspace_id });
where.push(scope.sql);
values.push(...scope.values);
}
} }
if (cron_job_id) { if (cron_job_id) {
src/functions/backend/db/usage/bucket-usage-days.test.ts
+101
View File
@@ -0,0 +1,101 @@
import { describe, expect, test } from "bun:test";
import bucketUsageDays, { localDate } from "./bucket-usage-days";
import { DayMs, startOfDay } from "./grab-usage-window";
function row(local_day: string, tokens: number) {
return {
local_day,
calls: 1,
prompt_tokens: tokens,
completion_tokens: 0,
cost: null,
};
}
const now = new Date(2026, 9, 23, 15, 0, 0).getTime();
const end = now;
const start = startOfDay({ at: now }) - 2 * DayMs;
describe("localDate", () => {
test("names the local calendar day, not the UTC one", () => {
/**
* Half an hour past local midnight is still the previous UTC day
* for anyone east of Greenwich, and the bug the chart had: the row
* was counted under one name and looked for under another.
*/
const at = new Date(2026, 9, 23, 0, 30, 0).getTime();
expect(localDate({ at })).toBe("2026-10-23");
});
test("pads a single-digit month and day", () => {
expect(localDate({ at: new Date(2026, 0, 5, 12, 0, 0).getTime() })).toBe("2026-01-05");
});
});
describe("bucketUsageDays", () => {
test("one bucket per day of the window, oldest first", () => {
const days = bucketUsageDays({ rows: [], start, end });
expect(days.length).toBe(3);
expect(days.map((day) => day.label)).toEqual(
[2, 1, 0].map((back) => localDate({ at: startOfDay({ at: now }) - back * DayMs })),
);
});
test("every row lands in its day, and yesterday's late call does not vanish", () => {
const today = localDate({ at: startOfDay({ at: now }) });
const yesterday = localDate({ at: startOfDay({ at: now }) - DayMs });
const days = bucketUsageDays({
rows: [row(today, 1500), row(yesterday, 400)],
start,
end,
});
const by_label = new Map(days.map((day) => [day.label, day.total_tokens]));
expect(by_label.get(today)).toBe(1500);
expect(by_label.get(yesterday)).toBe(400);
expect(days.reduce((total, day) => total + day.total_tokens, 0)).toBe(1900);
});
test("a day with nothing in it is still a bucket, at zero", () => {
const days = bucketUsageDays({
rows: [row(localDate({ at: startOfDay({ at: now }) }), 100)],
start,
end,
});
expect(days.filter((day) => day.total_tokens === 0).length).toBe(2);
expect(days.every((day) => day.calls === 0 || day.calls === 1)).toBe(true);
});
test("a bucket's midnight is that day's local midnight", () => {
const days = bucketUsageDays({ rows: [], start, end });
for (const day of days) {
expect(day.at).toBe(startOfDay({ at: day.at }));
expect(localDate({ at: day.at })).toBe(day.label);
}
});
test("cost and calls are summed, not counted twice", () => {
const label = localDate({ at: startOfDay({ at: now }) });
const days = bucketUsageDays({
rows: [
{ ...row(label, 100), calls: 3, cost: 0.25 },
{ ...row(label, 50), calls: 2, cost: 0.5 },
],
start,
end,
});
const bucket = days.find((day) => day.label === label);
expect(bucket?.total_tokens).toBe(150);
expect(bucket?.cost).toBe(0.75);
expect(days.reduce((total, day) => total + day.calls, 0)).toBe(5);
});
});
src/functions/backend/db/usage/bucket-usage-days.ts
+115
View File
@@ -0,0 +1,115 @@
import { DayMs, startOfDay } from "./grab-usage-window";
import { toSummary } from "./grab-usage-totals";
import type { TokenUsageSummary } from "@/src/harness/usage/types";
type Row = {
/**
* The day the call was made, `YYYY-MM-DD`, already in local time.
*/
local_day: string;
calls: number | null;
prompt_tokens: number | null;
completion_tokens: number | null;
cost: number | null;
};
type Params = {
rows: Row[];
start: number;
end: number;
};
export type UsageDay = {
/**
* The day's local midnight (epoch ms).
*/
at: number;
/**
* `YYYY-MM-DD`, in local time.
*/
label: string;
} & TokenUsageSummary;
/**
* The rows SQL grouped by local day, laid onto one bucket per day of the
* window, oldest first. Days with nothing in them are still there, at
* zero, so the bars keep their spacing, and every row lands in a bucket:
* a day outside the window would be a bug in the query's own bounds.
*/
export default function bucketUsageDays({ rows, start, end }: Params): UsageDay[] {
const by_day = new Map<string, TokenUsageSummary>();
for (const row of rows) {
/**
* Accumulated, not replaced: the query groups by day, so a repeated
* day means someone asked for two sets of rows, and the later one
* must add to the earlier rather than hide it.
*/
const label = String(row.local_day);
const summary = toSummary({ row });
by_day.set(label, mergeSummaries({ into: by_day.get(label), summary }));
}
const days: UsageDay[] = [];
const last = startOfDay({ at: end });
for (let at = startOfDay({ at: start }); at <= last; at += DayMs) {
const label = localDate({ at });
days.push({
at,
label,
...(by_day.get(label) || emptySummary()),
});
}
return days;
}
/**
* The day a local midnight falls on, in local time.
*
* Never `toISOString`: an epoch millisecond read in UTC is a different
* calendar day for anyone east or west of Greenwich, and a call counted
* under one day would then look for a bucket named after another and
* disappear from the chart (which is exactly what it did before).
*/
export function localDate({ at }: { at: number }) {
const date = new Date(at);
return [
date.getFullYear(),
String(date.getMonth() + 1).padStart(2, "0"),
String(date.getDate()).padStart(2, "0"),
].join("-");
}
function emptySummary(): TokenUsageSummary {
return { prompt_tokens: 0, completion_tokens: 0, total_tokens: 0, calls: 0 };
}
/**
* One summary added onto another. A day that reported no cost still costs
* nothing; only a real cost is carried, so an unpriced day doesn't turn a
* priced one into zero.
*/
function mergeSummaries({
into,
summary,
}: {
into?: TokenUsageSummary;
summary: TokenUsageSummary;
}): TokenUsageSummary {
const prompt_tokens = (into?.prompt_tokens || 0) + summary.prompt_tokens;
const completion_tokens = (into?.completion_tokens || 0) + summary.completion_tokens;
const cost = (into?.cost || 0) + (summary.cost || 0);
return {
prompt_tokens,
completion_tokens,
total_tokens: prompt_tokens + completion_tokens,
...(cost > 0 ? { cost } : {}),
calls: (into?.calls || 0) + summary.calls,
};
}
src/functions/backend/db/usage/grab-message-usage.ts
+56
View File
@@ -0,0 +1,56 @@
import BunSQLite from "@moduletrace/bun-sqlite";
import type { TokenUsageSummary } from "@/src/harness/usage/types";
type Params = {
/**
* The messages whose calls are summed. A message with no calls is
* absent from the answer.
*/
message_ids: number[];
};
/**
* What each of these messages cost, summed over every model call it took
* (chat turns, media readings, image generations). This is what a
* message's token line shows.
*/
export default async function grabMessageUsage({ message_ids }: Params) {
const totals = new Map<number, TokenUsageSummary>();
const ids = [...new Set(message_ids.map(Number).filter((id) => id > 0))];
if (!ids.length) return totals;
const res = await BunSQLite.sql<{
message_id: number;
prompt_tokens: number | null;
completion_tokens: number | null;
cost: number | null;
calls: number;
}>({
sql: `SELECT message_id,
SUM(prompt_tokens) AS prompt_tokens,
SUM(completion_tokens) AS completion_tokens,
SUM(cost) AS cost,
COUNT(*) AS calls
FROM token_usages
WHERE message_id IN (${ids.map(() => "?").join(", ")})
GROUP BY message_id`,
values: ids,
});
for (const row of res.payload || []) {
const prompt_tokens = Number(row.prompt_tokens) || 0;
const completion_tokens = Number(row.completion_tokens) || 0;
const cost = Number(row.cost);
totals.set(Number(row.message_id), {
prompt_tokens,
completion_tokens,
total_tokens: prompt_tokens + completion_tokens,
...(Number.isFinite(cost) && cost > 0 ? { cost } : {}),
calls: Number(row.calls) || 0,
});
}
return totals;
}
src/functions/backend/db/usage/grab-usage-days.ts
+42
View File
@@ -0,0 +1,42 @@
import BunSQLite from "@moduletrace/bun-sqlite";
import bucketUsageDays, { type UsageDay } from "./bucket-usage-days";
export type { UsageDay };
type Params = {
start: number;
end: number;
};
/**
* One bucket per day in the window, oldest first, each the tokens spent
* that day. Days with nothing in them are still there, at zero, so the
* bars keep their spacing.
*
* SQLite has no timezone support, so the day is taken with its `localtime`
* modifier, which reads the same server timezone the page's own day
* boundaries come from. Grouping by the bare UTC day instead put a late
* evening's calls in the previous day's bucket, and dropped them whenever
* no bucket carried that name.
*/
export default async function grabUsageDays({ start, end }: Params): Promise<UsageDay[]> {
const res = await BunSQLite.sql<{
local_day: string;
calls: number | null;
prompt_tokens: number | null;
completion_tokens: number | null;
cost: number | null;
}>({
sql: `SELECT date(created_at / 1000, 'unixepoch', 'localtime') AS local_day,
COUNT(*) AS calls,
SUM(prompt_tokens) AS prompt_tokens,
SUM(completion_tokens) AS completion_tokens,
SUM(cost) AS cost
FROM token_usages
WHERE created_at >= ? AND created_at < ?
GROUP BY local_day`,
values: [start, end],
});
return bucketUsageDays({ rows: res.payload || [], start, end });
}
src/functions/backend/db/usage/grab-usage-models.ts
+75
View File
@@ -0,0 +1,75 @@
import BunSQLite from "@moduletrace/bun-sqlite";
import { toSummary } from "./grab-usage-totals";
import type { TokenUsageSummary } from "@/src/harness/usage/types";
type Params = {
start: number;
end: number;
/**
* How many the page shows, busiest first.
*/
limit: number;
};
export type UsageModelRow = {
/**
* The provider's own id for the model, as the call recorded it
* (`deepseek/deepseek-chat`). This is what a person compares, not the
* catalog row's numeric id.
*/
model_slug: string;
/**
* The catalog's display name, when the model is still listed. A model
* a provider stopped listing keeps its slug and loses its name.
*/
model_name: string | null;
provider_name: string;
provider_id: number | null;
harness: string;
} & TokenUsageSummary;
/**
* Every model the period ran, busiest first (by tokens), with what it
* spent. This is how one works out what a model costs: its tokens against
* its calls, for the window asked for.
*/
export default async function grabUsageModels({ start, end, limit }: Params): Promise<UsageModelRow[]> {
const res = await BunSQLite.sql<{
model_slug: string | null;
model_name: string | null;
provider_name: string | null;
provider_id: number | null;
harness: string | null;
calls: number | null;
prompt_tokens: number | null;
completion_tokens: number | null;
cost: number | null;
}>({
sql: `SELECT u.model_slug,
m.name AS model_name,
u.provider_name,
u.provider_id,
u.harness,
COUNT(*) AS calls,
SUM(u.prompt_tokens) AS prompt_tokens,
SUM(u.completion_tokens) AS completion_tokens,
SUM(u.cost) AS cost
FROM token_usages u
LEFT JOIN harness_models m
ON m.provider_id = u.provider_id AND m.slug = u.model_slug
WHERE u.created_at >= ? AND u.created_at < ?
GROUP BY u.provider_name, u.model_slug, u.harness
ORDER BY SUM(u.prompt_tokens) + SUM(u.completion_tokens) DESC
LIMIT ?`,
values: [start, end, limit],
});
return (res.payload || []).map((row) => ({
model_slug: row.model_slug || "unknown",
model_name: row.model_name || null,
provider_name: row.provider_name || "unknown",
provider_id: row.provider_id == null ? null : Number(row.provider_id),
harness: String(row.harness || "text"),
...toSummary({ row }),
}));
}
src/functions/backend/db/usage/grab-usage-overview.ts
+53
View File
@@ -0,0 +1,53 @@
import { AppData } from "@/src/data/app-data";
import type { UsageOverview } from "@/src/types";
import grabUsageDays from "./grab-usage-days";
import grabUsageModels from "./grab-usage-models";
import grabUsageTotals from "./grab-usage-totals";
import grabUsageWindow, { DayMs } from "./grab-usage-window";
type Params = {
/**
* How many days back to total, today included.
*/
days?: number | string | null;
now?: number;
};
/**
* Everything the usage page shows for one window: the totals, the same day
* by day, per harness and per model, plus the day before the window and a
* rolling week, so the page can say whether work is going up or down.
*/
export default async function grabUsageOverview({
days,
now = Date.now(),
}: Params): Promise<UsageOverview> {
const window = grabUsageWindow({ days, now });
const previous_start = window.start - window.days * DayMs;
const [totals, days_by_day, models, previous, week] = await Promise.all([
grabUsageTotals({ start: window.start, end: window.end }),
grabUsageDays({ start: window.start, end: window.end }),
grabUsageModels({
start: window.start,
end: window.end,
limit: AppData.UsageModelPageSize,
}),
grabUsageTotals({ start: previous_start, end: window.start }),
grabUsageTotals({ start: window.end - 7 * DayMs, end: window.end }),
]);
return {
window: {
days: window.days,
start: window.start,
end: window.end,
},
totals: totals.totals,
by_harness: totals.by_harness,
days: days_by_day,
models,
previous: previous.totals,
week: week.totals,
};
}
src/functions/backend/db/usage/grab-usage-owner.ts
+47
View File
@@ -0,0 +1,47 @@
import BunSQLite from "@moduletrace/bun-sqlite";
type Params = {
/**
* The bot whose file it is, or the bot the search ran for.
*/
bot_id?: number | string | null;
/**
* The workspace whose file it is, when it shares one.
*/
workspace_id?: number | string | null;
};
/**
* Whose usage a call made outside anyone's run belongs to: a file ingested
* by the queue, or a search that ran with no conversation behind it. The
* workspace's creator when there is a workspace, else the bot's, so an
* embedding never goes uncounted.
*/
export default async function grabUsageOwnerUserId({
bot_id,
workspace_id,
}: Params): Promise<number | null> {
const workspace = Number(workspace_id);
if (Number.isFinite(workspace) && workspace > 0) {
const row = await BunSQLite.sql<{ user_id: number }>({
sql: `SELECT user_id FROM workspaces WHERE id = ? LIMIT 1`,
values: [workspace],
}).then((res) => res.payload?.[0]);
if (row?.user_id) return Number(row.user_id) || null;
}
const bot = Number(bot_id);
if (Number.isFinite(bot) && bot > 0) {
const row = await BunSQLite.sql<{ user_id: number }>({
sql: `SELECT user_id FROM bots WHERE id = ? LIMIT 1`,
values: [bot],
}).then((res) => res.payload?.[0]);
if (row?.user_id) return Number(row.user_id) || null;
}
return null;
}
src/functions/backend/db/usage/grab-usage-totals.ts
+97
View File
@@ -0,0 +1,97 @@
import BunSQLite from "@moduletrace/bun-sqlite";
import type { TokenUsageSummary } from "@/src/harness/usage/types";
type Params = {
/**
* The window's start and end (epoch ms), from `grabUsageWindow`.
*/
start: number;
end: number;
};
type UsageRow = {
calls: number | null;
prompt_tokens: number | null;
completion_tokens: number | null;
cost: number | null;
};
/**
* One total for a whole period, and the same split by the harnesses the
* calls were for.
*/
export type UsageTotals = {
totals: TokenUsageSummary;
by_harness: (TokenUsageSummary & { harness: string })[];
};
/**
* What every model call in the period cost, in total and per harness
* (chat, media reading, transcribing, speaking, images, embeddings).
*/
export default async function grabUsageTotals({ start, end }: Params): Promise<UsageTotals> {
const res = await BunSQLite.sql<UsageRow & { harness: string | null }>({
sql: `SELECT harness,
COUNT(*) AS calls,
SUM(prompt_tokens) AS prompt_tokens,
SUM(completion_tokens) AS completion_tokens,
SUM(cost) AS cost
FROM token_usages
WHERE created_at >= ? AND created_at < ?
GROUP BY harness`,
values: [start, end],
});
const by_harness = (res.payload || []).map((row) => ({
harness: String(row.harness || "text"),
...toSummary({ row }),
}));
return {
totals: sumSummary({ summaries: by_harness }),
by_harness: by_harness.sort((a, b) => b.total_tokens - a.total_tokens),
};
}
/**
* Adds summaries into one. A call that reported no tokens still counts as
* a call.
*/
export function sumSummary({ summaries }: { summaries: TokenUsageSummary[] }): TokenUsageSummary {
let prompt_tokens = 0;
let completion_tokens = 0;
let cost = 0;
let calls = 0;
for (const summary of summaries) {
prompt_tokens += summary.prompt_tokens;
completion_tokens += summary.completion_tokens;
cost += summary.cost || 0;
calls += summary.calls;
}
return {
prompt_tokens,
completion_tokens,
total_tokens: prompt_tokens + completion_tokens,
...(cost > 0 ? { cost } : {}),
calls,
};
}
/**
* One grouped row (any shape with the four sums) as a summary.
*/
export function toSummary({ row }: { row: UsageRow }): TokenUsageSummary {
const prompt_tokens = Number(row.prompt_tokens) || 0;
const completion_tokens = Number(row.completion_tokens) || 0;
const cost = Number(row.cost);
return {
prompt_tokens,
completion_tokens,
total_tokens: prompt_tokens + completion_tokens,
...(Number.isFinite(cost) && cost > 0 ? { cost } : {}),
calls: Number(row.calls) || 0,
};
}
src/functions/backend/db/usage/grab-usage-window.ts
+57
View File
@@ -0,0 +1,57 @@
import { AppData } from "@/src/data/app-data";
type Params = {
/**
* How many days back the window starts, today included. Clamped to
* what the page offers.
*/
days?: number | string | null;
/**
* The moment "now" is, so a whole page (its totals and its days) is
* read at one instant.
*/
now?: number;
};
/**
* The period the usage page totals: a start, an end and the day buckets
* between them, oldest first, each a local midnight.
*/
export default function grabUsageWindow({ days, now = Date.now() }: Params) {
const asked = Number(days);
const count =
Number.isFinite(asked) && asked > 0
? Math.min(Math.round(asked), AppData.UsageMaxDays)
: AppData.UsageDefaultDays;
const end = now;
/**
* Whole days back from the start of today, so every bucket is a day's
* work rather than a sliding window's slice.
*/
const today = startOfDay({ at: now });
const start = today - (count - 1) * DayMs;
return {
days: count,
start,
end,
/**
* One bucket per day in the window, oldest first.
*/
buckets: Array.from({ length: count }, (_, index) => start + index * DayMs),
};
}
export const DayMs = 24 * 60 * 60 * 1000;
/**
* Local midnight of the day the moment falls in.
*/
export function startOfDay({ at }: { at: number }) {
const date = new Date(at);
date.setHours(0, 0, 0, 0);
return date.getTime();
}
src/harness/audio/speak-text.ts
+17 -2
View File
@@ -2,7 +2,9 @@ import grabHarnessSetting from "@/src/functions/backend/db/harness/grab-harness-
import grabHarnessAdapter from "../providers/grab-harness-adapter"; import grabHarnessAdapter from "../providers/grab-harness-adapter";
import resolveHarnessCandidates from "../resolve/resolve-harness-candidates"; import resolveHarnessCandidates from "../resolve/resolve-harness-candidates";
import withCandidates from "../resolve/with-candidates"; import withCandidates from "../resolve/with-candidates";
import type { HarnessCandidate } from "../types"; import type { HarnessAudioStream, HarnessCandidate } from "../types";
import buildUsageCall from "../usage/build-usage-call";
import recordHarnessUsage from "../usage/record-harness-usage";
type Params = { type Params = {
text: string; text: string;
@@ -39,9 +41,22 @@ export default async function speakText({
grabHarnessSetting({ harness: "speech", bot_id, workspace_id }), grabHarnessSetting({ harness: "speech", bot_id, workspace_id }),
]); ]);
const { result } = await withCandidates({ const { result } = await withCandidates<HarnessAudioStream>({
harness: "speech", harness: "speech",
candidates, candidates,
/**
* Speech is billed per character and streamed, so there is no
* completion count to read: the call is recorded as it is, with
* the text's length standing in for what was spoken.
*/
on_success: async ({ candidate }) =>
await recordHarnessUsage({
call: buildUsageCall({
harness: "speech",
candidate,
usage: { prompt_tokens: text.length, completion_tokens: 0 },
}),
}),
run: async (candidate) => run: async (candidate) =>
await grabHarnessAdapter({ kind: candidate.provider.kind }).speak({ await grabHarnessAdapter({ kind: candidate.provider.kind }).speak({
candidate, candidate,
src/harness/audio/transcribe-audio.ts
+15 -5
View File
@@ -4,7 +4,9 @@ import HarnessError from "../harness-error";
import grabHarnessAdapter from "../providers/grab-harness-adapter"; import grabHarnessAdapter from "../providers/grab-harness-adapter";
import resolveHarnessCandidates from "../resolve/resolve-harness-candidates"; import resolveHarnessCandidates from "../resolve/resolve-harness-candidates";
import withCandidates from "../resolve/with-candidates"; import withCandidates from "../resolve/with-candidates";
import type { HarnessAudioInput } from "../types"; import type { HarnessAudioInput, HarnessUsage } from "../types";
import buildUsageCall from "../usage/build-usage-call";
import recordHarnessUsage from "../usage/record-harness-usage";
type Params = { type Params = {
audio: HarnessAudioInput; audio: HarnessAudioInput;
@@ -46,15 +48,23 @@ export default async function transcribeAudio({
const final_signal = signal || new AbortController().signal; const final_signal = signal || new AbortController().signal;
const { result } = await withCandidates({ const { result } = await withCandidates<{ text: string; usage?: HarnessUsage }>({
harness: "transcription", harness: "transcription",
candidates, candidates,
on_success: async ({ candidate, result: transcript }): Promise<void> =>
await recordHarnessUsage({
call: buildUsageCall({
harness: "transcription",
candidate,
usage: transcript.usage,
}),
}),
run: async (candidate) => { run: async (candidate) => {
const adapter = grabHarnessAdapter({ const adapter = grabHarnessAdapter({
kind: candidate.provider.kind, kind: candidate.provider.kind,
}); });
const { text } = await adapter.transcribe({ const { text, usage } = await adapter.transcribe({
candidate, candidate,
audio, audio,
language, language,
@@ -69,9 +79,9 @@ export default async function transcribeAudio({
}); });
} }
return text.trim(); return { text: text.trim(), usage };
}, },
}); });
return result; return result.text;
} }
src/harness/embeddings/embed-text.ts
+12 -1
View File
@@ -1,6 +1,9 @@
import grabHarnessAdapter from "../providers/grab-harness-adapter"; import grabHarnessAdapter from "../providers/grab-harness-adapter";
import resolveHarnessCandidates from "../resolve/resolve-harness-candidates"; import resolveHarnessCandidates from "../resolve/resolve-harness-candidates";
import withCandidates from "../resolve/with-candidates"; import withCandidates from "../resolve/with-candidates";
import type { HarnessEmbedResult } from "../types";
import buildUsageCall from "../usage/build-usage-call";
import recordHarnessUsage from "../usage/record-harness-usage";
import fitVectorDimensions from "./fit-vector-dimensions"; import fitVectorDimensions from "./fit-vector-dimensions";
type Params = { type Params = {
@@ -52,9 +55,17 @@ export default async function embedText({
workspace_id, workspace_id,
}); });
const { result } = await withCandidates({ const { result } = await withCandidates<HarnessEmbedResult>({
harness: "embedding", harness: "embedding",
candidates, candidates,
on_success: async ({ candidate, result: embedded }): Promise<void> =>
await recordHarnessUsage({
call: buildUsageCall({
harness: "embedding",
candidate,
usage: embedded.usage,
}),
}),
run: async (candidate) => run: async (candidate) =>
await grabHarnessAdapter({ kind: candidate.provider.kind }).embed({ await grabHarnessAdapter({ kind: candidate.provider.kind }).embed({
candidate, candidate,
src/harness/image/generate-image.ts
+12 -1
View File
@@ -1,6 +1,9 @@
import grabHarnessAdapter from "../providers/grab-harness-adapter"; import grabHarnessAdapter from "../providers/grab-harness-adapter";
import resolveHarnessCandidates from "../resolve/resolve-harness-candidates"; import resolveHarnessCandidates from "../resolve/resolve-harness-candidates";
import withCandidates from "../resolve/with-candidates"; import withCandidates from "../resolve/with-candidates";
import type { HarnessImageOutput, HarnessUsage } from "../types";
import buildUsageCall from "../usage/build-usage-call";
import recordHarnessUsage from "../usage/record-harness-usage";
type Params = { type Params = {
prompt: string; prompt: string;
@@ -35,9 +38,17 @@ export default async function generateImage({
}: Params) { }: Params) {
const candidates = await resolveHarnessCandidates({ harness: "image", bot_id, workspace_id }); const candidates = await resolveHarnessCandidates({ harness: "image", bot_id, workspace_id });
const { result } = await withCandidates({ const { result } = await withCandidates<{ images: HarnessImageOutput[]; usage?: HarnessUsage }>({
harness: "image", harness: "image",
candidates, candidates,
on_success: async ({ candidate, result: made }): Promise<void> =>
await recordHarnessUsage({
call: buildUsageCall({
harness: "image",
candidate,
usage: made.usage,
}),
}),
run: async (candidate) => run: async (candidate) =>
await grabHarnessAdapter({ await grabHarnessAdapter({
kind: candidate.provider.kind, kind: candidate.provider.kind,
src/harness/media/read-image.ts
+14 -3
View File
@@ -3,7 +3,10 @@ import HarnessError from "../harness-error";
import grabHarnessAdapter from "../providers/grab-harness-adapter"; import grabHarnessAdapter from "../providers/grab-harness-adapter";
import resolveHarnessCandidates from "../resolve/resolve-harness-candidates"; import resolveHarnessCandidates from "../resolve/resolve-harness-candidates";
import withCandidates from "../resolve/with-candidates"; import withCandidates from "../resolve/with-candidates";
import type { HarnessUsage } from "../types";
import grabChatParams from "../text/grab-chat-params"; import grabChatParams from "../text/grab-chat-params";
import buildUsageCall from "../usage/build-usage-call";
import recordHarnessUsage from "../usage/record-harness-usage";
type Params = { type Params = {
/** /**
@@ -42,9 +45,17 @@ export default async function readImage({ url, focus, signal, bot_id, workspace_
workspace_id, workspace_id,
}); });
const { result } = await withCandidates({ const { result } = await withCandidates<{ text: string; usage?: HarnessUsage }>({
harness: "vision", harness: "vision",
candidates, candidates,
on_success: async ({ candidate, result: answer }): Promise<void> =>
await recordHarnessUsage({
call: buildUsageCall({
harness: "vision",
candidate,
usage: answer.usage,
}),
}),
run: async (candidate) => { run: async (candidate) => {
const answer = await grabHarnessAdapter({ const answer = await grabHarnessAdapter({
kind: candidate.provider.kind, kind: candidate.provider.kind,
@@ -83,9 +94,9 @@ export default async function readImage({ url, focus, signal, bot_id, workspace_
}); });
} }
return answer.text.trim(); return { text: answer.text.trim(), usage: answer.usage };
}, },
}); });
return result; return result.text;
} }
src/harness/providers/openai-compatible/embed.ts
+13 -1
View File
@@ -13,6 +13,11 @@ const embed: HarnessAdapter["embed"] = async ({ candidate, input, signal }) => {
const { provider, model } = candidate; const { provider, model } = candidate;
const preset = grabProviderPreset({ kind: provider.kind }); const preset = grabProviderPreset({ kind: provider.kind });
const vectors: number[][] = []; const vectors: number[][] = [];
/**
* What the provider said the batch cost. Most report it once per
* request; the rows are opened before the loop so each batch counts.
*/
let prompt_tokens = 0;
/** /**
* Every OpenAI-compatible API puts embeddings at `/embeddings`. * Every OpenAI-compatible API puts embeddings at `/embeddings`.
@@ -47,6 +52,8 @@ const embed: HarnessAdapter["embed"] = async ({ candidate, input, signal }) => {
}); });
} }
prompt_tokens += Number(json?.usage?.prompt_tokens) || 0;
for (const item of items) { for (const item of items) {
const vector = Array.isArray(item) ? item : item?.embedding; const vector = Array.isArray(item) ? item : item?.embedding;
@@ -62,7 +69,12 @@ const embed: HarnessAdapter["embed"] = async ({ candidate, input, signal }) => {
} }
} }
return { vectors }; return {
vectors,
...(prompt_tokens
? { usage: { prompt_tokens, completion_tokens: 0 } }
: {}),
};
}; };
export default embed; export default embed;
src/harness/providers/openai-compatible/generate-image.ts
+14 -1
View File
@@ -81,7 +81,20 @@ const generateImage: HarnessAdapter["generateImage"] = async ({
}); });
} }
return { images }; /**
* Image models bill per image, not per token: only a provider that
* volunteers a usage figure is recorded with one.
*/
const usage =
json?.usage && typeof json.usage == "object"
? {
prompt_tokens: Number(json.usage.prompt_tokens) || 0,
completion_tokens: Number(json.usage.completion_tokens) || 0,
...(typeof json.usage.cost == "number" ? { cost: json.usage.cost } : {}),
}
: undefined;
return { images, ...(usage ? { usage } : {}) };
}; };
export default generateImage; export default generateImage;
src/harness/providers/openai-compatible/transcribe.ts
+16 -1
View File
@@ -50,7 +50,22 @@ const transcribe: HarnessAdapter["transcribe"] = async ({
}); });
} }
return { text: String(json?.text || "").trim() }; /**
* Providers that bill by audio seconds sometimes report a token count;
* the OpenAI shape puts it under `usage`. Left out when absent, so the
* row records that the call happened without a token count.
*/
const usage =
json?.usage && typeof json.usage == "object"
? {
prompt_tokens: Number(json.usage.prompt_tokens ?? json.usage.input_tokens) || 0,
completion_tokens:
Number(json.usage.completion_tokens ?? json.usage.output_tokens) || 0,
...(typeof json.usage.cost == "number" ? { cost: json.usage.cost } : {}),
}
: undefined;
return { text: String(json?.text || "").trim(), ...(usage ? { usage } : {}) };
}; };
export default transcribe; export default transcribe;
src/harness/resolve/with-candidates.ts
+15 -1
View File
@@ -14,6 +14,15 @@ type Params<T> = {
* uses, so it doesn't switch models mid-task without a reason. * uses, so it doesn't switch models mid-task without a reason.
*/ */
prefer_key?: string; prefer_key?: string;
/**
* Called with what the candidate that worked returned, before the call
* comes back to the caller: where a harness writes its usage down.
* Never fails the call (the caller passes a recorder that can't throw).
*/
on_success?: (params: {
candidate: HarnessCandidate;
result: T;
}) => Promise<void> | void;
cooldowns?: HarnessCooldowns; cooldowns?: HarnessCooldowns;
now?: () => number; now?: () => number;
}; };
@@ -42,6 +51,7 @@ export default async function withCandidates<T>({
candidates, candidates,
run, run,
prefer_key, prefer_key,
on_success,
cooldowns = DefaultCooldowns, cooldowns = DefaultCooldowns,
now = Date.now, now = Date.now,
}: Params<T>): Promise<{ result: T; candidate: HarnessCandidate }> { }: Params<T>): Promise<{ result: T; candidate: HarnessCandidate }> {
@@ -93,7 +103,11 @@ export default async function withCandidates<T>({
if (!isReady(candidate)) continue; if (!isReady(candidate)) continue;
try { try {
return { result: await run(candidate), candidate }; const result = await run(candidate);
await on_success?.({ candidate, result });
return { result, candidate };
} catch (error: unknown) { } catch (error: unknown) {
if (!(error instanceof HarnessError)) throw error; if (!(error instanceof HarnessError)) throw error;
if (error.code == "aborted") throw error; if (error.code == "aborted") throw error;
src/harness/text/generate-text.ts
+19 -1
View File
@@ -8,6 +8,8 @@ import type {
HarnessMessage, HarnessMessage,
HarnessToolDefinition, HarnessToolDefinition,
} from "../types"; } from "../types";
import buildUsageCall from "../usage/build-usage-call";
import recordHarnessUsage from "../usage/record-harness-usage";
import grabChatParams from "./grab-chat-params"; import grabChatParams from "./grab-chat-params";
import prepareMessagesForModel from "./prepare-messages-for-model"; import prepareMessagesForModel from "./prepare-messages-for-model";
import stripThinkTags from "./strip-think-tags"; import stripThinkTags from "./strip-think-tags";
@@ -47,6 +49,14 @@ export type GenerateTextResult = HarnessChatResult & {
candidate: HarnessCandidate; candidate: HarnessCandidate;
}; };
/**
* One turn as `withCandidates` carries it: the answer the provider gave,
* with the reasoning tags stripped and an empty answer turned into null.
*/
type ChatAnswer = Omit<HarnessChatResult, "text"> & {
text: string | null;
};
/** /**
* One model turn on the text harness, falling back across its models. * One model turn on the text harness, falling back across its models.
* Media the model can't take is read by the helper harnesses first. * Media the model can't take is read by the helper harnesses first.
@@ -71,10 +81,18 @@ export default async function generateText({
? undefined ? undefined
: await grabChatParams({ harness: "text", bot_id, workspace_id }); : await grabChatParams({ harness: "text", bot_id, workspace_id });
const { result, candidate } = await withCandidates({ const { result, candidate } = await withCandidates<ChatAnswer>({
harness: "text", harness: "text",
candidates: resolved, candidates: resolved,
prefer_key, prefer_key,
on_success: async ({ candidate: answered, result: answer }): Promise<void> =>
await recordHarnessUsage({
call: buildUsageCall({
harness: "text",
candidate: answered,
usage: answer.usage,
}),
}),
run: async (candidate) => { run: async (candidate) => {
const answer = await grabHarnessAdapter({ const answer = await grabHarnessAdapter({
kind: candidate.provider.kind, kind: candidate.provider.kind,
src/harness/tools/bot/delegate-task-tool.test.ts
+72
View File
@@ -0,0 +1,72 @@
import { describe, expect, test } from "bun:test";
import type { BUN_SQLITE_COBBLE_BOTS, BUN_SQLITE_COBBLE_WORKSPACES } from "@/db/types/db";
import delegateTaskTool from "./delegate-task-tool";
const worker = { id: 5, name: "Researcher", role: "research", kind: "worker" } as BUN_SQLITE_COBBLE_BOTS;
const workspace = { id: 3, name: "Launch", archived: 0 } as BUN_SQLITE_COBBLE_WORKSPACES;
const signal = new AbortController().signal;
function coordinatorTool({
workspace_id,
workers = [worker],
workspaces = [workspace],
}: { workspace_id?: number; workers?: BUN_SQLITE_COBBLE_BOTS[]; workspaces?: BUN_SQLITE_COBBLE_WORKSPACES[] } = {}) {
return delegateTaskTool({
user_id: 1,
coordinator_id: 1,
session_id: 10,
workspace_id: workspace_id ?? null,
coordinator_name: "Coordinator",
workers,
workspaces,
});
}
/**
* The tool's own confines, kept to what needs no database: the argument it
* accepts and the refusal that keeps a workspace hand-over in its
* workspace. Every run that gets past those reaches the DB.
*/
describe("delegateTaskTool", () => {
test("takes an optional workspace the task runs in", () => {
const tool = coordinatorTool();
expect(tool.name).toBe("delegate_task");
expect(tool.schema?.safeParse({ bot_id: 5, title: "Dig", instructions: "Read the docs." }).success).toBe(true);
expect(
tool.schema?.safeParse({ bot_id: 5, title: "Dig", instructions: "Read the docs.", workspace_id: 3 }).success,
).toBe(true);
});
test("names the workspaces it can hand work into", () => {
expect(coordinatorTool().description).toContain("3: Launch");
});
test("in a workspace, says the task runs there and offers no other", () => {
const tool = coordinatorTool({ workspace_id: 3 });
expect(tool.description).toContain("runs there");
expect(tool.schema?.safeParse({ bot_id: 5, title: "Dig", instructions: "x", workspace_id: 9 }).success).toBe(
true,
);
});
test("refuses to move a hand-over out of the workspace it came from", async () => {
await expect(
coordinatorTool({ workspace_id: 3 }).run({
args: { bot_id: 5, title: "Dig", instructions: "x", workspace_id: 9 },
signal,
}),
).rejects.toThrow(/runs in that workspace/);
});
test("accepts the workspace it came from", async () => {
/** Reaches the DB and fails there for a test id, never on the guard. */
await expect(
coordinatorTool({ workspace_id: 3 }).run({
args: { bot_id: 5, title: "Dig", instructions: "x", workspace_id: 3 },
signal,
}),
).rejects.not.toThrow(/runs in that workspace/);
});
});
src/harness/tools/bot/delegate-task-tool.ts
+123 -14
View File
@@ -1,8 +1,12 @@
import { z } from "zod"; import { z } from "zod";
import type { BUN_SQLITE_COBBLE_BOTS } from "@/db/types/db"; import type { BUN_SQLITE_COBBLE_BOTS, BUN_SQLITE_COBBLE_WORKSPACES } from "@/db/types/db";
import pushBotCreated from "@/src/bots-queue/push-bot-created";
import pushBotNotice from "@/src/bots-queue/push-bot-notice"; import pushBotNotice from "@/src/bots-queue/push-bot-notice";
import enqueueBotJob from "@/src/functions/backend/db/bots-queue/enqueue-bot-job"; import enqueueBotJob from "@/src/functions/backend/db/bots-queue/enqueue-bot-job";
import grabOpenBotTask from "@/src/functions/backend/db/bots-queue/grab-open-bot-task"; import grabOpenBotTask from "@/src/functions/backend/db/bots-queue/grab-open-bot-task";
import addBotToWorkspace from "@/src/functions/backend/db/workspaces/add-bot-to-workspace";
import grabUserWorkspaces from "@/src/functions/backend/db/workspaces/grab-user-workspaces";
import grabWorkspaceBots from "@/src/functions/backend/db/workspaces/grab-workspace-bots";
import insertBotMessage from "@/src/functions/backend/db/bots/insert-bot-message"; import insertBotMessage from "@/src/functions/backend/db/bots/insert-bot-message";
import grabCrewWorkers from "@/src/functions/backend/db/bots/grab-crew-workers"; import grabCrewWorkers from "@/src/functions/backend/db/bots/grab-crew-workers";
import grabBotComputers from "@/src/functions/backend/db/computers/grab-bot-computers"; import grabBotComputers from "@/src/functions/backend/db/computers/grab-bot-computers";
@@ -32,19 +36,32 @@ type Params = {
* The worker bots it can hand work to (in a workspace, its members). * The worker bots it can hand work to (in a workspace, its members).
*/ */
workers: BUN_SQLITE_COBBLE_BOTS[]; workers: BUN_SQLITE_COBBLE_BOTS[];
/**
* The user's live workspaces, for the picker. In a workspace there's
* nothing to pick: the task runs there.
*/
workspaces?: BUN_SQLITE_COBBLE_WORKSPACES[];
}; };
type Args = { type Args = {
bot_id: number; bot_id: number;
title: string; title: string;
instructions: string; instructions: string;
/**
* The workspace the worker runs the task in, from the coordinator's own
* chat. Left out, the worker runs it in its own chat, on its own
* memory and its direct computers only.
*/
workspace_id?: number;
}; };
/** /**
* Lets the coordinator hand a task to a worker bot. The worker runs it on * Lets the coordinator hand a task to a worker bot: in its own chat, or in
* the computer it's on (if it has one) and posts the result in its own * one of the user's workspaces, which decides the work's context entirely.
* conversation, where a note says who asked; the result (or failure) is * The worker runs it on the computer it's on there (if it has one) and
* also reported back in the coordinator's conversation. * posts the result in that conversation, where a note says who asked; the
* result (or failure) is also reported back in the coordinator's
* conversation.
*/ */
export default function delegateTaskTool({ export default function delegateTaskTool({
user_id, user_id,
@@ -53,33 +70,79 @@ export default function delegateTaskTool({
workspace_id, workspace_id,
coordinator_name, coordinator_name,
workers, workers,
workspaces = [],
}: Params): HarnessTool<Args> { }: Params): HarnessTool<Args> {
const crew = workers const crew = workers
.map((bot) => `${bot.id}: ${bot.name}${bot.role ? ` (${bot.role})` : ""}`) .map((bot) => `${bot.id}: ${bot.name}${bot.role ? ` (${bot.role})` : ""}`)
.join("; "); .join("; ");
const places = workspaces
.map((w) => `${w.id}: ${w.name}`)
.join("; ");
return { return {
name: "delegate_task", name: "delegate_task",
description: `Hand a task to a crew member. They run it in the background and post the result in their own conversation; it's also reported back here when they finish or fail. Write the instructions as a full brief: what to do, where, and how they'll know it's done. Crew: ${crew}.`, description: `Hand a task to a crew member. They run it in the background and post the result in their own conversation; it's also reported back here when they finish or fail. Write the instructions as a full brief: what to do, where, and how they'll know it's done. Crew: ${crew}.${
workspace_id
? ` You are working in a workspace, so the task runs there: in this workspace's context.`
: ` Leave workspace_id out to run it in the crew member's own chat: only its own memory, files and directly assigned computers apply there, and the user finds the result by talking to it on its own. Add workspace_id to run it in one of the user's workspaces instead: the workspace's shared context, the member's instructions and memory there, its files and its computers (the workspace's own and the member's) all apply, and the result lands in its conversation in that workspace, beside the project it belongs to. Pick a workspace when the work belongs to that project; the member is added to it when it isn't in it yet. Workspaces: ${places || "none"}.`
}`,
schema: z.object({ schema: z.object({
bot_id: z.number().int(), bot_id: z.number().int(),
title: z.string().min(1).max(80), title: z.string().min(1).max(80),
instructions: z.string().min(1).max(8000), instructions: z.string().min(1).max(8000),
workspace_id: z
.number()
.int()
.optional()
.describe(
workspace_id
? "Not available here: a task handed over in a workspace runs in that workspace."
: "Run the task in this workspace: its shared context, memory, files and computers apply. Leave it out to run it in the crew member's own chat.",
),
}), }),
run: async ({ args }) => { run: async ({ args }) => {
const fresh = await grabCrewWorkers({ workspace_id }); /**
* In a workspace the task can't leave it: the run is already
* there, and that is the whole point of the hand-over.
*/
if (workspace_id && args.workspace_id && args.workspace_id != Number(workspace_id)) {
throw new Error(
`A task handed over from a workspace runs in that workspace. Leave workspace_id out, or hand it over from your own chat.`,
);
}
const target_workspace_id = workspace_id
? Number(workspace_id)
: args.workspace_id
? Number(args.workspace_id)
: null;
const fresh = await grabCrewWorkers({ workspace_id: target_workspace_id });
const worker = fresh.find((bot) => bot.id == args.bot_id); const worker = fresh.find((bot) => bot.id == args.bot_id);
if (!worker?.id) { if (!worker?.id) {
throw new Error( throw new Error(
`No crew member has id ${args.bot_id}. Crew: ${fresh `No crew member has id ${args.bot_id}${
.map((bot) => `${bot.id}: ${bot.name}`) target_workspace_id ? ` in workspace ${target_workspace_id}` : ""
.join("; ") || "none"}.`, }. Crew: ${
fresh.map((bot) => `${bot.id}: ${bot.name}`).join("; ") || "none"
}.`,
); );
} }
const computers = await grabBotComputers({ bot_id: worker.id, workspace_id }); const joined = target_workspace_id
const worker_session = await ensureBotSession({ user_id, bot_id: worker.id, workspace_id }); ? await ensureWorkerInWorkspace({ user_id, worker, target_workspace_id })
: "";
const computers = await grabBotComputers({
bot_id: worker.id,
workspace_id: target_workspace_id,
});
const worker_session = await ensureBotSession({
user_id,
bot_id: worker.id,
workspace_id: target_workspace_id,
});
const open = await grabOpenBotTask({ const open = await grabOpenBotTask({
bot_id: worker.id, bot_id: worker.id,
session_id: Number(worker_session.id), session_id: Number(worker_session.id),
@@ -108,12 +171,58 @@ export default function delegateTaskTool({
message: await insertBotMessage({ message: await insertBotMessage({
bot_id: worker.id, bot_id: worker.id,
session_id: Number(worker_session.id), session_id: Number(worker_session.id),
workspace_id: workspace_id ? Number(workspace_id) : null, workspace_id: target_workspace_id,
text: `${coordinator_name} asked me to: **${args.title}**`, text: `${coordinator_name} asked me to: **${args.title}**`,
}), }),
}); });
return `Task #${job_id} is with ${worker.name}. Their result will be in their conversation and reported back here.`; return `Task #${job_id} is with ${worker.name}${
target_workspace_id ? ` in workspace ${target_workspace_id}` : " in their own chat"
}.${joined} Their result will be in that conversation and reported back here.`;
}, },
}; };
} }
type EnsureParams = {
user_id: number | string;
worker: BUN_SQLITE_COBBLE_BOTS;
target_workspace_id: number;
};
/**
* A worker can only run a task in a workspace it's in, and a workspace's
* members were already listed by `grabCrewWorkers`. Anyone else is added
* first, since delegation is exactly the moment the broker decides who
* does the work where.
*/
async function ensureWorkerInWorkspace({
user_id,
worker,
target_workspace_id,
}: EnsureParams) {
if (!worker.id) return "";
const workspaces = (await grabUserWorkspaces({ user_id })).filter((w) => !w.archived);
const workspace = workspaces.find((w) => w.id == target_workspace_id);
if (!workspace) {
throw new Error(
`No workspace has id ${target_workspace_id}. Workspaces: ${
workspaces.map((w) => `${w.id}: ${w.name}`).join("; ") || "none"
}. Hand the task over without workspace_id to run it in the crew member's own chat.`,
);
}
const members = await grabWorkspaceBots({ user_id, workspace_id: target_workspace_id });
if (members.some((bot) => bot.id == worker.id)) return "";
await addBotToWorkspace({
user_id,
workspace_id: target_workspace_id,
bot_id: Number(worker.id),
});
await pushBotCreated({ bot: worker });
return ` ${worker.name} wasn't in "${workspace.name}", so it was added to it first.`;
}
src/harness/tools/bot/grab-bot-chat-tools.ts
+4 -1
View File
@@ -61,7 +61,9 @@ type Params = {
* for the coordinator (the super admin's alone) create_workspace and * for the coordinator (the super admin's alone) create_workspace and
* update_workspace (a workspace and its shared context), create_bot to * update_workspace (a workspace and its shared context), create_bot to
* grow the crew, delegate_task to every worker (the workspace's in a * grow the crew, delegate_task to every worker (the workspace's in a
* workspace) plus, in its own chat, find_bot and find_workspace, * workspace, where the task runs; from its own chat, in the worker's own
* chat or in a workspace it names) plus, in its own chat, find_bot and
* find_workspace,
* add_bot_to_workspace and remove_bot_from_workspace, assign_computer, * add_bot_to_workspace and remove_bot_from_workspace, assign_computer,
* update_bot, delete_bot and delete_workspace (which workspace, which * update_bot, delete_bot and delete_workspace (which workspace, which
* computers, and which name, role and core instructions each crew member * computers, and which name, role and core instructions each crew member
@@ -150,6 +152,7 @@ export default async function grabBotChatTools({
workspace_id, workspace_id,
coordinator_name: bot.name || "The coordinator", coordinator_name: bot.name || "The coordinator",
workers, workers,
workspaces,
}), }),
); );
src/harness/types.ts
+25 -2
View File
@@ -138,6 +138,18 @@ export type HarnessUsage = {
cost?: number; cost?: number;
}; };
/**
* What a harness does with a model, for its usage rows and the totals
* that read them. Mirrors the `harness` column of `token_usages`.
*/
export type HarnessUsageKind =
| "text"
| "vision"
| "transcription"
| "speech"
| "image"
| "embedding";
export type HarnessChatParams = { export type HarnessChatParams = {
candidate: HarnessCandidate; candidate: HarnessCandidate;
messages: HarnessMessage[]; messages: HarnessMessage[];
@@ -238,11 +250,22 @@ export type HarnessEmbedResult = {
*/ */
export type HarnessAdapter = { export type HarnessAdapter = {
chat: (params: HarnessChatParams) => Promise<HarnessChatResult>; chat: (params: HarnessChatParams) => Promise<HarnessChatResult>;
transcribe: (params: HarnessTranscribeParams) => Promise<{ text: string }>; /**
* `usage` when the provider says how many tokens it heard, which
* OpenAI-style speech-to-text endpoints do only sometimes.
*/
transcribe: (params: HarnessTranscribeParams) => Promise<{
text: string;
usage?: HarnessUsage;
}>;
speak: (params: HarnessSpeakParams) => Promise<HarnessAudioStream>; speak: (params: HarnessSpeakParams) => Promise<HarnessAudioStream>;
/**
* Image models report what they made rather than how many tokens it
* took, so `usage` is only whatever the provider volunteers.
*/
generateImage: ( generateImage: (
params: HarnessImageParams, params: HarnessImageParams,
) => Promise<{ images: HarnessImageOutput[] }>; ) => Promise<{ images: HarnessImageOutput[]; usage?: HarnessUsage }>;
embed: (params: HarnessEmbedParams) => Promise<HarnessEmbedResult>; embed: (params: HarnessEmbedParams) => Promise<HarnessEmbedResult>;
listModels: (params: { listModels: (params: {
provider: HarnessProviderConfig; provider: HarnessProviderConfig;
src/harness/usage/build-usage-call.test.ts
+102
View File
@@ -0,0 +1,102 @@
import { describe, expect, test } from "bun:test";
import type { HarnessCandidate } from "../types";
import buildUsageCall from "./build-usage-call";
function model(overrides: { row_id?: number } = {}) {
return {
slug: "deepseek/deepseek-chat",
name: "DeepSeek Chat",
input_modalities: ["text"],
output_modalities: ["text"],
supports_tools: true,
...overrides,
} as HarnessCandidate["model"];
}
function candidate(overrides: Partial<HarnessCandidate> = {}): HarnessCandidate {
return {
provider: {
id: 2,
name: "OpenRouter",
kind: "openrouter",
} as HarnessCandidate["provider"],
key: "openrouter:deepseek/deepseek-chat",
source: "auto",
model: model({ row_id: 41 }),
...overrides,
};
}
describe("buildUsageCall", () => {
test("names the candidate that really answered and what it spent", () => {
const call = buildUsageCall({
harness: "text",
candidate: candidate(),
usage: { prompt_tokens: 1200, completion_tokens: 340, cost: 0.0042 },
});
expect(call).toEqual({
harness: "text",
provider_id: 2,
provider_name: "OpenRouter",
model_id: 41,
model_slug: "deepseek/deepseek-chat",
prompt_tokens: 1200,
completion_tokens: 340,
cost: 0.0042,
});
});
test("a provider with no name reads as its kind", () => {
const call = buildUsageCall({
harness: "image",
candidate: candidate({
provider: { id: 4, name: "", kind: "openai" } as HarnessCandidate["provider"],
}),
usage: { prompt_tokens: 0, completion_tokens: 0 },
});
expect(call.provider_name).toBe("openai");
});
test("a provider that reports no usage still gets a row", () => {
const call = buildUsageCall({ harness: "transcription", candidate: candidate() });
expect(call.prompt_tokens).toBe(0);
expect(call.completion_tokens).toBe(0);
expect(call.cost).toBeUndefined();
});
test("a candidate with no catalog row has no model id to point at", () => {
const call = buildUsageCall({
harness: "text",
candidate: candidate({
model: model({ slug: "made-up-model" } as any),
}),
usage: { prompt_tokens: 10, completion_tokens: 5 },
});
expect(call.model_id).toBeUndefined();
expect(call.model_slug).toBe("made-up-model");
});
test("a cost the provider did not report is left out, not zeroed", () => {
const call = buildUsageCall({
harness: "text",
candidate: candidate(),
usage: { prompt_tokens: 10, completion_tokens: 5, cost: Number.NaN },
});
expect(call.cost).toBeUndefined();
});
test("a free provider's zero cost is kept as zero", () => {
const call = buildUsageCall({
harness: "text",
candidate: candidate(),
usage: { prompt_tokens: 10, completion_tokens: 5, cost: 0 },
});
expect(call.cost).toBe(0);
});
});
src/harness/usage/build-usage-call.ts
+37
View File
@@ -0,0 +1,37 @@
import type { HarnessCandidate, HarnessUsage } from "../types";
import type { HarnessUsageCall } from "./types";
type Params = {
harness: HarnessUsageCall["harness"];
candidate: HarnessCandidate;
usage?: HarnessUsage;
};
/**
* The candidate that really answered a call, and what it reported, as the
* row `recordHarnessUsage` writes. Providers that report no usage at all
* still get a row (no tokens), so a call is never invisible.
*/
export default function buildUsageCall({ harness, candidate, usage }: Params): HarnessUsageCall {
return {
harness,
provider_id: candidate.provider.id,
provider_name: candidate.provider.name || candidate.provider.kind,
model_id: modelIdOf(candidate),
model_slug: candidate.model.slug,
prompt_tokens: Number(usage?.prompt_tokens) || 0,
completion_tokens: Number(usage?.completion_tokens) || 0,
cost: typeof usage?.cost == "number" && Number.isFinite(usage.cost) ? usage.cost : undefined,
};
}
/**
* The `harness_models` row id, when the candidate carries the catalog row
* it was resolved from (`withCandidates` doesn't drop it; a candidate made
* by hand may not have one).
*/
function modelIdOf(candidate: HarnessCandidate) {
const row_id = (candidate.model as { row_id?: number }).row_id;
return typeof row_id == "number" ? row_id : undefined;
}
src/harness/usage/harness-usage-scope.test.ts
+106
View File
@@ -0,0 +1,106 @@
import { describe, expect, test } from "bun:test";
import { grabHarnessUsageScope, runWithHarnessUsageScope } from "./harness-usage-scope";
describe("harness usage scope", () => {
test("nothing is attributed outside a run", () => {
expect(grabHarnessUsageScope()).toBeUndefined();
});
test("a run's scope is visible to everything it calls", async () => {
const scope = await runWithHarnessUsageScope({
scope: { user_id: 7, bot_id: 3, session_id: 11 },
task: async () => grabHarnessUsageScope(),
});
expect(scope).toEqual({ user_id: 7, bot_id: 3, session_id: 11 });
});
test("a nested run keeps what the outer one knew", async () => {
const scope = await runWithHarnessUsageScope({
scope: { user_id: 7, bot_id: 3, session_id: 11, job_id: 5 },
task: () =>
runWithHarnessUsageScope({
scope: { message_id: 42 },
task: async () => grabHarnessUsageScope(),
}),
});
expect(scope).toEqual({
user_id: 7,
bot_id: 3,
session_id: 11,
job_id: 5,
message_id: 42,
});
});
test("a nested run overrides only what it sets", async () => {
const scope = await runWithHarnessUsageScope({
scope: { user_id: 7, bot_id: 3, workspace_id: 9 },
task: () =>
runWithHarnessUsageScope({
scope: { bot_id: 4 },
task: async () => grabHarnessUsageScope(),
}),
});
expect(scope).toEqual({ user_id: 7, bot_id: 4, workspace_id: 9 });
});
test("an undefined field leaves the outer value alone", async () => {
const scope = await runWithHarnessUsageScope({
scope: { user_id: 7, bot_id: 3, job_id: 5 },
task: () =>
runWithHarnessUsageScope({
scope: { job_id: undefined, message_id: 42 },
task: async () => grabHarnessUsageScope(),
}),
});
expect(scope).toEqual({ user_id: 7, bot_id: 3, job_id: 5, message_id: 42 });
});
test("a null field clears the outer value, since it means 'none'", async () => {
const scope = await runWithHarnessUsageScope({
scope: { user_id: 7, bot_id: 3, workspace_id: 9, job_id: 5 },
task: () =>
runWithHarnessUsageScope({
scope: { workspace_id: null, job_id: null },
task: async () => grabHarnessUsageScope(),
}),
});
expect(scope).toEqual({ user_id: 7, bot_id: 3, workspace_id: null, job_id: null });
});
test("the scope is gone once the run returns", async () => {
await runWithHarnessUsageScope({
scope: { user_id: 7 },
task: async () => undefined,
});
expect(grabHarnessUsageScope()).toBeUndefined();
});
test("two concurrent runs never see each other's scope", async () => {
const [first, second] = await Promise.all([
runWithHarnessUsageScope({
scope: { user_id: 1 },
task: async () => {
await Bun.sleep(5);
return grabHarnessUsageScope();
},
}),
runWithHarnessUsageScope({
scope: { user_id: 2 },
task: async () => {
await Bun.sleep(1);
return grabHarnessUsageScope();
},
}),
]);
expect(first).toEqual({ user_id: 1 });
expect(second).toEqual({ user_id: 2 });
});
});
src/harness/usage/harness-usage-scope.ts
+63
View File
@@ -0,0 +1,63 @@
import { AsyncLocalStorage } from "node:async_hooks";
import type { HarnessUsageScope } from "./types";
/**
* The attribution a run's model calls carry.
*
* A bot's run is entered once (`runWithHarnessUsageScope`) and everything
* it calls underneath - the text loop's turns, a session summary, a
* transcription, an image, an embedding - is written down for that user,
* bot, session and job without threading the ids through every harness
* function. One of those calls is itself entered by a `run` that has to
* swap the scope it reports under (`withHarnessUsageScope`): the session
* namer runs outside the job's scope, on purpose, since the job it named
* is already complete.
*
* The store is merged, not replaced: a call that only knows the message id
* (the text loop attaching a turn's usage) keeps the run's user, bot and
* job.
*/
const storage = new AsyncLocalStorage<HarnessUsageScope>();
/**
* Runs `task` with `scope` as its attribution. Ambient values are merged
* under it, so a nested call keeps what the outer one set and overrides
* only what it knows better (null clears a field).
*/
export async function runWithHarnessUsageScope<T>(
{ scope, task }: { scope: HarnessUsageScope; task: () => Promise<T> },
): Promise<T> {
return await storage.run(mergeScopes({ base: storage.getStore(), overlay: scope }), task);
}
/**
* The attribution a call right now carries, if any.
*/
export function grabHarnessUsageScope(): HarnessUsageScope | undefined {
return storage.getStore();
}
function mergeScopes({
base,
overlay,
}: {
base?: HarnessUsageScope;
overlay: HarnessUsageScope;
}): HarnessUsageScope {
const merged: HarnessUsageScope = { ...base };
for (const [key, value] of Object.entries(overlay) as [
keyof HarnessUsageScope,
HarnessUsageScope[keyof HarnessUsageScope],
][]) {
if (value === undefined) continue;
/**
* A null is a real value here: it says "none", e.g. a bot's own
* chat has no workspace, and must clear what the outer scope had.
*/
(merged as Record<string, unknown>)[key] = value;
}
return merged;
}
src/harness/usage/record-harness-usage.ts
+92
View File
@@ -0,0 +1,92 @@
import BunSQLite from "@moduletrace/bun-sqlite";
import type {
BUN_SQLITE_COBBLE_TOKEN_USAGES,
BunSQLiteTables,
} from "@/db/types/db";
import { logError } from "@/src/functions/backend/logger";
import { grabHarnessUsageScope } from "./harness-usage-scope";
import type { HarnessUsageCall, HarnessUsageScope } from "./types";
type Params = {
/**
* The scope the call ran under. Left out, the ambient one
* (`grabHarnessUsageScope`) is used, which is what every entry point
* does: the ids came in through `runWithHarnessUsageScope`.
*/
scope?: HarnessUsageScope;
/**
* Which harness and which model answered, with what the provider
* reported it cost.
*/
call: HarnessUsageCall;
};
/**
* Writes one model call into `token_usages`: the tokens it spent, on which
* model and provider, for whom, and which message or job it was for. This
* is what a message's token line, a bot's and a provider's totals and the
* usage page all read.
*
* Never throws: a model that answered must not have its answer lost
* because the bookkeeping failed. A call without an owner (no scope, no
* user) is not written at all, since nothing could show it.
*/
export default async function recordHarnessUsage({ scope, call }: Params) {
const resolved = scope || grabHarnessUsageScope();
const user_id = Number(resolved?.user_id) || null;
if (!user_id) return;
const row: BUN_SQLITE_COBBLE_TOKEN_USAGES = {
user_id,
bot_id: numberOrNull(resolved?.bot_id),
workspace_id: numberOrNull(resolved?.workspace_id),
session_id: numberOrNull(resolved?.session_id),
message_id: numberOrNull(resolved?.message_id),
job_id: numberOrNull(resolved?.job_id),
harness: call.harness,
provider_id: call.provider_id,
provider_name: call.provider_name,
model_id: call.model_id,
model_slug: call.model_slug,
prompt_tokens: Math.max(0, Math.round(call.prompt_tokens || 0)),
completion_tokens: Math.max(0, Math.round(call.completion_tokens || 0)),
cost: typeof call.cost == "number" && Number.isFinite(call.cost) ? call.cost : undefined,
};
try {
const res = await BunSQLite.insert<
BUN_SQLITE_COBBLE_TOKEN_USAGES,
(typeof BunSQLiteTables)[number]
>({
table: "token_usages",
data: [row],
});
if (!res.success) {
throw new Error(res.msg || `The usage wasn't saved.`);
}
} catch (error: unknown) {
await logError({
title: "Model usage not recorded",
error,
metadata: {
harness: call.harness,
model_slug: call.model_slug || null,
provider_name: call.provider_name || null,
user_id,
},
}).catch(() => {});
}
}
/**
* SQLite stores NULL for what isn't given; `Number(undefined)` is NaN,
* which bun-sqlite would refuse. An absent id is `undefined`, not `null`:
* bun-sqlite's row types spell "no value" that way.
*/
function numberOrNull(value: unknown): number | undefined {
const number = Number(value);
return Number.isFinite(number) && number > 0 ? number : undefined;
}
src/harness/usage/run-with-fallback-scope.test.ts
+81
View File
@@ -0,0 +1,81 @@
import { describe, expect, test } from "bun:test";
import { grabHarnessUsageScope, runWithHarnessUsageScope } from "./harness-usage-scope";
import runWithFallbackScope from "./run-with-fallback-scope";
const owner = { user_id: 9, bot_id: 3, session_id: null, message_id: null, job_id: null };
describe("runWithFallbackScope", () => {
test("the enclosing run keeps the work it already owns", async () => {
const seen = await runWithHarnessUsageScope({
scope: { user_id: 7, bot_id: 3, job_id: 5 },
task: () =>
runWithFallbackScope({
scope: owner,
task: async () => grabHarnessUsageScope(),
}),
});
expect(seen).toEqual({ user_id: 7, bot_id: 3, job_id: 5 });
});
test("work with no run is attributed to its owner", async () => {
const seen = await runWithFallbackScope({
scope: owner,
task: async () => grabHarnessUsageScope(),
});
expect(seen).toEqual(owner);
});
test("nobody to charge leaves the work unattributed rather than failing", async () => {
const seen = await runWithFallbackScope({
scope: null,
task: async () => grabHarnessUsageScope(),
});
expect(seen).toBeUndefined();
});
test("a scope with no user is not an owner", async () => {
const seen = await runWithFallbackScope({
scope: { ...owner, user_id: undefined },
task: async () => grabHarnessUsageScope(),
});
expect(seen).toBeUndefined();
});
test("work that must be charged on its own overrides the enclosing run", async () => {
const seen = await runWithHarnessUsageScope({
scope: { user_id: 7, bot_id: 3, job_id: 5 },
task: () =>
runWithFallbackScope({
scope: owner,
mode: "own",
task: async () => grabHarnessUsageScope(),
}),
});
expect(seen).toEqual(owner);
});
test("the work's result is passed through", async () => {
const result = await runWithFallbackScope({
scope: owner,
task: async () => "the answer",
});
expect(result).toBe("the answer");
});
test("a failure inside the work is not swallowed", async () => {
expect(
runWithFallbackScope({
scope: owner,
task: async () => {
throw new Error("the provider refused");
},
}),
).rejects.toThrow("the provider refused");
});
});
src/harness/usage/run-with-fallback-scope.ts
+41
View File
@@ -0,0 +1,41 @@
import { grabHarnessUsageScope, runWithHarnessUsageScope } from "./harness-usage-scope";
import type { HarnessUsageScope } from "./types";
type Params<T> = {
/**
* What to attribute the work to when nothing already owns it: a file
* ingested by the queue has no run behind it, but its owner must still
* see what the embedding cost. Null when there is nobody to charge.
*/
scope: HarnessUsageScope | null;
task: () => Promise<T>;
/**
* How the work is attributed: `enclosing` keeps whatever run it is in,
* `own` always reports under `scope` (the session namer, whose job is
* already complete and must not be charged to it).
*/
mode?: "enclosing" | "own";
};
/**
* Runs work under an attribution the run may not have. Nothing is entered
* when a run is already in charge, so an ingest a bot asked for is charged
* to that bot's user and not to the file's owner.
*/
export default async function runWithFallbackScope<T>({
scope,
task,
mode = "enclosing",
}: Params<T>): Promise<T> {
if (mode == "enclosing" && grabHarnessUsageScope()?.user_id) {
return await task();
}
const user_id = Number(scope?.user_id);
if (!Number.isFinite(user_id) || user_id <= 0 || user_id === null) {
return await task();
}
return await runWithHarnessUsageScope({ scope: { ...scope, user_id }, task });
}
src/harness/usage/types.ts
+60
View File
@@ -0,0 +1,60 @@
/**
* Who and what a model call was for. Every harness call that reports its
* usage is written down with this (`/api/auth/crud` no: `token_usages`),
* so a message, a session, a bot or a provider can be totaled later.
*
* It is deliberately shaped slightly differently from the table's
* columns: `user_id` is required here (a row without an owner can't be
* shown or totaled for anyone), the rest are optional.
*/
export type HarnessUsageScope = {
/**
* Whose work it was: the user whose conversation the call ran for.
*/
user_id?: number | string;
bot_id?: number | string | null;
workspace_id?: number | string | null;
session_id?: number | string | null;
/**
* The queued job it ran for, when it ran for one.
*/
job_id?: number | string | null;
/**
* The message it produced, for the ones that produce one.
*/
message_id?: number | string | null;
};
/**
* What the caller knows about the call beyond its scope: which harness
* asked, and which candidate answered. Both are filled in by the call's
* own entry point, never by the code that runs it.
*/
export type HarnessUsageCall = {
harness: "text" | "vision" | "transcription" | "speech" | "image" | "embedding";
provider_id?: number;
provider_name?: string;
model_id?: number;
model_slug?: string;
prompt_tokens: number;
completion_tokens: number;
cost?: number;
};
/**
* A message's usage, as its thread shows it and as `token_usages` stores
* it: what one answer cost, summed over its model calls.
*/
export type TokenUsageSummary = {
prompt_tokens: number;
completion_tokens: number;
total_tokens: number;
/**
* In USD, when the provider reported it.
*/
cost?: number;
/**
* How many model calls it took.
*/
calls: number;
};
src/harness/usage/with-usage-scope.ts
+19
View File
@@ -0,0 +1,19 @@
import { runWithHarnessUsageScope } from "./harness-usage-scope";
import type { HarnessUsageScope } from "./types";
/**
* Runs one call the app makes on a user's behalf inside the scope its
* usage is written down with (transcribing a dictation, speaking a
* message, previewing a voice). The bot's run is already inside a scope of
* its own, so this only adds what the run didn't know: no job, no message,
* and the session the call belongs to.
*/
export default async function withUsageScope<T>({
scope,
task,
}: {
scope: HarnessUsageScope;
task: () => Promise<T>;
}): Promise<T> {
return await runWithHarnessUsageScope({ scope, task });
}
src/layouts/bots/(dict)/rail-user-menu-dict.ts
+8 -3
View File
@@ -1,4 +1,5 @@
import { LogOut, SlidersHorizontal, type LucideIcon } from "lucide-react"; import { LogOut, SlidersHorizontal, type LucideIcon } from "lucide-react";
import type { BUN_SQLITE_COBBLE_USER_TYPES } from "@/db/types/db";
import { SettingsDefaultPath } from "@/src/dict/settings-pages-dict"; import { SettingsDefaultPath } from "@/src/dict/settings-pages-dict";
import { RailPages } from "@/src/dict/rail-pages-dict"; import { RailPages } from "@/src/dict/rail-pages-dict";
@@ -18,12 +19,16 @@ export type RailUserMenuItem = {
* Shown to super admins only. * Shown to super admins only.
*/ */
super_admin_only?: boolean; super_admin_only?: boolean;
/**
* Who sees it, when that's an RBAC rule rather than super admins only.
*/
can_view?: (params: { user_types?: Pick<BUN_SQLITE_COBBLE_USER_TYPES, "user_type">[] | null }) => boolean;
}; };
/** /**
* The popover items: the rail pages (routines, activity, help), settings, * The popover items: the rail pages (routines, files, usage, activity,
* sign out. Nothing else lives in the rail: these destinations are only * help), settings, sign out. Nothing else lives in the rail: these
* reachable from here. * destinations are only reachable from here.
*/ */
export const RailUserMenuItems: RailUserMenuItem[] = [ export const RailUserMenuItems: RailUserMenuItem[] = [
...RailPages, ...RailPages,
src/layouts/bots/(partials)/rail-pages-list.tsx
+10 -4
View File
@@ -2,6 +2,7 @@ import { useContext } from "react";
import Divider from "@moduletrace/twui/layout/Divider"; import Divider from "@moduletrace/twui/layout/Divider";
import { RailPages } from "@/src/dict/rail-pages-dict"; import { RailPages } from "@/src/dict/rail-pages-dict";
import { AppContext } from "@/src/pages/__root"; import { AppContext } from "@/src/pages/__root";
import canSeeRailItem from "@/src/utils/rbac/can-see-rail-item";
import isPassThroughUser from "@/src/utils/rbac/is-pass-through-user"; import isPassThroughUser from "@/src/utils/rbac/is-pass-through-user";
import RailBackLink from "./rail-back-link"; import RailBackLink from "./rail-back-link";
import RailSettingsLink from "./rail-settings-link"; import RailSettingsLink from "./rail-settings-link";
@@ -11,8 +12,9 @@ type Props = {
}; };
/** /**
* The rail on routines, activity and help: back to the bots, then those * The rail on routines, usage, activity and help: back to the bots, then
* pages (super admin ones for super admins only). * those pages (super admin ones for super admins only, crew managers for
* usage).
*/ */
export default function RailPagesList({ pathname }: Props) { export default function RailPagesList({ pathname }: Props) {
const { pageProps } = useContext(AppContext); const { pageProps } = useContext(AppContext);
@@ -25,8 +27,12 @@ export default function RailPagesList({ pathname }: Props) {
<> <>
<RailBackLink /> <RailBackLink />
{/* <Divider className="shrink-0 mx-1 @rail:mx-3 my-2 w-auto border-hairline dark:border-hairline" /> */} {/* <Divider className="shrink-0 mx-1 @rail:mx-3 my-2 w-auto border-hairline dark:border-hairline" /> */}
{RailPages.filter( {RailPages.filter((page) =>
(page) => is_super_admin || !page.super_admin_only, canSeeRailItem({
item: page,
is_super_admin,
user_types: pageProps?.user_types,
}),
).map((page) => ( ).map((page) => (
<RailSettingsLink <RailSettingsLink
key={page.url} key={page.url}
src/layouts/bots/(partials)/rail-settings-list.tsx
+6 -3
View File
@@ -2,6 +2,7 @@ import { useContext } from "react";
import Divider from "@moduletrace/twui/layout/Divider"; import Divider from "@moduletrace/twui/layout/Divider";
import { SettingsPages } from "@/src/dict/settings-pages-dict"; import { SettingsPages } from "@/src/dict/settings-pages-dict";
import { AppContext } from "@/src/pages/__root"; import { AppContext } from "@/src/pages/__root";
import canSeeRailItem from "@/src/utils/rbac/can-see-rail-item";
import isPassThroughUser from "@/src/utils/rbac/is-pass-through-user"; import isPassThroughUser from "@/src/utils/rbac/is-pass-through-user";
import RailBackLink from "./rail-back-link"; import RailBackLink from "./rail-back-link";
import RailSettingsLink from "./rail-settings-link"; import RailSettingsLink from "./rail-settings-link";
@@ -27,9 +28,11 @@ export default function RailSettingsList({ pathname }: Props) {
<RailBackLink /> <RailBackLink />
{/* <Divider className="shrink-0 mx-1 @rail:mx-3 my-2 w-auto border-hairline dark:border-hairline" /> */} {/* <Divider className="shrink-0 mx-1 @rail:mx-3 my-2 w-auto border-hairline dark:border-hairline" /> */}
{SettingsPages.filter((page) => {SettingsPages.filter((page) =>
page.can_view canSeeRailItem({
? page.can_view({ user_types: pageProps?.user_types }) item: page,
: is_super_admin || !page.super_admin_only, is_super_admin,
user_types: pageProps?.user_types,
}),
).map((page) => ( ).map((page) => (
<RailSettingsLink <RailSettingsLink
key={page.url} key={page.url}
src/layouts/bots/(partials)/rail-user-menu.tsx
+9 -4
View File
@@ -9,18 +9,23 @@ import RailUserMenuEntry from "./rail-user-menu-entry";
import RailUserMenuAction from "./rail-user-menu-action"; import RailUserMenuAction from "./rail-user-menu-action";
import InstallAppModal from "./install-app-modal"; import InstallAppModal from "./install-app-modal";
import { AppContext } from "@/src/pages/__root"; import { AppContext } from "@/src/pages/__root";
import canSeeRailItem from "@/src/utils/rbac/can-see-rail-item";
import isPassThroughUser from "@/src/utils/rbac/is-pass-through-user"; import isPassThroughUser from "@/src/utils/rbac/is-pass-through-user";
import useInstallPrompt from "@/src/hooks/use-install-prompt"; import useInstallPrompt from "@/src/hooks/use-install-prompt";
/** /**
* 220px popover: the only element in the shell with elevation. Super * 220px popover: the only element in the shell with elevation. Items
* admin items show to super admins only. Install app shows while the * follow the same rule as the rail's lists: super admin ones show to
* browser can install it; Safari gets the Share menu steps instead. * super admins only, RBAC-gated ones by their own rule. Install app shows
* while the browser can install it; Safari gets the Share menu steps
* instead.
*/ */
export default function RailUserMenu() { export default function RailUserMenu() {
const { pageProps } = useContext(AppContext); const { pageProps } = useContext(AppContext);
const is_super_admin = isPassThroughUser({ user_types: pageProps?.user_types }); const is_super_admin = isPassThroughUser({ user_types: pageProps?.user_types });
const shown = RailUserMenuItems.filter((item) => is_super_admin || !item.super_admin_only); const shown = RailUserMenuItems.filter((item) =>
canSeeRailItem({ item, is_super_admin, user_types: pageProps?.user_types }),
);
const items = shown.filter((item) => !item.destructive); const items = shown.filter((item) => !item.destructive);
const destructive = shown.filter((item) => item.destructive); const destructive = shown.filter((item) => item.destructive);
src/pages/admin/bots/[bot_id]/(data)/stop-bot-context.ts
+21
View File
@@ -0,0 +1,21 @@
import React from "react";
import type { StopBot } from "../(hooks)/use-stop-bot";
export type StopBotState = StopBot & {
/**
* The bot has something open for the signed-in user right now (a reply
* it owes, a task it's on, a wait for a model or a computer), so
* stopping it has something to pull the plug on.
*/
can_stop: boolean;
};
/**
* The conversation's stop control, shared by the thinking row under the
* thread and the header's menu, so both pull the same plug.
*/
export const StopBotContext = React.createContext<StopBotState>({
stop: async () => {},
stopping: false,
can_stop: false,
});
src/pages/admin/bots/[bot_id]/(functions)/stop-bot.ts
+19
View File
@@ -0,0 +1,19 @@
import fetchApi from "@moduletrace/twui/utils/fetch/fetchApi";
import type { APIResponseObject } from "@moduletrace/bunext/types";
import type { BotConversationRequest } from "@/src/schemas/bot-schema";
import type { BotStoppedPayload } from "@/src/types";
/**
* Pulls the plug on the bot for the signed-in user: every job of it that's
* queued or running stops, wherever they chat with it. The messages that
* end the wait arrive over the WebSocket, so nothing here reloads.
*/
export default async function stopBot({ bot_id, workspace_id }: BotConversationRequest) {
return await fetchApi<
BotConversationRequest,
APIResponseObject<BotStoppedPayload>
>(`/api/admin/bots/stop`, {
method: "POST",
body: { bot_id, workspace_id },
});
}
src/pages/admin/bots/[bot_id]/(hooks)/use-bot-routines.ts
+54
View File
@@ -0,0 +1,54 @@
import fetchApi from "@moduletrace/twui/utils/fetch/fetchApi";
import type { APIResponseObject } from "@moduletrace/bunext/types";
import { useCallback, useEffect, useState } from "react";
import type { CronJobWithNames } from "@/src/types";
type Params = {
bot_id: number;
workspace_id?: number | null;
/**
* Load only once the modal opens, so the conversation's page props stay
* small.
*/
enabled?: boolean;
};
/**
* A conversation's routines: the schedules that run there, a bot's own in
* its chat or the ones that run in the workspace. Loaded on open and
* refreshed after every change, never part of the page props.
*/
export default function useBotRoutines({ bot_id, workspace_id, enabled = true }: Params) {
const [routines, setRoutines] = useState<CronJobWithNames[]>([]);
const [loading, setLoading] = useState(false);
const [error, setError] = useState<string>();
const load = useCallback(async () => {
setLoading(true);
setError(undefined);
const query = new URLSearchParams({ bot_id: String(bot_id) });
if (workspace_id) query.set("workspace_id", String(workspace_id));
const res = await fetchApi<{}, APIResponseObject<CronJobWithNames>>(
`/api/admin/bots/list-routines?${query.toString()}`,
{ method: "GET" },
).catch(() => null);
setLoading(false);
if (!res?.success) {
setError(res?.msg || `Couldn't load the routines.`);
return;
}
setRoutines(res.payload || []);
}, [bot_id, workspace_id]);
useEffect(() => {
if (enabled) load().catch(() => {});
}, [enabled, load]);
return { routines, setRoutines, loading, error, reload: load };
}
src/pages/admin/bots/[bot_id]/(hooks)/use-stop-bot.ts
+65
View File
@@ -0,0 +1,65 @@
import { useRef, useState } from "react";
import stopBot from "../(functions)/stop-bot";
export type StopBot = {
/**
* Stops every job of the bot for the signed-in user.
*/
stop: () => Promise<void>;
/**
* A stop is on its way to the server.
*/
stopping: boolean;
/**
* Why the last stop didn't work, for the button's title.
*/
error?: string;
};
type Params = {
bot_id: number;
workspace_id?: number | null;
};
/**
* The conversation's stop button: pulls the plug on the bot for the
* signed-in user, wherever they chat with it. It's quiet on the way out:
* the bot's closing message and idle face arrive as pushes, so nothing
* here reloads and a failed stop only surfaces as the button's title.
*/
export default function useStopBot({ bot_id, workspace_id }: Params): StopBot {
const [stopping, setStopping] = useState(false);
const [error, setError] = useState<string>();
/**
* A ref, not the state: two clicks in one tick both read the old
* `stopping` and would send two stops.
*/
const in_flight = useRef(false);
const stop = async () => {
if (in_flight.current) return;
in_flight.current = true;
setStopping(true);
setError(undefined);
try {
const res = await stopBot({
bot_id,
workspace_id: workspace_id || null,
}).catch(() => null);
/**
* A message without a failure is the count ("Stopped 2 jobs."),
* so only an unsuccessful answer is an error.
*/
if (!res?.success) setError(res?.msg || "Couldn't stop the bot.");
} finally {
in_flight.current = false;
setStopping(false);
}
};
return { stop, stopping, error };
}
src/pages/admin/bots/[bot_id]/(hooks)/use-thread-scroll.ts
+162
View File
@@ -0,0 +1,162 @@
import { useLayoutEffect, useRef, useState, type RefObject } from "react";
import type { BotMessageWithAttachments } from "@/src/types";
import {
ThreadJumpWindowMs,
scrollThreadToBottom,
shouldFollowThread,
threadOffset,
} from "../(utils)/thread-scroll";
type Params = {
/**
* The thread's messages, oldest first, pending ones included.
*/
messages: BotMessageWithAttachments[];
};
/**
* Keeps the thread's bottom in view: a chat sitting at the bottom follows
* every new message, a chat scrolled up stays where it is (except for a
* message the user just sent, which brings the thread to it) and offers a
* button to get back down. Whether it sits at the bottom is read from where
* the content ends, not from `scrollTop`, whose sign a reversed column
* reports differently across engines. Content that grows while the thread
* is at the bottom (images landing, a wider window) is followed too.
*/
export default function useThreadScroll({ messages }: Params) {
const column_ref = useRef<HTMLDivElement>(null);
const content_ref = useRef<HTMLDivElement>(null);
const [jump_visible, setJumpVisible] = useState(false);
/** Whether the bottom is being held, so nothing fights the user's scroll. */
const following = useRef(true);
/**
* A pressed jump is still animating, so the scrolling it causes is its
* own. It ends when the thread arrives, when a scroll moves away from
* the bottom (the user took over), or after the window, whichever first.
*/
const jump = useRef({ active: false, closest: 0, until: 0 });
const previous = useRef<{
last_id?: number | string | null;
count: number;
}>({ count: 0 });
const measure = () => {
const column = column_ref.current;
const content = content_ref.current;
if (!column || !content) return null;
return threadOffset({
content_bottom: content.getBoundingClientRect().bottom,
column_bottom: column.getBoundingClientRect().bottom,
});
};
/** The user's own scrolling: it decides whether the thread follows. */
const onScroll = () => {
const offset = measure();
if (!offset) return;
if (jump.current.active) {
const animating =
Date.now() < jump.current.until &&
offset.distance <= jump.current.closest;
jump.current.closest = offset.distance;
jump.current.active = animating;
if (animating) return;
}
following.current = offset.near_bottom;
setJumpVisible(!offset.near_bottom);
};
const jumpToBottom = () => {
const column = column_ref.current;
if (!column) return;
const reduced = Boolean(
globalThis.matchMedia?.("(prefers-reduced-motion: reduce)").matches,
);
jump.current = {
active: !reduced,
closest: Number.POSITIVE_INFINITY,
until: Date.now() + ThreadJumpWindowMs,
};
following.current = true;
setJumpVisible(false);
scrollThreadToBottom({
element: column,
behavior: reduced ? "auto" : "smooth",
});
};
useLayoutEffect(() => {
const column = column_ref.current;
const newest = messages[messages.length - 1];
const before = previous.current;
/**
* The list was replaced (a new session, the first render) when the
* message that was last is gone.
*/
const kept_last_message =
before.count > 0 &&
messages.some((message) => message.id == before.last_id);
previous.current = { last_id: newest?.id, count: messages.length };
if (!column) return;
const follow = shouldFollowThread({
near_bottom: following.current,
newest_author: newest?.author,
kept_last_message,
});
if (!follow) {
setJumpVisible(true);
return;
}
following.current = true;
jump.current.active = false;
setJumpVisible(false);
scrollThreadToBottom({ element: column });
}, [messages]);
useLayoutEffect(() => {
const column = column_ref.current;
const content = content_ref.current;
if (!column || !content) return;
/**
* A message that grows after it lands (an image, a widened window)
* keeps the bottom in view while the thread is there, and leaves a
* chat scrolled up alone.
*/
const observer = new ResizeObserver(() => {
if (!following.current) return;
scrollThreadToBottom({ element: column });
});
observer.observe(content);
return () => observer.disconnect();
}, []);
return {
column_ref,
content_ref: content_ref as RefObject<HTMLDivElement>,
jump_visible,
jump_to_bottom: jumpToBottom,
on_scroll: onScroll,
};
}
src/pages/admin/bots/[bot_id]/(partials)/bot-message-line.tsx
+2
View File
@@ -12,6 +12,7 @@ import GroupTimestamp from "./group-timestamp";
import MessageActions from "./message-actions"; import MessageActions from "./message-actions";
import MessageReactions from "./message-reactions"; import MessageReactions from "./message-reactions";
import MessageReplyQuote from "./message-reply-quote"; import MessageReplyQuote from "./message-reply-quote";
import MessageUsageChip from "./message-usage-chip";
import NeedsYouChip from "./needs-you-chip"; import NeedsYouChip from "./needs-you-chip";
import SpeakMessageButton from "./speak-message-button"; import SpeakMessageButton from "./speak-message-button";
@@ -74,6 +75,7 @@ function BotMessageLine({ message, bot_name, at }: Props) {
<Row className="gap-3! items-center"> <Row className="gap-3! items-center">
{message.needs_you ? <NeedsYouChip /> : null} {message.needs_you ? <NeedsYouChip /> : null}
{at ? <GroupTimestamp at={at} /> : null} {at ? <GroupTimestamp at={at} /> : null}
<MessageUsageChip usage={message.usage} />
{can_speak ? ( {can_speak ? (
<SpeakMessageButton message_id={Number(message.id)} /> <SpeakMessageButton message_id={Number(message.id)} />
) : null} ) : null}
src/pages/admin/bots/[bot_id]/(partials)/bot-routines-modal.tsx
+205
View File
@@ -0,0 +1,205 @@
import { useEffect, useState } from "react";
import Modal from "@moduletrace/twui/elements/Modal";
import H2 from "@moduletrace/twui/layout/H2";
import Row from "@moduletrace/twui/layout/Row";
import Span from "@moduletrace/twui/layout/Span";
import Stack from "@moduletrace/twui/layout/Stack";
import Button from "@moduletrace/twui/layout/Button";
import { Clock } from "lucide-react";
import type { BUN_SQLITE_COBBLE_WORKSPACES } from "@/db/types/db";
import type { BotWithLastMessage, CronJobWithNames } from "@/src/types";
import RoutineForm from "@/src/components/general/routine-form";
import RoutineRow from "@/src/components/general/routine-row";
import postAdminApi from "@/src/layouts/bots/(functions)/post-admin-api";
import SettingsList from "@/src/pages/admin/settings/(partials)/settings-list";
import SettingsStatus from "@/src/pages/admin/settings/(partials)/settings-status";
import { dropRoutine, replaceRoutine } from "@/src/utils/routine-list-changes";
import useBotRoutines from "../(hooks)/use-bot-routines";
import HeaderMenuItem from "./header-menu-item";
type Status = { error?: boolean; success?: boolean; msg?: string };
type Props = {
bot: BotWithLastMessage;
workspace?: BUN_SQLITE_COBBLE_WORKSPACES | null;
/**
* A row in the conversation header's menu instead of an icon beside it.
*/
menu?: boolean;
on_open?: () => void;
};
/**
* The routines that run in this conversation: the schedules whose result
* lands here, a bot's own on its chat or the ones it runs in this
* workspace. Open it from the header menu; the list loads then, so the
* page's props stay small. The user pauses, resumes, edits and deletes
* their own; the bot makes them when asked in chat.
*/
export default function BotRoutinesModal({ bot, workspace, menu, on_open }: Props) {
const [open, set_open] = useState(false);
const [editing, set_editing] = useState<CronJobWithNames>();
const [busy, set_busy] = useState(false);
const [status, set_status] = useState<Status>();
const { routines, setRoutines, loading, error } = useBotRoutines({
bot_id: Number(bot.id),
workspace_id: workspace?.id ? Number(workspace.id) : null,
enabled: open,
});
const name = bot.name || "This bot";
const place = workspace?.name || "this workspace";
const title = workspace?.id ? `${place} routines` : `${name}'s routines`;
useEffect(() => {
if (open) on_open?.();
}, [open]);
const replace = ({ cron_job }: { cron_job: CronJobWithNames }) =>
setRoutines((prev) => replaceRoutine({ routines: prev, cron_job }));
const toggle = async ({ cron_job }: { cron_job: CronJobWithNames }) => {
if (busy) return;
set_busy(true);
set_status(undefined);
const res = await postAdminApi<CronJobWithNames>({
path: "routines/update-routine",
body: { cron_job_id: cron_job.id, enabled: !cron_job.enabled },
});
set_busy(false);
if (!res.success || !res.singleRes) {
set_status({ error: true, msg: res.msg || "That didn't work." });
return;
}
replace({ cron_job: res.singleRes });
set_status({ success: true, msg: cron_job.enabled ? "Paused." : "Resumed." });
};
const remove = async ({ cron_job }: { cron_job: CronJobWithNames }) => {
if (busy) return;
if (!window.confirm(`Delete the routine "${cron_job.title}"? A run already started carries on.`)) return;
set_busy(true);
set_status(undefined);
const res = await postAdminApi({ path: "routines/delete-routine", body: { cron_job_id: cron_job.id } });
set_busy(false);
if (!res.success) {
set_status({ error: true, msg: res.msg || "That didn't work." });
return;
}
setRoutines((prev) => dropRoutine({ routines: prev, cron_job_id: cron_job.id }));
set_status({ success: true, msg: "Deleted." });
};
return (
<Modal
target={
menu ? (
<HeaderMenuItem icon={Clock} title="Schedules" description={title} loading={loading} />
) : (
<Button
title={title}
type="button"
variant="ghost"
color="text"
className="shrink-0 p-2! text-ink-secondary hover:text-ink hover:bg-sunken rounded-default"
>
<Clock size={18} aria-hidden />
</Button>
)
}
targetWrapperProps={{ className: menu ? "w-full" : "shrink-0" }}
open={open}
setOpen={(next) => {
set_open(next);
if (!next) {
set_editing(undefined);
set_status(undefined);
}
}}
className="max-w-xl bg-paper p-6"
>
<Stack className="w-full gap-1! items-stretch">
<Row className="w-full gap-2! flex-nowrap items-center">
<H2 className="grow text-lg! leading-6! tracking-normal! font-bold text-ink">
{editing ? editing.title || "Routine" : title}
</H2>
{editing ? (
<Button
title="Back to the list"
type="button"
variant="ghost"
color="gray"
size="small"
onClick={() => {
set_editing(undefined);
set_status(undefined);
}}
className="px-2! text-[12px]"
>
Back
</Button>
) : null}
</Row>
<Span className="text-[13px] leading-[18px] text-ink-secondary">
{editing
? `${name} gets this brief as a task at every run, and the result lands here.`
: workspace?.id
? `Work ${name} does on a schedule in ${place}. Each run is a task, and its result lands here.`
: `Work ${name} does on its own, on a schedule. Each run is a task, and its result lands here.`}
</Span>
</Stack>
<Stack className="mt-4 max-h-[55vh] w-full gap-0! items-stretch overflow-y-auto">
{editing ? (
<RoutineForm
key={editing.id}
cron_job={editing}
onCancel={() => set_editing(undefined)}
onSaved={({ cron_job }) => {
replace({ cron_job });
set_editing(undefined);
set_status({ success: true, msg: "Saved." });
}}
/>
) : (
<SettingsList
is_empty={!routines.length}
empty={loading ? "Loading routines..." : error || "No routines here yet."}
>
{routines.map((cron_job) => (
<RoutineRow
key={cron_job.id}
cron_job={cron_job}
compact
busy={busy}
onToggle={() => toggle({ cron_job }).catch(() => {})}
onEdit={() => set_editing(cron_job)}
onDelete={() => remove({ cron_job }).catch(() => {})}
/>
))}
</SettingsList>
)}
</Stack>
<Stack className="mt-3 gap-1! items-stretch">
<SettingsStatus status={status} />
{editing ? null : (
<Span className="text-[12px] leading-4 text-ink-secondary">
{`Ask ${name} in chat to set one up, or change one here.`}
</Span>
)}
</Stack>
</Modal>
);
}
src/pages/admin/bots/[bot_id]/(partials)/bot-thinking-row.tsx
+12 -18
View File
@@ -2,15 +2,11 @@ import Row from "@moduletrace/twui/layout/Row";
import Span from "@moduletrace/twui/layout/Span"; import Span from "@moduletrace/twui/layout/Span";
import Stack from "@moduletrace/twui/layout/Stack"; import Stack from "@moduletrace/twui/layout/Stack";
import BotAvatar from "@/src/components/general/bot-avatar"; import BotAvatar from "@/src/components/general/bot-avatar";
import type { BotFace } from "@/src/dict/bot-dict";
import botStoneProps from "@/src/utils/bot-stone-props"; import botStoneProps from "@/src/utils/bot-stone-props";
import { import { BotGettingMessageStatusLine } from "@/src/dict/bot-queue-dict";
BotComputerBusyStatusLine,
BotGettingMessageStatusLine,
BotReplyJobTitle,
BotWaitingStatusLine,
} from "@/src/dict/bot-queue-dict";
import type { BotWithLastMessage } from "@/src/types"; import type { BotWithLastMessage } from "@/src/types";
import botBusyState from "../(utils)/bot-busy-state";
import StopBotButton from "./stop-bot-button";
type Props = { type Props = {
bot: BotWithLastMessage; bot: BotWithLastMessage;
@@ -45,7 +41,7 @@ type Props = {
* reply shows from the thread alone: every user message queues one, so while * reply shows from the thread alone: every user message queues one, so while
* the user has the last word a reply is due here, even while the bot works * the user has the last word a reply is due here, even while the bot works
* elsewhere. Only the waits (for a model, for its computer) come from the * elsewhere. Only the waits (for a model, for its computer) come from the
* face. * face. While it shows, a stop button pulls the plug on the bot's work.
*/ */
export default function BotThinkingRow({ export default function BotThinkingRow({
bot, bot,
@@ -56,17 +52,14 @@ export default function BotThinkingRow({
has_task, has_task,
}: Props) { }: Props) {
const name = bot.name || "The bot"; const name = bot.name || "The bot";
const asleep = bot.face == "asleep"; const { waiting, getting, thinking, working, busy, stoppable } = botBusyState({
const waiting_model = asleep && bot.status_line == BotWaitingStatusLine && Boolean(awaiting_reply || has_task); bot,
const waiting_computer = awaiting_reply,
asleep && bot.status_line == BotComputerBusyStatusLine && Boolean(has_task) && !awaiting_reply; receiving,
const waiting = waiting_model || waiting_computer; has_task,
const getting = Boolean(receiving) && Boolean(awaiting_reply); });
const thinking = Boolean(awaiting_reply) && !waiting && !getting;
const working =
!awaiting_reply && bot.face == "working" && bot.status_line != BotReplyJobTitle && Boolean(has_task);
if (!getting && !thinking && !working && !waiting) return null; if (!busy) return null;
const label = getting const label = getting
? `${name} is ${BotGettingMessageStatusLine.toLowerCase()}…` ? `${name} is ${BotGettingMessageStatusLine.toLowerCase()}…`
@@ -95,6 +88,7 @@ export default function BotThinkingRow({
</Span> </Span>
) : null} ) : null}
</Stack> </Stack>
<StopBotButton bot_name={bot.name} disabled={!stoppable} />
</Row> </Row>
); );
} }
src/pages/admin/bots/[bot_id]/(partials)/conversation-header-menu.tsx
+13 -2
View File
@@ -12,9 +12,11 @@ import isPassThroughUser from "@/src/utils/rbac/is-pass-through-user";
import BotComputersModal from "./bot-computers-modal"; import BotComputersModal from "./bot-computers-modal";
import BotFilesModal from "./bot-files-modal"; import BotFilesModal from "./bot-files-modal";
import BotTasksModal from "./bot-tasks-modal"; import BotTasksModal from "./bot-tasks-modal";
import BotRoutinesModal from "./bot-routines-modal";
import BotModelsModal from "./bot-models-modal"; import BotModelsModal from "./bot-models-modal";
import HeaderMenuItem from "./header-menu-item"; import HeaderMenuItem from "./header-menu-item";
import NewSessionButton from "./new-session-button"; import NewSessionButton from "./new-session-button";
import StopBotButton from "./stop-bot-button";
type Props = { type Props = {
bot: BotWithLastMessage; bot: BotWithLastMessage;
@@ -47,8 +49,9 @@ type Props = {
/** /**
* The conversation header's menu: the sessions, a new session, the bot's * The conversation header's menu: the sessions, a new session, the bot's
* computer, its models and its settings. A row that opens a modal closes * computer, its models, its settings and, while it works, stopping it. A
* the menu first, so the modal never sits behind an open menu. * row that opens a modal closes the menu first, so the modal never sits
* behind an open menu.
*/ */
export default function ConversationHeaderMenu({ export default function ConversationHeaderMenu({
bot, bot,
@@ -108,6 +111,12 @@ export default function ConversationHeaderMenu({
workspace={workspace} workspace={workspace}
on_open={on_open} on_open={on_open}
/> />
<BotRoutinesModal
menu
bot={bot}
workspace={workspace}
on_open={on_open}
/>
<BotComputersModal <BotComputersModal
menu menu
bot={bot} bot={bot}
@@ -129,6 +138,8 @@ export default function ConversationHeaderMenu({
onSaved={on_model_saved} onSaved={on_model_saved}
/> />
</Stack> </Stack>
<Divider className="my-1 mx-3 w-auto border-hairline dark:border-hairline" />
<StopBotButton menu bot_name={bot.name} />
{can_open_settings ? ( {can_open_settings ? (
<> <>
<Divider className="my-1 mx-3 w-auto border-hairline dark:border-hairline" /> <Divider className="my-1 mx-3 w-auto border-hairline dark:border-hairline" />
src/pages/admin/bots/[bot_id]/(partials)/message-thread.tsx
+21 -7
View File
@@ -1,15 +1,17 @@
import { useMemo, useRef, type RefObject } from "react"; import { useMemo, type RefObject } from "react";
import Stack from "@moduletrace/twui/layout/Stack"; import Stack from "@moduletrace/twui/layout/Stack";
import type { import type {
BotMessageWithAttachments, BotMessageWithAttachments,
BotWithLastMessage, BotWithLastMessage,
} from "@/src/types"; } from "@/src/types";
import useMessageSelection from "../(hooks)/use-message-selection"; import useMessageSelection from "../(hooks)/use-message-selection";
import useThreadScroll from "../(hooks)/use-thread-scroll";
import groupMessages from "../(utils)/group-messages"; import groupMessages from "../(utils)/group-messages";
import BotThinkingRow from "./bot-thinking-row"; import BotThinkingRow from "./bot-thinking-row";
import MessageGroupBlock from "./message-group-block"; import MessageGroupBlock from "./message-group-block";
import OlderMessagesLoader from "./older-messages-loader"; import OlderMessagesLoader from "./older-messages-loader";
import QuoteSelectionButton from "./quote-selection-button"; import QuoteSelectionButton from "./quote-selection-button";
import ScrollToBottomButton from "./scroll-to-bottom-button";
type Props = { type Props = {
bot: BotWithLastMessage; bot: BotWithLastMessage;
@@ -51,7 +53,9 @@ type Props = {
* While the bot works on a reply or a task, a status row sits under the * While the bot works on a reply or a task, a status row sits under the
* last one. Text selected in a message can be quoted in the composer. * last one. Text selected in a message can be quoted in the composer.
* Older messages load as it scrolls to the top; the reversed column keeps * Older messages load as it scrolls to the top; the reversed column keeps
* the view where it was as they land above it. * the view where it was as they land above it. The thread follows new
* messages while it sits at its bottom, stays put while it's scrolled up
* and then offers a button to get back down.
*/ */
export default function MessageThread({ export default function MessageThread({
bot, bot,
@@ -64,14 +68,18 @@ export default function MessageThread({
older, older,
}: Props) { }: Props) {
const groups = useMemo(() => groupMessages({ messages }), [messages]); const groups = useMemo(() => groupMessages({ messages }), [messages]);
const ref = useRef<HTMLDivElement>(null); const scroll = useThreadScroll({ messages });
const { selection, clear } = useMessageSelection({ ref }); const { selection, clear } = useMessageSelection({ ref: scroll.column_ref });
return ( return (
<Stack className="relative grow min-h-0 gap-0! items-stretch">
<Stack <Stack
componentRef={ref as RefObject<HTMLDivElement>} componentRef={scroll.column_ref as RefObject<HTMLDivElement>}
className="grow min-h-0 gap-0! items-stretch overflow-y-auto flex-col-reverse"> className="grow min-h-0 gap-0! items-stretch overflow-y-auto flex-col-reverse"
onScroll={scroll.on_scroll}
>
<Stack <Stack
componentRef={scroll.content_ref as RefObject<HTMLDivElement>}
role="log" role="log"
aria-live="polite" aria-live="polite"
aria-label={`Conversation with ${bot.name}`} aria-label={`Conversation with ${bot.name}`}
@@ -95,11 +103,17 @@ export default function MessageThread({
bot={bot} bot={bot}
progress={progress} progress={progress}
reply_progress={reply_progress} reply_progress={reply_progress}
awaiting_reply={messages[messages.length - 1]?.author == "user"} awaiting_reply={
messages[messages.length - 1]?.author == "user"
}
receiving={receiving} receiving={receiving}
has_task={has_task} has_task={has_task}
/> />
</Stack> </Stack>
</Stack>
{scroll.jump_visible ? (
<ScrollToBottomButton onClick={scroll.jump_to_bottom} />
) : null}
{selection && onQuote ? ( {selection && onQuote ? (
<QuoteSelectionButton <QuoteSelectionButton
selection={selection} selection={selection}
src/pages/admin/bots/[bot_id]/(partials)/message-usage-chip.tsx
+35
View File
@@ -0,0 +1,35 @@
import Span from "@moduletrace/twui/layout/Span";
import type { TokenUsageSummary } from "@/src/harness/usage/types";
import formatTokens from "@/src/utils/format-tokens";
type Props = {
usage?: TokenUsageSummary | null;
};
/**
* What one answer cost, in the row under it beside its timestamp: the
* tokens every model call behind it spent (a chat turn, reading an image,
* making one), and the price when the provider reported it. The hover
* title breaks it down.
*/
export default function MessageUsageChip({ usage }: Props) {
if (!usage || usage.total_tokens <= 0) return null;
const { prompt_tokens, completion_tokens, total_tokens, cost, calls } = usage;
const title = [
`${formatTokens({ tokens: prompt_tokens })} in, ${formatTokens({ tokens: completion_tokens })} out`,
`${calls} model ${calls == 1 ? "call" : "calls"}`,
typeof cost == "number" && cost > 0 ? `$${cost.toFixed(4)}` : "",
]
.filter(Boolean)
.join(" · ");
return (
<Span
title={title}
className="text-[11px] leading-4 text-ink-secondary tabular-nums whitespace-nowrap cursor-help"
>
{formatTokens({ tokens: total_tokens })} tokens
</Span>
);
}
src/pages/admin/bots/[bot_id]/(partials)/scroll-to-bottom-button.tsx
+28
View File
@@ -0,0 +1,28 @@
import Button from "@moduletrace/twui/layout/Button";
import { ArrowDown } from "lucide-react";
type Props = {
onClick: () => void;
};
/**
* "Jump to latest", floating over the bottom of the thread while it sits
* above it: a bot's reply that landed out of sight, or a long scroll back
* through the conversation.
*/
export default function ScrollToBottomButton({ onClick }: Props) {
return (
<Button
type="button"
color="secondary"
title="Jump to the latest message"
aria-label="Jump to the latest message"
variant="outlined"
onClick={onClick}
className="absolute bottom-3 left-1/2 z-20 shrink-0 -translate-x-1/2 gap-1.5! px-2.5! py-1.5! rounded-default-sm border border-solid border-hairline bg-surface shadow-popover text-xs font-bold text-ink hover:bg-sunken focus-visible:ring-2 focus-visible:ring-ink"
>
<ArrowDown size={13} aria-hidden />
Jump to latest
</Button>
);
}
src/pages/admin/bots/[bot_id]/(partials)/stop-bot-button.tsx
+79
View File
@@ -0,0 +1,79 @@
import { useContext } from "react";
import Button from "@moduletrace/twui/layout/Button";
import { CircleStop } from "lucide-react";
import { BotStopButtonLabel } from "@/src/dict/bot-queue-dict";
import { StopBotContext } from "../(data)/stop-bot-context";
import HeaderMenuItem from "./header-menu-item";
type Props = {
/**
* Whose work stops, named in the tooltip so it's clear this pulls the
* bot's plug everywhere the user chats with it.
*/
bot_name?: string | null;
/**
* Shown as a row in the conversation header's menu instead of a button
* in the thinking row.
*/
menu?: boolean;
/**
* The caller already knows there's nothing to pull (a row that says the
* bot is only collecting the message), so the button sits out.
*/
disabled?: boolean;
};
/**
* Pulls the plug on the bot for the signed-in user: its queued and running
* work stops wherever they chat with it, so a bot stuck waiting for a model
* (or anything else) comes back. The thinking row carries it while
* something is running; the header's menu carries it too, disabled when
* there's nothing to stop.
*/
export default function StopBotButton({ bot_name, menu, disabled }: Props) {
const { stop, stopping, error, can_stop } = useContext(StopBotContext);
const name = bot_name || "the bot";
const off = Boolean(disabled) || !can_stop || stopping;
const title = error
? error
: off && !stopping
? `Nothing of ${name}'s is running for you.`
: `Stop ${name}. Its queued and running work stops, everywhere you chat with it.`;
if (menu) {
return (
<HeaderMenuItem
icon={CircleStop}
title={BotStopButtonLabel}
description={title}
hint={stopping ? "Stopping" : undefined}
loading={stopping}
disabled={off}
onClick={() => {
void stop();
}}
/>
);
}
return (
<Button
title={title}
aria-label={title}
type="button"
variant="ghost"
color="text"
size="smaller"
disabled={off}
loading={stopping}
loadingIconSize="small"
beforeIcon={<CircleStop size={14} aria-hidden />}
onClick={() => {
void stop();
}}
className="shrink-0 text-ink-secondary dark:text-ink-secondary hover:text-ink hover:dark:text-ink hover:bg-sunken hover:dark:bg-sunken rounded-default transition-colors duration-[var(--m-fast)] focus-visible:ring-2 focus-visible:ring-ink"
>
{BotStopButtonLabel}
</Button>
);
}
src/pages/admin/bots/[bot_id]/(sections)/conversation.tsx
+36 -1
View File
@@ -15,9 +15,15 @@ import { CrewBotsContext } from "../(data)/crew-bots-context";
import { CrewWorkspacesContext } from "../(data)/crew-workspaces-context"; import { CrewWorkspacesContext } from "../(data)/crew-workspaces-context";
import { BotMessageActionsContext } from "../(data)/message-actions-context"; import { BotMessageActionsContext } from "../(data)/message-actions-context";
import { SpeakMessageContext } from "../(data)/speak-message-context"; import { SpeakMessageContext } from "../(data)/speak-message-context";
import {
StopBotContext,
type StopBotState,
} from "../(data)/stop-bot-context";
import reactToBotMessage from "../(functions)/react-to-bot-message"; import reactToBotMessage from "../(functions)/react-to-bot-message";
import botBusyState from "../(utils)/bot-busy-state";
import dropSentMessage from "../(utils)/drop-sent-message"; import dropSentMessage from "../(utils)/drop-sent-message";
import useBotComputers from "../(hooks)/use-bot-computers"; import useBotComputers from "../(hooks)/use-bot-computers";
import useStopBot from "../(hooks)/use-stop-bot";
import useTaskProgress from "../(hooks)/use-task-progress"; import useTaskProgress from "../(hooks)/use-task-progress";
import useReplyProgress from "../(hooks)/use-reply-progress"; import useReplyProgress from "../(hooks)/use-reply-progress";
import useReplyCatchUp from "../(hooks)/use-reply-catch-up"; import useReplyCatchUp from "../(hooks)/use-reply-catch-up";
@@ -90,6 +96,10 @@ export default function Conversation() {
workspace_id, workspace_id,
computers: pageProps?.bot_computers || [], computers: pageProps?.bot_computers || [],
}); });
const stop_bot = useStopBot({
bot_id: bot?.id || 0,
workspace_id,
});
const speak = useSpeakMessage({ const speak = useSpeakMessage({
enabled: Boolean(pageProps?.bot_voice?.speech), enabled: Boolean(pageProps?.bot_voice?.speech),
@@ -233,7 +243,26 @@ export default function Conversation() {
if (!bot?.id || !live_bot) return null; if (!bot?.id || !live_bot) return null;
return ( /**
* What the stop control pulls the plug on, read the way the thinking
* row reads it: the same conditions, so the header's row is enabled
* exactly while the thread shows the bot at work.
*/
const stop_state: StopBotState = {
...stop_bot,
can_stop: botBusyState({
bot: live_bot,
awaiting_reply,
receiving: pending_messages.length > 0,
has_task: Boolean(progress.task_id),
}).stoppable,
};
/**
* The thread, kept intact so the stop control can wrap it without
* reindenting every provider.
*/
const thread = (
<CrewBotsContext.Provider value={bots}> <CrewBotsContext.Provider value={bots}>
<CrewWorkspacesContext.Provider value={workspaces}> <CrewWorkspacesContext.Provider value={workspaces}>
<BotComputersContext.Provider value={bot_computers}> <BotComputersContext.Provider value={bot_computers}>
@@ -316,4 +345,10 @@ export default function Conversation() {
</CrewWorkspacesContext.Provider> </CrewWorkspacesContext.Provider>
</CrewBotsContext.Provider> </CrewBotsContext.Provider>
); );
return (
<StopBotContext.Provider value={stop_state}>
{thread}
</StopBotContext.Provider>
);
} }
src/pages/admin/bots/[bot_id]/(utils)/bot-busy-state.test.ts
+193
View File
@@ -0,0 +1,193 @@
import { describe, expect, test } from "bun:test";
import {
BotComputerBusyStatusLine,
BotGettingMessageStatusLine,
BotReplyJobTitle,
BotWaitingStatusLine,
} from "@/src/dict/bot-queue-dict";
import type { BotWithLastMessage } from "@/src/types";
import botBusyState from "./bot-busy-state";
function bot(overrides: Partial<BotWithLastMessage>): BotWithLastMessage {
return { id: 1, name: "Ember", ...overrides } as BotWithLastMessage;
}
describe("botBusyState", () => {
test("a reply owed here is thinking", () => {
const state = botBusyState({ bot: bot({ face: "idle" }), awaiting_reply: true });
expect(state.thinking).toBe(true);
expect(state.busy).toBe(true);
});
test("a message still on its way is getting, not thinking", () => {
const state = botBusyState({
bot: bot({ face: "working" }),
awaiting_reply: true,
receiving: true,
});
expect(state.getting).toBe(true);
expect(state.thinking).toBe(false);
});
test("a task running while the bot has the last word is working", () => {
const state = botBusyState({
bot: bot({ face: "working", status_line: "Working" }),
awaiting_reply: false,
has_task: true,
});
expect(state.working).toBe(true);
expect(state.busy).toBe(true);
});
test("a reply's own turn never shows as working on a task", () => {
const state = botBusyState({
bot: bot({ face: "working", status_line: BotReplyJobTitle }),
has_task: true,
});
expect(state.working).toBe(false);
expect(state.busy).toBe(false);
});
test("waiting for a model is a wait, with a reply owed here", () => {
const state = botBusyState({
bot: bot({ face: "asleep", status_line: BotWaitingStatusLine }),
awaiting_reply: true,
});
expect(state.waiting).toBe(true);
expect(state.thinking).toBe(false);
expect(state.busy).toBe(true);
});
test("waiting for a model is a wait, with a task running", () => {
const state = botBusyState({
bot: bot({ face: "asleep", status_line: BotWaitingStatusLine }),
has_task: true,
});
expect(state.waiting).toBe(true);
expect(state.busy).toBe(true);
});
test("waiting for its computer is a wait only while a task runs", () => {
const waiting = botBusyState({
bot: bot({ face: "asleep", status_line: BotComputerBusyStatusLine }),
has_task: true,
});
const idle = botBusyState({
bot: bot({ face: "asleep", status_line: BotComputerBusyStatusLine }),
});
expect(waiting.waiting).toBe(true);
expect(idle.busy).toBe(false);
});
test("a message still on its way wins over the wait, so the row says what the user just did", () => {
const state = botBusyState({
bot: bot({ face: "asleep", status_line: BotWaitingStatusLine }),
awaiting_reply: true,
receiving: true,
});
expect(state.getting).toBe(true);
expect(state.waiting).toBe(true);
});
test("an idle bot with nothing owed and no task shows nothing", () => {
const state = botBusyState({ bot: bot({ face: "idle" }) });
expect(state.busy).toBe(false);
});
test("a done face with no task shows nothing", () => {
expect(botBusyState({ bot: bot({ face: "done" }) }).busy).toBe(false);
});
});
describe("botBusyState stoppable", () => {
test("busy in this thread is stoppable", () => {
expect(
botBusyState({
bot: bot({ face: "idle" }),
awaiting_reply: true,
}).stoppable,
).toBe(true);
});
test("working somewhere else is stoppable too, since the face is per user", () => {
const state = botBusyState({ bot: bot({ face: "working" }) });
expect(state.busy).toBe(false);
expect(state.stoppable).toBe(true);
});
test("asleep on a wait line is stoppable, even when this thread shows nothing", () => {
for (const status_line of [BotWaitingStatusLine, BotComputerBusyStatusLine]) {
expect(botBusyState({ bot: bot({ face: "asleep", status_line }) }).stoppable).toBe(true);
}
});
test("a bot simply resting is not stoppable", () => {
const state = botBusyState({
bot: bot({ face: "asleep", status_line: "Until tomorrow" }),
});
expect(state.busy).toBe(false);
expect(state.stoppable).toBe(false);
});
test("a message only being collected is not stoppable: its reply is queued after it", () => {
const state = botBusyState({
bot: bot({ face: "idle" }),
awaiting_reply: true,
receiving: true,
});
expect(state.getting).toBe(true);
expect(state.stoppable).toBe(false);
});
test("done, needs-you and idle mean nothing is running", () => {
for (const face of ["done", "needs-you", "idle"] as const) {
const state = botBusyState({ bot: bot({ face }) });
expect(state.stoppable).toBe(false);
}
});
test("a silently stopped bot is not stoppable", () => {
const state = botBusyState({
bot: bot({ face: "idle" }),
awaiting_reply: false,
has_task: false,
});
expect(state.stoppable).toBe(false);
});
test("a reply job's own turn with no task owed is not stoppable", () => {
/**
* The face says working while a reply runs somewhere else, which is
* real work: it's stoppable even though this thread owes nothing.
*/
const state = botBusyState({
bot: bot({ face: "working", status_line: BotReplyJobTitle }),
});
expect(state.stoppable).toBe(true);
});
test("the getting line still labels the row", () => {
const state = botBusyState({
bot: bot({ face: "idle", status_line: BotGettingMessageStatusLine }),
awaiting_reply: true,
receiving: true,
});
expect(state.getting).toBe(true);
});
});
src/pages/admin/bots/[bot_id]/(utils)/bot-busy-state.ts
+111
View File
@@ -0,0 +1,111 @@
import type { BotWithLastMessage } from "@/src/types";
import {
BotComputerBusyStatusLine,
BotReplyJobTitle,
BotWaitingStatusLine,
} from "@/src/dict/bot-queue-dict";
type Params = {
bot: BotWithLastMessage;
/**
* The last message in the open thread is the user's, so a reply is due
* here.
*/
awaiting_reply?: boolean;
/**
* A message is still on its way to the server.
*/
receiving?: boolean;
/**
* A task of the bot's is running.
*/
has_task?: boolean;
};
export type BotBusyState = {
/**
* Waiting for a model to come back, or for the bot's computer.
*/
waiting: boolean;
/**
* The user's message hasn't reached the server yet.
*/
getting: boolean;
/**
* A reply is due in this thread.
*/
thinking: boolean;
/**
* A task is running and the bot has the last word.
*/
working: boolean;
/**
* Anything of the above, so the row has something to say.
*/
busy: boolean;
/**
* There's work of the bot's to pull the plug on. A message still on
* its way doesn't count: its reply is queued after it lands, so a stop
* now would cancel nothing and be overtaken a moment later. The face
* covers work in the user's other conversations too, since it's one
* per user; `asleep` only counts on the two status lines the queue
* sets while it waits, never a bot that's simply resting.
*/
stoppable: boolean;
};
/**
* What a bot is doing in a conversation, read from the thread and its face.
* The thread decides the reply states: every user message queues one, so
* while the user has the last word a reply is due here, even while the bot
* works elsewhere. Only the waits (for a model, for its computer) come from
* the face. The thinking row reads it for its line, and the conversation
* reads it for the stop control.
*/
export default function botBusyState({
bot,
awaiting_reply,
receiving,
has_task,
}: Params): BotBusyState {
const asleep = bot.face == "asleep";
const waiting_model =
asleep && bot.status_line == BotWaitingStatusLine && Boolean(awaiting_reply || has_task);
const waiting_computer =
asleep &&
bot.status_line == BotComputerBusyStatusLine &&
Boolean(has_task) &&
!awaiting_reply;
const waiting = waiting_model || waiting_computer;
const getting = Boolean(receiving) && Boolean(awaiting_reply);
const thinking = Boolean(awaiting_reply) && !waiting && !getting;
const working =
!awaiting_reply &&
bot.face == "working" &&
bot.status_line != BotReplyJobTitle &&
Boolean(has_task);
/**
* Work runs here (a reply owed, a task of this thread's) or somewhere
* else of this user's (the face says `working`, or `asleep` on a wait
* line).
*/
const stoppable =
waiting ||
thinking ||
working ||
bot.face == "working" ||
(asleep &&
(bot.status_line == BotWaitingStatusLine ||
bot.status_line == BotComputerBusyStatusLine));
return {
waiting,
getting,
thinking,
working,
busy: getting || thinking || working || waiting,
stoppable,
};
}
src/pages/admin/bots/[bot_id]/(utils)/thread-scroll.test.ts
+98
View File
@@ -0,0 +1,98 @@
import { describe, expect, test } from "bun:test";
import {
ThreadBottomSlack,
shouldFollowThread,
threadOffset,
} from "./thread-scroll";
describe("threadOffset", () => {
test("content whose bottom edge is the column's is at the bottom", () => {
expect(
threadOffset({ content_bottom: 800, column_bottom: 800 }),
).toEqual({ distance: 0, near_bottom: true });
});
test("content ending just short of the column's bottom is at the bottom", () => {
expect(
threadOffset({ content_bottom: 780, column_bottom: 800 }),
).toEqual({ distance: 0, near_bottom: true });
});
test("a small offset past the bottom still counts as being there", () => {
const offset = threadOffset({
content_bottom: 800 + ThreadBottomSlack - 1,
column_bottom: 800,
});
expect(offset.near_bottom).toBe(true);
expect(offset.distance).toBe(ThreadBottomSlack - 1);
});
test("one pixel past the slack is not the bottom anymore", () => {
const offset = threadOffset({
content_bottom: 800 + ThreadBottomSlack + 1,
column_bottom: 800,
});
expect(offset.near_bottom).toBe(false);
expect(offset.distance).toBe(ThreadBottomSlack + 1);
});
test("scrolled up by a long way is far from the bottom", () => {
expect(
threadOffset({ content_bottom: 4800, column_bottom: 800 }),
).toEqual({ distance: 4000, near_bottom: false });
});
test("the slack is configurable", () => {
expect(
threadOffset({
content_bottom: 1000,
column_bottom: 800,
slack: 200,
}).near_bottom,
).toBe(true);
});
});
describe("shouldFollowThread", () => {
test("a chat at the bottom follows a bot's reply", () => {
expect(
shouldFollowThread({
near_bottom: true,
newest_author: "bot",
kept_last_message: true,
}),
).toBe(true);
});
test("a chat scrolled up stays put when a bot's reply lands", () => {
expect(
shouldFollowThread({
near_bottom: false,
newest_author: "bot",
kept_last_message: true,
}),
).toBe(false);
});
test("a message the user sent always brings the thread to it", () => {
expect(
shouldFollowThread({
near_bottom: false,
newest_author: "user",
kept_last_message: true,
}),
).toBe(true);
});
test("a list that replaced the old one opens at its bottom", () => {
expect(
shouldFollowThread({
near_bottom: false,
newest_author: "bot",
kept_last_message: false,
}),
).toBe(true);
});
});
src/pages/admin/bots/[bot_id]/(utils)/thread-scroll.ts
+75
View File
@@ -0,0 +1,75 @@
/**
* Pixels from the bottom that still count as being at it, so a mouse wheel
* tick or a rounding difference doesn't hide the latest message.
*/
export const ThreadBottomSlack = 48;
/**
* How long a smooth jump to the bottom runs, in milliseconds. The scrolling
* it causes is ignored for that long (so the button doesn't flicker back
* mid-animation), and the window ends on its own however it lands.
*/
export const ThreadJumpWindowMs = 800;
/**
* How far the thread's content sits below the bottom of its column, in
* pixels, and whether that still counts as being at the bottom. It reads
* the two rects rather than `scrollTop`, whose sign a reversed column
* (`flex-col-reverse`) reports differently across engines: the content's
* bottom edge is the column's when the thread is at its bottom, and past it
* by however far the thread has been scrolled up.
*/
export function threadOffset({
content_bottom,
column_bottom,
slack = ThreadBottomSlack,
}: {
content_bottom: number;
column_bottom: number;
slack?: number;
}) {
const distance = content_bottom - column_bottom;
const at_bottom = distance <= slack;
return {
distance: at_bottom && distance < 0 ? 0 : distance,
near_bottom: at_bottom,
};
}
/**
* Whether the thread should show a change of its messages, which keeps the
* bottom in view while the user is there: it follows while at the bottom, a
* message the user sent always counts, and a list that replaced the old one
* (a new session, a first render) opens at its bottom. A chat scrolled up
* stays where it is when a bot answers.
*/
export function shouldFollowThread({
near_bottom,
newest_author,
kept_last_message,
}: {
near_bottom: boolean;
newest_author?: string | null;
kept_last_message: boolean;
}) {
if (!kept_last_message) return true;
if (newest_author == "user") return true;
return near_bottom;
}
/**
* Puts the thread at its bottom, animated or at once. A reversed column
* clamps a past-the-end `scrollTop` back onto its bottom, so the same call
* serves either direction.
*/
export function scrollThreadToBottom({
element,
behavior = "auto",
}: {
element: HTMLElement;
behavior?: ScrollBehavior;
}) {
element.scrollTo({ top: element.scrollHeight, behavior });
}

Some files were not shown because too many files have changed in this diff Show More