Record what every model call spends, and show it
Nothing anywhere recorded what a model call cost: the harness used a provider's usage numbers only to decide whether a turn had finished, then dropped them. So a reply could not say what it took to make, one model could not be compared with another on price, and there was no way to ask what a day, a week or a month of the crew's work came to. - `token_usages` records one row per call: user, bot, workspace, session, job and message, the harness, the provider and model that actually answered, the prompt and completion counts, and the cost when the provider reports one. - `harness-usage-scope.ts` carries those ids in an AsyncLocalStorage scope that `run-bot-job.ts` enters once, so a run's turns, its session summary, its transcriptions, images and embeddings are all attributed without threading ids through every harness function. Nested runs merge rather than replace, and a null genuinely clears a field (a bot's own chat has no workspace). - Work with no run behind it is charged to its owner instead: `run-with-fallback-scope.ts` reaches the workspace's creator else the bot's, and deliberately does not override a real run, so an ingest a bot asked for is not misattributed to the bot's owner. - All six harnesses record through `withCandidates`'s `on_success`. Recording never throws: a model that answered must not lose its answer to a failed insert. - A job's calls land before its message exists, so `complete-bot-job.ts` back-fills `message_id` once the message is inserted. - The message's actions row shows what it cost, beside the timestamp and the play button, with the breakdown in its title. - `/admin/usage` totals the install's spending for 7, 30 or 90 days: the total against the previous period, a day-by-day chart, a split per kind of call, and a table per model with the price per thousand tokens worked out from cost and tokens, which is what makes two models comparable. Crew managers only, since it is the whole install's work rather than one user's conversation. It sits with Routines and Activity in the user popover rather than under Settings, because it is a record of what happened, not something to configure. Days are grouped in local time (`date(created_at / 1000, 'unixepoch', 'localtime')`), matching the local midnights the window builds and the `YYYY-MM-DD` buckets it looks up. Grouping by the bare UTC day put a late evening's calls under a name no bucket carried, so they vanished and the chart's total silently disagreed with the totals card; a test pins it. `canSeeRailItem` now decides who sees a popover or rail entry. The two each filtered with their own inline rule and only one understood `can_view`, so a gated page could be offered in the popover to everyone.
This commit is contained in:
@@ -206,15 +206,28 @@ Every folder on the data volume, browsable read-only: a bot's own, a workspace's
|
||||
- **Replies and tasks:** `run-bot-reply.ts` answers from the last `BotReplyHistoryMessages` messages (`botMessagesToHarness`) with `ChatReplyNote`, the bot's computers (`compose-computers-note.ts`: what they are and that they're reached through `start_task`) and the chat tools, at most `BotReplyMaxToolSteps` steps, and finishes silently if the bot already had the last word (`enqueueBotReply` reuses a pending reply, so a burst of messages gets one). `run-bot-task.ts` runs the brief (`compose-job-brief.ts`) with `TaskRunNote`, the bot tools, the file tools and the job's computer tools, closing the MCP connection after. Both add `BotFilesNote` (where a bot's files live, that they persist, and to search before saying it doesn't know) and collect `attachments` for files the bot wrote with `attach`, which `completeBotJob` links to the result message. Retryable harness errors go back to the queue's backoff, the rest are `BotJobFatalError`.
|
||||
- **API:** `api/admin/harness/` (`save-provider` also syncs, `sync-provider`, `delete-provider`, `save-settings`), super admin only, each returning the fresh `grabHarnessOverview()` page data.
|
||||
|
||||
### Token usage
|
||||
|
||||
Every model call is written down in `token_usages` as it happens, so a message can show what it cost and `/admin/usage` can total the install's spending by day, by model and by provider.
|
||||
|
||||
- **Whose call it was:** `src/harness/usage/harness-usage-scope.ts` is an `AsyncLocalStorage` scope (`user_id`, `bot_id`, `workspace_id`, `session_id`, `job_id`, `message_id`) that a run enters once, in `run-bot-job.ts` (`runWithHarnessUsageScope`), so everything underneath - the text loop's turns, a session summary, a transcription, an image, an embedding - is attributed without threading the ids through every harness function. The store is merged, not replaced: a nested run keeps the outer ids and overrides only what it knows better, and a `null` is a real value that clears a field (a bot's own chat has no workspace). `withUsageScope` (`with-usage-scope.ts`) wraps one call the app makes on a user's behalf outside any run (the chat's transcribe and speak routes), and `runWithFallbackScope` (`run-with-fallback-scope.ts`) attributes work whose run may be missing - a file the ingest queue embeds has no run, so it is charged to its owner (`grabUsageOwnerUserId`, the workspace's creator else the bot's) - while `mode: "own"` forces the given scope (the session namer, whose job is already complete and must not pay for it). Without an owner nothing is written: a row no one can be shown is not worth a row.
|
||||
- **Recording:** `record-harness-usage.ts` writes one row and never throws (a model that answered must not lose its answer to a failed insert; it logs with `logError` instead). `build-usage-call.ts` turns the candidate that really answered into the row's provider and model (`provider_id`/`provider_name` and the catalog `model_id` plus the provider's `model_slug`), and providers that report no usage still get a row, so a call is never invisible. Every entry point calls it in `withCandidates`'s `on_success`: `generate-text.ts`, `read-image.ts`, `transcribe-audio.ts`, `speak-text.ts` (billed per character, so the text's length stands in for what was spoken), `generate-image.ts`, `embed-text.ts`.
|
||||
- **Messages:** a reply or task job's calls carry the job's id as they happen, since its message doesn't exist yet; `complete-bot-job.ts` then runs `UPDATE token_usages SET message_id = ? WHERE job_id = ? AND message_id IS NULL` once the message is inserted, so the thread can show what the answer cost. Calls that produce no message (naming a session, embedding a file, transcribing a dictation) keep a null `message_id` and are still counted in the totals.
|
||||
- **Message usage:** `grab-message-usage.ts` sums a batch of messages' calls (one `GROUP BY message_id`) and `grab-bot-messages.ts` attaches it as `BotMessageWithAttachments.usage` (`TokenUsageSummary`: prompt, completion, total, cost, calls). `formatTokens` (`src/utils/format-tokens.ts`) writes a count as a person reads it (842 / 1.2K / 842K / 1.2M). `MessageUsageChip` (`bots/[bot_id]/(partials)/message-usage-chip.tsx`) sits in the actions row under a bot message beside its timestamp and play button, showing the total with the breakdown in its `title`, and renders nothing when a message has no calls.
|
||||
- **Usage page:** `/admin/usage`, a `RailPages` entry like Routines and Activity rather than a settings sub page, since it is the install's own record rather than a setting (super admins and admins only, `canManageCrew`; the dict entry's `can_view` hides it for anyone else, and `grab-usage-server-props.ts` redirects them to the bots). Helpers in `src/functions/backend/db/usage/`: `grab-usage-overview.ts` runs `grab-usage-totals.ts` (one total and the same split per harness), `grab-usage-days.ts` -> `bucket-usage-days.ts` (one bucket per day, empty days at zero), `grab-usage-models.ts` (per provider, model slug and harness, busiest first, joined to `harness_models` for a display name), plus the previous period of the same length and a rolling week. The window comes from `grab-usage-window.ts` (`UsageDefaultDays` 30, `UsageMaxDays` 365, clamped). API `api/admin/usage` (GET only) for another period, zod in `src/schemas/usage-schema.ts`.
|
||||
- **Days are local, not UTC:** `grabUsageDays` groups in SQL with `date(created_at / 1000, 'unixepoch', 'localtime')`, matching the local midnights `grabUsageWindow` builds and the `YYYY-MM-DD` `localDate` writes. Grouping by the bare UTC day put a late evening's calls in the previous bucket and silently dropped them whenever no bucket carried that name (the chart's total disagreed with the totals card); `bucket-usage-days.test.ts` pins this. Never name a local day with `toISOString().slice(0, 10)`.
|
||||
- **The page:** `index.server.ts` -> `(functions)/grab-usage-server-props.ts` (crew managers only, the period in the address), `index.tsx` in `SettingsShell` with `wide`, `(sections)/usage-section.tsx` (state in the address through `history.pushState`, `use-usage-overview.ts` loading another period), `(partials)/usage-totals.tsx` (tokens, model calls, cost, previous period with a change label), `usage-day-chart.tsx` + `usage-day-bar.tsx` (bars from static `h-*` classes in `(data)/usage-bar-heights.ts`, since inline styles are not allowed), `usage-harness-breakdown.tsx` + `usage-harness-row.tsx`, and `usage-models-table.tsx` + `usage-model-row.tsx` (calls, in, out, tokens, **price per thousand tokens** worked out from cost and tokens, which is what makes one model's cost comparable with another's).
|
||||
|
||||
## Admin Route Hierarchy
|
||||
|
||||
Live routes under `/admin` (see `AdminSections`): `/admin/bots`, `/admin/workspaces` (the index opens the first workspace), `/admin/routines`, `/admin/logs`, `/admin/help` and `/admin/settings`. `/admin` redirects to `/admin/bots`; a workspace's dashboard is `/admin/workspaces/<id>`. The routes in detail:
|
||||
|
||||
- `/admin/bots` (live) - the Cobble chat shell, plus `[bot_id]/`
|
||||
- `/admin/routines` (live) - every schedule the user's bots have (see Schedules), opened from "Routines" in the rail's user popover
|
||||
- `/admin/usage` (live) - what the crew's model calls spent, per day, per kind of call and per model, for crew managers (see Token usage), opened from "Usage" in the rail's user popover
|
||||
- `/admin/logs` (live) - Activity, the audit log: super admins only (others are redirected; the popover's "Activity" item shows to super admins only). Rendered in `SettingsShell` with `wide`
|
||||
- `/admin/help` (live) - familiar problems and their fixes, one `SettingsGroup` per topic (`(data)/help-topics.ts`, markdown rendered with the chat's `MessageMarkdownComponents`, the topic id as the group's anchor). First topic: signing in to Google and other strict sites with Cookie-Editor over the Desktop view (`HelpSignInTopicId` in `src/dict/help-dict.ts`; `ask_user_to_take_over` points bots to it)
|
||||
- Routines, Files, Activity and Help are the `RailPages` (`src/dict/rail-pages-dict.ts`), opened from the rail's user popover (which lists them, then Settings and Sign out). On them the rail shows its own aside (`(partials)/rail-pages-list.tsx`): a back link to the bots, then those pages, like settings
|
||||
- Routines, Files, Usage, Activity and Help are the `RailPages` (`src/dict/rail-pages-dict.ts`), opened from the rail's user popover (which lists them, then Settings and Sign out). On them the rail shows its own aside (`(partials)/rail-pages-list.tsx`): a back link to the bots, then those pages, like settings. A page carries `super_admin_only` or a `can_view` rule (Usage is `canManageCrew`), and `canSeeRailItem` (`src/utils/rbac/can-see-rail-item.ts`, tested) decides both the popover and the aside, so a page is never offered in one and hidden in the other
|
||||
- `/admin/settings/backups` - super admin only (listed in `SettingsPages`), two groups. Database snapshots: create, list, restore, download and delete in `db/.backups` (`bun-sqlite backup`); "Restore from file" beside "Create backup" opens `upload-restore-modal.tsx` (upload a SQLite database file, which is saved to `.backups` then restored with `bun-sqlite import <file>`, since `restore` is interactive). Whole-data archives: `write-data-backup-archive.ts` tars all of `DATA_DIR` (database, media, computer homes, bot and workspace files, MCP installs) into `BACKUP_ARCHIVE_DIR` as `cobble-data-<epoch-ms>.tar.gz` (via `tar -czf` into a `.part` name, renamed on success so a cut-off run is never listed, with a `--transform` putting every member under one root folder named after the archive), keeping the newest `AppData.DataBackupsKept` and never touching `db/.backups`; its own "Restore from file" uploads a tar.gz (`upload-restore-data.ts`, gzip header checked first). `restore-data-backup.ts` extracts an archive to a temp dir, requires exactly one root folder holding a `db` folder (`findArchiveRoot`, since an upload is saved under a fresh timestamp and its inner name no longer matches the file name), takes a safety archive before touching anything (`write-data-backup-archive.ts`, not the locked `create-data-backup.ts`), empties `DATA_DIR` (the snapshots included) and copies the archive's in, then `restart-app.ts` restarts the process through the docker socket after the response lands (`process.exit`, the container's restart policy brings it back), because the open SQLite handle points at the replaced file. Row actions are icon-only (`RowIconButton`, `src/components/general/row-icon-button.tsx`): restore, download, delete. `BACKUP_ARCHIVE_DIR` is a `ProcessOnlyEnvKeys` start-up path (the installer mounts `$COBBLE_HOME/backups-archives` there; outside docker it falls back to `~/.cobble/backups`). Every backup and restore, of either kind, runs one at a time via `with-backup-run-lock.ts`. Helpers in `src/functions/backend/backups/`, API routes in `api/admin/backups/`
|
||||
- `/admin/settings/database` - super admin only (listed in `SettingsPages`), a read-only browser of the install's SQLite database, no writes anywhere in it. `grabDatabaseTables` reads the catalog once (every table and its columns, SQLite's own `sqlite_*` tables and sqlite-vec's `bot_file_vectors_*` shadow tables left out) and adds each table's `COUNT(*)` and its size from `dbstat` (index pages folded back onto their table). `grabTableRecords` then returns one page (`AppData.DatabaseBrowserPageSize`, 50) of the open table's records newest key first, every value already cut in SQL at `AppData.DatabaseBrowserCellMaxChars` (400) so a table of long texts never ships a megabyte of JSON, and a `truncated` flag per cell; the search matches over the concatenation of every column's text (`buildSearchClause`: every word, or the whole phrase), which is what also works on a virtual table, where `MATCH` and `LIKE` over a vector blob do not. A record's key is its single primary key column, else its rowid, else nothing (a virtual table with no primary key reads as plain cells); clicking a cell's row opens `grabTableRecord` for that key, which returns the record whole up to `AppData.DatabaseBrowserValueMaxChars` (20000), the answer to a cell cut at three lines. `clampPage` puts a page past the end on the last one. The page's state (table, search, mode, page) lives in the address through `databaseHref`, so a table or a search can be linked to; the open table's first page comes with the page props and anything else loads through the API. Helpers in `src/functions/backend/database/` (`build-table-catalog-sql`, `build-table-sizes-sql`, `build-records-sql`, `build-search-clause`, `resolve-table-key`, `build-record-row`, `quote-sql-identifier`), API `api/admin/database/records` and `record` (GET only, `grabSuperAdminUser`), zod in `src/schemas/database-schema.ts`, page under `src/pages/admin/settings/database/`
|
||||
- `/admin/settings` (live) - redirects to `SettingsDefaultPath` (`/admin/settings/general`). Sub pages: `general/` (theme: System/Light/Dark per device via `useThemePreference`, stored in `localStorage[AppData.ThemeStorageKey]`, the same key `ThemeInitScript` and twui's `ColorSchemeSelector` read; applies instantly, no save bar; for super admins the prompt appendix (`prompt-section.tsx`, `system-prompt-append-form.tsx`) saving `api/admin/settings/system-prompt-append` into the `bot_system_prompt_append` app setting; and, for `canManageCrew`, the bot file quota: `storage-section.tsx`, `storage-quota-form.tsx` saving `api/admin/settings/storage-quota` into `app_settings`; and, for everyone, the running version: `version-section.tsx`, `version-line.tsx` showing `pageProps.running_version` from `runningVersion` (`src/utils/running-version.ts`: `COBBLE_VERSION`, the release tag the image was built with, else `package.json`'s version, in development too, with `(development)` after it and no such suffix in production), `profile/` (current user profile), `notifications/` (this device's Web Push switch, a test send, and every device the user subscribed), `change-password/` (set/change own password) `models/` (super admin only, `super_admin_only` in the dict hides it from the rail: model providers and which model each harness uses) and `environment/` (super admin only: app settings and the envs bots and computers use, per scope), each wrapped in `SettingsShell` (`(partials)/settings-shell.tsx`: 64px pane header like the conversation header, then one centered column). Content is built from `SettingsGroup` blocks (label + description left, controls right, hairline-separated, no cards) and ends with the sticky `SettingsSaveBar` (Save enabled only when dirty, Discard, one-line status). Saves don't reload the page. Keep save notices in component state: `useFormInit` clears `status` whenever the form changes. `bots/` (the crew: chat, edit with `BotForm`, models, delete a worker via `api/admin/bots/delete-bot`), `workspaces/` (open, create through the Add flow, delete), `computers/` (state and resources, live; super admins create, edit with `AddComputerForm`, retry, delete), `mcps/`, `tools/` and `skills/` (see MCPs, custom tools and skills: super admins add, edit, refresh or try, delete; admins only set Access, `extension-access-modal.tsx`; add, edit, access and try open in modals (`AddModalShell`). A skill has no status: nothing connects or runs, so `skill-row.tsx` puts its slug, who gets it (`extension-access-label.ts`) and its description where a server's or tool's status line goes), each MCP server or tool row with a status line under it (`extension-status-line.tsx`: connected or not, last run worked or failed, when, and why), those five for `canManageCrew` only (`grab-crew-settings-server-props.ts`, `can_view` in the dict) and `users/` (for `canListUsers`: super admins and admins; the dict's `can_view` gates it in the rail; add with a password (`canCreateUser`), edit (`canUpdateUser`), both with the one `UserForm` (`admin/settings/users/(partials)/user-form/`, `useFormInit`, photo included, saved in one request), revoke (adds `revoked`, `canRevokeUser`), delete (`canRevokeUser`'s rules, never yourself), pick their user types from `UserTypes` (`user-types-select.tsx`: only the ones `filterAssignableUserTypes` allows, never `UneditableUserTypes`; at least one, since login needs a type; `saveUser` checks each grant and removal with `userTypeAssignment`; the super admin's types are never shown or changed there, and they can't be revoked; the password is entered twice, `SaveUserRequestSchema` checks they match), delete with everything theirs; `api/admin/users/save-user`, `delete-user`). The lists share `SettingsList`, `SettingsListRow`, `SettingsRowAction` and `use-settings-list`. Sub pages are declared once in `src/dict/settings-pages-dict.ts`; on every `/admin/settings/*` page the rail swaps the crew for a back link plus that list. Add a sub page by adding its folder and a dict entry
|
||||
|
||||
Reference in New Issue
Block a user