Show how each bot message was generated
Every bot message gets an info button that opens its details: the provider and model behind it, tokens and cost, the job that ran it, each model used and every call in order. Each model call now records how it was made in token_usages.settings (generation settings sent, whose settings they were, pinned or automatic model, fallbacks), shown per model in the modal. Live replies now carry their token usage, from the job's push and the live relay alike, so it shows without a reload. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
@@ -211,9 +211,9 @@ Every folder on the data volume, browsable read-only: a bot's own, a workspace's
|
||||
Every model call is written down in `token_usages` as it happens, so a message can show what it cost and `/admin/usage` can total the install's spending by day, by model and by provider.
|
||||
|
||||
- **Whose call it was:** `src/harness/usage/harness-usage-scope.ts` is an `AsyncLocalStorage` scope (`user_id`, `bot_id`, `workspace_id`, `session_id`, `job_id`, `message_id`) that a run enters once, in `run-bot-job.ts` (`runWithHarnessUsageScope`), so everything underneath - the text loop's turns, a session summary, a transcription, an image, an embedding - is attributed without threading the ids through every harness function. The store is merged, not replaced: a nested run keeps the outer ids and overrides only what it knows better, and a `null` is a real value that clears a field (a bot's own chat has no workspace). `withUsageScope` (`with-usage-scope.ts`) wraps one call the app makes on a user's behalf outside any run (the chat's transcribe and speak routes), and `runWithFallbackScope` (`run-with-fallback-scope.ts`) attributes work whose run may be missing - a file the ingest queue embeds has no run, so it is charged to its owner (`grabUsageOwnerUserId`, the workspace's creator else the bot's) - while `mode: "own"` forces the given scope (the session namer, whose job is already complete and must not pay for it). Without an owner nothing is written: a row no one can be shown is not worth a row.
|
||||
- **Recording:** `record-harness-usage.ts` writes one row and never throws (a model that answered must not lose its answer to a failed insert; it logs with `logError` instead). `build-usage-call.ts` turns the candidate that really answered into the row's provider and model (`provider_id`/`provider_name` and the catalog `model_id` plus the provider's `model_slug`), and providers that report no usage still get a row, so a call is never invisible. Every entry point calls it in `withCandidates`'s `on_success`: `generate-text.ts`, `read-image.ts`, `transcribe-audio.ts`, `speak-text.ts` (billed per character, so the text's length stands in for what was spoken), `generate-image.ts`, `embed-text.ts`.
|
||||
- **Recording:** `record-harness-usage.ts` writes one row and never throws (a model that answered must not lose its answer to a failed insert; it logs with `logError` instead). `build-usage-call.ts` turns the candidate that really answered into the row's provider and model (`provider_id`/`provider_name` and the catalog `model_id` plus the provider's `model_slug`), and providers that report no usage still get a row, so a call is never invisible. Each row also keeps how the call was made in `token_usages.settings` (`HarnessCallSettings`, JSON): whether the model was picked or automatic, whose setting it used (`grabChatParams`' and `grabHarnessSetting`'s `source`), the generation settings actually sent (`flattenSentParams`: by setting key, so OpenRouter's `reasoning: { effort }` reads back as `reasoning_effort`), `max_tokens`, the speech voice, and the candidates that failed before it (`withCandidates` hands `failed` to `on_success`). Every entry point calls it in `withCandidates`'s `on_success`: `generate-text.ts`, `read-image.ts`, `transcribe-audio.ts`, `speak-text.ts` (billed per character, so the text's length stands in for what was spoken), `generate-image.ts`, `embed-text.ts`.
|
||||
- **Messages:** a reply or task job's calls carry the job's id as they happen, since its message doesn't exist yet; `complete-bot-job.ts` then runs `UPDATE token_usages SET message_id = ? WHERE job_id = ? AND message_id IS NULL` once the message is inserted, so the thread can show what the answer cost. Calls that produce no message (naming a session, embedding a file, transcribing a dictation) keep a null `message_id` and are still counted in the totals.
|
||||
- **Message usage:** `grab-message-usage.ts` sums a batch of messages' calls (one `GROUP BY message_id`) and `grab-bot-messages.ts` attaches it as `BotMessageWithAttachments.usage` (`TokenUsageSummary`: prompt, completion, total, cost, calls). `formatTokens` (`src/utils/format-tokens.ts`) writes a count as a person reads it (842 / 1.2K / 842K / 1.2M). `MessageUsageChip` (`bots/[bot_id]/(partials)/message-usage-chip.tsx`) sits in the actions row under a bot message beside its timestamp and play button, showing the total with the breakdown in its `title`, and renders nothing when a message has no calls.
|
||||
- **Message usage:** `grab-message-usage.ts` sums a batch of messages' calls (one `GROUP BY message_id`) and `grab-bot-messages.ts` attaches it as `BotMessageWithAttachments.usage` (`TokenUsageSummary`: prompt, completion, total, cost, calls). `formatTokens` (`src/utils/format-tokens.ts`) writes a count as a person reads it (842 / 1.2K / 842K / 1.2M). `MessageUsageChip` (`bots/[bot_id]/(partials)/message-usage-chip.tsx`) sits in the actions row under a bot message beside its timestamp and play button, showing the total with the breakdown in its `title`, and renders nothing when a message has no calls. Beside the reply and reaction actions, every bot message has an info button (`message-details-modal.tsx`, `use-message-details`) that opens its details, loaded when it opens: `GET api/admin/bots/message-details?message_id=` (`grabSpeakableMessage` for access, then `grabMessageDetails` in `src/functions/backend/db/usage/`) returns the provider and model of every call (`token_usages` joined to `harness_providers` and `harness_models` for the kind and display name), the calls summed per provider, model and harness (`groupMessageCalls`, `src/utils/group-message-calls.ts`), each model's settings (`parseCallSettings`, `describeCallSettings`, by their titles in `HarnessParameters`), the calls in order up to `AppData.MessageDetailsMaxCalls` (a fallback marked, settings on hover), the totals and the job that ran it (kind, status, attempts, how long it took and its steps, while `bots_queue` keeps it) (`MessageDetails` in `src/types/index.ts`).
|
||||
- **Usage page:** `/admin/usage`, a `RailPages` entry like Routines and Activity rather than a settings sub page, since it is the install's own record rather than a setting (super admins and admins only, `canManageCrew`; the dict entry's `can_view` hides it for anyone else, and `grab-usage-server-props.ts` redirects them to the bots). Helpers in `src/functions/backend/db/usage/`: `grab-usage-overview.ts` runs `grab-usage-totals.ts` (one total and the same split per harness), `grab-usage-days.ts` -> `bucket-usage-days.ts` (one bucket per day, empty days at zero), `grab-usage-models.ts` (per provider, model slug and harness, busiest first, joined to `harness_models` for a display name), plus the previous period of the same length and a rolling week. The window comes from `grab-usage-window.ts` (`UsageDefaultDays` 30, `UsageMaxDays` 365, clamped). API `api/admin/usage` (GET only) for another period, zod in `src/schemas/usage-schema.ts`.
|
||||
- **Days are local, not UTC:** `grabUsageDays` groups in SQL with `date(created_at / 1000, 'unixepoch', 'localtime')`, matching the local midnights `grabUsageWindow` builds and the `YYYY-MM-DD` `localDate` writes. Grouping by the bare UTC day put a late evening's calls in the previous bucket and silently dropped them whenever no bucket carried that name (the chart's total disagreed with the totals card); `bucket-usage-days.test.ts` pins this. Never name a local day with `toISOString().slice(0, 10)`.
|
||||
- **The page:** `index.server.ts` -> `(functions)/grab-usage-server-props.ts` (crew managers only, the period in the address), `index.tsx` in `SettingsShell` with `wide`, `(sections)/usage-section.tsx` (state in the address through `history.pushState`, `use-usage-overview.ts` loading another period), `(partials)/usage-totals.tsx` (tokens, model calls, cost, previous period with a change label), `usage-day-chart.tsx` + `usage-day-bar.tsx` (bars from static `h-*` classes in `(data)/usage-bar-heights.ts`, since inline styles are not allowed), `usage-harness-breakdown.tsx` + `usage-harness-row.tsx`, and `usage-models-table.tsx` + `usage-model-row.tsx` (calls, in, out, tokens, **price per thousand tokens** worked out from cost and tokens, which is what makes one model's cost comparable with another's).
|
||||
|
||||
Reference in New Issue
Block a user