diff --git a/AGENTS.md b/AGENTS.md index cb12cb8..ca5bd9e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -206,15 +206,28 @@ Every folder on the data volume, browsable read-only: a bot's own, a workspace's - **Replies and tasks:** `run-bot-reply.ts` answers from the last `BotReplyHistoryMessages` messages (`botMessagesToHarness`) with `ChatReplyNote`, the bot's computers (`compose-computers-note.ts`: what they are and that they're reached through `start_task`) and the chat tools, at most `BotReplyMaxToolSteps` steps, and finishes silently if the bot already had the last word (`enqueueBotReply` reuses a pending reply, so a burst of messages gets one). `run-bot-task.ts` runs the brief (`compose-job-brief.ts`) with `TaskRunNote`, the bot tools, the file tools and the job's computer tools, closing the MCP connection after. Both add `BotFilesNote` (where a bot's files live, that they persist, and to search before saying it doesn't know) and collect `attachments` for files the bot wrote with `attach`, which `completeBotJob` links to the result message. Retryable harness errors go back to the queue's backoff, the rest are `BotJobFatalError`. - **API:** `api/admin/harness/` (`save-provider` also syncs, `sync-provider`, `delete-provider`, `save-settings`), super admin only, each returning the fresh `grabHarnessOverview()` page data. +### Token usage + +Every model call is written down in `token_usages` as it happens, so a message can show what it cost and `/admin/usage` can total the install's spending by day, by model and by provider. + +- **Whose call it was:** `src/harness/usage/harness-usage-scope.ts` is an `AsyncLocalStorage` scope (`user_id`, `bot_id`, `workspace_id`, `session_id`, `job_id`, `message_id`) that a run enters once, in `run-bot-job.ts` (`runWithHarnessUsageScope`), so everything underneath - the text loop's turns, a session summary, a transcription, an image, an embedding - is attributed without threading the ids through every harness function. The store is merged, not replaced: a nested run keeps the outer ids and overrides only what it knows better, and a `null` is a real value that clears a field (a bot's own chat has no workspace). `withUsageScope` (`with-usage-scope.ts`) wraps one call the app makes on a user's behalf outside any run (the chat's transcribe and speak routes), and `runWithFallbackScope` (`run-with-fallback-scope.ts`) attributes work whose run may be missing - a file the ingest queue embeds has no run, so it is charged to its owner (`grabUsageOwnerUserId`, the workspace's creator else the bot's) - while `mode: "own"` forces the given scope (the session namer, whose job is already complete and must not pay for it). Without an owner nothing is written: a row no one can be shown is not worth a row. +- **Recording:** `record-harness-usage.ts` writes one row and never throws (a model that answered must not lose its answer to a failed insert; it logs with `logError` instead). `build-usage-call.ts` turns the candidate that really answered into the row's provider and model (`provider_id`/`provider_name` and the catalog `model_id` plus the provider's `model_slug`), and providers that report no usage still get a row, so a call is never invisible. Every entry point calls it in `withCandidates`'s `on_success`: `generate-text.ts`, `read-image.ts`, `transcribe-audio.ts`, `speak-text.ts` (billed per character, so the text's length stands in for what was spoken), `generate-image.ts`, `embed-text.ts`. +- **Messages:** a reply or task job's calls carry the job's id as they happen, since its message doesn't exist yet; `complete-bot-job.ts` then runs `UPDATE token_usages SET message_id = ? WHERE job_id = ? AND message_id IS NULL` once the message is inserted, so the thread can show what the answer cost. Calls that produce no message (naming a session, embedding a file, transcribing a dictation) keep a null `message_id` and are still counted in the totals. +- **Message usage:** `grab-message-usage.ts` sums a batch of messages' calls (one `GROUP BY message_id`) and `grab-bot-messages.ts` attaches it as `BotMessageWithAttachments.usage` (`TokenUsageSummary`: prompt, completion, total, cost, calls). `formatTokens` (`src/utils/format-tokens.ts`) writes a count as a person reads it (842 / 1.2K / 842K / 1.2M). `MessageUsageChip` (`bots/[bot_id]/(partials)/message-usage-chip.tsx`) sits in the actions row under a bot message beside its timestamp and play button, showing the total with the breakdown in its `title`, and renders nothing when a message has no calls. +- **Usage page:** `/admin/usage`, a `RailPages` entry like Routines and Activity rather than a settings sub page, since it is the install's own record rather than a setting (super admins and admins only, `canManageCrew`; the dict entry's `can_view` hides it for anyone else, and `grab-usage-server-props.ts` redirects them to the bots). Helpers in `src/functions/backend/db/usage/`: `grab-usage-overview.ts` runs `grab-usage-totals.ts` (one total and the same split per harness), `grab-usage-days.ts` -> `bucket-usage-days.ts` (one bucket per day, empty days at zero), `grab-usage-models.ts` (per provider, model slug and harness, busiest first, joined to `harness_models` for a display name), plus the previous period of the same length and a rolling week. The window comes from `grab-usage-window.ts` (`UsageDefaultDays` 30, `UsageMaxDays` 365, clamped). API `api/admin/usage` (GET only) for another period, zod in `src/schemas/usage-schema.ts`. +- **Days are local, not UTC:** `grabUsageDays` groups in SQL with `date(created_at / 1000, 'unixepoch', 'localtime')`, matching the local midnights `grabUsageWindow` builds and the `YYYY-MM-DD` `localDate` writes. Grouping by the bare UTC day put a late evening's calls in the previous bucket and silently dropped them whenever no bucket carried that name (the chart's total disagreed with the totals card); `bucket-usage-days.test.ts` pins this. Never name a local day with `toISOString().slice(0, 10)`. +- **The page:** `index.server.ts` -> `(functions)/grab-usage-server-props.ts` (crew managers only, the period in the address), `index.tsx` in `SettingsShell` with `wide`, `(sections)/usage-section.tsx` (state in the address through `history.pushState`, `use-usage-overview.ts` loading another period), `(partials)/usage-totals.tsx` (tokens, model calls, cost, previous period with a change label), `usage-day-chart.tsx` + `usage-day-bar.tsx` (bars from static `h-*` classes in `(data)/usage-bar-heights.ts`, since inline styles are not allowed), `usage-harness-breakdown.tsx` + `usage-harness-row.tsx`, and `usage-models-table.tsx` + `usage-model-row.tsx` (calls, in, out, tokens, **price per thousand tokens** worked out from cost and tokens, which is what makes one model's cost comparable with another's). + ## Admin Route Hierarchy Live routes under `/admin` (see `AdminSections`): `/admin/bots`, `/admin/workspaces` (the index opens the first workspace), `/admin/routines`, `/admin/logs`, `/admin/help` and `/admin/settings`. `/admin` redirects to `/admin/bots`; a workspace's dashboard is `/admin/workspaces/`. The routes in detail: - `/admin/bots` (live) - the Cobble chat shell, plus `[bot_id]/` - `/admin/routines` (live) - every schedule the user's bots have (see Schedules), opened from "Routines" in the rail's user popover +- `/admin/usage` (live) - what the crew's model calls spent, per day, per kind of call and per model, for crew managers (see Token usage), opened from "Usage" in the rail's user popover - `/admin/logs` (live) - Activity, the audit log: super admins only (others are redirected; the popover's "Activity" item shows to super admins only). Rendered in `SettingsShell` with `wide` - `/admin/help` (live) - familiar problems and their fixes, one `SettingsGroup` per topic (`(data)/help-topics.ts`, markdown rendered with the chat's `MessageMarkdownComponents`, the topic id as the group's anchor). First topic: signing in to Google and other strict sites with Cookie-Editor over the Desktop view (`HelpSignInTopicId` in `src/dict/help-dict.ts`; `ask_user_to_take_over` points bots to it) -- Routines, Files, Activity and Help are the `RailPages` (`src/dict/rail-pages-dict.ts`), opened from the rail's user popover (which lists them, then Settings and Sign out). On them the rail shows its own aside (`(partials)/rail-pages-list.tsx`): a back link to the bots, then those pages, like settings +- Routines, Files, Usage, Activity and Help are the `RailPages` (`src/dict/rail-pages-dict.ts`), opened from the rail's user popover (which lists them, then Settings and Sign out). On them the rail shows its own aside (`(partials)/rail-pages-list.tsx`): a back link to the bots, then those pages, like settings. A page carries `super_admin_only` or a `can_view` rule (Usage is `canManageCrew`), and `canSeeRailItem` (`src/utils/rbac/can-see-rail-item.ts`, tested) decides both the popover and the aside, so a page is never offered in one and hidden in the other - `/admin/settings/backups` - super admin only (listed in `SettingsPages`), two groups. Database snapshots: create, list, restore, download and delete in `db/.backups` (`bun-sqlite backup`); "Restore from file" beside "Create backup" opens `upload-restore-modal.tsx` (upload a SQLite database file, which is saved to `.backups` then restored with `bun-sqlite import `, since `restore` is interactive). Whole-data archives: `write-data-backup-archive.ts` tars all of `DATA_DIR` (database, media, computer homes, bot and workspace files, MCP installs) into `BACKUP_ARCHIVE_DIR` as `cobble-data-.tar.gz` (via `tar -czf` into a `.part` name, renamed on success so a cut-off run is never listed, with a `--transform` putting every member under one root folder named after the archive), keeping the newest `AppData.DataBackupsKept` and never touching `db/.backups`; its own "Restore from file" uploads a tar.gz (`upload-restore-data.ts`, gzip header checked first). `restore-data-backup.ts` extracts an archive to a temp dir, requires exactly one root folder holding a `db` folder (`findArchiveRoot`, since an upload is saved under a fresh timestamp and its inner name no longer matches the file name), takes a safety archive before touching anything (`write-data-backup-archive.ts`, not the locked `create-data-backup.ts`), empties `DATA_DIR` (the snapshots included) and copies the archive's in, then `restart-app.ts` restarts the process through the docker socket after the response lands (`process.exit`, the container's restart policy brings it back), because the open SQLite handle points at the replaced file. Row actions are icon-only (`RowIconButton`, `src/components/general/row-icon-button.tsx`): restore, download, delete. `BACKUP_ARCHIVE_DIR` is a `ProcessOnlyEnvKeys` start-up path (the installer mounts `$COBBLE_HOME/backups-archives` there; outside docker it falls back to `~/.cobble/backups`). Every backup and restore, of either kind, runs one at a time via `with-backup-run-lock.ts`. Helpers in `src/functions/backend/backups/`, API routes in `api/admin/backups/` - `/admin/settings/database` - super admin only (listed in `SettingsPages`), a read-only browser of the install's SQLite database, no writes anywhere in it. `grabDatabaseTables` reads the catalog once (every table and its columns, SQLite's own `sqlite_*` tables and sqlite-vec's `bot_file_vectors_*` shadow tables left out) and adds each table's `COUNT(*)` and its size from `dbstat` (index pages folded back onto their table). `grabTableRecords` then returns one page (`AppData.DatabaseBrowserPageSize`, 50) of the open table's records newest key first, every value already cut in SQL at `AppData.DatabaseBrowserCellMaxChars` (400) so a table of long texts never ships a megabyte of JSON, and a `truncated` flag per cell; the search matches over the concatenation of every column's text (`buildSearchClause`: every word, or the whole phrase), which is what also works on a virtual table, where `MATCH` and `LIKE` over a vector blob do not. A record's key is its single primary key column, else its rowid, else nothing (a virtual table with no primary key reads as plain cells); clicking a cell's row opens `grabTableRecord` for that key, which returns the record whole up to `AppData.DatabaseBrowserValueMaxChars` (20000), the answer to a cell cut at three lines. `clampPage` puts a page past the end on the last one. The page's state (table, search, mode, page) lives in the address through `databaseHref`, so a table or a search can be linked to; the open table's first page comes with the page props and anything else loads through the API. Helpers in `src/functions/backend/database/` (`build-table-catalog-sql`, `build-table-sizes-sql`, `build-records-sql`, `build-search-clause`, `resolve-table-key`, `build-record-row`, `quote-sql-identifier`), API `api/admin/database/records` and `record` (GET only, `grabSuperAdminUser`), zod in `src/schemas/database-schema.ts`, page under `src/pages/admin/settings/database/` - `/admin/settings` (live) - redirects to `SettingsDefaultPath` (`/admin/settings/general`). Sub pages: `general/` (theme: System/Light/Dark per device via `useThemePreference`, stored in `localStorage[AppData.ThemeStorageKey]`, the same key `ThemeInitScript` and twui's `ColorSchemeSelector` read; applies instantly, no save bar; for super admins the prompt appendix (`prompt-section.tsx`, `system-prompt-append-form.tsx`) saving `api/admin/settings/system-prompt-append` into the `bot_system_prompt_append` app setting; and, for `canManageCrew`, the bot file quota: `storage-section.tsx`, `storage-quota-form.tsx` saving `api/admin/settings/storage-quota` into `app_settings`; and, for everyone, the running version: `version-section.tsx`, `version-line.tsx` showing `pageProps.running_version` from `runningVersion` (`src/utils/running-version.ts`: `COBBLE_VERSION`, the release tag the image was built with, else `package.json`'s version, in development too, with `(development)` after it and no such suffix in production), `profile/` (current user profile), `notifications/` (this device's Web Push switch, a test send, and every device the user subscribed), `change-password/` (set/change own password) `models/` (super admin only, `super_admin_only` in the dict hides it from the rail: model providers and which model each harness uses) and `environment/` (super admin only: app settings and the envs bots and computers use, per scope), each wrapped in `SettingsShell` (`(partials)/settings-shell.tsx`: 64px pane header like the conversation header, then one centered column). Content is built from `SettingsGroup` blocks (label + description left, controls right, hairline-separated, no cards) and ends with the sticky `SettingsSaveBar` (Save enabled only when dirty, Discard, one-line status). Saves don't reload the page. Keep save notices in component state: `useFormInit` clears `status` whenever the form changes. `bots/` (the crew: chat, edit with `BotForm`, models, delete a worker via `api/admin/bots/delete-bot`), `workspaces/` (open, create through the Add flow, delete), `computers/` (state and resources, live; super admins create, edit with `AddComputerForm`, retry, delete), `mcps/`, `tools/` and `skills/` (see MCPs, custom tools and skills: super admins add, edit, refresh or try, delete; admins only set Access, `extension-access-modal.tsx`; add, edit, access and try open in modals (`AddModalShell`). A skill has no status: nothing connects or runs, so `skill-row.tsx` puts its slug, who gets it (`extension-access-label.ts`) and its description where a server's or tool's status line goes), each MCP server or tool row with a status line under it (`extension-status-line.tsx`: connected or not, last run worked or failed, when, and why), those five for `canManageCrew` only (`grab-crew-settings-server-props.ts`, `can_view` in the dict) and `users/` (for `canListUsers`: super admins and admins; the dict's `can_view` gates it in the rail; add with a password (`canCreateUser`), edit (`canUpdateUser`), both with the one `UserForm` (`admin/settings/users/(partials)/user-form/`, `useFormInit`, photo included, saved in one request), revoke (adds `revoked`, `canRevokeUser`), delete (`canRevokeUser`'s rules, never yourself), pick their user types from `UserTypes` (`user-types-select.tsx`: only the ones `filterAssignableUserTypes` allows, never `UneditableUserTypes`; at least one, since login needs a type; `saveUser` checks each grant and removal with `userTypeAssignment`; the super admin's types are never shown or changed there, and they can't be revoked; the password is entered twice, `SaveUserRequestSchema` checks they match), delete with everything theirs; `api/admin/users/save-user`, `delete-user`). The lists share `SettingsList`, `SettingsListRow`, `SettingsRowAction` and `use-settings-list`. Sub pages are declared once in `src/dict/settings-pages-dict.ts`; on every `/admin/settings/*` page the rail swaps the crew for a back link plus that list. Add a sub page by adding its folder and a dict entry diff --git a/db/schema.ts b/db/schema.ts index 508c18c..563771a 100644 --- a/db/schema.ts +++ b/db/schema.ts @@ -2866,6 +2866,149 @@ const schema: BUN_SQLITE_DatabaseSchemaType = { }, ], }, + { + tableName: "token_usages", + tableDescription: `One model call that reported its usage: the tokens it spent, on which model and provider, for whom and for what. A bot message's total is the sum of the calls it took; the usage page sums them by model, provider and period`, + fields: [ + { + fieldName: "user_id", + fieldDescription: `Whose work it was: the user whose conversation the call ran for`, + dataType: "INTEGER", + foreignKey: { + cascadeDelete: true, + destinationTableName: "users", + destinationTableColumnName: "id", + destinationTableColumnType: "INTEGER", + foreignKeyName: "token_usage_user_fk", + }, + }, + { + fieldName: "bot_id", + fieldDescription: `The bot it was for, when it was for one`, + dataType: "INTEGER", + foreignKey: { + cascadeDelete: true, + destinationTableName: "bots", + destinationTableColumnName: "id", + destinationTableColumnType: "INTEGER", + foreignKeyName: "token_usage_bot_fk", + }, + }, + { + fieldName: "workspace_id", + fieldDescription: `The workspace the call was in; null for a bot's own chat`, + dataType: "INTEGER", + foreignKey: { + cascadeDelete: true, + destinationTableName: "workspaces", + destinationTableColumnName: "id", + destinationTableColumnType: "INTEGER", + foreignKeyName: "token_usage_workspace_fk", + }, + }, + { + fieldName: "session_id", + fieldDescription: `The conversation it was in`, + dataType: "INTEGER", + foreignKey: { + cascadeDelete: true, + destinationTableName: "bot_sessions", + destinationTableColumnName: "id", + destinationTableColumnType: "INTEGER", + foreignKeyName: "token_usage_session_fk", + }, + }, + { + fieldName: "message_id", + fieldDescription: `The message it produced, so the thread can show what an answer cost. Null for calls that produce no message (naming a session, embedding a file)`, + dataType: "INTEGER", + foreignKey: { + cascadeDelete: true, + destinationTableName: "bot_messages", + destinationTableColumnName: "id", + destinationTableColumnType: "INTEGER", + foreignKeyName: "token_usage_message_fk", + }, + }, + { + fieldName: "job_id", + fieldDescription: `The queued job it ran for, when it ran for one`, + dataType: "INTEGER", + foreignKey: { + cascadeDelete: true, + destinationTableName: "bots_queue", + destinationTableColumnName: "id", + destinationTableColumnType: "INTEGER", + foreignKeyName: "token_usage_job_fk", + }, + }, + { + fieldName: "harness", + fieldDescription: `What the model was used for: a chat turn, reading media, transcribing, speaking, making an image, embedding`, + dataType: "TEXT", + options: Harnesses.map((h) => h.value), + }, + { + fieldName: "provider_id", + fieldDescription: `The provider that served it. Not a foreign key: it stays readable after the provider is deleted`, + dataType: "INTEGER", + }, + { + fieldName: "provider_name", + fieldDescription: `The provider's name as it was, so a deleted provider still reads`, + dataType: "TEXT", + }, + { + fieldName: "model_id", + fieldDescription: `The catalog row of the model. Not a foreign key: a model the provider stops listing still counts`, + dataType: "INTEGER", + }, + { + fieldName: "model_slug", + fieldDescription: `The provider's model id, as it was called`, + dataType: "TEXT", + }, + { + fieldName: "prompt_tokens", + fieldDescription: `Tokens sent: the prompt, the system instructions and the tool results`, + dataType: "INTEGER", + defaultValue: 0, + }, + { + fieldName: "completion_tokens", + fieldDescription: `Tokens written back: the answer, its tool calls and its reasoning`, + dataType: "INTEGER", + defaultValue: 0, + }, + { + fieldName: "cost", + fieldDescription: `The provider's price in USD, when it reports one`, + dataType: "REAL", + }, + ], + indexes: [ + { + indexName: "idx_token_usages_created_at", + indexTableFields: ["created_at"], + }, + { + indexName: "idx_token_usages_message_id", + indexTableFields: ["message_id"], + }, + { + indexName: "idx_token_usages_provider_model", + indexTableFields: ["provider_id", "model_slug"], + }, + { + indexName: "idx_token_usages_user_created_at", + indexTableFields: ["user_id", "created_at"], + }, + { + indexName: "idx_token_usages_bot_created_at", + indexTableFields: ["bot_id", "created_at"], + }, + ], + }, { tableName: "bot_file_vectors", tableDescription: `sqlite-vec vectors of a file's chunks, at \`AppData.EmbeddingDimensions\` wide whatever the model returns. A vec0 table is only recreated with \`bunx bun-sqlite schema --vector\`, so the entrypoint never passes it`, diff --git a/db/types/db.ts b/db/types/db.ts index b03a918..eff4364 100644 --- a/db/types/db.ts +++ b/db/types/db.ts @@ -48,6 +48,7 @@ export const BunSQLiteTables = [ "envs", "bot_files", "bot_file_chunks", + "token_usages", "bot_file_vectors", "file_ingest_queue", "app_settings", @@ -1717,6 +1718,77 @@ export type BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS = { token_estimate?: number | ""; } +export type BUN_SQLITE_COBBLE_TOKEN_USAGES = { + /** + * The unique identifier of the record. + */ + id?: number | ""; + /** + * The time when the record was created. (Unix Timestamp) + */ + created_at?: number | ""; + /** + * The time when the record was updated. (Unix Timestamp) + */ + updated_at?: number | ""; + /** + * Whose work it was: the user whose conversation the call ran for + */ + user_id?: number | ""; + /** + * The bot it was for, when it was for one + */ + bot_id?: number | ""; + /** + * The workspace the call was in; null for a bot's own chat + */ + workspace_id?: number | ""; + /** + * The conversation it was in + */ + session_id?: number | ""; + /** + * The message it produced, so the thread can show what an answer cost. Null for calls that produce no message (naming a session, embedding a file) + */ + message_id?: number | ""; + /** + * The queued job it ran for, when it ran for one + */ + job_id?: number | ""; + /** + * What the model was used for: a chat turn, reading media, transcribing, speaking, making an image, embedding + */ + harness?: "text" | "vision" | "transcription" | "speech" | "image" | "embedding" | ""; + /** + * The provider that served it. Not a foreign key: it stays readable after the provider is deleted + */ + provider_id?: number | ""; + /** + * The provider's name as it was, so a deleted provider still reads + */ + provider_name?: string; + /** + * The catalog row of the model. Not a foreign key: a model the provider stops listing still counts + */ + model_id?: number | ""; + /** + * The provider's model id, as it was called + */ + model_slug?: string; + /** + * Tokens sent: the prompt, the system instructions and the tool results + */ + prompt_tokens?: number | ""; + /** + * Tokens written back: the answer, its tool calls and its reasoning + */ + completion_tokens?: number | ""; + /** + * The provider's price in USD, when it reports one + */ + cost?: number | ""; +} + export type BUN_SQLITE_COBBLE_BOT_FILE_VECTORS = { /** * The unique identifier of the record. @@ -1800,4 +1872,4 @@ export type BUN_SQLITE_COBBLE_APP_SETTINGS = { value?: string; } -export type BUN_SQLITE_COBBLE_ALL_TYPEDEFS = BUN_SQLITE_COBBLE_USERS & BUN_SQLITE_COBBLE_USER_TYPES & BUN_SQLITE_COBBLE_SSO_LOGIN_CODES & BUN_SQLITE_COBBLE_USER_PASSWORDS & BUN_SQLITE_COBBLE_MEDIA & BUN_SQLITE_COBBLE_MEDIA_PARADIGMS & BUN_SQLITE_COBBLE_BOTS & BUN_SQLITE_COBBLE_BOT_SESSIONS & BUN_SQLITE_COBBLE_BOT_MESSAGES & BUN_SQLITE_COBBLE_BOT_MESSAGE_REACTIONS & BUN_SQLITE_COBBLE_BOT_TASKS & BUN_SQLITE_COBBLE_BOT_TASK_STEPS & BUN_SQLITE_COBBLE_TASKS & BUN_SQLITE_COBBLE_TASK_MEDIA & BUN_SQLITE_COBBLE_TASK_NOTES & BUN_SQLITE_COBBLE_BOTS_QUEUE & BUN_SQLITE_COBBLE_BOTS_QUEUE_UPDATES & BUN_SQLITE_COBBLE_CRON_JOBS & BUN_SQLITE_COBBLE_WORKSPACES & BUN_SQLITE_COBBLE_WORKSPACE_BOTS & BUN_SQLITE_COBBLE_WORKSPACE_USERS & BUN_SQLITE_COBBLE_BOT_USERS & BUN_SQLITE_COBBLE_BOT_USER_MEMORIES & BUN_SQLITE_COBBLE_BOT_USER_FACES & BUN_SQLITE_COBBLE_BOT_MESSAGE_MEDIA & BUN_SQLITE_COBBLE_CONTAINERS & BUN_SQLITE_COBBLE_COMPUTERS & BUN_SQLITE_COBBLE_BOT_COMPUTERS & BUN_SQLITE_COBBLE_WORKSPACE_COMPUTERS & BUN_SQLITE_COBBLE_HARNESS_PROVIDERS & BUN_SQLITE_COBBLE_HARNESS_MODELS & BUN_SQLITE_COBBLE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_BOT_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_WORKSPACE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_MCP_SERVERS & BUN_SQLITE_COBBLE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_SKILLS & BUN_SQLITE_COBBLE_BOT_MCP_SERVERS & BUN_SQLITE_COBBLE_WORKSPACE_MCP_SERVERS & BUN_SQLITE_COBBLE_BOT_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_WORKSPACE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_BOT_SKILLS & BUN_SQLITE_COBBLE_WORKSPACE_SKILLS & BUN_SQLITE_COBBLE_LOGS & BUN_SQLITE_COBBLE_USER_PRESENCE & BUN_SQLITE_COBBLE_PUSH_SUBSCRIPTIONS & BUN_SQLITE_COBBLE_ENVS & BUN_SQLITE_COBBLE_BOT_FILES & BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS & BUN_SQLITE_COBBLE_BOT_FILE_VECTORS & BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE & BUN_SQLITE_COBBLE_APP_SETTINGS \ No newline at end of file +export type BUN_SQLITE_COBBLE_ALL_TYPEDEFS = BUN_SQLITE_COBBLE_USERS & BUN_SQLITE_COBBLE_USER_TYPES & BUN_SQLITE_COBBLE_SSO_LOGIN_CODES & BUN_SQLITE_COBBLE_USER_PASSWORDS & BUN_SQLITE_COBBLE_MEDIA & BUN_SQLITE_COBBLE_MEDIA_PARADIGMS & BUN_SQLITE_COBBLE_BOTS & BUN_SQLITE_COBBLE_BOT_SESSIONS & BUN_SQLITE_COBBLE_BOT_MESSAGES & BUN_SQLITE_COBBLE_BOT_MESSAGE_REACTIONS & BUN_SQLITE_COBBLE_BOT_TASKS & BUN_SQLITE_COBBLE_BOT_TASK_STEPS & BUN_SQLITE_COBBLE_TASKS & BUN_SQLITE_COBBLE_TASK_MEDIA & BUN_SQLITE_COBBLE_TASK_NOTES & BUN_SQLITE_COBBLE_BOTS_QUEUE & BUN_SQLITE_COBBLE_BOTS_QUEUE_UPDATES & BUN_SQLITE_COBBLE_CRON_JOBS & BUN_SQLITE_COBBLE_WORKSPACES & BUN_SQLITE_COBBLE_WORKSPACE_BOTS & BUN_SQLITE_COBBLE_WORKSPACE_USERS & BUN_SQLITE_COBBLE_BOT_USERS & BUN_SQLITE_COBBLE_BOT_USER_MEMORIES & BUN_SQLITE_COBBLE_BOT_USER_FACES & BUN_SQLITE_COBBLE_BOT_MESSAGE_MEDIA & BUN_SQLITE_COBBLE_CONTAINERS & BUN_SQLITE_COBBLE_COMPUTERS & BUN_SQLITE_COBBLE_BOT_COMPUTERS & BUN_SQLITE_COBBLE_WORKSPACE_COMPUTERS & BUN_SQLITE_COBBLE_HARNESS_PROVIDERS & BUN_SQLITE_COBBLE_HARNESS_MODELS & BUN_SQLITE_COBBLE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_BOT_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_WORKSPACE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_MCP_SERVERS & BUN_SQLITE_COBBLE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_SKILLS & BUN_SQLITE_COBBLE_BOT_MCP_SERVERS & BUN_SQLITE_COBBLE_WORKSPACE_MCP_SERVERS & BUN_SQLITE_COBBLE_BOT_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_WORKSPACE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_BOT_SKILLS & BUN_SQLITE_COBBLE_WORKSPACE_SKILLS & BUN_SQLITE_COBBLE_LOGS & BUN_SQLITE_COBBLE_USER_PRESENCE & BUN_SQLITE_COBBLE_PUSH_SUBSCRIPTIONS & BUN_SQLITE_COBBLE_ENVS & BUN_SQLITE_COBBLE_BOT_FILES & BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS & BUN_SQLITE_COBBLE_TOKEN_USAGES & BUN_SQLITE_COBBLE_BOT_FILE_VECTORS & BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE & BUN_SQLITE_COBBLE_APP_SETTINGS \ No newline at end of file diff --git a/src/bots-queue/name-bot-session.ts b/src/bots-queue/name-bot-session.ts index cd52db9..33cb693 100644 --- a/src/bots-queue/name-bot-session.ts +++ b/src/bots-queue/name-bot-session.ts @@ -4,6 +4,7 @@ import { AppData } from "@/src/data/app-data"; import { SessionNamePrompt } from "@/src/data/harness-prompts"; import grabBotMessages from "@/src/functions/backend/db/bots/grab-bot-messages"; import generateText from "@/src/harness/text/generate-text"; +import { runWithHarnessUsageScope } from "@/src/harness/usage/harness-usage-scope"; import parseSessionName from "./parse-session-name"; type Params = { @@ -30,7 +31,7 @@ export default async function nameBotSession({ }: Params): Promise { const session = ( await BunSQLite.sql({ - sql: `SELECT id, title, summary FROM bot_sessions WHERE id = ? LIMIT 1`, + sql: `SELECT id, user_id, title, summary FROM bot_sessions WHERE id = ? LIMIT 1`, values: [session_id], }) ).payload?.[0]; @@ -57,14 +58,30 @@ export default async function nameBotSession({ ) .join("\n\n"); - const result = await generateText({ - messages: [ - { role: "system", content: SessionNamePrompt }, - { role: "user", content: transcript }, - ], - bot_id, - workspace_id, - signal, + /** + * The naming turn is a model call too, so it is recorded for this + * conversation. It runs after the job is complete, on purpose, so it + * has no job and no message of its own to charge it to. + */ + const result = await runWithHarnessUsageScope({ + scope: { + user_id: session.user_id || undefined, + bot_id, + workspace_id: workspace_id || null, + session_id: session.id, + job_id: null, + message_id: null, + }, + task: () => + generateText({ + messages: [ + { role: "system", content: SessionNamePrompt }, + { role: "user", content: transcript }, + ], + bot_id, + workspace_id, + signal, + }), }); const name = parseSessionName({ text: result.text }); diff --git a/src/bots-queue/run-bot-job.ts b/src/bots-queue/run-bot-job.ts index 74c5e68..eb605bb 100644 --- a/src/bots-queue/run-bot-job.ts +++ b/src/bots-queue/run-bot-job.ts @@ -1,6 +1,7 @@ import grabBotById from "@/src/functions/backend/db/bots/grab-bot-by-id"; import grabJobUserId from "@/src/functions/backend/db/bots-queue/grab-job-user-id"; import HarnessError from "@/src/harness/harness-error"; +import { runWithHarnessUsageScope } from "@/src/harness/usage/harness-usage-scope"; import BotJobFatalError from "./bot-job-fatal-error"; import runBotReply from "./run-bot-reply"; import runBotTask from "./run-bot-task"; @@ -33,9 +34,25 @@ const runBotJob: BotJobRunner = async (context) => { const owned = { ...bot, id: bot.id }; try { - return job.kind == "reply" - ? await runBotReply({ ...context, bot: owned, user_id }) - : await runBotTask({ ...context, bot: owned, user_id }); + /** + * Everything the job's models do is recorded for this user, this + * bot, this conversation and this job. A reply's text loop adds + * the message it is producing to the one-turn scope it opens. + */ + return await runWithHarnessUsageScope({ + scope: { + user_id, + bot_id: bot.id, + workspace_id: job.workspace_id || null, + session_id: job.session_id || null, + job_id: job.id || null, + message_id: null, + }, + task: () => + job.kind == "reply" + ? runBotReply({ ...context, bot: owned, user_id }) + : runBotTask({ ...context, bot: owned, user_id }), + }); } catch (error: unknown) { if ( error instanceof HarnessError && diff --git a/src/data/app-data.ts b/src/data/app-data.ts index e072fb3..6889297 100644 --- a/src/data/app-data.ts +++ b/src/data/app-data.ts @@ -328,6 +328,16 @@ export const AppData = { * How many older messages each scroll to the top of the thread loads. */ ConversationOlderMessages: 30, + /** + * Days of usage the usage page totals by default, and the most it will + * total at once. + */ + UsageDefaultDays: 30, + UsageMaxDays: 365, + /** + * Models listed per page on the usage page, the busiest first. + */ + UsageModelPageSize: 20, /** * How much of a bot's last message the rail's lists carry, as plain * text (`messagePreview`): a row shows one line of it. diff --git a/src/dict/admin-sections-dict.ts b/src/dict/admin-sections-dict.ts index bd1cde3..227611e 100644 --- a/src/dict/admin-sections-dict.ts +++ b/src/dict/admin-sections-dict.ts @@ -7,6 +7,7 @@ export const AdminSections = [ "/admin/workspaces", "/admin/routines", "/admin/files", + "/admin/usage", "/admin/logs", "/admin/help", "/admin/settings", diff --git a/src/dict/rail-pages-dict.ts b/src/dict/rail-pages-dict.ts index 06981bb..30fd63f 100644 --- a/src/dict/rail-pages-dict.ts +++ b/src/dict/rail-pages-dict.ts @@ -1,4 +1,6 @@ -import { Activity, CircleHelp, Clock, FolderOpen, type LucideIcon } from "lucide-react"; +import { Activity, BarChart3, CircleHelp, Clock, FolderOpen, type LucideIcon } from "lucide-react"; +import type { BUN_SQLITE_COBBLE_USER_TYPES } from "@/db/types/db"; +import canManageCrew from "@/src/utils/rbac/can-manage-crew"; import { HelpPath } from "./help-dict"; export type RailPage = { @@ -9,11 +11,16 @@ export type RailPage = { * Shown and served to super admins only. */ super_admin_only?: boolean; + /** + * Who sees it, when that's an RBAC rule rather than super admins only. + */ + can_view?: (params: { user_types?: Pick[] | null }) => boolean; }; export const RoutinesPath = "/admin/routines"; export const FilesPath = "/admin/files"; export const ActivityPath = "/admin/logs"; +export const UsagePath = "/admin/usage"; /** * Pages opened from the rail's user popover that have their own rail: a @@ -22,6 +29,7 @@ export const ActivityPath = "/admin/logs"; export const RailPages: RailPage[] = [ { title: "Routines", url: RoutinesPath, Icon: Clock }, { title: "Files", url: FilesPath, Icon: FolderOpen }, + { title: "Usage", url: UsagePath, Icon: BarChart3, can_view: canManageCrew }, { title: "Activity", url: ActivityPath, Icon: Activity, super_admin_only: true }, { title: "Help", url: HelpPath, Icon: CircleHelp }, ]; diff --git a/src/functions/backend/db/bot-files/ingest-bot-file.ts b/src/functions/backend/db/bot-files/ingest-bot-file.ts index 2f58cd7..faaed55 100644 --- a/src/functions/backend/db/bot-files/ingest-bot-file.ts +++ b/src/functions/backend/db/bot-files/ingest-bot-file.ts @@ -7,6 +7,10 @@ import type { import { AppData } from "@/src/data/app-data"; import embedText from "@/src/harness/embeddings/embed-text"; import HarnessError from "@/src/harness/harness-error"; +import { grabHarnessUsageScope } from "@/src/harness/usage/harness-usage-scope"; +import type { HarnessUsageScope } from "@/src/harness/usage/types"; +import runWithFallbackScope from "@/src/harness/usage/run-with-fallback-scope"; +import grabUsageOwnerUserId from "@/src/functions/backend/db/usage/grab-usage-owner"; import chunkText from "./chunk-text"; import extractBotFileText from "./extract-bot-file-text"; @@ -72,13 +76,27 @@ export default async function ingestBotFile({ return { status: "skipped", chunks: 0 }; } + /** + * A bot's own run already carries its user. The ingest queue has no run + * behind it, so everything this file costs - reading an image or an + * audio file, embedding its chunks - is entered under the owner's scope + * instead: it spends real tokens and must show on the usage page. + */ + const owner_scope: HarnessUsageScope | null = grabHarnessUsageScope()?.user_id + ? null + : await resolveOwnerScope({ file }); + try { - const text = await extractBotFileText({ - write_path: media?.media_write_path, - text_content: media?.text_content, - name: file.name, - mime_type: file.mime_type, - read_media, + const text = await runWithFallbackScope({ + scope: owner_scope, + task: () => + extractBotFileText({ + write_path: media?.media_write_path, + text_content: media?.text_content, + name: file.name, + mime_type: file.mime_type, + read_media, + }), }); await clearChunks({ file_id: file.id }); @@ -107,11 +125,11 @@ export default async function ingestBotFile({ */ for (let start = 0; start < chunks.length; start += AppData.EmbeddingBatchSize) { const batch = chunks.slice(start, start + AppData.EmbeddingBatchSize); - const { vectors } = await embedText({ - input: batch, - bot_id: file.bot_id!, - workspace_id: file.scope == "workspace" ? file.owner_id : null, + const { vectors } = await runEmbed({ + scope: owner_scope, signal, + file, + batch, }); for (let i = 0; i < vectors.length; i++) { @@ -252,3 +270,54 @@ async function markIngested({ values: [chunks > 0 ? 1 : 0, now, file_id], }).catch(() => {}); } + +/** + * One batch of a file's chunks, embedded under whatever attribution owns + * it: the run that asked for the ingest, else the file's owner. + */ +async function runEmbed({ + scope, + signal, + file, + batch, +}: { + scope: HarnessUsageScope | null; + signal?: AbortSignal; + file: BUN_SQLITE_COBBLE_BOT_FILES; + batch: string[]; +}) { + return await runWithFallbackScope({ + scope, + task: () => + embedText({ + input: batch, + bot_id: file.bot_id!, + workspace_id: file.scope == "workspace" ? file.owner_id : null, + signal, + }), + }); +} + +/** + * The attribution a file the queue ingests reports under: its workspace's + * creator when it shares one, else its bot's. + */ +async function resolveOwnerScope({ + file, +}: { + file: BUN_SQLITE_COBBLE_BOT_FILES; +}): Promise { + const workspace_id = file.scope == "workspace" ? file.owner_id : null; + const user_id = await grabUsageOwnerUserId({ bot_id: file.bot_id, workspace_id }); + + if (!user_id) return null; + + return { + user_id, + bot_id: file.scope == "bot" ? file.owner_id : file.bot_id, + workspace_id, + session_id: null, + message_id: null, + job_id: null, + }; +} diff --git a/src/functions/backend/db/bots-queue/complete-bot-job.ts b/src/functions/backend/db/bots-queue/complete-bot-job.ts index 1629755..ebeaf00 100644 --- a/src/functions/backend/db/bots-queue/complete-bot-job.ts +++ b/src/functions/backend/db/bots-queue/complete-bot-job.ts @@ -89,6 +89,18 @@ export default async function completeBotJob({ media_ids: attachments, }); + if (job.id && message?.id) { + /** + * The job's model calls were written down as they happened, before + * its message existed. Now that it does, they belong to it, so the + * thread can show what the answer cost. + */ + await BunSQLite.sql({ + sql: `UPDATE token_usages SET message_id = ? WHERE job_id = ? AND message_id IS NULL`, + values: [message.id, job.id], + }).catch(() => {}); + } + const user_id = await grabJobUserId({ job }); const bot = !user_id diff --git a/src/functions/backend/db/bots/grab-bot-messages.ts b/src/functions/backend/db/bots/grab-bot-messages.ts index 09d918b..5127584 100644 --- a/src/functions/backend/db/bots/grab-bot-messages.ts +++ b/src/functions/backend/db/bots/grab-bot-messages.ts @@ -1,6 +1,7 @@ import type { BUN_SQLITE_COBBLE_BOT_MESSAGES } from "@/db/types/db"; import type { BotMessageWithAttachments } from "@/src/types"; import BunSQLite from "@moduletrace/bun-sqlite"; +import grabMessageUsage from "../usage/grab-message-usage"; import grabMessagesMedia from "./grab-messages-media"; import grabMessagesReactions from "./grab-messages-reactions"; import grabMessagesReplies from "./grab-messages-replies"; @@ -53,12 +54,13 @@ export default async function grabBotMessages({ const messages = res.payload || []; const message_ids = messages.map((message) => Number(message.id)); - const [media, replies, reactions] = await Promise.all([ + const [media, replies, reactions, usage] = await Promise.all([ grabMessagesMedia({ message_ids }), grabMessagesReplies({ message_ids }), user_id ? grabMessagesReactions({ message_ids, user_id }) : Promise.resolve(new Map()), + grabMessageUsage({ message_ids }), ]); /** @@ -74,6 +76,8 @@ export default async function grabBotMessages({ const files = media.get(id); const reply_to = replies.get(id); + const spent = usage.get(id); + return { ...message, ...(files @@ -88,6 +92,7 @@ export default async function grabBotMessages({ : {}), ...(reply_to ? { reply_to } : {}), ...(user_id ? { reactions: reactions.get(id) || [] } : {}), + ...(spent ? { usage: spent } : {}), }; }); } diff --git a/src/functions/backend/db/usage/bucket-usage-days.test.ts b/src/functions/backend/db/usage/bucket-usage-days.test.ts new file mode 100644 index 0000000..eebc0e5 --- /dev/null +++ b/src/functions/backend/db/usage/bucket-usage-days.test.ts @@ -0,0 +1,101 @@ +import { describe, expect, test } from "bun:test"; +import bucketUsageDays, { localDate } from "./bucket-usage-days"; +import { DayMs, startOfDay } from "./grab-usage-window"; + +function row(local_day: string, tokens: number) { + return { + local_day, + calls: 1, + prompt_tokens: tokens, + completion_tokens: 0, + cost: null, + }; +} + +const now = new Date(2026, 9, 23, 15, 0, 0).getTime(); +const end = now; +const start = startOfDay({ at: now }) - 2 * DayMs; + +describe("localDate", () => { + test("names the local calendar day, not the UTC one", () => { + /** + * Half an hour past local midnight is still the previous UTC day + * for anyone east of Greenwich, and the bug the chart had: the row + * was counted under one name and looked for under another. + */ + const at = new Date(2026, 9, 23, 0, 30, 0).getTime(); + + expect(localDate({ at })).toBe("2026-10-23"); + }); + + test("pads a single-digit month and day", () => { + expect(localDate({ at: new Date(2026, 0, 5, 12, 0, 0).getTime() })).toBe("2026-01-05"); + }); +}); + +describe("bucketUsageDays", () => { + test("one bucket per day of the window, oldest first", () => { + const days = bucketUsageDays({ rows: [], start, end }); + + expect(days.length).toBe(3); + expect(days.map((day) => day.label)).toEqual( + [2, 1, 0].map((back) => localDate({ at: startOfDay({ at: now }) - back * DayMs })), + ); + }); + + test("every row lands in its day, and yesterday's late call does not vanish", () => { + const today = localDate({ at: startOfDay({ at: now }) }); + const yesterday = localDate({ at: startOfDay({ at: now }) - DayMs }); + + const days = bucketUsageDays({ + rows: [row(today, 1500), row(yesterday, 400)], + start, + end, + }); + + const by_label = new Map(days.map((day) => [day.label, day.total_tokens])); + + expect(by_label.get(today)).toBe(1500); + expect(by_label.get(yesterday)).toBe(400); + expect(days.reduce((total, day) => total + day.total_tokens, 0)).toBe(1900); + }); + + test("a day with nothing in it is still a bucket, at zero", () => { + const days = bucketUsageDays({ + rows: [row(localDate({ at: startOfDay({ at: now }) }), 100)], + start, + end, + }); + + expect(days.filter((day) => day.total_tokens === 0).length).toBe(2); + expect(days.every((day) => day.calls === 0 || day.calls === 1)).toBe(true); + }); + + test("a bucket's midnight is that day's local midnight", () => { + const days = bucketUsageDays({ rows: [], start, end }); + + for (const day of days) { + expect(day.at).toBe(startOfDay({ at: day.at })); + expect(localDate({ at: day.at })).toBe(day.label); + } + }); + + test("cost and calls are summed, not counted twice", () => { + const label = localDate({ at: startOfDay({ at: now }) }); + + const days = bucketUsageDays({ + rows: [ + { ...row(label, 100), calls: 3, cost: 0.25 }, + { ...row(label, 50), calls: 2, cost: 0.5 }, + ], + start, + end, + }); + + const bucket = days.find((day) => day.label === label); + + expect(bucket?.total_tokens).toBe(150); + expect(bucket?.cost).toBe(0.75); + expect(days.reduce((total, day) => total + day.calls, 0)).toBe(5); + }); +}); diff --git a/src/functions/backend/db/usage/bucket-usage-days.ts b/src/functions/backend/db/usage/bucket-usage-days.ts new file mode 100644 index 0000000..b729905 --- /dev/null +++ b/src/functions/backend/db/usage/bucket-usage-days.ts @@ -0,0 +1,115 @@ +import { DayMs, startOfDay } from "./grab-usage-window"; +import { toSummary } from "./grab-usage-totals"; +import type { TokenUsageSummary } from "@/src/harness/usage/types"; + +type Row = { + /** + * The day the call was made, `YYYY-MM-DD`, already in local time. + */ + local_day: string; + calls: number | null; + prompt_tokens: number | null; + completion_tokens: number | null; + cost: number | null; +}; + +type Params = { + rows: Row[]; + start: number; + end: number; +}; + +export type UsageDay = { + /** + * The day's local midnight (epoch ms). + */ + at: number; + /** + * `YYYY-MM-DD`, in local time. + */ + label: string; +} & TokenUsageSummary; + +/** + * The rows SQL grouped by local day, laid onto one bucket per day of the + * window, oldest first. Days with nothing in them are still there, at + * zero, so the bars keep their spacing, and every row lands in a bucket: + * a day outside the window would be a bug in the query's own bounds. + */ +export default function bucketUsageDays({ rows, start, end }: Params): UsageDay[] { + const by_day = new Map(); + + for (const row of rows) { + /** + * Accumulated, not replaced: the query groups by day, so a repeated + * day means someone asked for two sets of rows, and the later one + * must add to the earlier rather than hide it. + */ + const label = String(row.local_day); + const summary = toSummary({ row }); + + by_day.set(label, mergeSummaries({ into: by_day.get(label), summary })); + } + + const days: UsageDay[] = []; + const last = startOfDay({ at: end }); + + for (let at = startOfDay({ at: start }); at <= last; at += DayMs) { + const label = localDate({ at }); + + days.push({ + at, + label, + ...(by_day.get(label) || emptySummary()), + }); + } + + return days; +} + +/** + * The day a local midnight falls on, in local time. + * + * Never `toISOString`: an epoch millisecond read in UTC is a different + * calendar day for anyone east or west of Greenwich, and a call counted + * under one day would then look for a bucket named after another and + * disappear from the chart (which is exactly what it did before). + */ +export function localDate({ at }: { at: number }) { + const date = new Date(at); + + return [ + date.getFullYear(), + String(date.getMonth() + 1).padStart(2, "0"), + String(date.getDate()).padStart(2, "0"), + ].join("-"); +} + +function emptySummary(): TokenUsageSummary { + return { prompt_tokens: 0, completion_tokens: 0, total_tokens: 0, calls: 0 }; +} + +/** + * One summary added onto another. A day that reported no cost still costs + * nothing; only a real cost is carried, so an unpriced day doesn't turn a + * priced one into zero. + */ +function mergeSummaries({ + into, + summary, +}: { + into?: TokenUsageSummary; + summary: TokenUsageSummary; +}): TokenUsageSummary { + const prompt_tokens = (into?.prompt_tokens || 0) + summary.prompt_tokens; + const completion_tokens = (into?.completion_tokens || 0) + summary.completion_tokens; + const cost = (into?.cost || 0) + (summary.cost || 0); + + return { + prompt_tokens, + completion_tokens, + total_tokens: prompt_tokens + completion_tokens, + ...(cost > 0 ? { cost } : {}), + calls: (into?.calls || 0) + summary.calls, + }; +} diff --git a/src/functions/backend/db/usage/grab-message-usage.ts b/src/functions/backend/db/usage/grab-message-usage.ts new file mode 100644 index 0000000..40c8309 --- /dev/null +++ b/src/functions/backend/db/usage/grab-message-usage.ts @@ -0,0 +1,56 @@ +import BunSQLite from "@moduletrace/bun-sqlite"; +import type { TokenUsageSummary } from "@/src/harness/usage/types"; + +type Params = { + /** + * The messages whose calls are summed. A message with no calls is + * absent from the answer. + */ + message_ids: number[]; +}; + +/** + * What each of these messages cost, summed over every model call it took + * (chat turns, media readings, image generations). This is what a + * message's token line shows. + */ +export default async function grabMessageUsage({ message_ids }: Params) { + const totals = new Map(); + const ids = [...new Set(message_ids.map(Number).filter((id) => id > 0))]; + + if (!ids.length) return totals; + + const res = await BunSQLite.sql<{ + message_id: number; + prompt_tokens: number | null; + completion_tokens: number | null; + cost: number | null; + calls: number; + }>({ + sql: `SELECT message_id, + SUM(prompt_tokens) AS prompt_tokens, + SUM(completion_tokens) AS completion_tokens, + SUM(cost) AS cost, + COUNT(*) AS calls + FROM token_usages + WHERE message_id IN (${ids.map(() => "?").join(", ")}) + GROUP BY message_id`, + values: ids, + }); + + for (const row of res.payload || []) { + const prompt_tokens = Number(row.prompt_tokens) || 0; + const completion_tokens = Number(row.completion_tokens) || 0; + const cost = Number(row.cost); + + totals.set(Number(row.message_id), { + prompt_tokens, + completion_tokens, + total_tokens: prompt_tokens + completion_tokens, + ...(Number.isFinite(cost) && cost > 0 ? { cost } : {}), + calls: Number(row.calls) || 0, + }); + } + + return totals; +} diff --git a/src/functions/backend/db/usage/grab-usage-days.ts b/src/functions/backend/db/usage/grab-usage-days.ts new file mode 100644 index 0000000..36f5cdb --- /dev/null +++ b/src/functions/backend/db/usage/grab-usage-days.ts @@ -0,0 +1,42 @@ +import BunSQLite from "@moduletrace/bun-sqlite"; +import bucketUsageDays, { type UsageDay } from "./bucket-usage-days"; + +export type { UsageDay }; + +type Params = { + start: number; + end: number; +}; + +/** + * One bucket per day in the window, oldest first, each the tokens spent + * that day. Days with nothing in them are still there, at zero, so the + * bars keep their spacing. + * + * SQLite has no timezone support, so the day is taken with its `localtime` + * modifier, which reads the same server timezone the page's own day + * boundaries come from. Grouping by the bare UTC day instead put a late + * evening's calls in the previous day's bucket, and dropped them whenever + * no bucket carried that name. + */ +export default async function grabUsageDays({ start, end }: Params): Promise { + const res = await BunSQLite.sql<{ + local_day: string; + calls: number | null; + prompt_tokens: number | null; + completion_tokens: number | null; + cost: number | null; + }>({ + sql: `SELECT date(created_at / 1000, 'unixepoch', 'localtime') AS local_day, + COUNT(*) AS calls, + SUM(prompt_tokens) AS prompt_tokens, + SUM(completion_tokens) AS completion_tokens, + SUM(cost) AS cost + FROM token_usages + WHERE created_at >= ? AND created_at < ? + GROUP BY local_day`, + values: [start, end], + }); + + return bucketUsageDays({ rows: res.payload || [], start, end }); +} diff --git a/src/functions/backend/db/usage/grab-usage-models.ts b/src/functions/backend/db/usage/grab-usage-models.ts new file mode 100644 index 0000000..893db3e --- /dev/null +++ b/src/functions/backend/db/usage/grab-usage-models.ts @@ -0,0 +1,75 @@ +import BunSQLite from "@moduletrace/bun-sqlite"; +import { toSummary } from "./grab-usage-totals"; +import type { TokenUsageSummary } from "@/src/harness/usage/types"; + +type Params = { + start: number; + end: number; + /** + * How many the page shows, busiest first. + */ + limit: number; +}; + +export type UsageModelRow = { + /** + * The provider's own id for the model, as the call recorded it + * (`deepseek/deepseek-chat`). This is what a person compares, not the + * catalog row's numeric id. + */ + model_slug: string; + /** + * The catalog's display name, when the model is still listed. A model + * a provider stopped listing keeps its slug and loses its name. + */ + model_name: string | null; + provider_name: string; + provider_id: number | null; + harness: string; +} & TokenUsageSummary; + +/** + * Every model the period ran, busiest first (by tokens), with what it + * spent. This is how one works out what a model costs: its tokens against + * its calls, for the window asked for. + */ +export default async function grabUsageModels({ start, end, limit }: Params): Promise { + const res = await BunSQLite.sql<{ + model_slug: string | null; + model_name: string | null; + provider_name: string | null; + provider_id: number | null; + harness: string | null; + calls: number | null; + prompt_tokens: number | null; + completion_tokens: number | null; + cost: number | null; + }>({ + sql: `SELECT u.model_slug, + m.name AS model_name, + u.provider_name, + u.provider_id, + u.harness, + COUNT(*) AS calls, + SUM(u.prompt_tokens) AS prompt_tokens, + SUM(u.completion_tokens) AS completion_tokens, + SUM(u.cost) AS cost + FROM token_usages u + LEFT JOIN harness_models m + ON m.provider_id = u.provider_id AND m.slug = u.model_slug + WHERE u.created_at >= ? AND u.created_at < ? + GROUP BY u.provider_name, u.model_slug, u.harness + ORDER BY SUM(u.prompt_tokens) + SUM(u.completion_tokens) DESC + LIMIT ?`, + values: [start, end, limit], + }); + + return (res.payload || []).map((row) => ({ + model_slug: row.model_slug || "unknown", + model_name: row.model_name || null, + provider_name: row.provider_name || "unknown", + provider_id: row.provider_id == null ? null : Number(row.provider_id), + harness: String(row.harness || "text"), + ...toSummary({ row }), + })); +} diff --git a/src/functions/backend/db/usage/grab-usage-overview.ts b/src/functions/backend/db/usage/grab-usage-overview.ts new file mode 100644 index 0000000..339c03e --- /dev/null +++ b/src/functions/backend/db/usage/grab-usage-overview.ts @@ -0,0 +1,53 @@ +import { AppData } from "@/src/data/app-data"; +import type { UsageOverview } from "@/src/types"; +import grabUsageDays from "./grab-usage-days"; +import grabUsageModels from "./grab-usage-models"; +import grabUsageTotals from "./grab-usage-totals"; +import grabUsageWindow, { DayMs } from "./grab-usage-window"; + +type Params = { + /** + * How many days back to total, today included. + */ + days?: number | string | null; + now?: number; +}; + +/** + * Everything the usage page shows for one window: the totals, the same day + * by day, per harness and per model, plus the day before the window and a + * rolling week, so the page can say whether work is going up or down. + */ +export default async function grabUsageOverview({ + days, + now = Date.now(), +}: Params): Promise { + const window = grabUsageWindow({ days, now }); + const previous_start = window.start - window.days * DayMs; + + const [totals, days_by_day, models, previous, week] = await Promise.all([ + grabUsageTotals({ start: window.start, end: window.end }), + grabUsageDays({ start: window.start, end: window.end }), + grabUsageModels({ + start: window.start, + end: window.end, + limit: AppData.UsageModelPageSize, + }), + grabUsageTotals({ start: previous_start, end: window.start }), + grabUsageTotals({ start: window.end - 7 * DayMs, end: window.end }), + ]); + + return { + window: { + days: window.days, + start: window.start, + end: window.end, + }, + totals: totals.totals, + by_harness: totals.by_harness, + days: days_by_day, + models, + previous: previous.totals, + week: week.totals, + }; +} diff --git a/src/functions/backend/db/usage/grab-usage-owner.ts b/src/functions/backend/db/usage/grab-usage-owner.ts new file mode 100644 index 0000000..72a6f59 --- /dev/null +++ b/src/functions/backend/db/usage/grab-usage-owner.ts @@ -0,0 +1,47 @@ +import BunSQLite from "@moduletrace/bun-sqlite"; + +type Params = { + /** + * The bot whose file it is, or the bot the search ran for. + */ + bot_id?: number | string | null; + /** + * The workspace whose file it is, when it shares one. + */ + workspace_id?: number | string | null; +}; + +/** + * Whose usage a call made outside anyone's run belongs to: a file ingested + * by the queue, or a search that ran with no conversation behind it. The + * workspace's creator when there is a workspace, else the bot's, so an + * embedding never goes uncounted. + */ +export default async function grabUsageOwnerUserId({ + bot_id, + workspace_id, +}: Params): Promise { + const workspace = Number(workspace_id); + + if (Number.isFinite(workspace) && workspace > 0) { + const row = await BunSQLite.sql<{ user_id: number }>({ + sql: `SELECT user_id FROM workspaces WHERE id = ? LIMIT 1`, + values: [workspace], + }).then((res) => res.payload?.[0]); + + if (row?.user_id) return Number(row.user_id) || null; + } + + const bot = Number(bot_id); + + if (Number.isFinite(bot) && bot > 0) { + const row = await BunSQLite.sql<{ user_id: number }>({ + sql: `SELECT user_id FROM bots WHERE id = ? LIMIT 1`, + values: [bot], + }).then((res) => res.payload?.[0]); + + if (row?.user_id) return Number(row.user_id) || null; + } + + return null; +} diff --git a/src/functions/backend/db/usage/grab-usage-totals.ts b/src/functions/backend/db/usage/grab-usage-totals.ts new file mode 100644 index 0000000..75bea03 --- /dev/null +++ b/src/functions/backend/db/usage/grab-usage-totals.ts @@ -0,0 +1,97 @@ +import BunSQLite from "@moduletrace/bun-sqlite"; +import type { TokenUsageSummary } from "@/src/harness/usage/types"; + +type Params = { + /** + * The window's start and end (epoch ms), from `grabUsageWindow`. + */ + start: number; + end: number; +}; + +type UsageRow = { + calls: number | null; + prompt_tokens: number | null; + completion_tokens: number | null; + cost: number | null; +}; + +/** + * One total for a whole period, and the same split by the harnesses the + * calls were for. + */ +export type UsageTotals = { + totals: TokenUsageSummary; + by_harness: (TokenUsageSummary & { harness: string })[]; +}; + +/** + * What every model call in the period cost, in total and per harness + * (chat, media reading, transcribing, speaking, images, embeddings). + */ +export default async function grabUsageTotals({ start, end }: Params): Promise { + const res = await BunSQLite.sql({ + sql: `SELECT harness, + COUNT(*) AS calls, + SUM(prompt_tokens) AS prompt_tokens, + SUM(completion_tokens) AS completion_tokens, + SUM(cost) AS cost + FROM token_usages + WHERE created_at >= ? AND created_at < ? + GROUP BY harness`, + values: [start, end], + }); + + const by_harness = (res.payload || []).map((row) => ({ + harness: String(row.harness || "text"), + ...toSummary({ row }), + })); + + return { + totals: sumSummary({ summaries: by_harness }), + by_harness: by_harness.sort((a, b) => b.total_tokens - a.total_tokens), + }; +} + +/** + * Adds summaries into one. A call that reported no tokens still counts as + * a call. + */ +export function sumSummary({ summaries }: { summaries: TokenUsageSummary[] }): TokenUsageSummary { + let prompt_tokens = 0; + let completion_tokens = 0; + let cost = 0; + let calls = 0; + + for (const summary of summaries) { + prompt_tokens += summary.prompt_tokens; + completion_tokens += summary.completion_tokens; + cost += summary.cost || 0; + calls += summary.calls; + } + + return { + prompt_tokens, + completion_tokens, + total_tokens: prompt_tokens + completion_tokens, + ...(cost > 0 ? { cost } : {}), + calls, + }; +} + +/** + * One grouped row (any shape with the four sums) as a summary. + */ +export function toSummary({ row }: { row: UsageRow }): TokenUsageSummary { + const prompt_tokens = Number(row.prompt_tokens) || 0; + const completion_tokens = Number(row.completion_tokens) || 0; + const cost = Number(row.cost); + + return { + prompt_tokens, + completion_tokens, + total_tokens: prompt_tokens + completion_tokens, + ...(Number.isFinite(cost) && cost > 0 ? { cost } : {}), + calls: Number(row.calls) || 0, + }; +} diff --git a/src/functions/backend/db/usage/grab-usage-window.ts b/src/functions/backend/db/usage/grab-usage-window.ts new file mode 100644 index 0000000..1f1e3c7 --- /dev/null +++ b/src/functions/backend/db/usage/grab-usage-window.ts @@ -0,0 +1,57 @@ +import { AppData } from "@/src/data/app-data"; + +type Params = { + /** + * How many days back the window starts, today included. Clamped to + * what the page offers. + */ + days?: number | string | null; + /** + * The moment "now" is, so a whole page (its totals and its days) is + * read at one instant. + */ + now?: number; +}; + +/** + * The period the usage page totals: a start, an end and the day buckets + * between them, oldest first, each a local midnight. + */ +export default function grabUsageWindow({ days, now = Date.now() }: Params) { + const asked = Number(days); + const count = + Number.isFinite(asked) && asked > 0 + ? Math.min(Math.round(asked), AppData.UsageMaxDays) + : AppData.UsageDefaultDays; + + const end = now; + /** + * Whole days back from the start of today, so every bucket is a day's + * work rather than a sliding window's slice. + */ + const today = startOfDay({ at: now }); + const start = today - (count - 1) * DayMs; + + return { + days: count, + start, + end, + /** + * One bucket per day in the window, oldest first. + */ + buckets: Array.from({ length: count }, (_, index) => start + index * DayMs), + }; +} + +export const DayMs = 24 * 60 * 60 * 1000; + +/** + * Local midnight of the day the moment falls in. + */ +export function startOfDay({ at }: { at: number }) { + const date = new Date(at); + + date.setHours(0, 0, 0, 0); + + return date.getTime(); +} diff --git a/src/harness/audio/speak-text.ts b/src/harness/audio/speak-text.ts index e5e3854..6ff359e 100644 --- a/src/harness/audio/speak-text.ts +++ b/src/harness/audio/speak-text.ts @@ -2,7 +2,9 @@ import grabHarnessSetting from "@/src/functions/backend/db/harness/grab-harness- import grabHarnessAdapter from "../providers/grab-harness-adapter"; import resolveHarnessCandidates from "../resolve/resolve-harness-candidates"; import withCandidates from "../resolve/with-candidates"; -import type { HarnessCandidate } from "../types"; +import type { HarnessAudioStream, HarnessCandidate } from "../types"; +import buildUsageCall from "../usage/build-usage-call"; +import recordHarnessUsage from "../usage/record-harness-usage"; type Params = { text: string; @@ -39,9 +41,22 @@ export default async function speakText({ grabHarnessSetting({ harness: "speech", bot_id, workspace_id }), ]); - const { result } = await withCandidates({ + const { result } = await withCandidates({ harness: "speech", candidates, + /** + * Speech is billed per character and streamed, so there is no + * completion count to read: the call is recorded as it is, with + * the text's length standing in for what was spoken. + */ + on_success: async ({ candidate }) => + await recordHarnessUsage({ + call: buildUsageCall({ + harness: "speech", + candidate, + usage: { prompt_tokens: text.length, completion_tokens: 0 }, + }), + }), run: async (candidate) => await grabHarnessAdapter({ kind: candidate.provider.kind }).speak({ candidate, diff --git a/src/harness/audio/transcribe-audio.ts b/src/harness/audio/transcribe-audio.ts index 73fb0ff..fbaaf8a 100644 --- a/src/harness/audio/transcribe-audio.ts +++ b/src/harness/audio/transcribe-audio.ts @@ -4,7 +4,9 @@ import HarnessError from "../harness-error"; import grabHarnessAdapter from "../providers/grab-harness-adapter"; import resolveHarnessCandidates from "../resolve/resolve-harness-candidates"; import withCandidates from "../resolve/with-candidates"; -import type { HarnessAudioInput } from "../types"; +import type { HarnessAudioInput, HarnessUsage } from "../types"; +import buildUsageCall from "../usage/build-usage-call"; +import recordHarnessUsage from "../usage/record-harness-usage"; type Params = { audio: HarnessAudioInput; @@ -46,15 +48,23 @@ export default async function transcribeAudio({ const final_signal = signal || new AbortController().signal; - const { result } = await withCandidates({ + const { result } = await withCandidates<{ text: string; usage?: HarnessUsage }>({ harness: "transcription", candidates, + on_success: async ({ candidate, result: transcript }): Promise => + await recordHarnessUsage({ + call: buildUsageCall({ + harness: "transcription", + candidate, + usage: transcript.usage, + }), + }), run: async (candidate) => { const adapter = grabHarnessAdapter({ kind: candidate.provider.kind, }); - const { text } = await adapter.transcribe({ + const { text, usage } = await adapter.transcribe({ candidate, audio, language, @@ -69,9 +79,9 @@ export default async function transcribeAudio({ }); } - return text.trim(); + return { text: text.trim(), usage }; }, }); - return result; + return result.text; } diff --git a/src/harness/embeddings/embed-text.ts b/src/harness/embeddings/embed-text.ts index da98f3e..8caffdf 100644 --- a/src/harness/embeddings/embed-text.ts +++ b/src/harness/embeddings/embed-text.ts @@ -1,6 +1,9 @@ import grabHarnessAdapter from "../providers/grab-harness-adapter"; import resolveHarnessCandidates from "../resolve/resolve-harness-candidates"; import withCandidates from "../resolve/with-candidates"; +import type { HarnessEmbedResult } from "../types"; +import buildUsageCall from "../usage/build-usage-call"; +import recordHarnessUsage from "../usage/record-harness-usage"; import fitVectorDimensions from "./fit-vector-dimensions"; type Params = { @@ -52,9 +55,17 @@ export default async function embedText({ workspace_id, }); - const { result } = await withCandidates({ + const { result } = await withCandidates({ harness: "embedding", candidates, + on_success: async ({ candidate, result: embedded }): Promise => + await recordHarnessUsage({ + call: buildUsageCall({ + harness: "embedding", + candidate, + usage: embedded.usage, + }), + }), run: async (candidate) => await grabHarnessAdapter({ kind: candidate.provider.kind }).embed({ candidate, diff --git a/src/harness/image/generate-image.ts b/src/harness/image/generate-image.ts index 9d46e8d..20dca7d 100644 --- a/src/harness/image/generate-image.ts +++ b/src/harness/image/generate-image.ts @@ -1,6 +1,9 @@ import grabHarnessAdapter from "../providers/grab-harness-adapter"; import resolveHarnessCandidates from "../resolve/resolve-harness-candidates"; import withCandidates from "../resolve/with-candidates"; +import type { HarnessImageOutput, HarnessUsage } from "../types"; +import buildUsageCall from "../usage/build-usage-call"; +import recordHarnessUsage from "../usage/record-harness-usage"; type Params = { prompt: string; @@ -35,9 +38,17 @@ export default async function generateImage({ }: Params) { const candidates = await resolveHarnessCandidates({ harness: "image", bot_id, workspace_id }); - const { result } = await withCandidates({ + const { result } = await withCandidates<{ images: HarnessImageOutput[]; usage?: HarnessUsage }>({ harness: "image", candidates, + on_success: async ({ candidate, result: made }): Promise => + await recordHarnessUsage({ + call: buildUsageCall({ + harness: "image", + candidate, + usage: made.usage, + }), + }), run: async (candidate) => await grabHarnessAdapter({ kind: candidate.provider.kind, diff --git a/src/harness/media/read-image.ts b/src/harness/media/read-image.ts index 9cc1a7b..fb63bbd 100644 --- a/src/harness/media/read-image.ts +++ b/src/harness/media/read-image.ts @@ -3,7 +3,10 @@ import HarnessError from "../harness-error"; import grabHarnessAdapter from "../providers/grab-harness-adapter"; import resolveHarnessCandidates from "../resolve/resolve-harness-candidates"; import withCandidates from "../resolve/with-candidates"; +import type { HarnessUsage } from "../types"; import grabChatParams from "../text/grab-chat-params"; +import buildUsageCall from "../usage/build-usage-call"; +import recordHarnessUsage from "../usage/record-harness-usage"; type Params = { /** @@ -42,9 +45,17 @@ export default async function readImage({ url, focus, signal, bot_id, workspace_ workspace_id, }); - const { result } = await withCandidates({ + const { result } = await withCandidates<{ text: string; usage?: HarnessUsage }>({ harness: "vision", candidates, + on_success: async ({ candidate, result: answer }): Promise => + await recordHarnessUsage({ + call: buildUsageCall({ + harness: "vision", + candidate, + usage: answer.usage, + }), + }), run: async (candidate) => { const answer = await grabHarnessAdapter({ kind: candidate.provider.kind, @@ -83,9 +94,9 @@ export default async function readImage({ url, focus, signal, bot_id, workspace_ }); } - return answer.text.trim(); + return { text: answer.text.trim(), usage: answer.usage }; }, }); - return result; + return result.text; } diff --git a/src/harness/providers/openai-compatible/embed.ts b/src/harness/providers/openai-compatible/embed.ts index c581b94..56355ba 100644 --- a/src/harness/providers/openai-compatible/embed.ts +++ b/src/harness/providers/openai-compatible/embed.ts @@ -13,6 +13,11 @@ const embed: HarnessAdapter["embed"] = async ({ candidate, input, signal }) => { const { provider, model } = candidate; const preset = grabProviderPreset({ kind: provider.kind }); const vectors: number[][] = []; + /** + * What the provider said the batch cost. Most report it once per + * request; the rows are opened before the loop so each batch counts. + */ + let prompt_tokens = 0; /** * Every OpenAI-compatible API puts embeddings at `/embeddings`. @@ -47,6 +52,8 @@ const embed: HarnessAdapter["embed"] = async ({ candidate, input, signal }) => { }); } + prompt_tokens += Number(json?.usage?.prompt_tokens) || 0; + for (const item of items) { const vector = Array.isArray(item) ? item : item?.embedding; @@ -62,7 +69,12 @@ const embed: HarnessAdapter["embed"] = async ({ candidate, input, signal }) => { } } - return { vectors }; + return { + vectors, + ...(prompt_tokens + ? { usage: { prompt_tokens, completion_tokens: 0 } } + : {}), + }; }; export default embed; diff --git a/src/harness/providers/openai-compatible/generate-image.ts b/src/harness/providers/openai-compatible/generate-image.ts index 006bd12..1ed9f16 100644 --- a/src/harness/providers/openai-compatible/generate-image.ts +++ b/src/harness/providers/openai-compatible/generate-image.ts @@ -81,7 +81,20 @@ const generateImage: HarnessAdapter["generateImage"] = async ({ }); } - return { images }; + /** + * Image models bill per image, not per token: only a provider that + * volunteers a usage figure is recorded with one. + */ + const usage = + json?.usage && typeof json.usage == "object" + ? { + prompt_tokens: Number(json.usage.prompt_tokens) || 0, + completion_tokens: Number(json.usage.completion_tokens) || 0, + ...(typeof json.usage.cost == "number" ? { cost: json.usage.cost } : {}), + } + : undefined; + + return { images, ...(usage ? { usage } : {}) }; }; export default generateImage; diff --git a/src/harness/providers/openai-compatible/transcribe.ts b/src/harness/providers/openai-compatible/transcribe.ts index 88523df..7cd3d11 100644 --- a/src/harness/providers/openai-compatible/transcribe.ts +++ b/src/harness/providers/openai-compatible/transcribe.ts @@ -50,7 +50,22 @@ const transcribe: HarnessAdapter["transcribe"] = async ({ }); } - return { text: String(json?.text || "").trim() }; + /** + * Providers that bill by audio seconds sometimes report a token count; + * the OpenAI shape puts it under `usage`. Left out when absent, so the + * row records that the call happened without a token count. + */ + const usage = + json?.usage && typeof json.usage == "object" + ? { + prompt_tokens: Number(json.usage.prompt_tokens ?? json.usage.input_tokens) || 0, + completion_tokens: + Number(json.usage.completion_tokens ?? json.usage.output_tokens) || 0, + ...(typeof json.usage.cost == "number" ? { cost: json.usage.cost } : {}), + } + : undefined; + + return { text: String(json?.text || "").trim(), ...(usage ? { usage } : {}) }; }; export default transcribe; diff --git a/src/harness/resolve/with-candidates.ts b/src/harness/resolve/with-candidates.ts index 6d03274..714e2ad 100644 --- a/src/harness/resolve/with-candidates.ts +++ b/src/harness/resolve/with-candidates.ts @@ -14,6 +14,15 @@ type Params = { * uses, so it doesn't switch models mid-task without a reason. */ prefer_key?: string; + /** + * Called with what the candidate that worked returned, before the call + * comes back to the caller: where a harness writes its usage down. + * Never fails the call (the caller passes a recorder that can't throw). + */ + on_success?: (params: { + candidate: HarnessCandidate; + result: T; + }) => Promise | void; cooldowns?: HarnessCooldowns; now?: () => number; }; @@ -42,6 +51,7 @@ export default async function withCandidates({ candidates, run, prefer_key, + on_success, cooldowns = DefaultCooldowns, now = Date.now, }: Params): Promise<{ result: T; candidate: HarnessCandidate }> { @@ -93,7 +103,11 @@ export default async function withCandidates({ if (!isReady(candidate)) continue; try { - return { result: await run(candidate), candidate }; + const result = await run(candidate); + + await on_success?.({ candidate, result }); + + return { result, candidate }; } catch (error: unknown) { if (!(error instanceof HarnessError)) throw error; if (error.code == "aborted") throw error; diff --git a/src/harness/text/generate-text.ts b/src/harness/text/generate-text.ts index 4c0e0c1..16cfcb5 100644 --- a/src/harness/text/generate-text.ts +++ b/src/harness/text/generate-text.ts @@ -8,6 +8,8 @@ import type { HarnessMessage, HarnessToolDefinition, } from "../types"; +import buildUsageCall from "../usage/build-usage-call"; +import recordHarnessUsage from "../usage/record-harness-usage"; import grabChatParams from "./grab-chat-params"; import prepareMessagesForModel from "./prepare-messages-for-model"; import stripThinkTags from "./strip-think-tags"; @@ -47,6 +49,14 @@ export type GenerateTextResult = HarnessChatResult & { candidate: HarnessCandidate; }; +/** + * One turn as `withCandidates` carries it: the answer the provider gave, + * with the reasoning tags stripped and an empty answer turned into null. + */ +type ChatAnswer = Omit & { + text: string | null; +}; + /** * One model turn on the text harness, falling back across its models. * Media the model can't take is read by the helper harnesses first. @@ -71,10 +81,18 @@ export default async function generateText({ ? undefined : await grabChatParams({ harness: "text", bot_id, workspace_id }); - const { result, candidate } = await withCandidates({ + const { result, candidate } = await withCandidates({ harness: "text", candidates: resolved, prefer_key, + on_success: async ({ candidate: answered, result: answer }): Promise => + await recordHarnessUsage({ + call: buildUsageCall({ + harness: "text", + candidate: answered, + usage: answer.usage, + }), + }), run: async (candidate) => { const answer = await grabHarnessAdapter({ kind: candidate.provider.kind, diff --git a/src/harness/types.ts b/src/harness/types.ts index 6ad4a29..797175b 100644 --- a/src/harness/types.ts +++ b/src/harness/types.ts @@ -138,6 +138,18 @@ export type HarnessUsage = { cost?: number; }; +/** + * What a harness does with a model, for its usage rows and the totals + * that read them. Mirrors the `harness` column of `token_usages`. + */ +export type HarnessUsageKind = + | "text" + | "vision" + | "transcription" + | "speech" + | "image" + | "embedding"; + export type HarnessChatParams = { candidate: HarnessCandidate; messages: HarnessMessage[]; @@ -238,11 +250,22 @@ export type HarnessEmbedResult = { */ export type HarnessAdapter = { chat: (params: HarnessChatParams) => Promise; - transcribe: (params: HarnessTranscribeParams) => Promise<{ text: string }>; + /** + * `usage` when the provider says how many tokens it heard, which + * OpenAI-style speech-to-text endpoints do only sometimes. + */ + transcribe: (params: HarnessTranscribeParams) => Promise<{ + text: string; + usage?: HarnessUsage; + }>; speak: (params: HarnessSpeakParams) => Promise; + /** + * Image models report what they made rather than how many tokens it + * took, so `usage` is only whatever the provider volunteers. + */ generateImage: ( params: HarnessImageParams, - ) => Promise<{ images: HarnessImageOutput[] }>; + ) => Promise<{ images: HarnessImageOutput[]; usage?: HarnessUsage }>; embed: (params: HarnessEmbedParams) => Promise; listModels: (params: { provider: HarnessProviderConfig; diff --git a/src/harness/usage/build-usage-call.test.ts b/src/harness/usage/build-usage-call.test.ts new file mode 100644 index 0000000..f0e1f75 --- /dev/null +++ b/src/harness/usage/build-usage-call.test.ts @@ -0,0 +1,102 @@ +import { describe, expect, test } from "bun:test"; +import type { HarnessCandidate } from "../types"; +import buildUsageCall from "./build-usage-call"; + +function model(overrides: { row_id?: number } = {}) { + return { + slug: "deepseek/deepseek-chat", + name: "DeepSeek Chat", + input_modalities: ["text"], + output_modalities: ["text"], + supports_tools: true, + ...overrides, + } as HarnessCandidate["model"]; +} + +function candidate(overrides: Partial = {}): HarnessCandidate { + return { + provider: { + id: 2, + name: "OpenRouter", + kind: "openrouter", + } as HarnessCandidate["provider"], + key: "openrouter:deepseek/deepseek-chat", + source: "auto", + model: model({ row_id: 41 }), + ...overrides, + }; +} + +describe("buildUsageCall", () => { + test("names the candidate that really answered and what it spent", () => { + const call = buildUsageCall({ + harness: "text", + candidate: candidate(), + usage: { prompt_tokens: 1200, completion_tokens: 340, cost: 0.0042 }, + }); + + expect(call).toEqual({ + harness: "text", + provider_id: 2, + provider_name: "OpenRouter", + model_id: 41, + model_slug: "deepseek/deepseek-chat", + prompt_tokens: 1200, + completion_tokens: 340, + cost: 0.0042, + }); + }); + + test("a provider with no name reads as its kind", () => { + const call = buildUsageCall({ + harness: "image", + candidate: candidate({ + provider: { id: 4, name: "", kind: "openai" } as HarnessCandidate["provider"], + }), + usage: { prompt_tokens: 0, completion_tokens: 0 }, + }); + + expect(call.provider_name).toBe("openai"); + }); + + test("a provider that reports no usage still gets a row", () => { + const call = buildUsageCall({ harness: "transcription", candidate: candidate() }); + + expect(call.prompt_tokens).toBe(0); + expect(call.completion_tokens).toBe(0); + expect(call.cost).toBeUndefined(); + }); + + test("a candidate with no catalog row has no model id to point at", () => { + const call = buildUsageCall({ + harness: "text", + candidate: candidate({ + model: model({ slug: "made-up-model" } as any), + }), + usage: { prompt_tokens: 10, completion_tokens: 5 }, + }); + + expect(call.model_id).toBeUndefined(); + expect(call.model_slug).toBe("made-up-model"); + }); + + test("a cost the provider did not report is left out, not zeroed", () => { + const call = buildUsageCall({ + harness: "text", + candidate: candidate(), + usage: { prompt_tokens: 10, completion_tokens: 5, cost: Number.NaN }, + }); + + expect(call.cost).toBeUndefined(); + }); + + test("a free provider's zero cost is kept as zero", () => { + const call = buildUsageCall({ + harness: "text", + candidate: candidate(), + usage: { prompt_tokens: 10, completion_tokens: 5, cost: 0 }, + }); + + expect(call.cost).toBe(0); + }); +}); diff --git a/src/harness/usage/build-usage-call.ts b/src/harness/usage/build-usage-call.ts new file mode 100644 index 0000000..acfc194 --- /dev/null +++ b/src/harness/usage/build-usage-call.ts @@ -0,0 +1,37 @@ +import type { HarnessCandidate, HarnessUsage } from "../types"; +import type { HarnessUsageCall } from "./types"; + +type Params = { + harness: HarnessUsageCall["harness"]; + candidate: HarnessCandidate; + usage?: HarnessUsage; +}; + +/** + * The candidate that really answered a call, and what it reported, as the + * row `recordHarnessUsage` writes. Providers that report no usage at all + * still get a row (no tokens), so a call is never invisible. + */ +export default function buildUsageCall({ harness, candidate, usage }: Params): HarnessUsageCall { + return { + harness, + provider_id: candidate.provider.id, + provider_name: candidate.provider.name || candidate.provider.kind, + model_id: modelIdOf(candidate), + model_slug: candidate.model.slug, + prompt_tokens: Number(usage?.prompt_tokens) || 0, + completion_tokens: Number(usage?.completion_tokens) || 0, + cost: typeof usage?.cost == "number" && Number.isFinite(usage.cost) ? usage.cost : undefined, + }; +} + +/** + * The `harness_models` row id, when the candidate carries the catalog row + * it was resolved from (`withCandidates` doesn't drop it; a candidate made + * by hand may not have one). + */ +function modelIdOf(candidate: HarnessCandidate) { + const row_id = (candidate.model as { row_id?: number }).row_id; + + return typeof row_id == "number" ? row_id : undefined; +} diff --git a/src/harness/usage/harness-usage-scope.test.ts b/src/harness/usage/harness-usage-scope.test.ts new file mode 100644 index 0000000..bfe2845 --- /dev/null +++ b/src/harness/usage/harness-usage-scope.test.ts @@ -0,0 +1,106 @@ +import { describe, expect, test } from "bun:test"; +import { grabHarnessUsageScope, runWithHarnessUsageScope } from "./harness-usage-scope"; + +describe("harness usage scope", () => { + test("nothing is attributed outside a run", () => { + expect(grabHarnessUsageScope()).toBeUndefined(); + }); + + test("a run's scope is visible to everything it calls", async () => { + const scope = await runWithHarnessUsageScope({ + scope: { user_id: 7, bot_id: 3, session_id: 11 }, + task: async () => grabHarnessUsageScope(), + }); + + expect(scope).toEqual({ user_id: 7, bot_id: 3, session_id: 11 }); + }); + + test("a nested run keeps what the outer one knew", async () => { + const scope = await runWithHarnessUsageScope({ + scope: { user_id: 7, bot_id: 3, session_id: 11, job_id: 5 }, + task: () => + runWithHarnessUsageScope({ + scope: { message_id: 42 }, + task: async () => grabHarnessUsageScope(), + }), + }); + + expect(scope).toEqual({ + user_id: 7, + bot_id: 3, + session_id: 11, + job_id: 5, + message_id: 42, + }); + }); + + test("a nested run overrides only what it sets", async () => { + const scope = await runWithHarnessUsageScope({ + scope: { user_id: 7, bot_id: 3, workspace_id: 9 }, + task: () => + runWithHarnessUsageScope({ + scope: { bot_id: 4 }, + task: async () => grabHarnessUsageScope(), + }), + }); + + expect(scope).toEqual({ user_id: 7, bot_id: 4, workspace_id: 9 }); + }); + + test("an undefined field leaves the outer value alone", async () => { + const scope = await runWithHarnessUsageScope({ + scope: { user_id: 7, bot_id: 3, job_id: 5 }, + task: () => + runWithHarnessUsageScope({ + scope: { job_id: undefined, message_id: 42 }, + task: async () => grabHarnessUsageScope(), + }), + }); + + expect(scope).toEqual({ user_id: 7, bot_id: 3, job_id: 5, message_id: 42 }); + }); + + test("a null field clears the outer value, since it means 'none'", async () => { + const scope = await runWithHarnessUsageScope({ + scope: { user_id: 7, bot_id: 3, workspace_id: 9, job_id: 5 }, + task: () => + runWithHarnessUsageScope({ + scope: { workspace_id: null, job_id: null }, + task: async () => grabHarnessUsageScope(), + }), + }); + + expect(scope).toEqual({ user_id: 7, bot_id: 3, workspace_id: null, job_id: null }); + }); + + test("the scope is gone once the run returns", async () => { + await runWithHarnessUsageScope({ + scope: { user_id: 7 }, + task: async () => undefined, + }); + + expect(grabHarnessUsageScope()).toBeUndefined(); + }); + + test("two concurrent runs never see each other's scope", async () => { + const [first, second] = await Promise.all([ + runWithHarnessUsageScope({ + scope: { user_id: 1 }, + task: async () => { + await Bun.sleep(5); + return grabHarnessUsageScope(); + }, + }), + runWithHarnessUsageScope({ + scope: { user_id: 2 }, + task: async () => { + await Bun.sleep(1); + return grabHarnessUsageScope(); + }, + }), + ]); + + expect(first).toEqual({ user_id: 1 }); + expect(second).toEqual({ user_id: 2 }); + }); +}); diff --git a/src/harness/usage/harness-usage-scope.ts b/src/harness/usage/harness-usage-scope.ts new file mode 100644 index 0000000..2ea4a71 --- /dev/null +++ b/src/harness/usage/harness-usage-scope.ts @@ -0,0 +1,63 @@ +import { AsyncLocalStorage } from "node:async_hooks"; +import type { HarnessUsageScope } from "./types"; + +/** + * The attribution a run's model calls carry. + * + * A bot's run is entered once (`runWithHarnessUsageScope`) and everything + * it calls underneath - the text loop's turns, a session summary, a + * transcription, an image, an embedding - is written down for that user, + * bot, session and job without threading the ids through every harness + * function. One of those calls is itself entered by a `run` that has to + * swap the scope it reports under (`withHarnessUsageScope`): the session + * namer runs outside the job's scope, on purpose, since the job it named + * is already complete. + * + * The store is merged, not replaced: a call that only knows the message id + * (the text loop attaching a turn's usage) keeps the run's user, bot and + * job. + */ +const storage = new AsyncLocalStorage(); + +/** + * Runs `task` with `scope` as its attribution. Ambient values are merged + * under it, so a nested call keeps what the outer one set and overrides + * only what it knows better (null clears a field). + */ +export async function runWithHarnessUsageScope( + { scope, task }: { scope: HarnessUsageScope; task: () => Promise }, +): Promise { + return await storage.run(mergeScopes({ base: storage.getStore(), overlay: scope }), task); +} + +/** + * The attribution a call right now carries, if any. + */ +export function grabHarnessUsageScope(): HarnessUsageScope | undefined { + return storage.getStore(); +} + +function mergeScopes({ + base, + overlay, +}: { + base?: HarnessUsageScope; + overlay: HarnessUsageScope; +}): HarnessUsageScope { + const merged: HarnessUsageScope = { ...base }; + + for (const [key, value] of Object.entries(overlay) as [ + keyof HarnessUsageScope, + HarnessUsageScope[keyof HarnessUsageScope], + ][]) { + if (value === undefined) continue; + + /** + * A null is a real value here: it says "none", e.g. a bot's own + * chat has no workspace, and must clear what the outer scope had. + */ + (merged as Record)[key] = value; + } + + return merged; +} diff --git a/src/harness/usage/record-harness-usage.ts b/src/harness/usage/record-harness-usage.ts new file mode 100644 index 0000000..957de11 --- /dev/null +++ b/src/harness/usage/record-harness-usage.ts @@ -0,0 +1,92 @@ +import BunSQLite from "@moduletrace/bun-sqlite"; +import type { + BUN_SQLITE_COBBLE_TOKEN_USAGES, + BunSQLiteTables, +} from "@/db/types/db"; +import { logError } from "@/src/functions/backend/logger"; +import { grabHarnessUsageScope } from "./harness-usage-scope"; +import type { HarnessUsageCall, HarnessUsageScope } from "./types"; + +type Params = { + /** + * The scope the call ran under. Left out, the ambient one + * (`grabHarnessUsageScope`) is used, which is what every entry point + * does: the ids came in through `runWithHarnessUsageScope`. + */ + scope?: HarnessUsageScope; + /** + * Which harness and which model answered, with what the provider + * reported it cost. + */ + call: HarnessUsageCall; +}; + +/** + * Writes one model call into `token_usages`: the tokens it spent, on which + * model and provider, for whom, and which message or job it was for. This + * is what a message's token line, a bot's and a provider's totals and the + * usage page all read. + * + * Never throws: a model that answered must not have its answer lost + * because the bookkeeping failed. A call without an owner (no scope, no + * user) is not written at all, since nothing could show it. + */ +export default async function recordHarnessUsage({ scope, call }: Params) { + const resolved = scope || grabHarnessUsageScope(); + const user_id = Number(resolved?.user_id) || null; + + if (!user_id) return; + + const row: BUN_SQLITE_COBBLE_TOKEN_USAGES = { + user_id, + bot_id: numberOrNull(resolved?.bot_id), + workspace_id: numberOrNull(resolved?.workspace_id), + session_id: numberOrNull(resolved?.session_id), + message_id: numberOrNull(resolved?.message_id), + job_id: numberOrNull(resolved?.job_id), + harness: call.harness, + provider_id: call.provider_id, + provider_name: call.provider_name, + model_id: call.model_id, + model_slug: call.model_slug, + prompt_tokens: Math.max(0, Math.round(call.prompt_tokens || 0)), + completion_tokens: Math.max(0, Math.round(call.completion_tokens || 0)), + cost: typeof call.cost == "number" && Number.isFinite(call.cost) ? call.cost : undefined, + }; + + try { + const res = await BunSQLite.insert< + BUN_SQLITE_COBBLE_TOKEN_USAGES, + (typeof BunSQLiteTables)[number] + >({ + table: "token_usages", + data: [row], + }); + + if (!res.success) { + throw new Error(res.msg || `The usage wasn't saved.`); + } + } catch (error: unknown) { + await logError({ + title: "Model usage not recorded", + error, + metadata: { + harness: call.harness, + model_slug: call.model_slug || null, + provider_name: call.provider_name || null, + user_id, + }, + }).catch(() => {}); + } +} + +/** + * SQLite stores NULL for what isn't given; `Number(undefined)` is NaN, + * which bun-sqlite would refuse. An absent id is `undefined`, not `null`: + * bun-sqlite's row types spell "no value" that way. + */ +function numberOrNull(value: unknown): number | undefined { + const number = Number(value); + + return Number.isFinite(number) && number > 0 ? number : undefined; +} diff --git a/src/harness/usage/run-with-fallback-scope.test.ts b/src/harness/usage/run-with-fallback-scope.test.ts new file mode 100644 index 0000000..d6a6ee0 --- /dev/null +++ b/src/harness/usage/run-with-fallback-scope.test.ts @@ -0,0 +1,81 @@ +import { describe, expect, test } from "bun:test"; +import { grabHarnessUsageScope, runWithHarnessUsageScope } from "./harness-usage-scope"; +import runWithFallbackScope from "./run-with-fallback-scope"; + +const owner = { user_id: 9, bot_id: 3, session_id: null, message_id: null, job_id: null }; + +describe("runWithFallbackScope", () => { + test("the enclosing run keeps the work it already owns", async () => { + const seen = await runWithHarnessUsageScope({ + scope: { user_id: 7, bot_id: 3, job_id: 5 }, + task: () => + runWithFallbackScope({ + scope: owner, + task: async () => grabHarnessUsageScope(), + }), + }); + + expect(seen).toEqual({ user_id: 7, bot_id: 3, job_id: 5 }); + }); + + test("work with no run is attributed to its owner", async () => { + const seen = await runWithFallbackScope({ + scope: owner, + task: async () => grabHarnessUsageScope(), + }); + + expect(seen).toEqual(owner); + }); + + test("nobody to charge leaves the work unattributed rather than failing", async () => { + const seen = await runWithFallbackScope({ + scope: null, + task: async () => grabHarnessUsageScope(), + }); + + expect(seen).toBeUndefined(); + }); + + test("a scope with no user is not an owner", async () => { + const seen = await runWithFallbackScope({ + scope: { ...owner, user_id: undefined }, + task: async () => grabHarnessUsageScope(), + }); + + expect(seen).toBeUndefined(); + }); + + test("work that must be charged on its own overrides the enclosing run", async () => { + const seen = await runWithHarnessUsageScope({ + scope: { user_id: 7, bot_id: 3, job_id: 5 }, + task: () => + runWithFallbackScope({ + scope: owner, + mode: "own", + task: async () => grabHarnessUsageScope(), + }), + }); + + expect(seen).toEqual(owner); + }); + + test("the work's result is passed through", async () => { + const result = await runWithFallbackScope({ + scope: owner, + task: async () => "the answer", + }); + + expect(result).toBe("the answer"); + }); + + test("a failure inside the work is not swallowed", async () => { + expect( + runWithFallbackScope({ + scope: owner, + task: async () => { + throw new Error("the provider refused"); + }, + }), + ).rejects.toThrow("the provider refused"); + }); +}); diff --git a/src/harness/usage/run-with-fallback-scope.ts b/src/harness/usage/run-with-fallback-scope.ts new file mode 100644 index 0000000..b0d332f --- /dev/null +++ b/src/harness/usage/run-with-fallback-scope.ts @@ -0,0 +1,41 @@ +import { grabHarnessUsageScope, runWithHarnessUsageScope } from "./harness-usage-scope"; +import type { HarnessUsageScope } from "./types"; + +type Params = { + /** + * What to attribute the work to when nothing already owns it: a file + * ingested by the queue has no run behind it, but its owner must still + * see what the embedding cost. Null when there is nobody to charge. + */ + scope: HarnessUsageScope | null; + task: () => Promise; + /** + * How the work is attributed: `enclosing` keeps whatever run it is in, + * `own` always reports under `scope` (the session namer, whose job is + * already complete and must not be charged to it). + */ + mode?: "enclosing" | "own"; +}; + +/** + * Runs work under an attribution the run may not have. Nothing is entered + * when a run is already in charge, so an ingest a bot asked for is charged + * to that bot's user and not to the file's owner. + */ +export default async function runWithFallbackScope({ + scope, + task, + mode = "enclosing", +}: Params): Promise { + if (mode == "enclosing" && grabHarnessUsageScope()?.user_id) { + return await task(); + } + + const user_id = Number(scope?.user_id); + + if (!Number.isFinite(user_id) || user_id <= 0 || user_id === null) { + return await task(); + } + + return await runWithHarnessUsageScope({ scope: { ...scope, user_id }, task }); +} diff --git a/src/harness/usage/types.ts b/src/harness/usage/types.ts new file mode 100644 index 0000000..c33289c --- /dev/null +++ b/src/harness/usage/types.ts @@ -0,0 +1,60 @@ +/** + * Who and what a model call was for. Every harness call that reports its + * usage is written down with this (`/api/auth/crud` no: `token_usages`), + * so a message, a session, a bot or a provider can be totaled later. + * + * It is deliberately shaped slightly differently from the table's + * columns: `user_id` is required here (a row without an owner can't be + * shown or totaled for anyone), the rest are optional. + */ +export type HarnessUsageScope = { + /** + * Whose work it was: the user whose conversation the call ran for. + */ + user_id?: number | string; + bot_id?: number | string | null; + workspace_id?: number | string | null; + session_id?: number | string | null; + /** + * The queued job it ran for, when it ran for one. + */ + job_id?: number | string | null; + /** + * The message it produced, for the ones that produce one. + */ + message_id?: number | string | null; +}; + +/** + * What the caller knows about the call beyond its scope: which harness + * asked, and which candidate answered. Both are filled in by the call's + * own entry point, never by the code that runs it. + */ +export type HarnessUsageCall = { + harness: "text" | "vision" | "transcription" | "speech" | "image" | "embedding"; + provider_id?: number; + provider_name?: string; + model_id?: number; + model_slug?: string; + prompt_tokens: number; + completion_tokens: number; + cost?: number; +}; + +/** + * A message's usage, as its thread shows it and as `token_usages` stores + * it: what one answer cost, summed over its model calls. + */ +export type TokenUsageSummary = { + prompt_tokens: number; + completion_tokens: number; + total_tokens: number; + /** + * In USD, when the provider reported it. + */ + cost?: number; + /** + * How many model calls it took. + */ + calls: number; +}; diff --git a/src/harness/usage/with-usage-scope.ts b/src/harness/usage/with-usage-scope.ts new file mode 100644 index 0000000..31f140f --- /dev/null +++ b/src/harness/usage/with-usage-scope.ts @@ -0,0 +1,19 @@ +import { runWithHarnessUsageScope } from "./harness-usage-scope"; +import type { HarnessUsageScope } from "./types"; + +/** + * Runs one call the app makes on a user's behalf inside the scope its + * usage is written down with (transcribing a dictation, speaking a + * message, previewing a voice). The bot's run is already inside a scope of + * its own, so this only adds what the run didn't know: no job, no message, + * and the session the call belongs to. + */ +export default async function withUsageScope({ + scope, + task, +}: { + scope: HarnessUsageScope; + task: () => Promise; +}): Promise { + return await runWithHarnessUsageScope({ scope, task }); +} diff --git a/src/layouts/bots/(dict)/rail-user-menu-dict.ts b/src/layouts/bots/(dict)/rail-user-menu-dict.ts index dbb33cf..0db07a9 100644 --- a/src/layouts/bots/(dict)/rail-user-menu-dict.ts +++ b/src/layouts/bots/(dict)/rail-user-menu-dict.ts @@ -1,4 +1,5 @@ import { LogOut, SlidersHorizontal, type LucideIcon } from "lucide-react"; +import type { BUN_SQLITE_COBBLE_USER_TYPES } from "@/db/types/db"; import { SettingsDefaultPath } from "@/src/dict/settings-pages-dict"; import { RailPages } from "@/src/dict/rail-pages-dict"; @@ -18,12 +19,16 @@ export type RailUserMenuItem = { * Shown to super admins only. */ super_admin_only?: boolean; + /** + * Who sees it, when that's an RBAC rule rather than super admins only. + */ + can_view?: (params: { user_types?: Pick[] | null }) => boolean; }; /** - * The popover items: the rail pages (routines, activity, help), settings, - * sign out. Nothing else lives in the rail: these destinations are only - * reachable from here. + * The popover items: the rail pages (routines, files, usage, activity, + * help), settings, sign out. Nothing else lives in the rail: these + * destinations are only reachable from here. */ export const RailUserMenuItems: RailUserMenuItem[] = [ ...RailPages, diff --git a/src/layouts/bots/(partials)/rail-pages-list.tsx b/src/layouts/bots/(partials)/rail-pages-list.tsx index 37eeb03..19f13fa 100644 --- a/src/layouts/bots/(partials)/rail-pages-list.tsx +++ b/src/layouts/bots/(partials)/rail-pages-list.tsx @@ -2,6 +2,7 @@ import { useContext } from "react"; import Divider from "@moduletrace/twui/layout/Divider"; import { RailPages } from "@/src/dict/rail-pages-dict"; import { AppContext } from "@/src/pages/__root"; +import canSeeRailItem from "@/src/utils/rbac/can-see-rail-item"; import isPassThroughUser from "@/src/utils/rbac/is-pass-through-user"; import RailBackLink from "./rail-back-link"; import RailSettingsLink from "./rail-settings-link"; @@ -11,8 +12,9 @@ type Props = { }; /** - * The rail on routines, activity and help: back to the bots, then those - * pages (super admin ones for super admins only). + * The rail on routines, usage, activity and help: back to the bots, then + * those pages (super admin ones for super admins only, crew managers for + * usage). */ export default function RailPagesList({ pathname }: Props) { const { pageProps } = useContext(AppContext); @@ -25,8 +27,12 @@ export default function RailPagesList({ pathname }: Props) { <> {/* */} - {RailPages.filter( - (page) => is_super_admin || !page.super_admin_only, + {RailPages.filter((page) => + canSeeRailItem({ + item: page, + is_super_admin, + user_types: pageProps?.user_types, + }), ).map((page) => ( {/* */} {SettingsPages.filter((page) => - page.can_view - ? page.can_view({ user_types: pageProps?.user_types }) - : is_super_admin || !page.super_admin_only, + canSeeRailItem({ + item: page, + is_super_admin, + user_types: pageProps?.user_types, + }), ).map((page) => ( is_super_admin || !item.super_admin_only); + const shown = RailUserMenuItems.filter((item) => + canSeeRailItem({ item, is_super_admin, user_types: pageProps?.user_types }), + ); const items = shown.filter((item) => !item.destructive); const destructive = shown.filter((item) => item.destructive); diff --git a/src/pages/admin/bots/[bot_id]/(partials)/bot-message-line.tsx b/src/pages/admin/bots/[bot_id]/(partials)/bot-message-line.tsx index 8799a52..d14d137 100644 --- a/src/pages/admin/bots/[bot_id]/(partials)/bot-message-line.tsx +++ b/src/pages/admin/bots/[bot_id]/(partials)/bot-message-line.tsx @@ -12,6 +12,7 @@ import GroupTimestamp from "./group-timestamp"; import MessageActions from "./message-actions"; import MessageReactions from "./message-reactions"; import MessageReplyQuote from "./message-reply-quote"; +import MessageUsageChip from "./message-usage-chip"; import NeedsYouChip from "./needs-you-chip"; import SpeakMessageButton from "./speak-message-button"; @@ -74,6 +75,7 @@ function BotMessageLine({ message, bot_name, at }: Props) { {message.needs_you ? : null} {at ? : null} + {can_speak ? ( ) : null} diff --git a/src/pages/admin/bots/[bot_id]/(partials)/message-usage-chip.tsx b/src/pages/admin/bots/[bot_id]/(partials)/message-usage-chip.tsx new file mode 100644 index 0000000..fa8aa11 --- /dev/null +++ b/src/pages/admin/bots/[bot_id]/(partials)/message-usage-chip.tsx @@ -0,0 +1,35 @@ +import Span from "@moduletrace/twui/layout/Span"; +import type { TokenUsageSummary } from "@/src/harness/usage/types"; +import formatTokens from "@/src/utils/format-tokens"; + +type Props = { + usage?: TokenUsageSummary | null; +}; + +/** + * What one answer cost, in the row under it beside its timestamp: the + * tokens every model call behind it spent (a chat turn, reading an image, + * making one), and the price when the provider reported it. The hover + * title breaks it down. + */ +export default function MessageUsageChip({ usage }: Props) { + if (!usage || usage.total_tokens <= 0) return null; + + const { prompt_tokens, completion_tokens, total_tokens, cost, calls } = usage; + const title = [ + `${formatTokens({ tokens: prompt_tokens })} in, ${formatTokens({ tokens: completion_tokens })} out`, + `${calls} model ${calls == 1 ? "call" : "calls"}`, + typeof cost == "number" && cost > 0 ? `$${cost.toFixed(4)}` : "", + ] + .filter(Boolean) + .join(" ยท "); + + return ( + + {formatTokens({ tokens: total_tokens })} tokens + + ); +} diff --git a/src/pages/admin/usage/(data)/usage-bar-heights.ts b/src/pages/admin/usage/(data)/usage-bar-heights.ts new file mode 100644 index 0000000..d937afc --- /dev/null +++ b/src/pages/admin/usage/(data)/usage-bar-heights.ts @@ -0,0 +1,37 @@ +/** + * The heights a day's bar can take, shortest first. A day's tokens are + * placed on this scale rather than given a height in pixels, so every bar + * is a static utility class the stylesheet already carries: no inline + * style, and the theme still decides the colour. + */ +export const UsageBarHeights = [ + "h-0.5", + "h-1", + "h-1.5", + "h-2", + "h-2.5", + "h-3", + "h-4", + "h-5", + "h-6", + "h-8", + "h-10", + "h-12", + "h-14", + "h-16", + "h-20", + "h-24", +]; + +/** + * Where a value sits on that scale, against the window's busiest day. A + * day with work on it is never shorter than one step, so it is visible. + */ +export function usageBarHeight({ value, max }: { value: number; max: number }) { + if (value <= 0) return UsageBarHeights[0]!; + + const ratio = max > 0 ? value / max : 0; + const step = Math.max(1, Math.round(ratio * (UsageBarHeights.length - 1))); + + return UsageBarHeights[Math.min(step, UsageBarHeights.length - 1)]!; +} diff --git a/src/pages/admin/usage/(data)/usage-harness-titles.ts b/src/pages/admin/usage/(data)/usage-harness-titles.ts new file mode 100644 index 0000000..c133008 --- /dev/null +++ b/src/pages/admin/usage/(data)/usage-harness-titles.ts @@ -0,0 +1,12 @@ +/** + * What each harness is called on the usage page, in words rather than the + * harness's own key. + */ +export const UsageHarnessTitles: Record = { + text: "Chat and tasks", + vision: "Reading media", + transcription: "Transcription", + speech: "Speech", + image: "Image generation", + embedding: "Embeddings", +}; diff --git a/src/pages/admin/usage/(data)/usage-ranges.ts b/src/pages/admin/usage/(data)/usage-ranges.ts new file mode 100644 index 0000000..d1c42af --- /dev/null +++ b/src/pages/admin/usage/(data)/usage-ranges.ts @@ -0,0 +1,27 @@ +export type UsageRange = { + /** + * The days back the range totals, today included. + */ + days: number; + title: string; +}; + +/** + * The periods the usage page offers, shortest first. A week and a month + * are what most questions need; a quarter is for spotting a trend. + */ +export const UsageRanges: UsageRange[] = [ + { days: 7, title: "7 days" }, + { days: 30, title: "30 days" }, + { days: 90, title: "90 days" }, +]; + +/** + * The range asked for in the address, or the page's default when it isn't + * one of the offered ones. + */ +export function parseUsageRange({ value }: { value?: string | null }): UsageRange | null { + const days = Number(value); + + return UsageRanges.find((range) => range.days == days) || null; +} diff --git a/src/pages/admin/usage/(functions)/grab-usage-server-props.ts b/src/pages/admin/usage/(functions)/grab-usage-server-props.ts new file mode 100644 index 0000000..772676d --- /dev/null +++ b/src/pages/admin/usage/(functions)/grab-usage-server-props.ts @@ -0,0 +1,35 @@ +import type { PagePropsType } from "@/src/types"; +import type { BunextPageServerFn } from "@moduletrace/bunext/types"; +import { AdminHomePath } from "@/src/dict/admin-sections-dict"; +import userAuth from "@/src/functions/backend/auth/user-auth"; +import grabUsageOverview from "@/src/functions/backend/db/usage/grab-usage-overview"; +import canManageCrew from "@/src/utils/rbac/can-manage-crew"; +import { parseUsageRange } from "../(data)/usage-ranges"; + +type Params = Parameters>[0]; + +/** + * The usage page's data for the period in the address (the default when + * it carries none). Crew managers only, since it is the whole install's + * work rather than one user's conversation; everyone else lands on the + * bots. + */ +export default async function grabUsageServerProps(params: Params) { + const { user, user_types } = await userAuth({ req: params.req }); + + if (!user?.id) { + return { redirect: { destination: "/auth/login" } }; + } + + if (!canManageCrew({ user_types })) { + return { redirect: { destination: AdminHomePath } }; + } + + const range = parseUsageRange({ value: params.query?.range }); + + return { + props: { + usage: await grabUsageOverview({ days: range?.days }).catch(() => null), + }, + }; +} diff --git a/src/pages/admin/usage/(functions)/sum-tokens.ts b/src/pages/admin/usage/(functions)/sum-tokens.ts new file mode 100644 index 0000000..9fd3b15 --- /dev/null +++ b/src/pages/admin/usage/(functions)/sum-tokens.ts @@ -0,0 +1,13 @@ +import type { TokenUsageSummary } from "@/src/harness/usage/types"; + +type Params = { + rows: TokenUsageSummary[]; +}; + +/** + * The tokens every row of a breakdown spent together, which is the + * denominator of its shares. + */ +export default function sumTokens({ rows }: Params) { + return rows.reduce((total, row) => total + row.total_tokens, 0); +} diff --git a/src/pages/admin/usage/(functions)/usage-href.ts b/src/pages/admin/usage/(functions)/usage-href.ts new file mode 100644 index 0000000..1a7fe58 --- /dev/null +++ b/src/pages/admin/usage/(functions)/usage-href.ts @@ -0,0 +1,15 @@ +import { UsagePath } from "@/src/dict/rail-pages-dict"; + +type Params = { + days?: number; +}; + +/** + * The usage page's address for the period it is showing, so a range can be + * linked to and the browser's own back button walks back through them. + */ +export default function usageHref({ days }: Params): string { + if (!days) return UsagePath; + + return `${UsagePath}?range=${days}`; +} diff --git a/src/pages/admin/usage/(hooks)/use-usage-overview.ts b/src/pages/admin/usage/(hooks)/use-usage-overview.ts new file mode 100644 index 0000000..646e9e9 --- /dev/null +++ b/src/pages/admin/usage/(hooks)/use-usage-overview.ts @@ -0,0 +1,63 @@ +import fetchApi from "@moduletrace/twui/utils/fetch/fetchApi"; +import type { APIResponseObject } from "@moduletrace/bunext/types"; +import { useEffect, useState } from "react"; +import type { UsageOverview } from "@/src/types"; + +type Params = { + days: number; + /** + * The totals the page already carries, when it covers this period. + */ + page_usage?: UsageOverview | null; +}; + +/** + * The usage totals for one period. The page's own copy is used when it + * covers that period; another one loads from the API, so switching the + * range doesn't reload the page. + */ +export default function useUsageOverview({ days, page_usage }: Params) { + const from_page = Boolean(page_usage && page_usage.window.days == days); + const [usage, set_usage] = useState(from_page ? page_usage || null : null); + const [loading, set_loading] = useState(!from_page); + const [error, set_error] = useState(); + + useEffect(() => { + if (from_page) { + set_usage(page_usage || null); + set_loading(false); + set_error(undefined); + return; + } + + const controller = new AbortController(); + + const load = async () => { + set_loading(true); + set_error(undefined); + + const res = await fetchApi<{}, APIResponseObject>( + `/api/admin/usage?days=${days}`, + { method: "GET" }, + ).catch(() => null); + + if (controller.signal.aborted) return; + + set_loading(false); + + if (!res?.success || !res.singleRes) { + set_usage(null); + set_error(res?.msg || `Couldn't read the usage.`); + return; + } + + set_usage(res.singleRes); + }; + + load().catch(() => {}); + + return () => controller.abort(); + }, [days, from_page, page_usage]); + + return { usage, loading, error }; +} diff --git a/src/pages/admin/usage/(partials)/usage-day-bar.tsx b/src/pages/admin/usage/(partials)/usage-day-bar.tsx new file mode 100644 index 0000000..38540c7 --- /dev/null +++ b/src/pages/admin/usage/(partials)/usage-day-bar.tsx @@ -0,0 +1,39 @@ +import Stack from "@moduletrace/twui/layout/Stack"; +import { twMerge } from "tailwind-merge"; +import dayjs from "dayjs"; +import type { UsageOverview } from "@/src/types"; +import formatTokens from "@/src/utils/format-tokens"; +import { usageBarHeight } from "../(data)/usage-bar-heights"; + +type Props = { + day: UsageOverview["days"][number]; + max: number; +}; + +/** + * One day's bar in the chart: its tokens above nothing, the bar itself, + * and the title spells the day out with what it spent. Hovering it shows + * that title. + */ +export default function UsageDayBar({ day, max }: Props) { + const title = `${dayjs(day.at).format("ddd, MMM D")}: ${formatTokens({ tokens: day.total_tokens })} tokens, ${daysCalls({ calls: day.calls })}`; + + return ( + + + + ); +} + +function daysCalls({ calls }: { calls: number }) { + return `${calls} ${calls == 1 ? "call" : "calls"}`; +} diff --git a/src/pages/admin/usage/(partials)/usage-day-chart.tsx b/src/pages/admin/usage/(partials)/usage-day-chart.tsx new file mode 100644 index 0000000..3457a76 --- /dev/null +++ b/src/pages/admin/usage/(partials)/usage-day-chart.tsx @@ -0,0 +1,51 @@ +import Row from "@moduletrace/twui/layout/Row"; +import Span from "@moduletrace/twui/layout/Span"; +import Stack from "@moduletrace/twui/layout/Stack"; +import dayjs from "dayjs"; +import type { UsageOverview } from "@/src/types"; +import formatTokens from "@/src/utils/format-tokens"; +import SettingsGroup from "@/src/pages/admin/settings/(partials)/settings-group"; +import UsageDayBar from "./usage-day-bar"; + +type Props = { + days: UsageOverview["days"]; +}; + +/** + * The period day by day, one bar a day, so a spike or a quiet stretch is + * read at a glance. The busiest day's total labels the top. + */ +export default function UsageDayChart({ days }: Props) { + const max = days.reduce((highest, day) => Math.max(highest, day.total_tokens), 0); + const first = days[0]; + const last = days[days.length - 1]; + + return ( + + + + {days.map((day) => ( + + ))} + + {first && last ? ( + + + {dayjs(first.at).format("MMM D")} + + + {dayjs(last.at).format("MMM D")} + + + ) : null} + + + ); +} diff --git a/src/pages/admin/usage/(partials)/usage-harness-breakdown.tsx b/src/pages/admin/usage/(partials)/usage-harness-breakdown.tsx new file mode 100644 index 0000000..3c8b3e7 --- /dev/null +++ b/src/pages/admin/usage/(partials)/usage-harness-breakdown.tsx @@ -0,0 +1,55 @@ +import Center from "@moduletrace/twui/layout/Center"; +import Row from "@moduletrace/twui/layout/Row"; +import Span from "@moduletrace/twui/layout/Span"; +import Stack from "@moduletrace/twui/layout/Stack"; +import type { UsageOverview } from "@/src/types"; +import SettingsGroup from "@/src/pages/admin/settings/(partials)/settings-group"; +import sumTokens from "../(functions)/sum-tokens"; +import UsageHarnessRow from "./usage-harness-row"; + +type Props = { + rows: UsageOverview["by_harness"]; +}; + +/** + * What each kind of model call spent: a chat turn, reading an image, + * transcribing, speaking, making a picture, embedding a file. Their totals + * add up to the period's. + */ +export default function UsageHarnessBreakdown({ rows }: Props) { + if (!rows.length) { + return ( + +
+ Nothing to break down. +
+ + ); + } + + const total_tokens = sumTokens({ rows }); + + return ( + + + + + Kind + + + Share + + + Tokens + + + Cost + + + {rows.map((row) => ( + + ))} + + + ); +} diff --git a/src/pages/admin/usage/(partials)/usage-harness-row.tsx b/src/pages/admin/usage/(partials)/usage-harness-row.tsx new file mode 100644 index 0000000..77bcf99 --- /dev/null +++ b/src/pages/admin/usage/(partials)/usage-harness-row.tsx @@ -0,0 +1,38 @@ +import Row from "@moduletrace/twui/layout/Row"; +import Span from "@moduletrace/twui/layout/Span"; +import type { UsageOverview } from "@/src/types"; +import formatTokens from "@/src/utils/format-tokens"; +import { UsageHarnessTitles } from "../(data)/usage-harness-titles"; + +type Props = { + row: UsageOverview["by_harness"][number]; + /** + * Every harness's tokens in the period, for the share. + */ + total_tokens: number; +}; + +/** + * One harness's line in the breakdown: what it is, its share of the + * period's tokens, and what it spent. + */ +export default function UsageHarnessRow({ row, total_tokens }: Props) { + const share = total_tokens ? Math.round((row.total_tokens / total_tokens) * 100) : 0; + + return ( + + + {UsageHarnessTitles[row.harness] || row.harness} + + + {share}% + + + {formatTokens({ tokens: row.total_tokens })} + + + {row.cost ? `$${row.cost.toFixed(2)}` : "-"} + + + ); +} diff --git a/src/pages/admin/usage/(partials)/usage-model-head-cell.tsx b/src/pages/admin/usage/(partials)/usage-model-head-cell.tsx new file mode 100644 index 0000000..5a34d38 --- /dev/null +++ b/src/pages/admin/usage/(partials)/usage-model-head-cell.tsx @@ -0,0 +1,23 @@ +import Span from "@moduletrace/twui/layout/Span"; +import { twMerge } from "tailwind-merge"; + +type Props = { + title: string; + className?: string; +}; + +/** + * A column heading of the models table, right-aligned over its figures. + */ +export default function UsageModelHeadCell({ title, className }: Props) { + return ( + + {title} + + ); +} diff --git a/src/pages/admin/usage/(partials)/usage-model-row.tsx b/src/pages/admin/usage/(partials)/usage-model-row.tsx new file mode 100644 index 0000000..43ef94d --- /dev/null +++ b/src/pages/admin/usage/(partials)/usage-model-row.tsx @@ -0,0 +1,52 @@ +import Row from "@moduletrace/twui/layout/Row"; +import Span from "@moduletrace/twui/layout/Span"; +import Stack from "@moduletrace/twui/layout/Stack"; +import type { UsageOverview } from "@/src/types"; +import formatTokens from "@/src/utils/format-tokens"; +import { UsageHarnessTitles } from "../(data)/usage-harness-titles"; + +type Props = { + row: UsageOverview["models"][number]; +}; + +/** + * One model's line: which provider and model it was, what it was used for, + * how many calls and tokens it took, and the price of a thousand tokens + * worked out from both, which is what makes models comparable. + */ +export default function UsageModelRow({ row }: Props) { + const per_thousand = row.cost && row.total_tokens + ? `$${((row.cost / row.total_tokens) * 1000).toFixed(3)}` + : "-"; + + return ( + + + + {row.model_name || row.model_slug} + + + {row.provider_name} - {UsageHarnessTitles[row.harness] || row.harness} + + + + {formatTokens({ tokens: row.calls })} + + + {formatTokens({ tokens: row.prompt_tokens })} + + + {formatTokens({ tokens: row.completion_tokens })} + + + {formatTokens({ tokens: row.total_tokens })} + + + {per_thousand} + + + {row.cost ? `$${row.cost.toFixed(2)}` : "-"} + + + ); +} diff --git a/src/pages/admin/usage/(partials)/usage-models-table.tsx b/src/pages/admin/usage/(partials)/usage-models-table.tsx new file mode 100644 index 0000000..20b41e3 --- /dev/null +++ b/src/pages/admin/usage/(partials)/usage-models-table.tsx @@ -0,0 +1,55 @@ +import Center from "@moduletrace/twui/layout/Center"; +import Row from "@moduletrace/twui/layout/Row"; +import Span from "@moduletrace/twui/layout/Span"; +import Stack from "@moduletrace/twui/layout/Stack"; +import type { UsageOverview } from "@/src/types"; +import SettingsGroup from "@/src/pages/admin/settings/(partials)/settings-group"; +import UsageModelHeadCell from "./usage-model-head-cell"; +import UsageModelRow from "./usage-model-row"; + +type Props = { + rows: UsageOverview["models"]; + days: number; +}; + +/** + * Every model the period ran, busiest first. A row's price per thousand + * tokens is what makes one model's cost comparable with another's, and + * how many calls each took says which one earned it. + */ +export default function UsageModelsTable({ rows, days }: Props) { + if (!rows.length) { + return ( + +
+ Nothing to list. +
+ + ); + } + + return ( + + + + + + + + + + + + {rows.map((row) => ( + + ))} + + + ); +} diff --git a/src/pages/admin/usage/(partials)/usage-range-picker.tsx b/src/pages/admin/usage/(partials)/usage-range-picker.tsx new file mode 100644 index 0000000..82742b9 --- /dev/null +++ b/src/pages/admin/usage/(partials)/usage-range-picker.tsx @@ -0,0 +1,35 @@ +import Button from "@moduletrace/twui/layout/Button"; +import Row from "@moduletrace/twui/layout/Row"; +import { UsageRanges } from "../(data)/usage-ranges"; + +type Props = { + days: number; + loading?: boolean; + onChange: (days: number) => void; +}; + +/** + * The periods the page totals, one button each. The one showing is the + * one that reads as pressed. + */ +export default function UsageRangePicker({ days, loading, onChange }: Props) { + return ( + + {UsageRanges.map((range) => ( + + ))} + + ); +} diff --git a/src/pages/admin/usage/(partials)/usage-stat.tsx b/src/pages/admin/usage/(partials)/usage-stat.tsx new file mode 100644 index 0000000..6070dd2 --- /dev/null +++ b/src/pages/admin/usage/(partials)/usage-stat.tsx @@ -0,0 +1,27 @@ +import Span from "@moduletrace/twui/layout/Span"; +import Stack from "@moduletrace/twui/layout/Stack"; + +type Props = { + label: string; + value: string; + /** + * A line under the figure, e.g. the comparison with an earlier period. + */ + hint?: string; +}; + +/** + * One figure of the page's totals: what it is above, the number itself + * big, and a quiet line under it. + */ +export default function UsageStat({ label, value, hint }: Props) { + return ( + + {label} + {value} + {hint ? ( + {hint} + ) : null} + + ); +} diff --git a/src/pages/admin/usage/(partials)/usage-totals.tsx b/src/pages/admin/usage/(partials)/usage-totals.tsx new file mode 100644 index 0000000..766aa21 --- /dev/null +++ b/src/pages/admin/usage/(partials)/usage-totals.tsx @@ -0,0 +1,58 @@ +import Row from "@moduletrace/twui/layout/Row"; +import type { UsageOverview } from "@/src/types"; +import formatTokens from "@/src/utils/format-tokens"; +import { UsageHarnessTitles } from "../(data)/usage-harness-titles"; +import UsageStat from "./usage-stat"; + +type Props = { + usage: UsageOverview; +}; + +/** + * The page's headline figures for the period: what it spent, how many + * calls that took, the price when providers report it, the busiest + * harness, and how the total compares with the period before it. + */ +export default function UsageTotals({ usage }: Props) { + const { totals, previous, by_harness } = usage; + const busiest = by_harness[0]; + + return ( + + + + + + + ); +} + +/** + * How this period compares with the one before it: a share up or down, or + * that there is nothing to compare with yet. + */ +function changeLabel({ current, previous }: { current: number; previous: number }) { + if (!previous) return current ? "New" : "Nothing yet"; + + const change = Math.round(((current - previous) / previous) * 100); + + if (change == 0) return "Level"; + + return `${change > 0 ? "+" : ""}${change}%`; +} diff --git a/src/pages/admin/usage/(sections)/usage-section.tsx b/src/pages/admin/usage/(sections)/usage-section.tsx new file mode 100644 index 0000000..441dfaa --- /dev/null +++ b/src/pages/admin/usage/(sections)/usage-section.tsx @@ -0,0 +1,59 @@ +import Center from "@moduletrace/twui/layout/Center"; +import Row from "@moduletrace/twui/layout/Row"; +import Span from "@moduletrace/twui/layout/Span"; +import { useContext, useState } from "react"; +import { AppContext } from "@/src/pages/__root"; +import { UsageRanges } from "../(data)/usage-ranges"; +import usageHref from "../(functions)/usage-href"; +import useUsageOverview from "../(hooks)/use-usage-overview"; +import UsageDayChart from "../(partials)/usage-day-chart"; +import UsageHarnessBreakdown from "../(partials)/usage-harness-breakdown"; +import UsageModelsTable from "../(partials)/usage-models-table"; +import UsageRangePicker from "../(partials)/usage-range-picker"; +import UsageTotals from "../(partials)/usage-totals"; + +/** + * The usage page's state: which period is totalled, kept in the address so + * a range can be linked to. The default period comes with the page; + * another one loads through the API. + */ +export default function UsageSection() { + const { pageProps } = useContext(AppContext); + const page_usage = pageProps?.usage; + const [days, set_days] = useState(page_usage?.window.days || UsageRanges[1]!.days); + const { usage, loading, error } = useUsageOverview({ days, page_usage }); + + const changeRange = (next_days: number) => { + set_days(next_days); + + const url = new URL(usageHref({ days: next_days }), window.location.origin); + + history.pushState(null, "", url); + }; + + return ( + <> + + + Every model call the crew made, whether it answered in chat, read an image, + spoke or made one. + + + + {error ? ( +
+ {error} +
+ ) : !usage ? ( +
+ Reading the usage... +
+ ) : ( + + )} + {usage ? : null} + {usage ? : null} + {usage ? : null} + + ); +} diff --git a/src/pages/admin/usage/index.server.ts b/src/pages/admin/usage/index.server.ts new file mode 100644 index 0000000..aefdf0c --- /dev/null +++ b/src/pages/admin/usage/index.server.ts @@ -0,0 +1,7 @@ +import type { PagePropsType } from "@/src/types"; +import type { BunextPageServerFn } from "@moduletrace/bunext/types"; +import grabUsageServerProps from "./(functions)/grab-usage-server-props"; + +export const server: BunextPageServerFn = async (params) => { + return await grabUsageServerProps(params); +}; diff --git a/src/pages/admin/usage/index.tsx b/src/pages/admin/usage/index.tsx new file mode 100644 index 0000000..4027ae2 --- /dev/null +++ b/src/pages/admin/usage/index.tsx @@ -0,0 +1,23 @@ +import type { BunextPageModuleMetaFn } from "@moduletrace/bunext/types"; +import { SiteData } from "@/src/data/site-data"; +import SettingsShell from "@/src/pages/admin/settings/(partials)/settings-shell"; +import UsageSection from "./(sections)/usage-section"; + +export default function AdminUsagePage() { + return ( + + + + ); +} + +export const meta: BunextPageModuleMetaFn = async () => { + return { + title: `Usage | ${SiteData["SiteName"]}`, + description: "What the crew's model calls spent.", + }; +}; diff --git a/src/pages/api/admin/bots/speak-message.ts b/src/pages/api/admin/bots/speak-message.ts index a9815c8..88c5ffd 100644 --- a/src/pages/api/admin/bots/speak-message.ts +++ b/src/pages/api/admin/bots/speak-message.ts @@ -1,6 +1,7 @@ import userAuth from "@/src/functions/backend/auth/user-auth"; import grabSpeakableMessage from "@/src/functions/backend/db/bots/grab-speakable-message"; import speakText from "@/src/harness/audio/speak-text"; +import withUsageScope from "@/src/harness/usage/with-usage-scope"; import HarnessError from "@/src/harness/harness-error"; import { SpeakBotMessageRequestSchema } from "@/src/schemas/bot-schema"; import AppError from "@/src/utils/app-error"; @@ -47,11 +48,26 @@ export const handler: BunextAPIRouteHandler = async ({ req, q throw new AppError(`There's nothing to read in that message.`); } - const audio = await speakText({ - text, - bot_id: conversation.bot.id, - workspace_id: conversation.workspace_id, - signal: req.signal, + /** + * Reading a message aloud is billed like any other model call, so + * it is recorded against the message it reads. + */ + const audio = await withUsageScope({ + scope: { + user_id: user.id, + bot_id: conversation.bot.id, + workspace_id: conversation.workspace_id, + session_id: message.session_id || null, + job_id: null, + message_id: message.id || null, + }, + task: () => + speakText({ + text, + bot_id: conversation.bot.id, + workspace_id: conversation.workspace_id, + signal: req.signal, + }), }); return new Response(audio.stream, { diff --git a/src/pages/api/admin/bots/transcribe.ts b/src/pages/api/admin/bots/transcribe.ts index 08ee151..4d87d23 100644 --- a/src/pages/api/admin/bots/transcribe.ts +++ b/src/pages/api/admin/bots/transcribe.ts @@ -1,6 +1,8 @@ import userAuth from "@/src/functions/backend/auth/user-auth"; +import grabActiveBotSession from "@/src/functions/backend/db/bot-sessions/grab-active-bot-session"; import grabUserConversation from "@/src/functions/backend/db/bot-sessions/grab-user-conversation"; import transcribeAudio from "@/src/harness/audio/transcribe-audio"; +import withUsageScope from "@/src/harness/usage/with-usage-scope"; import { TranscribeVoiceRequestSchema } from "@/src/schemas/bot-schema"; import HarnessError from "@/src/harness/harness-error"; import AppError from "@/src/utils/app-error"; @@ -60,12 +62,33 @@ export const handler: BunextAPIRouteHandler< throw new AppError(`Nothing was recorded.`); } - const text = await transcribeAudio({ - audio: { data: new Uint8Array(buffer), format: AudioFormats[type] || type }, + /** + * The dictation is work done for this user in this conversation, + * even though no queued job or message is behind it. + */ + const session = await grabActiveBotSession({ + user_id: user.id, bot_id: conversation.bot.id, workspace_id: conversation.workspace_id, }); + const text = await withUsageScope({ + scope: { + user_id: user.id, + bot_id: conversation.bot.id, + workspace_id: conversation.workspace_id, + session_id: session?.id || null, + job_id: null, + message_id: null, + }, + task: () => + transcribeAudio({ + audio: { data: new Uint8Array(buffer), format: AudioFormats[type] || type }, + bot_id: conversation.bot.id, + workspace_id: conversation.workspace_id, + }), + }); + if (!text) { throw new AppError(`No speech was heard. Try again a little closer to the mic.`); } diff --git a/src/pages/api/admin/usage/index.ts b/src/pages/api/admin/usage/index.ts new file mode 100644 index 0000000..2839432 --- /dev/null +++ b/src/pages/api/admin/usage/index.ts @@ -0,0 +1,40 @@ +import userAuth from "@/src/functions/backend/auth/user-auth"; +import grabUsageOverview from "@/src/functions/backend/db/usage/grab-usage-overview"; +import { UsageRequestSchema } from "@/src/schemas/usage-schema"; +import type { UsageOverview } from "@/src/types"; +import canManageCrew from "@/src/utils/rbac/can-manage-crew"; +import parseWithSchema from "@/src/utils/parse-with-schema"; +import safeErrorMessage from "@/src/utils/safe-error-message"; +import type { APIResponseObject, BunextAPIRouteHandler } from "@moduletrace/bunext/types"; + +/** + * What the install's models spent over a period: in total, per day, per + * harness and per model. Super admins and admins only, since it is the + * whole install's work rather than one user's conversation. + */ +export const handler: BunextAPIRouteHandler> = async ({ req, query }) => { + try { + if (req.method != "GET") { + return { success: false, msg: `Method not allowed` }; + } + + const { user, user_types } = await userAuth({ req }); + + if (!user?.logged_in_status || !user.id) { + return { success: false, msg: `Unauthorized`, logoutUser: true }; + } + + if (!canManageCrew({ user_types })) { + return { success: false, msg: `You don't manage the crew.` }; + } + + const request = parseWithSchema({ schema: UsageRequestSchema, data: query }); + + return { + success: true, + singleRes: await grabUsageOverview({ days: request.days }), + }; + } catch (error: unknown) { + return { success: false, msg: safeErrorMessage({ error }) }; + } +}; diff --git a/src/schemas/usage-schema.ts b/src/schemas/usage-schema.ts new file mode 100644 index 0000000..c05ad3f --- /dev/null +++ b/src/schemas/usage-schema.ts @@ -0,0 +1,18 @@ +import { z } from "zod"; +import { AppData } from "@/src/data/app-data"; + +/** + * `GET api/admin/usage`: the period the usage page totals, in days back + * from today. Left out, it is the page's default; past the limit, the + * limit. + */ +export const UsageRequestSchema = z.object({ + days: z.coerce + .number() + .int() + .positive() + .max(AppData.UsageMaxDays, "That is too long a period") + .optional(), +}); + +export type UsageRequest = z.infer; diff --git a/src/types/index.ts b/src/types/index.ts index afa5a51..9693b82 100755 --- a/src/types/index.ts +++ b/src/types/index.ts @@ -32,6 +32,7 @@ import type { HarnessProviderKind } from "../dict/harness-provider-dict"; import type { EnvParadigm, KnownEnv } from "../dict/env-dict"; import type { ContainerStatus } from "../dict/computer-dict"; import type { BotFace, BotMessageReactionEmoji } from "../dict/bot-dict"; +import type { TokenUsageSummary } from "../harness/usage/types"; import type { McpServerStatus, McpTransport } from "../dict/mcp-dict"; import type { CustomToolHttpMethod, @@ -875,6 +876,36 @@ export type DatabaseTableColumn = { */ export type DatabaseSearchMode = "every_word" | "phrase"; +/** + * The period the usage page totals, in epoch ms. + */ +export type UsageWindow = { + days: number; + start: number; + end: number; +}; + +/** + * Everything the usage page shows for one window: what the install's + * model calls spent, in total, per day, per harness and per model, with + * the day before the window and the last seven days for comparison. + */ +export type UsageOverview = { + window: UsageWindow; + totals: TokenUsageSummary; + by_harness: (TokenUsageSummary & { harness: string })[]; + days: (TokenUsageSummary & { at: number; label: string })[]; + models: (TokenUsageSummary & { + model_slug: string; + model_name: string | null; + provider_name: string; + provider_id: number | null; + harness: string; + })[]; + previous: TokenUsageSummary; + week: TokenUsageSummary; +}; + /** * One table of the database, as the database browser lists it. */ @@ -988,6 +1019,11 @@ export type BotMessageWithAttachments = BUN_SQLITE_COBBLE_BOT_MESSAGES & { attachments?: BotMessageAttachment[]; reactions?: BotMessageReaction[]; reply_to?: BotMessageReply | null; + /** + * What the model calls behind this message spent (`token_usages`), + * summed. Only on bot messages that took at least one call. + */ + usage?: TokenUsageSummary | null; }; /** @@ -1177,6 +1213,11 @@ export type PagePropsType = { */ database_tables?: DatabaseTableSummary[] | null; database_records?: DatabaseRecordsPayload | null; + /** + * What the install's models spent, on the usage page (crew managers + * only). + */ + usage?: UsageOverview | null; /** * The open conversation's files (a bot's own, or a workspace's shared * ones), for the files modal and the chips that name one. diff --git a/src/utils/format-tokens.test.ts b/src/utils/format-tokens.test.ts new file mode 100644 index 0000000..6e9c2cb --- /dev/null +++ b/src/utils/format-tokens.test.ts @@ -0,0 +1,35 @@ +import { describe, expect, test } from "bun:test"; +import formatTokens from "./format-tokens"; + +describe("formatTokens", () => { + test("counts under a thousand are exact", () => { + expect(formatTokens({ tokens: 0 })).toBe("0"); + expect(formatTokens({ tokens: 842 })).toBe("842"); + expect(formatTokens({ tokens: 999 })).toBe("999"); + }); + + test("counts under ten thousand keep a decimal, without a trailing zero", () => { + expect(formatTokens({ tokens: 1000 })).toBe("1K"); + expect(formatTokens({ tokens: 1240 })).toBe("1.2K"); + expect(formatTokens({ tokens: 1240.7 })).toBe("1.2K"); + expect(formatTokens({ tokens: 9999 })).toBe("10K"); + }); + + test("counts under a million are whole thousands", () => { + expect(formatTokens({ tokens: 12_400 })).toBe("12K"); + expect(formatTokens({ tokens: 841_600 })).toBe("842K"); + expect(formatTokens({ tokens: 999_999 })).toBe("1000K"); + }); + + test("counts past a million are millions", () => { + expect(formatTokens({ tokens: 1_000_000 })).toBe("1M"); + expect(formatTokens({ tokens: 1_240_000 })).toBe("1.2M"); + expect(formatTokens({ tokens: 24_800_000 })).toBe("24.8M"); + }); + + test("a missing count reads as nothing spent", () => { + expect(formatTokens({ tokens: null })).toBe("0"); + expect(formatTokens({ tokens: undefined })).toBe("0"); + expect(formatTokens({ tokens: Number.NaN })).toBe("0"); + }); +}); diff --git a/src/utils/format-tokens.ts b/src/utils/format-tokens.ts new file mode 100644 index 0000000..7951ed8 --- /dev/null +++ b/src/utils/format-tokens.ts @@ -0,0 +1,19 @@ +type Params = { + tokens: number | null | undefined; +}; + +/** + * A token count as a person reads it: exact up to a thousand, then + * rounded to a decimal under ten thousand ("12.4K"), whole thousands up to + * a million ("842K") and millions after it ("1.2M"). A long run's total is + * never a wall of digits. + */ +export default function formatTokens({ tokens }: Params) { + const count = Number(tokens) || 0; + + if (count < 1000) return String(count); + if (count < 10_000) return `${(count / 1000).toFixed(1).replace(/\.0$/, "")}K`; + if (count < 1_000_000) return `${Math.round(count / 1000)}K`; + + return `${(count / 1_000_000).toFixed(1).replace(/\.0$/, "")}M`; +} diff --git a/src/utils/rbac/can-see-rail-item.test.ts b/src/utils/rbac/can-see-rail-item.test.ts new file mode 100644 index 0000000..b9b3eea --- /dev/null +++ b/src/utils/rbac/can-see-rail-item.test.ts @@ -0,0 +1,85 @@ +import { describe, expect, test } from "bun:test"; +import canSeeRailItem from "./can-see-rail-item"; +import canManageCrew from "./can-manage-crew"; +import canListUsers from "./can-list-users"; +import { RailPages, UsagePath } from "@/src/dict/rail-pages-dict"; + +function types(...user_types: string[]) { + return user_types.map((user_type) => ({ user_type })) as never; +} + +const super_admin = types("super_admin"); +const admin = types("member", "admin"); +const member = types("member"); +const read_only = types("read_only"); + +describe("canSeeRailItem", () => { + test("a page anyone may open shows to everyone", () => { + const page = {}; + + expect(canSeeRailItem({ item: page, is_super_admin: true, user_types: super_admin })).toBe(true); + expect(canSeeRailItem({ item: page, is_super_admin: false, user_types: member })).toBe(true); + expect(canSeeRailItem({ item: page, is_super_admin: false, user_types: read_only })).toBe(true); + }); + + test("a super admin page shows to super admins only", () => { + const page = { super_admin_only: true }; + + expect(canSeeRailItem({ item: page, is_super_admin: true, user_types: super_admin })).toBe(true); + expect(canSeeRailItem({ item: page, is_super_admin: false, user_types: admin })).toBe(false); + expect(canSeeRailItem({ item: page, is_super_admin: false, user_types: member })).toBe(false); + }); + + test("an RBAC gate is asked, not guessed from super admin", () => { + const page = { can_view: canManageCrew }; + + expect(canSeeRailItem({ item: page, is_super_admin: false, user_types: super_admin })).toBe(true); + expect(canSeeRailItem({ item: page, is_super_admin: false, user_types: admin })).toBe(true); + expect(canSeeRailItem({ item: page, is_super_admin: false, user_types: member })).toBe(false); + }); + + test("a can_view rule wins over super_admin_only, so a gate is never widened by it", () => { + /** + * The two together should follow the narrower rule the code asked + * for. No page sets both today, and this keeps a future one from + * quietly showing to every super admin when its rule says no. + */ + const page = { can_view: canListUsers, super_admin_only: true }; + + expect(canSeeRailItem({ item: page, is_super_admin: false, user_types: member })).toBe(false); + expect(canSeeRailItem({ item: page, is_super_admin: false, user_types: read_only })).toBe(false); + expect(canSeeRailItem({ item: page, is_super_admin: false, user_types: admin })).toBe(true); + }); +}); + +describe("RailPages", () => { + test("usage is a rail page, gated to the crew's managers", () => { + const usage = RailPages.find((page) => page.url == UsagePath); + + expect(usage?.title).toBe("Usage"); + expect(usage?.can_view).toBeDefined(); + expect(usage?.super_admin_only).toBeFalsy(); + }); + + test("activity stays super admin only", () => { + const activity = RailPages.find((page) => page.url == "/admin/logs"); + + expect(activity?.super_admin_only).toBe(true); + }); + + test("a member is offered routines, files and help but not usage or activity", () => { + const shown = RailPages.filter((page) => + canSeeRailItem({ item: page, is_super_admin: false, user_types: member }), + ).map((page) => page.title); + + expect(shown).toEqual(["Routines", "Files", "Help"]); + }); + + test("an admin is offered usage but still not activity", () => { + const shown = RailPages.filter((page) => + canSeeRailItem({ item: page, is_super_admin: false, user_types: admin }), + ).map((page) => page.title); + + expect(shown).toEqual(["Routines", "Files", "Usage", "Help"]); + }); +}); diff --git a/src/utils/rbac/can-see-rail-item.ts b/src/utils/rbac/can-see-rail-item.ts new file mode 100644 index 0000000..95bed45 --- /dev/null +++ b/src/utils/rbac/can-see-rail-item.ts @@ -0,0 +1,29 @@ +import type { BUN_SQLITE_COBBLE_USER_TYPES } from "@/db/types/db"; + +type Params = { + item: { + /** + * Who sees it, when that's an RBAC rule rather than super admins only. + */ + can_view?: (params: { + user_types?: Pick[] | null; + }) => boolean; + super_admin_only?: boolean; + }; + is_super_admin: boolean; + user_types?: Pick[] | null; +}; + +/** + * Whether a rail destination (a settings sub page, a popover page) shows + * to this user: its own RBAC rule when it has one, else super admins only + * when it's a super admin page, else everyone. + * + * Shared by the rail lists and the user popover, so a page gated in one is + * never offered in the other. + */ +export default function canSeeRailItem({ item, is_super_admin, user_types }: Params): boolean { + if (item.can_view) return item.can_view({ user_types }); + + return is_super_admin || !item.super_admin_only; +}