Record what every model call spends, and show it
Nothing anywhere recorded what a model call cost: the harness used a provider's usage numbers only to decide whether a turn had finished, then dropped them. So a reply could not say what it took to make, one model could not be compared with another on price, and there was no way to ask what a day, a week or a month of the crew's work came to. - `token_usages` records one row per call: user, bot, workspace, session, job and message, the harness, the provider and model that actually answered, the prompt and completion counts, and the cost when the provider reports one. - `harness-usage-scope.ts` carries those ids in an AsyncLocalStorage scope that `run-bot-job.ts` enters once, so a run's turns, its session summary, its transcriptions, images and embeddings are all attributed without threading ids through every harness function. Nested runs merge rather than replace, and a null genuinely clears a field (a bot's own chat has no workspace). - Work with no run behind it is charged to its owner instead: `run-with-fallback-scope.ts` reaches the workspace's creator else the bot's, and deliberately does not override a real run, so an ingest a bot asked for is not misattributed to the bot's owner. - All six harnesses record through `withCandidates`'s `on_success`. Recording never throws: a model that answered must not lose its answer to a failed insert. - A job's calls land before its message exists, so `complete-bot-job.ts` back-fills `message_id` once the message is inserted. - The message's actions row shows what it cost, beside the timestamp and the play button, with the breakdown in its title. - `/admin/usage` totals the install's spending for 7, 30 or 90 days: the total against the previous period, a day-by-day chart, a split per kind of call, and a table per model with the price per thousand tokens worked out from cost and tokens, which is what makes two models comparable. Crew managers only, since it is the whole install's work rather than one user's conversation. It sits with Routines and Activity in the user popover rather than under Settings, because it is a record of what happened, not something to configure. Days are grouped in local time (`date(created_at / 1000, 'unixepoch', 'localtime')`), matching the local midnights the window builds and the `YYYY-MM-DD` buckets it looks up. Grouping by the bare UTC day put a late evening's calls under a name no bucket carried, so they vanished and the chart's total silently disagreed with the totals card; a test pins it. `canSeeRailItem` now decides who sees a popover or rail entry. The two each filtered with their own inline rule and only one understood `can_view`, so a gated page could be offered in the popover to everyone.
This commit is contained in:
+143
@@ -2866,6 +2866,149 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
tableName: "token_usages",
|
||||
tableDescription: `One model call that reported its usage: the tokens it spent, on which model and provider, for whom and for what. A bot message's total is the sum of the calls it took; the usage page sums them by model, provider and period`,
|
||||
fields: [
|
||||
{
|
||||
fieldName: "user_id",
|
||||
fieldDescription: `Whose work it was: the user whose conversation the call ran for`,
|
||||
dataType: "INTEGER",
|
||||
foreignKey: {
|
||||
cascadeDelete: true,
|
||||
destinationTableName: "users",
|
||||
destinationTableColumnName: "id",
|
||||
destinationTableColumnType: "INTEGER",
|
||||
foreignKeyName: "token_usage_user_fk",
|
||||
},
|
||||
},
|
||||
{
|
||||
fieldName: "bot_id",
|
||||
fieldDescription: `The bot it was for, when it was for one`,
|
||||
dataType: "INTEGER",
|
||||
foreignKey: {
|
||||
cascadeDelete: true,
|
||||
destinationTableName: "bots",
|
||||
destinationTableColumnName: "id",
|
||||
destinationTableColumnType: "INTEGER",
|
||||
foreignKeyName: "token_usage_bot_fk",
|
||||
},
|
||||
},
|
||||
{
|
||||
fieldName: "workspace_id",
|
||||
fieldDescription: `The workspace the call was in; null for a bot's own chat`,
|
||||
dataType: "INTEGER",
|
||||
foreignKey: {
|
||||
cascadeDelete: true,
|
||||
destinationTableName: "workspaces",
|
||||
destinationTableColumnName: "id",
|
||||
destinationTableColumnType: "INTEGER",
|
||||
foreignKeyName: "token_usage_workspace_fk",
|
||||
},
|
||||
},
|
||||
{
|
||||
fieldName: "session_id",
|
||||
fieldDescription: `The conversation it was in`,
|
||||
dataType: "INTEGER",
|
||||
foreignKey: {
|
||||
cascadeDelete: true,
|
||||
destinationTableName: "bot_sessions",
|
||||
destinationTableColumnName: "id",
|
||||
destinationTableColumnType: "INTEGER",
|
||||
foreignKeyName: "token_usage_session_fk",
|
||||
},
|
||||
},
|
||||
{
|
||||
fieldName: "message_id",
|
||||
fieldDescription: `The message it produced, so the thread can show what an answer cost. Null for calls that produce no message (naming a session, embedding a file)`,
|
||||
dataType: "INTEGER",
|
||||
foreignKey: {
|
||||
cascadeDelete: true,
|
||||
destinationTableName: "bot_messages",
|
||||
destinationTableColumnName: "id",
|
||||
destinationTableColumnType: "INTEGER",
|
||||
foreignKeyName: "token_usage_message_fk",
|
||||
},
|
||||
},
|
||||
{
|
||||
fieldName: "job_id",
|
||||
fieldDescription: `The queued job it ran for, when it ran for one`,
|
||||
dataType: "INTEGER",
|
||||
foreignKey: {
|
||||
cascadeDelete: true,
|
||||
destinationTableName: "bots_queue",
|
||||
destinationTableColumnName: "id",
|
||||
destinationTableColumnType: "INTEGER",
|
||||
foreignKeyName: "token_usage_job_fk",
|
||||
},
|
||||
},
|
||||
{
|
||||
fieldName: "harness",
|
||||
fieldDescription: `What the model was used for: a chat turn, reading media, transcribing, speaking, making an image, embedding`,
|
||||
dataType: "TEXT",
|
||||
options: Harnesses.map((h) => h.value),
|
||||
},
|
||||
{
|
||||
fieldName: "provider_id",
|
||||
fieldDescription: `The provider that served it. Not a foreign key: it stays readable after the provider is deleted`,
|
||||
dataType: "INTEGER",
|
||||
},
|
||||
{
|
||||
fieldName: "provider_name",
|
||||
fieldDescription: `The provider's name as it was, so a deleted provider still reads`,
|
||||
dataType: "TEXT",
|
||||
},
|
||||
{
|
||||
fieldName: "model_id",
|
||||
fieldDescription: `The catalog row of the model. Not a foreign key: a model the provider stops listing still counts`,
|
||||
dataType: "INTEGER",
|
||||
},
|
||||
{
|
||||
fieldName: "model_slug",
|
||||
fieldDescription: `The provider's model id, as it was called`,
|
||||
dataType: "TEXT",
|
||||
},
|
||||
{
|
||||
fieldName: "prompt_tokens",
|
||||
fieldDescription: `Tokens sent: the prompt, the system instructions and the tool results`,
|
||||
dataType: "INTEGER",
|
||||
defaultValue: 0,
|
||||
},
|
||||
{
|
||||
fieldName: "completion_tokens",
|
||||
fieldDescription: `Tokens written back: the answer, its tool calls and its reasoning`,
|
||||
dataType: "INTEGER",
|
||||
defaultValue: 0,
|
||||
},
|
||||
{
|
||||
fieldName: "cost",
|
||||
fieldDescription: `The provider's price in USD, when it reports one`,
|
||||
dataType: "REAL",
|
||||
},
|
||||
],
|
||||
indexes: [
|
||||
{
|
||||
indexName: "idx_token_usages_created_at",
|
||||
indexTableFields: ["created_at"],
|
||||
},
|
||||
{
|
||||
indexName: "idx_token_usages_message_id",
|
||||
indexTableFields: ["message_id"],
|
||||
},
|
||||
{
|
||||
indexName: "idx_token_usages_provider_model",
|
||||
indexTableFields: ["provider_id", "model_slug"],
|
||||
},
|
||||
{
|
||||
indexName: "idx_token_usages_user_created_at",
|
||||
indexTableFields: ["user_id", "created_at"],
|
||||
},
|
||||
{
|
||||
indexName: "idx_token_usages_bot_created_at",
|
||||
indexTableFields: ["bot_id", "created_at"],
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
tableName: "bot_file_vectors",
|
||||
tableDescription: `sqlite-vec vectors of a file's chunks, at \`AppData.EmbeddingDimensions\` wide whatever the model returns. A vec0 table is only recreated with \`bunx bun-sqlite schema --vector\`, so the entrypoint never passes it`,
|
||||
|
||||
Reference in New Issue
Block a user