Nothing anywhere recorded what a model call cost: the harness used a
provider's usage numbers only to decide whether a turn had finished, then
dropped them. So a reply could not say what it took to make, one model
could not be compared with another on price, and there was no way to ask
what a day, a week or a month of the crew's work came to.
- `token_usages` records one row per call: user, bot, workspace, session,
job and message, the harness, the provider and model that actually
answered, the prompt and completion counts, and the cost when the
provider reports one.
- `harness-usage-scope.ts` carries those ids in an AsyncLocalStorage
scope that `run-bot-job.ts` enters once, so a run's turns, its session
summary, its transcriptions, images and embeddings are all attributed
without threading ids through every harness function. Nested runs merge
rather than replace, and a null genuinely clears a field (a bot's own
chat has no workspace).
- Work with no run behind it is charged to its owner instead:
`run-with-fallback-scope.ts` reaches the workspace's creator else the
bot's, and deliberately does not override a real run, so an ingest a
bot asked for is not misattributed to the bot's owner.
- All six harnesses record through `withCandidates`'s `on_success`.
Recording never throws: a model that answered must not lose its answer
to a failed insert.
- A job's calls land before its message exists, so `complete-bot-job.ts`
back-fills `message_id` once the message is inserted.
- The message's actions row shows what it cost, beside the timestamp and
the play button, with the breakdown in its title.
- `/admin/usage` totals the install's spending for 7, 30 or 90 days: the
total against the previous period, a day-by-day chart, a split per kind
of call, and a table per model with the price per thousand tokens
worked out from cost and tokens, which is what makes two models
comparable. Crew managers only, since it is the whole install's work
rather than one user's conversation. It sits with Routines and Activity
in the user popover rather than under Settings, because it is a record
of what happened, not something to configure.
Days are grouped in local time (`date(created_at / 1000, 'unixepoch',
'localtime')`), matching the local midnights the window builds and the
`YYYY-MM-DD` buckets it looks up. Grouping by the bare UTC day put a late
evening's calls under a name no bucket carried, so they vanished and the
chart's total silently disagreed with the totals card; a test pins it.
`canSeeRailItem` now decides who sees a popover or rail entry. The two
each filtered with their own inline rule and only one understood
`can_view`, so a gated page could be offered in the popover to everyone.
`text_content` is an extracted-text cache, but nothing stopped it from
holding a dump of bytes:
- `fileToText` ended its switch with `default: return buffer.toString("utf-8")`
and defaulted a missing extension to "txt", so any binary was decoded.
- Both extraction callers gated on `media_type == "document"`, and
`attachmentFileKind` classes everything non-image/audio/video as a
document: an SVG, a zip, a spreadsheet, a .bin.
- Nothing capped the length, so a 20 MB attachment could leave 20 MB of
text in the database.
- `storeChatAttachments` copied the text onto a second `media` row, and
`SELECT m.*` pulled the column into page props and API responses.
On a real install an SVG held 6852 characters of markup and a
`.designcraft` file held 142 KB of raw bytes.
Changes:
- `file-to-text.ts` lists the text extensions explicitly and returns
undefined for anything else; no extension is no text.
- `can-extract-text.ts` is the one allowlist (reusing
`BotTextFileExtensions`), used by both upload routes and by
`uploadAndRecordMedia`; `extension-of.ts` is shared with the storage
catalog instead of duplicated.
- `BotAttachmentTextStoreMaxChars` (200k) caps the column via
`cap-text-content.ts`.
- `store-chat-attachments.ts` no longer copies the text; ingestion
re-reads the bytes once.
- `media-columns.ts` selects every media column but `text_content`, which
now has to be asked for (`grab-messages-media`'s reply path and
ingestion do); the media CRUD routes sanitize rows they write.
- `ingest-bot-file.ts` decides on size alone, since a cached slice made
an oversized file look already read.
- `bun run media:prune-text` cleans rows that were already stored
(rerunnable, with a VACUUM hint).
Verified on the dev database: both bad rows cleared (146 KB), rerun
reports nothing to do.
tben