Record what every model call spends, and show it
Nothing anywhere recorded what a model call cost: the harness used a provider's usage numbers only to decide whether a turn had finished, then dropped them. So a reply could not say what it took to make, one model could not be compared with another on price, and there was no way to ask what a day, a week or a month of the crew's work came to. - `token_usages` records one row per call: user, bot, workspace, session, job and message, the harness, the provider and model that actually answered, the prompt and completion counts, and the cost when the provider reports one. - `harness-usage-scope.ts` carries those ids in an AsyncLocalStorage scope that `run-bot-job.ts` enters once, so a run's turns, its session summary, its transcriptions, images and embeddings are all attributed without threading ids through every harness function. Nested runs merge rather than replace, and a null genuinely clears a field (a bot's own chat has no workspace). - Work with no run behind it is charged to its owner instead: `run-with-fallback-scope.ts` reaches the workspace's creator else the bot's, and deliberately does not override a real run, so an ingest a bot asked for is not misattributed to the bot's owner. - All six harnesses record through `withCandidates`'s `on_success`. Recording never throws: a model that answered must not lose its answer to a failed insert. - A job's calls land before its message exists, so `complete-bot-job.ts` back-fills `message_id` once the message is inserted. - The message's actions row shows what it cost, beside the timestamp and the play button, with the breakdown in its title. - `/admin/usage` totals the install's spending for 7, 30 or 90 days: the total against the previous period, a day-by-day chart, a split per kind of call, and a table per model with the price per thousand tokens worked out from cost and tokens, which is what makes two models comparable. Crew managers only, since it is the whole install's work rather than one user's conversation. It sits with Routines and Activity in the user popover rather than under Settings, because it is a record of what happened, not something to configure. Days are grouped in local time (`date(created_at / 1000, 'unixepoch', 'localtime')`), matching the local midnights the window builds and the `YYYY-MM-DD` buckets it looks up. Grouping by the bare UTC day put a late evening's calls under a name no bucket carried, so they vanished and the chart's total silently disagreed with the totals card; a test pins it. `canSeeRailItem` now decides who sees a popover or rail entry. The two each filtered with their own inline rule and only one understood `can_view`, so a gated page could be offered in the popover to everyone.
This commit is contained in:
@@ -0,0 +1,39 @@
|
||||
import Stack from "@moduletrace/twui/layout/Stack";
|
||||
import { twMerge } from "tailwind-merge";
|
||||
import dayjs from "dayjs";
|
||||
import type { UsageOverview } from "@/src/types";
|
||||
import formatTokens from "@/src/utils/format-tokens";
|
||||
import { usageBarHeight } from "../(data)/usage-bar-heights";
|
||||
|
||||
type Props = {
|
||||
day: UsageOverview["days"][number];
|
||||
max: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* One day's bar in the chart: its tokens above nothing, the bar itself,
|
||||
* and the title spells the day out with what it spent. Hovering it shows
|
||||
* that title.
|
||||
*/
|
||||
export default function UsageDayBar({ day, max }: Props) {
|
||||
const title = `${dayjs(day.at).format("ddd, MMM D")}: ${formatTokens({ tokens: day.total_tokens })} tokens, ${daysCalls({ calls: day.calls })}`;
|
||||
|
||||
return (
|
||||
<Stack
|
||||
title={title}
|
||||
className="grow basis-[3px] min-w-[3px] max-w-10 h-24 justify-end items-stretch gap-0!"
|
||||
>
|
||||
<Stack
|
||||
className={twMerge(
|
||||
"w-full rounded-t-[2px]",
|
||||
day.total_tokens ? "bg-ink" : "bg-hairline",
|
||||
usageBarHeight({ value: day.total_tokens, max }),
|
||||
)}
|
||||
/>
|
||||
</Stack>
|
||||
);
|
||||
}
|
||||
|
||||
function daysCalls({ calls }: { calls: number }) {
|
||||
return `${calls} ${calls == 1 ? "call" : "calls"}`;
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
import Row from "@moduletrace/twui/layout/Row";
|
||||
import Span from "@moduletrace/twui/layout/Span";
|
||||
import Stack from "@moduletrace/twui/layout/Stack";
|
||||
import dayjs from "dayjs";
|
||||
import type { UsageOverview } from "@/src/types";
|
||||
import formatTokens from "@/src/utils/format-tokens";
|
||||
import SettingsGroup from "@/src/pages/admin/settings/(partials)/settings-group";
|
||||
import UsageDayBar from "./usage-day-bar";
|
||||
|
||||
type Props = {
|
||||
days: UsageOverview["days"];
|
||||
};
|
||||
|
||||
/**
|
||||
* The period day by day, one bar a day, so a spike or a quiet stretch is
|
||||
* read at a glance. The busiest day's total labels the top.
|
||||
*/
|
||||
export default function UsageDayChart({ days }: Props) {
|
||||
const max = days.reduce((highest, day) => Math.max(highest, day.total_tokens), 0);
|
||||
const first = days[0];
|
||||
const last = days[days.length - 1];
|
||||
|
||||
return (
|
||||
<SettingsGroup
|
||||
title="Day by day"
|
||||
description={
|
||||
max
|
||||
? `Busiest day: ${formatTokens({ tokens: max })} tokens.`
|
||||
: "Nothing spent in this period yet."
|
||||
}
|
||||
>
|
||||
<Stack className="w-full gap-2!">
|
||||
<Row className="w-full gap-1! items-end">
|
||||
{days.map((day) => (
|
||||
<UsageDayBar day={day} max={max} key={day.at} />
|
||||
))}
|
||||
</Row>
|
||||
{first && last ? (
|
||||
<Row className="w-full gap-2! items-center justify-between">
|
||||
<Span className="text-[11px] leading-4 text-ink-secondary">
|
||||
{dayjs(first.at).format("MMM D")}
|
||||
</Span>
|
||||
<Span className="text-[11px] leading-4 text-ink-secondary">
|
||||
{dayjs(last.at).format("MMM D")}
|
||||
</Span>
|
||||
</Row>
|
||||
) : null}
|
||||
</Stack>
|
||||
</SettingsGroup>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
import Center from "@moduletrace/twui/layout/Center";
|
||||
import Row from "@moduletrace/twui/layout/Row";
|
||||
import Span from "@moduletrace/twui/layout/Span";
|
||||
import Stack from "@moduletrace/twui/layout/Stack";
|
||||
import type { UsageOverview } from "@/src/types";
|
||||
import SettingsGroup from "@/src/pages/admin/settings/(partials)/settings-group";
|
||||
import sumTokens from "../(functions)/sum-tokens";
|
||||
import UsageHarnessRow from "./usage-harness-row";
|
||||
|
||||
type Props = {
|
||||
rows: UsageOverview["by_harness"];
|
||||
};
|
||||
|
||||
/**
|
||||
* What each kind of model call spent: a chat turn, reading an image,
|
||||
* transcribing, speaking, making a picture, embedding a file. Their totals
|
||||
* add up to the period's.
|
||||
*/
|
||||
export default function UsageHarnessBreakdown({ rows }: Props) {
|
||||
if (!rows.length) {
|
||||
return (
|
||||
<SettingsGroup title="By kind of call" description="No model calls in this period yet.">
|
||||
<Center className="p-8">
|
||||
<Span size="small">Nothing to break down.</Span>
|
||||
</Center>
|
||||
</SettingsGroup>
|
||||
);
|
||||
}
|
||||
|
||||
const total_tokens = sumTokens({ rows });
|
||||
|
||||
return (
|
||||
<SettingsGroup title="By kind of call" description="Where the period's tokens went.">
|
||||
<Stack className="w-full gap-0!">
|
||||
<Row className="w-full pb-2 gap-3! items-center border-0 border-b border-solid border-hairline">
|
||||
<Span className="text-[11px] uppercase tracking-wide text-ink-secondary grow">
|
||||
Kind
|
||||
</Span>
|
||||
<Span className="text-[11px] uppercase tracking-wide text-ink-secondary w-10 text-right">
|
||||
Share
|
||||
</Span>
|
||||
<Span className="text-[11px] uppercase tracking-wide text-ink-secondary w-20 text-right">
|
||||
Tokens
|
||||
</Span>
|
||||
<Span className="text-[11px] uppercase tracking-wide text-ink-secondary w-16 text-right">
|
||||
Cost
|
||||
</Span>
|
||||
</Row>
|
||||
{rows.map((row) => (
|
||||
<UsageHarnessRow row={row} total_tokens={total_tokens} key={row.harness} />
|
||||
))}
|
||||
</Stack>
|
||||
</SettingsGroup>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
import Row from "@moduletrace/twui/layout/Row";
|
||||
import Span from "@moduletrace/twui/layout/Span";
|
||||
import type { UsageOverview } from "@/src/types";
|
||||
import formatTokens from "@/src/utils/format-tokens";
|
||||
import { UsageHarnessTitles } from "../(data)/usage-harness-titles";
|
||||
|
||||
type Props = {
|
||||
row: UsageOverview["by_harness"][number];
|
||||
/**
|
||||
* Every harness's tokens in the period, for the share.
|
||||
*/
|
||||
total_tokens: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* One harness's line in the breakdown: what it is, its share of the
|
||||
* period's tokens, and what it spent.
|
||||
*/
|
||||
export default function UsageHarnessRow({ row, total_tokens }: Props) {
|
||||
const share = total_tokens ? Math.round((row.total_tokens / total_tokens) * 100) : 0;
|
||||
|
||||
return (
|
||||
<Row className="w-full py-2 gap-3! items-center border-0 border-b border-solid border-hairline last:border-b-0">
|
||||
<Span className="text-[13px] leading-5 text-ink grow min-w-0 truncate">
|
||||
{UsageHarnessTitles[row.harness] || row.harness}
|
||||
</Span>
|
||||
<Span className="text-[12px] leading-4 text-ink-secondary tabular-nums w-10 text-right">
|
||||
{share}%
|
||||
</Span>
|
||||
<Span className="text-[13px] leading-5 text-ink tabular-nums w-20 text-right">
|
||||
{formatTokens({ tokens: row.total_tokens })}
|
||||
</Span>
|
||||
<Span className="text-[12px] leading-4 text-ink-secondary tabular-nums w-16 text-right">
|
||||
{row.cost ? `$${row.cost.toFixed(2)}` : "-"}
|
||||
</Span>
|
||||
</Row>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
import Span from "@moduletrace/twui/layout/Span";
|
||||
import { twMerge } from "tailwind-merge";
|
||||
|
||||
type Props = {
|
||||
title: string;
|
||||
className?: string;
|
||||
};
|
||||
|
||||
/**
|
||||
* A column heading of the models table, right-aligned over its figures.
|
||||
*/
|
||||
export default function UsageModelHeadCell({ title, className }: Props) {
|
||||
return (
|
||||
<Span
|
||||
className={twMerge(
|
||||
"text-[11px] uppercase tracking-wide text-ink-secondary text-right shrink-0",
|
||||
className,
|
||||
)}
|
||||
>
|
||||
{title}
|
||||
</Span>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
import Row from "@moduletrace/twui/layout/Row";
|
||||
import Span from "@moduletrace/twui/layout/Span";
|
||||
import Stack from "@moduletrace/twui/layout/Stack";
|
||||
import type { UsageOverview } from "@/src/types";
|
||||
import formatTokens from "@/src/utils/format-tokens";
|
||||
import { UsageHarnessTitles } from "../(data)/usage-harness-titles";
|
||||
|
||||
type Props = {
|
||||
row: UsageOverview["models"][number];
|
||||
};
|
||||
|
||||
/**
|
||||
* One model's line: which provider and model it was, what it was used for,
|
||||
* how many calls and tokens it took, and the price of a thousand tokens
|
||||
* worked out from both, which is what makes models comparable.
|
||||
*/
|
||||
export default function UsageModelRow({ row }: Props) {
|
||||
const per_thousand = row.cost && row.total_tokens
|
||||
? `$${((row.cost / row.total_tokens) * 1000).toFixed(3)}`
|
||||
: "-";
|
||||
|
||||
return (
|
||||
<Row className="w-full py-2.5 gap-3! items-center border-0 border-b border-solid border-hairline last:border-b-0">
|
||||
<Stack className="grow min-w-0 gap-0!">
|
||||
<Span className="text-[13px] leading-5 text-ink truncate" title={row.model_slug}>
|
||||
{row.model_name || row.model_slug}
|
||||
</Span>
|
||||
<Span className="text-[11px] leading-4 text-ink-secondary truncate">
|
||||
{row.provider_name} - {UsageHarnessTitles[row.harness] || row.harness}
|
||||
</Span>
|
||||
</Stack>
|
||||
<Span className="text-[12px] leading-4 text-ink-secondary tabular-nums w-14 text-right">
|
||||
{formatTokens({ tokens: row.calls })}
|
||||
</Span>
|
||||
<Span className="text-[12px] leading-4 text-ink-secondary tabular-nums w-16 text-right">
|
||||
{formatTokens({ tokens: row.prompt_tokens })}
|
||||
</Span>
|
||||
<Span className="text-[12px] leading-4 text-ink-secondary tabular-nums w-16 text-right">
|
||||
{formatTokens({ tokens: row.completion_tokens })}
|
||||
</Span>
|
||||
<Span className="text-[13px] leading-5 text-ink tabular-nums w-16 text-right">
|
||||
{formatTokens({ tokens: row.total_tokens })}
|
||||
</Span>
|
||||
<Span className="text-[12px] leading-4 text-ink-secondary tabular-nums w-16 text-right">
|
||||
{per_thousand}
|
||||
</Span>
|
||||
<Span className="text-[12px] leading-4 text-ink-secondary tabular-nums w-14 text-right">
|
||||
{row.cost ? `$${row.cost.toFixed(2)}` : "-"}
|
||||
</Span>
|
||||
</Row>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
import Center from "@moduletrace/twui/layout/Center";
|
||||
import Row from "@moduletrace/twui/layout/Row";
|
||||
import Span from "@moduletrace/twui/layout/Span";
|
||||
import Stack from "@moduletrace/twui/layout/Stack";
|
||||
import type { UsageOverview } from "@/src/types";
|
||||
import SettingsGroup from "@/src/pages/admin/settings/(partials)/settings-group";
|
||||
import UsageModelHeadCell from "./usage-model-head-cell";
|
||||
import UsageModelRow from "./usage-model-row";
|
||||
|
||||
type Props = {
|
||||
rows: UsageOverview["models"];
|
||||
days: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Every model the period ran, busiest first. A row's price per thousand
|
||||
* tokens is what makes one model's cost comparable with another's, and
|
||||
* how many calls each took says which one earned it.
|
||||
*/
|
||||
export default function UsageModelsTable({ rows, days }: Props) {
|
||||
if (!rows.length) {
|
||||
return (
|
||||
<SettingsGroup title="By model" description="No model calls in this period yet.">
|
||||
<Center className="p-8">
|
||||
<Span size="small">Nothing to list.</Span>
|
||||
</Center>
|
||||
</SettingsGroup>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<SettingsGroup
|
||||
title="By model"
|
||||
description={`The models this install ran in the last ${days} days, busiest first. The price per thousand tokens is read from what the provider reported.`}
|
||||
>
|
||||
<Stack className="w-full gap-0! overflow-x-auto">
|
||||
<Row className="w-full pb-2 gap-3! items-center border-0 border-b border-solid border-hairline min-w-[640px]">
|
||||
<UsageModelHeadCell className="grow text-left" title="Model" />
|
||||
<UsageModelHeadCell className="w-14" title="Calls" />
|
||||
<UsageModelHeadCell className="w-16" title="In" />
|
||||
<UsageModelHeadCell className="w-16" title="Out" />
|
||||
<UsageModelHeadCell className="w-16" title="Tokens" />
|
||||
<UsageModelHeadCell className="w-16" title="Per 1K" />
|
||||
<UsageModelHeadCell className="w-14" title="Cost" />
|
||||
</Row>
|
||||
{rows.map((row) => (
|
||||
<UsageModelRow
|
||||
row={row}
|
||||
key={`${row.provider_name}-${row.model_slug}-${row.harness}`}
|
||||
/>
|
||||
))}
|
||||
</Stack>
|
||||
</SettingsGroup>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
import Button from "@moduletrace/twui/layout/Button";
|
||||
import Row from "@moduletrace/twui/layout/Row";
|
||||
import { UsageRanges } from "../(data)/usage-ranges";
|
||||
|
||||
type Props = {
|
||||
days: number;
|
||||
loading?: boolean;
|
||||
onChange: (days: number) => void;
|
||||
};
|
||||
|
||||
/**
|
||||
* The periods the page totals, one button each. The one showing is the
|
||||
* one that reads as pressed.
|
||||
*/
|
||||
export default function UsageRangePicker({ days, loading, onChange }: Props) {
|
||||
return (
|
||||
<Row className="gap-1! items-center">
|
||||
{UsageRanges.map((range) => (
|
||||
<Button
|
||||
key={range.days}
|
||||
title={range.title}
|
||||
type="button"
|
||||
size="small"
|
||||
variant={range.days == days ? "outlined" : "ghost"}
|
||||
color="text"
|
||||
disabled={loading}
|
||||
onClick={() => onChange(range.days)}
|
||||
className="px-2.5! py-1! text-[12px]! rounded-default-xs"
|
||||
>
|
||||
{range.title}
|
||||
</Button>
|
||||
))}
|
||||
</Row>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
import Span from "@moduletrace/twui/layout/Span";
|
||||
import Stack from "@moduletrace/twui/layout/Stack";
|
||||
|
||||
type Props = {
|
||||
label: string;
|
||||
value: string;
|
||||
/**
|
||||
* A line under the figure, e.g. the comparison with an earlier period.
|
||||
*/
|
||||
hint?: string;
|
||||
};
|
||||
|
||||
/**
|
||||
* One figure of the page's totals: what it is above, the number itself
|
||||
* big, and a quiet line under it.
|
||||
*/
|
||||
export default function UsageStat({ label, value, hint }: Props) {
|
||||
return (
|
||||
<Stack className="grow basis-[160px] min-w-0 gap-1!">
|
||||
<Span className="text-[11px] uppercase tracking-wide text-ink-secondary">{label}</Span>
|
||||
<Span className="text-2xl leading-8 font-bold text-ink tabular-nums">{value}</Span>
|
||||
{hint ? (
|
||||
<Span className="text-[12px] leading-4 text-ink-secondary">{hint}</Span>
|
||||
) : null}
|
||||
</Stack>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
import Row from "@moduletrace/twui/layout/Row";
|
||||
import type { UsageOverview } from "@/src/types";
|
||||
import formatTokens from "@/src/utils/format-tokens";
|
||||
import { UsageHarnessTitles } from "../(data)/usage-harness-titles";
|
||||
import UsageStat from "./usage-stat";
|
||||
|
||||
type Props = {
|
||||
usage: UsageOverview;
|
||||
};
|
||||
|
||||
/**
|
||||
* The page's headline figures for the period: what it spent, how many
|
||||
* calls that took, the price when providers report it, the busiest
|
||||
* harness, and how the total compares with the period before it.
|
||||
*/
|
||||
export default function UsageTotals({ usage }: Props) {
|
||||
const { totals, previous, by_harness } = usage;
|
||||
const busiest = by_harness[0];
|
||||
|
||||
return (
|
||||
<Row className="w-full py-8 gap-6! items-start flex-wrap border-0 border-b border-solid border-hairline">
|
||||
<UsageStat
|
||||
label="Tokens"
|
||||
value={formatTokens({ tokens: totals.total_tokens })}
|
||||
hint={`${formatTokens({ tokens: totals.prompt_tokens })} in, ${formatTokens({ tokens: totals.completion_tokens })} out`}
|
||||
/>
|
||||
<UsageStat
|
||||
label="Model calls"
|
||||
value={formatTokens({ tokens: totals.calls })}
|
||||
hint={busiest ? `${UsageHarnessTitles[busiest.harness] || busiest.harness} most of it` : undefined}
|
||||
/>
|
||||
<UsageStat
|
||||
label="Cost"
|
||||
value={totals.cost ? `$${totals.cost.toFixed(2)}` : "Not reported"}
|
||||
hint={totals.cost ? "As the providers reported it" : "Free models, or no price"}
|
||||
/>
|
||||
<UsageStat
|
||||
label="Previous period"
|
||||
value={changeLabel({ current: totals.total_tokens, previous: previous.total_tokens })}
|
||||
hint={`${formatTokens({ tokens: previous.total_tokens })} then`}
|
||||
/>
|
||||
</Row>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* How this period compares with the one before it: a share up or down, or
|
||||
* that there is nothing to compare with yet.
|
||||
*/
|
||||
function changeLabel({ current, previous }: { current: number; previous: number }) {
|
||||
if (!previous) return current ? "New" : "Nothing yet";
|
||||
|
||||
const change = Math.round(((current - previous) / previous) * 100);
|
||||
|
||||
if (change == 0) return "Level";
|
||||
|
||||
return `${change > 0 ? "+" : ""}${change}%`;
|
||||
}
|
||||
Reference in New Issue
Block a user