Add dictation and streamed read-aloud to the chat

- A mic in the composer records, transcribes with the bot's
  transcription model (api/admin/bots/transcribe) and writes the text
  into the input at the cursor to edit before sending. Escape cancels;
  recordings aren't stored.
- Every bot message has a read-aloud button that plays
  GET api/admin/bots/speak-message, streamed from the provider as the
  audio is made, so it starts in a couple of seconds. One message plays
  at a time; only the user's own sessions can be read.
- Both buttons stay visible but disabled while the bot has no model for
  them (bot_voice page prop).
- Transcription and speech no longer pick free models automatically:
  they're not available until a model is picked, and Settings says so.
  Transcription lists speech-to-text models only, so the chat-model
  transcript path and its missing-audio check are gone.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

AGENTS.md

@@ -93,6 +93,7 @@ Inspect live data with `bun llm-code-assistant/query-db.ts "<sql>"` - a read-onl
 - **Server:** `src/functions/backend/db/bots/` (`grab-bots`, `grab-bot-messages`, `grab-active-bot-task`, `grab-user-bot`, `create-bot`, `insert-user-message`). `api/admin/bots/send-message.ts` posts a user message to a bot the session user chats with (zod: `src/schemas/bot-schema.ts`), queues the bot's reply (`enqueueBotReply`) and wakes the queue (`src/bots-queue/wake-bots-queue.ts`; never import `src/bots-queue/index.ts` from a page or API route, it pulls the bunext runtime into the page bundles). `upload-attachment.ts` stores one file for the next message (private `media` row, paradigm `bot-attachment`, type from `attachmentFileKind`, documents get `text_content`, limits in `AppData.BotAttachment*`) and `send-message` takes their `media_ids` (checked to be the user's own by `grabAttachableMedia`), linked in `bot_message_media`; messages carry `attachments` (`BotMessageWithAttachments`). `mark-read.ts` clears a session's unread when a message arrives in the open conversation; `session-messages.ts` returns a session's latest messages, which the open conversation polls every `AppData.BotReplyCatchUpMs` while a reply is due (`use-reply-catch-up`), so a reply whose live push was missed (socket down) still lands without a reload (replayed as a `server:bot-message`, so the rail and unread take it too); `create-bot.ts` adds a worker (and puts it in a workspace when given one); `save-harness-settings.ts` saves a worker bot's models.
 - **Message rendering:** bot messages are markdown, rendered with `react-markdown` + `remark-gfm` (raw HTML never rendered) through the twui-based map in `[bot_id]/(data)/message-markdown-components.tsx` (`markdown-link`, `markdown-list`, `markdown-code-block` partials). User messages stay plain text.
 - **Attachments and thinking:** the composer uploads files as they're picked, pasted or dropped (`use-composer-attachments`), shows them as chips, and sends once none is still uploading; the thread shows images as thumbnails and other files as chips. Replies read the files of the last `BotReplyAttachmentMessages` messages with any (`load-attachment-parts.ts`: images and audio as parts, documents as their text up to `BotAttachmentMaxTextChars`, videos as a note); older ones are only named. Tasks don't get the files yet. While the user has the last word in a session (every user message queues a reply, so this needs no face and shows even while the bot works in another session) or the bot waits for a model (`BotWaitingStatusLine`) or a task for its computer (`BotComputerBusyStatusLine`), `BotThinkingRow` sits under the last message; while a task runs it reads "<name> is working…" with the task's latest step, and a reply working through tools shows its latest step under "is thinking…" (`use-reply-progress`). A coordinator with no workers gets `CoordinatorSoloNote` instead of `CoordinatorReplyNote`, so it never claims to have handed work to a worker. Live notices come from the process that ran the job at once, and from every other process through the live relay.
+- **Voice:** the composer's mic (`(partials)/composer-voice-button.tsx`, `use-voice-recorder`, shown once the browser can record, disabled with a note while transcription has no model to run on for the bot there, `bot_voice` page prop, from `resolveHarnessCandidates`) dictates: the recording goes to `api/admin/bots/transcribe.ts` (`transcribeAudio` with the bot's models, after `grabUserConversation`) and the text lands in the input at the cursor, to edit and send; Escape cancels, `AppData.ComposerVoiceMaxMs` stops it. Each bot message has a play button (disabled with a note while speech has no model) (`speak-message-button.tsx`, one player per conversation via `SpeakMessageContext` / `use-speak-message`) that plays `GET api/admin/bots/speak-message.ts?message_id=` straight in an `<audio>` (`speakText` streamed through as the provider makes it, so it starts in a couple of seconds; a failure answers in JSON, which the hook fetches again to show; only the user's own sessions, `grabSpeakableMessage`; text from `speechText`, cut at `AppData.SpeechMaxChars`). Neither recordings nor audio are stored. Harness errors (`HarnessError`, safe to show) come back as the buttons' titles.
 - **Pictures:** a bot or workspace can have a picture (`image_media_id`, a private `media` row, paradigm `bot-image`/`workspace-image`), set or removed through `api/admin/set-profile-image.ts` (`setProfileImage`, the old one deleted; served at `profileImageUrls`). `BotAvatar` shows a bot's picture as a rounded square, else its stone; its hue still applies. `WorkspaceAvatar` shows a workspace's, else the workspace icon. A bot is edited from the header avatar (`(partials)/bot-profile-modal.tsx`, the same `BotForm` as adding one, `api/admin/bots/update-bot.ts`; the coordinator keeps its name and stays hue-less).
 - **Avatars:** `BotStone` (`src/components/general/bot-stone.tsx`) renders the stone inline from `src/data/stone-geometry.ts`, hue as a fill token, face (`idle|working|done|asleep|needs-you`) drawn as eyes. The coordinator is hue-less (ink body, paper eyes; inverts in dark). Never put a stone in a circle/badge or add outline/shadow. Wordmark is vector (`Wordmark`, `src/data/wordmark-paths.ts`), never live text.
 - **Copy helpers:** `src/utils/crew-status.ts` (`3 working · 1 needs you`, "Three bots are working. One needs you."), `src/utils/relative-time.ts` (relative timestamps only).
@@ -134,11 +135,11 @@ Envs are set in Settings, under Environment (super admin only), not in `.env`. T
 
 `src/harness/` is the brain: every model call goes through it. Install-wide, managed by super admins at `/admin/settings/models`.
 
-- **Harnesses** (`Harnesses` in `src/dict/harness-dict.ts`): `text` (chats, plans, calls tools, drives computers; always on), `vision` ("Media reading": images/scans/screenshots for a text model that can't see), `transcription` (speech to text), `speech` (text to speech), `image` (image generation). Each has one `harness_settings` row (no row = automatic, on, with fallback): a `provider_id` (a `harness_providers` row the harness is limited to, for its pick and its automatic picks; null for any provider; with no row, the default provider, OpenRouter, `grabDefaultHarnessProviderId`), a pinned `model_id` (a `harness_models` row from that provider, not a foreign key) or null for automatic, `fallback` (also try the automatic picks), `enabled`, `options` (`voice`).
+- **Harnesses** (`Harnesses` in `src/dict/harness-dict.ts`): `text` (chats, plans, calls tools, drives computers; always on), `vision` ("Media reading": images/scans/screenshots for a text model that can't see), `transcription` (speech to text: speech-to-text models only, never chat models that hear audio), `speech` (text to speech: text-to-speech models only), `image` (image generation). Each has one `harness_settings` row (no row = automatic, on, with fallback): a `provider_id` (a `harness_providers` row the harness is limited to, for its pick and its automatic picks; null for any provider; with no row, the default provider, OpenRouter, `grabDefaultHarnessProviderId`), a pinned `model_id` (a `harness_models` row from that provider, not a foreign key) or null for automatic, `fallback` (also try the automatic picks), `enabled`, `options` (`voice`).
 - **Providers** (`HarnessProviderKinds` in `src/dict/harness-provider-dict.ts`): OpenRouter (default, created by `ensureDefaultHarnessProvider`), OpenAI, Anthropic, Gemini, DeepSeek, Groq, Mistral, xAI, Ollama (local, its models count as free), any OpenAI-compatible server. Rows in `harness_providers` (a private table: the key is encrypted with `src/utils/crypt.ts`, never sent to the client; falls back to the preset's env, e.g. `OPENROUTER_API_KEY`, via `grabEnv`). One adapter today, `src/harness/providers/openai-compatible/` (`chat`, `transcribe`, `speak`, `generateImage`, `listModels`), chosen in `grab-harness-adapter.ts`; a provider needing a native API gets its own adapter there (`HarnessAdapter` in `src/harness/types.ts`).
 - **Catalog:** `harness_models` per provider (modalities, tools support, voices, Artificial Analysis scores, `free`, `available`). The `sync-harness-models` cron (hourly, on start) refreshes every enabled provider (`syncAllHarnessModels`); unlisted models are kept but marked unavailable. OpenRouter lists every modality with capabilities; other providers list ids, so capabilities are guessed (`guess-model-modalities`). Free on OpenRouter means `:free` or `openrouter/free`, never zero token prices.
-- **Automatic picks** (`rank-auto-models`): free models that can do the job (on the setting's provider when it has one; a provider without free models needs a picked model) (`model-serves-harness`: text needs tools), by vendor `AutoModelVendorPreference` (DeepSeek, Qwen, NVIDIA, Liquid), then other vendors that look competent (`AppData.HarnessAutoMin*`), then routers; within a vendor by intelligence index, then full-size before small, newest first. Transcription puts real speech-to-text models first.
+- **Automatic picks** (`rank-auto-models`): only for the `automatic` harnesses in `Harnesses` (text, vision, image); transcription and speech have none and are not available until a model is picked (the model select reads "Not available", no fallback checkbox). Free models that can do the job (on the setting's provider when it has one; a provider without free models needs a picked model) (`model-serves-harness`: text needs tools), by vendor `AutoModelVendorPreference` (DeepSeek, Qwen, NVIDIA, Liquid), then other vendors that look competent (`AppData.HarnessAutoMin*`), then routers; within a vendor by intelligence index, then full-size before small, newest first.
-- **Calling:** `resolveHarnessCandidates` (pinned, then automatic) -> `withCandidates` tries them in order with process-wide cooldowns (`resolve/harness-cooldowns.ts`): rate limits and outages rest the model (until `Retry-After` when given), 401/402 rest the whole provider, 403/404 rest the model for an hour. Errors are `HarnessError` (`code` provider|unavailable|aborted, `retryable`), classified in `to-harness-error.ts`. Entry points: `text/generate-text.ts` (one turn), `text/run-text-loop.ts` (tool loop, sticks to one model, wraps up after `HarnessMaxToolSteps`, trims the oldest tool output before each turn to fit the model's context, `fit-transcript-to-context.ts`), `media/read-image.ts`, `media/read-media.ts` (documents locally via `fileToText`, images and audio through their harnesses), `audio/transcribe-audio.ts`, `audio/speak-text.ts`, `image/generate-image.ts`. A text model that can't take an image or audio part gets it read by the vision/transcription harness first (`prepare-messages-for-model.ts`).
+- **Calling:** `resolveHarnessCandidates` (pinned, then automatic) -> `withCandidates` tries them in order with process-wide cooldowns (`resolve/harness-cooldowns.ts`): rate limits and outages rest the model (until `Retry-After` when given), 401/402 rest the whole provider, 403/404 rest the model for an hour. Errors are `HarnessError` (`code` provider|unavailable|aborted, `retryable`), classified in `to-harness-error.ts`. Entry points: `text/generate-text.ts` (one turn), `text/run-text-loop.ts` (tool loop, sticks to one model, wraps up after `HarnessMaxToolSteps`, trims the oldest tool output before each turn to fit the model's context, `fit-transcript-to-context.ts`), `media/read-image.ts`, `media/read-media.ts` (documents locally via `fileToText`, images and audio through their harnesses), `audio/transcribe-audio.ts`, `audio/speak-text.ts` (returns the provider's audio as a stream, falling back to the next model only until one answers), `image/generate-image.ts`. A text model that can't take an image or audio part gets it read by the vision/transcription harness first (`prepare-messages-for-model.ts`).
 - **Per-bot models:** in a workspace, its `workspace_harness_settings` row wins (`source: "workspace"`). Otherwise a worker bot inherits the defaults per harness unless it has a `bot_harness_settings` row (same fields as `harness_settings`, unique per bot and harness). `grabHarnessSetting({ harness, bot_id })` returns the bot's row (`source: "bot"`) or the default; the coordinator always gets the default. Every entry point takes `bot_id` and passes it on (text loop -> media reading/transcription), so a bot's run uses its own models throughout. Saved by `saveBotHarnessSetting` (`inherit` deletes the row); the page data is `grabBotHarnessOverview`. A bot's chat header has a ghost model icon (`(partials)/header-icon-button.tsx`; the text model and provider show under the bot's name) that opens a modal with every harness (`(partials)/bot-models-modal.tsx`); the modal and `/admin/bots/[bot_id]/models` both use `(partials)/bot-harness-form.tsx`. On the coordinator the modal edits the defaults themselves (`api/admin/harness/save-settings`, the same settings as Settings, under Models, without the providers): super admins only, the icon is disabled for everyone else. Every one of these wraps `settings/models/(partials)/harness-settings-form.tsx`: per harness a provider select first (`harness-provider-select.tsx`: any provider, one provider, or on a bot "Same as the default"), then the model select narrowed to it. Removing a provider (`deleteHarnessProvider`) puts default and bot settings that picked it or its models back on any provider (`save` prop; "Same as the default" shows when the overview item has `default`).
 - **Tools:** `HarnessTool` (built per run with its bot bound): own tools validate with a zod `schema`; tools described elsewhere pass a JSON schema as `parameters` and validate themselves. `run-tool-call.ts` reports bad calls and failures back to the model as `Error: ...`. Bot tools in `src/harness/tools/bot/`: `update_memory`, `update_instructions` (`grabBotTools`, every run), `start_task` and, for the coordinator with workers, `delegate_task` (`grabBotChatTools`, chat replies only; the job's `requested_by_bot_id` is the coordinator, and `relayJobToRequester` posts the worker's result or failure in its conversation, pushed as `relay` by `pushBotNotice`), `ask_user_to_take_over` (see Take-over), plus for the coordinator `create_workspace` and `update_workspace` (a new workspace with its shared context, or changing one; both also in its tasks, so a task that gathers material can write it into the workspace), `create_bot` (add a worker with its name, role and instructions, and join it to a workspace straight away) and, in its own chat, `add_bot_to_workspace` (put an existing worker in one of the user's workspaces) and `remove_bot_from_workspace` (take it out, keeping its conversation there), `assign_computer` (grant or revoke a worker's direct access, `add/remove-bot-to/from-computer`; access through a workspace stays and the result says so), `update_bot` (change a worker's name, role or core instructions, `src/functions/backend/db/bots/update-bot.ts`), `find_bot` and `find_workspace` (look the crew, or the live workspaces, up by id, name, role/description or instructions/context), `delete_bot` (a worker with its conversations, tasks, schedules and workspace places; the coordinator can't go) and `delete_workspace` (its conversations, tasks, schedules and links; its bots and computers stay). `search_computers` (`src/harness/tools/computer/search-computers-tool.ts`, chat replies and tasks) runs one read-only command on all of a bot's computers at once over the sandbox REST API (`run-computer-command.ts`), each output under its `[computer:ID]`, secrets redacted, capped by `AppData.ComputerSearch*`.
 - **MCP:** `src/harness/mcp/`: `connectMcpServer` (Streamable HTTP, SSE fallback), `grabMcpTools` (each server tool becomes a `HarnessTool` with its JSON schema), `mcpResultToText` (text as is, images read by the vision harness, `isError` thrown). Computers are reached this way: `openComputerTools` (`src/harness/tools/computer/`) connects to the computer's AIO sandbox at `grabComputerMcpUrl` (the `MAIN_COMPUTER_URL` app setting for the main computer, for a dev server outside docker, e.g. `http://localhost:8170`, else `http://<hostname>:<port>/mcp`, or its IP address when the hostname can't be reached), leaves out `ComputerMcpExcludedTools`, and sets the computer's envs (see Envs).

src/data/app-data.ts

@@ -287,6 +287,14 @@ export const AppData = {
      * themselves to a reply; older ones are named.
      */
     BotReplyAttachmentMessages: 4,
+    /**
+     * Longest dictation the composer records before it stops on its own.
+     */
+    ComposerVoiceMaxMs: 5 * 60 * 1000,
+    /**
+     * Most of a bot message read aloud; the rest is cut.
+     */
+    SpeechMaxChars: 4000,
     /**
      * Largest picture a bot or a workspace can have.
      */

src/data/harness-prompts.ts

@@ -8,8 +8,6 @@ export const ReadImagePrompt = `Describe this image for someone who can't see it
 - For a screenshot, name the app or page, its state, and every control a user could click.
 - Don't guess at what you can't see. Plain text only, no preamble.`;
 
-export const TranscribeAudioPrompt = `Transcribe this audio exactly as spoken, in the language spoken. Output only the transcript: no preamble, no notes, no timestamps.`;
-
 /**
  * Added to the text model's prompt when a message part was read by another
  * harness, so it knows it is working from a description.

src/dict/harness-dict.ts

@@ -1,36 +1,43 @@
 /**
  * The harnesses: what a model is used for. Each harness has one model
- * setting (`harness_settings`), automatic by default. Text is the brain and
+ * setting (`harness_settings`). Text is the brain and can't be turned off;
- * can't be turned off; the others are helpers it calls on.
+ * the others are helpers it calls on. `automatic` ones pick a free model
+ * themselves when none is picked; the rest (transcription, speech) are
+ * not available until one is, since free picks there rarely work.
  */
 export const Harnesses = [
     {
         title: "Text",
         value: "text",
+        automatic: true,
         description: `Chats, plans and works on computers. Every bot thinks with it.`,
         required: true,
     },
     {
         title: "Media reading",
         value: "vision",
+        automatic: true,
         description: `Reads images, scans and screenshots for the text model when it can't see them itself.`,
         required: false,
     },
     {
         title: "Transcription",
         value: "transcription",
+        automatic: false,
         description: `Turns voice messages and recordings into text.`,
         required: false,
     },
     {
         title: "Speech",
         value: "speech",
+        automatic: false,
         description: `Reads bot replies aloud.`,
         required: false,
     },
     {
         title: "Image generation",
         value: "image",
+        automatic: true,
         description: `Makes images when a bot is asked for one.`,
         required: false,
     },

src/functions/backend/db/bot-sessions/grab-conversation-props.ts

@@ -6,6 +6,7 @@ import grabBotExtensionsOverview from "../extensions/grab-bot-extensions-overvie
 import grabBotComputers from "../computers/grab-bot-computers";
 import grabBotHarnessOverview from "../harness/grab-bot-harness-overview";
 import grabHarnessOverview from "../harness/grab-harness-overview";
+import resolveHarnessCandidates from "@/src/harness/resolve/resolve-harness-candidates";
 import grabUserWorkspaces from "../workspaces/grab-user-workspaces";
 import ensureBotSession from "./ensure-bot-session";
 import grabUserConversation from "./grab-user-conversation";
@@ -25,7 +26,8 @@ type Params = {
  * `/admin/workspaces/<id>/bots/<id>`): the rail's crew (the workspace's
  * in a workspace) and workspaces, the bot, its active session (marked
  * read) and its messages, its models, computers, tools and MCP servers
- * and active task there.
+ * and active task there, and whether its mic and read-aloud have a model
+ * to run on.
  * Null when the user can't chat with the bot there.
  */
 export default async function grabConversationProps({
@@ -64,7 +66,7 @@ export default async function grabConversationProps({
      * workspace the workspace's own models come first, so what runs now
      * shows its pick.
      */
-    const [bot_messages, bot_harness_overview, bot_computers, bot_task, bot_extensions] = await Promise.all([
+    const [bot_messages, bot_harness_overview, bot_computers, bot_task, bot_extensions, transcription, speech] = await Promise.all([
         grabBotMessages({ session_id: session.id! }),
         (bot.kind == "coordinator" && !scope
             ? grabHarnessOverview().then(({ harnesses }) => ({ harnesses }))
@@ -76,6 +78,8 @@ export default async function grabConversationProps({
         grabBotComputers({ bot_id: bot.id, workspace_id: scope }),
         grabActiveBotTask({ bot_id: bot.id, session_id: session.id }),
         grabBotExtensionsOverview({ user_id, bot_id: bot.id, workspace_id: scope }),
+        hasHarnessModel({ harness: "transcription", bot_id: bot.id, workspace_id: scope }),
+        hasHarnessModel({ harness: "speech", bot_id: bot.id, workspace_id: scope }),
     ]);
 
     return {
@@ -89,5 +93,15 @@ export default async function grabConversationProps({
         bot_computers,
         bot_task,
         bot_extensions,
+        bot_voice: { transcription, speech },
     };
 }
+
+/**
+ * Whether the harness is on and has a model to run on for the bot there.
+ */
+async function hasHarnessModel(params: Parameters<typeof resolveHarnessCandidates>[0]) {
+    return await resolveHarnessCandidates(params)
+        .then((candidates) => candidates.length > 0)
+        .catch(() => false);
+}

src/functions/backend/db/bots/grab-speakable-message.ts

@@ -0,0 +1,37 @@
+import BunSQLite from "@moduletrace/bun-sqlite";
+import type { BUN_SQLITE_COBBLE_BOT_MESSAGES } from "@/db/types/db";
+import AppError from "@/src/utils/app-error";
+import grabUserConversation from "../bot-sessions/grab-user-conversation";
+
+type Params = {
+    user_id: number;
+    message_id: number;
+};
+
+/**
+ * A bot message in one of the user's own sessions, for reading aloud,
+ * with the conversation it's in (so a lost grant blocks it). Throws when
+ * it isn't theirs.
+ */
+export default async function grabSpeakableMessage({ user_id, message_id }: Params) {
+    const message = (
+        await BunSQLite.sql<BUN_SQLITE_COBBLE_BOT_MESSAGES>({
+            sql: `SELECT m.* FROM bot_messages m
+                JOIN bot_sessions s ON s.id = m.session_id
+                WHERE m.id = ? AND m.author = 'bot' AND s.user_id = ?`,
+            values: [message_id, user_id],
+        })
+    ).payload?.[0];
+
+    if (!message?.id || !message.bot_id) {
+        throw new AppError(`Message not found`);
+    }
+
+    const conversation = await grabUserConversation({
+        user_id,
+        bot_id: message.bot_id,
+        workspace_id: message.workspace_id || null,
+    });
+
+    return { message, conversation };
+}

src/functions/backend/db/harness/grab-harness-overview-item.ts

@@ -55,6 +55,7 @@ export default async function grabHarnessOverviewItem({
         title: meta.title,
         description: meta.description,
         required: meta.required,
+        automatic: meta.automatic,
         model_id: setting.model_id,
         provider_id: setting.provider_id,
         providers,

src/harness/audio/looks-like-missing-audio.test.ts

@@ -1,27 +0,0 @@
-import { describe, expect, test } from "bun:test";
-import looksLikeMissingAudio from "./looks-like-missing-audio";
-
-describe("looksLikeMissingAudio", () => {
-    test("spots a model asking for the audio it never got", () => {
-        for (const text of [
-            "Please provide the audio file you’d like transcribed.",
-            "I’m sorry, but I can’t transcribe without the audio.",
-            "I cannot hear any audio in your message.",
-            "There is no audio attached.",
-            "I didn't receive any audio.",
-        ]) {
-            expect(looksLikeMissingAudio({ text })).toBe(true);
-        }
-    });
-
-    test("leaves real transcripts alone", () => {
-        expect(looksLikeMissingAudio({ text: "Hello from cobble." })).toBe(
-            false,
-        );
-        expect(
-            looksLikeMissingAudio({
-                text: `Can you send the audio report to finance? ${"And then more. ".repeat(30)}`,
-            }),
-        ).toBe(false);
-    });
-});

src/harness/audio/looks-like-missing-audio.ts

@@ -1,14 +0,0 @@
-type Params = {
-    text: string;
-};
-
-const MissingAudio =
-    /(provide|upload|share|attach|send)\b.{0,40}\baudio|can(?:'|’)?t (?:transcribe|hear|access|listen)|cannot (?:transcribe|hear|access|listen)|no audio|without (?:the |an |any )?audio|didn(?:'|’)?t (?:receive|get) (?:the |an |any )?audio/i;
-
-/**
- * Whether a chat model's "transcript" is really it saying it got no audio.
- * Some providers list audio input for a model but drop the audio.
- */
-export default function looksLikeMissingAudio({ text }: Params) {
-    return text.length < 300 && MissingAudio.test(text);
-}

src/harness/audio/speak-text.ts

@@ -22,8 +22,10 @@ type Params = {
 };
 
 /**
- * Reads text aloud with the speech harness. Uses the voice picked in the
+ * Reads text aloud with the speech harness, as a stream that can play
- * speech setting when the model has it, else the model's first voice.
+ * while the rest is still being made. Falls back to the next model only
+ * until one answers. Uses the voice picked in the speech setting when the
+ * model has it, else the model's first voice.
  */
 export default async function speakText({
     text,

src/harness/audio/transcribe-audio.ts

@@ -1,17 +1,10 @@
 import { AppData } from "@/src/data/app-data";
-import { TranscribeAudioPrompt } from "@/src/data/harness-prompts";
 import AppError from "@/src/utils/app-error";
 import HarnessError from "../harness-error";
 import grabHarnessAdapter from "../providers/grab-harness-adapter";
 import resolveHarnessCandidates from "../resolve/resolve-harness-candidates";
 import withCandidates from "../resolve/with-candidates";
 import type { HarnessAudioInput } from "../types";
-import looksLikeMissingAudio from "./looks-like-missing-audio";
-
-/**
- * A chat model that drops audio is skipped this long.
- */
-const MissingAudioCooldownMs = 60 * 60 * 1000;
 
 type Params = {
     audio: HarnessAudioInput;
@@ -31,9 +24,8 @@ type Params = {
 };
 
 /**
- * Turns speech into text with the transcription harness. Speech-to-text
+ * Turns speech into text with the transcription harness: its
- * models get the file at `/audio/transcriptions`; chat models that hear
+ * speech-to-text model gets the file at `/audio/transcriptions`.
- * audio are asked for a verbatim transcript.
  */
 export default async function transcribeAudio({
     audio,
@@ -62,40 +54,12 @@ export default async function transcribeAudio({
                 kind: candidate.provider.kind,
             });
 
-            const is_speech_to_text =
+            const { text } = await adapter.transcribe({
-                candidate.model.output_modalities.includes("transcription");
-
-            const text = is_speech_to_text
-                ? (
-                      await adapter.transcribe({
                 candidate,
                 audio,
                 language,
                 signal: final_signal,
-                      })
+            });
-                  ).text
-                : (
-                      await adapter.chat({
-                          candidate,
-                          messages: [
-                              {
-                                  role: "user",
-                                  content: [
-                                      { type: "text", text: TranscribeAudioPrompt },
-                                      {
-                                          type: "audio",
-                                          data: Buffer.from(audio.data).toString(
-                                              "base64",
-                                          ),
-                                          format: audio.format,
-                                      },
-                                  ],
-                              },
-                          ],
-                          temperature: 0,
-                          signal: final_signal,
-                      })
-                  ).text;
 
             if (text === null || text === undefined) {
                 throw new HarnessError({
@@ -105,15 +69,6 @@ export default async function transcribeAudio({
                 });
             }
 
-            if (!is_speech_to_text && looksLikeMissingAudio({ text })) {
-                throw new HarnessError({
-                    message: `${candidate.model.name || candidate.model.slug} didn't get the audio.`,
-                    retryable: true,
-                    retry_at: Date.now() + MissingAudioCooldownMs,
-                    candidate_key: candidate.key,
-                });
-            }
-
             return text.trim();
         },
     });

src/harness/catalog/model-serves-harness.test.ts

@@ -30,7 +30,7 @@ describe("modelServesHarness", () => {
         ).toBe(true);
     });
 
-    test("transcription takes speech-to-text or audio-hearing chat models", () => {
+    test("transcription takes speech-to-text models only", () => {
         expect(
             modelServesHarness({
                 model: {
@@ -46,7 +46,7 @@ describe("modelServesHarness", () => {
                 model: { ...chat, input_modalities: ["text", "audio"] },
                 harness: "transcription",
             }),
-        ).toBe(true);
+        ).toBe(false);
         expect(
             modelServesHarness({ model: chat, harness: "transcription" }),
         ).toBe(false);

src/harness/catalog/model-serves-harness.ts

@@ -13,7 +13,8 @@ type Params = {
  * Whether a model can do a harness's job:
  * - text: reads and writes text, and calls tools (bots work through tools)
  * - vision: reads images, writes text
- * - transcription: a speech-to-text model, or a chat model that hears audio
+ * - transcription: a speech-to-text model (never a chat model that hears
+ *   audio: free ones often drop it)
  * - speech: a text-to-speech model
  * - image: makes images
  */
@@ -31,10 +32,7 @@ export default function modelServesHarness({ model, harness }: Params) {
             return input.has("image") && output.has("text");
 
         case "transcription":
-            return (
+            return output.has("transcription");
-                output.has("transcription") ||
-                (input.has("audio") && output.has("text"))
-            );
 
         case "speech":
             return output.has("speech");

src/harness/catalog/rank-auto-models.test.ts

@@ -106,28 +106,24 @@ describe("rankAutoModels", () => {
             "google/gemma-4-31b-it:free",
             "openrouter/free",
         ]);
-        expect(
-            slugs(rankAutoModels({ models: live, harness: "transcription" })),
-        ).toEqual(["thinkingmachines/inkling-small:free"]);
-        expect(rankAutoModels({ models: live, harness: "speech" })).toEqual(
-            [],
-        );
     });
 
-    test("puts speech-to-text models ahead of chat models that hear audio", () => {
+    test("never picks for transcription or speech", () => {
-        const ranked = rankAutoModels({
+        const models = [
-            models: [
             model("deepseek/omni", { input_modalities: ["text", "audio"] }),
             model("acme/scribe", {
                 input_modalities: ["audio"],
                 output_modalities: ["transcription"],
                 supports_tools: false,
             }),
-            ],
+            model("acme/voice", {
-            harness: "transcription",
+                output_modalities: ["speech"],
-        });
+                supports_tools: false,
+            }),
+        ];
 
-        expect(slugs(ranked)).toEqual(["acme/scribe", "deepseek/omni"]);
+        expect(rankAutoModels({ models, harness: "transcription" })).toEqual([]);
+        expect(rankAutoModels({ models, harness: "speech" })).toEqual([]);
     });
 
     test("drops retired and unscored small models from other vendors", () => {

src/harness/catalog/rank-auto-models.ts

@@ -2,6 +2,7 @@ import { AppData } from "@/src/data/app-data";
 import {
     AutoModelRouterSlugs,
     AutoModelVendorPreference,
+    Harnesses,
     SmallModelPattern,
     SpecialistModelPattern,
     type HarnessKind,
@@ -26,13 +27,16 @@ type Params<T extends HarnessCatalogModel> = {
  * 3. routers such as `openrouter/free`, last
  *
  * Within a group, benchmarked models come first by intelligence index,
- * then the rest: full-size before small variants, newest first.
+ * then the rest: full-size before small variants, newest first. None for
+ * a harness that isn't `automatic` (transcription, speech).
  */
 export default function rankAutoModels<T extends HarnessCatalogModel>({
     models,
     harness,
     now = Date.now(),
 }: Params<T>): T[] {
+    if (!Harnesses.find((h) => h.value == harness)?.automatic) return [];
+
     const ranked = models
         .filter(
             (model) =>
@@ -50,23 +54,12 @@ export default function rankAutoModels<T extends HarnessCatalogModel>({
     return ranked
         .sort(
             (a, b) =>
-                (harness == "transcription"
-                    ? Number(!isSpeechToText(a.model)) -
-                      Number(!isSpeechToText(b.model))
-                    : 0) ||
                 a.tier - b.tier ||
                 compareInTier(a.model, b.model),
         )
         .map((entry) => entry.model);
 }
 
-/**
- * Dedicated speech-to-text models transcribe before chat models that hear
- * audio: some free chat endpoints list audio input but drop it.
- */
-function isSpeechToText(model: HarnessCatalogModel) {
-    return model.output_modalities.includes("transcription");
-}
 
 const OtherTier = AutoModelVendorPreference.length;
 const RouterTier = OtherTier + 1;

src/harness/providers/openai-compatible/speak.ts

@@ -1,8 +1,10 @@
 import type { HarnessAdapter } from "../../types";
+import HarnessError from "../../harness-error";
 import providerRequest from "./provider-request";
 
 /**
- * `POST /audio/speech`. Returns the audio bytes.
+ * `POST /audio/speech`. Resolves once the provider answers, with its
+ * audio as a stream (providers send it in chunks as it's made).
  */
 const speak: HarnessAdapter["speak"] = async ({
     candidate,
@@ -24,8 +26,16 @@ const speak: HarnessAdapter["speak"] = async ({
         candidate_key: candidate.key,
     });
 
+    if (!res.body) {
+        throw new HarnessError({
+            message: `The model returned no audio.`,
+            retryable: true,
+            candidate_key: candidate.key,
+        });
+    }
+
     return {
-        data: new Uint8Array(await res.arrayBuffer()),
+        stream: res.body,
         media_type:
             res.headers.get("content-type")?.split(";")[0] ||
             `audio/${format == "mp3" ? "mpeg" : format}`,

src/harness/types.ts

@@ -174,8 +174,12 @@ export type HarnessSpeakParams = {
     signal: AbortSignal;
 };
 
-export type HarnessAudioOutput = {
+/**
-    data: Uint8Array;
+ * Speech as the provider sends it: the stream starts once the provider
+ * answers, so it can play while the rest is still being made.
+ */
+export type HarnessAudioStream = {
+    stream: ReadableStream<Uint8Array>;
     media_type: string;
 };
 
@@ -206,7 +210,7 @@ export type HarnessImageOutput = {
 export type HarnessAdapter = {
     chat: (params: HarnessChatParams) => Promise<HarnessChatResult>;
     transcribe: (params: HarnessTranscribeParams) => Promise<{ text: string }>;
-    speak: (params: HarnessSpeakParams) => Promise<HarnessAudioOutput>;
+    speak: (params: HarnessSpeakParams) => Promise<HarnessAudioStream>;
     generateImage: (
         params: HarnessImageParams,
     ) => Promise<{ images: HarnessImageOutput[] }>;

src/pages/admin/bots/[bot_id]/(data)/speak-message-context.ts

@@ -0,0 +1,13 @@
+import React from "react";
+import type { SpeakMessage } from "../(hooks)/use-speak-message";
+
+/**
+ * The conversation's read-aloud, shared by every bot message's play
+ * button.
+ */
+export const SpeakMessageContext = React.createContext<SpeakMessage>({
+    enabled: false,
+    message_id: null,
+    state: "idle",
+    toggle: () => {},
+});

src/pages/admin/bots/[bot_id]/(functions)/transcribe-voice.ts

@@ -0,0 +1,37 @@
+import fetchApi from "@moduletrace/twui/utils/fetch/fetchApi";
+import fileInputToBase64 from "@moduletrace/twui/utils/form/fileInputToBase64";
+import type { APIResponseObject } from "@moduletrace/bunext/types";
+import type { TranscribeVoiceRequest } from "@/src/schemas/bot-schema";
+
+type Params = {
+    bot_id: number;
+    workspace_id?: number | null;
+    audio: Blob;
+};
+
+/**
+ * Sends a composer dictation to be turned into text with the bot's
+ * transcription model.
+ */
+export default async function transcribeVoice({
+    bot_id,
+    workspace_id,
+    audio,
+}: Params): Promise<APIResponseObject<{ text: string }>> {
+    const ext = audio.type.split("/").pop() || "webm";
+    const { fileBase64Full } = await fileInputToBase64({
+        inputFile: new File([audio], `dictation.${ext}`, { type: audio.type }),
+    });
+
+    if (!fileBase64Full) {
+        return { success: false, msg: `Couldn't read the recording.` };
+    }
+
+    return await fetchApi<
+        TranscribeVoiceRequest,
+        APIResponseObject<{ text: string }>
+    >(`/api/admin/bots/transcribe`, {
+        method: "POST",
+        body: { bot_id, workspace_id, data_url: fileBase64Full },
+    });
+}

src/pages/admin/bots/[bot_id]/(hooks)/use-speak-message.ts

@@ -0,0 +1,123 @@
+import { useEffect, useRef, useState } from "react";
+
+export type SpeakMessageState = "idle" | "loading" | "playing";
+
+export type SpeakMessage = {
+    /**
+     * The bot has a speech model here; the play buttons are disabled
+     * without one.
+     */
+    enabled: boolean;
+    /**
+     * The message being fetched or played, if any.
+     */
+    message_id: number | null;
+    state: SpeakMessageState;
+    error?: { message_id: number; text: string };
+    /**
+     * Plays the message, or stops it when it's the one playing. Starting
+     * one stops any other.
+     */
+    toggle: (params: { message_id: number }) => void;
+};
+
+/**
+ * The conversation's read-aloud: one audio element for every message,
+ * pointed at `api/admin/bots/speak-message`, which streams, so playing
+ * starts before the whole message is spoken. Stopping drops the stream.
+ */
+export default function useSpeakMessage({ enabled }: { enabled: boolean }): SpeakMessage {
+    const [message_id, setMessageId] = useState<number | null>(null);
+    const [state, setState] = useState<SpeakMessageState>("idle");
+    const [error, setError] = useState<SpeakMessage["error"]>();
+    const audio_ref = useRef<HTMLAudioElement | null>(null);
+    const current_ref = useRef<number | null>(null);
+
+    useEffect(() => () => release(), []);
+
+    function release() {
+        const audio = audio_ref.current;
+
+        if (!audio) return;
+
+        audio.pause();
+        audio.removeAttribute("src");
+        audio.load();
+    }
+
+    function reset() {
+        current_ref.current = null;
+        setMessageId(null);
+        setState("idle");
+    }
+
+    function stop() {
+        release();
+        reset();
+    }
+
+    function play({ message_id }: { message_id: number }) {
+        stop();
+        setError(undefined);
+        current_ref.current = message_id;
+        setMessageId(message_id);
+        setState("loading");
+
+        const src = `/api/admin/bots/speak-message?message_id=${message_id}`;
+        const audio = audio_ref.current || new Audio();
+        const isCurrent = () => current_ref.current == message_id;
+
+        audio_ref.current = audio;
+        audio.onplaying = () => {
+            if (isCurrent()) setState("playing");
+        };
+        audio.onended = () => {
+            if (isCurrent()) reset();
+        };
+        audio.onerror = () => {
+            if (!isCurrent()) return;
+
+            reset();
+            explainError({ src }).then((text) => {
+                setError({ message_id, text });
+            });
+        };
+        audio.src = src;
+        audio.play().catch(() => {});
+    }
+
+    function toggle({ message_id: id }: { message_id: number }) {
+        if (current_ref.current == id) {
+            stop();
+            return;
+        }
+
+        play({ message_id: id });
+    }
+
+    return { enabled, message_id, state, error, toggle };
+}
+
+/**
+ * The player only says it failed, so ask again for the reason: a failure
+ * answers in JSON, and audio coming back instead is dropped at once.
+ */
+async function explainError({ src }: { src: string }) {
+    const fallback = "Couldn't read it aloud. Try again.";
+    const controller = new AbortController();
+
+    try {
+        const res = await fetch(src, { signal: controller.signal, cache: "no-store" });
+
+        if (!res.headers.get("content-type")?.includes("json")) {
+            controller.abort();
+            return fallback;
+        }
+
+        const json = await res.json();
+
+        return json?.msg || fallback;
+    } catch {
+        return fallback;
+    }
+}

src/pages/admin/bots/[bot_id]/(hooks)/use-voice-recorder.ts

@@ -0,0 +1,136 @@
+import { useEffect, useRef, useState } from "react";
+import { AppData } from "@/src/data/app-data";
+
+export type VoiceRecorderState = "idle" | "recording" | "transcribing";
+
+type Params = {
+    /**
+     * Gets the finished recording. The hook stays `transcribing` until it
+     * resolves.
+     */
+    onRecorded: (params: { audio: Blob }) => Promise<void>;
+};
+
+/**
+ * Types the browser may record in, best first. Safari only has mp4.
+ */
+const RecordingTypes = ["audio/webm;codecs=opus", "audio/webm", "audio/mp4", "audio/ogg"];
+
+/**
+ * Records the microphone for the composer's dictation. Stops on its own
+ * after `AppData.ComposerVoiceMaxMs`; Escape cancels. The mic is released whenever
+ * recording ends, and on unmount. `supported` is false without a secure
+ * context or `MediaRecorder`.
+ */
+export default function useVoiceRecorder({ onRecorded }: Params) {
+    const [state, setState] = useState<VoiceRecorderState>("idle");
+    const [elapsed_ms, setElapsedMs] = useState(0);
+    const [error, setError] = useState<string>();
+    const [supported, setSupported] = useState(false);
+    const recorder_ref = useRef<MediaRecorder | null>(null);
+    const cancelled_ref = useRef(false);
+    const on_recorded_ref = useRef(onRecorded);
+
+    on_recorded_ref.current = onRecorded;
+
+    useEffect(() => {
+        setSupported(
+            typeof MediaRecorder != "undefined" &&
+                Boolean(navigator.mediaDevices?.getUserMedia),
+        );
+
+        return () => {
+            cancelled_ref.current = true;
+            recorder_ref.current?.stop();
+        };
+    }, []);
+
+    useEffect(() => {
+        if (state != "recording") return;
+
+        const started_at = Date.now();
+        const timer = setInterval(() => {
+            const ms = Date.now() - started_at;
+
+            setElapsedMs(ms);
+
+            if (ms >= AppData.ComposerVoiceMaxMs) stop();
+        }, 250);
+
+        const onKeyDown = (e: KeyboardEvent) => {
+            if (e.key != "Escape") return;
+
+            e.preventDefault();
+            cancel();
+        };
+
+        window.addEventListener("keydown", onKeyDown);
+
+        return () => {
+            clearInterval(timer);
+            window.removeEventListener("keydown", onKeyDown);
+        };
+    }, [state]);
+
+    async function start() {
+        if (state != "idle") return;
+
+        setError(undefined);
+        setElapsedMs(0);
+
+        let stream: MediaStream;
+
+        try {
+            stream = await navigator.mediaDevices.getUserMedia({ audio: true });
+        } catch {
+            setError("Microphone blocked. Allow it in the browser's site settings.");
+            return;
+        }
+
+        const type = RecordingTypes.find((t) => MediaRecorder.isTypeSupported(t));
+        const recorder = new MediaRecorder(stream, type ? { mimeType: type } : undefined);
+        const chunks: Blob[] = [];
+
+        cancelled_ref.current = false;
+        recorder.ondataavailable = (e) => {
+            if (e.data.size) chunks.push(e.data);
+        };
+        recorder.onstop = async () => {
+            stream.getTracks().forEach((track) => track.stop());
+            recorder_ref.current = null;
+
+            if (cancelled_ref.current || !chunks.length) {
+                setState("idle");
+                return;
+            }
+
+            setState("transcribing");
+
+            /**
+             * The bare type, without `;codecs=`, so it reads as a plain
+             * data URL.
+             */
+            const audio = new Blob(chunks, {
+                type: (recorder.mimeType || type || "audio/webm").split(";")[0],
+            });
+
+            await on_recorded_ref.current({ audio }).catch(() => {});
+            setState("idle");
+        };
+
+        recorder_ref.current = recorder;
+        recorder.start();
+        setState("recording");
+    }
+
+    function stop() {
+        recorder_ref.current?.stop();
+    }
+
+    function cancel() {
+        cancelled_ref.current = true;
+        recorder_ref.current?.stop();
+    }
+
+    return { state, elapsed_ms, error, setError, supported, start, stop, cancel };
+}

src/pages/admin/bots/[bot_id]/(partials)/bot-message-group.tsx

@@ -29,6 +29,7 @@ export default function BotMessageGroup({ bot, group }: Props) {
                 {group.messages.map((message) => (
                     <BotMessageLine
                         key={message.id}
+                        message_id={message.id || null}
                         text={message.text || ""}
                         needs_you={Boolean(message.needs_you)}
                     />

src/pages/admin/bots/[bot_id]/(partials)/bot-message-line.tsx

@@ -1,11 +1,14 @@
 import Markdown from "react-markdown";
 import remarkGfm from "remark-gfm";
+import Row from "@moduletrace/twui/layout/Row";
 import Stack from "@moduletrace/twui/layout/Stack";
 import linkComputerRefs from "@/src/utils/link-computer-refs";
 import { MessageMarkdownComponents } from "../(data)/message-markdown-components";
 import NeedsYouChip from "./needs-you-chip";
+import SpeakMessageButton from "./speak-message-button";
 
 type Props = {
+    message_id?: number | null;
     text: string;
     needs_you?: boolean;
 };
@@ -13,17 +16,30 @@ type Props = {
 /**
  * One bot message, rendered from markdown (GFM; raw HTML is never
  * rendered). Computers it names (`computerRef`) show as buttons that
- * switch the bot to them.
+ * switch the bot to them. A play button under it reads it aloud,
+ * disabled while the bot has no speech model.
  */
-export default function BotMessageLine({ text, needs_you }: Props) {
+export default function BotMessageLine({ message_id, text, needs_you }: Props) {
+    const can_speak = Boolean(message_id && text.trim());
+
     return (
         <Stack className="gap-1.5! items-start max-w-full">
             <Stack className="w-full gap-2.5! items-start break-words">
-                <Markdown remarkPlugins={[remarkGfm]} components={MessageMarkdownComponents}>
+                <Markdown
+                    remarkPlugins={[remarkGfm]}
+                    components={MessageMarkdownComponents}
+                >
                     {linkComputerRefs({ text })}
                 </Markdown>
             </Stack>
+            {needs_you || can_speak ? (
+                <Row className="gap-2! items-center">
                     {needs_you ? <NeedsYouChip /> : null}
+                    {can_speak ? (
+                        <SpeakMessageButton message_id={message_id!} />
+                    ) : null}
+                </Row>
+            ) : null}
         </Stack>
     );
 }

src/pages/admin/bots/[bot_id]/(partials)/composer-voice-button.tsx

@@ -0,0 +1,75 @@
+import Button from "@moduletrace/twui/layout/Button";
+import Row from "@moduletrace/twui/layout/Row";
+import Span from "@moduletrace/twui/layout/Span";
+import { Mic, Square } from "lucide-react";
+import type { VoiceRecorderState } from "../(hooks)/use-voice-recorder";
+
+const UnavailableTitle =
+    "Dictation needs a transcription model. A super admin can pick one in Settings, under Models.";
+
+type Props = {
+    state: VoiceRecorderState;
+    elapsed_ms: number;
+    error?: string;
+    /**
+     * No transcription model for the bot here: shown disabled, saying so.
+     */
+    unavailable?: boolean;
+    disabled?: boolean;
+    onStart: () => void;
+    onStop: () => void;
+};
+
+/**
+ * The composer's dictation button: a mic, a stop square with the time
+ * while recording, a spinner while the recording is turned into text.
+ * Its title says why it last failed, or that it needs a model.
+ */
+export default function ComposerVoiceButton({
+    state,
+    elapsed_ms,
+    error,
+    unavailable,
+    disabled,
+    onStart,
+    onStop,
+}: Props) {
+    const recording = state == "recording";
+    const seconds = Math.floor(elapsed_ms / 1000);
+
+    return (
+        <Row className="shrink-0 gap-1.5! flex-nowrap items-center">
+            {recording ? (
+                <Span className="text-xs tabular-nums text-ink-secondary" aria-live="off">
+                    {Math.floor(seconds / 60)}:{String(seconds % 60).padStart(2, "0")}
+                </Span>
+            ) : null}
+            <Button
+                title={
+                    unavailable
+                        ? UnavailableTitle
+                        : recording
+                          ? "Stop and write it out (Esc cancels)"
+                          : error || "Dictate"
+                }
+                aria-label={recording ? "Stop dictating" : "Dictate"}
+                aria-invalid={error ? true : undefined}
+                aria-pressed={recording}
+                type="button"
+                variant="ghost"
+                color="text"
+                disabled={disabled || unavailable}
+                loading={state == "transcribing"}
+                loadingIconSize="smaller"
+                onClick={recording ? onStop : onStart}
+                className="shrink-0 p-1! -mx-1 my-1 text-ink dark:text-ink rounded-default-xs focus-visible:ring-2 focus-visible:ring-ink"
+            >
+                {recording ? (
+                    <Square size={14} aria-hidden fill="currentColor" />
+                ) : (
+                    <Mic size={16} aria-hidden />
+                )}
+            </Button>
+        </Row>
+    );
+}

src/pages/admin/bots/[bot_id]/(partials)/composer.tsx

@@ -2,20 +2,25 @@ import { useRef, useState, type KeyboardEvent } from "react";
 import Row from "@moduletrace/twui/layout/Row";
 import Stack from "@moduletrace/twui/layout/Stack";
 import Textarea from "@moduletrace/twui/form/Textarea";
+import P from "@moduletrace/twui/layout/P";
 import type {
     BotExtensionsPayload,
     BotMessageWithAttachments,
+    BotVoicePayload,
     BotWithLastMessage,
 } from "@/src/types";
 import sendBotMessage from "../(functions)/send-bot-message";
+import transcribeVoice from "../(functions)/transcribe-voice";
 import useComposerAttachments from "../(hooks)/use-composer-attachments";
 import useComposerDraft from "../(hooks)/use-composer-draft";
 import useAutoGrowTextarea from "../(hooks)/use-auto-grow-textarea";
+import useVoiceRecorder from "../(hooks)/use-voice-recorder";
 import useOnlineStatus from "@/src/hooks/use-online-status";
 import ComposerAttachButton from "./composer-attach-button";
 import ComposerAttachments from "./composer-attachments";
 import ComposerSendButton from "./composer-send-button";
 import ComposerTools from "./composer-tools";
+import ComposerVoiceButton from "./composer-voice-button";
 
 type Props = {
     bot: BotWithLastMessage;
@@ -27,6 +32,10 @@ type Props = {
      * Its MCP servers and tools, listed under the input.
      */
     extensions?: BotExtensionsPayload | null;
+    /**
+     * The dictation mic is disabled while transcription has no model.
+     */
+    voice?: BotVoicePayload | null;
     onSent: (params: { message: BotMessageWithAttachments }) => void;
 };
 
@@ -36,9 +45,10 @@ type Props = {
  * just above it. A click anywhere on the row focuses the input. Files can
  * also be pasted or dropped in. Unsent text is kept per conversation
  * across reloads, with its uploaded files. The bot's MCP servers and tools
- * are listed under the input.
+ * are listed under the input. The mic dictates: what's said is written
+ * into the input at the cursor, to edit before sending.
  */
-export default function Composer({ bot, workspace_id, extensions, onSent }: Props) {
+export default function Composer({ bot, workspace_id, extensions, voice, onSent }: Props) {
     const { text, setText } = useComposerDraft({ bot_id: bot.id, workspace_id });
     const [sending, setSending] = useState(false);
     const [error, setError] = useState<string>();
@@ -48,12 +58,52 @@ export default function Composer({ bot, workspace_id, extensions, onSent }: Prop
 
     useAutoGrowTextarea({ ref: input_ref, value: text });
 
+    const recorder = useVoiceRecorder({
+        onRecorded: async ({ audio }) => {
+            if (!bot.id) return;
+
+            const res = await transcribeVoice({ bot_id: bot.id, workspace_id, audio });
+
+            if (!res.success || !res.singleRes?.text) {
+                recorder.setError(res.msg || "Couldn't write that out");
+                return;
+            }
+
+            insertText({ insert: res.singleRes.text });
+        },
+    });
+
     const can_send =
         Boolean(text.trim() || files.media_ids.length) &&
         !sending &&
         !files.uploading &&
+        recorder.state == "idle" &&
         online;
 
+    /**
+     * Puts dictated text at the cursor (or the end), spaced from what's
+     * around it, and leaves the cursor after it.
+     */
+    function insertText({ insert }: { insert: string }) {
+        const el = input_ref.current;
+        const start = el ? el.selectionStart : text.length;
+        const end = el ? el.selectionEnd : text.length;
+        const before = text.slice(0, start);
+        const after = text.slice(end);
+        const piece =
+            (before && !/\s$/.test(before) ? " " : "") +
+            insert +
+            (after && !/^\s/.test(after) ? " " : "");
+        const cursor = before.length + piece.length;
+
+        setText(before + piece + after);
+
+        requestAnimationFrame(() => {
+            input_ref.current?.focus();
+            input_ref.current?.setSelectionRange(cursor, cursor);
+        });
+    }
+
     async function send() {
         if (!can_send || !bot.id) return;
 
@@ -149,12 +199,28 @@ export default function Composer({ bot, workspace_id, extensions, onSent }: Prop
                         }}
                         autoFocus
                     />
+                    {recorder.supported ? (
+                        <ComposerVoiceButton
+                            state={recorder.state}
+                            elapsed_ms={recorder.elapsed_ms}
+                            error={recorder.error}
+                            unavailable={!voice?.transcription}
+                            disabled={!online}
+                            onStart={recorder.start}
+                            onStop={recorder.stop}
+                        />
+                    ) : null}
                     <ComposerSendButton
                         disabled={!can_send}
                         loading={sending || files.uploading}
                         onClick={send}
                     />
                 </Row>
+                {recorder.error ? (
+                    <P role="alert" className="w-full max-w-container mx-auto px-3 md:px-6 -mt-5 pb-3 text-xs text-ink-secondary">
+                        {recorder.error}
+                    </P>
+                ) : null}
                 <ComposerTools bot={bot} workspace_id={workspace_id} initial={extensions} />
             </Stack>
         </Stack>

src/pages/admin/bots/[bot_id]/(partials)/speak-message-button.tsx

@@ -0,0 +1,50 @@
+import { useContext } from "react";
+import Button from "@moduletrace/twui/layout/Button";
+import { Square, Volume2 } from "lucide-react";
+import { SpeakMessageContext } from "../(data)/speak-message-context";
+
+const UnavailableTitle =
+    "Reading aloud needs a speech model. A super admin can pick one in Settings, under Models.";
+
+type Props = {
+    message_id: number;
+};
+
+/**
+ * Reads a bot message aloud with its speech model, or stops it. Its title
+ * says why it last failed. Disabled, saying so, while speech has no model.
+ */
+export default function SpeakMessageButton({ message_id }: Props) {
+    const speak = useContext(SpeakMessageContext);
+    const active = speak.message_id == message_id;
+    const playing = active && speak.state == "playing";
+    const error = speak.error?.message_id == message_id ? speak.error.text : undefined;
+
+    return (
+        <Button
+            title={
+                !speak.enabled
+                    ? UnavailableTitle
+                    : playing
+                      ? "Stop"
+                      : error || "Read aloud"
+            }
+            aria-label={playing ? "Stop reading" : "Read aloud"}
+            aria-invalid={error ? true : undefined}
+            type="button"
+            disabled={!speak.enabled}
+            variant="ghost"
+            color="text"
+            loading={active && speak.state == "loading"}
+            loadingIconSize="smaller"
+            onClick={() => speak.toggle({ message_id })}
+            className="shrink-0 p-1! -mx-1 text-ink-secondary dark:text-ink-secondary hover:text-ink rounded-default-xs focus-visible:ring-2 focus-visible:ring-ink"
+        >
+            {playing ? (
+                <Square size={12} aria-hidden fill="currentColor" />
+            ) : (
+                <Volume2 size={14} aria-hidden />
+            )}
+        </Button>
+    );
+}

src/pages/admin/bots/[bot_id]/(sections)/conversation.tsx

@@ -6,10 +6,12 @@ import fetchApi from "@moduletrace/twui/utils/fetch/fetchApi";
 import useLiveBots from "@/src/hooks/use-live-bots";
 import useServerEvent from "@/src/hooks/use-server-event";
 import { BotComputersContext } from "../(data)/bot-computers-context";
+import { SpeakMessageContext } from "../(data)/speak-message-context";
 import useBotComputers from "../(hooks)/use-bot-computers";
 import useTaskProgress from "../(hooks)/use-task-progress";
 import useReplyProgress from "../(hooks)/use-reply-progress";
 import useReplyCatchUp from "../(hooks)/use-reply-catch-up";
+import useSpeakMessage from "../(hooks)/use-speak-message";
 import ConversationHeader from "../(partials)/conversation-header";
 import MessageThread from "../(partials)/message-thread";
 import Composer from "../(partials)/composer";
@@ -21,7 +23,8 @@ import ComputerViewerModal from "../(partials)/computer-viewer-modal";
  * bot's conversation, on its own or in a workspace (`pageProps.workspace`).
  * The bot's computers are shared with the header's modal and the computer
  * buttons in the thread, so a switch from either shows in both; either
- * can open one in the viewer.
+ * can open one in the viewer. Bot messages can be read aloud, one at a
+ * time.
  */
 export default function Conversation() {
     const { pageProps } = useContext(AppContext);
@@ -47,6 +50,8 @@ export default function Conversation() {
         computers: pageProps?.bot_computers || [],
     });
 
+    const speak = useSpeakMessage({ enabled: Boolean(pageProps?.bot_voice?.speech) });
+
     const progress = useTaskProgress({
         bot_id: bot?.id || 0,
         session_id,
@@ -112,6 +117,7 @@ export default function Conversation() {
 
     return (
         <BotComputersContext.Provider value={bot_computers}>
+            <SpeakMessageContext.Provider value={speak}>
                 <Stack className="grow min-w-0 h-full gap-0! items-stretch">
                     <ConversationHeader
                         bot={live_bot}
@@ -130,6 +136,7 @@ export default function Conversation() {
                         bot={bot}
                         workspace_id={workspace_id}
                         extensions={pageProps?.bot_extensions}
+                        voice={pageProps?.bot_voice}
                         onSent={({ message }) => {
                             if (startsNewSession(message) && message.session_id != session_id) {
                                 setSessionId(Number(message.session_id));
@@ -146,6 +153,7 @@ export default function Conversation() {
                     />
                 </Stack>
                 <ComputerViewerModal bot={live_bot} />
+            </SpeakMessageContext.Provider>
         </BotComputersContext.Provider>
     );
 }

src/pages/admin/settings/models/(partials)/harness-current-model.tsx

@@ -21,6 +21,8 @@ export default function HarnessCurrentModel({ harness }: Props) {
             }`
           : harness.required
             ? "Nothing to run on yet: no provider has a usable key."
+            : !harness.automatic
+              ? "Not available until a model is picked."
               : "No free model can do this right now. Pick one to turn it on.";
 
     return (

src/pages/admin/settings/models/(partials)/harness-model-select.tsx

@@ -12,17 +12,20 @@ type Props = {
     models: HarnessModelOption[];
     model_id: number | null;
     provider_name?: string;
+    automatic: boolean;
     onChange: (model_id: number | null) => void;
 };
 
 /**
- * Searchable model picker: automatic, or any model that can do the job.
+ * Searchable model picker: automatic (or not available, for a harness
+ * without automatic picks), or any model that can do the job.
  */
 export default function HarnessModelSelect({
     id,
     models,
     model_id,
     provider_name,
+    automatic,
     onChange,
 }: Props) {
     return (
@@ -32,7 +35,7 @@ export default function HarnessModelSelect({
             label="Model"
             showLabel
             placeholder="Search models"
-            options={harnessModelSelectOptions({ models, model_id, provider_name })}
+            options={harnessModelSelectOptions({ models, model_id, provider_name, automatic })}
             defaultValues={[model_id ? String(model_id) : AutoModelValue]}
             changeHandler={(value) => {
                 const picked = Array.isArray(value) ? value[0] : value;

src/pages/admin/settings/models/(partials)/harness-setting-group.tsx

@@ -14,7 +14,8 @@ type Props = {
 
 /**
  * One harness: its provider, then its model (narrowed to that provider),
- * whether it falls back to the automatic picks, whether it's on, and its
+ * whether it falls back to the automatic picks (for a harness that has
+ * them), whether it's on, and its
  * voice for speech. On a bot's page the provider select can also use the
  * default, which hides the rest.
  */
@@ -31,7 +32,7 @@ export default function HarnessSettingGroup({ harness, value, onChange }: Props)
         ? "turned off"
         : harness.default?.current
           ? `${harness.default.current.name} on ${harness.default.current.provider_name}`
-          : "nothing available";
+          : "not available";
 
     return (
         <SettingsGroup title={harness.title} description={harness.description}>
@@ -60,6 +61,7 @@ export default function HarnessSettingGroup({ harness, value, onChange }: Props)
                     models={models}
                     model_id={value.model_id}
                     provider_name={provider?.name}
+                    automatic={harness.automatic}
                     onChange={(model_id) => onChange({ ...value, model_id, voice: undefined })}
                 />
             )}
@@ -68,7 +70,7 @@ export default function HarnessSettingGroup({ harness, value, onChange }: Props)
             ) : (
                 <HarnessCurrentModel harness={harness} />
             )}
-            {!value.inherit && value.model_id ? (
+            {!value.inherit && value.model_id && harness.automatic ? (
                 <Checkbox
                     title="Fall back to the automatic picks if this model fails"
                     label={`Fall back to the automatic picks${provider ? ` on ${provider.name}` : ""} if this model fails`}

src/pages/admin/settings/models/(utils)/harness-model-select-options.ts

@@ -13,24 +13,33 @@ type Params = {
      * The picked provider's name; undefined for any provider.
      */
     provider_name?: string;
+    /**
+     * False: there's no automatic pick, so the empty choice reads "Not
+     * available".
+     */
+    automatic: boolean;
 };
 
 /**
  * The model picker's options: automatic (the best free model, on the
  * picked provider if there is one), then every model that can do the job
- * (free ones first, as the server sorts them).
+ * (free ones first, as the server sorts them). A harness without
+ * automatic picks starts with "Not available" instead.
  */
 export default function harnessModelSelectOptions({
     models,
     model_id,
     provider_name,
+    automatic,
 }: Params): TWUISelectOptionObject[] {
     const has_free = models.some((model) => model.free);
 
     return [
         {
             value: AutoModelValue,
-            title: provider_name
+            title: !automatic
+                ? "Not available (pick a model to turn it on)"
+                : provider_name
                 ? has_free
                     ? `Automatic (best free model on ${provider_name})`
                     : `Automatic (no free models on ${provider_name}: pick one)`

src/pages/api/admin/bots/speak-message.ts

@@ -0,0 +1,69 @@
+import userAuth from "@/src/functions/backend/auth/user-auth";
+import grabSpeakableMessage from "@/src/functions/backend/db/bots/grab-speakable-message";
+import speakText from "@/src/harness/audio/speak-text";
+import HarnessError from "@/src/harness/harness-error";
+import { SpeakBotMessageRequestSchema } from "@/src/schemas/bot-schema";
+import AppError from "@/src/utils/app-error";
+import parseWithSchema from "@/src/utils/parse-with-schema";
+import safeErrorMessage from "@/src/utils/safe-error-message";
+import speechText from "@/src/utils/speech-text";
+import type {
+    APIResponseObject,
+    BunextAPIRouteHandler,
+} from "@moduletrace/bunext/types";
+
+/**
+ * `GET ?message_id=`: reads one of the bot's messages in the user's own
+ * conversation aloud with the bot's speech model and voice, streamed as
+ * the provider makes it, so an `<audio>` pointed here starts playing
+ * within a couple of seconds. Closing the player stops the provider call.
+ * Nothing is stored; the browser may keep it for an hour.
+ */
+export const handler: BunextAPIRouteHandler<APIResponseObject> = async ({ req, query }) => {
+    try {
+        if (req.method != "GET") {
+            throw new AppError(`Method not allowed`);
+        }
+
+        const { user } = await userAuth({ req });
+
+        if (!user?.logged_in_status || !user.id) {
+            return { success: false, msg: `Unauthorized`, logoutUser: true };
+        }
+
+        const { message_id } = parseWithSchema({
+            schema: SpeakBotMessageRequestSchema,
+            data: query,
+        });
+
+        const { message, conversation } = await grabSpeakableMessage({
+            user_id: Number(user.id),
+            message_id,
+        });
+
+        const text = speechText({ text: message.text || "" });
+
+        if (!text) {
+            throw new AppError(`There's nothing to read in that message.`);
+        }
+
+        const audio = await speakText({
+            text,
+            bot_id: conversation.bot.id,
+            workspace_id: conversation.workspace_id,
+            signal: req.signal,
+        });
+
+        return new Response(audio.stream, {
+            headers: {
+                "Content-Type": audio.media_type,
+                "Cache-Control": "private, max-age=3600",
+            },
+        });
+    } catch (error: unknown) {
+        return {
+            success: false,
+            msg: error instanceof HarnessError ? error.message : safeErrorMessage({ error }),
+        };
+    }
+};

src/pages/api/admin/bots/transcribe.ts

@@ -0,0 +1,80 @@
+import userAuth from "@/src/functions/backend/auth/user-auth";
+import grabUserConversation from "@/src/functions/backend/db/bot-sessions/grab-user-conversation";
+import transcribeAudio from "@/src/harness/audio/transcribe-audio";
+import { TranscribeVoiceRequestSchema } from "@/src/schemas/bot-schema";
+import HarnessError from "@/src/harness/harness-error";
+import AppError from "@/src/utils/app-error";
+import bufferFromBase64 from "@/src/utils/buffer-from-base-64";
+import parseWithSchema from "@/src/utils/parse-with-schema";
+import isReadOnlyUser from "@/src/utils/rbac/is-read-only-user";
+import safeErrorMessage from "@/src/utils/safe-error-message";
+import type {
+    APIResponseObject,
+    BunextAPIRouteHandler,
+} from "@moduletrace/bunext/types";
+
+/**
+ * Recording types as the formats the harness takes.
+ */
+const AudioFormats: Record<string, string> = {
+    mp4: "m4a",
+    "x-m4a": "m4a",
+    mpeg: "mp3",
+    "x-wav": "wav",
+    wave: "wav",
+};
+
+/**
+ * Turns a composer dictation into text with the bot's transcription
+ * model, for the user to edit and send. The recording isn't kept.
+ */
+export const handler: BunextAPIRouteHandler<
+    APIResponseObject<{ text: string }>
+> = async ({ req, body }) => {
+    try {
+        if (req.method != "POST") {
+            throw new AppError(`Method not allowed`);
+        }
+
+        const { user, user_types } = await userAuth({ req });
+
+        if (!user?.logged_in_status || !user.id) {
+            return { success: false, msg: `Unauthorized`, logoutUser: true };
+        }
+
+        if (isReadOnlyUser({ user_types: user_types || [] })) {
+            throw new AppError(`This user can only read records.`);
+        }
+
+        const { bot_id, workspace_id, data_url } = parseWithSchema({
+            schema: TranscribeVoiceRequestSchema,
+            data: body,
+        });
+
+        const conversation = await grabUserConversation({ user_id: user.id, bot_id, workspace_id });
+
+        const type = data_url.match(/^data:audio\/([a-z0-9.+-]+)/i)?.[1]?.toLowerCase() || "webm";
+        const { buffer } = await bufferFromBase64(data_url);
+
+        if (!buffer.length) {
+            throw new AppError(`Nothing was recorded.`);
+        }
+
+        const text = await transcribeAudio({
+            audio: { data: new Uint8Array(buffer), format: AudioFormats[type] || type },
+            bot_id: conversation.bot.id,
+            workspace_id: conversation.workspace_id,
+        });
+
+        if (!text) {
+            throw new AppError(`No speech was heard. Try again a little closer to the mic.`);
+        }
+
+        return { success: true, singleRes: { text } };
+    } catch (error: unknown) {
+        return {
+            success: false,
+            msg: error instanceof HarnessError ? error.message : safeErrorMessage({ error }),
+        };
+    }
+};

src/schemas/bot-schema.ts

@@ -107,3 +107,20 @@ export const UploadBotAttachmentRequestSchema = z.object({
 });
 
 export type UploadBotAttachmentRequest = z.infer<typeof UploadBotAttachmentRequestSchema>;
+
+export const TranscribeVoiceRequestSchema = z.object({
+    bot_id: z.coerce.number().int().positive("Pick a bot"),
+    workspace_id: WorkspaceIdField,
+    /**
+     * The recording as a base64 data URL.
+     */
+    data_url: z.string().startsWith("data:audio/", "Send the recording as an audio data URL"),
+});
+
+export type TranscribeVoiceRequest = z.infer<typeof TranscribeVoiceRequestSchema>;
+
+export const SpeakBotMessageRequestSchema = z.object({
+    message_id: z.coerce.number().int().positive("Pick a message"),
+});
+
+export type SpeakBotMessageRequest = z.infer<typeof SpeakBotMessageRequestSchema>;

src/types/index.ts

@@ -277,6 +277,11 @@ export type HarnessOverview = {
     title: string;
     description: string;
     required: boolean;
+    /**
+     * Picks a free model itself when none is picked. Otherwise it isn't
+     * available until one is.
+     */
+    automatic: boolean;
     model_id?: number;
     /**
      * The provider the harness is limited to; undefined: any.
@@ -599,6 +604,11 @@ export type DbBackup = {
     size: number;
 };
 
+export type BotVoicePayload = {
+    transcription: boolean;
+    speech: boolean;
+};
+
 export type PagePropsType = {
     user?: User | null;
     user_types?: BUN_SQLITE_COBBLE_USER_TYPES[] | null;
@@ -678,6 +688,11 @@ export type PagePropsType = {
     mcps_overview?: McpServersOverviewPayload | null;
     tools_overview?: CustomToolsOverviewPayload | null;
     bot_extensions?: BotExtensionsPayload | null;
+    /**
+     * Whether the open conversation's mic (transcription) and read-aloud
+     * (speech) are turned on for the bot there.
+     */
+    bot_voice?: BotVoicePayload | null;
     workspace_extensions?: BotExtensionsPayload | null;
     envs?: {
         GOOGLE_CLIENT_ID?: string;

src/utils/speech-text.test.ts

@@ -0,0 +1,18 @@
+import { describe, expect, test } from "bun:test";
+import { AppData } from "@/src/data/app-data";
+import speechText from "./speech-text";
+
+describe("speechText", () => {
+    test("reads refs and markdown plainly", () => {
+        expect(
+            speechText({ text: "**Done.** Sign in on [computer:2]: [take-over:2:desktop]" }),
+        ).toBe("Done. Sign in on a computer: a take-over link");
+    });
+
+    test("cuts long messages", () => {
+        const out = speechText({ text: "a".repeat(AppData.SpeechMaxChars + 50) });
+
+        expect(out.length).toBe(AppData.SpeechMaxChars + 1);
+        expect(out.endsWith("…")).toBe(true);
+    });
+});

src/utils/speech-text.ts

@@ -0,0 +1,16 @@
+import { AppData } from "@/src/data/app-data";
+import messagePreview from "./message-preview";
+
+/**
+ * A bot message as text to read aloud: plain text (`messagePreview`),
+ * take-over refs as "a take-over link", cut at `AppData.SpeechMaxChars`.
+ */
+export default function speechText({ text }: { text: string }) {
+    const plain = messagePreview({
+        text: text.replace(/\[take-over:\d+:[a-z]+\](?!\()/g, "a take-over link"),
+    });
+
+    if (plain.length <= AppData.SpeechMaxChars) return plain;
+
+    return `${plain.slice(0, AppData.SpeechMaxChars).trimEnd()}…`;
+}