From 163d9a434cfaa132a156a0d1b11885d837624092 Mon Sep 17 00:00:00 2001 From: Benjamin Toby Date: Mon, 19 Oct 2026 21:21:04 +0100 Subject: [PATCH] Add dictation and streamed read-aloud to the chat - A mic in the composer records, transcribes with the bot's transcription model (api/admin/bots/transcribe) and writes the text into the input at the cursor to edit before sending. Escape cancels; recordings aren't stored. - Every bot message has a read-aloud button that plays GET api/admin/bots/speak-message, streamed from the provider as the audio is made, so it starts in a couple of seconds. One message plays at a time; only the user's own sessions can be read. - Both buttons stay visible but disabled while the bot has no model for them (bot_voice page prop). - Transcription and speech no longer pick free models automatically: they're not available until a model is picked, and Settings says so. Transcription lists speech-to-text models only, so the chat-model transcript path and its missing-audio check are gone. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 7 +- src/data/app-data.ts | 8 ++ src/data/harness-prompts.ts | 2 - src/dict/harness-dict.ts | 11 +- .../bot-sessions/grab-conversation-props.ts | 18 ++- .../backend/db/bots/grab-speakable-message.ts | 37 +++++ .../db/harness/grab-harness-overview-item.ts | 1 + .../audio/looks-like-missing-audio.test.ts | 27 ---- src/harness/audio/looks-like-missing-audio.ts | 14 -- src/harness/audio/speak-text.ts | 6 +- src/harness/audio/transcribe-audio.ts | 61 ++------ .../catalog/model-serves-harness.test.ts | 4 +- src/harness/catalog/model-serves-harness.ts | 8 +- src/harness/catalog/rank-auto-models.test.ts | 34 ++--- src/harness/catalog/rank-auto-models.ts | 17 +-- .../providers/openai-compatible/speak.ts | 14 +- src/harness/types.ts | 10 +- .../[bot_id]/(data)/speak-message-context.ts | 13 ++ .../[bot_id]/(functions)/transcribe-voice.ts | 37 +++++ .../[bot_id]/(hooks)/use-speak-message.ts | 123 ++++++++++++++++ .../[bot_id]/(hooks)/use-voice-recorder.ts | 136 ++++++++++++++++++ .../[bot_id]/(partials)/bot-message-group.tsx | 1 + .../[bot_id]/(partials)/bot-message-line.tsx | 24 +++- .../(partials)/composer-voice-button.tsx | 75 ++++++++++ .../bots/[bot_id]/(partials)/composer.tsx | 70 ++++++++- .../(partials)/speak-message-button.tsx | 50 +++++++ .../bots/[bot_id]/(sections)/conversation.tsx | 78 +++++----- .../(partials)/harness-current-model.tsx | 4 +- .../(partials)/harness-model-select.tsx | 7 +- .../(partials)/harness-setting-group.tsx | 8 +- .../(utils)/harness-model-select-options.ts | 13 +- src/pages/api/admin/bots/speak-message.ts | 69 +++++++++ src/pages/api/admin/bots/transcribe.ts | 80 +++++++++++ src/schemas/bot-schema.ts | 17 +++ src/types/index.ts | 15 ++ src/utils/speech-text.test.ts | 18 +++ src/utils/speech-text.ts | 16 +++ 37 files changed, 936 insertions(+), 197 deletions(-) create mode 100644 src/functions/backend/db/bots/grab-speakable-message.ts delete mode 100644 src/harness/audio/looks-like-missing-audio.test.ts delete mode 100644 src/harness/audio/looks-like-missing-audio.ts create mode 100644 src/pages/admin/bots/[bot_id]/(data)/speak-message-context.ts create mode 100644 src/pages/admin/bots/[bot_id]/(functions)/transcribe-voice.ts create mode 100644 src/pages/admin/bots/[bot_id]/(hooks)/use-speak-message.ts create mode 100644 src/pages/admin/bots/[bot_id]/(hooks)/use-voice-recorder.ts create mode 100644 src/pages/admin/bots/[bot_id]/(partials)/composer-voice-button.tsx create mode 100644 src/pages/admin/bots/[bot_id]/(partials)/speak-message-button.tsx create mode 100644 src/pages/api/admin/bots/speak-message.ts create mode 100644 src/pages/api/admin/bots/transcribe.ts create mode 100644 src/utils/speech-text.test.ts create mode 100644 src/utils/speech-text.ts diff --git a/AGENTS.md b/AGENTS.md index 75ca5fe..014a0e6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -93,6 +93,7 @@ Inspect live data with `bun llm-code-assistant/query-db.ts ""` - a read-onl - **Server:** `src/functions/backend/db/bots/` (`grab-bots`, `grab-bot-messages`, `grab-active-bot-task`, `grab-user-bot`, `create-bot`, `insert-user-message`). `api/admin/bots/send-message.ts` posts a user message to a bot the session user chats with (zod: `src/schemas/bot-schema.ts`), queues the bot's reply (`enqueueBotReply`) and wakes the queue (`src/bots-queue/wake-bots-queue.ts`; never import `src/bots-queue/index.ts` from a page or API route, it pulls the bunext runtime into the page bundles). `upload-attachment.ts` stores one file for the next message (private `media` row, paradigm `bot-attachment`, type from `attachmentFileKind`, documents get `text_content`, limits in `AppData.BotAttachment*`) and `send-message` takes their `media_ids` (checked to be the user's own by `grabAttachableMedia`), linked in `bot_message_media`; messages carry `attachments` (`BotMessageWithAttachments`). `mark-read.ts` clears a session's unread when a message arrives in the open conversation; `session-messages.ts` returns a session's latest messages, which the open conversation polls every `AppData.BotReplyCatchUpMs` while a reply is due (`use-reply-catch-up`), so a reply whose live push was missed (socket down) still lands without a reload (replayed as a `server:bot-message`, so the rail and unread take it too); `create-bot.ts` adds a worker (and puts it in a workspace when given one); `save-harness-settings.ts` saves a worker bot's models. - **Message rendering:** bot messages are markdown, rendered with `react-markdown` + `remark-gfm` (raw HTML never rendered) through the twui-based map in `[bot_id]/(data)/message-markdown-components.tsx` (`markdown-link`, `markdown-list`, `markdown-code-block` partials). User messages stay plain text. - **Attachments and thinking:** the composer uploads files as they're picked, pasted or dropped (`use-composer-attachments`), shows them as chips, and sends once none is still uploading; the thread shows images as thumbnails and other files as chips. Replies read the files of the last `BotReplyAttachmentMessages` messages with any (`load-attachment-parts.ts`: images and audio as parts, documents as their text up to `BotAttachmentMaxTextChars`, videos as a note); older ones are only named. Tasks don't get the files yet. While the user has the last word in a session (every user message queues a reply, so this needs no face and shows even while the bot works in another session) or the bot waits for a model (`BotWaitingStatusLine`) or a task for its computer (`BotComputerBusyStatusLine`), `BotThinkingRow` sits under the last message; while a task runs it reads " is working…" with the task's latest step, and a reply working through tools shows its latest step under "is thinking…" (`use-reply-progress`). A coordinator with no workers gets `CoordinatorSoloNote` instead of `CoordinatorReplyNote`, so it never claims to have handed work to a worker. Live notices come from the process that ran the job at once, and from every other process through the live relay. +- **Voice:** the composer's mic (`(partials)/composer-voice-button.tsx`, `use-voice-recorder`, shown once the browser can record, disabled with a note while transcription has no model to run on for the bot there, `bot_voice` page prop, from `resolveHarnessCandidates`) dictates: the recording goes to `api/admin/bots/transcribe.ts` (`transcribeAudio` with the bot's models, after `grabUserConversation`) and the text lands in the input at the cursor, to edit and send; Escape cancels, `AppData.ComposerVoiceMaxMs` stops it. Each bot message has a play button (disabled with a note while speech has no model) (`speak-message-button.tsx`, one player per conversation via `SpeakMessageContext` / `use-speak-message`) that plays `GET api/admin/bots/speak-message.ts?message_id=` straight in an `