Add dictation and streamed read-aloud to the chat

- A mic in the composer records, transcribes with the bot's
  transcription model (api/admin/bots/transcribe) and writes the text
  into the input at the cursor to edit before sending. Escape cancels;
  recordings aren't stored.
- Every bot message has a read-aloud button that plays
  GET api/admin/bots/speak-message, streamed from the provider as the
  audio is made, so it starts in a couple of seconds. One message plays
  at a time; only the user's own sessions can be read.
- Both buttons stay visible but disabled while the bot has no model for
  them (bot_voice page prop).
- Transcription and speech no longer pick free models automatically:
  they're not available until a model is picked, and Settings says so.
  Transcription lists speech-to-text models only, so the chat-model
  transcript path and its missing-audio check are gone.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
2026-10-19 21:21:04 +01:00
co-authored by Claude Opus 5.5
parent 727cbf53c8
commit 163d9a434c
37 changed files with 936 additions and 197 deletions
+7 -3
View File
@@ -174,8 +174,12 @@ export type HarnessSpeakParams = {
signal: AbortSignal;
};
export type HarnessAudioOutput = {
data: Uint8Array;
/**
* Speech as the provider sends it: the stream starts once the provider
* answers, so it can play while the rest is still being made.
*/
export type HarnessAudioStream = {
stream: ReadableStream<Uint8Array>;
media_type: string;
};
@@ -206,7 +210,7 @@ export type HarnessImageOutput = {
export type HarnessAdapter = {
chat: (params: HarnessChatParams) => Promise<HarnessChatResult>;
transcribe: (params: HarnessTranscribeParams) => Promise<{ text: string }>;
speak: (params: HarnessSpeakParams) => Promise<HarnessAudioOutput>;
speak: (params: HarnessSpeakParams) => Promise<HarnessAudioStream>;
generateImage: (
params: HarnessImageParams,
) => Promise<{ images: HarnessImageOutput[] }>;