Add dictation and streamed read-aloud to the chat
- A mic in the composer records, transcribes with the bot's transcription model (api/admin/bots/transcribe) and writes the text into the input at the cursor to edit before sending. Escape cancels; recordings aren't stored. - Every bot message has a read-aloud button that plays GET api/admin/bots/speak-message, streamed from the provider as the audio is made, so it starts in a couple of seconds. One message plays at a time; only the user's own sessions can be read. - Both buttons stay visible but disabled while the bot has no model for them (bot_voice page prop). - Transcription and speech no longer pick free models automatically: they're not available until a model is picked, and Settings says so. Transcription lists speech-to-text models only, so the chat-model transcript path and its missing-audio check are gone. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
@@ -174,8 +174,12 @@ export type HarnessSpeakParams = {
|
||||
signal: AbortSignal;
|
||||
};
|
||||
|
||||
export type HarnessAudioOutput = {
|
||||
data: Uint8Array;
|
||||
/**
|
||||
* Speech as the provider sends it: the stream starts once the provider
|
||||
* answers, so it can play while the rest is still being made.
|
||||
*/
|
||||
export type HarnessAudioStream = {
|
||||
stream: ReadableStream<Uint8Array>;
|
||||
media_type: string;
|
||||
};
|
||||
|
||||
@@ -206,7 +210,7 @@ export type HarnessImageOutput = {
|
||||
export type HarnessAdapter = {
|
||||
chat: (params: HarnessChatParams) => Promise<HarnessChatResult>;
|
||||
transcribe: (params: HarnessTranscribeParams) => Promise<{ text: string }>;
|
||||
speak: (params: HarnessSpeakParams) => Promise<HarnessAudioOutput>;
|
||||
speak: (params: HarnessSpeakParams) => Promise<HarnessAudioStream>;
|
||||
generateImage: (
|
||||
params: HarnessImageParams,
|
||||
) => Promise<{ images: HarnessImageOutput[] }>;
|
||||
|
||||
Reference in New Issue
Block a user