Let bots take in messages sent while they reply

A reply read the session once at the start, so a message sent while it
ran was never seen, and the follow-up reply queued for it then finished
silently: the message was saved before the reply, so the thread ended
with the bot. Those messages went unanswered.

The text loop now reads new user messages after every round of tools and
before an answer goes out, and the bot decides from what each one means
whether it changes the work (folded in, or the unsent answer redrafted)
or is a separate request, which it acknowledges at once with the new
send_message tool and does after the current work. Messages sent that
way are marked interim, so the thinking row stays up while the bot
works. A reply job records the newest user message it read, and a queued
reply only stays silent when a finished reply already read everything.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
2026-10-25 04:49:10 +01:00
co-authored by Claude Opus 5.5
parent c13b9ea826
commit b0fe619ae8
25 changed files with 754 additions and 20 deletions
+1 -1
View File
@@ -203,7 +203,7 @@ Every folder on the data volume, browsable read-only: a bot's own, a workspace's
- **Custom tools** (`CustomToolKinds` in `src/dict/custom-tool-dict.ts`): a name (`checkCustomToolName`: snake_case, never a built-in's, `ReservedToolNames`), a description, a JSON-schema `parameters` and a `config` per kind, built by `buildCustomTool` (`src/harness/tools/custom/`): `http` (`runHttpTool`: the app fetches, arguments URL-encoded in the URL, no body template sends the arguments as JSON), `script` (`runScriptTool`: bash/python/javascript on the run's computer, the task's or in chat the bot's current one, over the sandbox REST API in a subshell, computer envs set, arguments as JSON in `COBBLE_ARGS` and on stdin; left out without a computer), `prompt` (`runPromptTool`: one `generateText` turn with the bot's models). Super admins try one from the page (`testCustomTool`, the main computer for scripts): a saved one from its row (Try), or the form as it is, saved or not (`draft`, nothing recorded), from the Test panel in its modal (`custom-tool-test-panel.tsx`). Every run (a bot's call or a try; not bad arguments or a cancel) is saved as the tool's `status`, `last_error` and `last_checked_at` (`recordCustomToolRun`); editing resets it to `untested`.
- **MCP servers** (`McpTransports` in `src/dict/mcp-dict.ts`): `remote` (URL plus templated headers, `connectMcpServer`) or `stdio` (a command run with the host's shell, `hostShell`, in `DATA_DIR/mcp/<id>` (`grabMcpDir`), after its `setup_command`, which `runMcpSetup` reruns whenever it changes (marker file) or on Reinstall; its env is only its own `env` map plus the shell basics, `grabMcpStdioEnv`, never the app's). Their tools are listed into `tools_cache` (`syncMcpServer`: on save, Refresh, and hourly for remote ones by the `sync-mcp-servers` cron) and named `<slug>__<tool>` (`mcpToolName`); the modal's Test connection checks the form as it is, saved or not, outside the pool (`testMcpDraft`, `testMcpServer`, `api/admin/mcps/test-mcp`, a new stdio server's setup runs in `mcp/0`); a run that can't connect, or connects again after failing, updates its `status` too (`recordMcpServerConnection`); runs build tools from the cache (`mcpHarnessTool`) and connect on the first call (`openMcpServer`). stdio servers live in a per-process pool (`src/harness/mcp/mcp-stdio-pool.ts`), stopped after `AppData.McpStdioIdleMs` idle, on save or delete.
- **Check before and after acting:** every reply and task gets `ActionCheckNote` (`src/data/harness-prompts.ts`): look for an existing record before creating one (update it instead), and verify every action took effect before saying it's done. The tools back it: `update_memory` and `update_instructions` read the save back (`checkSavedText`, `grabBotSavedText`) and fail if it didn't take; `create_bot` refuses a name already on the crew, `create_schedule` a twin of one of the bot's schedules (`findDuplicateCronJob`: same place and same title, or same timing and brief), and `start_task`/`delegate_task` a task with the same title still open in that session (`grabOpenBotTask`); `create_workspace` already refuses a name in use.
- **Replies and tasks:** `run-bot-reply.ts` answers from the last `BotReplyHistoryMessages` messages (`botMessagesToHarness`) with `ChatReplyNote`, the bot's computers (`compose-computers-note.ts`: what they are and that they're reached through `start_task`) and the chat tools, at most `grabReplyMaxToolSteps()` steps (the `bot_reply_max_tool_steps` app setting, else `AppData.BotReplyMaxToolSteps`), and finishes silently if the bot already had the last word (`enqueueBotReply` reuses a pending reply, so a burst of messages gets one). `run-bot-task.ts` runs the brief (`compose-job-brief.ts`) with `TaskRunNote`, the bot tools, the file tools and the job's computer tools, closing the MCP connection after. Both add `BotFilesNote` (where a bot's files live, that they persist, and to search before saying it doesn't know) and collect `attachments` for files the bot wrote with `attach`, which `completeBotJob` links to the result message. Retryable harness errors go back to the queue's backoff, the rest are `BotJobFatalError`.
- **Replies and tasks:** `run-bot-reply.ts` answers from the last `BotReplyHistoryMessages` messages (`botMessagesToHarness`) with `ChatReplyNote`, the bot's computers (`compose-computers-note.ts`: what they are and that they're reached through `start_task`) and the chat tools, at most `grabReplyMaxToolSteps()` steps (the `bot_reply_max_tool_steps` app setting, else `AppData.BotReplyMaxToolSteps`), and takes in what the user sends while it works: the text loop's `incoming` reads the session's new user messages after every round of tools and before an answer goes out, joining them to the transcript (`IncomingMessagesNote`), and an answer they arrive in time for is written again with them in mind (`RedraftNote`, the draft and tool turns kept so nothing is called twice; not on the wrap-up). The bot decides from what a message means, never from a keyword, whether it belongs to the work: one that changes or adds to it is folded in, a separate request is acknowledged at once with `send_message` (`src/harness/tools/bot/send-message-tool.ts`, chat replies only, at most `AppData.BotReplyInterimMessagesMax` per reply) and done after the current work in the same run. A message it sends that way is `bot_messages.interim`, and the conversation keeps the thinking row up while the thread ends with one and the bot's face is `working` (`(utils)/is-awaiting-reply.ts`). The job then records the newest user message it read as its `message_id` (`markReplyAnswered`), and a queued reply finishes silently only when a finished reply already read every user message (`hasUnansweredMessage`, `grabAnsweredMessageId`), never by who wrote last: a message sent mid-reply is saved before that reply, so the thread ends with the bot even when it never read it. `enqueueBotReply` reuses a pending reply, so a burst of messages gets one. `run-bot-task.ts` runs the brief (`compose-job-brief.ts`) with `TaskRunNote`, the bot tools, the file tools and the job's computer tools, closing the MCP connection after. Both add `BotFilesNote` (where a bot's files live, that they persist, and to search before saying it doesn't know) and collect `attachments` for files the bot wrote with `attach`, which `completeBotJob` links to the result message. Retryable harness errors go back to the queue's backoff, the rest are `BotJobFatalError`.
- **API:** `api/admin/harness/` (`save-provider` also syncs, `sync-provider`, `delete-provider`, `save-settings`), super admin only, each returning the fresh `grabHarnessOverview()` page data.
### Token usage
+7
View File
@@ -439,6 +439,13 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
options: [0, 1],
defaultValue: 0,
},
{
fieldName: "interim",
fieldDescription: `Sent mid-reply with send_message (an acknowledgement, a heads-up): the bot is still working on the reply, so the thinking row stays up while its face is working`,
dataType: "INTEGER",
options: [0, 1],
defaultValue: 0,
},
{
fieldName: "reply_to_message_id",
fieldDescription: `The message this one replies to, quoted above it. Not a foreign key, so deleting the quoted message only drops the quote`,
+4
View File
@@ -309,6 +309,10 @@ export type BUN_SQLITE_COBBLE_BOT_MESSAGES = {
* Renders the Needs you chip under the message
*/
needs_you?: 0 | 1 | "";
/**
* Sent mid-reply with send_message (an acknowledgement, a heads-up): the bot is still working on the reply, so the thinking row stays up while its face is working
*/
interim?: 0 | 1 | "";
/**
* The message this one replies to, quoted above it. Not a foreign key, so deleting the quoted message only drops the quote
*/
@@ -0,0 +1,35 @@
import { describe, expect, test } from "bun:test";
import hasUnansweredMessage from "./has-unanswered-message";
const user = (id: number) => ({ id, author: "user" as const });
const bot = (id: number) => ({ id, author: "bot" as const });
describe("hasUnansweredMessage", () => {
test("a message sent while the last reply ran is answered, though the thread ends with the bot", () => {
/**
* A (10) was being answered when B (11) came in; the reply (12)
* was saved after B but its run never read it.
*/
expect(hasUnansweredMessage({ history: [user(10), user(11), bot(12)], answered_id: 10 })).toBe(true);
});
test("a message the last reply took in is not answered twice", () => {
expect(hasUnansweredMessage({ history: [user(10), user(11), bot(12)], answered_id: 11 })).toBe(false);
});
test("a burst after the answer is answered", () => {
expect(
hasUnansweredMessage({ history: [user(10), bot(12), user(13), user(14)], answered_id: 10 }),
).toBe(true);
});
test("without a record, the user having the last word decides", () => {
expect(hasUnansweredMessage({ history: [user(1)], answered_id: null })).toBe(true);
expect(hasUnansweredMessage({ history: [user(1), bot(2)], answered_id: null })).toBe(false);
});
test("nothing from the user, nothing to answer", () => {
expect(hasUnansweredMessage({ history: [], answered_id: null })).toBe(false);
expect(hasUnansweredMessage({ history: [bot(4)], answered_id: 2 })).toBe(false);
});
});
+34
View File
@@ -0,0 +1,34 @@
import type { BUN_SQLITE_COBBLE_BOT_MESSAGES } from "@/db/types/db";
type Params = {
/**
* The session's latest messages, oldest first.
*/
history: Pick<BUN_SQLITE_COBBLE_BOT_MESSAGES, "id" | "author">[];
/**
* The newest user message a finished reply answered
* (`grabAnsweredMessageId`), null when none is on record.
*/
answered_id: number | null;
};
/**
* Whether a reply has anything left to answer. A user message that came in
* while the last reply ran is saved before that reply, so the thread ends
* with the bot's message even though the bot never read it: what the last
* reply read decides, not who wrote last. Without a record (no reply has
* finished in the session), the user having the last word does.
*/
export default function hasUnansweredMessage({ history, answered_id }: Params) {
const user_ids = history
.filter((message) => message.author == "user")
.map((message) => Number(message.id));
if (!user_ids.length) return false;
if (answered_id) {
return Math.max(...user_ids) > answered_id;
}
return history[history.length - 1]?.author == "user";
}
+71 -12
View File
@@ -28,6 +28,12 @@ import jobSessionId from "@/src/functions/backend/db/bots-queue/job-session-id";
import loadAttachmentParts from "./load-attachment-parts";
import grabMessageRefNames from "@/src/functions/backend/db/bots/grab-message-ref-names";
import withTakeOverRefs from "./with-take-over-refs";
import hasUnansweredMessage from "./has-unanswered-message";
import grabAnsweredMessageId from "@/src/functions/backend/db/bots-queue/grab-answered-message-id";
import markReplyAnswered from "@/src/functions/backend/db/bots-queue/mark-reply-answered";
import sendMessageTool from "@/src/harness/tools/bot/send-message-tool";
import type { BotMessageWithAttachments } from "@/src/types";
import type { HarnessMessage } from "@/src/harness/types";
import type { BotJobContext, BotJobOutcome } from "./types";
type Params = BotJobContext & {
@@ -47,8 +53,12 @@ type Params = BotJobContext & {
* instructions, start_task, and for the coordinator delegate_task) plus
* its MCP servers' and custom tools (`openBotExtensions`, script tools on
* its current computer).
* Finishes silently when the last message is already the bot's, i.e. an
* earlier reply answered everything.
* Messages the user sends while it works join the run (`incoming`): one
* that changes the work changes the answer before it goes out, and one
* that's separate can be acknowledged at once (`send_message`) and done
* after the current work. The job records the newest
* one it read (`markReplyAnswered`). Finishes silently when an earlier
* reply already read everything (`hasUnansweredMessage`).
*/
export default async function runBotReply({
bot,
@@ -65,10 +75,48 @@ export default async function runBotReply({
limit: AppData.BotReplyHistoryMessages,
});
if (history[history.length - 1]?.author != "user") {
const answered_id = await grabAnsweredMessageId({ session_id });
if (!hasUnansweredMessage({ history, answered_id })) {
return { result: "", silent: true };
}
/**
* The newest message the run has read, and of those the newest user
* message, which is what the reply answers.
*/
let seen_id = Math.max(0, ...history.map((message) => Number(message.id)));
let answering_id = newestUserMessageId({ messages: history });
const toTurns = async ({ messages }: { messages: BotMessageWithAttachments[] }) =>
botMessagesToHarness({
messages,
parts: await loadAttachmentParts({ messages }),
refs: await grabMessageRefNames({
texts: messages.filter((m) => m.author == "user").map((m) => m.text || ""),
user_id,
}),
});
/**
* What the user wrote since the run last looked.
*/
const incoming = async (): Promise<HarnessMessage[]> => {
const fresh = await grabBotMessages({
session_id,
after_id: seen_id,
author: "user",
limit: AppData.BotReplyHistoryMessages,
});
if (!fresh.length) return [];
seen_id = Math.max(seen_id, ...fresh.map((message) => Number(message.id)));
answering_id = Math.max(answering_id, newestUserMessageId({ messages: fresh }));
return await toTurns({ messages: fresh });
};
const prompt = await grabBotPrompt({ bot_id: bot.id, user_id, workspace_id });
const computers = await grabBotComputers({ bot_id: bot.id, workspace_id });
@@ -165,22 +213,26 @@ export default async function runBotReply({
]
.filter(Boolean)
.join("\n\n"),
messages: botMessagesToHarness({
messages: history,
parts: await loadAttachmentParts({ messages: history }),
refs: await grabMessageRefNames({
texts: history.filter((m) => m.author == "user").map((m) => m.text || ""),
user_id,
}),
}),
tools: [...tools, ...file_tools, ...extensions.tools, ...skills.tools],
messages: await toTurns({ messages: history }),
tools: [
...tools,
sendMessageTool({ bot_id: bot.id, session_id: Number(session_id), workspace_id }),
...file_tools,
...extensions.tools,
...skills.tools,
],
max_steps: await grabReplyMaxToolSteps(),
bot_id: bot.id,
workspace_id,
signal,
onStep: report,
incoming,
});
if (job.id && answering_id) {
await markReplyAnswered({ job_id: job.id, message_id: answering_id });
}
return {
...withTakeOverRefs({ text: result.text, refs: take_overs }),
attachments: attachments.length ? attachments : undefined,
@@ -189,3 +241,10 @@ export default async function runBotReply({
await extensions.close();
}
}
function newestUserMessageId({ messages }: { messages: Pick<BotMessageWithAttachments, "id" | "author">[] }) {
return Math.max(
0,
...messages.filter((message) => message.author == "user").map((message) => Number(message.id)),
);
}
+6
View File
@@ -402,6 +402,12 @@ export const AppData = {
* reads it through `grabReplyMaxToolSteps`.
*/
BotReplyMaxToolSteps: 20,
/**
* Messages a chat reply may send while it works (`send_message`):
* acknowledging something the user wrote meanwhile, a heads-up. Its
* answer still comes at the end.
*/
BotReplyInterimMessagesMax: 3,
/**
* The most steps a saved run limit may ask for, for a task run or a
* chat reply. A typo in the settings page can't turn one run into an
+1
View File
@@ -111,6 +111,7 @@ export const JobStepToolLabels: Record<string, string> = {
update_memory: "Updating memory",
update_instructions: "Updating instructions",
load_skill: "Reading a skill",
send_message: "Sending a message",
};
/**
+1
View File
@@ -69,6 +69,7 @@ export const ReservedToolNames = [
"load_skill",
"report_run_failed",
"search_computers",
"send_message",
"set_bot_workspace_instructions",
"set_picture",
"set_workspace_computer",
@@ -0,0 +1,46 @@
import { Database } from "bun:sqlite";
import { describe, expect, test } from "bun:test";
import answeredMessageIdSql from "./answered-message-id-sql";
/**
* What decides whether a queued reply still has something to answer, run
* against real SQLite: only finished replies in the same session count, so
* a failed or still running one never hides a message.
*/
const db = new Database(":memory:");
db.run(
`CREATE TABLE bots_queue (id INTEGER PRIMARY KEY, session_id INTEGER, kind TEXT, status TEXT, message_id INTEGER)`,
);
db.run(`INSERT INTO bots_queue VALUES
(1, 100, 'reply', 'completed', 10),
(2, 100, 'reply', 'completed', 14),
(3, 100, 'reply', 'processing', 20),
(4, 100, 'reply', 'failed', 22),
(5, 100, 'task', 'completed', 30),
(6, 101, 'reply', 'completed', 40),
(7, 102, 'reply', 'pending', 50)`);
function answered(session_id: number) {
const { sql, values } = answeredMessageIdSql({ session_id });
return (db.query(sql).get(...values) as { id: number | null }).id;
}
describe("answeredMessageIdSql", () => {
test("is the newest message a finished reply in the session read", () => {
expect(answered(100)).toBe(14);
expect(answered(101)).toBe(40);
});
test("ignores running and failed replies, and tasks", () => {
expect(answered(100)).not.toBe(20);
expect(answered(100)).not.toBe(22);
expect(answered(100)).not.toBe(30);
});
test("is null where no reply has finished", () => {
expect(answered(102)).toBe(null);
expect(answered(999)).toBe(null);
});
});
@@ -0,0 +1,20 @@
import type { SqlCondition } from "@/src/utils/crew-access/types";
type Params = {
session_id: number | string;
};
/**
* The SQL for the newest user message a finished reply in the session
* answered: a reply job's `message_id` is the newest one it read
* (`markReplyAnswered`), so a message with a higher id came in after the
* bot last looked and still needs an answer, wherever its reply landed in
* the thread. Selects one `id` column, null when no reply has finished.
*/
export default function answeredMessageIdSql({ session_id }: Params): SqlCondition {
return {
sql: `SELECT MAX(message_id) AS id FROM bots_queue
WHERE session_id = ? AND kind = 'reply' AND status = 'completed'`,
values: [Number(session_id)],
};
}
@@ -0,0 +1,19 @@
import BunSQLite from "@moduletrace/bun-sqlite";
import answeredMessageIdSql from "./answered-message-id-sql";
type Params = {
session_id: number | string;
};
/**
* The newest user message a finished reply in the session answered
* (`answeredMessageIdSql`), or null when none is on record.
*/
export default async function grabAnsweredMessageId({ session_id }: Params) {
const { sql, values } = answeredMessageIdSql({ session_id });
const res = await BunSQLite.sql<{ id: number | null }>({ sql, values });
const id = res.payload?.[0]?.id;
return id ? Number(id) : null;
}
@@ -0,0 +1,22 @@
import BunSQLite from "@moduletrace/bun-sqlite";
type Params = {
job_id: number | string;
/**
* The newest user message the reply read.
*/
message_id: number;
};
/**
* Records the newest user message a running reply read, so a reply queued
* for a message it already took in finishes silently and one it missed is
* still answered (`grabAnsweredMessageId`). Never lowers it.
*/
export default async function markReplyAnswered({ job_id, message_id }: Params) {
await BunSQLite.sql({
sql: `UPDATE bots_queue SET message_id = ?
WHERE id = ? AND status = 'processing' AND (message_id IS NULL OR message_id < ?)`,
values: [message_id, job_id, message_id],
});
}
@@ -14,6 +14,15 @@ type Params = {
* Only messages older than this one, for loading further back.
*/
before_id?: number | null;
/**
* Only messages saved after this one (by id, which is insertion
* order), for what arrived while a reply ran.
*/
after_id?: number | null;
/**
* Only one side's messages.
*/
author?: "user" | "bot";
/**
* Whose conversation it is, so a reaction can show as the user's own.
* Left out for the bot's own history reads, which don't need it.
@@ -30,9 +39,12 @@ export default async function grabBotMessages({
session_id,
limit = 100,
before_id,
after_id,
author,
user_id,
}: Params): Promise<BotMessageWithAttachments[]> {
let sql = ``;
const values: (string | number)[] = [session_id];
sql += `SELECT * FROM (\n`;
sql += ` SELECT * FROM bot_messages\n`;
@@ -40,6 +52,17 @@ export default async function grabBotMessages({
if (before_id) {
sql += ` AND (created_at, id) < (SELECT created_at, id FROM bot_messages WHERE id = ?)\n`;
values.push(before_id);
}
if (after_id) {
sql += ` AND id > ?\n`;
values.push(after_id);
}
if (author) {
sql += ` AND author = ?\n`;
values.push(author);
}
sql += ` ORDER BY created_at DESC, id DESC\n`;
@@ -48,7 +71,7 @@ export default async function grabBotMessages({
const res = await BunSQLite.sql<BUN_SQLITE_COBBLE_BOT_MESSAGES>({
sql,
values: before_id ? [session_id, before_id, limit] : [session_id, limit],
values: [...values, limit],
});
const messages = res.payload || [];
@@ -13,6 +13,10 @@ type Params = {
workspace_id?: number | null;
text: string;
needs_you?: boolean;
/**
* Sent mid-reply (`send_message`): the reply is still being worked on.
*/
interim?: boolean;
/**
* Files the bot saved and sent with it (`media` ids): a PDF it made, a
* generated image, a document it wrote.
@@ -30,10 +34,12 @@ export default async function insertBotMessage({
workspace_id,
text,
needs_you,
interim,
media_ids = [],
}: Params): Promise<BUN_SQLITE_COBBLE_BOT_MESSAGES> {
const now = Date.now();
const needs_you_flag = needs_you ? 1 : 0;
const interim_flag = interim ? 1 : 0;
const res = await BunSQLite.insert<
BUN_SQLITE_COBBLE_BOT_MESSAGES,
@@ -48,6 +54,7 @@ export default async function insertBotMessage({
author: "bot",
text,
needs_you: needs_you_flag,
interim: interim_flag,
},
],
});
@@ -78,6 +85,7 @@ export default async function insertBotMessage({
author: "bot",
text,
needs_you: needs_you_flag,
interim: interim_flag,
created_at: now,
updated_at: now,
};
+40
View File
@@ -0,0 +1,40 @@
import { describe, expect, test } from "bun:test";
import type { HarnessMessage } from "../types";
import pushUserTurn from "./push-user-turn";
describe("pushUserTurn", () => {
test("adds a turn after the bot's", () => {
const transcript: HarnessMessage[] = [{ role: "assistant", content: "Hi." }];
pushUserTurn({ transcript, content: "Hello" });
expect(transcript).toEqual([
{ role: "assistant", content: "Hi." },
{ role: "user", content: "Hello" },
]);
});
test("joins text onto a user turn, so roles never repeat", () => {
const transcript: HarnessMessage[] = [{ role: "user", content: "One" }];
pushUserTurn({ transcript, content: "Two" });
expect(transcript).toEqual([{ role: "user", content: "One\n\nTwo" }]);
});
test("joins parts once either side has files", () => {
const transcript: HarnessMessage[] = [{ role: "user", content: "Look" }];
pushUserTurn({ transcript, content: [{ type: "image", url: "u" }] });
expect(transcript).toEqual([
{
role: "user",
content: [
{ type: "text", text: "Look" },
{ type: "image", url: "u" },
],
},
]);
});
});
+35
View File
@@ -0,0 +1,35 @@
import type { HarnessContentPart, HarnessMessage } from "../types";
type UserContent = Extract<HarnessMessage, { role: "user" }>["content"];
type Params = {
transcript: HarnessMessage[];
content: UserContent;
};
/**
* Adds a user turn to the transcript, joined onto the last one when that is
* the user's too, since some providers reject repeated roles. Text stays a
* string while both sides are text.
*/
export default function pushUserTurn({ transcript, content }: Params) {
const last = transcript[transcript.length - 1];
if (last?.role != "user") {
transcript.push({ role: "user", content });
return;
}
if (typeof last.content == "string" && typeof content == "string") {
last.content = [last.content, content].filter(Boolean).join("\n\n");
return;
}
last.content = [...toParts({ content: last.content }), ...toParts({ content })];
}
function toParts({ content }: { content: UserContent }): HarnessContentPart[] {
if (typeof content != "string") return content;
return content ? [{ type: "text", text: content }] : [];
}
+137 -1
View File
@@ -3,7 +3,7 @@ import { z } from "zod";
import HarnessError from "../harness-error";
import type { HarnessCandidate, HarnessTool } from "../types";
import type { GenerateTextParams, GenerateTextResult } from "./generate-text";
import runTextLoop from "./run-text-loop";
import runTextLoop, { IncomingMessagesNote, RedraftNote } from "./run-text-loop";
const candidate: HarnessCandidate = {
key: "1:m",
@@ -152,4 +152,140 @@ describe("runTextLoop", () => {
expect(error).toBeInstanceOf(HarnessError);
expect(error.code).toBe("aborted");
});
test("takes in what the user wrote while tools ran, before the next turn", async () => {
memory.length = 0;
const { generate, seen } = scripted([
{ tool_calls: [{ id: "c1", name: "remember", arguments: `{"note":"tea"}` }] },
{ text: "Coffee it is." },
]);
const arrivals = [[{ role: "user" as const, content: "Actually, coffee." }]];
const result = await runTextLoop({
system: "s",
messages: [{ role: "user", content: "I like tea." }],
tools: [remember],
generate,
resolveCandidates,
incoming: async () => arrivals.shift() || [],
});
expect(result.text).toBe("Coffee it is.");
expect(seen).toHaveLength(2);
expect(seen[1]!.messages.slice(-2)).toEqual([
{ role: "tool", tool_call_id: "c1", content: "Saved." },
{ role: "user", content: `${IncomingMessagesNote}\n\nActually, coffee.` },
]);
});
test("writes an answer again when the user wrote past it, keeping the draft and tool turns", async () => {
const { generate, seen } = scripted([
{ text: "Here's the summary in English." },
{ text: "Here's the summary in French." },
]);
const arrivals = [
[
{ role: "user" as const, content: "In French, please." },
{ role: "user" as const, content: "And short." },
],
];
const result = await runTextLoop({
system: "s",
messages: [{ role: "user", content: "Summarise it." }],
generate,
resolveCandidates,
incoming: async () => arrivals.shift() || [],
});
expect(result.text).toBe("Here's the summary in French.");
expect(result.steps).toBe(2);
expect(seen[1]!.messages.slice(-2)).toEqual([
{ role: "assistant", content: "Here's the summary in English." },
{
role: "user",
content: `${RedraftNote}\n\nIn French, please.\n\nAnd short.`,
},
]);
});
test("joins files from new messages onto the same user turn", async () => {
const { generate, seen } = scripted([{ text: "Draft." }, { text: "With the photo." }]);
const arrivals = [
[
{
role: "user" as const,
content: [
{ type: "text" as const, text: "Use this one." },
{ type: "image" as const, url: "data:image/png;base64,x" },
],
},
],
];
await runTextLoop({
system: "s",
messages: [{ role: "user", content: "Write the caption." }],
generate,
resolveCandidates,
incoming: async () => arrivals.shift() || [],
});
expect(seen[1]!.messages.at(-1)).toEqual({
role: "user",
content: [
{ type: "text", text: RedraftNote },
{ type: "text", text: "Use this one." },
{ type: "image", url: "data:image/png;base64,x" },
],
});
});
test("returns at once when nothing new came in", async () => {
const { generate, seen } = scripted([{ text: "Hi." }]);
let looks = 0;
const result = await runTextLoop({
system: "s",
messages: [{ role: "user", content: "Hello" }],
generate,
resolveCandidates,
incoming: async () => {
looks++;
return [];
},
});
expect(result.text).toBe("Hi.");
expect(seen).toHaveLength(1);
expect(looks).toBe(1);
});
test("doesn't redraft the wrap up: a later reply picks the new messages up", async () => {
const loop_call = {
tool_calls: [{ id: "c", name: "remember", arguments: `{"note":"x"}` }],
};
const { generate, seen } = scripted([loop_call, { text: "Out of steps." }]);
let looks = 0;
const result = await runTextLoop({
system: "s",
messages: [{ role: "user", content: "go" }],
tools: [remember],
max_steps: 1,
generate,
resolveCandidates,
incoming: async () => {
looks++;
return looks == 1 ? [{ role: "user", content: "Also this." }] : [];
},
});
expect(result.text).toBe("Out of steps.");
expect(seen).toHaveLength(2);
expect(looks).toBe(1);
expect(typeof seen[1]!.messages.at(-1)!.content).toBe("string");
expect(seen[1]!.messages.at(-1)!.content).toContain("Also this.");
expect(seen[1]!.messages.at(-1)!.content).toContain("You've used every step");
});
});
+59 -1
View File
@@ -12,6 +12,7 @@ import type {
} from "../types";
import fitTranscriptToContext from "./fit-transcript-to-context";
import generateText from "./generate-text";
import pushUserTurn from "./push-user-turn";
export type TextLoopStep = {
index: number;
@@ -37,6 +38,14 @@ type Params = {
* progress and checkpoints.
*/
onStep?: (step: TextLoopStep) => Promise<void> | void;
/**
* The user's turns that arrived since the run started (and since the
* last call), read after each round of tools and before an answer is
* returned. They join the transcript, so a message sent while the bot
* works can change what it does, and an answer they arrive in time for
* is written again with them in mind.
*/
incoming?: () => Promise<HarnessMessage[]>;
/**
* The bot the run is for: its own models are used where it has them.
*/
@@ -65,6 +74,25 @@ export type TextLoopResult = {
const WrapUpPrompt = `You've used every step this run allows. Don't call any more tools. Report what you did, what is left, and anything you need from the user.`;
/**
* Told with what the user wrote while the run worked. Whether a message
* belongs to the work is the model's call, from what it says: a correction
* changes the work, a separate request is acknowledged (`send_message`,
* where the run has it) and done after.
*/
export const IncomingMessagesNote = `The user wrote again while you were working (below). Decide what it is from what it means, not how it's worded:
- If it changes, corrects or adds to what you're doing, take it into account and carry on with the updated request.
- If it's a separate request, don't drop what you're doing: if you can send_message, acknowledge it in a line now (say you'll get to it once this is done), then finish the current work, deal with the new request, and cover both in your answer.
- If it only needs a quick answer, give it with send_message if you can and carry on.`;
/**
* Told when the user wrote past an answer that hadn't gone out yet.
*/
export const RedraftNote = `Your answer above was not sent: the user wrote again before it went out (below). Decide what it is from what it means, not how it's worded:
- If it changes or adds to what you were answering, write the answer again with it in mind.
- If it's separate, keep your answer as it is and deal with the new message too: in the same answer, or send the first with send_message and then answer the new one.
Don't repeat tool calls you have already made.`;
/**
* The text harness at work: the model answers or calls tools, tool results
* go back to it, and so on until it answers without calling a tool. Tools
@@ -78,6 +106,7 @@ export default async function runTextLoop({
max_steps = AppData.HarnessMaxToolSteps,
signal = new AbortController().signal,
onStep,
incoming,
bot_id,
workspace_id,
generate = generateText,
@@ -102,7 +131,7 @@ export default async function runTextLoop({
const wrap_up = index == max_steps;
if (wrap_up) {
transcript.push({ role: "user", content: WrapUpPrompt });
pushUserTurn({ transcript, content: WrapUpPrompt });
}
/**
@@ -148,6 +177,14 @@ export default async function runTextLoop({
candidate: answer.candidate,
});
/**
* An answer the user wrote past is drafted again. Not on the wrap
* up: the run is out of steps, and a later reply picks them up.
*/
if (!tool_calls.length && !wrap_up && (await takeIncoming({ note: RedraftNote }))) {
continue;
}
if (!tool_calls.length) {
return {
text:
@@ -167,7 +204,28 @@ export default async function runTextLoop({
content: await runToolCall({ call, tools, signal }),
});
}
await takeIncoming({ note: IncomingMessagesNote });
}
throw new Error(`Unreachable: the text loop ended without an answer.`);
/**
* Adds what the user wrote since the last look, after `note`. True when
* there was anything.
*/
async function takeIncoming({ note }: { note: string }) {
const turns = (await incoming?.()) || [];
const user_turns = turns.filter((turn) => turn.role == "user");
if (!user_turns.length) return false;
pushUserTurn({ transcript, content: note });
for (const turn of user_turns) {
pushUserTurn({ transcript, content: turn.content });
}
return true;
}
}
@@ -0,0 +1,52 @@
import { describe, expect, test } from "bun:test";
import { AppData } from "@/src/data/app-data";
import sendMessageTool from "./send-message-tool";
const signal = new AbortController().signal;
function tool() {
const posted: string[] = [];
const sent: number[] = [];
const send = sendMessageTool({
bot_id: 2,
session_id: 100,
post: async ({ text }) => {
posted.push(text);
return { id: posted.length };
},
onSend: ({ message_id }) => sent.push(message_id),
});
return { send, posted, sent };
}
describe("sendMessageTool", () => {
test("posts the message now and tells the bot to carry on", async () => {
const { send, posted, sent } = tool();
const result = await send.run({ args: { text: "Noted, I'll do that once this is done." }, signal });
expect(posted).toEqual(["Noted, I'll do that once this is done."]);
expect(sent).toEqual([1]);
expect(result).toContain("Carry on");
});
test("stops at the cap, so a reply can't flood the chat", async () => {
const { send, posted } = tool();
for (let i = 0; i < AppData.BotReplyInterimMessagesMax; i++) {
await send.run({ args: { text: `Heads-up ${i}` }, signal });
}
const error = await Promise.resolve(send.run({ args: { text: "One more" }, signal })).catch(
(e) => e,
);
expect(error).toBeInstanceOf(Error);
expect(posted).toHaveLength(AppData.BotReplyInterimMessagesMax);
});
test("refuses an empty message", () => {
expect(tool().send.schema?.safeParse({ text: " " }).success).toBe(false);
});
});
@@ -0,0 +1,71 @@
import { z } from "zod";
import pushBotNotice from "@/src/bots-queue/push-bot-notice";
import { AppData } from "@/src/data/app-data";
import insertBotMessage from "@/src/functions/backend/db/bots/insert-bot-message";
import type { HarnessTool } from "../../types";
type Params = {
bot_id: number;
/**
* The session being answered: the message goes in it.
*/
session_id: number;
workspace_id?: number | null;
/**
* Called with each message sent, so the reply knows it said something.
*/
onSend?: (params: { message_id: number }) => void;
/**
* Saves and pushes the message (`insertBotMessage`, `pushBotNotice`).
*/
post?: (params: { text: string }) => Promise<{ id?: number | "" }>;
};
type Args = {
text: string;
};
/**
* Lets a reply post a message in the conversation now, without ending the
* run: the bot acknowledges something the user wrote while it works ("Noted,
* I'll do that once this is done") and carries on. The message is marked
* `interim`, so the conversation keeps showing the bot at work, and is
* pushed at once. At most `AppData.BotReplyInterimMessagesMax` per run.
*/
export default function sendMessageTool({
bot_id,
session_id,
workspace_id,
onSend,
post = async ({ text }) => {
const message = await insertBotMessage({ bot_id, session_id, workspace_id, text, interim: true });
await pushBotNotice({ bot_id, message });
return message;
},
}: Params): HarnessTool<Args> {
let sent = 0;
return {
name: "send_message",
description: `Send the user a message now and keep working: your reply still comes when you finish. Use it to acknowledge something they wrote while you were working that is separate from what you're doing (say you'll get to it, then carry on), or for a short heads-up on long work. Not for your answer, and not for something that changes what you're doing: take that into account instead. At most ${AppData.BotReplyInterimMessagesMax} per reply.`,
schema: z.object({
text: z.string().trim().min(1).max(2000).describe(`The message (markdown), short`),
}),
run: async ({ args }) => {
if (sent >= AppData.BotReplyInterimMessagesMax) {
throw new Error(
`You've sent ${sent} messages in this reply already. Carry on, and put the rest in your answer.`,
);
}
const message = await post({ text: args.text });
sent++;
onSend?.({ message_id: Number(message.id) });
return `Sent. Carry on with what you were doing.`;
},
};
}
@@ -1,4 +1,5 @@
import { useMemo, type RefObject } from "react";
import isAwaitingReply from "../(utils)/is-awaiting-reply";
import Stack from "@moduletrace/twui/layout/Stack";
import type {
BotMessageWithAttachments,
@@ -103,9 +104,10 @@ export default function MessageThread({
bot={bot}
progress={progress}
reply_progress={reply_progress}
awaiting_reply={
messages[messages.length - 1]?.author == "user"
}
awaiting_reply={isAwaitingReply({
last_message: messages[messages.length - 1],
face: bot.face,
})}
receiving={receiving}
has_task={has_task}
/>
@@ -1,4 +1,5 @@
import { useContext, useEffect, useMemo, useRef, useState } from "react";
import isAwaitingReply from "../(utils)/is-awaiting-reply";
import Stack from "@moduletrace/twui/layout/Stack";
import type { BotMessageReactionEmoji } from "@/src/dict/bot-dict";
import type {
@@ -160,7 +161,7 @@ export default function Conversation() {
const last_saved_id = messages[messages.length - 1]?.id || null;
const last_thread_message = thread_messages[thread_messages.length - 1];
const awaiting_reply = last_thread_message?.author == "user";
const awaiting_reply = isAwaitingReply({ last_message: last_thread_message, face: bot?.face });
const reply_progress = useReplyProgress({
bot_id: bot?.id || 0,
@@ -0,0 +1,29 @@
import { describe, expect, test } from "bun:test";
import isAwaitingReply from "./is-awaiting-reply";
describe("isAwaitingReply", () => {
test("the user having the last word owes a reply", () => {
expect(isAwaitingReply({ last_message: { author: "user" }, face: "idle" })).toBe(true);
});
test("an acknowledgement sent mid-reply keeps it due while the bot works", () => {
expect(isAwaitingReply({ last_message: { author: "bot", interim: 1 }, face: "working" })).toBe(
true,
);
});
test("once the reply is over, an acknowledgement left last owes nothing", () => {
expect(isAwaitingReply({ last_message: { author: "bot", interim: 1 }, face: "idle" })).toBe(false);
});
test("an answer owes nothing, even while the bot works elsewhere", () => {
expect(isAwaitingReply({ last_message: { author: "bot", interim: 0 }, face: "working" })).toBe(
false,
);
expect(isAwaitingReply({ last_message: { author: "bot" }, face: "working" })).toBe(false);
});
test("an empty thread owes nothing", () => {
expect(isAwaitingReply({ last_message: null, face: "working" })).toBe(false);
});
});
@@ -0,0 +1,25 @@
import type { BUN_SQLITE_COBBLE_BOT_MESSAGES } from "@/db/types/db";
type Params = {
/**
* The thread's last message.
*/
last_message?: Pick<BUN_SQLITE_COBBLE_BOT_MESSAGES, "author" | "interim"> | null;
/**
* The bot's face for this user.
*/
face?: string | null;
};
/**
* Whether a reply is due in the thread: the user has the last word, or the
* bot's last word was sent mid-reply (`interim`, an acknowledgement while
* it works) and it is still working. Once the reply ends its face moves
* on, so a reply that had nothing more to say never leaves the row up.
*/
export default function isAwaitingReply({ last_message, face }: Params) {
if (!last_message) return false;
if (last_message.author == "user") return true;
return Boolean(last_message.interim) && face == "working";
}