diff --git a/AGENTS.md b/AGENTS.md index 0a103d0..50e077b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -101,7 +101,7 @@ Inspect live data with `bun llm-code-assistant/query-db.ts ""` - a read-onl - **Threads and sessions:** each user has one conversation with a bot in its own chat and one per workspace it's in (the coordinator in every workspace). Each is split into `bot_sessions` (`user_id`, `bot_id`, `workspace_id` null for its own chat, `status` active|closed, `title` and `summary` written by the bot from the first exchange, `summary_message_count` (how many messages the summary covers), `previous_session_id`, `unread`, `last_activity_at`); `bot_messages`, `bot_tasks` and `bots_queue` carry `session_id` and `workspace_id`. Helpers in `src/functions/backend/db/bot-sessions/`: `ensureBotSession` (the active one, else a new one; `rollover` starts the next once it has `AppData.BotSessionMaxMessages`), `startBotSession`, `reopenBotSession`, `grabBotSessions` (search over title, summary and message text), `grabUserConversation` (access and membership), `grabConversationProps` (the conversation pages' props). Once a session's first exchange is in, `name-bot-session.ts` has the bot name it (the title the rail and list show) and write its summary, after the reply is pushed; a new session's first reply gets the previous one's summary (`grab-session-carry-over.ts`), rewritten first when it covers fewer messages than that session ended with. The header's sessions icon (`src/components/general/bot-sessions-modal.tsx`) lists, searches, starts and reopens sessions, and the new-session icon beside it (`(partials)/new-session-button.tsx`) starts one (`api/admin/bots/list-sessions`, `new-session`, `reopen-session`). The rail lists the open bot's latest `AppData.RailRecentSessions` sessions under its row (`rail-bot-sessions.tsx`), and once there are more, "All sessions" opens the same modal. Unread, the rail's last message and activity are per thread, from its sessions. Job results go into the job's own session, even after it closes. - **Coordinator:** the install has exactly one `kind: coordinator` bot, the super admin's alone, that coordinates the crew and delegates to every worker. It is created at signup (`bun run seed`) and, as a fallback, by the super admin's `/admin/bots`, all via `ensureCoordinatorBot` (`src/functions/backend/db/bots/ensure-coordinator-bot.ts`, idempotent). Name in `CoordinatorBotName` (`src/dict/bot-dict.ts`). Its default prompt is `CoordinatorSystemPrompt` (`src/data/coordinator-system-prompt.ts`): attached at creation, and backfilled onto a coordinator whose prompt is empty, never over an edited one. - **Routes:** `/admin/bots` opens the bot this browser was last in (`readLastVisitedBot` from the single `cobble_last_visited_bot_id` cookie, which the conversation pages set through `withLastVisitedBotCookies` in their `res_transform`, and which `/auth/logout` clears; a conversation inside a workspace is never remembered, so the landing page always reopens the bot's own chat), but only while `grabReachableConversation` says the user still reaches it. Otherwise it ensures the coordinator and redirects the super admin to its chat (`/admin/bots/1` on a fresh install), anyone else to their latest bot, else their first workspace, else an empty page; `/admin/bots/[bot_id]` renders the conversation (header, thread and composer, the last two held to 760px; there is no right panel); `/admin/bots/[bot_id]/models` is a worker bot's own model settings, for super admins and admins (the coordinator's redirects to its chat); `/admin/workspaces/[workspace_id]` is a workspace's dashboard (`(sections)/workspace-dashboard.tsx`: read-only picture, name, description and context, latest activity, bots with their per-workspace instructions and memory, computers; its header links into `/admin/settings/workspaces/[workspace_id]`, whose `workspace-overview.tsx` has the editable groups) and `/admin/workspaces/[workspace_id]/bots/[bot_id]` is the same `Conversation` section scoped to the workspace (`pageProps.workspace`; only the coordinator and member bots). All render in the full-bleed `src/layouts/bots/` layout (rail of conversations + user popover), which `__root.tsx` uses for every `/admin/*` page. -- **Server:** `src/functions/backend/db/bots/` (`grab-bots`, `grab-bot-messages`, `grab-active-bot-task`, `grab-user-bot`, `create-bot`, `insert-user-message`). `api/admin/bots/send-message.ts` posts a user message to a bot the session user chats with (zod: `src/schemas/bot-schema.ts`), queues the bot's reply (`enqueueBotReply`) and wakes the queue (`src/bots-queue/wake-bots-queue.ts`; never import `src/bots-queue/index.ts` from a page or API route, it pulls the bunext runtime into the page bundles). `upload-attachment.ts` stores one file for the next message (private `media` row, paradigm `bot-attachment`, type from `attachmentFileKind`, documents get `text_content`, limits in `AppData.BotAttachment*`) and `send-message` takes their `media_ids` (checked to be the user's own by `grabAttachableMedia`), linked in `bot_message_media`; messages carry `attachments` (`BotMessageWithAttachments`). `mark-read.ts` clears a session's unread when a message arrives in the open conversation; The page renders a session's latest `AppData.ConversationFirstMessages` messages (`bot_messages_more` says there are older), and the thread loads older ones `AppData.ConversationOlderMessages` at a time as it scrolls to its top (`older-messages.ts`, `use-older-messages`, `older-messages-loader.tsx`). Bot and workspace lists in page props go through `railBots` / `railWorkspaces` (`src/utils/`): no core instructions, memory or workspace context, and the last message cut to a preview (`AppData.RailPreviewMaxChars`); the header avatar's edit form loads the instructions when it opens (`bot-instructions.ts`, `use-bot-instructions`). `session-messages.ts` returns a session's latest messages, which the open conversation polls every `AppData.BotReplyCatchUpMs` while a reply is due (`use-reply-catch-up`), so a reply whose live push was missed (socket down) still lands without a reload (replayed as a `server:bot-message`, so the rail and unread take it too); `create-bot.ts` adds a worker (and puts it in a workspace when given one); `save-harness-settings.ts` saves a worker bot's models. +- **Server:** `src/functions/backend/db/bots/` (`grab-bots`, `grab-bot-messages`, `grab-active-bot-task`, `grab-user-bot`, `create-bot`, `insert-user-message`). `api/admin/bots/send-message.ts` posts a user message to a bot the session user chats with (zod: `src/schemas/bot-schema.ts`), queues the bot's reply (`enqueueBotReply`) and wakes the queue (`src/bots-queue/wake-bots-queue.ts`; never import `src/bots-queue/index.ts` from a page or API route, it pulls the bunext runtime into the page bundles). `upload-attachment.ts` stores one file for the next message (private `media` row, paradigm `bot-attachment`, type from `attachmentFileKind`, only a file whose extension the extractor unpacks gets `text_content` - `canExtractText`, cut to `AppData.BotAttachmentTextStoreMaxChars` - limits in `AppData.BotAttachment*`) and `send-message` takes their `media_ids` (checked to be the user's own by `grabAttachableMedia`), linked in `bot_message_media`; messages carry `attachments` (`BotMessageWithAttachments`). `mark-read.ts` clears a session's unread when a message arrives in the open conversation; The page renders a session's latest `AppData.ConversationFirstMessages` messages (`bot_messages_more` says there are older), and the thread loads older ones `AppData.ConversationOlderMessages` at a time as it scrolls to its top (`older-messages.ts`, `use-older-messages`, `older-messages-loader.tsx`). Bot and workspace lists in page props go through `railBots` / `railWorkspaces` (`src/utils/`): no core instructions, memory or workspace context, and the last message cut to a preview (`AppData.RailPreviewMaxChars`); the header avatar's edit form loads the instructions when it opens (`bot-instructions.ts`, `use-bot-instructions`). `session-messages.ts` returns a session's latest messages, which the open conversation polls every `AppData.BotReplyCatchUpMs` while a reply is due (`use-reply-catch-up`), so a reply whose live push was missed (socket down) still lands without a reload (replayed as a `server:bot-message`, so the rail and unread take it too); `create-bot.ts` adds a worker (and puts it in a workspace when given one); `save-harness-settings.ts` saves a worker bot's models. - **Composer:** a Tiptap editor (`(hooks)/use-composer-editor.ts`, `(partials)/composer-editor.tsx`), not a textarea, loaded with the page and focused automatically, that reads and writes markdown (`@tiptap/markdown`), so drafts and sent messages are markdown. Drafts are written to `localStorage` on a 250ms debounce (`useComposerDraft`), flushed on unmount. Bold, italic, strike, code, lists, quotes and links (no headings), from shortcuts, markdown typing or the selection bubble (`composer-format-bubble.tsx`, items in `(data)/composer-formats.ts`). `@` mentions a bot and `#` a workspace (`(data)/composer-mention-kinds.ts`, menu `composer-mention-list.tsx` above the input, items from `CrewBotsContext` and `CrewWorkspacesContext` via `searchMentionItems`), shown as chips (`composer-mention-chip.tsx`) and written as `[bot:ID]` / `[workspace:ID]` (`(utils)/mention-markdown.ts`, `src/utils/workspace-ref.ts`). Enter sends, Shift+Enter breaks the line; in a list or code block Enter is the editor's and Mod+Enter sends. Sending shows the bubble at once and empties the input (with its quote and files), so a slow connection never waits on the request: the optimistic message rides in `pending_messages` in `conversation.tsx` until the response or a push replaces it with the saved one (`(utils)/drop-sent-message.ts`), and a failed send takes it back and restores the text and files. Selecting text in a message (`data-quotable`, `use-message-selection`) shows a Quote button (`quote-selection-button.tsx`) that inserts it as a blockquote at the cursor (`ComposerHandle.insertQuote`). Before a reply, `grabMessageRefNames` names the refs in the user's messages that the user reaches, and `botMessagesToHarness` adds a `[Mentions: ...]` line after each one (`describeMessageRefs`). `@tiptap/react` needs `use-sync-external-store`, which is CommonJS and calls `require("react")`; bunext keeps React external in page bundles, so the app installs an ESM stand-in from `packages/use-sync-external-store-esm` (a `file:` dependency plus an override; the Dockerfile copies `packages/` before `bun install`). - **Message rendering:** bot messages are markdown, rendered with `react-markdown` + `remark-gfm` (raw HTML never rendered) through the twui-based map in `[bot_id]/(data)/message-markdown-components.tsx` (`markdown-link`, `markdown-list`, `markdown-code-block` partials). A bot names a crew member as `[bot:ID]` (`src/utils/bot-ref.ts`, linked to `#bot-ID` by `link-computer-refs.ts`): `(partials)/bot-ref-chip.tsx` shows its avatar and name, linked to its conversation (in the open workspace when it's in the crew there), from the live crew (`CrewBotsContext`), so a bot created in that reply links too; the coordinator is told to write them (`CoordinatorReplyNote`), and `find_bot` and `create_bot` return them. A workspace is named as `[workspace:ID]` (`#workspace-ID`, `(partials)/workspace-ref-chip.tsx`, from `CrewWorkspacesContext`). User messages are markdown too, rendered in the ink bubble by `user-bubble.tsx` with `UserMarkdownComponents` and `remark-breaks` (a single line break stays one, so older plain messages read as typed). - **Replies and reactions:** every bot message has a row under it (`(partials)/message-actions.tsx`, handlers from `BotMessageActionsContext`) to reply or react; the user's own messages only show the bot's reactions, read-only. Replying quotes the message above the composer (`composer-reply-preview.tsx`) and the sent one carries `bot_messages.reply_to_message_id` (not a foreign key: a deleted quote only drops it); `send-message` takes any message of the same conversation (the user's, same bot and workspace, any session, so a rollover or a moved-on thread still sends), and the conversation drops a picked quote when its session changes. Messages carry `reply_to` (`grabMessagesReplies`, also on relayed messages), shown by `message-reply-quote.tsx`, and the bot reads it as `[Replying to ...]` before the user's turn (`botMessagesToHarness`). Reactions are a fixed set (`BotMessageReactions` in `src/dict/bot-dict.ts`) in `bot_message_reactions` (one row per message, emoji and user, or bot); `api/admin/bots/react-message.ts` toggles the user's own on a bot message in their sessions (`grabUserBotMessage`, `toggleMessageReaction`), and every bot gets `react_to_message` (`src/harness/tools/bot/react-to-message-tool.ts`, chat replies and tasks: a message in the session it answers, the user's latest by default). Both push the message's fresh reactions (`grabMessagesReactions`, `mine` for the session user) to the session's user as `server:bot-reaction` (`BotMessageReactionData`), which replaces the message's own; `grabBotMessages` adds them when given `user_id`. Reactions aren't relayed between processes (production runs one). diff --git a/package.json b/package.json index 9cb17fc..3b32a8f 100644 --- a/package.json +++ b/package.json @@ -13,6 +13,7 @@ "db:backup": "bunx bun-sqlite backup", "seed": "bun src/scripts/seed-super-admin.tsx", "seed:bots": "bun src/scripts/seed-bots.ts", + "media:prune-text": "bun src/scripts/prune-media-text-content.ts", "start": "MAIN_SERVER=1 NODE_ENV=production bun src/server.ts", "typecheck": "tsc --noEmit", "test": "bun test", diff --git a/src/bots-queue/load-attachment-parts.ts b/src/bots-queue/load-attachment-parts.ts index 5ef8606..19dc6c8 100644 --- a/src/bots-queue/load-attachment-parts.ts +++ b/src/bots-queue/load-attachment-parts.ts @@ -27,7 +27,10 @@ export default async function loadAttachmentParts({ messages }: Params) { .slice(-AppData.BotReplyAttachmentMessages) .map((message) => Number(message.id)); - const media = await grabMessagesMedia({ message_ids: recent }); + const media = await grabMessagesMedia({ + message_ids: recent, + with_text_content: true, + }); const parts = new Map(); for (const [message_id, files] of media) { diff --git a/src/data/app-data.ts b/src/data/app-data.ts index eb1981d..72fa07a 100644 --- a/src/data/app-data.ts +++ b/src/data/app-data.ts @@ -388,6 +388,13 @@ export const AppData = { BotAttachmentMaxBytes: 20 * 1024 * 1024, BotAttachmentMaxFiles: 6, BotAttachmentMaxTextChars: 20000, + /** + * How much of a document's text is kept on its `media` row. A ceiling + * on the column itself: without it a 20 MB PDF left 20 MB of text in + * the database, even though a reply reads at most + * `BotAttachmentMaxTextChars` of it. + */ + BotAttachmentTextStoreMaxChars: 200000, /** * Only this many of the latest messages with files hand the files * themselves to a reply; older ones are named. diff --git a/src/functions/backend/db/bot-files/grab-bot-file.ts b/src/functions/backend/db/bot-files/grab-bot-file.ts index d73e8c0..c4f69b0 100644 --- a/src/functions/backend/db/bot-files/grab-bot-file.ts +++ b/src/functions/backend/db/bot-files/grab-bot-file.ts @@ -3,6 +3,7 @@ import type { BUN_SQLITE_COBBLE_BOT_FILES, BUN_SQLITE_COBBLE_MEDIA, } from "@/db/types/db"; +import mediaColumns from "@/src/functions/backend/db/media/media-columns"; import { canReachScope, type StorageScope } from "@/src/utils/storage-access"; type Params = { @@ -59,7 +60,7 @@ export default async function grabBotFile({ const media = file.media_id ? await BunSQLite.sql({ - sql: `SELECT * FROM media WHERE id = ? LIMIT 1`, + sql: `SELECT ${mediaColumns({ alias: "m" })} FROM media m WHERE m.id = ? LIMIT 1`, values: [file.media_id], }).then((r) => r.payload?.[0] || null) : null; diff --git a/src/functions/backend/db/bot-files/ingest-bot-file.ts b/src/functions/backend/db/bot-files/ingest-bot-file.ts index ad5caa3..2f58cd7 100644 --- a/src/functions/backend/db/bot-files/ingest-bot-file.ts +++ b/src/functions/backend/db/bot-files/ingest-bot-file.ts @@ -57,7 +57,13 @@ export default async function ingestBotFile({ }).then((res) => res.payload?.[0]) : undefined; - if (Number(file.size) > AppData.IngestMaxBytes && !media?.text_content) { + /** + * Decided on the file's size alone. A cached `text_content` is not a + * reason to read an oversized file: it was only ever a slice of it, + * and taking it as the whole document made a file too big to unpack + * look already read. + */ + if (Number(file.size) > AppData.IngestMaxBytes) { await BunSQLite.sql({ sql: `UPDATE bot_files SET ingest_error = ?, updated_at = ? WHERE id = ?`, values: [`Too large to read (over ${AppData.IngestMaxBytes / (1024 * 1024)} MB).`, Date.now(), file.id], diff --git a/src/functions/backend/db/bot-files/media-type-for-file.ts b/src/functions/backend/db/bot-files/media-type-for-file.ts index 6a287c3..efa3038 100644 --- a/src/functions/backend/db/bot-files/media-type-for-file.ts +++ b/src/functions/backend/db/bot-files/media-type-for-file.ts @@ -1,5 +1,6 @@ import type { MediaType } from "@/src/types"; import { BotImageFileExtensions } from "@/src/dict/bot-file-dict"; +import extensionOf from "@/src/utils/extension-of"; type Params = { /** @@ -27,19 +28,10 @@ export default function mediaTypeForFile({ name, mime_type }: Params): MediaType } /** - * Its extension, lower case, without the dot. + * Its extension, lower case, without the dot. Shared, so the storage + * catalog and the text extractor read a name the same way. */ -export function extensionOf({ name, mime_type }: Params) { - const from_name = name?.includes(".") ? name.split(".").pop() : undefined; - - if (from_name) return from_name.toLowerCase().replace(/[^a-z0-9]/g, ""); - - const from_mime = mime_type && !mime_type.includes("octet-stream") - ? mime_type.split("/").pop() - : undefined; - - return from_mime ? from_mime.toLowerCase().replace(/[^a-z0-9]/g, "") : undefined; -} +export { extensionOf }; /** * A MIME type for the row, from the extension when there isn't one. diff --git a/src/functions/backend/db/bot-files/store-chat-attachments.test.ts b/src/functions/backend/db/bot-files/store-chat-attachments.test.ts new file mode 100644 index 0000000..27f667e --- /dev/null +++ b/src/functions/backend/db/bot-files/store-chat-attachments.test.ts @@ -0,0 +1,34 @@ +import { describe, expect, test } from "bun:test"; + +/** + * The module's own source, since the regression it guards is what the code + * writes to the database, not what it returns. + */ +async function source() { + return await Bun.file( + `${import.meta.dir}/store-chat-attachments.ts`, + ).text(); +} + +describe("storeChatAttachments", () => { + /** + * The same text used to be copied onto the new `bot-file` media row, + * so a chat upload kept a second copy of its extracted text (and left + * it behind when the upload was pruned). The text belongs only on the + * message's own `bot-attachment` row, where a reply reads it; + * ingestion re-reads the bytes through `extractBotFileText`. + */ + test("never writes text_content onto the file's media row", async () => { + const code = await source(); + + expect(code).not.toContain("SET text_content"); + expect(code).not.toContain("text_content = ?"); + }); + + test("still saves the bytes and queues the ingest", async () => { + const code = await source(); + + expect(code).toContain("recordBotFile("); + expect(code).toContain("enqueueFileIngest("); + }); +}); diff --git a/src/functions/backend/db/bot-files/store-chat-attachments.ts b/src/functions/backend/db/bot-files/store-chat-attachments.ts index e5384e1..e2c29b6 100644 --- a/src/functions/backend/db/bot-files/store-chat-attachments.ts +++ b/src/functions/backend/db/bot-files/store-chat-attachments.ts @@ -1,4 +1,3 @@ -import BunSQLite from "@moduletrace/bun-sqlite"; import type { BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db"; import { BotUploadFolder } from "@/src/dict/bot-file-dict"; import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest"; @@ -44,6 +43,13 @@ export default async function storeChatAttachments({ */ const name = safeName({ name: file.title || `file-${file.id}` }); + /** + * The text stays on the message's own `bot-attachment` row, + * where a reply reads it. Copying it onto the file's row kept + * the same text twice and left the copy behind when the + * upload was pruned; ingestion re-reads the bytes once + * through `extractBotFileText` instead. + */ const saved = await recordBotFile({ bot_id, scope: workspace_id ? "workspace" : "bot", @@ -57,17 +63,6 @@ export default async function storeChatAttachments({ user_id, }); - /** - * The same text the upload already extracted, so a PDF isn't - * unpacked a second time. - */ - if (file.text_content && saved.media_id) { - await BunSQLite.sql({ - sql: `UPDATE media SET text_content = ? WHERE id = ?`, - values: [file.text_content, saved.media_id], - }).catch(() => {}); - } - await enqueueFileIngest({ file_id: saved.id! }); } catch (error) { await logError({ diff --git a/src/functions/backend/db/bots/grab-messages-media.test.ts b/src/functions/backend/db/bots/grab-messages-media.test.ts new file mode 100644 index 0000000..22ebee9 --- /dev/null +++ b/src/functions/backend/db/bots/grab-messages-media.test.ts @@ -0,0 +1,72 @@ +import { describe, expect, test } from "bun:test"; +import { Database } from "bun:sqlite"; + +const db = new Database(":memory:"); + +db.run( + `CREATE TABLE media ( + id INTEGER PRIMARY KEY, + created_at INTEGER, + updated_at INTEGER, + user_id INTEGER, + title TEXT, + description TEXT, + summary TEXT, + media_type TEXT, + media_write_path TEXT, + media_thumbnail_write_path TEXT, + mime_type TEXT, + text_content TEXT, + is_primary INTEGER, + is_rag_ingested INTEGER, + archived INTEGER + )`, +); +db.run(`CREATE TABLE bot_message_media (id INTEGER PRIMARY KEY, message_id INTEGER, media_id INTEGER)`); +db.run( + `INSERT INTO media (id, title, media_type, media_write_path, mime_type, text_content) + VALUES (1, 'invoice.pdf', 'document', 'media/1.pdf', 'application/pdf', 'Total: 12')`, +); +db.run(`INSERT INTO bot_message_media (id, message_id, media_id) VALUES (1, 10, 1)`); + +/** + * The query `grabMessagesMedia` builds, with the text column included or + * left out. It is the largest column in the table, so only the reply path + * that hands a document to a model asks for it. + */ +function read({ with_text_content }: { with_text_content: boolean }) { + const columns = [ + "m.id", + "m.title", + "m.media_type", + "m.media_write_path", + "m.mime_type", + ...(with_text_content ? ["m.text_content"] : []), + ].join(", "); + + return db + .query( + `SELECT ${columns}, l.message_id FROM bot_message_media l + JOIN media m ON m.id = l.media_id + WHERE l.message_id = 10`, + ) + .get() as Record; +} + +describe("grabMessagesMedia", () => { + test("a plain read never drags the extracted text along", () => { + expect(read({ with_text_content: false })).not.toHaveProperty("text_content"); + }); + + test("the reply path can ask for it", () => { + expect(read({ with_text_content: true }).text_content).toBe("Total: 12"); + }); + + test("the row's name and path are always there", () => { + expect(read({ with_text_content: false })).toMatchObject({ + title: "invoice.pdf", + media_write_path: "media/1.pdf", + message_id: 10, + }); + }); +}); diff --git a/src/functions/backend/db/bots/grab-messages-media.ts b/src/functions/backend/db/bots/grab-messages-media.ts index 0737995..bf4bb8f 100644 --- a/src/functions/backend/db/bots/grab-messages-media.ts +++ b/src/functions/backend/db/bots/grab-messages-media.ts @@ -1,8 +1,15 @@ import BunSQLite from "@moduletrace/bun-sqlite"; import type { BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db"; +import mediaColumns from "@/src/functions/backend/db/media/media-columns"; type Params = { message_ids: number[]; + /** + * Also read `text_content`, the extracted text a reply hands a + * document to the model as. Off by default: it is the largest column + * in the table and nothing else here uses it. + */ + with_text_content?: boolean; }; export type MessageMediaRow = BUN_SQLITE_COBBLE_MEDIA & { message_id: number }; @@ -11,16 +18,21 @@ export type MessageMediaRow = BUN_SQLITE_COBBLE_MEDIA & { message_id: number }; * The files attached to these messages, by message id, in the order they * were attached. */ -export default async function grabMessagesMedia({ message_ids }: Params) { +export default async function grabMessagesMedia({ + message_ids, + with_text_content, +}: Params) { const by_message = new Map(); const ids = [...new Set(message_ids.filter(Boolean))]; if (!ids.length) return by_message; + const columns = mediaColumns({ alias: "m", with_text_content }); + const rows = ( await BunSQLite.sql({ - sql: `SELECT m.*, l.message_id FROM bot_message_media l + sql: `SELECT ${columns}, l.message_id FROM bot_message_media l JOIN media m ON m.id = l.media_id WHERE l.message_id IN (${ids.map(() => "?").join(", ")}) ORDER BY l.id ASC`, diff --git a/src/functions/backend/db/media/grab-media.ts b/src/functions/backend/db/media/grab-media.ts index 29cecd1..0ffccfd 100644 --- a/src/functions/backend/db/media/grab-media.ts +++ b/src/functions/backend/db/media/grab-media.ts @@ -6,6 +6,7 @@ import BunSQLite, { type APIResponseObject } from "@moduletrace/bun-sqlite"; import type { SQLInsertGenValueType } from "@moduletrace/bun-sqlite"; import _ from "lodash"; import appendUsersToMedia from "./append-users-to-media"; +import mediaColumns from "./media-columns"; type Params = { query?: ApiReqParams; @@ -23,7 +24,7 @@ export default async function grabMedia( const hasParadigms = Boolean(query?.media_paradigms?.length); let sql = `SELECT\n`; - sql += ` m.*,\n`; + sql += ` ${mediaColumns({ alias: "m" })},\n`; sql += ` GROUP_CONCAT(DISTINCT mp.media_paradigm) AS media_paradigms\n`; sql += `FROM media m\n`; sql += `LEFT JOIN media_paradigms mp ON m.id = mp.media_id\n`; diff --git a/src/functions/backend/db/media/media-columns.test.ts b/src/functions/backend/db/media/media-columns.test.ts new file mode 100644 index 0000000..391e6bc --- /dev/null +++ b/src/functions/backend/db/media/media-columns.test.ts @@ -0,0 +1,55 @@ +import { describe, expect, test } from "bun:test"; +import schema from "@/db/schema"; +import mediaColumns from "./media-columns"; + +/** + * The schema's own list, so a column added later can't quietly be left + * out of (or forgotten in) the select this guards. + */ +function mediaSchemaFields(): string[] { + const tables = (schema as { tables?: { tableName: string; fields: { fieldName: string }[] }[] }).tables || []; + const media = tables.find((table) => table.tableName == "media"); + + /** + * `id`, `created_at` and `updated_at` are added to every table by the + * ORM, so they aren't in the schema's own field list. + */ + return [...(media?.fields || []).map((field) => field.fieldName), "id", "created_at", "updated_at"]; +} + +describe("mediaColumns", () => { + test("names every media column except the extracted text", () => { + const listed = mediaColumns({ alias: "m" }) + .split(", ") + .map((column) => column.replace(/^m\./, "")) + .sort(); + + const expected = mediaSchemaFields() + .filter((field) => field != "text_content") + .sort(); + + expect(listed).toEqual(expected); + }); + + test("never selects text_content unless asked", () => { + expect(mediaColumns()).not.toContain("text_content"); + expect(mediaColumns({ with_text_content: true })).toContain("m.text_content"); + }); + + test("asking for the text adds it to the same full list", () => { + const with_text = mediaColumns({ with_text_content: true }); + + expect(with_text).toContain(mediaColumns()); + expect(with_text.split(", ")).toHaveLength(mediaColumns().split(", ").length + 1); + }); + + test("qualifies every column, so a join can't make one ambiguous", () => { + for (const column of mediaColumns({ alias: "media" }).split(", ")) { + expect(column.startsWith("media.")).toBe(true); + } + }); + + test("the alias defaults to m", () => { + expect(mediaColumns().startsWith("m.id")).toBe(true); + }); +}); diff --git a/src/functions/backend/db/media/media-columns.ts b/src/functions/backend/db/media/media-columns.ts new file mode 100644 index 0000000..add056e --- /dev/null +++ b/src/functions/backend/db/media/media-columns.ts @@ -0,0 +1,40 @@ +/** + * The `media` columns a normal read takes, with a table alias. + * + * `text_content` is deliberately left out: it holds up to + * `AppData.BotAttachmentTextStoreMaxChars` of extracted document text, and + * every caller that only needs a file's name, type or path (`SELECT m.*` + * did) pulled it into memory, into page props and into API responses. + * Read it explicitly where it is really used: + * `src/bots-queue/load-attachment-parts.ts` and + * `src/functions/backend/db/bot-files/ingest-bot-file.ts`. + */ +export default function mediaColumns({ + alias = "m", + with_text_content, +}: { alias?: string; with_text_content?: boolean } = {}) { + const columns = [ + "id", + "created_at", + "updated_at", + "user_id", + "title", + "description", + "summary", + "media_type", + "media_write_path", + "media_thumbnail_write_path", + "mime_type", + "is_primary", + "is_rag_ingested", + "archived", + ]; + + /** + * Only the two paths that really use it ask for it: the reply that + * hands a document to a model, and ingestion. + */ + if (with_text_content) columns.push("text_content"); + + return columns.map((column) => `${alias}.${column}`).join(", "); +} diff --git a/src/functions/backend/db/media/plan-text-content-prune.test.ts b/src/functions/backend/db/media/plan-text-content-prune.test.ts new file mode 100644 index 0000000..0f9b2fc --- /dev/null +++ b/src/functions/backend/db/media/plan-text-content-prune.test.ts @@ -0,0 +1,65 @@ +import { describe, expect, test } from "bun:test"; +import { AppData } from "@/src/data/app-data"; +import planTextContentPrune from "./plan-text-content-prune"; + +describe("planTextContentPrune", () => { + test("clears the raw bytes stored for a type the extractor can't read", () => { + const changes = planTextContentPrune({ + rows: [ + { id: 1, title: "logo.svg", mime_type: "image/svg+xml", text_content: "" }, + { id: 2, title: "bundle.zip", mime_type: "application/zip", text_content: "PK\u0003\u0004" }, + { id: 3, title: "sheet.xlsx", mime_type: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", text_content: "PK\u0003\u0004" }, + ], + }); + + expect(changes.map((change) => change.media_id)).toEqual([1, 2, 3]); + expect(changes.every((change) => change.text_content === null)).toBe(true); + expect(changes.every((change) => change.after_chars === 0)).toBe(true); + }); + + test("leaves a document's text alone", () => { + const changes = planTextContentPrune({ + rows: [ + { id: 4, title: "invoice.pdf", mime_type: "application/pdf", text_content: "Total: 12" }, + { id: 5, title: "notes.md", mime_type: "text/markdown", text_content: "# Hi" }, + { id: 6, title: "rows.csv", mime_type: "text/csv", text_content: "a,b\n1,2" }, + ], + }); + + expect(changes).toEqual([]); + }); + + test("cuts a document over the store ceiling", () => { + const text = "a".repeat(AppData.BotAttachmentTextStoreMaxChars + 100); + const changes = planTextContentPrune({ + rows: [{ id: 7, title: "long.txt", mime_type: "text/plain", text_content: text }], + }); + + expect(changes).toHaveLength(1); + expect(changes[0]!.after_chars).toBeLessThanOrEqual(AppData.BotAttachmentTextStoreMaxChars); + expect(changes[0]!.text_content).toContain("Truncated"); + expect(changes[0]!.before_chars).toBe(text.length); + }); + + test("nothing to say about a row with no text or no id", () => { + expect( + planTextContentPrune({ + rows: [ + { id: 8, title: "empty.pdf", text_content: "" }, + { id: 9, title: "blank.pdf", text_content: " \n " }, + { id: 10, title: "none.pdf" }, + { title: "no id.pdf", text_content: "text" }, + ], + }), + ).toEqual([]); + }); + + test("counting what each change reclaims is left to the caller", () => { + const changes = planTextContentPrune({ + rows: [{ id: 11, title: "icon.svg", mime_type: "image/svg+xml", text_content: "12345" }], + }); + + expect(changes).toHaveLength(1); + expect(changes[0]!.before_chars - changes[0]!.after_chars).toBe(5); + }); +}); diff --git a/src/functions/backend/db/media/plan-text-content-prune.ts b/src/functions/backend/db/media/plan-text-content-prune.ts new file mode 100644 index 0000000..cf2df7d --- /dev/null +++ b/src/functions/backend/db/media/plan-text-content-prune.ts @@ -0,0 +1,55 @@ +import type { BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db"; +import decideTextContent from "@/src/utils/decide-text-content"; + +export type TextContentChange = { + media_id: number; + name: string; + before_chars: number; + after_chars: number; + /** + * What the row's `text_content` should become: `null` clears it. + */ + text_content: string | null; +}; + +type Params = { + /** + * The `media` rows that have any `text_content`. + */ + rows: Partial[]; +}; + +/** + * What `pruneMediaTextContent` would change, worked out without touching + * the database: one entry per row whose text belongs nowhere or is over + * the store ceiling, and nothing at all for a row that is already right. + */ +export default function planTextContentPrune({ rows }: Params): TextContentChange[] { + const changes: TextContentChange[] = []; + + for (const row of rows) { + if (!row.id) continue; + + const before = row.text_content || ""; + + if (!before.trim()) continue; + + const text_content = decideTextContent({ + name: row.title, + mime_type: row.mime_type, + text: before, + }); + + if (text_content === before) continue; + + changes.push({ + media_id: Number(row.id), + name: row.title || "no name", + before_chars: before.length, + after_chars: text_content?.length || 0, + text_content, + }); + } + + return changes; +} diff --git a/src/functions/backend/db/media/prune-media-text-content.test.ts b/src/functions/backend/db/media/prune-media-text-content.test.ts new file mode 100644 index 0000000..84bf8f9 --- /dev/null +++ b/src/functions/backend/db/media/prune-media-text-content.test.ts @@ -0,0 +1,120 @@ +import { describe, expect, test } from "bun:test"; +import { Database } from "bun:sqlite"; +import { AppData } from "@/src/data/app-data"; +import pruneMediaTextContent, { type SqlRunner } from "./prune-media-text-content"; + +type Row = { id: number; title: string; mime_type: string; media_type: string; text_content: string | null; updated_at?: number }; + +function makeDb({ rows }: { rows: Row[] }) { + const db = new Database(":memory:"); + + db.run( + `CREATE TABLE media ( + id INTEGER PRIMARY KEY, + title TEXT, + mime_type TEXT, + media_type TEXT, + text_content TEXT, + updated_at INTEGER + )`, + ); + + const insert = db.prepare( + `INSERT INTO media (id, title, mime_type, media_type, text_content) VALUES (?, ?, ?, ?, ?)`, + ); + + for (const row of rows) insert.run(row.id, row.title, row.mime_type, row.media_type, row.text_content); + + const run: SqlRunner = async ({ sql, values }) => { + const statement = db.prepare(sql); + + if (sql.trim().toUpperCase().startsWith("SELECT")) { + return { payload: statement.all(...((values || []) as never[])) as unknown[] }; + } + + statement.run(...((values || []) as never[])); + + return { payload: [] }; + }; + + const read = (id: number) => + (db.query(`SELECT text_content FROM media WHERE id = ?`).get(id) as { text_content: string | null }).text_content; + + return { db, run, read }; +} + +describe("pruneMediaTextContent", () => { + test("clears raw bytes from a type the extractor can't read, and keeps a document's text", async () => { + const { run, read } = makeDb({ + rows: [ + { id: 1, title: "logo.svg", mime_type: "image/svg+xml", media_type: "document", text_content: "" }, + { id: 2, title: "bundle.zip", mime_type: "application/zip", media_type: "document", text_content: "PK\u0003\u0004binary" }, + { id: 3, title: "invoice.pdf", mime_type: "application/pdf", media_type: "document", text_content: "Total: 12" }, + ], + }); + + const result = await pruneMediaTextContent({ run }); + + expect(result.checked).toBe(3); + expect(result.cleared).toBe(2); + expect(result.truncated).toBe(0); + expect(read(1)).toBeNull(); + expect(read(2)).toBeNull(); + expect(read(3)).toBe("Total: 12"); + }); + + test("cuts an oversized document to the store ceiling", async () => { + const text = "a".repeat(AppData.BotAttachmentTextStoreMaxChars + 1000); + const { run, read } = makeDb({ + rows: [{ id: 4, title: "huge.txt", mime_type: "text/plain", media_type: "document", text_content: text }], + }); + + const result = await pruneMediaTextContent({ run }); + const after = read(4)!; + + expect(result.truncated).toBe(1); + expect(result.cleared).toBe(0); + expect(after.length).toBeLessThanOrEqual(AppData.BotAttachmentTextStoreMaxChars); + expect(after).toContain("Truncated"); + expect(result.reclaimed_bytes).toBeGreaterThan(0); + }); + + test("is safe to rerun: a second sweep changes nothing", async () => { + const { run, read } = makeDb({ + rows: [{ id: 5, title: "icon.svg", mime_type: "image/svg+xml", media_type: "document", text_content: "raw" }], + }); + + await pruneMediaTextContent({ run }); + const second = await pruneMediaTextContent({ run }); + + expect(second.checked).toBe(0); + expect(second.changes).toEqual([]); + expect(second.reclaimed_bytes).toBe(0); + expect(read(5)).toBeNull(); + }); + + test("stamps updated_at on the rows it rewrites", async () => { + const now = 1_700_000_000_000; + const { run, db } = makeDb({ + rows: [{ id: 6, title: "x.zip", mime_type: "application/zip", media_type: "document", text_content: "raw" }], + }); + + await pruneMediaTextContent({ run, now }); + + expect( + (db.query(`SELECT updated_at FROM media WHERE id = 6`).get() as { updated_at: number }).updated_at, + ).toBe(now); + }); + + test("reports every change with its row, name and sizes", async () => { + const { run } = makeDb({ + rows: [{ id: 7, title: "sheet.xlsx", mime_type: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", media_type: "document", text_content: "PK\u0003\u0004" }], + }); + + const result = await pruneMediaTextContent({ run }); + + expect(result.changes).toEqual([ + { media_id: 7, name: "sheet.xlsx", before_chars: 4, after_chars: 0, text_content: null }, + ]); + }); +}); diff --git a/src/functions/backend/db/media/prune-media-text-content.ts b/src/functions/backend/db/media/prune-media-text-content.ts new file mode 100644 index 0000000..8bb0299 --- /dev/null +++ b/src/functions/backend/db/media/prune-media-text-content.ts @@ -0,0 +1,75 @@ +import BunSQLite from "@moduletrace/bun-sqlite"; +import type { BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db"; +import planTextContentPrune, { type TextContentChange } from "./plan-text-content-prune"; + +/** + * One statement against the database. `BunSQLite.sql` is the real one; a + * test passes its own, so the whole sweep can be run without the app's + * database. + */ +export type SqlRunner = (params: { + sql: string; + values?: (string | number | null)[]; +}) => Promise<{ payload?: unknown[] | null }>; + +export type PruneMediaTextResult = { + checked: number; + cleared: number; + truncated: number; + reclaimed_bytes: number; + changes: TextContentChange[]; +}; + +type Params = { + run?: SqlRunner; + /** + * When the sweep runs, for the rows it writes. + */ + now?: number; +}; + +/** + * Brings every `media.text_content` in line with what may be stored there + * (`planTextContentPrune`): no text on a type the extractor can't read, + * and no text longer than the store ceiling. + * + * Run once per install after the allowlist landed; safe to rerun, since a + * row that is already right is not written. + */ +export default async function pruneMediaTextContent({ + run, + now = Date.now(), +}: Params = {}): Promise { + const sql: SqlRunner = + run || (({ sql, values }) => BunSQLite.sql({ sql, values })); + + const res = await sql({ + sql: `SELECT id, title, mime_type, media_type, text_content + FROM media + WHERE text_content IS NOT NULL AND TRIM(text_content) != ''`, + }); + + const rows = (res.payload || []) as BUN_SQLITE_COBBLE_MEDIA[]; + const changes = planTextContentPrune({ rows }); + const result: PruneMediaTextResult = { + checked: rows.length, + cleared: 0, + truncated: 0, + reclaimed_bytes: 0, + changes, + }; + + for (const change of changes) { + await sql({ + sql: `UPDATE media SET text_content = ?, updated_at = ? WHERE id = ?`, + values: [change.text_content, now, change.media_id], + }); + + result.reclaimed_bytes += change.before_chars - change.after_chars; + + if (change.text_content === null) result.cleared += 1; + else result.truncated += 1; + } + + return result; +} diff --git a/src/functions/backend/db/media/sanitize-media-row-text-content.test.ts b/src/functions/backend/db/media/sanitize-media-row-text-content.test.ts new file mode 100644 index 0000000..5326983 --- /dev/null +++ b/src/functions/backend/db/media/sanitize-media-row-text-content.test.ts @@ -0,0 +1,53 @@ +import { describe, expect, test } from "bun:test"; +import { AppData } from "@/src/data/app-data"; +import sanitizeMediaRowTextContent from "./sanitize-media-row-text-content"; + +describe("sanitizeMediaRowTextContent", () => { + test("keeps a document's text", () => { + const row = sanitizeMediaRowTextContent({ + row: { title: "invoice.pdf", mime_type: "application/pdf", text_content: "Total: 12" }, + }); + + expect(row.text_content).toBe("Total: 12"); + }); + + test("refuses raw bytes typed into a row that names a binary", () => { + const row = sanitizeMediaRowTextContent({ + row: { title: "logo.svg", mime_type: "image/svg+xml", text_content: "" }, + }); + + expect(row.text_content).toBeUndefined(); + }); + + test("an update is judged on the file it already is", () => { + const row = sanitizeMediaRowTextContent({ + row: { text_content: "PK\u0003\u0004" }, + existing: { title: "bundle.zip", mime_type: "application/zip" }, + }); + + expect(row.text_content).toBeUndefined(); + }); + + test("an update with no text is left as it is", () => { + const row = sanitizeMediaRowTextContent({ + row: { title: "renamed.pdf" }, + existing: { title: "invoice.pdf" }, + }); + + expect(row).toEqual({ title: "renamed.pdf" }); + }); + + test("a long text without a name is still cut, not dropped", () => { + const text = "a".repeat(AppData.BotAttachmentTextStoreMaxChars + 10); + const row = sanitizeMediaRowTextContent({ row: { text_content: text } }); + + expect(row.text_content).toContain("Truncated"); + expect(row.text_content!.length).toBeLessThanOrEqual(AppData.BotAttachmentTextStoreMaxChars); + }); + + test("empty text becomes unset", () => { + const row = sanitizeMediaRowTextContent({ row: { title: "a.pdf", text_content: "" } }); + + expect(row.text_content).toBeUndefined(); + }); +}); diff --git a/src/functions/backend/db/media/sanitize-media-row-text-content.ts b/src/functions/backend/db/media/sanitize-media-row-text-content.ts new file mode 100644 index 0000000..6182f87 --- /dev/null +++ b/src/functions/backend/db/media/sanitize-media-row-text-content.ts @@ -0,0 +1,49 @@ +import type { BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db"; +import capTextContent from "@/src/utils/cap-text-content"; +import decideTextContent from "@/src/utils/decide-text-content"; + +type Params = { + /** + * The row as the request sent it. + */ + row: Partial; + /** + * The row already in the database, for an update: its name and type + * decide whether the new text may be kept when the update doesn't + * repeat them. + */ + existing?: Partial | null; +}; + +/** + * A `media` row whose `text_content` is what the column may hold. + * + * The upload path decides this itself, but the CRUD API writes rows + * straight through, so a typed-in `text_content` could store raw bytes or + * megabytes of text. An update that carries no text is left alone. + */ +export default function sanitizeMediaRowTextContent({ + row, + existing, +}: Params): Partial { + const text = row.text_content; + + if (text === undefined) return row; + + const name = row.title || existing?.title || null; + const mime_type = row.mime_type || existing?.mime_type || null; + + /** + * With no name and no type there is nothing to judge it by, so it is + * only cut: dropping text that a rename would have kept would be + * worse than a slightly generous cache. + */ + const text_content = + name || mime_type + ? decideTextContent({ name, mime_type, text }) + : text + ? capTextContent({ text }) + : null; + + return { ...row, text_content: text_content ?? undefined }; +} diff --git a/src/functions/backend/db/users/media/file-to-text.test.ts b/src/functions/backend/db/users/media/file-to-text.test.ts new file mode 100644 index 0000000..bf523ea --- /dev/null +++ b/src/functions/backend/db/users/media/file-to-text.test.ts @@ -0,0 +1,35 @@ +import { describe, expect, test } from "bun:test"; +import fileToText from "./file-to-text"; + +function stream({ text }: { text: string }) { + return Buffer.from(text, "utf-8") as Buffer; +} + +describe("fileToText", () => { + test("reads the plain text types as they are", async () => { + for (const file_type of ["txt", "text", "md", "markdown", "csv", "tsv", "json", "html", "htm", "xml", "yml", "yaml", "log"]) { + expect(await fileToText({ file_type, file_stream: stream({ text: "hello" }) })).toBe("hello"); + } + }); + + test("keeps the real case of the text", async () => { + expect(await fileToText({ file_type: "md", file_stream: stream({ text: "# Heading\n\nBody" }) })).toBe( + "# Heading\n\nBody", + ); + }); + + /** + * The bug this guards: the switch used to fall through to + * `buffer.toString("utf-8")`, so a zip, an SVG or a spreadsheet was + * "extracted" as garbled bytes and stored in `media.text_content`. + */ + test("refuses a binary instead of decoding its bytes", async () => { + for (const file_type of ["svg", "zip", "xlsx", "pptx", "bin", "exe", ""]) { + expect(await fileToText({ file_type, file_stream: stream({ text: "PK\u0003\u0004binary" }) })).toBeUndefined(); + } + }); + + test("no buffer is no text", async () => { + expect(await fileToText({ file_type: "txt" })).toBeUndefined(); + }); +}); diff --git a/src/functions/backend/db/users/media/file-to-text.ts b/src/functions/backend/db/users/media/file-to-text.ts index 2138b7f..c0381fc 100644 --- a/src/functions/backend/db/users/media/file-to-text.ts +++ b/src/functions/backend/db/users/media/file-to-text.ts @@ -36,9 +36,15 @@ export default async function fileToText({ buffer = file_path ? Buffer.from(await Bun.file(file_path).arrayBuffer()) : file_stream; - ext = (file_type || file_path?.split(".").pop() || "txt") - .toString() - .toLowerCase(); + const known_ext = file_type || file_path?.split(".").pop(); + + /** + * No extension and no type: nothing says this is text, so nothing + * gets decoded. Guessing "txt" is how a binary became text. + */ + if (!known_ext) return undefined; + + ext = known_ext.toString().toLowerCase(); } if (!buffer) return undefined; @@ -76,9 +82,32 @@ export default async function fileToText({ // }).join("\n\n"); // } - case "md": + /** + * The text types read as they are. Listed one by one, never a + * catch-all: decoding a zip, an SVG or a spreadsheet as UTF-8 + * fills the media row with raw bytes instead of text. + */ case "txt": - default: + case "text": + case "md": + case "markdown": + case "csv": + case "tsv": + case "json": + case "html": + case "htm": + case "xml": + case "yml": + case "yaml": + case "log": return buffer.toString("utf-8"); + + default: + /** + * Nothing readable locally: the caller decides whether to read + * it another way (an image or audio through the harness) or to + * leave its text unset. + */ + return undefined; } } diff --git a/src/functions/backend/db/users/media/upload-and-record-media.ts b/src/functions/backend/db/users/media/upload-and-record-media.ts index 910b3cc..f36bc61 100644 --- a/src/functions/backend/db/users/media/upload-and-record-media.ts +++ b/src/functions/backend/db/users/media/upload-and-record-media.ts @@ -17,6 +17,8 @@ import BunSQLite, { type APIResponseObject } from "@moduletrace/bun-sqlite"; import { existsSync, mkdirSync } from "fs"; import path from "path"; import sharp from "sharp"; +import canExtractText from "@/src/utils/can-extract-text"; +import capTextContent from "@/src/utils/cap-text-content"; import fileToText from "./file-to-text"; import MediaDataToArrayBuffer from "./media-data-to-array-buffer"; import sniffMediaExtension from "./sniff-media-ext"; @@ -180,14 +182,22 @@ export default async function uploadAndRecordMedia({ mode: 1, }); - if (extract_media_text_content) { + /** + * Only an extension the extractor really unpacks is read: any + * other binary has no text, and decoding it anyway stored raw + * bytes in the row. + */ + if ( + extract_media_text_content && + canExtractText({ name: media_name, mime_type: media_mime_type }) + ) { const media_text = await fileToText({ file_type: media_ext, file_stream: Buffer.from(media_array_buffer), }); if (media_text) { - media_text_content = media_text; + media_text_content = capTextContent({ text: media_text }); } } } diff --git a/src/harness/media/read-media.ts b/src/harness/media/read-media.ts index 717c15e..ef7c959 100644 --- a/src/harness/media/read-media.ts +++ b/src/harness/media/read-media.ts @@ -1,3 +1,4 @@ +import { BotTextFileExtensions } from "@/src/dict/bot-file-dict"; import fileToText from "@/src/functions/backend/db/users/media/file-to-text"; import AppError from "@/src/utils/app-error"; import transcribeAudio from "../audio/transcribe-audio"; @@ -24,7 +25,11 @@ type Params = { workspace_id?: number | string | null; }; -const DocumentExtensions = ["pdf", "docx", "txt", "md", "csv", "json", "html", "xml", "yaml", "yml"]; +/** + * One list, the same one the local extractor reads: keeping a second copy + * here is how a type got read by one and silently missed by the other. + */ +const DocumentExtensions: readonly string[] = BotTextFileExtensions; const TypeExtensions: Record = { "application/pdf": "pdf", diff --git a/src/pages/api/admin/bots/upload-attachment.ts b/src/pages/api/admin/bots/upload-attachment.ts index ac24a85..029a53a 100644 --- a/src/pages/api/admin/bots/upload-attachment.ts +++ b/src/pages/api/admin/bots/upload-attachment.ts @@ -8,6 +8,7 @@ import type { BotMessageAttachment } from "@/src/types"; import AppError from "@/src/utils/app-error"; import attachmentFileKind from "@/src/utils/attachment-file-kind"; import bufferFromBase64 from "@/src/utils/buffer-from-base-64"; +import canExtractText from "@/src/utils/can-extract-text"; import parseWithSchema from "@/src/utils/parse-with-schema"; import isReadOnlyUser from "@/src/utils/rbac/is-read-only-user"; import safeErrorMessage from "@/src/utils/safe-error-message"; @@ -64,7 +65,7 @@ export const handler: BunextAPIRouteHandler< media_mime_type: ext, media_paradigms: ["bot-attachment"], media: { title: name }, - extract_media_text_content: media_type == "document", + extract_media_text_content: canExtractText({ name, mime_type: mime }), user: { id: Number(user.id) }, }).catch(async (error) => { await logError({ title: `Attachment ${name} couldn't be saved`, error }).catch(() => {}); diff --git a/src/pages/api/admin/crud/(tables)/media/post.ts b/src/pages/api/admin/crud/(tables)/media/post.ts index e5f6508..3f56bc0 100644 --- a/src/pages/api/admin/crud/(tables)/media/post.ts +++ b/src/pages/api/admin/crud/(tables)/media/post.ts @@ -1,5 +1,6 @@ import type { BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db"; import checkUserAccess from "@/src/functions/backend/auth/check-user-access"; +import sanitizeMediaRowTextContent from "@/src/functions/backend/db/media/sanitize-media-row-text-content"; import type { AdminCrudAPIParams } from "@/src/types"; import BunSQLite from "@moduletrace/bun-sqlite"; import type { APIResponseObject } from "@moduletrace/bunext/types"; @@ -10,9 +11,13 @@ export default async function ( ): Promise { const { table, body, user_types, user } = params; + /** + * A row written straight through the API still may not carry raw + * bytes or megabytes in `text_content`. + */ let final_insert_data: BUN_SQLITE_COBBLE_MEDIA[] = [ ...(body?.insert_data || []), - ].map((d) => ({ ...d, user_id: user.id })); + ].map((d) => ({ ...sanitizeMediaRowTextContent({ row: d }), user_id: user.id })); const is_user_allowed_to_post_any = checkUserAccess({ includes: ["admin"], diff --git a/src/pages/api/admin/crud/(tables)/media/put.ts b/src/pages/api/admin/crud/(tables)/media/put.ts index a198027..16e8615 100644 --- a/src/pages/api/admin/crud/(tables)/media/put.ts +++ b/src/pages/api/admin/crud/(tables)/media/put.ts @@ -1,4 +1,6 @@ +import type { BunSQLiteTables, BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db"; import checkUserAccess from "@/src/functions/backend/auth/check-user-access"; +import sanitizeMediaRowTextContent from "@/src/functions/backend/db/media/sanitize-media-row-text-content"; import type { AdminCrudAPIParams } from "@/src/types"; import BunSQLite from "@moduletrace/bun-sqlite"; import type { APIResponseObject } from "@moduletrace/bunext/types"; @@ -29,9 +31,21 @@ export default async function ( targetId = user.id; } + /** + * The row it replaces decides whether the new text belongs there, so + * an update that only sends `text_content` is judged on the file's + * real name and type. + */ + const existing = targetId + ? await BunSQLite.select({ + table, + query: { query: { id: { value: targetId } } }, + }).then((res) => res.payload?.[0] || null) + : null; + const PUT = await BunSQLite.update({ table, - data: body?.update_data, + data: sanitizeMediaRowTextContent({ row: body.update_data, existing }), targetId, query: final_sql_query, }); diff --git a/src/pages/api/admin/tasks/attach-file.ts b/src/pages/api/admin/tasks/attach-file.ts index e7c61d4..66ab44a 100644 --- a/src/pages/api/admin/tasks/attach-file.ts +++ b/src/pages/api/admin/tasks/attach-file.ts @@ -9,6 +9,7 @@ import type { TaskDetail } from "@/src/types"; import AppError from "@/src/utils/app-error"; import attachmentFileKind from "@/src/utils/attachment-file-kind"; import bufferFromBase64 from "@/src/utils/buffer-from-base-64"; +import canExtractText from "@/src/utils/can-extract-text"; import parseWithSchema from "@/src/utils/parse-with-schema"; import isReadOnlyUser from "@/src/utils/rbac/is-read-only-user"; import safeErrorMessage from "@/src/utils/safe-error-message"; @@ -65,7 +66,7 @@ export const handler: BunextAPIRouteHandler> = asy media_mime_type: ext, media_paradigms: ["task-media"], media: { title: name }, - extract_media_text_content: media_type == "document", + extract_media_text_content: canExtractText({ name, mime_type: mime }), user: { id: Number(user.id) }, }); diff --git a/src/scripts/prune-media-text-content.ts b/src/scripts/prune-media-text-content.ts new file mode 100644 index 0000000..4a95bb6 --- /dev/null +++ b/src/scripts/prune-media-text-content.ts @@ -0,0 +1,40 @@ +import { AppData } from "@/src/data/app-data"; +import pruneMediaTextContent from "@/src/functions/backend/db/media/prune-media-text-content"; + +/** + * Cleans the `media.text_content` that should never have been stored. + * + * Before the extractor had an allowlist, every non-image/audio/video file + * was decoded as UTF-8 and its bytes were written to `text_content`: an + * SVG, a zip, a spreadsheet and an unknown binary all left raw (garbled) + * content in the database. Nothing capped the length either, so a large + * document could leave megabytes of text behind. + * + * Safe to rerun: it only writes a row whose text has to change. + * + * `bun src/scripts/prune-media-text-content.ts` + */ +async function main() { + const result = await pruneMediaTextContent(); + + for (const change of result.changes) { + console.log( + change.after_chars === 0 + ? `Media ${change.media_id} (${change.name}): dropped ${change.before_chars} characters.` + : `Media ${change.media_id} (${change.name}): cut ${change.before_chars} characters to ${change.after_chars}.`, + ); + } + + console.log( + `Done: ${result.cleared} row(s) cleared, ${result.truncated} truncated, about ${Math.round(result.reclaimed_bytes / 1024)} KB reclaimed from ${result.checked} row(s) with text.`, + ); + + if (result.reclaimed_bytes > 0) { + console.log( + `The freed space stays in the database file until it is vacuumed: run "bunx bun-sqlite vacuum" (or restore from a fresh backup) to reclaim it on disk. Kept text is capped at AppData.BotAttachmentTextStoreMaxChars (${AppData.BotAttachmentTextStoreMaxChars}).`, + ); + } +} + +await main(); +process.exit(0); diff --git a/src/utils/can-extract-text.test.ts b/src/utils/can-extract-text.test.ts new file mode 100644 index 0000000..d49029c --- /dev/null +++ b/src/utils/can-extract-text.test.ts @@ -0,0 +1,26 @@ +import { describe, expect, test } from "bun:test"; +import canExtractText from "./can-extract-text"; + +describe("canExtractText", () => { + test("accepts the types the extractor really unpacks", () => { + for (const name of ["a.pdf", "b.docx", "c.txt", "d.md", "e.csv", "f.tsv", "g.json", "h.html", "i.xml", "j.yml", "k.yaml", "l.log", "m.markdown"]) { + expect(canExtractText({ name })).toBe(true); + } + }); + + test("refuses a binary that only has bytes, not text", () => { + for (const name of ["icon.svg", "bundle.zip", "sheet.xlsx", "slides.pptx", "app.bin", "noext", "archive.tar.gz"]) { + expect(canExtractText({ name })).toBe(false); + } + }); + + test("reads the type when the name has no extension", () => { + expect(canExtractText({ name: "noext", mime_type: "application/pdf" })).toBe(true); + expect(canExtractText({ name: "noext", mime_type: "image/svg+xml" })).toBe(false); + }); + + test("an octet-stream says nothing, so the name decides", () => { + expect(canExtractText({ name: "notes.md", mime_type: "application/octet-stream" })).toBe(true); + expect(canExtractText({ name: "logo.svg", mime_type: "application/octet-stream" })).toBe(false); + }); +}); diff --git a/src/utils/can-extract-text.ts b/src/utils/can-extract-text.ts new file mode 100644 index 0000000..a28a20d --- /dev/null +++ b/src/utils/can-extract-text.ts @@ -0,0 +1,30 @@ +import { BotTextFileExtensions } from "@/src/dict/bot-file-dict"; +import extensionOf from "./extension-of"; + +type Params = { + /** + * The file's name, e.g. `invoice.pdf`. + */ + name?: string | null; + /** + * Its MIME type, for a file with no extension in its name. + */ + mime_type?: string | null; +}; + +/** + * Whether a file's text is worth extracting and keeping on its `media` row. + * + * Only the extensions the local extractor really unpacks count + * (`BotTextFileExtensions`). Everything else - a zip, an SVG, a + * spreadsheet, a binary with an unknown name - has no text to store, and + * decoding its bytes anyway is how raw file content ended up in the + * database. + */ +export default function canExtractText({ name, mime_type }: Params): boolean { + const ext = extensionOf({ name, mime_type }); + + if (!ext) return false; + + return (BotTextFileExtensions as readonly string[]).includes(ext); +} diff --git a/src/utils/cap-text-content.test.ts b/src/utils/cap-text-content.test.ts new file mode 100644 index 0000000..942bfc9 --- /dev/null +++ b/src/utils/cap-text-content.test.ts @@ -0,0 +1,24 @@ +import { describe, expect, test } from "bun:test"; +import { AppData } from "@/src/data/app-data"; +import capTextContent from "./cap-text-content"; + +describe("capTextContent", () => { + test("keeps text under the ceiling as it is", () => { + expect(capTextContent({ text: "short" })).toBe("short"); + }); + + test("keeps exactly the ceiling", () => { + const text = "a".repeat(AppData.BotAttachmentTextStoreMaxChars); + expect(capTextContent({ text })).toBe(text); + }); + + test("cuts it and says so", () => { + const text = "a".repeat(AppData.BotAttachmentTextStoreMaxChars + 500); + const capped = capTextContent({ text }); + + expect(capped.length).toBeLessThanOrEqual(AppData.BotAttachmentTextStoreMaxChars); + expect(capped.startsWith("a")).toBe(true); + expect(capped).toContain("Truncated"); + expect(capped).toContain(String(text.length)); + }); +}); diff --git a/src/utils/cap-text-content.ts b/src/utils/cap-text-content.ts new file mode 100644 index 0000000..67a6f5a --- /dev/null +++ b/src/utils/cap-text-content.ts @@ -0,0 +1,31 @@ +import { AppData } from "@/src/data/app-data"; + +type Params = { + /** + * The text extracted from a file. + */ + text: string; +}; + +/** + * Extracted document text, cut to what the `media` row keeps + * (`AppData.BotAttachmentTextStoreMaxChars`) with a line saying so. + * + * The column is a cache, not a copy of the file: a 20 MB document would + * otherwise put 20 MB of text into the database, and no reader takes more + * than a slice of it anyway. + */ +export default function capTextContent({ text }: Params) { + const max = AppData.BotAttachmentTextStoreMaxChars; + + if (text.length <= max) return text; + + const note = `\n\n[Truncated: this document is ${text.length} characters long, so only its beginning was kept. Read the file itself for the rest.]`; + /** + * The note counts against the ceiling too: the column is capped, not + * the text plus whatever was appended to it. + */ + const kept = Math.max(0, max - note.length); + + return `${text.slice(0, kept)}${note}`; +} diff --git a/src/utils/decide-text-content.test.ts b/src/utils/decide-text-content.test.ts new file mode 100644 index 0000000..439eb41 --- /dev/null +++ b/src/utils/decide-text-content.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, test } from "bun:test"; +import { AppData } from "@/src/data/app-data"; +import decideTextContent from "./decide-text-content"; + +describe("decideTextContent", () => { + test("keeps the text of a document the extractor reads", () => { + expect(decideTextContent({ name: "invoice.pdf", text: "Total: 12" })).toBe("Total: 12"); + expect(decideTextContent({ name: "notes.md", text: "# Hi" })).toBe("# Hi"); + expect(decideTextContent({ name: "rows.csv", text: "a,b" })).toBe("a,b"); + }); + + test("drops the raw bytes of anything else", () => { + expect(decideTextContent({ name: "icon.svg", text: "..." })).toBeNull(); + expect(decideTextContent({ name: "bundle.zip", text: "PK\u0003\u0004binary" })).toBeNull(); + expect(decideTextContent({ name: "sheet.xlsx", text: "PK\u0003\u0004binary" })).toBeNull(); + expect(decideTextContent({ name: "app.bin", text: "\u0000\u0001\u0002" })).toBeNull(); + }); + + test("nothing to store is null, not an empty string", () => { + expect(decideTextContent({ name: "invoice.pdf", text: "" })).toBeNull(); + expect(decideTextContent({ name: "invoice.pdf", text: " \n " })).toBeNull(); + expect(decideTextContent({ name: "invoice.pdf" })).toBeNull(); + }); + + test("cuts a document over the ceiling", () => { + const text = "a".repeat(AppData.BotAttachmentTextStoreMaxChars + 10); + const decided = decideTextContent({ name: "long.txt", text }); + + expect(decided).toContain("Truncated"); + expect(decided!.length).toBeLessThanOrEqual(AppData.BotAttachmentTextStoreMaxChars); + }); + + test("a document over the ceiling is still cut when the overflow is small", () => { + const text = "a".repeat(AppData.BotAttachmentTextStoreMaxChars + 10); + const decided = decideTextContent({ name: "long.txt", text })!; + + expect(decided).toContain("Truncated"); + expect(decided.length).toBeLessThanOrEqual(AppData.BotAttachmentTextStoreMaxChars); + }); +}); diff --git a/src/utils/decide-text-content.ts b/src/utils/decide-text-content.ts new file mode 100644 index 0000000..db71daa --- /dev/null +++ b/src/utils/decide-text-content.ts @@ -0,0 +1,36 @@ +import canExtractText from "./can-extract-text"; +import capTextContent from "./cap-text-content"; + +type Params = { + /** + * The file's name, when the row kept it (`media.title`). + */ + name?: string | null; + /** + * Its MIME type (`media.mime_type`). For old rows this is often just + * the extension. + */ + mime_type?: string | null; +}; + +/** + * What a `media` row's `text_content` should be: the text itself when it + * belongs there, `null` when the row should not carry any. + * + * A row qualifies when its file is one the extractor really unpacks + * (`canExtractText`); its text is then cut to the store ceiling + * (`capTextContent`). Anything else - a zip, an SVG, a spreadsheet, a + * binary that only ever had its bytes decoded - loses its `text_content`, + * because that column is an extracted-text cache, not a copy of the file. + */ +export default function decideTextContent({ + name, + mime_type, + text, +}: Params & { text?: string | null }): string | null { + if (!text || !text.trim()) return null; + + if (!canExtractText({ name, mime_type })) return null; + + return capTextContent({ text }); +} diff --git a/src/utils/extension-of.ts b/src/utils/extension-of.ts new file mode 100644 index 0000000..2ed3f93 --- /dev/null +++ b/src/utils/extension-of.ts @@ -0,0 +1,28 @@ +type Params = { + /** + * A file name, e.g. `report.pdf`. + */ + name?: string | null; + /** + * Its MIME type, used when the name has no extension. + */ + mime_type?: string | null; +}; + +/** + * A file's extension, lower case and without the dot: from its name, else + * from its MIME type. Anything but letters and digits is stripped, so a + * hostile name can't turn into a path. + */ +export default function extensionOf({ name, mime_type }: Params) { + const from_name = name?.includes(".") ? name.split(".").pop() : undefined; + + if (from_name) return from_name.toLowerCase().replace(/[^a-z0-9]/g, ""); + + const from_mime = + mime_type && !mime_type.includes("octet-stream") + ? mime_type.split("/").pop() + : undefined; + + return from_mime ? from_mime.toLowerCase().replace(/[^a-z0-9]/g, "") : undefined; +}