From ac3ebc43d42df7964bc50b8aeb4fba7b854c715f Mon Sep 17 00:00:00 2001 From: Benjamin Toby Date: Wed, 28 Oct 2026 04:44:46 +0100 Subject: [PATCH] Keep searchable text only in knowledgebases Every bot file was chunked into bot_file_chunks and embedded into bot_file_vectors, and uploads kept their extracted text in media.text_content, which together took most of the database. Bot files are no longer indexed: bots find them by name or path with list_files and read them with read_file, and search_files, ingest_file and the files modal's re-read button are gone. Attachments keep no text; a reply reads a document from the file. Only knowledgebase documents keep text. The schema drops both tables and the ingest columns. At boot, dropStoredFileText removes what older versions stored (file jobs, upload text) and vacuums once much of the database is free. The media:prune-text script it replaces is removed. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 6 +- db/schema.ts | 108 +----- db/types/db.ts | 68 +--- package.json | 3 +- src/bots-queue/load-attachment-parts.ts | 18 +- src/cron/ingest-pending-files/index.ts | 27 +- src/data/app-data.ts | 16 +- src/data/harness-prompts.ts | 10 +- src/file-ingest/enqueue-file-ingest.ts | 37 -- src/file-ingest/index.ts | 6 +- src/file-ingest/process-ingest-job.ts | 41 +-- .../database/build-table-catalog-sql.ts | 2 +- .../backend/db/bot-files/delete-bot-file.ts | 17 +- .../db/bot-files/grab-conversation-files.ts | 2 - .../backend/db/bot-files/ingest-bot-file.ts | 329 ------------------ .../backend/db/bot-files/purge-storage.ts | 16 - .../backend/db/bot-files/record-bot-file.ts | 4 +- .../backend/db/bot-files/search-bot-files.ts | 271 --------------- .../bot-files/store-chat-attachments.test.ts | 10 +- .../db/bot-files/store-chat-attachments.ts | 9 +- .../db/bot-files/vector-search.test.ts | 132 ------- .../db/bots/grab-messages-media.test.ts | 16 +- .../backend/db/bots/grab-messages-media.ts | 9 +- .../backend/db/media/media-columns.ts | 8 +- .../db/media/plan-text-content-prune.test.ts | 65 ---- .../db/media/plan-text-content-prune.ts | 55 --- .../db/media/prune-media-text-content.test.ts | 120 ------- .../db/media/prune-media-text-content.ts | 75 ---- .../drop-stored-file-text-sql.test.ts | 65 ++++ .../stored-text/drop-stored-file-text-sql.ts | 17 + .../db/stored-text/drop-stored-file-text.ts | 46 +++ .../db/stored-text/should-vacuum.test.ts | 24 ++ .../backend/db/stored-text/should-vacuum.ts | 22 ++ .../embeddings/fit-vector-dimensions.ts | 2 +- .../tools/computer/import-file-tool.ts | 9 +- .../tools/storage/generate-image-tool.ts | 3 - .../tools/storage/grab-storage-tools.ts | 4 - src/harness/tools/storage/ingest-file-tool.ts | 44 --- src/harness/tools/storage/list-files-tool.ts | 3 +- src/harness/tools/storage/make-pdf-tool.ts | 3 - .../tools/storage/search-files-tool.ts | 49 --- src/harness/tools/storage/write-file-tool.ts | 7 +- .../[bot_id]/(partials)/bot-files-modal.tsx | 19 +- .../bots/[bot_id]/(partials)/file-row.tsx | 27 +- .../files/(partials)/files-search-input.tsx | 2 +- .../(partials)/database-table-row.test.tsx | 6 +- src/pages/api/admin/bots/files.ts | 4 - src/pages/api/admin/bots/ingest-file.ts | 82 ----- src/pages/api/admin/bots/upload-attachment.ts | 6 +- src/pages/api/admin/tasks/attach-file.ts | 2 - src/pages/api/admin/upload-media.ts | 2 - src/schemas/bot-file-schema.ts | 8 - src/scripts/prune-media-text-content.ts | 40 --- src/server.ts | 4 + src/types/index.ts | 3 - 55 files changed, 271 insertions(+), 1712 deletions(-) delete mode 100644 src/file-ingest/enqueue-file-ingest.ts delete mode 100644 src/functions/backend/db/bot-files/ingest-bot-file.ts delete mode 100644 src/functions/backend/db/bot-files/search-bot-files.ts delete mode 100644 src/functions/backend/db/bot-files/vector-search.test.ts delete mode 100644 src/functions/backend/db/media/plan-text-content-prune.test.ts delete mode 100644 src/functions/backend/db/media/plan-text-content-prune.ts delete mode 100644 src/functions/backend/db/media/prune-media-text-content.test.ts delete mode 100644 src/functions/backend/db/media/prune-media-text-content.ts create mode 100644 src/functions/backend/db/stored-text/drop-stored-file-text-sql.test.ts create mode 100644 src/functions/backend/db/stored-text/drop-stored-file-text-sql.ts create mode 100644 src/functions/backend/db/stored-text/drop-stored-file-text.ts create mode 100644 src/functions/backend/db/stored-text/should-vacuum.test.ts create mode 100644 src/functions/backend/db/stored-text/should-vacuum.ts delete mode 100644 src/harness/tools/storage/ingest-file-tool.ts delete mode 100644 src/harness/tools/storage/search-files-tool.ts delete mode 100644 src/pages/api/admin/bots/ingest-file.ts delete mode 100644 src/scripts/prune-media-text-content.ts diff --git a/AGENTS.md b/AGENTS.md index 8236adc..833adb8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -97,7 +97,7 @@ Enables SQLite foreign keys (`enableForeignKeys`), `bunextInit()`, `ensureDocker - **Rendering:** markdown via `react-markdown` + `remark-gfm`, raw HTML never rendered, component map in `[bot_id]/(data)/message-markdown-components.tsx`. Inline refs, all linked by `link-computer-refs.ts` and rendered by chips in `(partials)/`: `[bot:ID]`, `[workspace:ID]`, `[computer:ID]`, `[file:ID]` (a saved file), `[folder:::]` (opens `/admin/files`), `[take-over:ID:mode]`, `[proposal:ID]`, `[hook-proposal:ID]`, `[secret:ID]` (masked env, Reveal/Copy), `[secret-request:ID]` (masked input card), `[graphic:ID]` (a bot graphic, framed inline), `[room:ID]` (a chatroom). Parsers in `src/utils/*-ref.ts`. A ref quoted alone in a code span still links; one inside longer code stays literal (`linkComputerRefs` splits out code first). A file chip opens a file from another bot's folder for a user who reaches that folder (`userReachesFileScope`, the same rule as its bytes), so answers passed on from chatrooms keep working. - **Copy chat** (header menu): the active session (the one on screen) as one markdown document (`api/admin/bots/conversation-export` -> `conversationMarkdown`, refs as names, message headings demoted). The clipboard write starts on the click with a pending `ClipboardItem` (`copyPendingText`), since Safari drops the gesture after a fetch. - **Replies/reactions:** `reply_to_message_id` (not an FK); reactions from the fixed `BotMessageReactions`, toggled by `react-message.ts` and the `react_to_message` tool, pushed as `server:bot-reaction`. -- **Attachments:** uploaded on pick (`upload-attachment.ts`, private `media`, text extracted when possible), linked via `bot_message_media`. Replies read files of the last `BotReplyAttachmentMessages` messages. Tasks don't get files yet. +- **Attachments:** uploaded on pick (`upload-attachment.ts`, private `media`, no text kept: a reply reads a document's text from the file), linked via `bot_message_media`. Replies read files of the last `BotReplyAttachmentMessages` messages. Tasks don't get files yet. - **Thinking row / stop:** `bot-busy-state.ts` decides the row's line and whether Stop shows. Stop (`api/admin/bots/stop.ts`, `stop-bot-jobs`) fails every open job of that bot for the user, aborts the running handler (`job-abort-registry`) and drops late face updates. - **Voice:** mic -> `api/admin/bots/transcribe.ts`; play -> `GET speak-message` streamed into `