Keep searchable text only in knowledgebases
Every bot file was chunked into bot_file_chunks and embedded into bot_file_vectors, and uploads kept their extracted text in media.text_content, which together took most of the database. Bot files are no longer indexed: bots find them by name or path with list_files and read them with read_file, and search_files, ingest_file and the files modal's re-read button are gone. Attachments keep no text; a reply reads a document from the file. Only knowledgebase documents keep text. The schema drops both tables and the ingest columns. At boot, dropStoredFileText removes what older versions stored (file jobs, upload text) and vacuums once much of the database is free. The media:prune-text script it replaces is removed. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Claude Opus 5.5
@@ -97,7 +97,7 @@ Enables SQLite foreign keys (`enableForeignKeys`), `bunextInit()`, `ensureDocker
|
||||
- **Rendering:** markdown via `react-markdown` + `remark-gfm`, raw HTML never rendered, component map in `[bot_id]/(data)/message-markdown-components.tsx`. Inline refs, all linked by `link-computer-refs.ts` and rendered by chips in `(partials)/`: `[bot:ID]`, `[workspace:ID]`, `[computer:ID]`, `[file:ID]` (a saved file), `[folder:<volume>:<owner>:<path>]` (opens `/admin/files`), `[take-over:ID:mode]`, `[proposal:ID]`, `[hook-proposal:ID]`, `[secret:ID]` (masked env, Reveal/Copy), `[secret-request:ID]` (masked input card), `[graphic:ID]` (a bot graphic, framed inline), `[room:ID]` (a chatroom). Parsers in `src/utils/*-ref.ts`. A ref quoted alone in a code span still links; one inside longer code stays literal (`linkComputerRefs` splits out code first). A file chip opens a file from another bot's folder for a user who reaches that folder (`userReachesFileScope`, the same rule as its bytes), so answers passed on from chatrooms keep working.
|
||||
- **Copy chat** (header menu): the active session (the one on screen) as one markdown document (`api/admin/bots/conversation-export` -> `conversationMarkdown`, refs as names, message headings demoted). The clipboard write starts on the click with a pending `ClipboardItem` (`copyPendingText`), since Safari drops the gesture after a fetch.
|
||||
- **Replies/reactions:** `reply_to_message_id` (not an FK); reactions from the fixed `BotMessageReactions`, toggled by `react-message.ts` and the `react_to_message` tool, pushed as `server:bot-reaction`.
|
||||
- **Attachments:** uploaded on pick (`upload-attachment.ts`, private `media`, text extracted when possible), linked via `bot_message_media`. Replies read files of the last `BotReplyAttachmentMessages` messages. Tasks don't get files yet.
|
||||
- **Attachments:** uploaded on pick (`upload-attachment.ts`, private `media`, no text kept: a reply reads a document's text from the file), linked via `bot_message_media`. Replies read files of the last `BotReplyAttachmentMessages` messages. Tasks don't get files yet.
|
||||
- **Thinking row / stop:** `bot-busy-state.ts` decides the row's line and whether Stop shows. Stop (`api/admin/bots/stop.ts`, `stop-bot-jobs`) fails every open job of that bot for the user, aborts the running handler (`job-abort-registry`) and drops late face updates.
|
||||
- **Voice:** mic -> `api/admin/bots/transcribe.ts`; play -> `GET speak-message` streamed into `<audio>`. Nothing recorded is stored. Both disabled with a note when the harness has no model.
|
||||
- **Pictures:** `image_media_id` (private media, `bot-image`/`workspace-image`), via `set-profile-image.ts`, the `set_picture` tool or `update_workspace`. Bytes are **copied**, never moved; only picture-paradigm rows are deleted on replace. Format kept by `imageOutputFormat`.
|
||||
@@ -177,7 +177,7 @@ Set in Settings > Environment (super admin), not `.env`. `envs` (private; encryp
|
||||
- Files live on the data volume, not on computers: `DATA_DIR/bots/<id>` and `DATA_DIR/workspaces/<id>`. Catalog `bot_files`, bytes in private `media` (`bot-file`) written by `recordBotFile`. Reach via `grabStorageScopes`; paths via `resolveBotPath` (no absolute, `..`, NUL, symlink escape); quota checked by `assertStorageRoom`.
|
||||
- Tools in `src/harness/tools/storage/` (every run). `attach` puts a file on the reply. `grabConversationFiles` must list the **whole** folder tree, or nested `[file:ID]` refs render dead. A workspace run saves to the bot's own folder unless it picks the workspace's, so a chip in a workspace conversation looks up both (`api/admin/bots/files?file_id=` with `bot_id` and `workspace_id`).
|
||||
- `import_file` copies from a computer through the sandbox's streaming `GET /v1/file/download`, **never through its shell** (`/v1/shell/exec` caps output at ~30K chars and silently truncated imports). Size read first, `change_policy=abort`, length checked.
|
||||
- Ingestion: chunks + embeddings into `bot_file_chunks` / `bot_file_vectors` (vec0), worked by `src/file-ingest/`; without an embedding model search falls back to words and says so. sqlite-vec limits: metadata columns are NOT NULL; side-car columns can't be filtered with `MATCH` (filter in code). **Never pass `--vector` to `bunx bun-sqlite schema` in `entrypoint.sh`** (wipes vectors on boot). Changing `AppData.EmbeddingDimensions` needs a manual `--vector` run and a re-ingest.
|
||||
- **Only knowledgebases store text.** Bot files and uploads are never indexed and keep no extracted text (`media.text_content` only for knowledgebase documents); bots find files by name or path (`list_files` `search`) and read them with `read_file`, and the Files searches match names only. Text that should be searchable goes in a knowledgebase. `dropStoredFileText` clears what older versions stored, at boot, and vacuums when much of the file is free.
|
||||
- **Foreign keys are off per connection** unless `enableForeignKeys` ran (only `src/server.ts` does). Scripts never fire `ON DELETE CASCADE`, so **any helper that deletes a parent row must delete its children explicitly** (as `deleteBotFile`, `purgeStorage`, `deleteSkill` do). A leftover child fails the next `bun run db:schema` with a foreign key violation.
|
||||
|
||||
### Graphics (`bot_graphics`)
|
||||
@@ -188,6 +188,8 @@ A bot shows a concept as a visual with `create_graphic` (plus `update_graphic`/`
|
||||
|
||||
A fourth extension kind (`ExtensionTables.knowledgebase`): `knowledgebases` given per bot (`bot_knowledgebases`), per workspace (`workspace_knowledgebases`) or every bot (`all_bots`), through the same add/remove/access/default paths as skills (coordinator extension tools leave them out). Documents (`knowledgebase_documents`: an upload in private `media`, paradigm `knowledgebase-document`, or written `content`) are chunked into `knowledgebase_chunks` and embedded into the `knowledgebase_vectors` vec0 table by the file-ingest worker (`file_ingest_queue.document_id`, `ingestKnowledgebaseDocument`), always with the install's embedding model; old passages stay until the new ones are in, `embedding_model` records where vectors came from; whatever can change the install's model (default settings, a provider saved or deleted, the hourly catalog sync) calls `reindexAfterEmbeddingChange`, and Re-index reads a knowledgebase again by hand. Word search runs on `knowledgebase_chunks_fts`, an FTS5 table (`unicode61 remove_diacritics 2`) made by `ensureKnowledgebaseFts` (FTS5 can't be declared in the schema; sync leaves undeclared tables alone), backfilled at boot by `syncKnowledgebaseFts`, ranked by BM25 with a scored `LIKE` fallback for unspaced scripts (`knowledgebaseWordQuery`); every write and delete of `knowledgebase_chunks` updates it. Ingest jobs (`settleFailedIngestJob`) wait without spending an attempt while every embedding model rests, fail at once on an error retrying can't fix, and a document whose model refused keeps its new passages for word search (counted as needing re-index, not failed). Runs get `search_knowledgebase`/`read_knowledgebase_document` (`openBotKnowledgebases`, limited to the run's ones) and replies start with passages found for the unanswered messages (`grabKnowledgebaseContext`); passages are fenced (`knowledgebasePassageBlock`) and `KnowledgebasesNote` says they're never instructions. UI: Settings > Knowledgebases (super admin edits and fills; admins set access), a Knowledgebases group on bot/workspace settings, and the composer's own button beside Tools (`ComposerToolsModal` with `composerKnowledgebasesCopy`). Deleting one deletes its documents' vectors, chunks, media and links explicitly.
|
||||
|
||||
sqlite-vec limits: metadata columns are NOT NULL; side-car columns can't be filtered with `MATCH` (filter in code). **Never pass `--vector` to `bunx bun-sqlite schema` in `entrypoint.sh`** (wipes vectors on boot). Changing `AppData.EmbeddingDimensions` needs a manual `--vector` run and a re-index.
|
||||
|
||||
### Files browser (`/admin/files`)
|
||||
|
||||
Read-only browser of every volume (`src/utils/storage-volumes.ts`: bots, workspaces, computers, media, mcp, db). `resolveStorageSpec` is the only access gate (members see only their bots' and their own user folders inside a computer, `computerPathAllowed`); `resolveStoragePath` applies the path rules. Links built only by `storageSpecHref`. Deletion stays in the chat's files modal and bot tools.
|
||||
|
||||
+2
-106
@@ -3792,16 +3792,6 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
|
||||
options: BotFileSources.map((s) => s.value),
|
||||
defaultValue: "bot-write",
|
||||
},
|
||||
{
|
||||
fieldName: "ingested_at",
|
||||
fieldDescription: `When its text was chunked and embedded. Null: not ingested (yet)`,
|
||||
dataType: "INTEGER",
|
||||
},
|
||||
{
|
||||
fieldName: "ingest_error",
|
||||
fieldDescription: `Why the last ingestion failed, when it did`,
|
||||
dataType: "TEXT",
|
||||
},
|
||||
],
|
||||
uniqueConstraints: [
|
||||
{
|
||||
@@ -3825,56 +3815,6 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
tableName: "bot_file_chunks",
|
||||
tableDescription: `A piece of a bot file's text, the unit a search returns. Replaced whole when the file is ingested again`,
|
||||
fields: [
|
||||
{
|
||||
fieldName: "file_id",
|
||||
dataType: "INTEGER",
|
||||
foreignKey: {
|
||||
cascadeDelete: true,
|
||||
destinationTableName: "bot_files",
|
||||
destinationTableColumnName: "id",
|
||||
destinationTableColumnType: "INTEGER",
|
||||
foreignKeyName: "bot_file_chunks_file_fk",
|
||||
},
|
||||
},
|
||||
{
|
||||
fieldName: "scope",
|
||||
dataType: "TEXT",
|
||||
options: BotFileScopes.map((s) => s.value),
|
||||
},
|
||||
{
|
||||
fieldName: "owner_id",
|
||||
fieldDescription: `The bot or workspace the file belongs to, so a search filters without a join`,
|
||||
dataType: "INTEGER",
|
||||
},
|
||||
{
|
||||
fieldName: "chunk_index",
|
||||
dataType: "INTEGER",
|
||||
},
|
||||
{
|
||||
fieldName: "text",
|
||||
dataType: "TEXT",
|
||||
},
|
||||
{
|
||||
fieldName: "token_estimate",
|
||||
fieldDescription: `A rough count (characters / 4), for trimming results`,
|
||||
dataType: "INTEGER",
|
||||
},
|
||||
],
|
||||
indexes: [
|
||||
{
|
||||
indexName: "idx_bot_file_chunks_file_id",
|
||||
indexTableFields: ["file_id", "chunk_index"],
|
||||
},
|
||||
{
|
||||
indexName: "idx_bot_file_chunks_owner",
|
||||
indexTableFields: ["scope", "owner_id"],
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
tableName: "token_usages",
|
||||
tableDescription: `One model call that reported its usage: the tokens it spent, on which model and provider, for whom and for what. A bot message's total is the sum of the calls it took; the usage page sums them by model, provider and period`,
|
||||
@@ -4023,44 +3963,9 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
tableName: "bot_file_vectors",
|
||||
tableDescription: `sqlite-vec vectors of a file's chunks, at \`AppData.EmbeddingDimensions\` wide whatever the model returns. A vec0 table is only recreated with \`bunx bun-sqlite schema --vector\`, so the entrypoint never passes it`,
|
||||
isVector: true,
|
||||
vectorType: "vec0",
|
||||
fields: [
|
||||
{
|
||||
fieldName: "embedding",
|
||||
fieldDescription: `Normalized, padded or cut to \`AppData.EmbeddingDimensions\``,
|
||||
dataType: "BLOB",
|
||||
isVector: true,
|
||||
vectorSize: AppData.EmbeddingDimensions,
|
||||
},
|
||||
{
|
||||
fieldName: "chunk_id",
|
||||
dataType: "INTEGER",
|
||||
sideCar: true,
|
||||
},
|
||||
{
|
||||
fieldName: "file_id",
|
||||
dataType: "INTEGER",
|
||||
sideCar: true,
|
||||
},
|
||||
{
|
||||
fieldName: "scope",
|
||||
dataType: "TEXT",
|
||||
sideCar: true,
|
||||
},
|
||||
{
|
||||
fieldName: "owner_id",
|
||||
dataType: "INTEGER",
|
||||
sideCar: true,
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
tableName: "file_ingest_queue",
|
||||
tableDescription: `Files (and knowledgebase documents) waiting to have their text chunked and embedded. A queue, so a restart never strands a file and an upload never waits on a model. A finished job is deleted: \`bot_files.ingested_at\` and \`ingest_error\` keep the outcome`,
|
||||
tableDescription: `Knowledgebase documents waiting to have their text chunked and embedded (bot files are never indexed). A queue, so a restart never strands a document and an upload never waits on a model. A finished job is deleted: \`knowledgebase_documents.ingested_at\` and \`ingest_error\` keep the outcome`,
|
||||
isQueue: {
|
||||
maxAttempts: 3,
|
||||
lockTimeout: 10 * 60 * 1000,
|
||||
@@ -4068,22 +3973,13 @@ const schema: BUN_SQLITE_DatabaseSchemaType = {
|
||||
deleteOnComplete: true,
|
||||
},
|
||||
fields: [
|
||||
{
|
||||
fieldName: "file_id",
|
||||
fieldDescription: `The \`bot_files\` row to (re)ingest`,
|
||||
dataType: "INTEGER",
|
||||
},
|
||||
{
|
||||
fieldName: "document_id",
|
||||
fieldDescription: `The \`knowledgebase_documents\` row to (re)ingest, instead of a file`,
|
||||
fieldDescription: `The \`knowledgebase_documents\` row to (re)ingest`,
|
||||
dataType: "INTEGER",
|
||||
},
|
||||
],
|
||||
indexes: [
|
||||
{
|
||||
indexName: "idx_file_ingest_queue_file_id",
|
||||
indexTableFields: ["file_id"],
|
||||
},
|
||||
{
|
||||
indexName: "idx_file_ingest_queue_document_id",
|
||||
indexTableFields: ["document_id"],
|
||||
|
||||
+2
-66
@@ -61,9 +61,7 @@ export const BunSQLiteTables = [
|
||||
"push_subscriptions",
|
||||
"envs",
|
||||
"bot_files",
|
||||
"bot_file_chunks",
|
||||
"token_usages",
|
||||
"bot_file_vectors",
|
||||
"file_ingest_queue",
|
||||
"app_settings",
|
||||
] as const
|
||||
@@ -2335,41 +2333,6 @@ export type BUN_SQLITE_COBBLE_BOT_FILES = {
|
||||
* How it got here: the user attached it to a message, the bot wrote it, or a tool made it
|
||||
*/
|
||||
source?: "chat-upload" | "bot-write" | "tool" | "";
|
||||
/**
|
||||
* When its text was chunked and embedded. Null: not ingested (yet)
|
||||
*/
|
||||
ingested_at?: number | "";
|
||||
/**
|
||||
* Why the last ingestion failed, when it did
|
||||
*/
|
||||
ingest_error?: string;
|
||||
}
|
||||
|
||||
export type BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS = {
|
||||
/**
|
||||
* The unique identifier of the record.
|
||||
*/
|
||||
id?: number | "";
|
||||
/**
|
||||
* The time when the record was created. (Unix Timestamp)
|
||||
*/
|
||||
created_at?: number | "";
|
||||
/**
|
||||
* The time when the record was updated. (Unix Timestamp)
|
||||
*/
|
||||
updated_at?: number | "";
|
||||
file_id?: number | "";
|
||||
scope?: "bot" | "workspace" | "";
|
||||
/**
|
||||
* The bot or workspace the file belongs to, so a search filters without a join
|
||||
*/
|
||||
owner_id?: number | "";
|
||||
chunk_index?: number | "";
|
||||
text?: string;
|
||||
/**
|
||||
* A rough count (characters / 4), for trimming results
|
||||
*/
|
||||
token_estimate?: number | "";
|
||||
}
|
||||
|
||||
export type BUN_SQLITE_COBBLE_TOKEN_USAGES = {
|
||||
@@ -2447,29 +2410,6 @@ export type BUN_SQLITE_COBBLE_TOKEN_USAGES = {
|
||||
settings?: string;
|
||||
}
|
||||
|
||||
export type BUN_SQLITE_COBBLE_BOT_FILE_VECTORS = {
|
||||
/**
|
||||
* The unique identifier of the record.
|
||||
*/
|
||||
id?: number | "";
|
||||
/**
|
||||
* The time when the record was created. (Unix Timestamp)
|
||||
*/
|
||||
created_at?: number | "";
|
||||
/**
|
||||
* The time when the record was updated. (Unix Timestamp)
|
||||
*/
|
||||
updated_at?: number | "";
|
||||
/**
|
||||
* Normalized, padded or cut to `AppData.EmbeddingDimensions`
|
||||
*/
|
||||
embedding?: Float32Array<ArrayBuffer> | Buffer<ArrayBuffer> | null;
|
||||
chunk_id?: number | "";
|
||||
file_id?: number | "";
|
||||
scope?: string;
|
||||
owner_id?: number | "";
|
||||
}
|
||||
|
||||
export type BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE = {
|
||||
/**
|
||||
* The unique identifier of the record.
|
||||
@@ -2484,11 +2424,7 @@ export type BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE = {
|
||||
*/
|
||||
updated_at?: number | "";
|
||||
/**
|
||||
* The `bot_files` row to (re)ingest
|
||||
*/
|
||||
file_id?: number | "";
|
||||
/**
|
||||
* The `knowledgebase_documents` row to (re)ingest, instead of a file
|
||||
* The `knowledgebase_documents` row to (re)ingest
|
||||
*/
|
||||
document_id?: number | "";
|
||||
/**
|
||||
@@ -2534,4 +2470,4 @@ export type BUN_SQLITE_COBBLE_APP_SETTINGS = {
|
||||
value?: string;
|
||||
}
|
||||
|
||||
export type BUN_SQLITE_COBBLE_ALL_TYPEDEFS = BUN_SQLITE_COBBLE_USERS & BUN_SQLITE_COBBLE_USER_TYPES & BUN_SQLITE_COBBLE_SSO_LOGIN_CODES & BUN_SQLITE_COBBLE_USER_PASSWORDS & BUN_SQLITE_COBBLE_MEDIA & BUN_SQLITE_COBBLE_MEDIA_PARADIGMS & BUN_SQLITE_COBBLE_BOTS & BUN_SQLITE_COBBLE_CHATROOMS & BUN_SQLITE_COBBLE_BOT_SESSIONS & BUN_SQLITE_COBBLE_BOT_MESSAGES & BUN_SQLITE_COBBLE_BOT_MESSAGE_REACTIONS & BUN_SQLITE_COBBLE_BOT_TASKS & BUN_SQLITE_COBBLE_BOT_TASK_STEPS & BUN_SQLITE_COBBLE_TASKS & BUN_SQLITE_COBBLE_TASK_MEDIA & BUN_SQLITE_COBBLE_TASK_NOTES & BUN_SQLITE_COBBLE_BOTS_QUEUE & BUN_SQLITE_COBBLE_BOTS_QUEUE_UPDATES & BUN_SQLITE_COBBLE_CRON_JOBS & BUN_SQLITE_COBBLE_BOT_HOOKS & BUN_SQLITE_COBBLE_BOT_HOOK_EVENTS & BUN_SQLITE_COBBLE_HOOK_PROPOSALS & BUN_SQLITE_COBBLE_SECRET_REQUESTS & BUN_SQLITE_COBBLE_BOT_GRAPHICS & BUN_SQLITE_COBBLE_WORKSPACES & BUN_SQLITE_COBBLE_WORKSPACE_BOTS & BUN_SQLITE_COBBLE_BOT_TYPES & BUN_SQLITE_COBBLE_WORKSPACE_USERS & BUN_SQLITE_COBBLE_BOT_USERS & BUN_SQLITE_COBBLE_BOT_USER_MEMORIES & BUN_SQLITE_COBBLE_BOT_USER_FACES & BUN_SQLITE_COBBLE_BOT_MESSAGE_MEDIA & BUN_SQLITE_COBBLE_CONTAINERS & BUN_SQLITE_COBBLE_COMPUTERS & BUN_SQLITE_COBBLE_BOT_COMPUTERS & BUN_SQLITE_COBBLE_WORKSPACE_COMPUTERS & BUN_SQLITE_COBBLE_HARNESS_PROVIDERS & BUN_SQLITE_COBBLE_HARNESS_MODELS & BUN_SQLITE_COBBLE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_BOT_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_WORKSPACE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_MCP_SERVERS & BUN_SQLITE_COBBLE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_SKILLS & BUN_SQLITE_COBBLE_EXTENSION_PROPOSALS & BUN_SQLITE_COBBLE_BOT_MCP_SERVERS & BUN_SQLITE_COBBLE_WORKSPACE_MCP_SERVERS & BUN_SQLITE_COBBLE_BOT_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_WORKSPACE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_BOT_SKILLS & BUN_SQLITE_COBBLE_WORKSPACE_SKILLS & BUN_SQLITE_COBBLE_KNOWLEDGEBASES & BUN_SQLITE_COBBLE_KNOWLEDGEBASE_DOCUMENTS & BUN_SQLITE_COBBLE_KNOWLEDGEBASE_CHUNKS & BUN_SQLITE_COBBLE_KNOWLEDGEBASE_VECTORS & BUN_SQLITE_COBBLE_BOT_KNOWLEDGEBASES & BUN_SQLITE_COBBLE_WORKSPACE_KNOWLEDGEBASES & BUN_SQLITE_COBBLE_LOGS & BUN_SQLITE_COBBLE_USER_PRESENCE & BUN_SQLITE_COBBLE_PUSH_SUBSCRIPTIONS & BUN_SQLITE_COBBLE_ENVS & BUN_SQLITE_COBBLE_BOT_FILES & BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS & BUN_SQLITE_COBBLE_TOKEN_USAGES & BUN_SQLITE_COBBLE_BOT_FILE_VECTORS & BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE & BUN_SQLITE_COBBLE_APP_SETTINGS
|
||||
export type BUN_SQLITE_COBBLE_ALL_TYPEDEFS = BUN_SQLITE_COBBLE_USERS & BUN_SQLITE_COBBLE_USER_TYPES & BUN_SQLITE_COBBLE_SSO_LOGIN_CODES & BUN_SQLITE_COBBLE_USER_PASSWORDS & BUN_SQLITE_COBBLE_MEDIA & BUN_SQLITE_COBBLE_MEDIA_PARADIGMS & BUN_SQLITE_COBBLE_BOTS & BUN_SQLITE_COBBLE_CHATROOMS & BUN_SQLITE_COBBLE_BOT_SESSIONS & BUN_SQLITE_COBBLE_BOT_MESSAGES & BUN_SQLITE_COBBLE_BOT_MESSAGE_REACTIONS & BUN_SQLITE_COBBLE_BOT_TASKS & BUN_SQLITE_COBBLE_BOT_TASK_STEPS & BUN_SQLITE_COBBLE_TASKS & BUN_SQLITE_COBBLE_TASK_MEDIA & BUN_SQLITE_COBBLE_TASK_NOTES & BUN_SQLITE_COBBLE_BOTS_QUEUE & BUN_SQLITE_COBBLE_BOTS_QUEUE_UPDATES & BUN_SQLITE_COBBLE_CRON_JOBS & BUN_SQLITE_COBBLE_BOT_HOOKS & BUN_SQLITE_COBBLE_BOT_HOOK_EVENTS & BUN_SQLITE_COBBLE_HOOK_PROPOSALS & BUN_SQLITE_COBBLE_SECRET_REQUESTS & BUN_SQLITE_COBBLE_BOT_GRAPHICS & BUN_SQLITE_COBBLE_WORKSPACES & BUN_SQLITE_COBBLE_WORKSPACE_BOTS & BUN_SQLITE_COBBLE_BOT_TYPES & BUN_SQLITE_COBBLE_WORKSPACE_USERS & BUN_SQLITE_COBBLE_BOT_USERS & BUN_SQLITE_COBBLE_BOT_USER_MEMORIES & BUN_SQLITE_COBBLE_BOT_USER_FACES & BUN_SQLITE_COBBLE_BOT_MESSAGE_MEDIA & BUN_SQLITE_COBBLE_CONTAINERS & BUN_SQLITE_COBBLE_COMPUTERS & BUN_SQLITE_COBBLE_BOT_COMPUTERS & BUN_SQLITE_COBBLE_WORKSPACE_COMPUTERS & BUN_SQLITE_COBBLE_HARNESS_PROVIDERS & BUN_SQLITE_COBBLE_HARNESS_MODELS & BUN_SQLITE_COBBLE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_BOT_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_WORKSPACE_HARNESS_SETTINGS & BUN_SQLITE_COBBLE_MCP_SERVERS & BUN_SQLITE_COBBLE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_SKILLS & BUN_SQLITE_COBBLE_EXTENSION_PROPOSALS & BUN_SQLITE_COBBLE_BOT_MCP_SERVERS & BUN_SQLITE_COBBLE_WORKSPACE_MCP_SERVERS & BUN_SQLITE_COBBLE_BOT_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_WORKSPACE_CUSTOM_TOOLS & BUN_SQLITE_COBBLE_BOT_SKILLS & BUN_SQLITE_COBBLE_WORKSPACE_SKILLS & BUN_SQLITE_COBBLE_KNOWLEDGEBASES & BUN_SQLITE_COBBLE_KNOWLEDGEBASE_DOCUMENTS & BUN_SQLITE_COBBLE_KNOWLEDGEBASE_CHUNKS & BUN_SQLITE_COBBLE_KNOWLEDGEBASE_VECTORS & BUN_SQLITE_COBBLE_BOT_KNOWLEDGEBASES & BUN_SQLITE_COBBLE_WORKSPACE_KNOWLEDGEBASES & BUN_SQLITE_COBBLE_LOGS & BUN_SQLITE_COBBLE_USER_PRESENCE & BUN_SQLITE_COBBLE_PUSH_SUBSCRIPTIONS & BUN_SQLITE_COBBLE_ENVS & BUN_SQLITE_COBBLE_BOT_FILES & BUN_SQLITE_COBBLE_TOKEN_USAGES & BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE & BUN_SQLITE_COBBLE_APP_SETTINGS
|
||||
+1
-2
@@ -8,12 +8,11 @@
|
||||
"dev": "bun src/server.ts",
|
||||
"gen:favicon": "bun src/scripts/generate-brand-assets.ts",
|
||||
"build": "bunx bunext build",
|
||||
"db:schema": "bunx bun-sqlite schema -t",
|
||||
"db:schema": "bunx bun-sqlite schema -t -y",
|
||||
"db:admin": "bunx bun-sqlite admin",
|
||||
"db:backup": "bunx bun-sqlite backup",
|
||||
"seed": "bun src/scripts/seed-super-admin.tsx",
|
||||
"seed:bots": "bun src/scripts/seed-bots.ts",
|
||||
"media:prune-text": "bun src/scripts/prune-media-text-content.ts",
|
||||
"start": "MAIN_SERVER=1 NODE_ENV=production bun src/server.ts",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"test": "bun test",
|
||||
|
||||
@@ -5,6 +5,7 @@ import grabMessagesMedia, {
|
||||
} from "@/src/functions/backend/db/bots/grab-messages-media";
|
||||
import type { HarnessContentPart } from "@/src/harness/types";
|
||||
import type { BotMessageWithAttachments } from "@/src/types";
|
||||
import extractBotFileText from "@/src/functions/backend/db/bot-files/extract-bot-file-text";
|
||||
import describeAttachment from "./describe-attachment";
|
||||
|
||||
type Params = {
|
||||
@@ -27,10 +28,7 @@ export default async function loadAttachmentParts({ messages }: Params) {
|
||||
.slice(-AppData.BotReplyAttachmentMessages)
|
||||
.map((message) => Number(message.id));
|
||||
|
||||
const media = await grabMessagesMedia({
|
||||
message_ids: recent,
|
||||
with_text_content: true,
|
||||
});
|
||||
const media = await grabMessagesMedia({ message_ids: recent });
|
||||
const parts = new Map<number, HarnessContentPart[]>();
|
||||
|
||||
for (const [message_id, files] of media) {
|
||||
@@ -53,7 +51,17 @@ async function fileToParts({ file }: { file: MessageMediaRow }): Promise<Harness
|
||||
const label = describeAttachment({ name, id: file.id! });
|
||||
|
||||
if (file.media_type == "document") {
|
||||
const text = file.text_content?.trim();
|
||||
/**
|
||||
* Read from the file each time: no text is kept outside
|
||||
* knowledgebases.
|
||||
*/
|
||||
const text = (
|
||||
await extractBotFileText({
|
||||
write_path: file.media_write_path,
|
||||
name: file.title,
|
||||
mime_type: file.mime_type,
|
||||
}).catch(() => null)
|
||||
)?.trim();
|
||||
|
||||
if (!text) {
|
||||
return [{ type: "text", text: `[Attached file ${label}: its text couldn't be read.]` }];
|
||||
|
||||
@@ -1,31 +1,12 @@
|
||||
import BunSQLite from "@moduletrace/bun-sqlite";
|
||||
import type { BUN_SQLITE_COBBLE_BOT_FILES } from "@/db/types/db";
|
||||
import enqueueDocumentIngest from "@/src/file-ingest/enqueue-document-ingest";
|
||||
import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
|
||||
|
||||
/**
|
||||
* Queues any bot file or knowledgebase document that never got ingested (an upload that happened
|
||||
* while the worker was down, a restart mid-ingest), so nothing is left
|
||||
* unsearchable. Runs every ten minutes.
|
||||
* Queues any knowledgebase document that never got ingested (an upload
|
||||
* that happened while the worker was down, a restart mid-ingest), so
|
||||
* nothing is left unsearchable. Runs every ten minutes.
|
||||
*/
|
||||
export default async function ingestPendingFiles() {
|
||||
const res = await BunSQLite.sql<BUN_SQLITE_COBBLE_BOT_FILES>({
|
||||
sql: `SELECT f.id FROM bot_files f
|
||||
WHERE f.ingested_at IS NULL
|
||||
AND f.ingest_error IS NULL
|
||||
AND f.created_at > ?
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM file_ingest_queue q
|
||||
WHERE q.file_id = f.id AND q.status IN ('pending', 'processing')
|
||||
)
|
||||
LIMIT 25`,
|
||||
values: [Date.now() - 24 * 60 * 60 * 1000],
|
||||
});
|
||||
|
||||
for (const file of res.payload || []) {
|
||||
if (file.id) await enqueueFileIngest({ file_id: file.id });
|
||||
}
|
||||
|
||||
const documents = await BunSQLite.sql<{ id: number }>({
|
||||
sql: `SELECT d.id FROM knowledgebase_documents d
|
||||
WHERE d.ingested_at IS NULL
|
||||
@@ -41,5 +22,5 @@ export default async function ingestPendingFiles() {
|
||||
if (document.id) await enqueueDocumentIngest({ document_id: document.id });
|
||||
}
|
||||
|
||||
return (res.payload?.length || 0) + (documents.payload?.length || 0);
|
||||
return documents.payload?.length || 0;
|
||||
}
|
||||
|
||||
+12
-4
@@ -661,8 +661,9 @@ export const AppData = {
|
||||
*/
|
||||
BotFileImportMaxBytes: 20 * 1024 * 1024,
|
||||
/**
|
||||
* Ingesting a bot file: how much text one chunk holds, how much of the
|
||||
* previous chunk it repeats, and the largest file it reads at all.
|
||||
* Ingesting a knowledgebase document: how much text one chunk holds,
|
||||
* how much of the previous chunk it repeats, and the largest file it
|
||||
* reads at all (also the largest attachment a reply reads).
|
||||
*/
|
||||
ChunkTargetChars: 1200,
|
||||
ChunkOverlapChars: 200,
|
||||
@@ -681,12 +682,19 @@ export const AppData = {
|
||||
EmbeddingBatchSize: 32,
|
||||
EmbeddingMaxInputChars: 8000,
|
||||
/**
|
||||
* How many chunks a file search returns, and how many more it asks
|
||||
* the vector index for before filtering to the run's scopes.
|
||||
* How many chunks a knowledgebase search returns, and how many more it
|
||||
* asks the vector index for before filtering to the run's ones.
|
||||
*/
|
||||
RagTopChunks: 8,
|
||||
RagOverFetch: 4,
|
||||
RagMaxK: 60,
|
||||
/**
|
||||
* The database is vacuumed at boot once this share of its pages, and
|
||||
* at least this many bytes, sit free (after a big delete, such as the
|
||||
* bot file index being dropped).
|
||||
*/
|
||||
DbVacuumMinFreeRatio: 0.2,
|
||||
DbVacuumMinFreeBytes: 4 * 1024 * 1024,
|
||||
/**
|
||||
* Knowledgebases: the most one document holds (an upload, or text
|
||||
* written in), how much of a pasted text a form takes, how many
|
||||
|
||||
@@ -60,18 +60,18 @@ export const botFilesNote = ({ bot_id }: { bot_id: number | string }) => `# Your
|
||||
You have a folder of your own, kept with the app (not on a computer), and it stays between runs and conversations.
|
||||
|
||||
- list_files shows what's in it; read_file opens one (by its [file:ID] ref or path); write_file saves a text file, make_pdf a PDF, generate_image an image. Move and delete with move_file and delete_file when something is out of date, so the folder stays useful.
|
||||
- Text files are chunked and searchable as soon as they're saved. search_files finds the passages that answer a question, by meaning when an embedding model is set and by words otherwise. Search your files before saying you don't know something they may hold, and when the user asks about a document they gave you.
|
||||
- Uploaded files land in the same folder, under uploads/ (uploads/2026-10/logo.png), so what the user sent you earlier is searchable later too.
|
||||
- Files are found by name or path, never by what's in them: list_files with "search" matches names and paths, then read_file reads the one you want. Look for a file and read it before saying you don't know something it may hold, and when the user asks about a document they gave you. Searching by content is what knowledgebases are for; your files aren't indexed.
|
||||
- Uploaded files land in the same folder, under uploads/ (uploads/2026-10/logo.png), so what the user sent you earlier is there to list and read later too.
|
||||
- A picture can be set from a file: set_picture makes a file your own avatar, and update_workspace sets a workspace's. Both take the [file:ID] ref, so a logo the user sent you is usually ready to go: list_files for it, then set it. An SVG or another kind of file is turned into a PNG first. Its picture is free to change and never removes your file.
|
||||
- A file and a folder are two different refs, and you never guess which one from the name:
|
||||
- A file you saved is a [file:ID] ref (from write_file, list_files, search_files or import_file). The user sees a chip that opens the file.
|
||||
- A file you saved is a [file:ID] ref (from write_file, list_files or import_file). The user sees a chip that opens the file.
|
||||
- A folder is a folder ref, e.g. ${ownFolderRef({ bot_id })}: the user sees a chip that opens that folder in Files, and nothing is downloaded. Use the volume of the folder you mean ("bot" for your own, "workspace" for a workspace's shared one).
|
||||
- Never write a path as plain text and expect the user to find it; never write a [file:ID] ref for something that is a folder.
|
||||
- Write refs as they are, never in backticks or a code block: inside code they show as plain text, not a chip.
|
||||
- A file you only described is not saved. Save it first, then say it's saved.
|
||||
- Save what the user will want before the long tail of a run. A step limit ends a run where it stands: write the report first with write_file (or copy it off the computer with import_file), then carry on verifying.
|
||||
- One copy of a thing is enough. Before you write or import something, list_files (or search_files) for it: if it's already there, update it (write_file over the same path) instead of saving a second copy under a new name. Never split a file into parts unless the user asked for parts.
|
||||
- Your files don't disappear. If you can't find something, list_files or search_files before saying anything is missing, and never claim the folder is empty without having just listed it.
|
||||
- One copy of a thing is enough. Before you write or import something, list_files for it: if it's already there, update it (write_file over the same path) instead of saving a second copy under a new name. Never split a file into parts unless the user asked for parts.
|
||||
- Your files don't disappear. If you can't find something, list_files (with and without "search") before saying anything is missing, and never claim the folder is empty without having just listed it.
|
||||
- Report what the tools actually returned. Don't invent limits, sizes or caps you haven't seen in a tool result: if an import or a write failed, say what the tool said.`;
|
||||
|
||||
/**
|
||||
|
||||
@@ -1,37 +0,0 @@
|
||||
import BunSQLite from "@moduletrace/bun-sqlite";
|
||||
import type { BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE } from "@/db/types/db";
|
||||
import wakeFileIngest from "./wake-file-ingest";
|
||||
|
||||
type Params = {
|
||||
file_id: number | string;
|
||||
};
|
||||
|
||||
/**
|
||||
* Queues a file to be chunked and embedded, unless it's already waiting.
|
||||
* Never fails the caller: a file that can't be queued is picked up by the
|
||||
* `ingest-pending-files` cron instead.
|
||||
*/
|
||||
export default async function enqueueFileIngest({ file_id }: Params) {
|
||||
try {
|
||||
const open = await BunSQLite.sql<BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE>({
|
||||
sql: `SELECT id FROM file_ingest_queue WHERE file_id = ? AND status IN ('pending', 'processing') LIMIT 1`,
|
||||
values: [file_id],
|
||||
});
|
||||
|
||||
if (open.payload?.[0]?.id) {
|
||||
wakeFileIngest();
|
||||
return;
|
||||
}
|
||||
|
||||
await BunSQLite.queue.add({
|
||||
table: "file_ingest_queue",
|
||||
data: [{ file_id: Number(file_id) }],
|
||||
});
|
||||
|
||||
wakeFileIngest();
|
||||
} catch {
|
||||
/**
|
||||
* The cron sweeps it up.
|
||||
*/
|
||||
}
|
||||
}
|
||||
@@ -14,9 +14,9 @@ let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
/**
|
||||
* Starts the file ingestion worker: polls every `AppData.FileIngestPollMs`
|
||||
* for `file_ingest_queue` jobs and runs up to
|
||||
* `AppData.FileIngestConcurrency` at once (chunking and embedding a bot's
|
||||
* file). Gated in production to the `MAIN_SERVER` container, like cron and
|
||||
* the bots queue.
|
||||
* `AppData.FileIngestConcurrency` at once (chunking and embedding a
|
||||
* knowledgebase document). Gated in production to the `MAIN_SERVER`
|
||||
* container, like cron and the bots queue.
|
||||
*/
|
||||
export default async function fileIngest() {
|
||||
if (process.env.NODE_ENV == "production" && !process.env.MAIN_SERVER) {
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
import BunSQLite from "@moduletrace/bun-sqlite";
|
||||
import type { BUN_SQLITE_COBBLE_FILE_INGEST_QUEUE } from "@/db/types/db";
|
||||
import ingestBotFile from "@/src/functions/backend/db/bot-files/ingest-bot-file";
|
||||
import ingestKnowledgebaseDocument from "@/src/functions/backend/db/knowledgebases/ingest-knowledgebase-document";
|
||||
import { logError } from "@/src/functions/backend/logger";
|
||||
import settleFailedIngestJob from "./settle-failed-ingest-job";
|
||||
@@ -10,56 +9,30 @@ type Params = {
|
||||
};
|
||||
|
||||
/**
|
||||
* Ingests one queued file (or knowledgebase document) and completes (or
|
||||
* fails) its job. One that was deleted meanwhile is simply done.
|
||||
* Ingests one queued knowledgebase document and completes (or fails) its
|
||||
* job. One that was deleted meanwhile is simply done, and so is a job
|
||||
* without a document (bot files are no longer indexed).
|
||||
*/
|
||||
export default async function processIngestJob({ job }: Params) {
|
||||
if (!job.id) return;
|
||||
|
||||
if (job.document_id) return await processDocumentJob({ job_id: job.id, document_id: job.document_id });
|
||||
|
||||
if (!job.file_id) {
|
||||
if (!job.document_id) {
|
||||
await BunSQLite.queue.complete({ table: "file_ingest_queue", id: job.id }).catch(() => {});
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await ingestBotFile({ file_id: job.file_id });
|
||||
|
||||
await ingestKnowledgebaseDocument({ document_id: job.document_id });
|
||||
await BunSQLite.queue.complete({ table: "file_ingest_queue", id: job.id });
|
||||
|
||||
if (result.status == "ingested") {
|
||||
BunSQLite.sql({
|
||||
sql: `UPDATE bot_files SET ingest_error = NULL WHERE id = ?`,
|
||||
values: [job.file_id],
|
||||
}).catch(() => {});
|
||||
}
|
||||
} catch (error) {
|
||||
const { deferred } = await settleFailedIngestJob({ job_id: job.id, error });
|
||||
|
||||
if (deferred) return;
|
||||
|
||||
await logError({
|
||||
title: `Bot file #${job.file_id} couldn't be ingested`,
|
||||
title: `Knowledgebase document #${job.document_id} couldn't be ingested`,
|
||||
error,
|
||||
metadata: { file_id: job.file_id },
|
||||
}).catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
async function processDocumentJob({ job_id, document_id }: { job_id: number; document_id: number }) {
|
||||
try {
|
||||
await ingestKnowledgebaseDocument({ document_id });
|
||||
await BunSQLite.queue.complete({ table: "file_ingest_queue", id: job_id });
|
||||
} catch (error) {
|
||||
const { deferred } = await settleFailedIngestJob({ job_id, error });
|
||||
|
||||
if (deferred) return;
|
||||
|
||||
await logError({
|
||||
title: `Knowledgebase document #${document_id} couldn't be ingested`,
|
||||
error,
|
||||
metadata: { document_id },
|
||||
metadata: { document_id: job.document_id },
|
||||
}).catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
* table name as an argument the way a table-valued function does.
|
||||
*
|
||||
* Left out: SQLite's own `sqlite_*` tables, and the shadow tables a
|
||||
* virtual table keeps its internals in (the `bot_file_vectors_*` tables
|
||||
* virtual table keeps its internals in (the `knowledgebase_vectors_*` tables
|
||||
* behind the sqlite-vec index), which are storage, not rows anyone
|
||||
* browses.
|
||||
*/
|
||||
|
||||
@@ -15,7 +15,7 @@ type Params = {
|
||||
};
|
||||
|
||||
/**
|
||||
* Deletes a bot file: the catalog row (its chunks go with it), its bytes
|
||||
* Deletes a bot file: the catalog row, its bytes
|
||||
* and its `media` row. Refuses anything outside the run's directories.
|
||||
*/
|
||||
export default async function deleteBotFile({ scopes, file_id }: Params) {
|
||||
@@ -32,22 +32,7 @@ export default async function deleteBotFile({ scopes, file_id }: Params) {
|
||||
rmSync(absolute, { force: true });
|
||||
}
|
||||
|
||||
await BunSQLite.sql({ sql: `DELETE FROM bot_file_vectors WHERE file_id = ?`, values: [file.id] });
|
||||
/**
|
||||
* Deleted here rather than left to the schema's cascade: SQLite has
|
||||
* foreign keys off per connection unless `enableForeignKeys` ran, and
|
||||
* a script or CLI process that deletes a file would otherwise leave
|
||||
* its chunks behind, which then fail the next schema sync.
|
||||
*/
|
||||
await BunSQLite.sql({ sql: `DELETE FROM bot_file_chunks WHERE file_id = ?`, values: [file.id] });
|
||||
await BunSQLite.sql({ sql: `DELETE FROM bot_files WHERE id = ?`, values: [file.id] });
|
||||
/**
|
||||
* Its pending ingestion would only find nothing left.
|
||||
*/
|
||||
await BunSQLite.sql({
|
||||
sql: `DELETE FROM file_ingest_queue WHERE file_id = ? AND status IN ('pending', 'processing')`,
|
||||
values: [file.id],
|
||||
}).catch(() => {});
|
||||
|
||||
if (file.media_id) {
|
||||
const media = await BunSQLite.sql<{ media_write_path?: string; media_thumbnail_write_path?: string }>({
|
||||
|
||||
@@ -53,8 +53,6 @@ export default async function grabConversationFiles({
|
||||
purpose: String(file.purpose || "writing"),
|
||||
scope: String(file.scope || "bot"),
|
||||
owner_id: Number(file.owner_id) || 0,
|
||||
ingested_at: file.ingested_at ? Number(file.ingested_at) : undefined,
|
||||
ingest_error: file.ingest_error || undefined,
|
||||
created_at: Number(file.created_at) || 0,
|
||||
url: media_url || "",
|
||||
};
|
||||
|
||||
@@ -1,329 +0,0 @@
|
||||
import BunSQLite from "@moduletrace/bun-sqlite";
|
||||
import type {
|
||||
BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS,
|
||||
BUN_SQLITE_COBBLE_BOT_FILES,
|
||||
BunSQLiteTables,
|
||||
} from "@/db/types/db";
|
||||
import { AppData } from "@/src/data/app-data";
|
||||
import embedText from "@/src/harness/embeddings/embed-text";
|
||||
import HarnessError from "@/src/harness/harness-error";
|
||||
import { grabHarnessUsageScope } from "@/src/harness/usage/harness-usage-scope";
|
||||
import type { HarnessUsageScope } from "@/src/harness/usage/types";
|
||||
import runWithFallbackScope from "@/src/harness/usage/run-with-fallback-scope";
|
||||
import grabUsageOwnerUserId from "@/src/functions/backend/db/usage/grab-usage-owner";
|
||||
import chunkText from "./chunk-text";
|
||||
import extractBotFileText from "./extract-bot-file-text";
|
||||
|
||||
type Params = {
|
||||
file_id: number | string;
|
||||
/**
|
||||
* Read images and audio too (a bot asking for it).
|
||||
*/
|
||||
read_media?: boolean;
|
||||
signal?: AbortSignal;
|
||||
};
|
||||
|
||||
export type IngestResult = {
|
||||
/**
|
||||
* `ingested`, `empty` (no text to read), `skipped` (too large) or
|
||||
* `unavailable` (no embedding model), which keeps the file's text but
|
||||
* leaves it searchable by words only.
|
||||
*/
|
||||
status: "ingested" | "empty" | "skipped" | "unavailable";
|
||||
chunks: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Reads a bot file's text, splits it into chunks and embeds them into
|
||||
* `bot_file_vectors`, replacing whatever was stored before. The chunks
|
||||
* stay in `bot_file_chunks` even when there's no embedding model, so a
|
||||
* search can still match words in them.
|
||||
*
|
||||
* Failures are recorded on the file (`ingest_error`) and rethrown, so the
|
||||
* queue can retry.
|
||||
*/
|
||||
export default async function ingestBotFile({
|
||||
file_id,
|
||||
read_media,
|
||||
signal,
|
||||
}: Params): Promise<IngestResult> {
|
||||
const file = await BunSQLite.sql<BUN_SQLITE_COBBLE_BOT_FILES>({
|
||||
sql: `SELECT * FROM bot_files WHERE id = ? LIMIT 1`,
|
||||
values: [file_id],
|
||||
}).then((res) => res.payload?.[0]);
|
||||
|
||||
if (!file?.id) return { status: "empty", chunks: 0 };
|
||||
|
||||
const media = file.media_id
|
||||
? await BunSQLite.sql<{ media_write_path?: string; text_content?: string }>({
|
||||
sql: `SELECT media_write_path, text_content FROM media WHERE id = ? LIMIT 1`,
|
||||
values: [file.media_id],
|
||||
}).then((res) => res.payload?.[0])
|
||||
: undefined;
|
||||
|
||||
/**
|
||||
* Decided on the file's size alone. A cached `text_content` is not a
|
||||
* reason to read an oversized file: it was only ever a slice of it,
|
||||
* and taking it as the whole document made a file too big to unpack
|
||||
* look already read.
|
||||
*/
|
||||
if (Number(file.size) > AppData.IngestMaxBytes) {
|
||||
await BunSQLite.sql({
|
||||
sql: `UPDATE bot_files SET ingest_error = ?, updated_at = ? WHERE id = ?`,
|
||||
values: [`Too large to read (over ${AppData.IngestMaxBytes / (1024 * 1024)} MB).`, Date.now(), file.id],
|
||||
});
|
||||
|
||||
return { status: "skipped", chunks: 0 };
|
||||
}
|
||||
|
||||
/**
|
||||
* A bot's own run already carries its user. The ingest queue has no run
|
||||
* behind it, so everything this file costs - reading an image or an
|
||||
* audio file, embedding its chunks - is entered under the owner's scope
|
||||
* instead: it spends real tokens and must show on the usage page.
|
||||
*/
|
||||
const owner_scope: HarnessUsageScope | null = grabHarnessUsageScope()?.user_id
|
||||
? null
|
||||
: await resolveOwnerScope({ file });
|
||||
|
||||
try {
|
||||
const text = await runWithFallbackScope({
|
||||
scope: owner_scope,
|
||||
task: () =>
|
||||
extractBotFileText({
|
||||
write_path: media?.media_write_path,
|
||||
text_content: media?.text_content,
|
||||
name: file.name,
|
||||
mime_type: file.mime_type,
|
||||
read_media,
|
||||
}),
|
||||
});
|
||||
|
||||
await clearChunks({ file_id: file.id });
|
||||
|
||||
if (!text?.trim()) {
|
||||
await markIngested({ file_id: file.id, chunks: 0 });
|
||||
|
||||
return { status: "empty", chunks: 0 };
|
||||
}
|
||||
|
||||
const chunks = chunkText({ text });
|
||||
|
||||
if (!chunks.length) {
|
||||
await markIngested({ file_id: file.id, chunks: 0 });
|
||||
|
||||
return { status: "empty", chunks: 0 };
|
||||
}
|
||||
|
||||
const chunk_ids = await insertChunks({ file, chunks });
|
||||
let embedded = 0;
|
||||
|
||||
try {
|
||||
/**
|
||||
* A big file's chunks go in batches, so one provider refusal
|
||||
* doesn't throw away the whole file.
|
||||
*/
|
||||
for (let start = 0; start < chunks.length; start += AppData.EmbeddingBatchSize) {
|
||||
const batch = chunks.slice(start, start + AppData.EmbeddingBatchSize);
|
||||
const { vectors } = await runEmbed({
|
||||
scope: owner_scope,
|
||||
signal,
|
||||
file,
|
||||
batch,
|
||||
});
|
||||
|
||||
for (let i = 0; i < vectors.length; i++) {
|
||||
await insertVector({
|
||||
vector: vectors[i]!,
|
||||
chunk_id: chunk_ids[start + i]!,
|
||||
file_id: file.id,
|
||||
scope: String(file.scope),
|
||||
owner_id: Number(file.owner_id),
|
||||
});
|
||||
|
||||
embedded++;
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
if (error instanceof HarnessError && error.code == "unavailable") {
|
||||
/**
|
||||
* No embedding model: the chunks still let a search match
|
||||
* words, so the file isn't a failure.
|
||||
*/
|
||||
await markIngested({
|
||||
file_id: file.id,
|
||||
chunks: chunks.length,
|
||||
error: "No embedding model is set, so this file is searchable by words only.",
|
||||
});
|
||||
|
||||
return { status: "unavailable", chunks: chunks.length };
|
||||
}
|
||||
|
||||
throw error;
|
||||
}
|
||||
|
||||
await markIngested({ file_id: file.id, chunks: chunks.length });
|
||||
|
||||
return { status: "ingested", chunks: embedded };
|
||||
} catch (error) {
|
||||
/**
|
||||
* Every model resting means nothing was tried: the job waits for
|
||||
* them, and the file keeps whatever it said before.
|
||||
*/
|
||||
if (error instanceof HarnessError && error.resting) throw error;
|
||||
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
|
||||
await BunSQLite.sql({
|
||||
sql: `UPDATE bot_files SET ingest_error = ?, updated_at = ? WHERE id = ?`,
|
||||
values: [message.slice(0, 500), Date.now(), file.id],
|
||||
}).catch(() => {});
|
||||
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function clearChunks({ file_id }: { file_id: number }) {
|
||||
await BunSQLite.sql({ sql: `DELETE FROM bot_file_vectors WHERE file_id = ?`, values: [file_id] });
|
||||
await BunSQLite.sql({ sql: `DELETE FROM bot_file_chunks WHERE file_id = ?`, values: [file_id] });
|
||||
}
|
||||
|
||||
async function insertChunks({
|
||||
file,
|
||||
chunks,
|
||||
}: {
|
||||
file: BUN_SQLITE_COBBLE_BOT_FILES;
|
||||
chunks: string[];
|
||||
}) {
|
||||
const ids: number[] = [];
|
||||
|
||||
for (let index = 0; index < chunks.length; index++) {
|
||||
const text = chunks[index]!;
|
||||
|
||||
const res = await BunSQLite.insert<
|
||||
BUN_SQLITE_COBBLE_BOT_FILE_CHUNKS,
|
||||
(typeof BunSQLiteTables)[number]
|
||||
>({
|
||||
table: "bot_file_chunks",
|
||||
data: [
|
||||
{
|
||||
file_id: file.id,
|
||||
scope: file.scope as any,
|
||||
owner_id: Number(file.owner_id),
|
||||
chunk_index: index,
|
||||
text,
|
||||
token_estimate: Math.ceil(text.length / 4),
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const id = res.postInsertReturn?.insertId;
|
||||
|
||||
if (id) ids.push(id);
|
||||
}
|
||||
|
||||
return ids;
|
||||
}
|
||||
|
||||
async function insertVector({
|
||||
vector,
|
||||
chunk_id,
|
||||
file_id,
|
||||
scope,
|
||||
owner_id,
|
||||
}: {
|
||||
vector: Float32Array;
|
||||
chunk_id: number;
|
||||
file_id: number;
|
||||
scope: string;
|
||||
owner_id: number;
|
||||
}) {
|
||||
/**
|
||||
* vec0's metadata columns (its id and timestamps) are NOT NULL, so a
|
||||
* raw insert has to fill them; `BunSQLite.insert` would add them but
|
||||
* the vector needs `vec_f32`.
|
||||
*/
|
||||
const now = Date.now();
|
||||
|
||||
const res = await BunSQLite.sql({
|
||||
sql: `INSERT INTO bot_file_vectors (created_at, updated_at, embedding, chunk_id, file_id, scope, owner_id) VALUES (?, ?, vec_f32(?), ?, ?, ?, ?)`,
|
||||
values: [now, now, vector as any, chunk_id, file_id, scope, owner_id],
|
||||
});
|
||||
|
||||
if (!res.success) {
|
||||
throw new Error(`Couldn't store an embedding: ${res.error || res.msg}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function markIngested({
|
||||
file_id,
|
||||
chunks,
|
||||
error,
|
||||
}: {
|
||||
file_id: number;
|
||||
chunks: number;
|
||||
error?: string;
|
||||
}) {
|
||||
const now = Date.now();
|
||||
|
||||
await BunSQLite.sql({
|
||||
sql: `UPDATE bot_files SET ingested_at = ?, ingest_error = ?, updated_at = ? WHERE id = ?`,
|
||||
values: [now, error || null, now, file_id],
|
||||
});
|
||||
|
||||
await BunSQLite.sql({
|
||||
sql: `UPDATE media SET is_rag_ingested = ?, updated_at = ? WHERE id = (SELECT media_id FROM bot_files WHERE id = ?)`,
|
||||
values: [chunks > 0 ? 1 : 0, now, file_id],
|
||||
}).catch(() => {});
|
||||
}
|
||||
|
||||
/**
|
||||
* One batch of a file's chunks, embedded under whatever attribution owns
|
||||
* it: the run that asked for the ingest, else the file's owner.
|
||||
*/
|
||||
async function runEmbed({
|
||||
scope,
|
||||
signal,
|
||||
file,
|
||||
batch,
|
||||
}: {
|
||||
scope: HarnessUsageScope | null;
|
||||
signal?: AbortSignal;
|
||||
file: BUN_SQLITE_COBBLE_BOT_FILES;
|
||||
batch: string[];
|
||||
}) {
|
||||
return await runWithFallbackScope({
|
||||
scope,
|
||||
task: () =>
|
||||
embedText({
|
||||
input: batch,
|
||||
bot_id: file.bot_id!,
|
||||
workspace_id: file.scope == "workspace" ? file.owner_id : null,
|
||||
signal,
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* The attribution a file the queue ingests reports under: its workspace's
|
||||
* creator when it shares one, else its bot's.
|
||||
*/
|
||||
async function resolveOwnerScope({
|
||||
file,
|
||||
}: {
|
||||
file: BUN_SQLITE_COBBLE_BOT_FILES;
|
||||
}): Promise<HarnessUsageScope | null> {
|
||||
const workspace_id = file.scope == "workspace" ? file.owner_id : null;
|
||||
const user_id = await grabUsageOwnerUserId({ bot_id: file.bot_id, workspace_id });
|
||||
|
||||
if (!user_id) return null;
|
||||
|
||||
return {
|
||||
user_id,
|
||||
bot_id: file.scope == "bot" ? file.owner_id : file.bot_id,
|
||||
workspace_id,
|
||||
session_id: null,
|
||||
message_id: null,
|
||||
job_id: null,
|
||||
};
|
||||
}
|
||||
@@ -54,22 +54,6 @@ export default async function purgeStorage({ scope, owner_id, media_ids }: Param
|
||||
}).catch(() => {});
|
||||
}
|
||||
|
||||
await BunSQLite.sql({
|
||||
sql: `DELETE FROM bot_file_vectors WHERE scope = ? AND owner_id = ?`,
|
||||
values: [scope, owner_id],
|
||||
}).catch(() => {});
|
||||
|
||||
/**
|
||||
* Deleted here rather than left to the cascade, since SQLite has
|
||||
* foreign keys off per connection unless `enableForeignKeys` ran (a
|
||||
* script or CLI process never does), and a chunk left behind fails
|
||||
* the next schema sync.
|
||||
*/
|
||||
await BunSQLite.sql({
|
||||
sql: `DELETE FROM bot_file_chunks WHERE scope = ? AND owner_id = ?`,
|
||||
values: [scope, owner_id],
|
||||
}).catch(() => {});
|
||||
|
||||
/**
|
||||
* Usually gone already, with the bot or workspace (the cascade).
|
||||
*/
|
||||
|
||||
@@ -104,7 +104,7 @@ export default async function recordBotFile({
|
||||
if (existing?.id) {
|
||||
await BunSQLite.sql({
|
||||
sql: `UPDATE bot_files SET media_id = ?, name = ?, folder = ?, mime_type = ?, media_type = ?, size = ?,
|
||||
purpose = ?, source = ?, ingested_at = NULL, ingest_error = NULL, updated_at = ? WHERE id = ?`,
|
||||
purpose = ?, source = ?, updated_at = ? WHERE id = ?`,
|
||||
values: [
|
||||
media_id,
|
||||
resolved.name,
|
||||
@@ -129,8 +129,6 @@ export default async function recordBotFile({
|
||||
size: bytes.byteLength,
|
||||
purpose,
|
||||
source,
|
||||
ingested_at: undefined,
|
||||
ingest_error: undefined,
|
||||
updated_at: now,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1,271 +0,0 @@
|
||||
import BunSQLite from "@moduletrace/bun-sqlite";
|
||||
import { AppData } from "@/src/data/app-data";
|
||||
import { embedQuery } from "@/src/harness/embeddings/embed-text";
|
||||
import HarnessError from "@/src/harness/harness-error";
|
||||
import type { StorageScope } from "@/src/utils/storage-access";
|
||||
|
||||
type Params = {
|
||||
scopes: StorageScope[];
|
||||
query: string;
|
||||
/**
|
||||
* Only this file's chunks.
|
||||
*/
|
||||
file_id?: number | string | null;
|
||||
limit?: number;
|
||||
signal?: AbortSignal;
|
||||
/**
|
||||
* Whose embedding model to use.
|
||||
*/
|
||||
bot_id?: number | string;
|
||||
workspace_id?: number | string | null;
|
||||
};
|
||||
|
||||
export type BotFileSearchHit = {
|
||||
file_id: number;
|
||||
chunk_id: number;
|
||||
chunk_index: number;
|
||||
name: string;
|
||||
path: string;
|
||||
scope: string;
|
||||
/**
|
||||
* Cosine distance when the vector index answered, null for a word
|
||||
* match.
|
||||
*/
|
||||
distance: number | null;
|
||||
text: string;
|
||||
};
|
||||
|
||||
export type BotFileSearchResult = {
|
||||
hits: BotFileSearchHit[];
|
||||
/**
|
||||
* How the search ran: `vector`, `words` (no embedding model, or nothing
|
||||
* matched) or `none` (no files at all).
|
||||
*/
|
||||
mode: "vector" | "words" | "none";
|
||||
/**
|
||||
* Said plainly for the bot's own reading of what it got back.
|
||||
*/
|
||||
note: string;
|
||||
};
|
||||
|
||||
/**
|
||||
* Searches a run's files: by meaning through the embedding vectors when a
|
||||
* model is set, else (or when nothing matches) by words over the stored
|
||||
* chunks. Never throws for an unavailable model: the bot always gets
|
||||
* something it can act on.
|
||||
*/
|
||||
export default async function searchBotFiles({
|
||||
scopes,
|
||||
query,
|
||||
file_id,
|
||||
limit = AppData.RagTopChunks,
|
||||
signal,
|
||||
bot_id,
|
||||
workspace_id,
|
||||
}: Params): Promise<BotFileSearchResult> {
|
||||
const text = query.trim();
|
||||
|
||||
if (!scopes.length || !text) {
|
||||
return { hits: [], mode: "none", note: "No files to search." };
|
||||
}
|
||||
|
||||
try {
|
||||
const vector = await embedQuery({
|
||||
text,
|
||||
signal,
|
||||
bot_id,
|
||||
workspace_id,
|
||||
});
|
||||
|
||||
const hits = await vectorSearch({
|
||||
scopes,
|
||||
query_vector: vector,
|
||||
file_id,
|
||||
limit,
|
||||
seek: Math.min(limit * AppData.RagOverFetch, AppData.RagMaxK),
|
||||
});
|
||||
|
||||
if (hits.length) {
|
||||
return {
|
||||
hits,
|
||||
mode: "vector",
|
||||
note: `${hits.length} passage(s), found by meaning.`,
|
||||
};
|
||||
}
|
||||
} catch (error) {
|
||||
if (!(error instanceof HarnessError)) throw error;
|
||||
}
|
||||
|
||||
const hits = await wordSearch({ scopes, query: text, file_id, limit });
|
||||
|
||||
return {
|
||||
hits,
|
||||
mode: hits.length ? "words" : "none",
|
||||
note: hits.length
|
||||
? `${hits.length} passage(s), found by matching words (no embedding model, or nothing matched by meaning).`
|
||||
: `Nothing in your files matched "${text}".`,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The nearest chunks by vector. sqlite-vec refuses a filter on a vec0
|
||||
* side-car column in the same query as the match, so the index is asked
|
||||
* for its nearest `seek` and the run's scopes are applied to that list
|
||||
* here, before the chunk text is read.
|
||||
*/
|
||||
async function vectorSearch({
|
||||
scopes,
|
||||
query_vector,
|
||||
file_id,
|
||||
limit,
|
||||
seek,
|
||||
}: {
|
||||
scopes: StorageScope[];
|
||||
query_vector: Float32Array;
|
||||
file_id?: number | string | null;
|
||||
limit: number;
|
||||
seek: number;
|
||||
}): Promise<BotFileSearchHit[]> {
|
||||
const nearest = await BunSQLite.sql<{
|
||||
chunk_id: number;
|
||||
file_id: number;
|
||||
scope: string;
|
||||
owner_id: number;
|
||||
distance: number;
|
||||
}>({
|
||||
sql: `SELECT chunk_id, file_id, scope, owner_id, distance FROM bot_file_vectors WHERE embedding MATCH vec_f32(?) AND k = ? ORDER BY distance`,
|
||||
values: [query_vector as any, seek],
|
||||
});
|
||||
|
||||
/**
|
||||
* `BunSQLite.sql` resolves with `success: false` instead of throwing, so
|
||||
* a broken vector query would otherwise read as "nothing matched" and
|
||||
* silently fall back to words. A failure here is a real error.
|
||||
*/
|
||||
if (!nearest.success) {
|
||||
throw new Error(`The file search index couldn't be read: ${nearest.error || nearest.msg}`);
|
||||
}
|
||||
|
||||
const allowed = nearest.payload?.filter((row) =>
|
||||
scopes.some((scope) => scope.scope == row.scope && Number(scope.owner_id) == Number(row.owner_id)),
|
||||
);
|
||||
|
||||
const picked = (allowed || [])
|
||||
.filter((row) => !file_id || Number(row.file_id) == Number(file_id))
|
||||
.slice(0, limit);
|
||||
|
||||
if (!picked.length) return [];
|
||||
|
||||
const chunk_ids = picked.map((row) => Number(row.chunk_id));
|
||||
const placeholders = chunk_ids.map(() => "?").join(", ");
|
||||
|
||||
const chunks = await BunSQLite.sql<{
|
||||
id: number;
|
||||
text: string;
|
||||
chunk_index: number;
|
||||
name: string;
|
||||
path: string;
|
||||
}>({
|
||||
sql: `SELECT c.id, c.text, c.chunk_index, f.name, f.path FROM bot_file_chunks c
|
||||
JOIN bot_files f ON f.id = c.file_id
|
||||
WHERE c.id IN (${placeholders})`,
|
||||
values: chunk_ids,
|
||||
});
|
||||
|
||||
const by_id = new Map((chunks.payload || []).map((row) => [Number(row.id), row]));
|
||||
|
||||
return picked.flatMap((row) => {
|
||||
const chunk = by_id.get(Number(row.chunk_id));
|
||||
|
||||
if (!chunk) return [];
|
||||
|
||||
return [
|
||||
toHit({
|
||||
chunk_id: row.chunk_id,
|
||||
file_id: row.file_id,
|
||||
scope: row.scope,
|
||||
distance: Number(row.distance),
|
||||
text: chunk.text,
|
||||
chunk_index: chunk.chunk_index,
|
||||
name: chunk.name,
|
||||
path: chunk.path,
|
||||
}),
|
||||
];
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Chunks whose text has the query's words, newest files first. Used when
|
||||
* there's no embedding model, and as the fallback when the vectors
|
||||
* answered nothing.
|
||||
*/
|
||||
async function wordSearch({
|
||||
scopes,
|
||||
query,
|
||||
file_id,
|
||||
limit,
|
||||
}: {
|
||||
scopes: StorageScope[];
|
||||
query: string;
|
||||
file_id?: number | string | null;
|
||||
limit: number;
|
||||
}): Promise<BotFileSearchHit[]> {
|
||||
const words = [
|
||||
...new Set(
|
||||
query
|
||||
.toLowerCase()
|
||||
.split(/[^a-z0-9]+/)
|
||||
.filter((word) => word.length > 2),
|
||||
),
|
||||
].slice(0, 6);
|
||||
|
||||
if (!words.length) return [];
|
||||
|
||||
const scope_sql = scopes.map(() => `(c.scope = ? AND c.owner_id = ?)`).join(" OR ");
|
||||
const scope_values = scopes.flatMap((s) => [s.scope, s.owner_id]);
|
||||
const word_sql = words.map(() => `lower(c.text) LIKE ?`).join(" OR ");
|
||||
const word_values = words.map((word) => `%${word}%`);
|
||||
const file_clause = file_id ? "AND c.file_id = ?" : "";
|
||||
|
||||
const res = await BunSQLite.sql<{
|
||||
chunk_id: number;
|
||||
file_id: number;
|
||||
text: string;
|
||||
chunk_index: number;
|
||||
name: string;
|
||||
path: string;
|
||||
scope: string;
|
||||
}>({
|
||||
sql: `SELECT c.id AS chunk_id, c.file_id, c.text, c.chunk_index, c.scope, f.name, f.path
|
||||
FROM bot_file_chunks c
|
||||
JOIN bot_files f ON f.id = c.file_id
|
||||
WHERE (${scope_sql}) ${file_clause} AND (${word_sql})
|
||||
ORDER BY f.updated_at DESC, c.chunk_index
|
||||
LIMIT ?`,
|
||||
values: [...scope_values, ...(file_id ? [file_id] : []), ...word_values, limit],
|
||||
});
|
||||
|
||||
return (res.payload || []).map((row) => toHit({ ...row, distance: null }));
|
||||
}
|
||||
|
||||
function toHit(row: {
|
||||
chunk_id: number;
|
||||
file_id: number;
|
||||
text: string;
|
||||
chunk_index: number;
|
||||
name: string;
|
||||
path: string;
|
||||
scope: string;
|
||||
distance: number | null;
|
||||
}): BotFileSearchHit {
|
||||
return {
|
||||
file_id: Number(row.file_id),
|
||||
chunk_id: Number(row.chunk_id),
|
||||
chunk_index: Number(row.chunk_index),
|
||||
name: row.name || "",
|
||||
path: row.path || "",
|
||||
scope: row.scope || "bot",
|
||||
distance: row.distance == null ? null : Number(row.distance),
|
||||
text: row.text || "",
|
||||
};
|
||||
}
|
||||
@@ -13,10 +13,8 @@ async function source() {
|
||||
describe("storeChatAttachments", () => {
|
||||
/**
|
||||
* The same text used to be copied onto the new `bot-file` media row,
|
||||
* so a chat upload kept a second copy of its extracted text (and left
|
||||
* it behind when the upload was pruned). The text belongs only on the
|
||||
* message's own `bot-attachment` row, where a reply reads it;
|
||||
* ingestion re-reads the bytes through `extractBotFileText`.
|
||||
* so a chat upload kept a second copy of its extracted text. Bot files
|
||||
* keep no text at all: a bot reads the bytes when it needs them.
|
||||
*/
|
||||
test("never writes text_content onto the file's media row", async () => {
|
||||
const code = await source();
|
||||
@@ -25,10 +23,10 @@ describe("storeChatAttachments", () => {
|
||||
expect(code).not.toContain("text_content = ?");
|
||||
});
|
||||
|
||||
test("still saves the bytes and queues the ingest", async () => {
|
||||
test("saves the bytes and indexes nothing", async () => {
|
||||
const code = await source();
|
||||
|
||||
expect(code).toContain("recordBotFile(");
|
||||
expect(code).toContain("enqueueFileIngest(");
|
||||
expect(code).not.toContain("enqueueFileIngest(");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
import mediaFilePath from "@/src/utils/media-file-path";
|
||||
import type { BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db";
|
||||
import { BotUploadFolder } from "@/src/dict/bot-file-dict";
|
||||
import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
|
||||
import { logError } from "@/src/functions/backend/logger";
|
||||
import recordBotFile from "./record-bot-file";
|
||||
|
||||
@@ -14,7 +13,7 @@ type Params = {
|
||||
|
||||
/**
|
||||
* Copies the files the user attached to a message into the conversation's
|
||||
* folder (`uploads/<yyyy-mm>/<name>`), so they stay searchable after the
|
||||
* folder (`uploads/<yyyy-mm>/<name>`), so they stay findable after the
|
||||
* message scrolls away and a later run can find them. Best effort: a
|
||||
* failure is logged, never shown, and the message still goes out.
|
||||
*/
|
||||
@@ -48,8 +47,8 @@ export default async function storeChatAttachments({
|
||||
* The text stays on the message's own `bot-attachment` row,
|
||||
* where a reply reads it. Copying it onto the file's row kept
|
||||
* the same text twice and left the copy behind when the
|
||||
* upload was pruned; ingestion re-reads the bytes once
|
||||
* through `extractBotFileText` instead.
|
||||
* upload was pruned; a bot reads the bytes with `read_file`
|
||||
* instead.
|
||||
*/
|
||||
const saved = await recordBotFile({
|
||||
bot_id,
|
||||
@@ -63,8 +62,6 @@ export default async function storeChatAttachments({
|
||||
mime_type: file.mime_type || undefined,
|
||||
user_id,
|
||||
});
|
||||
|
||||
await enqueueFileIngest({ file_id: saved.id! });
|
||||
} catch (error) {
|
||||
await logError({
|
||||
title: `A chat attachment couldn't be saved to the bot's files`,
|
||||
|
||||
@@ -1,132 +0,0 @@
|
||||
import { afterAll, describe, expect, test } from "bun:test";
|
||||
import { Database } from "bun:sqlite";
|
||||
import * as sqliteVec from "sqlite-vec";
|
||||
import fitVectorDimensions from "@/src/harness/embeddings/fit-vector-dimensions";
|
||||
|
||||
/**
|
||||
* The exact insert and nearest-neighbour shape `ingestBotFile` and
|
||||
* `searchBotFiles` use, on a scratch database: vec0's metadata columns are
|
||||
* NOT NULL, and a side-car filter can't share the match query.
|
||||
*/
|
||||
function createDb() {
|
||||
const db = new Database(":memory:");
|
||||
sqliteVec.load(db);
|
||||
db.run(`CREATE VIRTUAL TABLE bot_file_vectors USING vec0(
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
created_at INTEGER,
|
||||
updated_at INTEGER,
|
||||
embedding float[8],
|
||||
+chunk_id INTEGER,
|
||||
+file_id INTEGER,
|
||||
+scope TEXT,
|
||||
+owner_id INTEGER
|
||||
)`);
|
||||
|
||||
return db;
|
||||
}
|
||||
|
||||
const vector = (values: number[]) => fitVectorDimensions({ vector: values, dimensions: 8 });
|
||||
|
||||
const insert = (db: Database, row: { chunk_id: number; file_id: number; scope: string; owner_id: number; v: number[] }) => {
|
||||
const now = Date.now();
|
||||
|
||||
db.run(
|
||||
`INSERT INTO bot_file_vectors (created_at, updated_at, embedding, chunk_id, file_id, scope, owner_id) VALUES (?, ?, vec_f32(?), ?, ?, ?, ?)`,
|
||||
[now, now, vector(row.v), row.chunk_id, row.file_id, row.scope, row.owner_id],
|
||||
);
|
||||
};
|
||||
|
||||
describe("bot file vectors", () => {
|
||||
test("stores a vector and matches the nearest one", () => {
|
||||
const db = createDb();
|
||||
|
||||
insert(db, { chunk_id: 1, file_id: 10, scope: "bot", owner_id: 1, v: [1, 0, 0, 0] });
|
||||
insert(db, { chunk_id: 2, file_id: 11, scope: "bot", owner_id: 1, v: [0, 1, 0, 0] });
|
||||
|
||||
const rows = db
|
||||
.query(
|
||||
`SELECT chunk_id, file_id, distance FROM bot_file_vectors WHERE embedding MATCH vec_f32(?1) AND k = 2 ORDER BY distance`,
|
||||
)
|
||||
.all(vector([0.9, 0.1, 0, 0])) as { chunk_id: number; distance: number }[];
|
||||
|
||||
expect(rows.length).toBe(2);
|
||||
expect(rows[0]!.chunk_id).toBe(1);
|
||||
expect(rows[0]!.distance).toBeLessThan(rows[1]!.distance);
|
||||
});
|
||||
|
||||
test("filters scopes in the app, not in the match query", () => {
|
||||
const db = createDb();
|
||||
|
||||
insert(db, { chunk_id: 1, file_id: 10, scope: "bot", owner_id: 1, v: [1, 0] });
|
||||
insert(db, { chunk_id: 2, file_id: 20, scope: "bot", owner_id: 2, v: [1, 0.05] });
|
||||
insert(db, { chunk_id: 3, file_id: 30, scope: "workspace", owner_id: 3, v: [1, 0.1] });
|
||||
|
||||
const nearest = db
|
||||
.query(`SELECT chunk_id, scope, owner_id, distance FROM bot_file_vectors WHERE embedding MATCH vec_f32(?1) AND k = 10 ORDER BY distance`)
|
||||
.all(vector([1, 0])) as { chunk_id: number; scope: string; owner_id: number }[];
|
||||
|
||||
expect(nearest.length).toBe(3);
|
||||
|
||||
const mine = nearest.filter((row) => row.scope == "bot" && row.owner_id == 1);
|
||||
|
||||
expect(mine.map((row) => row.chunk_id)).toEqual([1]);
|
||||
});
|
||||
|
||||
test("a side-car filter in the match query is refused", () => {
|
||||
const db = createDb();
|
||||
|
||||
insert(db, { chunk_id: 1, file_id: 10, scope: "bot", owner_id: 1, v: [1, 0] });
|
||||
|
||||
expect(() =>
|
||||
db
|
||||
.query(
|
||||
`SELECT chunk_id FROM bot_file_vectors WHERE embedding MATCH vec_f32(?1) AND k = 2 AND scope = 'bot'`,
|
||||
)
|
||||
.all(vector([1, 0])),
|
||||
).toThrow(/illegal WHERE constraint/);
|
||||
});
|
||||
|
||||
test("an insert without vec0's metadata columns is refused", () => {
|
||||
const db = createDb();
|
||||
|
||||
expect(() =>
|
||||
db.run(
|
||||
`INSERT INTO bot_file_vectors (embedding, chunk_id, file_id, scope, owner_id) VALUES (vec_f32(?1), 1, 10, 'bot', 1)`,
|
||||
[vector([1, 0])],
|
||||
),
|
||||
).toThrow();
|
||||
});
|
||||
|
||||
test("deletes a file's vectors by its side-car column", () => {
|
||||
const db = createDb();
|
||||
|
||||
insert(db, { chunk_id: 1, file_id: 10, scope: "bot", owner_id: 1, v: [1, 0] });
|
||||
|
||||
db.run(`DELETE FROM bot_file_vectors WHERE file_id = ?`, [10]);
|
||||
|
||||
const count = db.query(`SELECT COUNT(*) AS n FROM bot_file_vectors`).get() as { n: number };
|
||||
|
||||
expect(count.n).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("fitVectorDimensions", () => {
|
||||
test("pads a short vector and normalizes it", () => {
|
||||
const out = fitVectorDimensions({ vector: [3, 4], dimensions: 4 });
|
||||
const norm = Math.sqrt([...out].reduce((sum, value) => sum + value * value, 0));
|
||||
|
||||
expect(out.length).toBe(4);
|
||||
expect(Math.round(norm * 1000) / 1000).toBe(1);
|
||||
expect(Math.round(out[0]! * 1000) / 1000).toBe(0.6);
|
||||
});
|
||||
|
||||
test("cuts a long vector", () => {
|
||||
expect(fitVectorDimensions({ vector: [1, 1, 1, 1, 1, 1], dimensions: 2 }).length).toBe(2);
|
||||
});
|
||||
|
||||
test("leaves a zero vector as zeros", () => {
|
||||
expect([...fitVectorDimensions({ vector: [0, 0], dimensions: 2 })]).toEqual([0, 0]);
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(() => {});
|
||||
@@ -30,18 +30,16 @@ db.run(
|
||||
db.run(`INSERT INTO bot_message_media (id, message_id, media_id) VALUES (1, 10, 1)`);
|
||||
|
||||
/**
|
||||
* The query `grabMessagesMedia` builds, with the text column included or
|
||||
* left out. It is the largest column in the table, so only the reply path
|
||||
* that hands a document to a model asks for it.
|
||||
* The query `grabMessagesMedia` builds. The text column is never read: a
|
||||
* reply reads an attachment's text from the file itself.
|
||||
*/
|
||||
function read({ with_text_content }: { with_text_content: boolean }) {
|
||||
function read() {
|
||||
const columns = [
|
||||
"m.id",
|
||||
"m.title",
|
||||
"m.media_type",
|
||||
"m.media_write_path",
|
||||
"m.mime_type",
|
||||
...(with_text_content ? ["m.text_content"] : []),
|
||||
].join(", ");
|
||||
|
||||
return db
|
||||
@@ -55,15 +53,11 @@ function read({ with_text_content }: { with_text_content: boolean }) {
|
||||
|
||||
describe("grabMessagesMedia", () => {
|
||||
test("a plain read never drags the extracted text along", () => {
|
||||
expect(read({ with_text_content: false })).not.toHaveProperty("text_content");
|
||||
});
|
||||
|
||||
test("the reply path can ask for it", () => {
|
||||
expect(read({ with_text_content: true }).text_content).toBe("Total: 12");
|
||||
expect(read()).not.toHaveProperty("text_content");
|
||||
});
|
||||
|
||||
test("the row's name and path are always there", () => {
|
||||
expect(read({ with_text_content: false })).toMatchObject({
|
||||
expect(read()).toMatchObject({
|
||||
title: "invoice.pdf",
|
||||
media_write_path: "media/1.pdf",
|
||||
message_id: 10,
|
||||
|
||||
@@ -4,12 +4,6 @@ import mediaColumns from "@/src/functions/backend/db/media/media-columns";
|
||||
|
||||
type Params = {
|
||||
message_ids: number[];
|
||||
/**
|
||||
* Also read `text_content`, the extracted text a reply hands a
|
||||
* document to the model as. Off by default: it is the largest column
|
||||
* in the table and nothing else here uses it.
|
||||
*/
|
||||
with_text_content?: boolean;
|
||||
};
|
||||
|
||||
export type MessageMediaRow = BUN_SQLITE_COBBLE_MEDIA & { message_id: number };
|
||||
@@ -20,14 +14,13 @@ export type MessageMediaRow = BUN_SQLITE_COBBLE_MEDIA & { message_id: number };
|
||||
*/
|
||||
export default async function grabMessagesMedia({
|
||||
message_ids,
|
||||
with_text_content,
|
||||
}: Params) {
|
||||
const by_message = new Map<number, MessageMediaRow[]>();
|
||||
const ids = [...new Set(message_ids.filter(Boolean))];
|
||||
|
||||
if (!ids.length) return by_message;
|
||||
|
||||
const columns = mediaColumns({ alias: "m", with_text_content });
|
||||
const columns = mediaColumns({ alias: "m" });
|
||||
|
||||
const rows =
|
||||
(
|
||||
|
||||
@@ -5,9 +5,8 @@
|
||||
* `AppData.BotAttachmentTextStoreMaxChars` of extracted document text, and
|
||||
* every caller that only needs a file's name, type or path (`SELECT m.*`
|
||||
* did) pulled it into memory, into page props and into API responses.
|
||||
* Read it explicitly where it is really used:
|
||||
* `src/bots-queue/load-attachment-parts.ts` and
|
||||
* `src/functions/backend/db/bot-files/ingest-bot-file.ts`.
|
||||
* Only knowledgebase documents keep it now; read it explicitly where it
|
||||
* is really used.
|
||||
*/
|
||||
export default function mediaColumns({
|
||||
alias = "m",
|
||||
@@ -31,8 +30,7 @@ export default function mediaColumns({
|
||||
];
|
||||
|
||||
/**
|
||||
* Only the two paths that really use it ask for it: the reply that
|
||||
* hands a document to a model, and ingestion.
|
||||
* Only a caller that really uses it asks for it.
|
||||
*/
|
||||
if (with_text_content) columns.push("text_content");
|
||||
|
||||
|
||||
@@ -1,65 +0,0 @@
|
||||
import { describe, expect, test } from "bun:test";
|
||||
import { AppData } from "@/src/data/app-data";
|
||||
import planTextContentPrune from "./plan-text-content-prune";
|
||||
|
||||
describe("planTextContentPrune", () => {
|
||||
test("clears the raw bytes stored for a type the extractor can't read", () => {
|
||||
const changes = planTextContentPrune({
|
||||
rows: [
|
||||
{ id: 1, title: "logo.svg", mime_type: "image/svg+xml", text_content: "<svg><path/></svg>" },
|
||||
{ id: 2, title: "bundle.zip", mime_type: "application/zip", text_content: "PK\u0003\u0004" },
|
||||
{ id: 3, title: "sheet.xlsx", mime_type: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", text_content: "PK\u0003\u0004" },
|
||||
],
|
||||
});
|
||||
|
||||
expect(changes.map((change) => change.media_id)).toEqual([1, 2, 3]);
|
||||
expect(changes.every((change) => change.text_content === null)).toBe(true);
|
||||
expect(changes.every((change) => change.after_chars === 0)).toBe(true);
|
||||
});
|
||||
|
||||
test("leaves a document's text alone", () => {
|
||||
const changes = planTextContentPrune({
|
||||
rows: [
|
||||
{ id: 4, title: "invoice.pdf", mime_type: "application/pdf", text_content: "Total: 12" },
|
||||
{ id: 5, title: "notes.md", mime_type: "text/markdown", text_content: "# Hi" },
|
||||
{ id: 6, title: "rows.csv", mime_type: "text/csv", text_content: "a,b\n1,2" },
|
||||
],
|
||||
});
|
||||
|
||||
expect(changes).toEqual([]);
|
||||
});
|
||||
|
||||
test("cuts a document over the store ceiling", () => {
|
||||
const text = "a".repeat(AppData.BotAttachmentTextStoreMaxChars + 100);
|
||||
const changes = planTextContentPrune({
|
||||
rows: [{ id: 7, title: "long.txt", mime_type: "text/plain", text_content: text }],
|
||||
});
|
||||
|
||||
expect(changes).toHaveLength(1);
|
||||
expect(changes[0]!.after_chars).toBeLessThanOrEqual(AppData.BotAttachmentTextStoreMaxChars);
|
||||
expect(changes[0]!.text_content).toContain("Truncated");
|
||||
expect(changes[0]!.before_chars).toBe(text.length);
|
||||
});
|
||||
|
||||
test("nothing to say about a row with no text or no id", () => {
|
||||
expect(
|
||||
planTextContentPrune({
|
||||
rows: [
|
||||
{ id: 8, title: "empty.pdf", text_content: "" },
|
||||
{ id: 9, title: "blank.pdf", text_content: " \n " },
|
||||
{ id: 10, title: "none.pdf" },
|
||||
{ title: "no id.pdf", text_content: "text" },
|
||||
],
|
||||
}),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
test("counting what each change reclaims is left to the caller", () => {
|
||||
const changes = planTextContentPrune({
|
||||
rows: [{ id: 11, title: "icon.svg", mime_type: "image/svg+xml", text_content: "12345" }],
|
||||
});
|
||||
|
||||
expect(changes).toHaveLength(1);
|
||||
expect(changes[0]!.before_chars - changes[0]!.after_chars).toBe(5);
|
||||
});
|
||||
});
|
||||
@@ -1,55 +0,0 @@
|
||||
import type { BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db";
|
||||
import decideTextContent from "@/src/utils/decide-text-content";
|
||||
|
||||
export type TextContentChange = {
|
||||
media_id: number;
|
||||
name: string;
|
||||
before_chars: number;
|
||||
after_chars: number;
|
||||
/**
|
||||
* What the row's `text_content` should become: `null` clears it.
|
||||
*/
|
||||
text_content: string | null;
|
||||
};
|
||||
|
||||
type Params = {
|
||||
/**
|
||||
* The `media` rows that have any `text_content`.
|
||||
*/
|
||||
rows: Partial<BUN_SQLITE_COBBLE_MEDIA>[];
|
||||
};
|
||||
|
||||
/**
|
||||
* What `pruneMediaTextContent` would change, worked out without touching
|
||||
* the database: one entry per row whose text belongs nowhere or is over
|
||||
* the store ceiling, and nothing at all for a row that is already right.
|
||||
*/
|
||||
export default function planTextContentPrune({ rows }: Params): TextContentChange[] {
|
||||
const changes: TextContentChange[] = [];
|
||||
|
||||
for (const row of rows) {
|
||||
if (!row.id) continue;
|
||||
|
||||
const before = row.text_content || "";
|
||||
|
||||
if (!before.trim()) continue;
|
||||
|
||||
const text_content = decideTextContent({
|
||||
name: row.title,
|
||||
mime_type: row.mime_type,
|
||||
text: before,
|
||||
});
|
||||
|
||||
if (text_content === before) continue;
|
||||
|
||||
changes.push({
|
||||
media_id: Number(row.id),
|
||||
name: row.title || "no name",
|
||||
before_chars: before.length,
|
||||
after_chars: text_content?.length || 0,
|
||||
text_content,
|
||||
});
|
||||
}
|
||||
|
||||
return changes;
|
||||
}
|
||||
@@ -1,120 +0,0 @@
|
||||
import { describe, expect, test } from "bun:test";
|
||||
import { Database } from "bun:sqlite";
|
||||
import { AppData } from "@/src/data/app-data";
|
||||
import pruneMediaTextContent, { type SqlRunner } from "./prune-media-text-content";
|
||||
|
||||
type Row = { id: number; title: string; mime_type: string; media_type: string; text_content: string | null; updated_at?: number };
|
||||
|
||||
function makeDb({ rows }: { rows: Row[] }) {
|
||||
const db = new Database(":memory:");
|
||||
|
||||
db.run(
|
||||
`CREATE TABLE media (
|
||||
id INTEGER PRIMARY KEY,
|
||||
title TEXT,
|
||||
mime_type TEXT,
|
||||
media_type TEXT,
|
||||
text_content TEXT,
|
||||
updated_at INTEGER
|
||||
)`,
|
||||
);
|
||||
|
||||
const insert = db.prepare(
|
||||
`INSERT INTO media (id, title, mime_type, media_type, text_content) VALUES (?, ?, ?, ?, ?)`,
|
||||
);
|
||||
|
||||
for (const row of rows) insert.run(row.id, row.title, row.mime_type, row.media_type, row.text_content);
|
||||
|
||||
const run: SqlRunner = async ({ sql, values }) => {
|
||||
const statement = db.prepare(sql);
|
||||
|
||||
if (sql.trim().toUpperCase().startsWith("SELECT")) {
|
||||
return { payload: statement.all(...((values || []) as never[])) as unknown[] };
|
||||
}
|
||||
|
||||
statement.run(...((values || []) as never[]));
|
||||
|
||||
return { payload: [] };
|
||||
};
|
||||
|
||||
const read = (id: number) =>
|
||||
(db.query(`SELECT text_content FROM media WHERE id = ?`).get(id) as { text_content: string | null }).text_content;
|
||||
|
||||
return { db, run, read };
|
||||
}
|
||||
|
||||
describe("pruneMediaTextContent", () => {
|
||||
test("clears raw bytes from a type the extractor can't read, and keeps a document's text", async () => {
|
||||
const { run, read } = makeDb({
|
||||
rows: [
|
||||
{ id: 1, title: "logo.svg", mime_type: "image/svg+xml", media_type: "document", text_content: "<svg><path d='M0 0'/></svg>" },
|
||||
{ id: 2, title: "bundle.zip", mime_type: "application/zip", media_type: "document", text_content: "PK\u0003\u0004binary" },
|
||||
{ id: 3, title: "invoice.pdf", mime_type: "application/pdf", media_type: "document", text_content: "Total: 12" },
|
||||
],
|
||||
});
|
||||
|
||||
const result = await pruneMediaTextContent({ run });
|
||||
|
||||
expect(result.checked).toBe(3);
|
||||
expect(result.cleared).toBe(2);
|
||||
expect(result.truncated).toBe(0);
|
||||
expect(read(1)).toBeNull();
|
||||
expect(read(2)).toBeNull();
|
||||
expect(read(3)).toBe("Total: 12");
|
||||
});
|
||||
|
||||
test("cuts an oversized document to the store ceiling", async () => {
|
||||
const text = "a".repeat(AppData.BotAttachmentTextStoreMaxChars + 1000);
|
||||
const { run, read } = makeDb({
|
||||
rows: [{ id: 4, title: "huge.txt", mime_type: "text/plain", media_type: "document", text_content: text }],
|
||||
});
|
||||
|
||||
const result = await pruneMediaTextContent({ run });
|
||||
const after = read(4)!;
|
||||
|
||||
expect(result.truncated).toBe(1);
|
||||
expect(result.cleared).toBe(0);
|
||||
expect(after.length).toBeLessThanOrEqual(AppData.BotAttachmentTextStoreMaxChars);
|
||||
expect(after).toContain("Truncated");
|
||||
expect(result.reclaimed_bytes).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test("is safe to rerun: a second sweep changes nothing", async () => {
|
||||
const { run, read } = makeDb({
|
||||
rows: [{ id: 5, title: "icon.svg", mime_type: "image/svg+xml", media_type: "document", text_content: "raw" }],
|
||||
});
|
||||
|
||||
await pruneMediaTextContent({ run });
|
||||
const second = await pruneMediaTextContent({ run });
|
||||
|
||||
expect(second.checked).toBe(0);
|
||||
expect(second.changes).toEqual([]);
|
||||
expect(second.reclaimed_bytes).toBe(0);
|
||||
expect(read(5)).toBeNull();
|
||||
});
|
||||
|
||||
test("stamps updated_at on the rows it rewrites", async () => {
|
||||
const now = 1_700_000_000_000;
|
||||
const { run, db } = makeDb({
|
||||
rows: [{ id: 6, title: "x.zip", mime_type: "application/zip", media_type: "document", text_content: "raw" }],
|
||||
});
|
||||
|
||||
await pruneMediaTextContent({ run, now });
|
||||
|
||||
expect(
|
||||
(db.query(`SELECT updated_at FROM media WHERE id = 6`).get() as { updated_at: number }).updated_at,
|
||||
).toBe(now);
|
||||
});
|
||||
|
||||
test("reports every change with its row, name and sizes", async () => {
|
||||
const { run } = makeDb({
|
||||
rows: [{ id: 7, title: "sheet.xlsx", mime_type: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", media_type: "document", text_content: "PK\u0003\u0004" }],
|
||||
});
|
||||
|
||||
const result = await pruneMediaTextContent({ run });
|
||||
|
||||
expect(result.changes).toEqual([
|
||||
{ media_id: 7, name: "sheet.xlsx", before_chars: 4, after_chars: 0, text_content: null },
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -1,75 +0,0 @@
|
||||
import BunSQLite from "@moduletrace/bun-sqlite";
|
||||
import type { BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db";
|
||||
import planTextContentPrune, { type TextContentChange } from "./plan-text-content-prune";
|
||||
|
||||
/**
|
||||
* One statement against the database. `BunSQLite.sql` is the real one; a
|
||||
* test passes its own, so the whole sweep can be run without the app's
|
||||
* database.
|
||||
*/
|
||||
export type SqlRunner = (params: {
|
||||
sql: string;
|
||||
values?: (string | number | null)[];
|
||||
}) => Promise<{ payload?: unknown[] | null }>;
|
||||
|
||||
export type PruneMediaTextResult = {
|
||||
checked: number;
|
||||
cleared: number;
|
||||
truncated: number;
|
||||
reclaimed_bytes: number;
|
||||
changes: TextContentChange[];
|
||||
};
|
||||
|
||||
type Params = {
|
||||
run?: SqlRunner;
|
||||
/**
|
||||
* When the sweep runs, for the rows it writes.
|
||||
*/
|
||||
now?: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Brings every `media.text_content` in line with what may be stored there
|
||||
* (`planTextContentPrune`): no text on a type the extractor can't read,
|
||||
* and no text longer than the store ceiling.
|
||||
*
|
||||
* Run once per install after the allowlist landed; safe to rerun, since a
|
||||
* row that is already right is not written.
|
||||
*/
|
||||
export default async function pruneMediaTextContent({
|
||||
run,
|
||||
now = Date.now(),
|
||||
}: Params = {}): Promise<PruneMediaTextResult> {
|
||||
const sql: SqlRunner =
|
||||
run || (({ sql, values }) => BunSQLite.sql<BUN_SQLITE_COBBLE_MEDIA>({ sql, values }));
|
||||
|
||||
const res = await sql({
|
||||
sql: `SELECT id, title, mime_type, media_type, text_content
|
||||
FROM media
|
||||
WHERE text_content IS NOT NULL AND TRIM(text_content) != ''`,
|
||||
});
|
||||
|
||||
const rows = (res.payload || []) as BUN_SQLITE_COBBLE_MEDIA[];
|
||||
const changes = planTextContentPrune({ rows });
|
||||
const result: PruneMediaTextResult = {
|
||||
checked: rows.length,
|
||||
cleared: 0,
|
||||
truncated: 0,
|
||||
reclaimed_bytes: 0,
|
||||
changes,
|
||||
};
|
||||
|
||||
for (const change of changes) {
|
||||
await sql({
|
||||
sql: `UPDATE media SET text_content = ?, updated_at = ? WHERE id = ?`,
|
||||
values: [change.text_content, now, change.media_id],
|
||||
});
|
||||
|
||||
result.reclaimed_bytes += change.before_chars - change.after_chars;
|
||||
|
||||
if (change.text_content === null) result.cleared += 1;
|
||||
else result.truncated += 1;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
import { Database } from "bun:sqlite";
|
||||
import { describe, expect, test } from "bun:test";
|
||||
import dropStoredFileTextSql from "./drop-stored-file-text-sql";
|
||||
|
||||
function scratch() {
|
||||
const db = new Database(":memory:");
|
||||
|
||||
db.run(`CREATE TABLE media (id INTEGER PRIMARY KEY, text_content TEXT)`);
|
||||
db.run(`CREATE TABLE knowledgebase_documents (id INTEGER PRIMARY KEY, media_id INTEGER)`);
|
||||
db.run(`CREATE TABLE file_ingest_queue (id INTEGER PRIMARY KEY, document_id INTEGER)`);
|
||||
db.run(`CREATE TABLE bot_file_chunks (id INTEGER PRIMARY KEY, text TEXT)`);
|
||||
|
||||
db.run(`INSERT INTO media (id, text_content) VALUES (1, 'an attachment'), (2, 'a knowledgebase upload'), (3, NULL)`);
|
||||
db.run(`INSERT INTO knowledgebase_documents (id, media_id) VALUES (1, 2), (2, NULL)`);
|
||||
db.run(`INSERT INTO file_ingest_queue (id, document_id) VALUES (1, NULL), (2, 1)`);
|
||||
db.run(`INSERT INTO bot_file_chunks (id, text) VALUES (1, 'old passage')`);
|
||||
|
||||
return db;
|
||||
}
|
||||
|
||||
function run(db: Database) {
|
||||
for (const sql of dropStoredFileTextSql()) db.run(sql);
|
||||
}
|
||||
|
||||
describe("dropStoredFileTextSql", () => {
|
||||
test("drops the bot file index", () => {
|
||||
const db = scratch();
|
||||
|
||||
run(db);
|
||||
|
||||
const tables = db.query(`SELECT name FROM sqlite_master WHERE name LIKE 'bot_file_%'`).all();
|
||||
|
||||
expect(tables).toEqual([]);
|
||||
});
|
||||
|
||||
test("clears upload text but keeps a knowledgebase document's", () => {
|
||||
const db = scratch();
|
||||
|
||||
run(db);
|
||||
|
||||
const rows = db.query(`SELECT id, text_content FROM media ORDER BY id`).all();
|
||||
|
||||
expect(rows).toEqual([
|
||||
{ id: 1, text_content: null },
|
||||
{ id: 2, text_content: "a knowledgebase upload" },
|
||||
{ id: 3, text_content: null },
|
||||
]);
|
||||
});
|
||||
|
||||
test("keeps only document jobs on the ingest queue", () => {
|
||||
const db = scratch();
|
||||
|
||||
run(db);
|
||||
|
||||
expect(db.query(`SELECT id FROM file_ingest_queue`).all()).toEqual([{ id: 2 }]);
|
||||
});
|
||||
|
||||
test("runs again on a clean database without failing", () => {
|
||||
const db = scratch();
|
||||
|
||||
run(db);
|
||||
|
||||
expect(() => run(db)).not.toThrow();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,17 @@
|
||||
/**
|
||||
* The statements that remove text stored outside knowledgebases: the old
|
||||
* bot file index (chunks and their vectors; the vec0 drop takes its shadow
|
||||
* tables with it), file jobs left on the ingest queue, and the extracted
|
||||
* text uploads used to keep. A knowledgebase document's media keeps its
|
||||
* text. Every statement is a no-op once it has run.
|
||||
*/
|
||||
export default function dropStoredFileTextSql(): string[] {
|
||||
return [
|
||||
`DROP TABLE IF EXISTS bot_file_vectors`,
|
||||
`DROP TABLE IF EXISTS bot_file_chunks`,
|
||||
`DELETE FROM file_ingest_queue WHERE document_id IS NULL`,
|
||||
`UPDATE media SET text_content = NULL
|
||||
WHERE text_content IS NOT NULL
|
||||
AND id NOT IN (SELECT media_id FROM knowledgebase_documents WHERE media_id IS NOT NULL)`,
|
||||
];
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
import BunSQLite from "@moduletrace/bun-sqlite";
|
||||
import logEvent, { logError } from "@/src/functions/backend/logger";
|
||||
import dropStoredFileTextSql from "./drop-stored-file-text-sql";
|
||||
import shouldVacuum from "./should-vacuum";
|
||||
|
||||
/**
|
||||
* Knowledgebases are the only place text is kept for search. Removes what
|
||||
* bot files and uploads stored before that (`dropStoredFileTextSql`), then
|
||||
* vacuums once if the database has a lot of free pages, so the space comes
|
||||
* back on disk. Safe on every start: once clean, it only reads two pragmas.
|
||||
*/
|
||||
export default async function dropStoredFileText() {
|
||||
for (const sql of dropStoredFileTextSql()) {
|
||||
const res = await BunSQLite.sql({ sql });
|
||||
|
||||
if (!res.success) throw new Error(`${sql.split("\n")[0]}: ${res.error || res.msg}`);
|
||||
}
|
||||
|
||||
const [page_count, freelist_count, page_size] = await Promise.all(
|
||||
["page_count", "freelist_count", "page_size"].map((pragma) =>
|
||||
BunSQLite.sql<Record<string, number>>({ sql: `PRAGMA ${pragma}` }).then((res) =>
|
||||
Number(res.payload?.[0]?.[pragma]) || 0,
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
if (!shouldVacuum({ page_count: page_count!, freelist_count: freelist_count!, page_size: page_size! })) return;
|
||||
|
||||
const res = await BunSQLite.sql({ sql: `VACUUM` });
|
||||
|
||||
if (!res.success) {
|
||||
await logError({
|
||||
title: "Could not vacuum the database",
|
||||
error: new Error(`${res.error || res.msg}. It needs about as much free disk as the database takes.`),
|
||||
metadata: { free_bytes: freelist_count! * page_size! },
|
||||
}).catch(() => {});
|
||||
return;
|
||||
}
|
||||
|
||||
await logEvent({
|
||||
title: "Vacuumed the database",
|
||||
category: "system",
|
||||
plain_text: `Gave back ${Math.round((freelist_count! * page_size!) / (1024 * 1024))} MB of free pages to the disk.`,
|
||||
metadata: { freed_bytes: freelist_count! * page_size! },
|
||||
}).catch(() => {});
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
import { describe, expect, test } from "bun:test";
|
||||
import { AppData } from "@/src/data/app-data";
|
||||
import shouldVacuum from "./should-vacuum";
|
||||
|
||||
const page_size = 4096;
|
||||
const min_pages = Math.ceil(AppData.DbVacuumMinFreeBytes / page_size);
|
||||
|
||||
describe("shouldVacuum", () => {
|
||||
test("vacuums when a large share of a big file is free", () => {
|
||||
expect(shouldVacuum({ page_count: min_pages * 2, freelist_count: min_pages, page_size })).toBe(true);
|
||||
});
|
||||
|
||||
test("leaves a database with little free space alone", () => {
|
||||
expect(shouldVacuum({ page_count: min_pages * 100, freelist_count: min_pages, page_size })).toBe(false);
|
||||
});
|
||||
|
||||
test("leaves a small database alone even when mostly free", () => {
|
||||
expect(shouldVacuum({ page_count: 20, freelist_count: 15, page_size })).toBe(false);
|
||||
});
|
||||
|
||||
test("never vacuums on missing numbers", () => {
|
||||
expect(shouldVacuum({ page_count: 0, freelist_count: 0, page_size })).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,22 @@
|
||||
import { AppData } from "@/src/data/app-data";
|
||||
|
||||
type Params = {
|
||||
page_count: number;
|
||||
freelist_count: number;
|
||||
page_size: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Whether enough of the database file sits free to be worth a `VACUUM`
|
||||
* (`AppData.DbVacuumMinFreeRatio` of its pages and at least
|
||||
* `AppData.DbVacuumMinFreeBytes`). Deleted rows only free pages inside the
|
||||
* file; it shrinks on disk only when vacuumed.
|
||||
*/
|
||||
export default function shouldVacuum({ page_count, freelist_count, page_size }: Params) {
|
||||
if (!(page_count > 0) || !(freelist_count > 0)) return false;
|
||||
|
||||
return (
|
||||
freelist_count / page_count >= AppData.DbVacuumMinFreeRatio &&
|
||||
freelist_count * page_size >= AppData.DbVacuumMinFreeBytes
|
||||
);
|
||||
}
|
||||
@@ -4,7 +4,7 @@ type Params = {
|
||||
vector: number[];
|
||||
/**
|
||||
* Defaults to `AppData.EmbeddingDimensions`, the width of the fixed
|
||||
* `bot_file_vectors` table.
|
||||
* `knowledgebase_vectors` table.
|
||||
*/
|
||||
dimensions?: number;
|
||||
};
|
||||
|
||||
@@ -3,7 +3,6 @@ import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { z } from "zod";
|
||||
import { AppData } from "@/src/data/app-data";
|
||||
import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
|
||||
import recordBotFile, { grabBotFileRow } from "@/src/functions/backend/db/bot-files/record-bot-file";
|
||||
import grabComputerMcpUrl from "@/src/functions/backend/db/computers/grab-computer-mcp-url";
|
||||
import computerNotReadyMessage from "@/src/utils/computer-not-ready-message";
|
||||
@@ -49,7 +48,7 @@ type Args = {
|
||||
* Copies what a bot made on its computer into its own files: one file, or
|
||||
* a whole folder (taken as a gzipped tar and unpacked into one row per
|
||||
* file). The copies are kept with the app, stay after the computer is
|
||||
* rebuilt, and are searchable at once; the originals stay on the computer,
|
||||
* rebuilt, and are readable at once; the originals stay on the computer,
|
||||
* where the message's folder ref points at the folder holding them.
|
||||
*
|
||||
* The bytes come back over the sandbox's streaming download endpoint
|
||||
@@ -72,7 +71,7 @@ export default function importFileTool({
|
||||
name: "import_file",
|
||||
description: `Copy a file or folder from this computer into your own files, so it is kept with the app and stays after the computer is rebuilt. Give its path on the computer, relative to your folder there (${dirs.cwd}), e.g. "seo/coderank/raw/crawl.json" or "seo/coderank/raw"; a folder comes in as it is, in one go. Up to ${max_mb} MB. Check with list_files and the computer's own tools what you already imported: importing the same thing twice just writes over the copy, so do it once.${
|
||||
own?.scope == "workspace" ? " The copies land in the workspace's shared files." : ""
|
||||
} Imports are searchable at once. Name a copy with its [file:ID] ref. Import the report you wrote before the last long steps of a run, so it is never lost to the step limit.`,
|
||||
} Imports are readable at once. Name a copy with its [file:ID] ref. Import the report you wrote before the last long steps of a run, so it is never lost to the step limit.`,
|
||||
schema: z.object({
|
||||
path: z
|
||||
.string()
|
||||
@@ -190,8 +189,6 @@ async function importOneFile({
|
||||
user_id: target.user_id,
|
||||
});
|
||||
|
||||
if (saved.id) await enqueueFileIngest({ file_id: saved.id });
|
||||
|
||||
if (attach && saved.media_id) {
|
||||
onAttach?.({ media_id: Number(saved.media_id), name: saved.name || name });
|
||||
}
|
||||
@@ -427,8 +424,6 @@ async function unpackArchive({
|
||||
|
||||
if (!row?.id) continue;
|
||||
|
||||
await enqueueFileIngest({ file_id: row.id }).catch(() => {});
|
||||
|
||||
saved.push({ id: Number(row.id), path: row.path || `${folder}/${relative}` });
|
||||
}
|
||||
} finally {
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
import { z } from "zod";
|
||||
import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
|
||||
import recordBotFile from "@/src/functions/backend/db/bot-files/record-bot-file";
|
||||
import generateImage from "@/src/harness/image/generate-image";
|
||||
import type { HarnessTool } from "../../types";
|
||||
@@ -98,8 +97,6 @@ export default function generateImageTool(context: StorageToolContext): HarnessT
|
||||
user_id: context.user_id,
|
||||
});
|
||||
|
||||
await enqueueFileIngest({ file_id: file.id! });
|
||||
|
||||
if (args.attach && file.media_id) {
|
||||
context.onAttach?.({ media_id: Number(file.media_id), name: file.name || "" });
|
||||
}
|
||||
|
||||
@@ -1,11 +1,9 @@
|
||||
import type { HarnessTool } from "../../types";
|
||||
import copyFileTool from "./copy-file-tool";
|
||||
import deleteFileTool from "./delete-file-tool";
|
||||
import ingestFileTool from "./ingest-file-tool";
|
||||
import listFilesTool from "./list-files-tool";
|
||||
import moveFileTool from "./move-file-tool";
|
||||
import readFileTool from "./read-file-tool";
|
||||
import searchFilesTool from "./search-files-tool";
|
||||
import type { StorageToolContext } from "./storage-tool-context";
|
||||
import writeFileTool from "./write-file-tool";
|
||||
|
||||
@@ -19,10 +17,8 @@ export default function grabStorageTools(context: StorageToolContext): HarnessTo
|
||||
listFilesTool(context),
|
||||
readFileTool(context),
|
||||
writeFileTool(context),
|
||||
searchFilesTool(context),
|
||||
moveFileTool(context),
|
||||
deleteFileTool(context),
|
||||
ingestFileTool(context),
|
||||
...(context.scopes.length > 1 ? [copyFileTool(context)] : []),
|
||||
];
|
||||
}
|
||||
|
||||
@@ -1,44 +0,0 @@
|
||||
import { z } from "zod";
|
||||
import { AppData } from "@/src/data/app-data";
|
||||
import ingestBotFile from "@/src/functions/backend/db/bot-files/ingest-bot-file";
|
||||
import type { HarnessTool } from "../../types";
|
||||
import type { StorageToolContext } from "./storage-tool-context";
|
||||
|
||||
type Args = {
|
||||
file_id: number;
|
||||
read_media?: boolean;
|
||||
};
|
||||
|
||||
/**
|
||||
* Ingests a file again: reads its text, chunks it and embeds it, so a
|
||||
* changed or previously failed file is searchable. Images and audio are
|
||||
* only read when asked for, since that costs a model call.
|
||||
*/
|
||||
export default function ingestFileTool(_context: StorageToolContext): HarnessTool<Args> {
|
||||
return {
|
||||
name: "ingest_file",
|
||||
description: `Make one of your files searchable again (a course correction when search misses something you know is in it), by its ref id. Images and scans are only read when "read_media" is true.`,
|
||||
schema: z.object({
|
||||
file_id: z.number().int().positive(),
|
||||
read_media: z.boolean().optional(),
|
||||
}),
|
||||
run: async ({ args, signal }) => {
|
||||
const result = await ingestBotFile({
|
||||
file_id: args.file_id,
|
||||
read_media: args.read_media,
|
||||
signal,
|
||||
});
|
||||
|
||||
switch (result.status) {
|
||||
case "ingested":
|
||||
return `Ingested: ${result.chunks} passage(s), searchable now.`;
|
||||
case "unavailable":
|
||||
return `Its text is stored and searchable by words, but no embedding model is set, so meaning search is off.`;
|
||||
case "skipped":
|
||||
return `It's too large to read (over ${Math.round(AppData.IngestMaxBytes / (1024 * 1024))} MB).`;
|
||||
default:
|
||||
return `It has no text to read.`;
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -50,9 +50,8 @@ export default function listFilesTool(context: StorageToolContext): HarnessTool<
|
||||
|
||||
const lines = files.map((file) => {
|
||||
const where = file.scope == "workspace" ? ` (workspace ${file.owner_id})` : "";
|
||||
const ingested = file.ingested_at ? "" : " - not searchable yet";
|
||||
|
||||
return `- ${file.path} - ${formatBytes({ bytes: Number(file.size) || 0 })}${where}${ingested} ${fileRef({ file_id: file.id! })}`;
|
||||
return `- ${file.path} - ${formatBytes({ bytes: Number(file.size) || 0 })}${where} ${fileRef({ file_id: file.id! })}`;
|
||||
});
|
||||
|
||||
const more =
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
import { z } from "zod";
|
||||
import { PDFDocument, StandardFonts, rgb } from "pdf-lib";
|
||||
import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
|
||||
import recordBotFile from "@/src/functions/backend/db/bot-files/record-bot-file";
|
||||
import type { HarnessTool } from "../../types";
|
||||
import {
|
||||
@@ -69,8 +68,6 @@ export default function makePdfTool(context: StorageToolContext): HarnessTool<Ar
|
||||
user_id: context.user_id,
|
||||
});
|
||||
|
||||
await enqueueFileIngest({ file_id: file.id! });
|
||||
|
||||
if (args.attach && file.media_id) {
|
||||
context.onAttach?.({ media_id: Number(file.media_id), name: file.name || "" });
|
||||
}
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
import { z } from "zod";
|
||||
import { AppData } from "@/src/data/app-data";
|
||||
import searchBotFiles from "@/src/functions/backend/db/bot-files/search-bot-files";
|
||||
import type { HarnessTool } from "../../types";
|
||||
import { fileRef, type StorageToolContext } from "./storage-tool-context";
|
||||
|
||||
type Args = {
|
||||
query: string;
|
||||
file_id?: number;
|
||||
limit?: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Searches the bot's saved files by meaning (through the embedding
|
||||
* vectors), falling back to words, and returns the passages with their
|
||||
* file's ref so the bot can read more.
|
||||
*/
|
||||
export default function searchFilesTool(context: StorageToolContext): HarnessTool<Args> {
|
||||
return {
|
||||
name: "search_files",
|
||||
description: `Search the files you have saved (documents the user gave you and what you wrote) and get the passages that answer "query", with their file's ref. Search before saying you don't know something that may be in a file, and before writing a document from memory.`,
|
||||
schema: z.object({
|
||||
query: z.string().min(1).max(500),
|
||||
file_id: z.number().int().positive().optional().describe(`Only this file's passages`),
|
||||
limit: z.number().int().min(1).max(20).optional(),
|
||||
}),
|
||||
run: async ({ args, signal }) => {
|
||||
const result = await searchBotFiles({
|
||||
scopes: context.scopes,
|
||||
query: args.query,
|
||||
file_id: args.file_id,
|
||||
limit: args.limit || AppData.RagTopChunks,
|
||||
signal,
|
||||
bot_id: context.bot_id,
|
||||
workspace_id: context.workspace_id,
|
||||
});
|
||||
|
||||
if (!result.hits.length) return result.note;
|
||||
|
||||
const passages = result.hits.map((hit, index) => {
|
||||
const where = hit.scope == "workspace" ? ` (workspace ${hit.scope})` : "";
|
||||
|
||||
return `## ${index + 1}. ${hit.name}${where} ${fileRef({ file_id: hit.file_id })}\n${hit.text.trim()}`;
|
||||
});
|
||||
|
||||
return `${result.note}\n\n${passages.join("\n\n")}`;
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -1,7 +1,6 @@
|
||||
import { z } from "zod";
|
||||
import { AppData } from "@/src/data/app-data";
|
||||
import type { BotFilePurpose } from "@/src/dict/bot-file-dict";
|
||||
import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
|
||||
import recordBotFile from "@/src/functions/backend/db/bot-files/record-bot-file";
|
||||
import type { HarnessTool } from "../../types";
|
||||
import {
|
||||
@@ -23,7 +22,7 @@ type Args = {
|
||||
/**
|
||||
* Writes a text file (markdown, CSV, JSON, code, plain text) into one of
|
||||
* the bot's directories, making folders as needed. The file stays for
|
||||
* later runs and becomes searchable.
|
||||
* later runs.
|
||||
*/
|
||||
export default function writeFileTool(context: StorageToolContext): HarnessTool<Args> {
|
||||
const own = context.scopes[0];
|
||||
@@ -32,7 +31,7 @@ export default function writeFileTool(context: StorageToolContext): HarnessTool<
|
||||
name: "write_file",
|
||||
description: `Save a text file for later: markdown, CSV, JSON, code or plain text. Give it a path like "reports/q3.md"; folders are made as needed. Files belong to ${
|
||||
own?.scope == "workspace" ? "the workspace you're working in" : "you"
|
||||
} and stay between runs: write a file before telling the user something is saved, and search_files to find one you wrote before. It is readable and searchable at once.${
|
||||
} and stay between runs: write a file before telling the user something is saved, and list_files (by name or path) to find one you wrote before. It is readable at once.${
|
||||
context.scopes.length > 1
|
||||
? ` "scope" picks a directory: ${context.scopes.map((s) => `${s.scope}:${s.owner_id}`).join(", ")}.`
|
||||
: ""
|
||||
@@ -74,8 +73,6 @@ export default function writeFileTool(context: StorageToolContext): HarnessTool<
|
||||
user_id: context.user_id,
|
||||
});
|
||||
|
||||
await enqueueFileIngest({ file_id: file.id! });
|
||||
|
||||
if (args.attach && file.media_id) {
|
||||
context.onAttach?.({ media_id: Number(file.media_id), name: file.name || "" });
|
||||
}
|
||||
|
||||
@@ -29,10 +29,9 @@ type Props = {
|
||||
|
||||
/**
|
||||
* The files a conversation works with: a bot's own folder, or the
|
||||
* workspace's shared one. Each can be opened, read again so it can be
|
||||
* searched, or deleted (a crew manager only). The list loads when the
|
||||
* modal opens, and search goes through the same endpoint, so the page
|
||||
* props stay small.
|
||||
* workspace's shared one. Each can be opened or deleted (a crew manager
|
||||
* only). The list loads when the modal opens, and search (by name or path)
|
||||
* goes through the same endpoint, so the page props stay small.
|
||||
*/
|
||||
export default function BotFilesModal({ bot, workspace, menu, on_open }: Props) {
|
||||
const { pageProps } = useContext(AppContext);
|
||||
@@ -105,7 +104,7 @@ export default function BotFilesModal({ bot, workspace, menu, on_open }: Props)
|
||||
<H2 className="text-lg! leading-6! tracking-normal! font-bold text-ink">{title}</H2>
|
||||
<Span className="text-[13px] leading-[18px] text-ink-secondary">
|
||||
{workspace?.id
|
||||
? `Files every bot here can read and search. They're kept with the app, not on a computer.`
|
||||
? `Files every bot here can find and read. They're kept with the app, not on a computer.`
|
||||
: `What ${name} has saved: documents you sent, files it wrote and images it made. They're kept with the app, not on a computer.${can_manage ? " Deleting one removes it for good." : ""}`}
|
||||
</Span>
|
||||
{usage ? (
|
||||
@@ -128,7 +127,7 @@ export default function BotFilesModal({ bot, workspace, menu, on_open }: Props)
|
||||
|
||||
<Stack className="w-full mt-4 gap-2! items-stretch">
|
||||
<Input
|
||||
placeholder="Search files"
|
||||
placeholder="Search by name or path"
|
||||
value={search}
|
||||
onChange={(event) => setSearch(event.target.value)}
|
||||
onKeyDown={(event) => {
|
||||
@@ -146,14 +145,6 @@ export default function BotFilesModal({ bot, workspace, menu, on_open }: Props)
|
||||
file={file}
|
||||
can_manage={can_manage}
|
||||
busy={busy_id == file.id}
|
||||
onIngest={() =>
|
||||
run({
|
||||
url: `/api/admin/bots/ingest-file`,
|
||||
payload: { file_id: file.id },
|
||||
file_id: file.id,
|
||||
done: `${file.name} will be read again shortly.`,
|
||||
})
|
||||
}
|
||||
onDelete={() =>
|
||||
run({
|
||||
url: `/api/admin/bots/delete-file`,
|
||||
|
||||
@@ -3,7 +3,7 @@ import Row from "@moduletrace/twui/layout/Row";
|
||||
import Span from "@moduletrace/twui/layout/Span";
|
||||
import Stack from "@moduletrace/twui/layout/Stack";
|
||||
import Button from "@moduletrace/twui/layout/Button";
|
||||
import { RefreshCw, Trash2 } from "lucide-react";
|
||||
import { Trash2 } from "lucide-react";
|
||||
import type { MediaType, PublicBotFile } from "@/src/types";
|
||||
import relativeTime from "@/src/utils/relative-time";
|
||||
import AttachmentIcon from "./attachment-icon";
|
||||
@@ -11,19 +11,18 @@ import AttachmentIcon from "./attachment-icon";
|
||||
type Props = {
|
||||
file: PublicBotFile;
|
||||
/**
|
||||
* A crew manager may delete and re-ingest.
|
||||
* A crew manager may delete it.
|
||||
*/
|
||||
can_manage: boolean;
|
||||
busy?: boolean;
|
||||
onIngest: () => void;
|
||||
onDelete: () => void;
|
||||
};
|
||||
|
||||
/**
|
||||
* One file in the files modal: its type, path, size and age, and what can
|
||||
* be done with it. A file that isn't searchable says why.
|
||||
* be done with it.
|
||||
*/
|
||||
export default function FileRow({ file, can_manage, busy, onIngest, onDelete }: Props) {
|
||||
export default function FileRow({ file, can_manage, busy, onDelete }: Props) {
|
||||
return (
|
||||
<Row className="w-full gap-3! flex-nowrap items-start py-2 border-0 border-b border-solid border-hairline last:border-b-0">
|
||||
<Row className="mt-0.5 text-ink-secondary">
|
||||
@@ -44,11 +43,6 @@ export default function FileRow({ file, can_manage, busy, onIngest, onDelete }:
|
||||
file.scope == "workspace" ? "Workspace" : "Bot files",
|
||||
`${Math.max(1, Math.round(file.size / 1024))} KB`,
|
||||
file.created_at ? relativeTime({ at: file.created_at }) : "",
|
||||
file.ingest_error
|
||||
? file.ingest_error
|
||||
: file.ingested_at
|
||||
? "searchable"
|
||||
: "not searchable yet",
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join(" · ")}
|
||||
@@ -56,19 +50,6 @@ export default function FileRow({ file, can_manage, busy, onIngest, onDelete }:
|
||||
</Stack>
|
||||
{can_manage ? (
|
||||
<Row className="shrink-0 gap-0!">
|
||||
<Button
|
||||
title="Read it again so it can be searched"
|
||||
aria-label="Reingest"
|
||||
type="button"
|
||||
variant="ghost"
|
||||
color="gray"
|
||||
size="small"
|
||||
disabled={busy}
|
||||
onClick={onIngest}
|
||||
className="p-1.5!"
|
||||
>
|
||||
<RefreshCw size={14} aria-hidden />
|
||||
</Button>
|
||||
<Button
|
||||
title="Delete it"
|
||||
aria-label="Delete"
|
||||
|
||||
@@ -14,7 +14,7 @@ type Props = {
|
||||
export default function FilesSearchInput({ value, loading, onChange, onSubmit }: Props) {
|
||||
return (
|
||||
<Input
|
||||
placeholder="Search this folder"
|
||||
placeholder="Search this folder by name"
|
||||
value={value}
|
||||
disabled={loading}
|
||||
onChange={(event) => onChange(event.target.value)}
|
||||
|
||||
@@ -4,7 +4,7 @@ import type { DatabaseTableSummary } from "@/src/types";
|
||||
import DatabaseTableRow from "./database-table-row";
|
||||
|
||||
const table: DatabaseTableSummary = {
|
||||
name: "bot_file_vectors",
|
||||
name: "knowledgebase_vectors",
|
||||
type: "virtual",
|
||||
rows: 12,
|
||||
size_bytes: 2048,
|
||||
@@ -16,7 +16,7 @@ describe("DatabaseTableRow", () => {
|
||||
test("names the table, its record count and its size", () => {
|
||||
const html = renderToString(<DatabaseTableRow table={table} selected={false} onOpen={() => {}} />);
|
||||
|
||||
expect(html).toContain("bot_file_vectors");
|
||||
expect(html).toContain("knowledgebase_vectors");
|
||||
expect(html).toContain("12");
|
||||
expect(html).toContain("2.0 KB");
|
||||
});
|
||||
@@ -24,7 +24,7 @@ describe("DatabaseTableRow", () => {
|
||||
test("links to the table's own address", () => {
|
||||
const html = renderToString(<DatabaseTableRow table={table} selected={false} onOpen={() => {}} />);
|
||||
|
||||
expect(html).toContain(`href="/admin/settings/database?table=bot_file_vectors"`);
|
||||
expect(html).toContain(`href="/admin/settings/database?table=knowledgebase_vectors"`);
|
||||
});
|
||||
|
||||
test("a table with no count yet shows a dash", () => {
|
||||
|
||||
@@ -132,8 +132,6 @@ function toPublicFile({ file }: { file: BotFileWithMediaLite }) {
|
||||
purpose: file.purpose || "writing",
|
||||
scope: file.scope || "bot",
|
||||
owner_id: Number(file.owner_id) || 0,
|
||||
ingested_at: file.ingested_at ? Number(file.ingested_at) : undefined,
|
||||
ingest_error: file.ingest_error || undefined,
|
||||
created_at: Number(file.created_at) || 0,
|
||||
url: media_url || "",
|
||||
};
|
||||
@@ -149,8 +147,6 @@ type BotFileWithMediaLite = {
|
||||
purpose?: string;
|
||||
scope?: string;
|
||||
owner_id?: number | "";
|
||||
ingested_at?: number | "";
|
||||
ingest_error?: string;
|
||||
created_at?: number | "";
|
||||
media?: Record<string, unknown> | null;
|
||||
};
|
||||
|
||||
@@ -1,82 +0,0 @@
|
||||
import userAuth from "@/src/functions/backend/auth/user-auth";
|
||||
import grabBotFile from "@/src/functions/backend/db/bot-files/grab-bot-file";
|
||||
import grabUserBot from "@/src/functions/backend/db/bots/grab-user-bot";
|
||||
import grabUserWorkspace from "@/src/functions/backend/db/workspaces/grab-user-workspace";
|
||||
import { IngestBotFileRequestSchema } from "@/src/schemas/bot-file-schema";
|
||||
import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
|
||||
import AppError from "@/src/utils/app-error";
|
||||
import parseWithSchema from "@/src/utils/parse-with-schema";
|
||||
import canManageCrew from "@/src/utils/rbac/can-manage-crew";
|
||||
import isReadOnlyUser from "@/src/utils/rbac/is-read-only-user";
|
||||
import safeErrorMessage from "@/src/utils/safe-error-message";
|
||||
import type { StorageScope } from "@/src/utils/storage-access";
|
||||
import type { APIResponseObject, BunextAPIRouteHandler } from "@moduletrace/bunext/types";
|
||||
|
||||
/**
|
||||
* Queues a conversation's file to be read again and re-embedded, for when
|
||||
* a search misses something the user knows is in it.
|
||||
*/
|
||||
export const handler: BunextAPIRouteHandler<APIResponseObject> = async ({ req, body }) => {
|
||||
try {
|
||||
if (req.method != "POST") throw new AppError(`Method not allowed`);
|
||||
|
||||
const { user, user_types } = await userAuth({ req });
|
||||
|
||||
if (!user?.id) return { success: false, msg: `Unauthorized`, logoutUser: true };
|
||||
if (isReadOnlyUser({ user_types: user_types || [] })) {
|
||||
throw new AppError(`This user can only read records.`);
|
||||
}
|
||||
if (!canManageCrew({ user_types: user_types || [] })) {
|
||||
throw new AppError(`Only a crew manager can re-read files.`);
|
||||
}
|
||||
|
||||
const { file_id, read_media } = parseWithSchema({
|
||||
schema: IngestBotFileRequestSchema,
|
||||
data: body,
|
||||
});
|
||||
|
||||
const scopes = await reachableScopes({ user_id: user.id, file_id });
|
||||
|
||||
const file = await grabBotFile({ scopes, file_id });
|
||||
|
||||
if (!file?.id) throw new AppError(`That file isn't here.`);
|
||||
|
||||
await enqueueFileIngest({ file_id: file.id });
|
||||
|
||||
return { success: true, msg: `${file.name || "The file"} will be read again shortly.` };
|
||||
} catch (error: unknown) {
|
||||
return { success: false, msg: safeErrorMessage({ error }) };
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* The directories the user reaches for this file.
|
||||
*/
|
||||
async function reachableScopes({
|
||||
user_id,
|
||||
file_id,
|
||||
}: {
|
||||
user_id: number;
|
||||
file_id: number;
|
||||
}): Promise<StorageScope[]> {
|
||||
const { default: BunSQLite } = await import("@moduletrace/bun-sqlite");
|
||||
|
||||
const res = await BunSQLite.sql<{ scope: string; owner_id: number }>({
|
||||
sql: `SELECT scope, owner_id FROM bot_files WHERE id = ? LIMIT 1`,
|
||||
values: [file_id],
|
||||
});
|
||||
|
||||
const file = res.payload?.[0];
|
||||
|
||||
if (!file?.owner_id) return [];
|
||||
|
||||
if (file.scope == "workspace") {
|
||||
const workspace = await grabUserWorkspace({ workspace_id: file.owner_id, user_id });
|
||||
|
||||
return workspace?.id ? [{ scope: "workspace", owner_id: Number(file.owner_id) }] : [];
|
||||
}
|
||||
|
||||
const bot = await grabUserBot({ bot_id: file.owner_id, user_id });
|
||||
|
||||
return bot?.id ? [{ scope: "bot", owner_id: Number(file.owner_id) }] : [];
|
||||
}
|
||||
@@ -8,7 +8,6 @@ import type { BotMessageAttachment } from "@/src/types";
|
||||
import AppError from "@/src/utils/app-error";
|
||||
import attachmentFileKind from "@/src/utils/attachment-file-kind";
|
||||
import bufferFromBase64 from "@/src/utils/buffer-from-base-64";
|
||||
import canExtractText from "@/src/utils/can-extract-text";
|
||||
import parseWithSchema from "@/src/utils/parse-with-schema";
|
||||
import isReadOnlyUser from "@/src/utils/rbac/is-read-only-user";
|
||||
import safeErrorMessage from "@/src/utils/safe-error-message";
|
||||
@@ -19,8 +18,8 @@ import type {
|
||||
|
||||
/**
|
||||
* Stores one file for a chat message, private to the session user, before
|
||||
* the message is sent with its id. Images are resized; documents have
|
||||
* their text pulled out now, so bots can read them later.
|
||||
* the message is sent with its id. Images are resized; a document keeps
|
||||
* no extracted text (a reply reads it from the file).
|
||||
*/
|
||||
export const handler: BunextAPIRouteHandler<
|
||||
APIResponseObject<BotMessageAttachment>
|
||||
@@ -65,7 +64,6 @@ export const handler: BunextAPIRouteHandler<
|
||||
media_mime_type: ext,
|
||||
media_paradigms: ["bot-attachment"],
|
||||
media: { title: name },
|
||||
extract_media_text_content: canExtractText({ name, mime_type: mime }),
|
||||
user: { id: Number(user.id) },
|
||||
}).catch(async (error) => {
|
||||
await logError({ title: `Attachment ${name} couldn't be saved`, error }).catch(() => {});
|
||||
|
||||
@@ -9,7 +9,6 @@ import type { TaskDetail } from "@/src/types";
|
||||
import AppError from "@/src/utils/app-error";
|
||||
import attachmentFileKind from "@/src/utils/attachment-file-kind";
|
||||
import bufferFromBase64 from "@/src/utils/buffer-from-base-64";
|
||||
import canExtractText from "@/src/utils/can-extract-text";
|
||||
import parseWithSchema from "@/src/utils/parse-with-schema";
|
||||
import isReadOnlyUser from "@/src/utils/rbac/is-read-only-user";
|
||||
import safeErrorMessage from "@/src/utils/safe-error-message";
|
||||
@@ -66,7 +65,6 @@ export const handler: BunextAPIRouteHandler<APIResponseObject<TaskDetail>> = asy
|
||||
media_mime_type: ext,
|
||||
media_paradigms: ["task-media"],
|
||||
media: { title: name },
|
||||
extract_media_text_content: canExtractText({ name, mime_type: mime }),
|
||||
user: { id: Number(user.id) },
|
||||
});
|
||||
|
||||
|
||||
@@ -44,7 +44,6 @@ export const handler: BunextAPIRouteHandler<APIResponseObject> = async (
|
||||
is_media_primary,
|
||||
update_on_duplicate,
|
||||
reauth,
|
||||
return_media_text_content,
|
||||
media,
|
||||
media_name,
|
||||
} = body;
|
||||
@@ -100,7 +99,6 @@ export const handler: BunextAPIRouteHandler<APIResponseObject> = async (
|
||||
media_type,
|
||||
is_primary: is_media_primary,
|
||||
update_on_duplicate,
|
||||
extract_media_text_content: return_media_text_content,
|
||||
media,
|
||||
media_mime_type: ext,
|
||||
user: { id: target_user_id },
|
||||
|
||||
@@ -7,14 +7,6 @@ export const DeleteBotFileRequestSchema = z.object({
|
||||
file_id: z.coerce.number().int().positive(),
|
||||
});
|
||||
|
||||
/**
|
||||
* Read one file again so it can be searched.
|
||||
*/
|
||||
export const IngestBotFileRequestSchema = z.object({
|
||||
file_id: z.coerce.number().int().positive(),
|
||||
read_media: z.boolean().optional(),
|
||||
});
|
||||
|
||||
/**
|
||||
* The General settings' storage quota, in MB.
|
||||
*/
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
import { AppData } from "@/src/data/app-data";
|
||||
import pruneMediaTextContent from "@/src/functions/backend/db/media/prune-media-text-content";
|
||||
|
||||
/**
|
||||
* Cleans the `media.text_content` that should never have been stored.
|
||||
*
|
||||
* Before the extractor had an allowlist, every non-image/audio/video file
|
||||
* was decoded as UTF-8 and its bytes were written to `text_content`: an
|
||||
* SVG, a zip, a spreadsheet and an unknown binary all left raw (garbled)
|
||||
* content in the database. Nothing capped the length either, so a large
|
||||
* document could leave megabytes of text behind.
|
||||
*
|
||||
* Safe to rerun: it only writes a row whose text has to change.
|
||||
*
|
||||
* `bun src/scripts/prune-media-text-content.ts`
|
||||
*/
|
||||
async function main() {
|
||||
const result = await pruneMediaTextContent();
|
||||
|
||||
for (const change of result.changes) {
|
||||
console.log(
|
||||
change.after_chars === 0
|
||||
? `Media ${change.media_id} (${change.name}): dropped ${change.before_chars} characters.`
|
||||
: `Media ${change.media_id} (${change.name}): cut ${change.before_chars} characters to ${change.after_chars}.`,
|
||||
);
|
||||
}
|
||||
|
||||
console.log(
|
||||
`Done: ${result.cleared} row(s) cleared, ${result.truncated} truncated, about ${Math.round(result.reclaimed_bytes / 1024)} KB reclaimed from ${result.checked} row(s) with text.`,
|
||||
);
|
||||
|
||||
if (result.reclaimed_bytes > 0) {
|
||||
console.log(
|
||||
`The freed space stays in the database file until it is vacuumed: run "bunx bun-sqlite vacuum" (or restore from a fresh backup) to reclaim it on disk. Kept text is capped at AppData.BotAttachmentTextStoreMaxChars (${AppData.BotAttachmentTextStoreMaxChars}).`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
await main();
|
||||
process.exit(0);
|
||||
@@ -11,6 +11,7 @@ import liveRelay from "./live-relay";
|
||||
import enableForeignKeys from "./functions/backend/db/enable-foreign-keys";
|
||||
import migrateBotTypes from "./functions/backend/db/bot-types/migrate-bot-types";
|
||||
import clearImplicitSessionLinks from "./functions/backend/db/bot-sessions/clear-implicit-session-links";
|
||||
import dropStoredFileText from "./functions/backend/db/stored-text/drop-stored-file-text";
|
||||
import syncKnowledgebaseFts from "./functions/backend/db/knowledgebases/fts/sync-knowledgebase-fts";
|
||||
import handleComputerProxy from "./functions/server/computers/handle-computer-proxy";
|
||||
import handleGraphicsRequest from "./functions/server/graphics/handle-graphics-request";
|
||||
@@ -40,6 +41,9 @@ await migrateBotTypes().catch((error) =>
|
||||
await clearImplicitSessionLinks().catch((error) =>
|
||||
logError({ title: "Could not clear implicit session links", error }).catch(() => {}),
|
||||
);
|
||||
await dropStoredFileText().catch((error) =>
|
||||
logError({ title: "Could not drop the stored file text", error }).catch(() => {}),
|
||||
);
|
||||
await syncKnowledgebaseFts().catch((error) =>
|
||||
logError({ title: "Could not sync the knowledgebase search index", error }).catch(() => {}),
|
||||
);
|
||||
|
||||
@@ -1277,8 +1277,6 @@ export type PublicBotFile = {
|
||||
purpose: string;
|
||||
scope: string;
|
||||
owner_id: number;
|
||||
ingested_at?: number;
|
||||
ingest_error?: string;
|
||||
created_at: number;
|
||||
/**
|
||||
* Where the bytes are served (`/media/<media id>`, private).
|
||||
@@ -2357,7 +2355,6 @@ export type ApiReqParams<
|
||||
search_term_user_type?: string | string[];
|
||||
|
||||
text?: string;
|
||||
return_media_text_content?: boolean;
|
||||
media?: BUN_SQLITE_COBBLE_MEDIA;
|
||||
backup_name?: string;
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user