Moduletrace/cobble
2
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases 28 Wiki Activity

Keep searchable text only in knowledgebases

Browse Source 
Every bot file was chunked into bot_file_chunks and embedded into
bot_file_vectors, and uploads kept their extracted text in
media.text_content, which together took most of the database. Bot
files are no longer indexed: bots find them by name or path with
list_files and read them with read_file, and search_files, ingest_file
and the files modal's re-read button are gone. Attachments keep no
text; a reply reads a document from the file. Only knowledgebase
documents keep text.

The schema drops both tables and the ingest columns. At boot,
dropStoredFileText removes what older versions stored (file jobs,
upload text) and vacuums once much of the database is free. The
media:prune-text script it replaces is removed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
tben
2026-10-28 04:44:46 +01:00
co-authored by Claude Opus 5.5
parent 2896e76256
commit ac3ebc43d4
55 changed files with 271 additions and 1712 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/harness/tools/computer/import-file-tool.ts
+2 -7
View File
@@ -3,7 +3,6 @@ import { tmpdir } from "node:os";
import path from "node:path";
import { z } from "zod";
import { AppData } from "@/src/data/app-data";
import enqueueFileIngest from "@/src/file-ingest/enqueue-file-ingest";
import recordBotFile, { grabBotFileRow } from "@/src/functions/backend/db/bot-files/record-bot-file";
import grabComputerMcpUrl from "@/src/functions/backend/db/computers/grab-computer-mcp-url";
import computerNotReadyMessage from "@/src/utils/computer-not-ready-message";
@@ -49,7 +48,7 @@ type Args = {
* Copies what a bot made on its computer into its own files: one file, or
* a whole folder (taken as a gzipped tar and unpacked into one row per
* file). The copies are kept with the app, stay after the computer is
* rebuilt, and are searchable at once; the originals stay on the computer,
* rebuilt, and are readable at once; the originals stay on the computer,
* where the message's folder ref points at the folder holding them.
*
* The bytes come back over the sandbox's streaming download endpoint
@@ -72,7 +71,7 @@ export default function importFileTool({
name: "import_file",
description: `Copy a file or folder from this computer into your own files, so it is kept with the app and stays after the computer is rebuilt. Give its path on the computer, relative to your folder there (${dirs.cwd}), e.g. "seo/coderank/raw/crawl.json" or "seo/coderank/raw"; a folder comes in as it is, in one go. Up to ${max_mb} MB. Check with list_files and the computer's own tools what you already imported: importing the same thing twice just writes over the copy, so do it once.${
own?.scope == "workspace" ? " The copies land in the workspace's shared files." : ""
} Imports are searchable at once. Name a copy with its [file:ID] ref. Import the report you wrote before the last long steps of a run, so it is never lost to the step limit.`,
} Imports are readable at once. Name a copy with its [file:ID] ref. Import the report you wrote before the last long steps of a run, so it is never lost to the step limit.`,
schema: z.object({
path: z
.string()
@@ -190,8 +189,6 @@ async function importOneFile({
user_id: target.user_id,
});
if (saved.id) await enqueueFileIngest({ file_id: saved.id });
if (attach && saved.media_id) {
onAttach?.({ media_id: Number(saved.media_id), name: saved.name || name });
}
@@ -427,8 +424,6 @@ async function unpackArchive({
if (!row?.id) continue;
await enqueueFileIngest({ file_id: row.id }).catch(() => {});
saved.push({ id: Number(row.id), path: row.path || `${folder}/${relative}` });
}
} finally {