Stop storing raw file bytes in media.text_content

`text_content` is an extracted-text cache, but nothing stopped it from
holding a dump of bytes:

- `fileToText` ended its switch with `default: return buffer.toString("utf-8")`
  and defaulted a missing extension to "txt", so any binary was decoded.
- Both extraction callers gated on `media_type == "document"`, and
  `attachmentFileKind` classes everything non-image/audio/video as a
  document: an SVG, a zip, a spreadsheet, a .bin.
- Nothing capped the length, so a 20 MB attachment could leave 20 MB of
  text in the database.
- `storeChatAttachments` copied the text onto a second `media` row, and
  `SELECT m.*` pulled the column into page props and API responses.

On a real install an SVG held 6852 characters of markup and a
`.designcraft` file held 142 KB of raw bytes.

Changes:

- `file-to-text.ts` lists the text extensions explicitly and returns
  undefined for anything else; no extension is no text.
- `can-extract-text.ts` is the one allowlist (reusing
  `BotTextFileExtensions`), used by both upload routes and by
  `uploadAndRecordMedia`; `extension-of.ts` is shared with the storage
  catalog instead of duplicated.
- `BotAttachmentTextStoreMaxChars` (200k) caps the column via
  `cap-text-content.ts`.
- `store-chat-attachments.ts` no longer copies the text; ingestion
  re-reads the bytes once.
- `media-columns.ts` selects every media column but `text_content`, which
  now has to be asked for (`grab-messages-media`'s reply path and
  ingestion do); the media CRUD routes sanitize rows they write.
- `ingest-bot-file.ts` decides on size alone, since a cached slice made
  an oversized file look already read.
- `bun run media:prune-text` cleans rows that were already stored
  (rerunnable, with a VACUUM hint).

Verified on the dev database: both bad rows cleared (146 KB), rerun
reports nothing to do.
This commit is contained in:
2026-10-23 05:22:04 +01:00
parent 5475241371
commit d3357cb7df
36 changed files with 1034 additions and 43 deletions
@@ -0,0 +1,72 @@
import { describe, expect, test } from "bun:test";
import { Database } from "bun:sqlite";
const db = new Database(":memory:");
db.run(
`CREATE TABLE media (
id INTEGER PRIMARY KEY,
created_at INTEGER,
updated_at INTEGER,
user_id INTEGER,
title TEXT,
description TEXT,
summary TEXT,
media_type TEXT,
media_write_path TEXT,
media_thumbnail_write_path TEXT,
mime_type TEXT,
text_content TEXT,
is_primary INTEGER,
is_rag_ingested INTEGER,
archived INTEGER
)`,
);
db.run(`CREATE TABLE bot_message_media (id INTEGER PRIMARY KEY, message_id INTEGER, media_id INTEGER)`);
db.run(
`INSERT INTO media (id, title, media_type, media_write_path, mime_type, text_content)
VALUES (1, 'invoice.pdf', 'document', 'media/1.pdf', 'application/pdf', 'Total: 12')`,
);
db.run(`INSERT INTO bot_message_media (id, message_id, media_id) VALUES (1, 10, 1)`);
/**
* The query `grabMessagesMedia` builds, with the text column included or
* left out. It is the largest column in the table, so only the reply path
* that hands a document to a model asks for it.
*/
function read({ with_text_content }: { with_text_content: boolean }) {
const columns = [
"m.id",
"m.title",
"m.media_type",
"m.media_write_path",
"m.mime_type",
...(with_text_content ? ["m.text_content"] : []),
].join(", ");
return db
.query(
`SELECT ${columns}, l.message_id FROM bot_message_media l
JOIN media m ON m.id = l.media_id
WHERE l.message_id = 10`,
)
.get() as Record<string, unknown>;
}
describe("grabMessagesMedia", () => {
test("a plain read never drags the extracted text along", () => {
expect(read({ with_text_content: false })).not.toHaveProperty("text_content");
});
test("the reply path can ask for it", () => {
expect(read({ with_text_content: true }).text_content).toBe("Total: 12");
});
test("the row's name and path are always there", () => {
expect(read({ with_text_content: false })).toMatchObject({
title: "invoice.pdf",
media_write_path: "media/1.pdf",
message_id: 10,
});
});
});