Stop storing raw file bytes in media.text_content

`text_content` is an extracted-text cache, but nothing stopped it from
holding a dump of bytes:

- `fileToText` ended its switch with `default: return buffer.toString("utf-8")`
  and defaulted a missing extension to "txt", so any binary was decoded.
- Both extraction callers gated on `media_type == "document"`, and
  `attachmentFileKind` classes everything non-image/audio/video as a
  document: an SVG, a zip, a spreadsheet, a .bin.
- Nothing capped the length, so a 20 MB attachment could leave 20 MB of
  text in the database.
- `storeChatAttachments` copied the text onto a second `media` row, and
  `SELECT m.*` pulled the column into page props and API responses.

On a real install an SVG held 6852 characters of markup and a
`.designcraft` file held 142 KB of raw bytes.

Changes:

- `file-to-text.ts` lists the text extensions explicitly and returns
  undefined for anything else; no extension is no text.
- `can-extract-text.ts` is the one allowlist (reusing
  `BotTextFileExtensions`), used by both upload routes and by
  `uploadAndRecordMedia`; `extension-of.ts` is shared with the storage
  catalog instead of duplicated.
- `BotAttachmentTextStoreMaxChars` (200k) caps the column via
  `cap-text-content.ts`.
- `store-chat-attachments.ts` no longer copies the text; ingestion
  re-reads the bytes once.
- `media-columns.ts` selects every media column but `text_content`, which
  now has to be asked for (`grab-messages-media`'s reply path and
  ingestion do); the media CRUD routes sanitize rows they write.
- `ingest-bot-file.ts` decides on size alone, since a cached slice made
  an oversized file look already read.
- `bun run media:prune-text` cleans rows that were already stored
  (rerunnable, with a VACUUM hint).

Verified on the dev database: both bad rows cleared (146 KB), rerun
reports nothing to do.
This commit is contained in:
2026-10-23 05:22:04 +01:00
parent 5475241371
commit d3357cb7df
36 changed files with 1034 additions and 43 deletions
+40
View File
@@ -0,0 +1,40 @@
import { describe, expect, test } from "bun:test";
import { AppData } from "@/src/data/app-data";
import decideTextContent from "./decide-text-content";
describe("decideTextContent", () => {
test("keeps the text of a document the extractor reads", () => {
expect(decideTextContent({ name: "invoice.pdf", text: "Total: 12" })).toBe("Total: 12");
expect(decideTextContent({ name: "notes.md", text: "# Hi" })).toBe("# Hi");
expect(decideTextContent({ name: "rows.csv", text: "a,b" })).toBe("a,b");
});
test("drops the raw bytes of anything else", () => {
expect(decideTextContent({ name: "icon.svg", text: "<svg>...</svg>" })).toBeNull();
expect(decideTextContent({ name: "bundle.zip", text: "PK\u0003\u0004binary" })).toBeNull();
expect(decideTextContent({ name: "sheet.xlsx", text: "PK\u0003\u0004binary" })).toBeNull();
expect(decideTextContent({ name: "app.bin", text: "\u0000\u0001\u0002" })).toBeNull();
});
test("nothing to store is null, not an empty string", () => {
expect(decideTextContent({ name: "invoice.pdf", text: "" })).toBeNull();
expect(decideTextContent({ name: "invoice.pdf", text: " \n " })).toBeNull();
expect(decideTextContent({ name: "invoice.pdf" })).toBeNull();
});
test("cuts a document over the ceiling", () => {
const text = "a".repeat(AppData.BotAttachmentTextStoreMaxChars + 10);
const decided = decideTextContent({ name: "long.txt", text });
expect(decided).toContain("Truncated");
expect(decided!.length).toBeLessThanOrEqual(AppData.BotAttachmentTextStoreMaxChars);
});
test("a document over the ceiling is still cut when the overflow is small", () => {
const text = "a".repeat(AppData.BotAttachmentTextStoreMaxChars + 10);
const decided = decideTextContent({ name: "long.txt", text })!;
expect(decided).toContain("Truncated");
expect(decided.length).toBeLessThanOrEqual(AppData.BotAttachmentTextStoreMaxChars);
});
});