tben
d3357cb7df
`text_content` is an extracted-text cache, but nothing stopped it from
holding a dump of bytes:
- `fileToText` ended its switch with `default: return buffer.toString("utf-8")`
and defaulted a missing extension to "txt", so any binary was decoded.
- Both extraction callers gated on `media_type == "document"`, and
`attachmentFileKind` classes everything non-image/audio/video as a
document: an SVG, a zip, a spreadsheet, a .bin.
- Nothing capped the length, so a 20 MB attachment could leave 20 MB of
text in the database.
- `storeChatAttachments` copied the text onto a second `media` row, and
`SELECT m.*` pulled the column into page props and API responses.
On a real install an SVG held 6852 characters of markup and a
`.designcraft` file held 142 KB of raw bytes.
Changes:
- `file-to-text.ts` lists the text extensions explicitly and returns
undefined for anything else; no extension is no text.
- `can-extract-text.ts` is the one allowlist (reusing
`BotTextFileExtensions`), used by both upload routes and by
`uploadAndRecordMedia`; `extension-of.ts` is shared with the storage
catalog instead of duplicated.
- `BotAttachmentTextStoreMaxChars` (200k) caps the column via
`cap-text-content.ts`.
- `store-chat-attachments.ts` no longer copies the text; ingestion
re-reads the bytes once.
- `media-columns.ts` selects every media column but `text_content`, which
now has to be asked for (`grab-messages-media`'s reply path and
ingestion do); the media CRUD routes sanitize rows they write.
- `ingest-bot-file.ts` decides on size alone, since a cached slice made
an oversized file look already read.
- `bun run media:prune-text` cleans rows that were already stored
(rerunnable, with a VACUUM hint).
Verified on the dev database: both bad rows cleared (146 KB), rerun
reports nothing to do.
50 lines
1.6 KiB
TypeScript
50 lines
1.6 KiB
TypeScript
import type { BUN_SQLITE_COBBLE_MEDIA } from "@/db/types/db";
|
|
import capTextContent from "@/src/utils/cap-text-content";
|
|
import decideTextContent from "@/src/utils/decide-text-content";
|
|
|
|
type Params = {
|
|
/**
|
|
* The row as the request sent it.
|
|
*/
|
|
row: Partial<BUN_SQLITE_COBBLE_MEDIA>;
|
|
/**
|
|
* The row already in the database, for an update: its name and type
|
|
* decide whether the new text may be kept when the update doesn't
|
|
* repeat them.
|
|
*/
|
|
existing?: Partial<BUN_SQLITE_COBBLE_MEDIA> | null;
|
|
};
|
|
|
|
/**
|
|
* A `media` row whose `text_content` is what the column may hold.
|
|
*
|
|
* The upload path decides this itself, but the CRUD API writes rows
|
|
* straight through, so a typed-in `text_content` could store raw bytes or
|
|
* megabytes of text. An update that carries no text is left alone.
|
|
*/
|
|
export default function sanitizeMediaRowTextContent({
|
|
row,
|
|
existing,
|
|
}: Params): Partial<BUN_SQLITE_COBBLE_MEDIA> {
|
|
const text = row.text_content;
|
|
|
|
if (text === undefined) return row;
|
|
|
|
const name = row.title || existing?.title || null;
|
|
const mime_type = row.mime_type || existing?.mime_type || null;
|
|
|
|
/**
|
|
* With no name and no type there is nothing to judge it by, so it is
|
|
* only cut: dropping text that a rename would have kept would be
|
|
* worse than a slightly generous cache.
|
|
*/
|
|
const text_content =
|
|
name || mime_type
|
|
? decideTextContent({ name, mime_type, text })
|
|
: text
|
|
? capTextContent({ text })
|
|
: null;
|
|
|
|
return { ...row, text_content: text_content ?? undefined };
|
|
}
|