Stop storing raw file bytes in media.text_content
`text_content` is an extracted-text cache, but nothing stopped it from
holding a dump of bytes:
- `fileToText` ended its switch with `default: return buffer.toString("utf-8")`
and defaulted a missing extension to "txt", so any binary was decoded.
- Both extraction callers gated on `media_type == "document"`, and
`attachmentFileKind` classes everything non-image/audio/video as a
document: an SVG, a zip, a spreadsheet, a .bin.
- Nothing capped the length, so a 20 MB attachment could leave 20 MB of
text in the database.
- `storeChatAttachments` copied the text onto a second `media` row, and
`SELECT m.*` pulled the column into page props and API responses.
On a real install an SVG held 6852 characters of markup and a
`.designcraft` file held 142 KB of raw bytes.
Changes:
- `file-to-text.ts` lists the text extensions explicitly and returns
undefined for anything else; no extension is no text.
- `can-extract-text.ts` is the one allowlist (reusing
`BotTextFileExtensions`), used by both upload routes and by
`uploadAndRecordMedia`; `extension-of.ts` is shared with the storage
catalog instead of duplicated.
- `BotAttachmentTextStoreMaxChars` (200k) caps the column via
`cap-text-content.ts`.
- `store-chat-attachments.ts` no longer copies the text; ingestion
re-reads the bytes once.
- `media-columns.ts` selects every media column but `text_content`, which
now has to be asked for (`grab-messages-media`'s reply path and
ingestion do); the media CRUD routes sanitize rows they write.
- `ingest-bot-file.ts` decides on size alone, since a cached slice made
an oversized file look already read.
- `bun run media:prune-text` cleans rows that were already stored
(rerunnable, with a VACUUM hint).
Verified on the dev database: both bad rows cleared (146 KB), rerun
reports nothing to do.
This commit is contained in:
@@ -0,0 +1,40 @@
|
||||
import { AppData } from "@/src/data/app-data";
|
||||
import pruneMediaTextContent from "@/src/functions/backend/db/media/prune-media-text-content";
|
||||
|
||||
/**
|
||||
* Cleans the `media.text_content` that should never have been stored.
|
||||
*
|
||||
* Before the extractor had an allowlist, every non-image/audio/video file
|
||||
* was decoded as UTF-8 and its bytes were written to `text_content`: an
|
||||
* SVG, a zip, a spreadsheet and an unknown binary all left raw (garbled)
|
||||
* content in the database. Nothing capped the length either, so a large
|
||||
* document could leave megabytes of text behind.
|
||||
*
|
||||
* Safe to rerun: it only writes a row whose text has to change.
|
||||
*
|
||||
* `bun src/scripts/prune-media-text-content.ts`
|
||||
*/
|
||||
async function main() {
|
||||
const result = await pruneMediaTextContent();
|
||||
|
||||
for (const change of result.changes) {
|
||||
console.log(
|
||||
change.after_chars === 0
|
||||
? `Media ${change.media_id} (${change.name}): dropped ${change.before_chars} characters.`
|
||||
: `Media ${change.media_id} (${change.name}): cut ${change.before_chars} characters to ${change.after_chars}.`,
|
||||
);
|
||||
}
|
||||
|
||||
console.log(
|
||||
`Done: ${result.cleared} row(s) cleared, ${result.truncated} truncated, about ${Math.round(result.reclaimed_bytes / 1024)} KB reclaimed from ${result.checked} row(s) with text.`,
|
||||
);
|
||||
|
||||
if (result.reclaimed_bytes > 0) {
|
||||
console.log(
|
||||
`The freed space stays in the database file until it is vacuumed: run "bunx bun-sqlite vacuum" (or restore from a fresh backup) to reclaim it on disk. Kept text is capped at AppData.BotAttachmentTextStoreMaxChars (${AppData.BotAttachmentTextStoreMaxChars}).`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
await main();
|
||||
process.exit(0);
|
||||
Reference in New Issue
Block a user