tben
d3357cb7df
`text_content` is an extracted-text cache, but nothing stopped it from
holding a dump of bytes:
- `fileToText` ended its switch with `default: return buffer.toString("utf-8")`
and defaulted a missing extension to "txt", so any binary was decoded.
- Both extraction callers gated on `media_type == "document"`, and
`attachmentFileKind` classes everything non-image/audio/video as a
document: an SVG, a zip, a spreadsheet, a .bin.
- Nothing capped the length, so a 20 MB attachment could leave 20 MB of
text in the database.
- `storeChatAttachments` copied the text onto a second `media` row, and
`SELECT m.*` pulled the column into page props and API responses.
On a real install an SVG held 6852 characters of markup and a
`.designcraft` file held 142 KB of raw bytes.
Changes:
- `file-to-text.ts` lists the text extensions explicitly and returns
undefined for anything else; no extension is no text.
- `can-extract-text.ts` is the one allowlist (reusing
`BotTextFileExtensions`), used by both upload routes and by
`uploadAndRecordMedia`; `extension-of.ts` is shared with the storage
catalog instead of duplicated.
- `BotAttachmentTextStoreMaxChars` (200k) caps the column via
`cap-text-content.ts`.
- `store-chat-attachments.ts` no longer copies the text; ingestion
re-reads the bytes once.
- `media-columns.ts` selects every media column but `text_content`, which
now has to be asked for (`grab-messages-media`'s reply path and
ingestion do); the media CRUD routes sanitize rows they write.
- `ingest-bot-file.ts` decides on size alone, since a cached slice made
an oversized file look already read.
- `bun run media:prune-text` cleans rows that were already stored
(rerunnable, with a VACUUM hint).
Verified on the dev database: both bad rows cleared (146 KB), rerun
reports nothing to do.
41 lines
1.2 KiB
TypeScript
41 lines
1.2 KiB
TypeScript
/**
|
|
* The `media` columns a normal read takes, with a table alias.
|
|
*
|
|
* `text_content` is deliberately left out: it holds up to
|
|
* `AppData.BotAttachmentTextStoreMaxChars` of extracted document text, and
|
|
* every caller that only needs a file's name, type or path (`SELECT m.*`
|
|
* did) pulled it into memory, into page props and into API responses.
|
|
* Read it explicitly where it is really used:
|
|
* `src/bots-queue/load-attachment-parts.ts` and
|
|
* `src/functions/backend/db/bot-files/ingest-bot-file.ts`.
|
|
*/
|
|
export default function mediaColumns({
|
|
alias = "m",
|
|
with_text_content,
|
|
}: { alias?: string; with_text_content?: boolean } = {}) {
|
|
const columns = [
|
|
"id",
|
|
"created_at",
|
|
"updated_at",
|
|
"user_id",
|
|
"title",
|
|
"description",
|
|
"summary",
|
|
"media_type",
|
|
"media_write_path",
|
|
"media_thumbnail_write_path",
|
|
"mime_type",
|
|
"is_primary",
|
|
"is_rag_ingested",
|
|
"archived",
|
|
];
|
|
|
|
/**
|
|
* Only the two paths that really use it ask for it: the reply that
|
|
* hands a document to a model, and ingestion.
|
|
*/
|
|
if (with_text_content) columns.push("text_content");
|
|
|
|
return columns.map((column) => `${alias}.${column}`).join(", ");
|
|
}
|