Every bot file was chunked into bot_file_chunks and embedded into
bot_file_vectors, and uploads kept their extracted text in
media.text_content, which together took most of the database. Bot
files are no longer indexed: bots find them by name or path with
list_files and read them with read_file, and search_files, ingest_file
and the files modal's re-read button are gone. Attachments keep no
text; a reply reads a document from the file. Only knowledgebase
documents keep text.
The schema drops both tables and the ingest columns. At boot,
dropStoredFileText removes what older versions stored (file jobs,
upload text) and vacuums once much of the database is free. The
media:prune-text script it replaces is removed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
- Media rows can no longer be created through the CRUD API, updates only
take description fields on the caller's own row, and every media read
or delete goes through mediaFilePath, which refuses paths outside
DATA_DIR or inside the database folder.
- Uploads strip the file name and extension to safe characters and keep
only title, description and summary from the request's media object.
- Folder imports no longer follow symlinks (walkRegularFiles), and the
source path is quoted in the computer's shell.
- User and bot files are served with nosniff, and HTML, SVG, XML and
scripts are sandboxed by CSP.
- Sandbox cookies are kept to the computer's prefix, and websocket
upgrades on /ws and the computer proxy must come from the app's own
address (README and Help note nginx's Host header).
- user_types CRUD writes go through the assignment matrix; members only
read their own user record; read-only users can read CRUD again;
grabUsers builds valid SQL for array search terms and bad pages; an
admin media DELETE needs an id; read-only users can't delete media.
- Rate limits and the audit log use the connection's address, trusting
forwarded headers only from a proxy, and login limits are split per
address, per account from that address and per account overall.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
`text_content` is an extracted-text cache, but nothing stopped it from
holding a dump of bytes:
- `fileToText` ended its switch with `default: return buffer.toString("utf-8")`
and defaulted a missing extension to "txt", so any binary was decoded.
- Both extraction callers gated on `media_type == "document"`, and
`attachmentFileKind` classes everything non-image/audio/video as a
document: an SVG, a zip, a spreadsheet, a .bin.
- Nothing capped the length, so a 20 MB attachment could leave 20 MB of
text in the database.
- `storeChatAttachments` copied the text onto a second `media` row, and
`SELECT m.*` pulled the column into page props and API responses.
On a real install an SVG held 6852 characters of markup and a
`.designcraft` file held 142 KB of raw bytes.
Changes:
- `file-to-text.ts` lists the text extensions explicitly and returns
undefined for anything else; no extension is no text.
- `can-extract-text.ts` is the one allowlist (reusing
`BotTextFileExtensions`), used by both upload routes and by
`uploadAndRecordMedia`; `extension-of.ts` is shared with the storage
catalog instead of duplicated.
- `BotAttachmentTextStoreMaxChars` (200k) caps the column via
`cap-text-content.ts`.
- `store-chat-attachments.ts` no longer copies the text; ingestion
re-reads the bytes once.
- `media-columns.ts` selects every media column but `text_content`, which
now has to be asked for (`grab-messages-media`'s reply path and
ingestion do); the media CRUD routes sanitize rows they write.
- `ingest-bot-file.ts` decides on size alone, since a cached slice made
an oversized file look already read.
- `bun run media:prune-text` cleans rows that were already stored
(rerunnable, with a VACUUM hint).
Verified on the dev database: both bad rows cleared (146 KB), rerun
reports nothing to do.
tbenandClaude Opus 5.5