Stream computer files instead of reading them through the shell

import_file sent a computer's bytes back with base64 over /v1/shell/exec,
whose stdout is capped at 30,047 characters. Every import was therefore
cut at 22,528 bytes and saved anyway, with the tool reporting success: a
1.12 MiB archive became a 22 KB file that would not open, and a crawler's
crawl.json lost half its pages.

The bytes now come from the sandbox's streaming GET /v1/file/download.
The size is read first so nothing over BotFileImportMaxBytes is
transferred, the download asks for change_policy=abort so a file that
changes mid-send is refused rather than half-saved, and the received
length is checked against the size read, so a cut-short transfer fails
loudly. Folders go the same way, as a gzipped tar unpacked into one row
per file.

Bots also name a path as [path:volume:owner:path] now, so what a bot
leaves on a computer is a link into the files browser instead of text.
This commit is contained in:
2026-10-22 07:37:19 +01:00
parent 956942bba9
commit f1b7ba535f
7 changed files with 658 additions and 6 deletions
+11
View File
@@ -417,6 +417,17 @@ export const AppData = {
* The files modal and the bot's file list: one page of them.
*/
BotFilesPageSize: 50,
/**
* The files browser at `/admin/files`: how many entries one folder
* shows, and the most of a file it hands back at once.
*/
StorageBrowserPageSize: 200,
StorageBrowserReadMaxBytes: 50 * 1024 * 1024,
/**
* The most a bot copies off a computer into its own files in one
* call, so a folder of raw evidence can't fill the data volume.
*/
BotFileImportMaxBytes: 20 * 1024 * 1024,
/**
* Ingesting a bot file: how much text one chunk holds, how much of the
* previous chunk it repeats, and the largest file it reads at all.