Add knowledgebases

Knowledgebases give bots RAG over documents, scoped like extensions: to
particular bots, to a workspace, or to every bot, with their own button
beside Tools in the composer. Documents are chunked and embedded by the
file-ingest worker with the install's embedding model, which is now an
install-wide setting rather than a per-bot one.

- Word search runs on an FTS5 index ranked by BM25, with a scored LIKE
  fallback for scripts written without spaces.
- Changing the embedding model re-indexes documents automatically.
- Ingest jobs wait without spending attempts while every embedding model
  rests, fail at once on errors retrying can't fix, and keep passages
  searchable by words when the model refuses.
- Settings > Knowledgebases shows each knowledgebase's and document's
  index state: spinner, warning, grey check (words only) or green check.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
2026-10-26 16:41:01 +01:00
co-authored by Claude Opus 5.5
parent 2259b6895f
commit e7b23b7454
111 changed files with 4573 additions and 148 deletions
@@ -0,0 +1,21 @@
import { describe, expect, test } from "bun:test";
import knowledgebaseSummary from "./knowledgebase-summary";
const empty = { document_count: 0, pending_count: 0, error_count: 0, stale_count: 0, size: 0 };
describe("knowledgebaseSummary", () => {
test("an empty knowledgebase says so", () => {
expect(knowledgebaseSummary({ knowledgebase: empty })).toBe("No documents yet");
});
test("counts documents and names only the problems there are", () => {
expect(knowledgebaseSummary({ knowledgebase: { ...empty, document_count: 1, size: 2048 } })).toBe(
"1 document · 2.0 KB",
);
expect(
knowledgebaseSummary({
knowledgebase: { document_count: 3, pending_count: 1, error_count: 1, stale_count: 2, size: 2048 },
}),
).toBe("3 documents · 2.0 KB · 1 being read · 1 couldn't be read · 2 need re-indexing");
});
});