Moduletrace/cobble
2
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases 32 Wiki Activity

Let generate_image work from reference images, with an image editing model

Browse Source 
generate_image takes reference images: files the run reaches as
[file:ID] refs, or the user's uploads by media id, up to four, each
checked like a picture source and limited to raster images.

Calls with references go to a new Image editing harness (no automatic
picks). While it has no model or is off, image generation takes them;
with its fallback on, image generation steps in after its pick fails.
Only models that take image input are tried with references, so none
drops them unseen. OpenRouter gets them on /images as input_references;
other providers as a multipart upload to /images/edits.

The Models pages show image editing's unset state as "Same as image
generation", and the models cache version is bumped so it shows at once.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
tben
2026-10-28 18:43:10 +01:00
co-authored by Claude Opus 5.5
parent 243c19e0c5
commit de28d4d9f7
22 changed files with 564 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+2 -1
View File
@@ -198,7 +198,8 @@ Read-only browser of every volume (`src/utils/storage-volumes.ts`: bots, workspa
### Harness (the model layer, `src/harness/`)
- **Harnesses** (`src/dict/harness-dict.ts`): `text` (always on), `vision`, `transcription`, `speech`, `image`, `embedding`. One `harness_settings` row each (provider, pinned model or automatic, `fallback`, `enabled`, `options`). Transcription, speech and embedding have no automatic picks: unavailable until a model is picked.
- **Harnesses** (`src/dict/harness-dict.ts`): `text` (always on), `vision`, `transcription`, `speech`, `image`, `image_edit`, `embedding`. One `harness_settings` row each (provider, pinned model or automatic, `fallback`, `enabled`, `options`). Transcription, speech and embedding have no automatic picks: unavailable until a model is picked.
- **Images:** `generate_image` takes reference images (`references` as `[file:ID]`, `reference_media_ids` for uploads, checked like a picture source by `loadReferenceImages`). With references the call goes to `image_edit` (no automatic picks; `HarnessFallbacks` hands it to `image` while unpicked or off, and after its pick fails when `fallback` is on), and only models that take image input are tried (`orderImageCandidates`), never one that would drop them. OpenRouter takes references on `/images` (`input_references`), other providers as multipart on `image_edit_path` (`/images/edits`).
- **Providers** (`src/dict/harness-provider-dict.ts`): OpenRouter default plus OpenAI-compatible ones; one adapter today (`src/harness/providers/openai-compatible/`). Keys encrypted in private `harness_providers`, falling back to the preset env.
- **Catalog:** `harness_models`, synced hourly. OpenRouter "free" means `:free` or `openrouter/free`, never zero prices. Auto picks: `rank-auto-models`.
- **Calling:** `resolveHarnessCandidates` -> `withCandidates` with process-wide cooldowns; errors are `HarnessError`. Entry points under `text/`, `media/`, `audio/`, `image/`, `embeddings/`. `run-text-loop.ts` sticks to one model, caps steps (`grabHarnessMaxToolSteps`, `grabReplyMaxToolSteps`), trims old tool output to fit context. Models that can't take an image/audio part get it read by vision/transcription first.