# Multimodal: image extraction + indexing for documents **Status:** Spec, not started. Branch will be cut from `main` at commit `63d8538`. **Goal:** When a user ingests a PDF / PPTX / DOCX that contains images (charts, diagrams, photos, screenshots), the images become discoverable via the existing wiki search + chat flow alongside the document's text. **Non-goals (this round):** - "Search by image" / image-to-image retrieval — deferred to Phase 5. - Editing or annotating images post-ingest. - OCR-only path (Tesseract). VLM caption is strictly more capable; if cost becomes the issue we'll add OCR as a fallback later. - Replacing / changing the existing chunker, embedding API, search ranking, or RAG pipeline. This work strictly **adds** a vision step on top. --- ## Current state (audit, not assumption) `src-tauri/src/commands/fs.rs::preprocess_file`: | Format | Current behavior | Image handling | |---|---|---| | PDF | `pdfium_render` → `page.text().all()` | **Ignored.** Embedded images, scans, charts all dropped. | | PPTX | unzip → parse `ppt/slides/slideN.xml` | **Ignored.** `ppt/media/*.png|jpg` already in the ZIP, just not read. | | DOCX | unzip → parse `word/document.xml` | **Ignored.** `word/media/*` same as above. | | XLSX/ODS | `calamine` → cell text | Ignored. | | Standalone images (.png/.jpg/...) | Read as binary in `read_file`; preview UI shows them | **Do NOT enter the ingest pipeline.** Never become wiki pages. | Whole TS chain (`text-chunker.ts`, `embedding.ts`, `search.ts`, `chat-panel.tsx`) is text-only. LanceDB v2 schema field is `chunk_text: Utf8` — no provision for image bytes or paths. Dependencies already present we can lean on: - `pdfium-render` 0.9 — supports `page.objects()` iteration, including `PdfPageObjectType::Image` extraction - `zip` 2.x — direct access to PPTX/DOCX `media/` directories - LLM provider abstraction in `llm-providers.ts` — every provider (OpenAI / Anthropic / Gemini / Claude Code CLI) supports vision-message input on its native wire; we just don't expose it --- ## Design: caption-first hybrid (Option C from planning) Three rejected alternatives are documented at the bottom of this file for posterity. The chosen path: 1. **Extract images** from PDF / PPTX / DOCX during preprocess 2. **Save originals** to `/wiki/media//.` 3. **Caption with vision LLM** ("describe factually, include any text, chart axes, key visual elements; 2–4 sentences") 4. **Inject as markdown** `![](media//.png)` into the source content fed to the analysis / generation prompts — so the LLM that builds the wiki page can place these images contextually 5. **Captions are ordinary text** — they flow through the existing `chunkMarkdown` → `embedPage` → `vector_upsert_chunks` pipeline with zero changes 6. **chat-message renders the markdown image** — the existing `react-markdown` setup already does this; it just needs the path to resolve to the right place ### Why this design - **No schema change** to LanceDB. Captions are text chunks. Search works without modification. - **No retrieval-quality regression.** Existing text-only retrieval paths are untouched. The chunker just sees more text (the captions) which makes images cite-able by their semantic content. - **User sees the actual image** in chat replies, not just a textual description. - **Provider-agnostic.** Every LLM provider we support has a vision format; we abstract over them in `buildBody`. - **Phased.** Each phase is independently shippable and reversible. ### What this design does NOT solve (and that's OK for v1) - Retrieving "an image that LOOKS like X" (visual similarity) — needs multimodal embedding (Phase 5, deferred). - Captions that miss subtle details (e.g. "the third bar is taller than the second") — limited by VLM quality. Pro-tier models help; Flash Lite captions will be shallow. - Image dedup across files (same logo / icon appearing 50 times) — handled by a SHA-256 hash cache, see Phase 1 risks below. --- ## Implementation phases ### Phase 1: Rust-side image extraction New commands in `src-tauri/src/commands/fs.rs` (or a new `src-tauri/src/commands/extract_images.rs` if `fs.rs` is getting too big — currently 1100+ lines, leaning toward new file). Public API shape (Tauri commands, callable from TS): ```rust #[derive(Serialize)] struct ExtractedImage { /// 1-based image index within the document (for filename) index: u32, /// PNG / JPEG / etc., as a MIME type mime_type: String, /// Page (PDF) or slide (PPTX) the image came from. None for DOCX. page: Option, /// Pixel width / height — used to filter out logos / icons. width: u32, height: u32, /// Image bytes, base64-encoded for IPC. data_base64: String, } #[tauri::command] async fn extract_pdf_images(path: String) -> Result, String> #[tauri::command] async fn extract_office_images(path: String) -> Result, String> ``` Implementation notes: - **PDF**: iterate `doc.pages()` → `page.objects()` → filter `PdfPageObjectType::Image` → `as_image_object().get_raw_image()` → encode to PNG via `image` crate (already a transitive dep through pdfium-render). - **PPTX/DOCX**: open as ZIP, iterate file names matching `^(ppt|word)/media/.*\.(png|jpe?g|gif|webp|bmp)$`, read bytes directly — already in their native format. - **Size filter**: drop images smaller than 100×100 (configurable later). Saves VLM cost on logos / decorations / cropping artifacts. ~80% noise removal in practice for slide decks. - **Memory**: extract images in a `for` loop, not `collect()` — a 100-page PDF with 50 images is ~50 MB before base64 (~67 MB after). Streaming through a `Vec` is OK for IPC but we should be defensive against a pathological 5000-image document — add a `max_images: 500` cap. Tests (`src-tauri/src/commands/extract_images.rs::tests`): - Synthetic PDF with 1 known image → extract returns 1 entry with expected dims and non-empty bytes. - Real PPTX from `tests/fixtures/` with multiple slides containing images → counts and sizes match. - DOCX with no images → returns `Ok([])`, not an error. - Password-protected PDF → returns the same error string the text extractor returns (consistent UX). ### Phase 2: Vision-message support in LLM abstraction `src/lib/llm-providers.ts`: ```ts // New union — replaces the existing `content: string` on ChatMessage export type ContentBlock = | { type: "text"; text: string } | { type: "image"; mediaType: string; dataBase64: string } export interface ChatMessage { role: "system" | "user" | "assistant" // Backwards-compatible: providers that don't get an image keep // calling sites working with plain strings. Block-array form // unlocks vision input. content: string | ContentBlock[] } ``` Each provider's `buildBody` learns to translate `ContentBlock[]`: - **OpenAI**: `[{type:"text",...}, {type:"image_url",image_url:{url:"data:image/png;base64,..."}}]` - **Anthropic**: `[{type:"text",...}, {type:"image",source:{type:"base64",media_type:"image/png",data:"..."}}]` - **Gemini**: `parts:[{text:"..."},{inline_data:{mime_type:"image/png",data:"..."}}]` - **Claude Code CLI**: already takes content blocks (PR #61), just add `image` block type passthrough. - **Ollama**: `messages[].images: [base64]` (separate field, not inline blocks). Conditional on the model — only `llava`, `qwen2.5-vl`, etc. accept it. Existing test files (`llm-providers.test.ts`, `__tests__/claude-cli-transport.test.ts`) need vision cases added. ### Phase 3: Captioning helper + ingest integration `src/lib/vision-caption.ts` (new): ```ts export async function captionImage( imageBase64: string, mediaType: string, llmConfig: LlmConfig, signal?: AbortSignal, ): Promise ``` Implementation: build a `streamChat` call with a single user message whose content is `[{type:"text",text:CAPTION_PROMPT},{type:"image",...}]`, collect all tokens, return the joined string. Caption prompt (pinned, factual, no markdown): > Describe this image factually for a knowledge-base index. Include: > any visible text verbatim, chart axes and values, diagram structure > (boxes/arrows/labels), key visual elements. Do NOT speculate or > editorialize. 2 to 4 sentences. Output plain text only — no > markdown, no preamble. `src/lib/ingest.ts` integration: After `preprocess_file` returns text, BEFORE the analysis stage: ```ts const images = await invoke('extract_pdf_images' or 'extract_office_images', { path }) const captioned = [] for (const img of images) { const relPath = `wiki/media/${slug}/img-${img.index}.${ext}` await writeFile(`${pp}/${relPath}`, base64ToBytes(img.data_base64)) const caption = await captionImage(img.data_base64, img.mime_type, llmConfig, signal) captioned.push({ relPath, caption, page: img.page }) } // Inject into sourceContent so the LLM sees them in context const imageSection = captioned.length > 0 ? '\n\n## Embedded Images\n\n' + captioned.map(c => c.page ? `**[Page ${c.page}]** ![${c.caption}](${c.relPath})` : `![${c.caption}](${c.relPath})` ).join('\n\n') : '' const enrichedSource = sourceContent + imageSection // ... rest of autoIngest uses enrichedSource ``` Per-image cache keyed by SHA-256 of image bytes — same logo across 50 PDFs = 1 caption call, not 50. Cache lives in `/.llm-wiki/image-caption-cache.json` mapping `hash → caption` (and image dimensions, mime, optionally the cached file path so we deduplicate file storage too). ### Phase 4: Settings toggle + cost guardrails `src/components/settings/sections/embedding-section.tsx` (or a new "Multimodal" section if it grows): add a toggle. ``` ☐ Index images from documents (uses extra LLM credits) Each image is captioned with a vision model. A 100-page paper with 30 images = 30 vision calls per ingest. Max images per document: [500] Skip images smaller than: [100]px on either side ``` Stored in `useWikiStore.embeddingConfig` (or a sibling `multimodalConfig` if we want to keep them separate). Read by `autoIngest` to decide whether to run Phase 1 + 3 at all. **Default off.** Users opt in. README / changelog notes the cost implication clearly. ### Phase 5 (deferred, NOT this round): multimodal embedding Add a parallel embedding path that hits a multimodal endpoint (`/v1/embeddings` with image input — supported by Voyage Multimodal, Jina CLIP v2, some local CLIP servers). Store image-vector alongside text chunk-vector in LanceDB (either same table with a `kind` field, or a sibling `wiki_images` table). This unlocks "find an image that looks like X" but is **strictly additive** — caption-based retrieval keeps working as-is. Requires user to have a multimodal embedding endpoint, which their current LM Studio `qwen3-embedding-0.6b` is NOT. --- ## Open questions (resolve before / during Phase 1) 1. **Provider matrix**: which providers should the vision toggle actually enable? OpenAI / Anthropic / Gemini / Claude Code CLI all work. Ollama needs a vision-capable model (must check `cfg.model` against a known list). Custom endpoint depends on user's setup. MiniMax — uncertain, needs probe. **Tentative answer**: silently skip vision step on providers that don't support it; show a banner in Settings. 2. **Image size threshold**: 100×100 vs 80×80 vs 5KB byte threshold. Small images are usually icons / decorations. **Tentative**: 100×100 default, exposed in Settings. 3. **Dedup strategy**: SHA-256 hash of image bytes → cache caption for that hash project-wide. **Tentative**: yes, default on. Cache invalidation tied to caption-prompt version. 4. **Per-document VLM cap**: a 500-page slide deck with 1500 images could blow up costs unnoticed. **Tentative**: hard cap at 500 images per document, configurable. Beyond that, surface a warning in the activity panel and skip. 5. **What if the VLM call fails / times out?** Caption-less image should still be saved to disk and embedded as `![image](path)` without a caption — it's at least visible to the user, just not searchable by content. Soft failure, not hard. 6. **Standalone .png / .jpg imports**: do we treat them as single-image "documents" and run them through the caption path? **Tentative**: yes, but as a follow-up after Phase 1–4 land for embedded images. 7. **Image sub-dir naming**: `wiki/media//` or flat `wiki/media/-.`? Subdirs are cleaner; conflicts resolved by source-delete cascade automatically. **Tentative**: subdirs. --- ## Risks - **Cost**: Phase 3 is the expensive step. Mitigated by Phase 4 toggle (default off) + dedup cache + per-doc cap. - **Caption quality**: Flash Lite produces near-useless 1-sentence captions. Document this in Settings hint; recommend Sonnet+ for multimodal. - **Performance**: each ingest now does N additional LLM calls in series. For a 30-image PDF, that's 30 × ~3s = 90s extra latency. We can parallelize the caption calls with `Promise.all` (the caption mutex doesn't apply — they're independent). - **PDF extraction quality**: pdfium's image extraction returns the raw embedded image; for vector graphics (which PDFs sometimes use for charts) this fails — those are paths/text, not Image objects. We'd miss them. **Mitigation**: render the entire page to a PNG as fallback when no Image objects found AND the page has structural complexity. Defer to Phase 1.5 if Phase 1 results are weak. --- ## Testing strategy Per phase, in priority order: **Phase 1 (Rust extraction):** - Unit tests with synthetic + real fixtures - Test on a known-good PDF (e.g. an arxiv paper) — verify image count matches manual count **Phase 2 (vision message format):** - Per-provider unit tests: assert correct wire format for each - Mock-server test that the bytes-on-wire match each provider's documented schema **Phase 3 (captioning + ingest):** - Real-LLM test (gated by `RUN_LLM_TESTS=1`): pass a known image, verify caption is non-trivial and contains expected keywords - Integration test: full autoIngest on a small fixture PDF with 2 known images → assert wiki/media/ has the files + the generated page references them in markdown **Phase 4 (toggle):** - UI smoke test (manual) - Unit test: when toggle is off, extract_*_images is never called --- ## Rejected alternatives (for posterity) - **Pure VLM caption** (Option A): same as our chosen path BUT without saving the original image. User loses ability to see the image in retrieval. Rejected — UX regression. - **Pure multimodal embedding** (Option B, no caption): no LLM cost at index time, true semantic image retrieval. Rejected because user's current embedding endpoint is text-only, AND we lose the ability to feed image content into LLM context (no caption text). - **OCR-only**: useless for non-textual images (charts, photos). Rejected for v1; could be added as a fallback for VLM failures. --- ## Branch + delivery plan - Cut branch `feat/multimodal-images` from `main` @ `63d8538` - Phase 1 → 1 commit, ~3-4 days work - Phase 2 → 1 commit, ~2 days - Phase 3 → 2 commits (caption helper + ingest integration), ~3 days - Phase 4 → 1 commit, ~1 day - Each commit independently runnable + tested. Merge phases into branch as they land. Final merge to main as one big feature, OR as 4 separate PRs depending on review preference. Total estimate: ~10 days of focused work.