修复快照完整性：raw/llm_wiki 由 gitlink 转为普通目录（.git 备份为 .git.bak），新增 .gitignore 排除 __pycache__/pyc 与子仓库元数据
This commit is contained in:
AAsige
@@ -0,0 +1,389 @@
|
||||
# Multimodal: image extraction + indexing for documents
|
||||
|
||||
**Status:** Spec, not started. Branch will be cut from `main` at commit `63d8538`.
|
||||
|
||||
**Goal:** When a user ingests a PDF / PPTX / DOCX that contains images
|
||||
(charts, diagrams, photos, screenshots), the images become discoverable
|
||||
via the existing wiki search + chat flow alongside the document's text.
|
||||
|
||||
**Non-goals (this round):**
|
||||
- "Search by image" / image-to-image retrieval — deferred to Phase 5.
|
||||
- Editing or annotating images post-ingest.
|
||||
- OCR-only path (Tesseract). VLM caption is strictly more capable; if
|
||||
cost becomes the issue we'll add OCR as a fallback later.
|
||||
- Replacing / changing the existing chunker, embedding API, search
|
||||
ranking, or RAG pipeline. This work strictly **adds** a vision step
|
||||
on top.
|
||||
|
||||
---
|
||||
|
||||
## Current state (audit, not assumption)
|
||||
|
||||
`src-tauri/src/commands/fs.rs::preprocess_file`:
|
||||
|
||||
| Format | Current behavior | Image handling |
|
||||
|---|---|---|
|
||||
| PDF | `pdfium_render` → `page.text().all()` | **Ignored.** Embedded images, scans, charts all dropped. |
|
||||
| PPTX | unzip → parse `ppt/slides/slideN.xml` | **Ignored.** `ppt/media/*.png|jpg` already in the ZIP, just not read. |
|
||||
| DOCX | unzip → parse `word/document.xml` | **Ignored.** `word/media/*` same as above. |
|
||||
| XLSX/ODS | `calamine` → cell text | Ignored. |
|
||||
| Standalone images (.png/.jpg/...) | Read as binary in `read_file`; preview UI shows them | **Do NOT enter the ingest pipeline.** Never become wiki pages. |
|
||||
|
||||
Whole TS chain (`text-chunker.ts`, `embedding.ts`, `search.ts`,
|
||||
`chat-panel.tsx`) is text-only. LanceDB v2 schema field is
|
||||
`chunk_text: Utf8` — no provision for image bytes or paths.
|
||||
|
||||
Dependencies already present we can lean on:
|
||||
- `pdfium-render` 0.9 — supports `page.objects()` iteration, including
|
||||
`PdfPageObjectType::Image` extraction
|
||||
- `zip` 2.x — direct access to PPTX/DOCX `media/` directories
|
||||
- LLM provider abstraction in `llm-providers.ts` — every provider
|
||||
(OpenAI / Anthropic / Gemini / Claude Code CLI) supports
|
||||
vision-message input on its native wire; we just don't expose it
|
||||
|
||||
---
|
||||
|
||||
## Design: caption-first hybrid (Option C from planning)
|
||||
|
||||
Three rejected alternatives are documented at the bottom of this file
|
||||
for posterity. The chosen path:
|
||||
|
||||
1. **Extract images** from PDF / PPTX / DOCX during preprocess
|
||||
2. **Save originals** to `<project>/wiki/media/<source-slug>/<n>.<ext>`
|
||||
3. **Caption with vision LLM** ("describe factually, include any text,
|
||||
chart axes, key visual elements; 2–4 sentences")
|
||||
4. **Inject as markdown** ``
|
||||
into the source content fed to the analysis / generation prompts —
|
||||
so the LLM that builds the wiki page can place these images
|
||||
contextually
|
||||
5. **Captions are ordinary text** — they flow through the existing
|
||||
`chunkMarkdown` → `embedPage` → `vector_upsert_chunks` pipeline
|
||||
with zero changes
|
||||
6. **chat-message renders the markdown image** — the existing
|
||||
`react-markdown` setup already does this; it just needs the path
|
||||
to resolve to the right place
|
||||
|
||||
### Why this design
|
||||
|
||||
- **No schema change** to LanceDB. Captions are text chunks. Search
|
||||
works without modification.
|
||||
- **No retrieval-quality regression.** Existing text-only retrieval
|
||||
paths are untouched. The chunker just sees more text (the
|
||||
captions) which makes images cite-able by their semantic content.
|
||||
- **User sees the actual image** in chat replies, not just a textual
|
||||
description.
|
||||
- **Provider-agnostic.** Every LLM provider we support has a vision
|
||||
format; we abstract over them in `buildBody`.
|
||||
- **Phased.** Each phase is independently shippable and reversible.
|
||||
|
||||
### What this design does NOT solve (and that's OK for v1)
|
||||
|
||||
- Retrieving "an image that LOOKS like X" (visual similarity) — needs
|
||||
multimodal embedding (Phase 5, deferred).
|
||||
- Captions that miss subtle details (e.g. "the third bar is taller
|
||||
than the second") — limited by VLM quality. Pro-tier models help;
|
||||
Flash Lite captions will be shallow.
|
||||
- Image dedup across files (same logo / icon appearing 50 times) —
|
||||
handled by a SHA-256 hash cache, see Phase 1 risks below.
|
||||
|
||||
---
|
||||
|
||||
## Implementation phases
|
||||
|
||||
### Phase 1: Rust-side image extraction
|
||||
|
||||
New commands in `src-tauri/src/commands/fs.rs` (or a new
|
||||
`src-tauri/src/commands/extract_images.rs` if `fs.rs` is getting too
|
||||
big — currently 1100+ lines, leaning toward new file).
|
||||
|
||||
Public API shape (Tauri commands, callable from TS):
|
||||
|
||||
```rust
|
||||
#[derive(Serialize)]
|
||||
struct ExtractedImage {
|
||||
/// 1-based image index within the document (for filename)
|
||||
index: u32,
|
||||
/// PNG / JPEG / etc., as a MIME type
|
||||
mime_type: String,
|
||||
/// Page (PDF) or slide (PPTX) the image came from. None for DOCX.
|
||||
page: Option<u32>,
|
||||
/// Pixel width / height — used to filter out logos / icons.
|
||||
width: u32,
|
||||
height: u32,
|
||||
/// Image bytes, base64-encoded for IPC.
|
||||
data_base64: String,
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
async fn extract_pdf_images(path: String) -> Result<Vec<ExtractedImage>, String>
|
||||
|
||||
#[tauri::command]
|
||||
async fn extract_office_images(path: String) -> Result<Vec<ExtractedImage>, String>
|
||||
```
|
||||
|
||||
Implementation notes:
|
||||
- **PDF**: iterate `doc.pages()` → `page.objects()` → filter
|
||||
`PdfPageObjectType::Image` → `as_image_object().get_raw_image()`
|
||||
→ encode to PNG via `image` crate (already a transitive dep
|
||||
through pdfium-render).
|
||||
- **PPTX/DOCX**: open as ZIP, iterate file names matching
|
||||
`^(ppt|word)/media/.*\.(png|jpe?g|gif|webp|bmp)$`, read bytes
|
||||
directly — already in their native format.
|
||||
- **Size filter**: drop images smaller than 100×100 (configurable
|
||||
later). Saves VLM cost on logos / decorations / cropping
|
||||
artifacts. ~80% noise removal in practice for slide decks.
|
||||
- **Memory**: extract images in a `for` loop, not `collect()` — a
|
||||
100-page PDF with 50 images is ~50 MB before base64 (~67 MB
|
||||
after). Streaming through a `Vec<ExtractedImage>` is OK for IPC
|
||||
but we should be defensive against a pathological 5000-image
|
||||
document — add a `max_images: 500` cap.
|
||||
|
||||
Tests (`src-tauri/src/commands/extract_images.rs::tests`):
|
||||
- Synthetic PDF with 1 known image → extract returns 1 entry with
|
||||
expected dims and non-empty bytes.
|
||||
- Real PPTX from `tests/fixtures/` with multiple slides containing
|
||||
images → counts and sizes match.
|
||||
- DOCX with no images → returns `Ok([])`, not an error.
|
||||
- Password-protected PDF → returns the same error string the text
|
||||
extractor returns (consistent UX).
|
||||
|
||||
### Phase 2: Vision-message support in LLM abstraction
|
||||
|
||||
`src/lib/llm-providers.ts`:
|
||||
|
||||
```ts
|
||||
// New union — replaces the existing `content: string` on ChatMessage
|
||||
export type ContentBlock =
|
||||
| { type: "text"; text: string }
|
||||
| { type: "image"; mediaType: string; dataBase64: string }
|
||||
|
||||
export interface ChatMessage {
|
||||
role: "system" | "user" | "assistant"
|
||||
// Backwards-compatible: providers that don't get an image keep
|
||||
// calling sites working with plain strings. Block-array form
|
||||
// unlocks vision input.
|
||||
content: string | ContentBlock[]
|
||||
}
|
||||
```
|
||||
|
||||
Each provider's `buildBody` learns to translate `ContentBlock[]`:
|
||||
- **OpenAI**: `[{type:"text",...}, {type:"image_url",image_url:{url:"data:image/png;base64,..."}}]`
|
||||
- **Anthropic**: `[{type:"text",...}, {type:"image",source:{type:"base64",media_type:"image/png",data:"..."}}]`
|
||||
- **Gemini**: `parts:[{text:"..."},{inline_data:{mime_type:"image/png",data:"..."}}]`
|
||||
- **Claude Code CLI**: already takes content blocks (PR #61), just
|
||||
add `image` block type passthrough.
|
||||
- **Ollama**: `messages[].images: [base64]` (separate field, not
|
||||
inline blocks). Conditional on the model — only `llava`,
|
||||
`qwen2.5-vl`, etc. accept it.
|
||||
|
||||
Existing test files (`llm-providers.test.ts`,
|
||||
`__tests__/claude-cli-transport.test.ts`) need vision cases added.
|
||||
|
||||
### Phase 3: Captioning helper + ingest integration
|
||||
|
||||
`src/lib/vision-caption.ts` (new):
|
||||
|
||||
```ts
|
||||
export async function captionImage(
|
||||
imageBase64: string,
|
||||
mediaType: string,
|
||||
llmConfig: LlmConfig,
|
||||
signal?: AbortSignal,
|
||||
): Promise<string>
|
||||
```
|
||||
|
||||
Implementation: build a `streamChat` call with a single user message
|
||||
whose content is `[{type:"text",text:CAPTION_PROMPT},{type:"image",...}]`,
|
||||
collect all tokens, return the joined string.
|
||||
|
||||
Caption prompt (pinned, factual, no markdown):
|
||||
|
||||
> Describe this image factually for a knowledge-base index. Include:
|
||||
> any visible text verbatim, chart axes and values, diagram structure
|
||||
> (boxes/arrows/labels), key visual elements. Do NOT speculate or
|
||||
> editorialize. 2 to 4 sentences. Output plain text only — no
|
||||
> markdown, no preamble.
|
||||
|
||||
`src/lib/ingest.ts` integration:
|
||||
|
||||
After `preprocess_file` returns text, BEFORE the analysis stage:
|
||||
|
||||
```ts
|
||||
const images = await invoke('extract_pdf_images' or 'extract_office_images', { path })
|
||||
const captioned = []
|
||||
for (const img of images) {
|
||||
const relPath = `wiki/media/${slug}/img-${img.index}.${ext}`
|
||||
await writeFile(`${pp}/${relPath}`, base64ToBytes(img.data_base64))
|
||||
const caption = await captionImage(img.data_base64, img.mime_type, llmConfig, signal)
|
||||
captioned.push({ relPath, caption, page: img.page })
|
||||
}
|
||||
|
||||
// Inject into sourceContent so the LLM sees them in context
|
||||
const imageSection = captioned.length > 0
|
||||
? '\n\n## Embedded Images\n\n' +
|
||||
captioned.map(c =>
|
||||
c.page
|
||||
? `**[Page ${c.page}]** `
|
||||
: ``
|
||||
).join('\n\n')
|
||||
: ''
|
||||
const enrichedSource = sourceContent + imageSection
|
||||
// ... rest of autoIngest uses enrichedSource
|
||||
```
|
||||
|
||||
Per-image cache keyed by SHA-256 of image bytes — same logo across
|
||||
50 PDFs = 1 caption call, not 50. Cache lives in
|
||||
`<project>/.llm-wiki/image-caption-cache.json` mapping
|
||||
`hash → caption` (and image dimensions, mime, optionally the cached
|
||||
file path so we deduplicate file storage too).
|
||||
|
||||
### Phase 4: Settings toggle + cost guardrails
|
||||
|
||||
`src/components/settings/sections/embedding-section.tsx` (or a new
|
||||
"Multimodal" section if it grows): add a toggle.
|
||||
|
||||
```
|
||||
☐ Index images from documents (uses extra LLM credits)
|
||||
Each image is captioned with a vision model. A 100-page paper
|
||||
with 30 images = 30 vision calls per ingest.
|
||||
Max images per document: [500]
|
||||
Skip images smaller than: [100]px on either side
|
||||
```
|
||||
|
||||
Stored in `useWikiStore.embeddingConfig` (or a sibling
|
||||
`multimodalConfig` if we want to keep them separate). Read by
|
||||
`autoIngest` to decide whether to run Phase 1 + 3 at all.
|
||||
|
||||
**Default off.** Users opt in. README / changelog notes the cost
|
||||
implication clearly.
|
||||
|
||||
### Phase 5 (deferred, NOT this round): multimodal embedding
|
||||
|
||||
Add a parallel embedding path that hits a multimodal endpoint
|
||||
(`/v1/embeddings` with image input — supported by Voyage Multimodal,
|
||||
Jina CLIP v2, some local CLIP servers). Store image-vector alongside
|
||||
text chunk-vector in LanceDB (either same table with a `kind` field,
|
||||
or a sibling `wiki_images` table).
|
||||
|
||||
This unlocks "find an image that looks like X" but is **strictly
|
||||
additive** — caption-based retrieval keeps working as-is.
|
||||
|
||||
Requires user to have a multimodal embedding endpoint, which their
|
||||
current LM Studio `qwen3-embedding-0.6b` is NOT.
|
||||
|
||||
---
|
||||
|
||||
## Open questions (resolve before / during Phase 1)
|
||||
|
||||
1. **Provider matrix**: which providers should the vision toggle
|
||||
actually enable? OpenAI / Anthropic / Gemini / Claude Code CLI all
|
||||
work. Ollama needs a vision-capable model (must check `cfg.model`
|
||||
against a known list). Custom endpoint depends on user's setup.
|
||||
MiniMax — uncertain, needs probe. **Tentative answer**: silently
|
||||
skip vision step on providers that don't support it; show a banner
|
||||
in Settings.
|
||||
|
||||
2. **Image size threshold**: 100×100 vs 80×80 vs 5KB byte threshold.
|
||||
Small images are usually icons / decorations. **Tentative**:
|
||||
100×100 default, exposed in Settings.
|
||||
|
||||
3. **Dedup strategy**: SHA-256 hash of image bytes → cache caption
|
||||
for that hash project-wide. **Tentative**: yes, default on.
|
||||
Cache invalidation tied to caption-prompt version.
|
||||
|
||||
4. **Per-document VLM cap**: a 500-page slide deck with 1500 images
|
||||
could blow up costs unnoticed. **Tentative**: hard cap at 500
|
||||
images per document, configurable. Beyond that, surface a
|
||||
warning in the activity panel and skip.
|
||||
|
||||
5. **What if the VLM call fails / times out?** Caption-less image
|
||||
should still be saved to disk and embedded as ``
|
||||
without a caption — it's at least visible to the user, just not
|
||||
searchable by content. Soft failure, not hard.
|
||||
|
||||
6. **Standalone .png / .jpg imports**: do we treat them as
|
||||
single-image "documents" and run them through the caption path?
|
||||
**Tentative**: yes, but as a follow-up after Phase 1–4 land for
|
||||
embedded images.
|
||||
|
||||
7. **Image sub-dir naming**: `wiki/media/<source-slug>/` or flat
|
||||
`wiki/media/<slug>-<n>.<ext>`? Subdirs are cleaner; conflicts
|
||||
resolved by source-delete cascade automatically. **Tentative**:
|
||||
subdirs.
|
||||
|
||||
---
|
||||
|
||||
## Risks
|
||||
|
||||
- **Cost**: Phase 3 is the expensive step. Mitigated by Phase 4
|
||||
toggle (default off) + dedup cache + per-doc cap.
|
||||
- **Caption quality**: Flash Lite produces near-useless 1-sentence
|
||||
captions. Document this in Settings hint; recommend Sonnet+ for
|
||||
multimodal.
|
||||
- **Performance**: each ingest now does N additional LLM calls in
|
||||
series. For a 30-image PDF, that's 30 × ~3s = 90s extra latency.
|
||||
We can parallelize the caption calls with `Promise.all` (the
|
||||
caption mutex doesn't apply — they're independent).
|
||||
- **PDF extraction quality**: pdfium's image extraction returns the
|
||||
raw embedded image; for vector graphics (which PDFs sometimes
|
||||
use for charts) this fails — those are paths/text, not Image
|
||||
objects. We'd miss them. **Mitigation**: render the entire page
|
||||
to a PNG as fallback when no Image objects found AND the page
|
||||
has structural complexity. Defer to Phase 1.5 if Phase 1
|
||||
results are weak.
|
||||
|
||||
---
|
||||
|
||||
## Testing strategy
|
||||
|
||||
Per phase, in priority order:
|
||||
|
||||
**Phase 1 (Rust extraction):**
|
||||
- Unit tests with synthetic + real fixtures
|
||||
- Test on a known-good PDF (e.g. an arxiv paper) — verify image
|
||||
count matches manual count
|
||||
|
||||
**Phase 2 (vision message format):**
|
||||
- Per-provider unit tests: assert correct wire format for each
|
||||
- Mock-server test that the bytes-on-wire match each provider's
|
||||
documented schema
|
||||
|
||||
**Phase 3 (captioning + ingest):**
|
||||
- Real-LLM test (gated by `RUN_LLM_TESTS=1`): pass a known image,
|
||||
verify caption is non-trivial and contains expected keywords
|
||||
- Integration test: full autoIngest on a small fixture PDF with 2
|
||||
known images → assert wiki/media/ has the files + the generated
|
||||
page references them in markdown
|
||||
|
||||
**Phase 4 (toggle):**
|
||||
- UI smoke test (manual)
|
||||
- Unit test: when toggle is off, extract_*_images is never called
|
||||
|
||||
---
|
||||
|
||||
## Rejected alternatives (for posterity)
|
||||
|
||||
- **Pure VLM caption** (Option A): same as our chosen path BUT
|
||||
without saving the original image. User loses ability to see the
|
||||
image in retrieval. Rejected — UX regression.
|
||||
- **Pure multimodal embedding** (Option B, no caption): no LLM cost
|
||||
at index time, true semantic image retrieval. Rejected because
|
||||
user's current embedding endpoint is text-only, AND we lose the
|
||||
ability to feed image content into LLM context (no caption text).
|
||||
- **OCR-only**: useless for non-textual images (charts, photos).
|
||||
Rejected for v1; could be added as a fallback for VLM failures.
|
||||
|
||||
---
|
||||
|
||||
## Branch + delivery plan
|
||||
|
||||
- Cut branch `feat/multimodal-images` from `main` @ `63d8538`
|
||||
- Phase 1 → 1 commit, ~3-4 days work
|
||||
- Phase 2 → 1 commit, ~2 days
|
||||
- Phase 3 → 2 commits (caption helper + ingest integration), ~3 days
|
||||
- Phase 4 → 1 commit, ~1 day
|
||||
- Each commit independently runnable + tested. Merge phases into
|
||||
branch as they land. Final merge to main as one big feature, OR
|
||||
as 4 separate PRs depending on review preference.
|
||||
|
||||
Total estimate: ~10 days of focused work.
|
||||
Reference in New Issue
Block a user