Multimodal content

Spectron can ingest more than text. Images, audio recordings, and video files are all first-class document types in authoritative knowledge. Depending on the ingestion profile you select, Spectron applies OCR, speech-to-text transcription, and visual embedding to extract retrievable content from non-textual sources.

Ingestion profiles

Context.config.ingestion_profile selects which provider-bearing stages run. Profiles are monotonic (each richer profile is a strict superset). Stages whose provider is absent are skipped, not errored.

Default: MultimodalFull. Uploaded images and audio get content understanding (vision caption, CLIP, audio embeddings) when the backing providers are configured. Dial down to StandardMultimodal, TextPlusKeyword, or TextOnly to reduce cost.

Wire names are PascalCase as in the table (MultimodalFull, not MultimodalFull).

Specify the profile on the Context configuration (or at upload when your client supports it):

doc = await memory.knowledge.upload(
    file=open("product-diagram.png", "rb"),
    title="Product Architecture Diagram",
    profile="StandardMultimodal",
    scope=["org/acme"],
)

const doc = await memory.knowledge.upload({
    file: imageFile,
    title: "Product Architecture Diagram",
    profile: "StandardMultimodal",
    scope: ["org/acme"],
});

Supported formats

Images

Audio

Video

OCR

OCR (optical character recognition) recognises printed and handwritten text in images and scanned PDFs. The recognised text is extracted, split into chunks, and embedded in exactly the same way as text from a native PDF or HTML document. Chunks derived from OCR carry a source: "ocr" annotation in their metadata.

OCR is available with StandardMultimodal and MultimodalFull profiles.

Scanned and low-text PDFs

When a PDF has no extractable text layer (image-only / scanned pages), Spectron rasterises pages and runs OCR when the local-ml-providers feature and an OCR provider are configured. Without OCR, ingest fails loudly with an actionable error — the document never lands in ready with an empty body.

For partially scanned PDFs (a sparse text layer on some pages), a low-text trigger rasterises and OCRs all pages when the recovered text exceeds what the native layer provided — all-or-nothing per document, not per-page splicing.

Embedded PDF images

PDFs with embedded JPEG images (including common filter chains such as ASCII85/Hex + DCTDecode) extract image bytes and run the same OCR, vision, and CLIP path as standalone image uploads. OCR and vision text is folded into the chunk spine before embedding; visual embeddings land on image_chunk rows linked to the page's knowledge_section.

doc = await memory.knowledge.upload(
    file=open("scanned-invoice.pdf", "rb"),
    title="Invoice 1042",
    profile="StandardMultimodal",
    scope=["org/acme"],
)

Pipeline stages during OCR processing include extracting and rendering. Once the document is ready, its chunks are searchable by text query.

Speech-to-text

For audio and video documents, Spectron generates a full transcript and appends it to the chunk body so the spoken content is retrievable by semantic and keyword search. In addition to the full transcript, the pipeline produces time-coded audio_chunk segments that record approximately which part of the recording a retrieved chunk came from.

An audio_chunk segment looks like:

{
  "chunk_id": "chunk:01hy2…",
  "start_ms": 14200,
  "end_ms": 28700,
  "text": "The return window for unopened items is thirty days from purchase."
}

This lets your application link a retrieved fact back to the precise moment in a recording – useful for surfacing relevant clips or timestamped citations.

Transcription is available with StandardMultimodal and MultimodalFull profiles.

Transcript segments participate in vector recall alongside text chunks (same embedding model and ranker leg). A query can return an audio_chunk hit with the segment text and timestamps even when the parent document chunk spine phrased the content differently.

When a document is reprocessed, previous audio_chunk and image_chunk rows are cleared at the start of the run before modality stages execute, so stale transcript or frame rows cannot survive a profile or content change.

doc = await memory.knowledge.upload(
    file=open("support-call.mp3", "rb"),
    title="Support Call 2026-05-12",
    profile="StandardMultimodal",
    scope=["org/acme/user/alice"],
)

The pipeline status will pass through transcribing before reaching ready.

CLIP visual embeddings

For image content processed under MultimodalFull, Spectron generates a CLIP visual embedding alongside any OCR text. CLIP embeddings capture the semantic content of the image independently of its textual labels, enabling retrieval by visual similarity.

When a query is issued against a context that includes images, the CLIP embeddings participate in the vector search alongside text embeddings. A query such as "product dimension diagram" can retrieve a relevant engineering drawing even if that drawing contains no recognisable text.

Video captions

Under MultimodalFull, Spectron generates per-frame captions for video documents. Captions are indexed as additional text chunks associated with the video document. This allows text-based queries to retrieve video content described by the visual scene, not only by the spoken transcript.

Video caption extraction is the most resource-intensive stage and is therefore opt-in, available only with MultimodalFull.

HTTP provider integration

By default, Spectron uses built-in or feature-gated local OCR and speech-to-text providers. For deployments with existing multimodal infrastructure, configure deployment-level HTTP endpoints via environment variables (same pattern as SPECTRON_RERANKER_*):

When a URL is set, its _MODEL is required and Spectron calls the endpoint at ingest time (worker role). HTTP providers take precedence over local fallbacks. CLIP output must match the 512-dim image embedding width — dimension mismatch is rejected at embed time.

Vision LLMs: under MultimodalFull, image understanding runs when the worker resolves an extraction LLM for the Context — per-Context models.extraction and provider keys apply even without a deployment-wide default LLM. Vision descriptions are appended to the document body before chunking (same spine as OCR text). OpenAI and Anthropic providers send image content blocks when media is attached.

Per-Context ingest LLM: document jobs resolve the extraction LLM from the Context config on each run (shared budget-enforced resolver with maintenance jobs). A Context with its own provider key but no deployment-wide SPECTRON_LLM_PROVIDER gets ingest-time extraction and vision when configured.

See Configuration for the full variable list.

Checking pipeline progress

Multimodal documents take longer to process than text-only uploads. The status field reflects the current pipeline stage precisely, so you can surface progress information in your application:

import asyncio

stages = []

while True:
    doc = await memory.knowledge.get(doc.id)
    if doc.status not in stages:
        stages.append(doc.status)
        print(f"Stage: {doc.status}")

    if doc.status in ("ready", "failed"):
        break

    await asyncio.sleep(3)

Example output for a video document processed with MultimodalFull:

Stage: queued
Stage: extracting
Stage: transcribing
Stage: captioning
Stage: chunking
Stage: embedding
Stage: keywording
Stage: rendering
Stage: ready

PII redaction

PII redaction can be enabled per-Context and applies to all ingested content, including OCR-recognised text and speech transcripts, before chunking. See Context configuration for details on enabling PII redaction.