# Multimodal content

Ingesting images, audio, and video into the SurrealDB Agent Memory knowledge layer.

SurrealDB Agent Memory can ingest more than text. Images, audio recordings, and video files are all first-class document types in authoritative knowledge. Depending on the ingestion profile you select, it will apply OCR, speech-to-text transcription, and visual embedding to extract retrievable content from non-textual sources.

## Ingestion profiles

`Context.config.ingestion_profile` selects which provider-bearing stages run. Profiles are monotonic (each richer profile is a strict superset). Stages whose provider is absent are skipped, not errored.

| Profile | Text + chunks + embeds | Keywords + sections + doc links | OCR + speech-to-text | CLIP + vision caption + audio/video |
|---|---|---|---|---|
| `TextOnly` | Yes | No | No | No |
| `TextPlusKeyword` | Yes | Yes | No | No |
| `StandardMultimodal` | Yes | Yes | Yes | No |
| `MultimodalFull` | Yes | Yes | Yes | Yes |

**Default: `MultimodalFull`.** Uploaded images and audio get content understanding (vision caption, CLIP, audio embeddings) when the backing providers are configured. Dial down to `StandardMultimodal`, `TextPlusKeyword`, or `TextOnly` to reduce cost.

Wire names are PascalCase as in the table (`MultimodalFull`, not `MultimodalFull`).

Specify the profile on the Context configuration (or at upload when your client supports it):

```python
doc = await memory.documents.upload(
    file=open("product-diagram.png", "rb"),
    title="Product Architecture Diagram",
    profile="StandardMultimodal",
    scope=["org/acme"],
)
```

```javascript
const doc = await memory.documents.upload({
    file: imageFile,
    title: "Product Architecture Diagram",
    profile: "StandardMultimodal",
    scope: ["org/acme"],
});
```

## Supported formats

### Images

| Format | MIME type | Notes |
|---|---|---|
| PNG | `image/png` | Lossless; recommended for diagrams and screenshots |
| JPEG | `image/jpeg` | Suitable for photographs |
| WebP | `image/webp` | Modern format; lossless or lossy |
| GIF | `image/gif` | Static and animated; first frame used for embedding |

### Audio

| Format | MIME type |
|---|---|
| WAV | `audio/wav` |
| MP3 | `audio/mpeg` |
| OGG | `audio/ogg` |
| FLAC | `audio/flac` |
| AAC | `audio/aac` |

### Video

| Format | MIME type |
|---|---|
| MP4 | `video/mp4` |
| WebM | `video/webm` |
| MOV | `video/quicktime` |

## OCR

OCR (optical character recognition) recognises printed and handwritten text in images and scanned PDFs. The recognised text is extracted, split into chunks, and embedded in exactly the same way as text from a native PDF or HTML document. Chunks derived from OCR carry a `source: "ocr"` annotation in their metadata.

OCR is available with `StandardMultimodal` and `MultimodalFull` profiles.

### Scanned and low-text PDFs

When a PDF has **no extractable text layer** (image-only / scanned pages), SurrealDB Agent Memory rasterises pages and runs OCR when the **`local-ml-providers`** feature and an OCR provider are configured. Without OCR, ingest **fails loudly** with an actionable error - the document never lands in **`ready`** with an empty body.

For **partially scanned** PDFs (a sparse text layer on some pages), a low-text trigger rasterises and OCRs all pages when the recovered text exceeds what the native layer provided - all-or-nothing per document, not per-page splicing.

### Embedded PDF images

PDFs with embedded JPEG images (including common filter chains such as ASCII85/Hex + DCTDecode) extract image bytes and run the same OCR, vision, and CLIP path as standalone image uploads. OCR and vision text is folded into the chunk spine before embedding; visual embeddings land on **`image_chunk`** rows linked to the page's **`knowledge_section`**.

```python
doc = await memory.documents.upload(
    file=open("scanned-invoice.pdf", "rb"),
    title="Invoice 1042",
    profile="StandardMultimodal",
    scope=["org/acme"],
)
```

Pipeline stages during OCR processing include `extracting` and `rendering`. Once the document is `ready`, its chunks are searchable by text query.

## Speech-to-text

For audio and video documents, SurrealDB Agent Memory generates a full transcript and appends it to the chunk body so the spoken content is retrievable by semantic and keyword search. In addition to the full transcript, the pipeline produces time-coded `audio_chunk` segments that record approximately which part of the recording a retrieved chunk came from.

An `audio_chunk` segment looks like:

```json
{
  "chunk_id": "chunk:01hy2…",
  "start_ms": 14200,
  "end_ms": 28700,
  "text": "The return window for unopened items is thirty days from purchase."
}
```

This lets your application link a retrieved fact back to the precise moment in a recording - useful for surfacing relevant clips or timestamped citations.

Transcription is available with `StandardMultimodal` and `MultimodalFull` profiles.

Transcript segments participate in **vector recall** alongside text chunks (same embedding model and ranker leg). A query can return an `audio_chunk` hit with the segment text and timestamps even when the parent document chunk spine phrased the content differently.

When a document is **reprocessed**, previous `audio_chunk` and `image_chunk` rows are cleared at the start of the run before modality stages execute, so stale transcript or frame rows cannot survive a profile or content change.

```python
doc = await memory.documents.upload(
    file=open("support-call.mp3", "rb"),
    title="Support Call 2026-05-12",
    profile="StandardMultimodal",
    scope=["org/acme/user/alice"],
)
```

The pipeline status will pass through `transcribing` before reaching `ready`.

## CLIP visual embeddings

For image content processed under `MultimodalFull`, SurrealDB Agent Memory generates a CLIP visual embedding alongside any OCR text. CLIP embeddings capture the semantic content of the image independently of its textual labels, enabling retrieval by visual similarity.

When a query is issued against a context that includes images, the CLIP embeddings participate in the vector search alongside text embeddings. A query such as "product dimension diagram" can retrieve a relevant engineering drawing even if that drawing contains no recognisable text.

## Video captions

Under `MultimodalFull`, SurrealDB Agent Memory generates per-frame captions for video documents. Captions are indexed as additional text chunks associated with the video document. This allows text-based queries to retrieve video content described by the visual scene, not only by the spoken transcript.

Video caption extraction is the most resource-intensive stage and is therefore opt-in, available only with `MultimodalFull`.

## HTTP provider integration

By default, SurrealDB Agent Memory uses built-in or feature-gated local OCR and speech-to-text providers. For deployments with existing multimodal infrastructure, configure **deployment-level** HTTP endpoints via environment variables (same pattern as `SPECTRON_RERANKER_*`):

| Modality | Variables |
| --- | --- |
| OCR | `SPECTRON_OCR_URL`, `SPECTRON_OCR_MODEL`, `SPECTRON_OCR_API_KEY` (optional) |
| CLIP (visual embeddings) | `SPECTRON_CLIP_URL`, `SPECTRON_CLIP_MODEL`, `SPECTRON_CLIP_API_KEY` (optional) |
| Speech-to-text | `SPECTRON_STT_URL`, `SPECTRON_STT_MODEL`, `SPECTRON_STT_API_KEY` (optional) |

When a URL is set, its **`_MODEL`** is required and SurrealDB Agent Memory calls the endpoint at ingest time (worker role). HTTP providers take precedence over local fallbacks. CLIP output must match the **512-dim** image embedding width - dimension mismatch is rejected at embed time.

**Vision LLMs:** under **`MultimodalFull`**, image understanding runs when the worker resolves an extraction LLM for the Context - per-Context **`models.extraction`** and provider keys apply even without a deployment-wide default LLM. Vision descriptions are appended to the document body **before chunking** (same spine as OCR text). OpenAI and Anthropic providers send image content blocks when media is attached.

**Per-Context ingest LLM:** document jobs resolve the extraction LLM from the Context config on each run (shared budget-enforced resolver with maintenance jobs). A Context with its own provider key but no deployment-wide `SPECTRON_LLM_PROVIDER` gets ingest-time extraction and vision when configured.

See [Configuration](/docs/agent-memory/reference/configuration.md#multimodal-providers-optional) for the full variable list.

## Checking pipeline progress

Multimodal documents take longer to process than text-only uploads. The `status` field reflects the current pipeline stage precisely, so you can surface progress information in your application:

```python
import asyncio

stages = []

while True:
    doc = await memory.documents.get(doc.id)
    if doc.status not in stages:
        stages.append(doc.status)
        print(f"Stage: {doc.status}")

    if doc.status in ("ready", "failed"):
        break

    await asyncio.sleep(3)
```

Example output for a video document processed with `MultimodalFull`:

```
Stage: queued
Stage: extracting
Stage: transcribing
Stage: captioning
Stage: chunking
Stage: embedding
Stage: keywording
Stage: rendering
Stage: ready
```

## PII redaction

PII redaction can be enabled per-Context and applies to all ingested content, including OCR-recognised text and speech transcripts, before chunking. See [Extraction tuning](/docs/agent-memory/reference/configuration.md#extraction-tuning) for the **`pii_redaction_enabled`** field.