> Full SurrealDB documentation index: https://surrealdb.com/docs/llms.txt

# Multimodal content

Ingesting images, audio, and video into the SurrealDB Agent Memory knowledge layer.

SurrealDB Agent Memory can ingest more than text. Images, audio recordings, and video files are all first-class document types in authoritative knowledge. Depending on the Context's ingestion profile, it will apply OCR, speech-to-text transcription, and visual embedding to extract retrievable content from non-textual sources.

## Ingestion profiles

`Context.config.ingestion_profile` selects which provider-bearing stages run. Profiles are monotonic (each richer profile is a strict superset). Stages whose provider is absent are skipped, not errored.

| Profile | Text + chunks + embeds | Keywords + sections + doc links | OCR + speech-to-text | CLIP + vision caption + audio/video |
|---|---|---|---|---|
| `TextOnly` | Yes | No | No | No |
| `TextPlusKeyword` | Yes | Yes | No | No |
| `StandardMultimodal` | Yes | Yes | Yes | No |
| `MultimodalFull` | Yes | Yes | Yes | Yes |

**Default: `MultimodalFull`.** Uploaded images and audio get content understanding (vision caption, CLIP, audio embeddings) when the backing providers are configured. Dial down to `StandardMultimodal`, `TextPlusKeyword`, or `TextOnly` to reduce cost.

Wire names are PascalCase as in the table (`MultimodalFull`, not `multimodal_full`).

The profile is a Context setting, `ingestion_profile` in the Context configuration, and applies to every upload in that Context. An upload request has no profile field, so an image is uploaded like any other document:

```python
doc = await memory.documents.upload(
    "product-diagram.png",
    title="Product Architecture Diagram",
    scopes=["org/acme"],
)
```

```javascript
const doc = await memory.documents.upload({
    file: imageFile,
    contentType: "image/png",
    title: "Product Architecture Diagram",
    scopes: ["org/acme"],
});
```

## Supported formats

### Images

| Format | MIME type | Notes |
|---|---|---|
| PNG | `image/png` | Lossless; recommended for diagrams and screenshots |
| JPEG | `image/jpeg` | Suitable for photographs |
| WebP | `image/webp` | Modern format; lossless or lossy |
| GIF | `image/gif` | Static and animated; first frame used for embedding |

### Audio

| Format | MIME type |
|---|---|
| WAV | `audio/wav` |
| MP3 | `audio/mpeg` |
| OGG | `audio/ogg` |
| FLAC | `audio/flac` |
| AAC | `audio/aac` |

### Video

| Format | MIME type |
|---|---|
| MP4 | `video/mp4` |
| WebM | `video/webm` |
| MOV | `video/quicktime` |

## OCR

OCR (optical character recognition) recognises printed and handwritten text in images and scanned PDFs. The recognised text is extracted, split into chunks, and embedded in exactly the same way as text from a native PDF or HTML document. Chunks do not record whether their text came from OCR.

OCR is available with `StandardMultimodal` and `MultimodalFull` profiles.

### Scanned and low-text PDFs

When a PDF has **no extractable text layer** (image-only / scanned pages), SurrealDB Agent Memory first sends the whole PDF to the Gemini document parser when the deployment has one. If there is no parser or it recovers no text, it rasterises pages and runs OCR when the deployment has an OCR provider. When neither recovers text, ingest **fails loudly** with an actionable error - the document never lands in **`ready`** with an empty body.

For **partially scanned** PDFs (a sparse text layer on some pages), a low-text trigger rasterises and OCRs all pages when the recovered text exceeds what the native layer provided - all-or-nothing per document, not per-page splicing.

### Embedded PDF images

PDFs with embedded JPEG images (including common filter chains such as ASCII85/Hex + DCTDecode) extract image bytes and run the same OCR, vision, and CLIP path as standalone image uploads. OCR and vision text is folded into the chunk spine before embedding; visual embeddings land on **`image_chunk`** records linked to the page's **`knowledge_section`**.

```python
doc = await memory.documents.upload(
    "scanned-invoice.pdf",
    title="Invoice 1042",
    scopes=["org/acme"],
)
```

OCR runs while the document status is `extracting`. Once the document is `ready`, its chunks are searchable by text query.

## Speech-to-text

For audio and video documents, SurrealDB Agent Memory generates a full transcript and appends it to the chunk body so the spoken content is retrievable by semantic and keyword search. In addition to the full transcript, the pipeline produces time-coded `audio_chunk` segments that record approximately which part of the recording a retrieved chunk came from.

An `audio_chunk` segment record looks like this, with times in seconds from the start of the recording:

```json
{
  "document": "document:01hx9…",
  "position": 3,
  "start_seconds": 14.2,
  "end_seconds": 28.7,
  "source": "stt",
  "text": "The return window for unopened items is thirty days from purchase."
}
```

This lets your application link a retrieved fact back to the precise moment in a recording - useful for surfacing relevant clips or timestamped citations.

Transcription is available with `StandardMultimodal` and `MultimodalFull` profiles. The time-coded `audio_chunk` segments are embedded and written only under `MultimodalFull`; under `StandardMultimodal` the transcript is kept in the document text only.

Transcript segments participate in **vector recall** alongside text chunks (same embedding model and ranker leg). A query can return an `audio_chunk` hit with the segment text and timestamps even when the parent document chunk spine phrased the content differently.

When a document is **reprocessed**, previous `audio_chunk` and `image_chunk` records are cleared at the start of the run before modality stages execute, so stale transcript or frame records cannot survive a profile or content change.

```python
doc = await memory.documents.upload(
    "support-call.mp3",
    title="Support Call 2026-05-12",
    scopes=["org/acme/user/alice"],
)
```

There is no separate transcription status. The transcript is produced while the document is `extracting`, and the time-coded segments are written before it reaches `ready`.

## CLIP visual embeddings

For image content processed under `MultimodalFull`, SurrealDB Agent Memory generates a CLIP visual embedding alongside any OCR text. CLIP embeddings capture the semantic content of the image independently of its textual labels, enabling retrieval by visual similarity.

When a query is issued against a context that includes images, the CLIP embeddings participate in the vector search alongside text embeddings. A query such as "product dimension diagram" can retrieve a relevant engineering drawing even if that drawing contains no recognisable text.

Text-to-image retrieval needs the image vectors in the same embedding space as the text embedder, so it needs the deployment's Gemini image embedding provider. A separate CLIP provider embeds images into its own space, and those images are not returned for a text query.

## Video captions

Under `MultimodalFull`, SurrealDB Agent Memory generates per-frame captions for video documents. Captions are indexed as additional text chunks associated with the video document. This allows text-based queries to retrieve video content described by the visual scene, not only by the spoken transcript.

Video caption extraction is the most resource-intensive stage and is therefore opt-in, available only with `MultimodalFull`.

## Models for multimodal ingest

OCR, CLIP and speech-to-text providers are set for the deployment, not per Context. The extraction LLM is set per Context.

**Vision LLMs:** under **`MultimodalFull`**, image understanding runs when the worker resolves an extraction LLM for the Context. Per-Context **`models.extraction`** and provider keys apply even without a deployment-wide default LLM. Vision descriptions are appended to the document body **before chunking** (same spine as OCR text). OpenAI and Anthropic providers send image content blocks when media is attached.

**Per-Context ingest LLM:** document jobs resolve the extraction LLM from the Context config on each run (shared budget-enforced resolver with maintenance jobs). A Context with its own provider key gets ingest-time extraction and vision even when the deployment has no default LLM. See [Models per stage](/docs/agent-memory/memory-and-knowledge/tuning/models-per-stage.md).

## Checking pipeline progress

Multimodal documents take longer to process than text-only uploads. The `status` field reflects the current pipeline stage precisely, so you can surface progress information in your application:

```python
import asyncio

stages = []

while True:
    doc = await memory.documents.get(doc.id)
    if doc.status not in stages:
        stages.append(doc.status)
        print(f"Stage: {doc.status}")

    if doc.status in ("ready", "failed"):
        break

    await asyncio.sleep(3)
```

Example output for a video document processed with `MultimodalFull`:

```text
Stage: queued
Stage: extracting
Stage: chunking
Stage: embedding
Stage: keywording
Stage: extracting_nodes
Stage: ready
```

## PII redaction

PII redaction can be enabled per-Context and applies to all ingested content, including OCR-recognised text and speech transcripts, before chunking. See [Extraction tuning](/docs/agent-memory/reference/configuration.md#extraction-tuning) for the **`pii_redaction_enabled`** field.
