Multimodal content

SurrealDB Agent Memory can ingest more than text. Images, audio recordings, and video files are all first-class document types in authoritative knowledge. Depending on the Context's ingestion profile, it will apply OCR, speech-to-text transcription, and visual embedding to extract retrievable content from non-textual sources.

Ingestion profiles

Context.config.ingestion_profile selects which provider-bearing stages run. Profiles are monotonic (each richer profile is a strict superset). Stages whose provider is absent are skipped, not errored.

Default: MultimodalFull. Uploaded images and audio get content understanding (vision caption, CLIP, audio embeddings) when the backing providers are configured. Dial down to StandardMultimodal, TextPlusKeyword, or TextOnly to reduce cost.

Wire names are PascalCase as in the table (MultimodalFull, not multimodal_full).

The profile is a Context setting, ingestion_profile in the Context configuration, and applies to every upload in that Context. An upload request has no profile field, so an image is uploaded like any other document:

doc = await memory.documents.upload(
    "product-diagram.png",
    title="Product Architecture Diagram",
    scopes=["org/acme"],
)

const doc = await memory.documents.upload({
    file: imageFile,
    contentType: "image/png",
    title: "Product Architecture Diagram",
    scopes: ["org/acme"],
});

Supported formats

Images

Audio

Video

OCR

OCR (optical character recognition) recognises printed and handwritten text in images and scanned PDFs. The recognised text is extracted, split into chunks, and embedded in exactly the same way as text from a native PDF or HTML document. Chunks do not record whether their text came from OCR.

OCR is available with StandardMultimodal and MultimodalFull profiles.

Scanned and low-text PDFs

When a PDF has no extractable text layer (image-only / scanned pages), SurrealDB Agent Memory first sends the whole PDF to the Gemini document parser when the deployment has one. If there is no parser or it recovers no text, it rasterises pages and runs OCR when the deployment has an OCR provider. When neither recovers text, ingest fails loudly with an actionable error - the document never lands in ready with an empty body.

For partially scanned PDFs (a sparse text layer on some pages), a low-text trigger rasterises and OCRs all pages when the recovered text exceeds what the native layer provided - all-or-nothing per document, not per-page splicing.

Embedded PDF images

PDFs with embedded JPEG images (including common filter chains such as ASCII85/Hex + DCTDecode) extract image bytes and run the same OCR, vision, and CLIP path as standalone image uploads. OCR and vision text is folded into the chunk spine before embedding; visual embeddings land on image_chunk records linked to the page's knowledge_section.

doc = await memory.documents.upload(
    "scanned-invoice.pdf",
    title="Invoice 1042",
    scopes=["org/acme"],
)

OCR runs while the document status is extracting. Once the document is ready, its chunks are searchable by text query.

Speech-to-text

For audio and video documents, SurrealDB Agent Memory generates a full transcript and appends it to the chunk body so the spoken content is retrievable by semantic and keyword search. In addition to the full transcript, the pipeline produces time-coded audio_chunk segments that record approximately which part of the recording a retrieved chunk came from.

An audio_chunk segment record looks like this, with times in seconds from the start of the recording:

{
  "document": "document:01hx9…",
  "position": 3,
  "start_seconds": 14.2,
  "end_seconds": 28.7,
  "source": "stt",
  "text": "The return window for unopened items is thirty days from purchase."
}

This lets your application link a retrieved fact back to the precise moment in a recording - useful for surfacing relevant clips or timestamped citations.

Transcription is available with StandardMultimodal and MultimodalFull profiles. The time-coded audio_chunk segments are embedded and written only under MultimodalFull; under StandardMultimodal the transcript is kept in the document text only.

Transcript segments participate in vector recall alongside text chunks (same embedding model and ranker leg). A query can return an audio_chunk hit with the segment text and timestamps even when the parent document chunk spine phrased the content differently.

When a document is reprocessed, previous audio_chunk and image_chunk records are cleared at the start of the run before modality stages execute, so stale transcript or frame records cannot survive a profile or content change.

doc = await memory.documents.upload(
    "support-call.mp3",
    title="Support Call 2026-05-12",
    scopes=["org/acme/user/alice"],
)

There is no separate transcription status. The transcript is produced while the document is extracting, and the time-coded segments are written before it reaches ready.

CLIP visual embeddings

For image content processed under MultimodalFull, SurrealDB Agent Memory generates a CLIP visual embedding alongside any OCR text. CLIP embeddings capture the semantic content of the image independently of its textual labels, enabling retrieval by visual similarity.

When a query is issued against a context that includes images, the CLIP embeddings participate in the vector search alongside text embeddings. A query such as "product dimension diagram" can retrieve a relevant engineering drawing even if that drawing contains no recognisable text.

Text-to-image retrieval needs the image vectors in the same embedding space as the text embedder, so it needs the deployment's Gemini image embedding provider. A separate CLIP provider embeds images into its own space, and those images are not returned for a text query.

Video captions

Under MultimodalFull, SurrealDB Agent Memory generates per-frame captions for video documents. Captions are indexed as additional text chunks associated with the video document. This allows text-based queries to retrieve video content described by the visual scene, not only by the spoken transcript.

Video caption extraction is the most resource-intensive stage and is therefore opt-in, available only with MultimodalFull.

Models for multimodal ingest

OCR, CLIP and speech-to-text providers are set for the deployment, not per Context. The extraction LLM is set per Context.

Vision LLMs: under MultimodalFull, image understanding runs when the worker resolves an extraction LLM for the Context. Per-Context models.extraction and provider keys apply even without a deployment-wide default LLM. Vision descriptions are appended to the document body before chunking (same spine as OCR text). OpenAI and Anthropic providers send image content blocks when media is attached.

Per-Context ingest LLM: document jobs resolve the extraction LLM from the Context config on each run (shared budget-enforced resolver with maintenance jobs). A Context with its own provider key gets ingest-time extraction and vision even when the deployment has no default LLM. See Models per stage.

Checking pipeline progress

Multimodal documents take longer to process than text-only uploads. The status field reflects the current pipeline stage precisely, so you can surface progress information in your application:

import asyncio

stages = []

while True:
    doc = await memory.documents.get(doc.id)
    if doc.status not in stages:
        stages.append(doc.status)
        print(f"Stage: {doc.status}")

    if doc.status in ("ready", "failed"):
        break

    await asyncio.sleep(3)

Example output for a video document processed with MultimodalFull:

Stage: queued
Stage: extracting
Stage: chunking
Stage: embedding
Stage: keywording
Stage: extracting_nodes
Stage: ready

PII redaction

PII redaction can be enabled per-Context and applies to all ingested content, including OCR-recognised text and speech transcripts, before chunking. See Extraction tuning for the pii_redaction_enabled field.