Telemetry (OTLP)

SurrealDB can be monitored by enabling built-in observability.

Enable observability

To enable observability, the SURREAL_TELEMETRY_PROVIDER environment variable has to be set to otlp. If set to anything else, no observability will be available.

If enabled, SurrealDB sends metrics and/or traces to an OpenTelemetry Collector. Configuration of the collector follows the OpenTelemetry environment variable conventions. The most important variable is OTEL_EXPORTER_OTLP_ENDPOINT. By default this points at localhost. Set it to the gRPC endpoint of your collector. For example, if your collector named my-collector runs in Kubernetes in the monitoring namespace:

OTEL_EXPORTER_OTLP_ENDPOINT="http://my-collector.monitoring.svc.cluster.local:4317"

Metrics can be disabled (even if SURREAL_TELEMETRY_PROVIDER is set to otlp) by setting SURREAL_TELEMETRY_DISABLE_METRICS to true. Similarly, traces can be disabled with SURREAL_TELEMETRY_DISABLE_TRACING=true.

Metrics

Metrics are gathered every minute and sent to the collector. The following metrics are present:

The metrics are shown here in the form required by the OpenTelemetry Metrics Semantic Conventions with a . separator. When ingested into Prometheus the . separator is replaced with _. For example rpc.server.active_connections becomes rpc_server_active_connections.

For async runtime introspection (tasks, polls, scheduling), you can use the Tokio console which is separate from OTLP.

For server logging (levels, JSON, files, sockets, slow-query logging), see Logging.

From SurrealDB 3.1, OpenTelemetry is the single source of truth for metrics and, where configured, for log records emitted on the shared logger provider — including audit and slow-query events on SurrealDB Enterprise when you opt in with SURREAL_AUDIT_OTEL_EXPORT / SURREAL_SLOW_QUERY_OTEL_EXPORT. One meter provider and one logger provider route measurements to multiple exporters, including the Prometheus text exposition at GET /metrics. An OTLP subscriber receives the same metric surface that a fully authenticated Prometheus scrape sees, plus traces and any log signals you have enabled (subject to the /metrics allowlist only for the pull path, not for OTLP).

For Prometheus pull — allowlists, naming, migration from older series, multi-tenant guidance, and Community versus Enterprise context — see Observability (metrics and Prometheus) and Enterprise observability.

What flows over OTLP

Configure the OTLP push pipeline with:

SURREAL_TELEMETRY_PROVIDER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317   # gRPC by default

When SURREAL_TELEMETRY_PROVIDER is unset (or any value other than otlp), OTLP exporters are not built; metrics still flow to Prometheus when /metrics is enabled. SurrealDB Enterprise audit and slow-query file sinks are independent of OTLP — they keep writing whenever configured.

When SURREAL_TELEMETRY_DISABLE_METRICS=true, the OTLP metrics reader is skipped while logs and traces continue to push.

When SURREAL_TELEMETRY_DISABLE_TRACING=true, the OTLP trace exporter is skipped while metrics and logs continue to push.

Push interval

Metric push frequency follows the OpenTelemetry specification and is controlled by OTEL_METRIC_EXPORT_INTERVAL (milliseconds; default 60000). The SDK reads this at startup and applies it to the periodic reader behind OTLP metrics. Sub-minute intervals (10–15 s) are common when you want responsive dashboards.

Process metric freshness

surrealdb.process.memory and surrealdb.process.cpu_percent are observable gauges backed by a process-wide cache. A background task refreshes the cache on a fixed cadence so values stay fresh whether scrapers hit /metrics or OTLP pushes on its own schedule.

The task runs only when at least one metrics reader is configured (Prometheus and/or OTLP). OTLP-only deployments get the same freshness guarantee as Prometheus scrapers.

Histogram bucket views

The meter provider applies three default views:

• Instruments named *.duration with unit seconds use a quasi-exponential bucket family from 5 ms to 30 s.

• Instruments named *.duration with unit milliseconds use a parallel millisecond-scale family for the legacy HTTP/RPC pipeline.

• Instruments named *.size with unit bytes use a 1 KiB–100 MiB byte family.

Operators with custom views can override these before building the provider; refer to the OpenTelemetry SDK documentation.

Backward-compatibility instrument set

Legacy OpenTelemetry HTTP/RPC instruments are still recorded under separate meter scopes (surrealdb.http, surrealdb.rpc). They coexist with the new surrealdb.* families so existing OTLP dashboards that pivot on semantic-convention names keep working.

Deprecated environment variables

The following variables are still parsed for backwards compatibility but are no longer applied. The server logs a deprecation warning at startup if either is set:

• SURREAL_TELEMETRY_NAMESPACE — the namespace attribute was removed from telemetry metrics because it identifies tenants in multi-tenant deployments.

• SURREAL_TELEMETRY_RPC_LIVE_ID — per-notification OTLP attribution by rpc.live_id was removed when WebSocket telemetry was unified into the execution observer pipeline.

Local development

From the repository root, start the observability stack under dev/docker (collector, Grafana, Prometheus, Tempo, and Loki):

docker compose -f dev/docker/compose.yaml up -d
SURREAL_TELEMETRY_PROVIDER=otlp OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" surreal start

Open http://localhost:3000 in a browser; default Grafana credentials are admin / admin.

For OpenTelemetry log and trace verbosity, file output, sockets, and other logging flags used alongside this stack, see Logging and surreal start.