Live queries and dashboards

Spectron's observability story is built on SurrealDB's native capabilities. Because all state lives in SurrealDB, you can use live queries to stream real-time updates, build dashboards with direct database queries, and connect any BI tool that can speak to SurrealDB.

Live queries

SurrealDB's LIVE SELECT statement streams changes from any table in real time. This is useful for building real-time dashboards, monitoring ingestion queues, and watching memory state evolve.

Monitor document ingestion

-- Stream status changes for all documents being processed
LIVE SELECT id,
  title,
  status,
  error,
  processing_started_at,
  processing_completed_at
FROM document
WHERE status IN ["queued",
  "extracting",
  "chunking",
  "embedding",
  "keywording"];

Every status transition – queued → extracting → chunking → embedding → keywording → ready – triggers a live update. A failed document triggers an update with status = "failed" and the error field populated.

Monitor extraction activity

-- Stream new turns being processed
LIVE SELECT id, session, role, content, created_at
FROM turn
ORDER BY created_at DESC;

-- Stream new entities as they are extracted
LIVE SELECT id, name, type, memory_category, scope, created_at
FROM entity;

Monitor trace activity

-- Stream real-time operation traces
LIVE SELECT query,
  tier,
  duration_ms,
  token_cost,
  api_key_id,
  created_at
FROM decision_trace;

Use this to watch which queries are hitting the cache versus falling through to retrieval, and to identify unexpectedly slow operations.

Operational dashboards

You can build dashboards by querying SurrealDB directly from any tool that supports SurrealDB as a data source, or by running scheduled queries via the Spectron REST API.

Key metrics to track

Ingestion pipeline health:

SELECT status, count() AS count
FROM document
WHERE created_at > time::now() - 1d
GROUP BY status;

Memory growth rate:

SELECT
    time::format(created_at, "%Y-%m-%d") AS day,
    count() AS new_entities
FROM entity
WHERE created_at > time::now() - 30d
GROUP BY day
ORDER BY day;

Query resolution tier distribution:

SELECT tier, count() AS calls, math::mean(duration_ms) AS avg_ms
FROM decision_trace
WHERE created_at > time::now() - 1d
GROUP BY tier;

A healthy deployment typically shows:

30–60% direct (exact attribute match, very fast)
10–20% cache (semantic response cache hit)
20–50% hybrid (retrieval + synthesis)

Error rate:

SELECT count() AS failed_docs
FROM document
WHERE status = "failed"
  AND created_at > time::now() - 24h;

Cache effectiveness

-- Cache hit rate over the last hour
LET $total = (SELECT count() FROM decision_trace
  WHERE created_at > time::now() - 1h);
LET $cache = (SELECT count() FROM decision_trace WHERE tier = "cache"
  AND created_at > time::now() - 1h);
RETURN ($cache / $total) * 100;

Low cache hit rates (<10%) in a production workload may indicate the cache TTL is too short or the similarity threshold too high.

Surrealist (SurrealDB Cloud UI)

Surrealist is the SurrealDB Cloud dashboard. For Spectron on Cloud, use Surrealist to create contexts, manage Spectron billing, and mint API keys — see Surrealist dashboard quickstart. That is separate from this operator-focused observability page.

For self-hosted deployments, Surrealist can also connect to the raw SurrealDB instance behind Spectron (advanced debugging — not the Spectron product UI):

Host: ws://your-surrealdb:8000/rpc
Namespace: <context namespace>
Database: <context database>

Use this only when you operate the database directly. Cloud customers typically use the context host and REST API instead.

Prometheus metrics

Spectron exposes Prometheus metrics at /metrics:

Metric	Type	Description
`spectron_requests_total`	Counter	Total HTTP requests, labelled by endpoint and status
`spectron_request_duration_seconds`	Histogram	Request latency distribution
`spectron_llm_tokens_total`	Counter	Total LLM tokens, labelled by context and stage
`spectron_extraction_turns_total`	Counter	Total turns processed, labelled by context
`spectron_documents_queued`	Gauge	Documents currently in the ingestion queue
`spectron_cache_hits_total`	Counter	Semantic response cache hits
`spectron_cache_misses_total`	Counter	Semantic response cache misses
`spectron_storage_bytes_written_total`	Counter	Bytes written to object storage
`spectron_storage_bytes_read_total`	Counter	Bytes read from object storage
`spectron_documents_bytes_ingested_total`	Counter	Raw document bytes accepted for ingest

Byte-valued counters omit a doubled bytes suffix in Prometheus — the OTel exporter does not repeat a unit word already present in the metric name.

Use these metrics with Prometheus + Grafana for production dashboarding without direct SurrealDB access.

Alerting recommendations

Set up alerts for:

High ingestion error rate – spectron_documents_queued growing without a corresponding increase in status = "ready" documents
Token budget approaching limit – alert at 80% of the monthly token_limit
Low cache hit rate – below 5% in production indicates a caching configuration issue
Slow operations – P99 latency above 5s for hybrid retrieval
Authentication errors – elevated 401/403 response rates may indicate key rotation issues or misconfigured clients