Observability

Tracing

Retrieval, decision, and response traces stored in the substrate.

Spectron records what happened during reads and writes as three trace kinds — first-class graph nodes in the same database as your memory, not an external log stream alone. Together they answer “why did the system return this?” and “what did we believe when we wrote that?”

Trace kindEmitted byHolds (conceptually)
retrieval_traceEvery /query, /context, and ranked readQuery text, candidate sets, fused scores, returned hits, model metadata
decision_traceEvery reconciliation (turn ingest, document extraction, reflection writes, …)Extraction input, confidence, trust, per-record outcomes, acting principal, optional on-behalf-of target, session scope
response_trace/chat and /reflectUser message, assembled prompt, model response, tokens, cost, latency; may link reused_from when tier-2 cache hits

See Traces and memory evolution for how traces feed back into ranking and consolidation.

Every REST response and MCP tool result includes a trace_id. Pass it to the trace endpoint for the one-hop neighbourhood around that operation:

GET /api/v1/{context_id}/traces/{trace_id}

Example response footer:

{
"tier": "hybrid",
"trace_id": "retrieval_trace:01HF3X..."
}
GET /api/v1/{context_id}/traces?scope[org]=acme&limit=100
GET /api/v1/{context_id}/traces?since=2026-05-01T00:00:00Z&limit=50
GET /api/v1/{context_id}/traces/stats?scope[org]=acme&since=2026-05-01T00:00:00Z

Access control: holders of manage see every trace in a Context. Other callers see only traces whose session scope falls within their read grant — trace listing follows the same boundary model as memory reads.

When an agent returns an unexpected answer:

  1. Read trace_id from the response.

  2. Fetch the trace and check tier — direct lookup, response reuse, hybrid retrieval, or full-context escalation.

  3. For reads, follow retrieval_trace edges to see which entities, attributes, and passages were considered versus returned.

  4. For recent writes, follow decision_trace to see supersession, uncertainty flags, and source kinds.

  5. For /chat, inspect response_trace for token cost and whether reused_from indicates a cache hit.

If a stale response-reuse hit is suspect, invalidate dependent facts or tune reuse thresholds per Context. If the wrong entity was matched, inspect entity attributes and their provenance links.

Alongside the trace graph, Spectron emits structured audit events for operations traces do not fully capture on their own — including destructive deletes, scope-vocabulary changes, background jobs, and denied authorisation attempts. Denied reads and writes are always recorded so security reviews can distinguish “no data matched” from “access was refused”.

For compliance exports, combine trace listing, SurrealDB export of trace tables, and your log pipeline’s capture of audit events. See Profiles and audit trails.

Trace volume grows with traffic. decision_trace records are small and high-value; many deployments retain them longer than retrieval and response traces. Configure per-Context retention via management API or CLI — defaults often keep decision traces indefinitely while pruning retrieval and response traces after roughly 90 days.

Export traces before pruning if your compliance window exceeds the configured retention.

Was this page helpful?