Tracing | Spectron

Spectron records what happened during reads and writes as three trace kinds — first-class graph nodes in the same database as your memory, not an external log stream alone. Together they answer “why did the system return this?” and “what did we believe when we wrote that?”

Trace kind	Emitted by	Holds (conceptually)
`retrieval_trace`	Every `/query`, `/context`, and ranked read	Query text, candidate sets, fused scores, returned hits, model metadata
`decision_trace`	Every reconciliation (turn ingest, document extraction, reflection writes, …)	Extraction input, confidence, trust, per-record outcomes, acting principal, optional on-behalf-of target, session scope

OpenTelemetry spans and data-plane audit events also carry spectron.principal.id (and audit.principal.id on audit rows) — distinct from the authenticating API key id — so operators can correlate work to the resolved principal.

`response_trace`	`/chat` and `/reflect`	User message, assembled prompt, model response, tokens, cost, latency; may link `reused_from` when tier-2 cache hits

See Traces and memory evolution for how traces feed back into ranking and consolidation.

Trace IDs in responses

Trace correlation IDs appear on the wire in camelCase, depending on the endpoint:

Endpoint	Where to find the ID
`POST /query`, `POST /context`	`trace.traceId` on the response
`POST /reflect`	`traceId` on the response
`GET /audit`	`traceId` on each row in `rows`

Pass the ID to the trace endpoint:

GET /api/v1/{context_id}/traces/{traceId}

Example /query footer:

{
  "tier": "hybrid",
  "queryMs": 42,
  "trace": {
    "traceId": "jj3AnFO3Mi1WxFO6",
    "resolutionTier": "hybrid",
    "latencyMs": 42
  }
}

Graph-resident trace tables in SurrealDB use snake_case names (retrieval_trace, decision_trace, response_trace). GET /audit?kind=decision uses the wire enum values decision, retrieval, and response.

Listing and filtering traces

GET /api/v1/{context_id}/traces?scope[org]=acme&limit=100
GET /api/v1/{context_id}/traces?since=2026-05-01T00:00:00Z&limit=50
GET /api/v1/{context_id}/traces/stats?scope[org]=acme&windowHours=24

/traces/stats returns operational aggregates for the requested scope and time window — use windowHours for rolling windows (defaults apply when omitted). Scope-enforcing keys receive zeroed aggregates outside their grant (same rule as trace listing); only callers with grant:manage see Context-wide totals.

Access control: principals with grant:manage or broad memory:read see traces in their region. Other callers see only traces whose session scope falls within their read grant — trace listing follows the same boundary model as memory reads.

Using traces for debugging

When an agent returns an unexpected answer:

Read trace.traceId (from /query) or traceId (from /reflect or /audit).
Fetch GET .../traces/{traceId} and check resolutionTier — direct lookup, response reuse, hybrid retrieval, or full-context escalation.
For reads, follow retrieval_trace edges to see which entities, attributes, and passages were considered versus returned.
For recent writes, follow decision_trace to see supersession, uncertainty flags, and source kinds.
For /chat, inspect response_trace for token cost and whether reused_from indicates a cache hit.

If a stale response-reuse hit is suspect, invalidate dependent facts or tune reuse thresholds per Context. If the wrong entity was matched, inspect entity attributes and their provenance links.

Operational audit events

Alongside the trace graph, Spectron emits structured audit events for operations traces do not fully capture on their own — including destructive deletes, scope-vocabulary changes, background jobs, and denied authorisation attempts. Denied reads and writes are always recorded so security reviews can distinguish “no data matched” from “access was refused”.

GET /audit returns records with a TraceKind filter (decision, retrieval, or response) when correlating operational audit with graph-resident traces.

For compliance exports, combine trace listing, SurrealDB export of trace tables, and your log pipeline’s capture of audit events. See Profiles and audit trails.

Trace retention

Trace volume grows with traffic. decision_trace records are small and high-value; many deployments retain them longer than retrieval and response traces. Configure per-Context retention via management API or CLI — defaults often keep decision traces indefinitely while pruning retrieval and response traces after roughly 90 days.

Export traces before pruning if your compliance window exceeds the configured retention.