Historical and archaeological data

Left to right: the Domesday volumes, engraved for William Andrews, "Historic Byways and Highways of Old England" (1900). The amphitheatre dedication of Quinctius Valgus and Marcius Porcius, Pompeii, photographed by Gary Todd. A figurine of the Vinča culture, Cleveland Museum of Art 2000.202.

Overview

SurrealDB Agent Memory can be used to answer questions that no single historical record answers on its own.

Land surveys, inscriptions, census returns, excavation catalogues, and parish registers share a shape, but tend to be short and fragmentary and may lack a stable identifier. The useful answers in this case come from stitching, in which SurrealDB Agent Memory composes many scattered records into one claim.

This page describes the general workflow:

• •

  Prepare a corpus and keep it checkable.

• •

  Stamp historical time rather than ingest time.

• •

  Choose questions the material can answer.

• •

  Widen retrieval enough for totals.

• •

  Prove that an answer came from your corpus, not from a model's training data.

Three case studies inform the advice. The first is a 1086 Domesday survey of one English hundred, at 52 records. The second is 127 Roman electoral notices from the walls of Pompeii, painted before the disaster in AD 79. The third is 91 site reports built from 1,822 radiocarbon measurements from the Serbian Danube (largely from the Vinča culture), in which one excavation appears under as many as six spellings.

What makes these corpora distinct

Historical and archaeological records are painstakingly put together and are not frequently updated. This leads to a number of interesting properties that must be taken into account:

Some tips for working with such data are:

Ingest prose, not a graph

Do not resolve entities in your own pipeline. If you upload holding → held_by → william_of_warenne, you have already done the identity work, and SurrealDB Agent Memory cannot help with the hard part.

Write each record the way the source reports it, with no identifiers. Extraction then works out that the William of Warenne in one place and the one in another are the same man.

A prose record from a land survey looks like this:

Scottow, in the hundred of South Erpingham in Norfolk.

Before the Conquest, in King Edward's time:
one free man held land assessed at 0.10 carucates for geld.

In 1086: Ribald brother of Count Alan holds it of Count Alan,
who holds of the King, worth £1 4s.

The survey records a mill for grinding corn,
6 acres of meadow and woodland for 20 pigs here.

Domesday Book, Norfolk 4,36.

Three rules hold for every corpus type:

• •

  One record per source unit. Use the unit a historian cites: one survey entry, one inscription, one burial. Citations then match the scholarly reference, so a reader can check any claim.

• •

  Keep the source text. Where the original is in another language or notation, put it in the record beside the gloss. The record stays checkable, and keyword retrieval gets the original terms.

• •

  End with the citation. Retrieval returns the citation with the passage, so every answer carries provenance.

For an inscription corpus, one record carries both layers:

A campaign notice painted on a street wall at Pompeii.

The muleteers ask the voters to elect Gaius Iulius Polybius as duumvir.

The Latin reads: caium iulium polybium iivirum muliones rogant.

CIL 04, 00113.

Upload records as documents rather than as many synchronous facts. See Documents (upload).

Normalise surface forms, not identities

Normalise the spelling of a name. Do not merge two people by hand.

• •

  Put inflected names into one citation form. Latin Marcum Holconium Priscum becomes Marcus Holconius Priscus. The same man then reads identically wherever he appears. Do the same for declined place names.

• •

  Expand editorial conventions into running prose. Print editions mark supplied letters (M(arcum)) and lost text ([3]). Strip the brackets for readable text. Keep a marker where the source is illegible.

• •

  Leave similar names alone. Holconius, Marcus Holconius, and Marcus Holconius Priscus may be one person or three. Let resolution and reconciliation decide. A manual merge destroys the evidence for the decision.

• •

  Keep the "nothing recorded" sentinel distinct from zero. Sources use -, n/a, or a blank to mean not surveyed. Write "the survey records no mill here", or omit the sentence. Never write "0 mills".

• •

  Say what the blank applies to. "No coordinates are recorded under this spelling" is a better sentence than "no coordinates". The first tells a reader, and the retrieval layer, that another spelling may hold the value. The second reads as a fact about the place.

Keep the join key

Some corpora carry an identifier that is not a name: a laboratory sample number, a museum accession number, a catalogue reference, a DOI. Where one exists, keep it in the record text and keep it verbatim. Two records that cite the same identifier describe the same subject, whatever their names say. Resolution can then be scored instead of argued.

Identifiers settle identity questions that names cannot. Several spellings that read as variants of one name may resolve on shared sample numbers into two separate subjects, hundreds of kilometres apart. Spellings that look unrelated may resolve into one. String similarity gets both cases wrong, and an identifier gets both right.

An identifier also gives you a second, independent check on an answer. Where a claim rests on two records being about one subject, you can confirm the join yourself.

Stamp historical time, not ingest time

By default, derived facts date to the moment you upload them. For historical material that date is wrong, and it breaks every temporal question. Set observedAt on upload to the known time of the record.

POST /api/v1/{context_id}/documents
Content-Type: multipart/form-data

file=<binary>
metadata={"title":"scottow-4-36","observedAt":"1086-01-01T00:00:00Z","labels":["corpus=domesday","place=scottow"]}

Reads then accept asOf. You can ask what the corpus attests for one moment rather than for all time. See Temporal validity and spoiler-safe narrative memory for more on this subject.

Decide three cases before you ingest:

• •

  A record that describes two eras. Survey entries often report a holding before and after a conquest. Split the record in two, with one observedAt each. Or keep one record and give each era its own paragraph, with the date stated in prose. Do not interleave the eras in one sentence.

• •

  A date that is a range. Stamp observedAt with the earliest plausible instant. State the range in prose ("between AD 62 and AD 79"). Synthesis quotes the prose. Ordering uses the timestamp.

Some corpora must hide later material from the reader. An excavation published in phases works this way, as does a narrative released in instalments. The same mechanism gives spoiler-safe playback.

Label provenance and certainty

Labels are key=value tags for filtering. On a historical corpus they carry the distinctions a historian needs.

certainty matters most. A gap an editor filled is not evidence, nor is your own inference. State the distinction on the label and in the prose, because synthesis reads the prose.

Choose questions the corpus can answer

Question shape matters more than prompt wording. It decides which surface to ask, and whether you drive retrieval yourself.

Set breadth to match the question

k on /query defaults to 10 and deployments cap it at 50. The internal retrieval pool is set separately from k. See Answer size vs search breadth.

A count needs a window wide enough to cover every record that bears on it. Where the candidate set is larger than the window, the answer describes what the window saw rather than what the corpus holds. Raising k to the size of the candidate set completes it. Nothing is missing from the index in the meantime, only unseen.

Records in one corpus share vocabulary, so relevance scores tend to band tightly. Rank order inside a narrow band carries little information. What a count sees therefore depends on the width of the window rather than on the ranking.

Counts are worth driving yourself, with k set from what you already know about the corpus:

import os
import httpx

SPECTRON = os.environ["SPECTRON_BASE_URL"].rstrip("/")
CTX = os.environ["SPECTRON_CONTEXT_ID"]
KEY = os.environ["SPECTRON_API_KEY"]

client = httpx.Client(
    base_url=f"{SPECTRON}/api/v1/{CTX}",
    headers={"Authorization": f"Bearer {KEY}", "api-version": "1"},
    timeout=60.0,
)

def hits(query: str, k: int = 50) -> list[dict]:
    r = client.post("/query", json={"query": query, "k": k})
    r.raise_for_status()
    return r.json()["hits"]

# Fan out one query per subject instead of asking for a ranking in one call.
places = ["Scottow", "Aylsham", "Wickmere", "Calthorpe"]  # from your own index
mills = {place: hits(f"mill for grinding corn at {place}") for place in places}

for place, found in mills.items():
    print(place, len(found))

Three practices follow:

1. 1.

   Enumerate subjects from your own index. You know which places or people exist. Ask once per subject, then combine the results yourself.

2. 2.

   Report coverage with every total. State how many records contributed. "£51 from 11 of 52 records" is usable. "£51" misleads.

3. 3.

   Check the margin before you report a ranking. Where the leader wins by a wide margin, the order holds up on partial coverage. Where the field is close, the order depends on having seen every contributor. Widen k until the coverage line stops moving, then name a winner.

Enumerate both sides of a set operation

Coverage acts on a total and on a set operation differently, and the difference is arithmetic rather than anything to do with reasoning.

A total built from part of the corpus comes out low, in proportion to what it saw. A coverage line makes the shortfall visible. An intersection built from part of the corpus is not a partial intersection: it is the intersection of two samples, which is a different quantity, and its size gives no clue to the true one.

The gap can be wide. Suppose two entries hold a hundred sample numbers each, and the evidence names three of them. An intersection over that evidence can report no overlap at all between entries that share dozens. Every identifier in the evidence may be genuine, and every step over it may follow. Set operations do not survive sampling.

So treat intersections, differences, and "do these two overlap" questions the way you treat totals. Enumerate both sides with /query at raised k, then compute the set operation in your own code.

import re

# Whatever shape the identifier takes in your corpus: OxA-13613, Bln-873, AA-57774.
LAB_NUMBER = re.compile(r"\b[A-Z][A-Za-z]{1,3}-\d{3,5}\b")

def samples(entry: str) -> set[str]:
    found = set()
    for hit in hits(f"laboratory samples listed for {entry}", k=50):
        found.update(LAB_NUMBER.findall(hit["text"]))
    return found

shared = samples("Vlasac") & samples("Lepenski Vir")
print(f"{len(shared)} shared, from {len(samples('Vlasac'))} and {len(samples('Lepenski Vir'))}")

Bridge period vocabulary and modern questions

Sources use the vocabulary of their time and readers ask in theirs. "Where can I get bread?" and a mill for grinding corn have no words in common. A modern question can therefore miss the records that answer it, or be read as a question about the modern world. Retrieval may return the right records while the wording at both ends keeps them apart. The distance to close sits in the wording, not in the index.

Close it from either end. Four ways, cheapest first:

• •

  Ask in period terms. Re-framed as "the year is 1086; a traveller wants corn ground into flour", the same corpus answered well.

• •

  Add one bridging sentence per record. A short modern gloss ("A mill grinds corn into flour for bread.") gives retrieval and synthesis the hop. Keep it separate from the source text.

• •

  Upload a glossary. Put period terms and their modern equivalents in one document. The vocabulary then sits in the corpus, not in your prompts.

• •

  Turn on query expansion for document retrieval. /documents/query accepts useHyde and decomposeQuery. Both help when query wording and corpus wording diverge. See Hybrid search.

Where the corpus names things plainly, the gap does not appear. A corpus that says "the bakers" outright needs no bridge for the bread question.

Handle records that cover several subjects

One survey entry can cover ten places. One inscription can name several candidates. These records retrieve well, and what a reader can do with them depends on how you write them. A leading list of ten names followed by figures for the group supports a group answer, because that is the only claim the record makes. There is nothing in it to attach to any single place. A record written that way yields group answers, for that reason.

Write each subject so it carries its own claim. Two options:

• •

  Repeat each subject in its own sentence inside the record. Each name then appears with its own resources, not only in a leading list.

• •

  Emit one record per subject. Each carries the shared citation and a group=<citation> label, plus a sentence that states the source entry covers several subjects and shares the figures.

The second option costs duplication and buys attribution. Prefer it when per-subject questions matter.

Apply synthesis

Synthesis turns a fragmentary corpus into an answer. The claims worth the operation are the ones no single record makes:

• •

  A change in structure across a period. One form of tenure or allegiance gives way to another. The claim comes from the shape of the graph at each end, not from any one entry.

• •

  Two subjects acting together. A shared ticket, partnership, or workshop, read from co-occurrence across separate records.

• •

  Rivalry. Subjects contending for the same office, market, or territory in the same season, which no record states outright.

• •

  A value recovered under another name. The record you asked about holds the subject but not the value, and a variant spelling holds the value.

The last is the clearest case for the pattern. The value sits in the corpus under a name nobody would think to search for, and no single record could produce the answer. Ask for the source spelling alongside the value, so the recovery stays checkable.

Use the operations in this order:

1. 1.

   /elaborate before relational questions, so links are pre-formed rather than inferred inside one answer window.

2. 2.

   /consolidate to merge name variants into single entities. Run it before any question that depends on counting people.

3. 3.

   /reflect for interpretive claims across many records. Reflection always calls a model. It returns synthesised text with its evidence. See Reflection.

essay = client.post(
    "/reflect",
    json={
        "query": (
            "Across these records, how did the pattern of lordship change "
            "between 1066 and 1086? Cite the records for each claim."
        ),
        "persist": False,
    },
).json()

Ask for reasoning to be shown. A structural claim with its supporting records attached is checkable. The same claim alone is not.

Prove the answer came from the corpus

Most well-known historical sources sit in model training data, so a correct answer does not prove that retrieval worked. Here are some tips to work with sources that may already form part of general knowledge known by LLM tools:

Perturbation is the strongest of the four, because it separates retrieval from prior knowledge directly. Rewrite one record with an invented holder and value. Delete the original. Ingest the forgery, then ask. If the answer names the invented holder and never the real one, the answer came from your corpus.

# 1. remove the genuine record
curl -X DELETE "$SPECTRON_BASE_URL/api/v1/$SPECTRON_CONTEXT_ID/documents/$DOC_ID" \
  -H "Authorization: Bearer $SPECTRON_API_KEY"

# 2. ingest the altered record in its place
curl -X POST "$SPECTRON_BASE_URL/api/v1/$SPECTRON_CONTEXT_ID/documents" \
  -H "Authorization: Bearer $SPECTRON_API_KEY" \
  -F "file=@perturbed/cawston.txt" \
  -F 'metadata={"title":"cawston-perturbed","labels":["control=perturbation"]}'

Run the held-out control with a real subject from a neighbouring region, not an invented one. A model that knows the source may recognise a genuine name, so a refusal to answer about it is the stronger result. Keep the invented-subject probe as well. It catches confabulation rather than leakage.

Restore the perturbed record when the control finishes. Note in your results that the corpus was altered.

Build ground truth that can adjudicate

Score answers rather than read them. Compute the expected totals, graphs, and counts from the same source data that generated the records. Store them beside the corpus.

Two failures are common.

• •

  A count by name string fragments people. One man spread across five spellings can score 3 where the merged figure is 9. An answer that reasons from rank or office rather than raw frequency is then marked wrong while being defensible. Merge name variants before a count-based figure adjudicates "most prominent" or "wealthiest".

• •

  A join can lose records silently. Where a citation key maps many-to-many in both directions, an inner join drops rows without an error. Check row counts at each join. Verify the total against a figure published with the source.

Record which source records produced each ground-truth figure. When an answer disagrees, you can then find out whether the answer or the ground truth is wrong.

Include pairs that must not merge

Ground truth that holds only true merges cannot detect over-merging. A system that joins everything on string similarity scores full marks against it.

Add negative controls: subjects whose names invite a merge that the evidence does not support. Most corpora hold good cases already. Look for two names that differ by a letter or two and sit in the same region. Keep the pair if they share no identifiers, and if the evidence puts them a thousand years or a few hundred kilometres apart. For example, Banja and Banjica, or Hajdučka Vodenica and Hadučka Vodenica.

A passing distinct-pair control is what makes the merges elsewhere worth reporting. Record the expected verdict for each pair, not only the expected clusters.

Build three kinds of case into the same file:

Notes for bulk ingest

• •

  Verify one write first. Post a single small document to a new Context. Confirm it becomes retrievable before you send hundreds.

• •

  Confirm retrievability, not only status. Document processing is asynchronous. A record can answer queries while its reported stage still lags. Gate your run on a successful /query for a known string.

• •

  Page correctly when you list documents. GET /documents is zero-indexed and uses camelCase parameters (page=0&pageSize=100).

• •

  Poll gently during ingest. Listing endpoints stay under load while the pipeline drains. Back off and retry rather than poll tightly.

• •

  Use a fresh Context per experiment. Operator questions and prior answers become hits, so a reused Context contaminates the next run. See Contexts and scope.

• •

  Keep the answer out of metadata. A document title, label, or probe id that names the expected result teaches the model the answer.

• •

  Check the count out against the count in. Any normalisation in your own pipeline can perform the resolution you are trying to measure. A filename function that maps every non-ASCII character to _ collapses č and a mojibake variant onto one name. Documents then overwrite each other before the upload starts, and part of the resolution arrives pre-solved. The run still looks like a clean pass. Compare the number of subjects in your source data with the number of files, documents, and dictionary keys at every stage.

Checklist

• Records are prose with no pre-resolved identifiers
• One record per citation unit, ending with its citation
• Original-language text sits beside the gloss
• Laboratory, accession, or catalogue numbers kept verbatim in the record
• Name spellings normalised; identities left unmerged
• "Nothing recorded" stays distinct from zero, and names the spelling it applies to
• observedAt set to historical time; two-era records split or dated in prose
• certainty stated on labels and in prose
• Aggregate and set questions use fan-out /query at raised k
• Set operations computed in your own code, not read out of an answer
• Every total reports how many records contributed
• Period vocabulary bridged by a gloss line or a glossary document
• Multi-subject records either repeat subjects or split with a shared citation
• /consolidate run before any count-based question
• Synthesis persisted with derived=true or to a separate Context
• Empty-context, perturbation, held-out, and invented-subject controls all run
• Ground truth holds merge, split, and distinct cases, and records its contributing records
• Document count out matches subject count in, at every pipeline stage