Integrating Ultima VII with Spectron, part II: the testing

Engineering

Aug 6, 20268 min read

Dave MacLeod

Show all posts

Integrating Ultima VII with Spectron, part II: the testing

This is a follow-up to last week's post that explains what makes the RPG Ultima VII an ideal solution for testing out a memory context layer like Spectron.

As the second of three posts, in this one we will get into how the integration was put together and what sort of data was sent to Spectron. The integration itself can be seen at this link if you'd like to try it out yourself.

Our newsletter

Get tutorials, AI agent recipes, webinars, and early product updates in your inbox every two weeks

Setting up testing

Since testing a game requires actually playing it for a while, I wanted to have a way to see the actual data being fed to Spectron as it happened. The Exult engine used to play the game keeps track of game events, and those are easy enough to pipe into a window in the terminal.

However, I also wanted to see the responses from Spectron at the game progressed.

To do this, a "sidecar" was put together that uses egui (a GUI built in Rust) to show the gamer the mind of the Avatar (the main character) as the game progresses. The idea is to have the sidecar sit in between the game and Spectron as it does the following:

Send info to Spectron as you play the game
Query Spectron at certain points to show what the Avatar might be thinking. For the gamer, this should provide helpful hints about what is going on and what to do next.

Here are some examples of what I wanted the Avatar's mind to display as the game progressed:

"I just finished talking with Finnigan. Maybe I'll ask him about the Fellowship next time." (hinting at dialog questions that were never clicked on)
"I now know this fighter is Leavell, a bodyguard for Master Robin." (when learning more about an NPC)
"The skies are clearing now, after the rain. It's a clear day for walking." (change in weather state)

It also includes a thin observer added to the original Exult code that posted certain events to an endpoint (in this case http://127.0.0.1:8765/event) that the Rust app could pick up. These events were posted to Spectron, with other POST methods used to retrieve the Avatar's current thoughts.

The setup looked like this:

  Ultima VII (Exult)              Sidecar (Rust / egui)           Spectron
  ==================              =====================           ========
  play Britannia                  mind window (thoughts)
         |                               ^
         | talk / walk / sign / …        | first-person /chat
         v                               |
  spectron_bridge.cc  --HTTP JSON-->  /event listener ---------> POST /facts
  (C++ in Exult)      localhost:8765     |                       POST /chat
                                         |                         |
                                         +-- prompts filled -------+
                                            (player never types)

You don't just want to funnel everything Exult could possibly give you, as that would quickly just turn Spectron into an omniscient tool. The goal here was to send just the events that most clearly represented the experience of the Avatar and the gamer so that Spectron could get data that most clearly represents a real-life experience. This post is mostly about how this refinement went.

Let's take a look at the events sent to Spectron and how they were refined during the testing.

Moving and looking

Exult tracks every movement and direction you make. But we in real life don't track our movements that way. Instead, we become gradually aware of large changes in space and the scenery around us.

To represent that, the sidecar sends an event every 50 steps that shows the net change in direction. For example, if you moved a few steps to the top right and then went down, it would send something like "moved about 30 steps SSE".

During the testing I ended up adding more and more sight- and location-based data as time went on, as long as they represented information that a real person or character would have.

I first started by adding experiences that came from direct mouse clicks:

Conversations

What you see when you click on something ("a nobleman", "a sign", "a dog")
Signs. This part was particularly interesting as Ultima signs are traditionally written in runic, while in Ultima VII a new group called the Fellowship has shown up that promises a more "modern" interpretation of how to live life. (i.e. a cult is beginning to take over the world) Their signs are written in the Latin alphabet. So I had the sidecar transcribe the runes as a convenience, while marking Latin signs as a little unsettling for the Avatar.

This worked fairly well, though I soon realised that the experience was still a fairly blind one because it required you to directly click on trees, walls, and all the rest for Spectron to have any idea of where the Avatar is. I needed to give it more of the passive visual info that you get as the Avatar.

Since Exult does provide the Avatar's current view at all times (all the tiles visible on the screen), I opted to pass that on every fifty steps along with the direction moved. This feels pretty organic too: "I've now moved in this direction and I see X and Y" is basically how we work as humans.

At first this resulted in too much information, as now the Avatar was "viewing" every object and item inside the gaming area, including the insides of houses. Fortunately, there is enough context to refine this by map chunk, in which it's inside if Map_chunk::is_roof(), which can be used on the Avatar's current spot as well to see if you are inside or outside and filter based on that.

Encountering people

I first had Exult pass on all the details emitted from every conversation, and refined from there. At first there was too much information, including internal details such as official NPC numbers names even before you've asked them their name. Even this charming fellow who refuses to give you his name ("I am a guard") apparently has the internal name "The Juggernaut" which was likely what the developers at Origin Systems (the creators of Ultima) decided to call him. That was too much information to pass on to Spectron.

So the info passed on to Spectron was slimmed down. One other notable change is that since every NPC conversation always has the options name and job, this allowed the Rust structs that worked with this info to add them as fields which made the shape sent to Spectron a bit more predictible.

#[derive(Debug, Clone, Default)]
pub struct NpcCard {
    /// Exult schedule slot when known (`-1` / `0` = unknown).
    pub npc_id: i32,
    /// Shape / role label from the conversation face ("a paladin", "Spark").
    pub appears_as: String,
    /// Personal name if learned (or Spectron reports one).
    pub name: Option<String>,
    /// Job / role if learned.
    pub job: Option<String>,
    /// Spectron (or local stub) summary of what the Avatar knows.
    pub summary: String,
}

Dying and reloading

Dying and reloading are two events where in-game character experiences can diverge from those of you, the player.

After dying: In some games you go back to a certain savepoint, while in others the character miraculously revives with memories intact. In Ultima VII you wake up in a different location but have your memories intact.
After reloading: All games erase the in-game character's memory here.

Here, I erred on the side of gamer knowledge and didn't try to match Spectron's memory with the slots where progress is saved. That said, this can be implemented in Spectron. The next post is the one that focuses on Spectron's internal details, so we will get into that subject then.

What the sidecar sent to Spectron

Here are some examples of what Spectron actually received through the sidecar while playing the game. Most of these are sent to the /facts endpoint.

After a conversation with a guard

Note the instructions that precede it; those ensure that unnecessary details don't show up in the Avatar's thoughts.

{
  "text": "Ultima VII conversation with npc_id=14 (appeared as: guard).\nUse a personal name only if the transcript shows it was revealed in dialogue; otherwise keep the role label.\nThis is spoken testimony — treat it as what was said, not as proven fact.\n…\nTranscript:\nYou see an alert and no-nonsense guard.\n\"Johnson.\"\nAvatar chooses: name\nAvatar chooses: job\n\"I have the morning watch guarding the dock…\"\n…\n\"Thou dost not know the password… The Mayor may give thee the proper password.\"\nAvatar chooses: bye",
  "infer": "full",
  "memory_category": "context",
  "labels": [
    "source=exult",
    "kind=conversation",
    "authority=spoken"
  ]
}

Reading books

Since you can read books in Ultima VII, this data gets sent to Spectron too every time the Avatar clicks on one.

{
  "text": "Epistemic frame: the Avatar has just READ this document in Ultima VII. That is a glance / study event…\nTitle (as printed): \"THE HONORABLE HOUND\".\nHow to store this:\n- Prefer ONE document / written-work memory…\n- If this is a guest register… store that named place…\n…\nFull text:\nTHE HONORABLE HOUND\nRegister*\n~ ~Walter of Britain\nJaffe of Yew\nJaana\n…",
  "infer": "full",
  "memory_category": "knowledge",
  "labels": [
    "source=exult",
    "kind=book",
    "authority=written",
    "epistemic=read"
  ]
}

Viewing signs

Signs have the sign itself as well as the letters used (runic or not). Some weather info is passed on too.

{
  "text": "The Avatar read a distinct plaque or street sign. The readable text on THIS marker was:\n\"STRAND\"\n…\nScript appearance for this marker only: traditional Britannian runes (gump=woodsign).\n… It is dawn, weather clear, outdoors.",
  "infer": "full",
  "memory_category": "knowledge",
  "labels": [
    "source=exult",
    "kind=sign",
    "authority=written",
    "epistemic=read",
    "sign=STRAND"
  ]
}

Getting the Avatar's thoughts

This is sent to the /chat endpoint every once in a while to give an idea of what the Avatar might be thinking.

{
  "prompt": "You are me: the Avatar… This window is my inner voice — always first person…\n\nI am now beginning to converse with someone who appears as \"guard\" (npc_id 258)…\nRight now: It is day, weather clear, outdoors..\nFrom what I already remember about this npc_id only: …\nReply in 2–4 short sentences. Under ~55 words…",
  "stream": false
}

Post-game testing

Post-game testing usually took place after completing the first mission, which involves lots of talking to NPCs in the first city. The testing pertains more to Spectron itself which will be detailed in the next post, but it involved seeing what connections had been made, viewing the memory entities directly, and directly asking questions such as "Who is Christopher's son"? (It's this boy here who ends up joining your team)

Code for the Ultima VII sidecar

The code for the Ultima VII sidecar can be seen at this fork of the Exult project. On its own you can test out a dry run of the sidecar, but the true experience requires the game itself which you can find at GOG.com or EA's website. Buying it will set you back...six dollars, I believe it was.

Spectron is currently rolling out in early access, but once it is available and you have access to your own context, these are the commands you can use to run it.

export SPECTRON_BASE_URL=https:...
export SPECTRON_CONTEXT_ID=...
export SPECTRON_API_KEY='sp-Lz...'
cargo run

Here's a visual of the sidecar together with Exult to show what it's like to test Spectron while playing the game.

In the next post, we will go into details of Spectron itself, the endpoints available, how the Ultima VII data ends up on Spectron's side, and whatever else comes to mind.