Kafka data can be imported to SurrealDB using the Surreal Sync tool.
Importing JSON Lines data using Surreal Sync
Kafka data imported to SurrealDB works as follows:
The specified Kafka topic becomes a table in SurrealDB, where each message from the topic becomes a record in the table, with configurable deduplication.
Each Kafka message must be encoded using Protobuf. The Kafka source decodes every message payload using Protobuf and converts it into SurrealDB Upsert queries, transforming the payloads into SurrealDB records.
Message Key Strategy: Use Kafka message keys as record IDs (base64 encoded)
Field Strategy: Extract IDs from a field in the message payload (default: “id” field)
For more on how to import Kafka data to SurrealDB, please see the Kafka Source Usage Guide page in the Surreal Sync repo.
