Persistent Transparency Ledger

Persistent Transparency Ledger

This tutorial shows the operational shape of the LSM-backed Sovereign Transparency Ledger store.

Time: 20 minutes Level: Intermediate Prerequisites: Events, BLAKE3-derived event ids, and basic use imports.

Open the Substrate

LSMStore borrows a bytes-keyed LSM store. Open the substrate first, then pass a pointer to make_store.

use std.db.lsm
use std.stl.lsm_store
use std.stl.store

func main() -> i32 do
    var gs = lsm.gs_open_bytes("/tmp/tutorial-stl.wal", 0x6503)
    var stl = lsm_store.make_store(&gs)

    if not lsm_store.is_open(&stl) do
        _ = lsm.gs_close_bytes(&gs)
        return 1
    end

    _ = lsm.gs_close_bytes(&gs)
    return 0
end

The LSMStore does not close the GrainStoreBytes. That ownership stays with the caller.

Append Events

Use append_and_derive when the event id should be derived from the canonical event body at append time.

var e: event.Event = make_event()
let rc: i64 = lsm_store.append_and_derive(&stl, &e)
if rc != store.STORE_OK do
    _ = lsm.gs_close_bytes(&gs)
    return 2
end

Each successful append writes the primary event record and the rank sidecar. The primary key gives get_by_id; the sidecar gives append-order lookup.

Read by Rank

Auditors and replay validators usually walk the ledger by insertion rank.

let got = lsm_store.get_by_insertion_rank(&stl, 0)
if got == null do
    _ = lsm.gs_close_bytes(&gs)
    return 3
end

If make_store detects sidecar damage, the store opens in MODE_DEGRADED. In that state, by-id reads still work, but rank reads refuse until repair succeeds.

Flush Batches

Flush periodically to move the current MemTable into an SSTable. Use flush_with_manifest when the batch must remain recoverable after WAL truncation or rotation.

if lsm_store.flush_with_manifest(
    &stl,
    "/tmp/tutorial-stl-l0.sst",
    "/tmp/tutorial-stl.manifest",
    "/tmp/tutorial-stl.manifest.tmp",
    "/tmp",
) != store.STORE_OK do
    _ = lsm.gs_close_bytes(&gs)
    return 4
end

The flush resets the bytes-owned pool after the MemTable has been encoded into the SSTable, so a long-running producer can keep appending in batches. The manifest save path runs after the SSTable attach and writes to a temp file, fsyncs it, renames it into place, and fsyncs the directory. The manifest records attached L0 files. It does not snapshot the current MemTable.

On restart, open the WAL first, then construct the STL store with make_store_recovered.

var reopened = lsm.gs_open_bytes("/tmp/tutorial-stl.wal", 0x6503)
var recovered = lsm_store.make_store_recovered(&reopened, "/tmp/tutorial-stl.manifest")
if lsm_store.manifest_status(&recovered) != store.STORE_OK do
    _ = lsm.gs_close_bytes(&reopened)
    return 5
end

Recovered open validates the manifest header, body CRC, slot order, SSTable file size, footer entry count, and SSTable image fingerprint before publishing the L0 list and running the sidecar probe. Torn manifests, missing or truncated SSTables, and stale same-shape SSTables are refused and set MODE_RECOVERY_DEGRADED.

Repair the Rank Sidecar

On reopen, make_store checks primary and sidecar consistency.

if stl.mode == lsm_store.MODE_DEGRADED do
    let repaired: i64 = lsm_store.rebuild_rank_sidecar(&stl)
    if repaired != store.STORE_OK do
        _ = lsm.gs_close_bytes(&gs)
        return 6
    end
end

The rebuild scans primary entries, rewrites sidecar mappings through the WAL, and restores MODE_HEALTHY when it completes. A later clean close/open replays the repaired sidecar.

Operational Rule

Treat the primary keyspace as ledger truth and the rank sidecar as a repairable index. When the sidecar is healthy, positional APIs are available. When it is degraded, repair it before auditing, proving inclusion, or replaying append order.