std.testing

std.testing

std.testing is the canonical Janus testing module. It works with the language surface:

test "reread count is preserved" do
    const reread = try read_index(...)
    try testing.expect_equal[usize](1, reread.count)
end

It is not a second test language. The runner discovers test "..." do ... end and bench "..." do ... end; std.testing supplies the assertions and test-scoped authority.

The Laws

Rule	Meaning
Failure is an error value	Assertions return TestError!void; tests use try.
Expected comes first	Comparison calls read as law, then evidence.
No hidden authority	File, network, time, random, allocator, and process power use TestCtx or capabilities.
Diagnostics beat cleverness	Failures show the smallest useful mismatch.
No second DSL	Janus test syntax stays ordinary Janus.

Runner Commands

janus test tests/parser_test.jan
janus test tests/parser_test.jan --only "parser/invalid digit"
janus test tests/parser_test.jan --only "parser/*"
janus test tests/parser_test.jan --skip-tag slow
janus test tests/parser_test.jan --seed 12345
janus test tests/parser_test.jan --jobs 8
janus test tests/parser_test.jan --bench
janus test tests/parser_test.jan --update-golden
janus test --help

The runner executes tests in deterministic source order by default. It prints stable IDs such as T0001, T0002, B0001, and fails the process if any test fails, unexpectedly passes an xfail, leaks through a TestingAllocator, or fails a benchmark body at runtime.

Core Assertions

use std.testing

try testing.expect(ok)
try testing.expect_msg(ok, "index must be non-empty")
try testing.expect_equal[usize](1, count)
try testing.expect_not_equal[i32](0, status)
try testing.expect_equal_slices[u8]("janus", actual)
try testing.expect_approx_abs(1.0, actual, 0.001)
try testing.expect_approx_rel(100.0, actual, 0.02)

Compatibility aliases exist for Zig-heritage migrations:

try testing.expectEqual[usize](1, count)
try testing.expectEqualSlices[u8](expected, actual)

Prefer the canonical snake_case names in new code.

Error Testing

Errors are values. Test the returned error union directly:

error ParseError {
    InvalidMagic,
}

func parse_header(bytes: []const u8) -> ParseError!usize do
    if bytes.len == 0 do
        fail ParseError.InvalidMagic
    end
    return bytes.len
end

test "invalid header is rejected" do
    const result = parse_header("")
    try testing.expect_error[ParseError, usize](ParseError.InvalidMagic, result)
end

test "valid header returns length" do
    const len = try testing.expect_no_error[ParseError, usize](parse_header("JANUS"))
    try testing.expect_equal[usize](5, len)
end

Use expect_panic only for boundary checks such as FFI panic quarantine, compiler traps, or invariant tests:

let panics = func() -> void do
    panic("expected trap")
end

try testing.expect_panic("expected", panics)

Invalid input should normally return an error value, not panic.

Diagnostics

A scalar mismatch reports expected and actual values at the test source location:

FAIL T0001 katana/diagnostic equality

Failures:
  "katana/diagnostic equality": at tests/std_testing_diagnostics_smoke.jan:8
value mismatch
  expected: 1
  actual:   2

A slice mismatch reports length and the first differing index:

slice mismatch
  length: expected 5, actual 5
  first differing index: 1
  expected[1]: 97
  actual[1]:   120

The output is intentionally small. It should show the wound, not a novel.

Test Context And Authority

TestCtx carries test-scoped authority:

test "writes config" do
    var t = testing.context()
    let fs = t.fs_readonly("/tmp")
    testing.write_file(fs, "/tmp/config.kdl", "port=8080") catch return
    const data = testing.read_file(fs, "/tmp/config.kdl")
    try testing.expect_equal_slices[u8]("port=8080", data)
end

Profile behavior:

Profile	Behavior
:script	May use ergonomic ambient test filesystem, stdio, allocator, and temporary directory helpers.
:core	No ambient effects. Tests stay pure unless authority is explicit and effect-clean.
:service and above	Resource-touching helpers require explicit TestCtx or a capability-backed argument.

The path-only form below is intentionally rejected in :service:

{.profile: service.}

use std.testing

pub func main() -> i32 do
    let data = testing.read_file("/tmp/input")
    _ = data
    return 0
end

Use testing.context() and t.fs_readonly(...) instead.

TestingAllocator

TestingAllocator is test-only authority. It does not create a production global allocator.

test "no leaks" do
    var alloc = testing.allocator()
    testing.record_alloc(&alloc)
    testing.record_free(&alloc)
    try testing.expect_no_leaks(&alloc)
end

The runner also checks allocator accounting at test end. If a test records an allocation and does not record a matching free, janus test fails:

leak detected
  allocations: 1
  frees: 0
  outstanding: 1

This makes leaks visible even when the test body forgets to call expect_no_leaks.

Subtests

Subtests are ordinary function bodies grouped under a parent test:

test "parse integer cases" do
    var t = testing.context()

    try t.subtest("zero", do
        try testing.expect_equal[i64](0, parse_i64("0"))
    end)

    try t.subtest("negative", do
        try testing.expect_equal[i64](-7, parse_i64("-7"))
    end)
end

Subtest names become slash-separated paths:

parse integer cases/zero
parse integer cases/negative

Run one subtest with:

janus test tests/parser_test.jan --only "parse integer cases/negative"

Skips, Tags, Xfail, And Xpass

Tags are selection metadata:

@test.tag(.slow)
test "large corpus parse" do
    try run_large_corpus()
end

Skip slow tests:

janus test tests/parser_test.jan --skip-tag slow

Dynamic skips return TestError.Skipped through the normal error path:

test "network/live endpoint" do
    var t = testing.context()
    try t.skip("disabled in offline mode")
end

Expected failures pass only when the body fails. If the test unexpectedly passes, the runner reports XPASS and exits non-zero.

Compile-Fail Tests

Compiler negative tests are first-class:

test "non-SBI payload is rejected" do
    try testing.compile_fails(testing.CompileFailCase {
        source: "message Bad { Ref { x: *u8 } }",
        error_code: "E2530",
        message_contains: "non-SBI-conformant",
        span_contains: "Ref",
    })
end

Use structured fields as the contract. Full diagnostic text can change; error codes and required fragments should not.

Golden Tests

Golden updates are explicit:

test "formatter output" do
    const actual = format_module(source)
    try testing.expect_golden("tests/golden/formatter/basic.out", actual)
end

By default a mismatch fails. To update the artifact:

janus test tests/formatter_test.jan --update-golden

The runner prints every changed path under a Golden updates: section.

Benchmarks

Benchmarks share discovery with tests but run only with --bench:

bench "parse small module" do
    var b = testing.benchmark_context()
    const source = b.read_fixture("tests/golden/std_testing/bench_fixture.txt")

    while b.keep_running() do
        _ = source.len
    end
end

Output includes timing and allocation counters when available:

Benchmark Summary:
BENCH B0001 katana/bench loop
  iterations: 1
  median: 0.274ms
  p95: 0.274ms
  p99: 0.274ms
  allocations: 0
  bytes: 0

Benchmarks fail only for runtime failures in the benchmark body. Performance thresholds belong in separate policy.