Chaos Testing & Devnet Scenarios

Overview

Ashen ships two complementary chaos tools:

1. devnet binary: quick deterministic simulations with latency/loss knobs.
2. devnet_replay tests: scripted scenarios with golden-state checks.

Use devnet for fast iteration and devnet_replay for regression coverage and state-root determinism.

Quick Chaos Runs (devnet binary)

just devnet-small
just devnet-chaos

devnet-chaos injects packet loss and latency via env vars:

DEVNET_SUCCESS_RATE=0.9 \
DEVNET_LINK_LATENCY_MS=25 \
DEVNET_LINK_JITTER_MS=10 \
just devnet-chaos

See /guides/devnet/ for the full parameter list.

Deterministic Scenario Suite (devnet_replay)

Run the full suite:

cargo test --test devnet_replay

Regenerate goldens:

REGENERATE_GOLDENS=1 cargo test --test devnet_replay

Run a single scenario by test name:

cargo test --test devnet_replay baseline_4_validators_happy_path
cargo test --test devnet_replay network_partition_and_heal

Golden files live under tests/goldens/ and assert state-root stability across nodes and runs.

Scenario Catalog (Common)

These scenarios are defined in tests/devnet_replay.rs:

Scenario	What it stresses
baseline_4v_happy	4 validators, happy path
baseline_2v_happy	2 validators, happy path
partition_heal	network split + heal
high_latency	sustained high latency
lossy_network	packet loss
high_jitter	reordering and jitter
delayed_link_0_to_1	targeted slow link
node_crash_recovery	node isolate + restore
silent_byzantine	node receives but stops sending
leader_isolation	isolate leader and observe handoff
cascading_failures	sequential node isolation
txpool_burst	burst transaction load
sustained_tx_load	continuous tx pressure
combined_stress	mixed network + tx stress
soak_*	long-running soak variants
replay_*	record/replay determinism

See the test file for the complete list and parameters.

Network Events Used by Scenarios

The scenario harness supports these events:

• Partition { at_step, partitioned_nodes }
• HealPartition { at_step }
• ModifyLink { at_step, from_node, to_node, latency_ms, jitter_ms, success_rate }
• IsolateNode { at_step, node_idx }
• RestoreNode { at_step, node_idx }
• SilenceNode { at_step, node_idx }

These events are applied at deterministic steps, making failures reproducible under a fixed seed.

Tips

• Use a fixed seed for reproducibility.
• Keep steps small when iterating; increase for soak tests.
• Use -- --nocapture to see detailed logs from the test runner.

Related

• /guides/devnet/ for devnet binary usage
• tests/devnet_replay.rs for scenario definitions
• tests/goldens/ for golden outputs