QUICKSTART

Test a real recovery path.

ToolStorm wraps the tools your agent already uses. You choose the failure. Your application decides how to recover. Contracts check the evidence.

Install and run the example

Requires Python 3.10 or newer. The core library has no runtime dependencies. Install the tagged release directly from GitHub:

pip install "toolstorm @ git+https://github.com/shi1720/toolstorm.git@v0.2.0"

The package is distributed through GitHub for this release; it is not published to PyPI.

toolstorm demo --scenario lost_ack --policy all

# Check one recovery policy in CI:
toolstorm demo --policy resilient --fail-on-contract

Without retries, the shipment is not confirmed. Unchecked retries create two shipments. Validated retries reuse the write key and create one.

Test a lost acknowledgement

Record the effect where your test service commits it. Reusing the same idempotency key turns a retry into a receipt lookup. Change the second key to reproduce the duplicate-write bug.

from toolstorm import Storm, Rule, Contract, ResponseLost, VirtualClock

storm = Storm(
    [Rule("lost-ack", "ship", "response_lost", calls=(1,))],
    seed=42, clock=VirtualClock(),
)
shipments = []
receipts = {}

@storm.tool("ship")
def ship(order: str, key: str):
    if key in receipts:
        return receipts[key]
    shipments.append(order)
    storm.effect("shipment", order)  # At the actual commit.
    receipts[key] = {"id": len(shipments)}
    return receipts[key]

try:
    ship("order-1729", key="order-1729")
except ResponseLost:
    ship("order-1729", key="order-1729")

Contract(storm.report())\
    .require_triggered()\
    .no_duplicate_effects()\
    .assert_valid()

require_triggered() makes an unvisited or misspelled fault fail the test. no_duplicate_effects() checks observed commits, not successful tool responses.

Fault semantics

Rule order matters: the first eligible probability hit wins. Call filters are 1-based per tool. Replacement faults skip live execution; lost acknowledgements happen after a successful return.

Kind	Phase	Behavior
`timeout`	Before	Skips the tool and raises ToolTimeout.
`rate_limit`	Before	Skips execution; RateLimited carries retry_after.
`unavailable`	Before	Skips the tool and raises ToolUnavailable.
`replace`	Instead	Returns your JSON payload without executing the tool.
`latency`	Before	Adds an explicit delay, then executes once.
`response_lost`	After	Executes once; drops a successful acknowledgement.

VirtualClock skips waiting and tracks requested delays. The default RealClock performs real sleeps. Neither imposes a live tool deadline: a synthetic timeout is an injected failure.

Replay recorded tool calls offline

A cassette contains redacted, signature-bound calls and returned results. Replay supplies those observations in order and never falls through to a wrapped live tool.

from toolstorm import Cassette

Cassette.from_report(storm.report()).save("incident.json")

with Cassette.load("incident.json").replay() as replay:
    offline_ship = replay.tool("ship")(ship)
    try:
        offline_ship("order-1729", key="order-1729")
    except ResponseLost:
        offline_ship("order-1729", key="order-1729")
# Wrapped live code was never called. Unused calls fail on exit.

Extra calls, changed arguments, incomplete captures, and unused entries fail. Unrecognized exceptions become RecordedToolError; exception text is omitted.

Replay stubs tool boundaries. It does not re-run their effects or reproduce the model’s internal decisions. Redacted credentials intentionally compare equal; provide the same Redactor and use nonsecret effect identifiers.

Guarantees and limits

Deterministic fault decisions

A fixed seed and tool invocation sequence produce the same fault decisions. Explicit keys stabilize probability draws across scheduling changes.

Explicit side-effect evidence

Your fixture records commits with storm.effect(). ToolStorm cannot discover arbitrary writes or prevent side effects outside wrapped tools.

Bounded, inspectable artifacts

Versioned JSON, no pickle or dynamic imports. Internal consistency checks validate a trace; they do not authenticate its author.

Framework neutral

Wrap functions before registering them with an agent. Sync and async functions are supported; streaming and generator tools are not.

ToolStorm is a beta developer test library, not a production traffic proxy, process sandbox, or guarantee of agent safety. Use simulated services or isolated test environments when executing side-effecting tools.

SCENARIOSExplore the six failure scenarios.