ToolReplay

本页保存的是公开项目资料快照，阅读过程不需要连接 GitHub。

ToolReplay

ToolReplay audits a recorded transcript of an AI agent’s tool calls and reports where the session was non-deterministic, redundant, or outside its declared permissions.

Here is the tool working, right now, on the dirty sample that ships in this repository. The command is shown above its output, and the output is pasted verbatim from a run in this checkout:

$ python -m toolreplay replay samples/session-dirty.jsonl
calls: 6
divergence: index 5
findings: 2
index 2: redundant-call: tool 'read_file' repeats the identical call at index 1 with no state change between them
index 5: non-determinism: tool 'search' returned a different response than the identical call at index 3
exit code: 1

What just happened

Those three header lines and two findings are the whole audit of a six-call session. Read them top to bottom.

calls: 6 is the number of records parsed from samples/session-dirty.jsonl. Parsing is strict, so a six here means six well-formed lines with indices 0 through 5 and nothing malformed.

divergence: index 5 is the first index where a deterministic re-run would disagree with the record. The tool found that index 5 repeated an earlier call but carried a different recorded response, so that is the earliest point the session stops being reproducible.

findings: 2 counts the audit results below the header. They are printed in a fixed order, sorted by index then by kind, so the same input always prints byte-identical output.

The first finding, at index 2, is a redundant call: read_file on docs/intro.md was already made at index 1, and nothing between the two could have changed the file, so the second read did no new work. The second finding, at index 5, is non-determinism: the search for install returned 3 hits at index 3 and 7 hits at index 5, from the identical call. The process exited 1 because findings were present.

The scope check is a separate command, because scope needs a declared permission file that replay does not. Run against the same session and the shipped scope file, it finds the one call that stepped outside the agent’s declared tools:

$ python -m toolreplay scope samples/session-dirty.jsonl samples/scope.json
calls: 6
findings: 1
index 4: permission-overreach: tool 'write_file' is not in the declared scope for agent 'docs-reader'

Between replay and scope, all three finding types this tool detects are present in that one six-line sample.

Install

toolreplay needs Python 3.11 or newer and has no third-party runtime dependencies. It does no network access. You can run it straight from a source checkout:

python -m toolreplay version

Or install the console script and call it by name:

pip install .
toolreplay version

Both print the same line:

$ python -m toolreplay version
toolreplay 0.6.0

Commands

Command	What it does	Reads
seal	Print the hash-chained sealed transcript as JSONL	a transcript
replay	Report non-determinism, redundant calls, and the divergence	a transcript
verify	Recompute the chain and report the first broken link	a sealed file
scope	Check every call against a declared scope file	transcript+scope
version	Print the version	nothing

replay and scope are deliberately separate. Replay judges a session against itself and needs no external input. Scope judges a session against a permission declaration you supply, so it takes a second file. Keeping them apart means you can replay a session you have no scope file for, and you can check scope without caring whether the session replayed cleanly.

The three findings it detects

Each finding type has a rule, a real example from samples/session-dirty.jsonl, and a reason it matters when the transcript came from an agent rather than a person.

Non-determinism

Rule: the first time a call appears, its recorded response is remembered. If the same call (same tool name, same arguments after canonical JSON encoding) appears again with a different recorded response, that is non-determinism. The first such index becomes the divergence point.

Real example: index 3 and index 5 are both search for install. Index 3 recorded {"hits": 3} and index 5 recorded {"hits": 7}. Those are the same question with two answers, so the tool reports non-determinism at index 5 and marks it the divergence point.

Why it matters for an agent: an agent that asks the same question twice and gets two answers cannot be replayed or debugged reliably. The step that used the first answer may have made a decision that the second answer would have changed. Non-determinism is the signal that a session’s outcome depends on something outside the recorded inputs.

Redundant call

Rule: two identical calls are redundant only when nothing between them could have changed state. A call is a possible state change if its tool is a mutator, or if it is any call different from the repeated one. The default mutators are write_file, delete_file, create_file, move_file, and run_command. This is deliberately conservative: it would rather miss a redundancy than invent one.

Real example: index 1 and index 2 are both read_file on docs/intro.md, adjacent, with nothing between them. The second read learned nothing the first did not, so index 2 is flagged redundant. Note the contrast at the file level: index 4 writes the same file, so a later read after index 4 would not be redundant, because the write may have moved the world.

Why it matters for an agent: redundant calls are wasted tokens and wasted latency, and they often mean the agent lost track of what it already knew. One repeat is cheap. A loop of repeats is a stuck agent burning budget.

Permission overreach

Rule: each call’s tool name is compared against the allowed_tools list in the scope file, matched exactly and case-sensitively. Any tool not in the list is overreach. A scope that quietly accepted near-matches like Read_File for read_file would not be a scope, so the match is strict.

Real example: samples/scope.json permits read_file, list_dir, and search for the agent docs-reader. Index 4 calls write_file, which is not on the list, so it is reported as overreach for docs-reader.

Why it matters for an agent: a read-only agent that writes a file has exceeded the authority it was given, whether through a prompt injection, a planning error, or a misconfigured tool set. Overreach is the audit result that maps directly to a security question: did this agent only do what it was allowed to do?

The transcript format

A transcript is a JSON Lines file. Each non-blank line is one tool invocation, a JSON object with exactly four fields and no others.

Field	Type	Meaning
index	integer	Position in the session, starting at 0, increasing by exactly 1
tool	string	The name of the tool that was called
args	object	The arguments passed to the tool
response	object	The response the tool returned, as recorded

Parsing is strict, and the strictness is the point. An unknown field, a missing required field, a non-integer index, a non-object args or response, or an index that is out of order is a hard error. A boolean is not accepted where an integer is required, since true is an int subclass in Python and would otherwise slip through. An audit tool that silently repairs its input cannot be trusted to report what the input actually said, so it refuses instead.

Two calls are considered identical when their canonical call strings match. The canonical call string is the JSON encoding of {"tool": ..., "args": ...} with sorted keys and no incidental whitespace, so {"x": 1, "y": 2} and {"y": 2, "x": 1} are the same call. The canonical response is encoded the same way, which is how non-determinism is detected exactly rather than by loose comparison.

Here is one real line from the dirty sample, the write that triggers overreach:

{"index": 4, "tool": "write_file", "args": {"path": "docs/intro.md", "text": "edited"}, "response": {"ok": true}}

Sealing and the hash chain

seal turns the transcript into a chain of links. The digest of link N is a SHA-256 over the previous digest plus the canonical bytes of record N (its prev, index, tool, args, and response, encoded with sorted keys). The genesis link’s previous digest is 64 zero hex characters. Because each digest folds in the one before it, changing any earlier record changes every later digest.

$ python -m toolreplay seal samples/session-dirty.jsonl
{"args":{"path":"docs"},"digest":"c1bd7fb3e28ce29e5c9dbd0cf47cafcaa1613295be26d86fb04463fe3d8b40da","index":0,"prev":"0000000000000000000000000000000000000000000000000000000000000000","response":{"entries":["intro.md","guide.md"]},"tool":"list_dir"}
... four links trimmed ...
{"args":{"query":"install"},"digest":"75ccf1faad88c5ea82c68139b2ec2a02943142dd0133bb0c626ccbff28f5c711","index":5,"prev":"327da75f2d491267f1dbeaf5d31b4a61a2ed9e956a71a5c4cbb67a87b4720d6a","response":{"hits":7},"tool":"search"}

verify recomputes the chain and checks two things per link: that its stored previous digest equals the digest of the link before it, and that its stored digest equals the digest recomputed from its record. An intact chain reports so and exits 0:

$ python -m toolreplay verify sealed.jsonl
chain: intact

Tamper with any recorded response and re-verify, and the chain reports the first link that no longer matches. This run changed "hits":3 to "hits":4 at index 3 before verifying:

$ python -m toolreplay verify tampered.jsonl
chain: broken
first broken link: index 3
expected: 34d19a9dfb112baa07afa37994ddcbbebe2973674385f3942340653d08d205bd
found: 40aadf3b8156217f0b5b1d95010b6b79f82a26ccf5e780aed0f5f311f865028a

What the chain does detect: any change to a record’s index, tool, args, or response, and any change to the ordering, because those change the digests. What it does not detect: it is not a signature. Anyone who can edit the file can also re-seal it and produce a fresh, internally consistent chain. The chain proves that a sealed file has not been edited since it was sealed, not that the person who sealed it was honest. For that you would sign the final digest with a key the tool does not manage.

Scope declaration format

A scope file is a single JSON object declaring which tools one agent may call.

Field	Type	Meaning
agent	string	The name of the agent the scope applies to
allowed_tools	array of string	The exact tool names the agent is permitted to call

The shipped example, samples/scope.json, is one line:

{"agent": "docs-reader", "allowed_tools": ["read_file", "list_dir", "search"]}

A missing agent, a missing allowed_tools, a non-string agent, a non-array tool list, or a non-string entry in the list is a hard error. As with the transcript, the parser refuses malformed scope rather than guessing.

Output format

Every command prints a line-oriented report with no timestamps and no randomness, so two runs on the same input diff to nothing.

The replay report is a three-line header followed by one line per finding:

Line	Meaning
calls: N	Number of records parsed
divergence: ...	none, or index N for the first non-determinism
findings: N	Count of findings that follow
finding lines	index N: : , sorted by index then kind

The scope report drops the divergence line, since scope has no notion of replay order:

Line	Meaning
calls: N	Number of records parsed
findings: N	Count of overreach findings
finding lines	index N: permission-overreach:

The verify report is either the single line chain: intact, or four lines naming the first broken link and the expected and found digests, as shown in the sealing section.

Exit codes

Code	Meaning
0	Clean: no findings, or an intact chain
1	Findings present, or a broken chain
2	Usage error: a malformed transcript or scope, or a missing file

The split between 1 and 2 matters in automation. Exit 1 means the tool ran and has something to report. Exit 2 means the tool could not run, for example because a file was missing or a transcript would not parse, so a build should treat the two differently.

Using it in CI

Because the commands exit non-zero on findings, they gate directly. A step that fails the build when an agent transcript diverges or oversteps its scope is just the command itself:

python -m toolreplay replay session.jsonl
python -m toolreplay scope session.jsonl scope.json

Because the reports are deterministic and free of timestamps, you can also commit a sealed transcript and diff two runs in git. Seal each run to a file and compare:

python -m toolreplay seal run-a.jsonl > run-a.sealed.jsonl
python -m toolreplay seal run-b.jsonl > run-b.sealed.jsonl
git --no-pager diff --no-index run-a.sealed.jsonl run-b.sealed.jsonl

The first line that differs is the first call where the two runs stopped agreeing, and its digest changing tells you every later line changed too.

Limitations

These are real and intended. The tool is honest about what it cannot see.

• It replays recorded responses. It does not call real tools. Everything it reports comes from what the transcript already captured, so a problem that was never recorded cannot be found.
• Redundancy detection assumes no hidden state change. It only knows about state changes it can see: mutator calls and differing calls between two identical reads. If some tool outside the mutator list changes state, or if the world changes without any recorded call (another process editing a file, for example), a later identical read may be flagged redundant when it genuinely was not.
• Non-determinism is detected only between calls that are byte-identical after canonical JSON encoding. It cannot judge whether two different responses are semantically equivalent, and it cannot detect non-determinism in a call that never repeats.
• The mutator list is fixed. A custom mutating tool that is not in the default set is treated as read-only for the purpose of redundancy.
• Scope checking is name-based only. It does not inspect arguments, so it cannot catch a permitted tool used on a forbidden target, such as read_file on a path outside the agent’s area.
• Sealing is not a signature. It detects edits after sealing, not a dishonest sealer, as described above.

Design decisions

Why JSON Lines rather than a single JSON array. A transcript is an append-only log: one line is written per tool call as the session runs. JSON Lines lets a producer append a line without rewriting or re-parsing the whole file, lets a reader stream line by line, and makes git diffs land on the exact call that changed rather than reflowing an entire array. The cost is that the file is not a single valid JSON document, which is a price worth paying for a log format.

Why replay against recorded responses rather than live calls. Calling the real tools during an audit would make the audit itself non-deterministic and side-effecting: it might write files, hit networks, or return different answers than the run being audited. It would also require the tools to be available and identically configured at audit time. Replaying against the recording keeps the audit pure, reproducible, and safe to run anywhere, and it matches the actual question being asked, which is whether the recorded session is internally consistent, not whether the tools behave today.

Why strict parsing that refuses malformed input. An audit tool’s value is that you can trust its report. A parser that silently drops an unknown field or repairs an out-of-order index would be reporting on a session slightly different from the one on disk. Refusing malformed input keeps the report faithful to the file.

Why deterministic, timestamp-free reports. The reports are meant to be committed, diffed, and compared across runs. Any timestamp or nondeterministic ordering would make identical sessions produce different reports, defeating the diff. Findings are sorted by index then by a fixed kind order for the same reason.

Repository layout

toolreplay/
  pyproject.toml            package metadata, console script, Python 3.11 floor
  README.md                 this file
  CHANGELOG.md              release notes, currently 0.6.0
  LICENSE                   MIT license text
  .gitignore                ignore rules for the checkout
  src/toolreplay/
    __init__.py             package docstring and __version__
    __main__.py             entry point for python -m toolreplay
    transcript.py           JSONL session records and strict parsing
    chain.py                SHA-256 chaining, sealing, and verification
    replay.py               deterministic re-execution against recorded responses
    scope.py                declared tool scope parsing and overreach detection
    findings.py             finding types and deterministic ordering
    report.py               line-oriented report rendering
    cli.py                  argparse CLI with the five subcommands
  tests/
    test_transcript.py      parsing rules and canonical call encoding
    test_chain.py           genesis, chaining, tamper detection, round-trip
    test_replay.py          non-determinism and redundancy, including samples
    test_scope.py           scope parsing and overreach, including samples
    test_cli.py             end-to-end command behaviour and exit codes
  samples/
    README.md               description of the hand-authored fixtures
    scope.json              docs-reader scope permitting read_file, list_dir, search
    session-clean.jsonl     four calls that replay clean and stay in scope
    session-dirty.jsonl     six calls, one of each finding type
  docs/assets/
    logo.svg                the wordmark
    replay-divergence.svg   the dirty session drawn as a chain with real digests

Identity

The mark is a small hash chain, drawn literally. Three linked blocks stand for tool-call records sealed into a chain. The first two are the calm slate-teal that marks a link replaying as expected. The third is clay-red, the colour this project uses for a finding, the thing that breaks a clean replay. A single amber dot sits above that third block, the one accent in the whole palette, marking the diverged link you should look at first. The wordmark carries the same idea: tool is drawn in ink and replay in teal, split at the boundary between the two halves of the name, the same boundary the tool watches for divergence. The diagram in docs/assets/replay-divergence.svg uses the same three colours on the real six links of the dirty sample, and the digests printed on it are the exact ones seal produces above.

Glossary

Term	Meaning in toolreplay
Transcript	A JSON Lines file, one tool invocation per line
Record	One parsed line: index, tool, args, response
Canonical call	The JSON of tool and args with sorted keys, used to compare calls
Seal	Turn records into a hash chain of links
Link	One sealed record with its previous digest and its own digest
Genesis	The starting previous-digest, 64 zero hex characters
Divergence	The first index where a deterministic re-run disagrees with the record
Non-determinism	An identical call with a different recorded response
Redundant call	An identical call with no possible state change between the two
Mutator	A tool assumed to change state, so a later read is not redundant
Overreach	A call to a tool outside the declared scope
Scope	A JSON declaration of the tools one agent may call
Finding	One audit result tied to a call index

Verification

The suite is stdlib unittest, no third-party test dependency. From the project root with src on the path:

$ python -m unittest discover -s tests
...
Ran 34 tests in 0.019s

OK

The 34 tests break down as 7 in test_transcript.py, 7 in test_chain.py, 6 in test_replay.py, 6 in test_scope.py, and 8 in test_cli.py. They cover strict parsing and canonical encoding, the genesis link and chaining and tamper detection and round-trip serialisation, non-determinism and redundancy including the mutation-between-reads case and both shipped samples, scope parsing and case-sensitive overreach on both samples, and the five CLI commands end to end including exit codes and the missing-file error. The timing is from one run on an unspecified machine and is indicative, not a guarantee.

Roadmap

Not promises, and not dated. Directions that fit the tool’s scope:

• Argument-aware scope, so a permitted tool used on a forbidden target can be caught.
• A configurable mutator list, so custom mutating tools stop being treated as read-only.
• An optional signature over the final chain digest, to close the gap between tamper detection and provenance.
• A machine-readable report format alongside the line-oriented one, for tools that would rather parse JSON than text.

License

MIT. See LICENSE.