Public agent-benchmark rulebooks, broken into single clauses by hand, with human verdicts on how well an automated matcher aligned them. A precision figure means something only if you can read the answer key.
git clone https://github.com/cognitive-fab/polyx-bench
ABCD, the Action-Based Conversations Dataset: 499 clauses from its knowledge base and guidelines. τ²-bench: its retail policy, decomposed by hand.
The gold file holds human verdicts on a stratified sample of the matcher’s alignments, including the ones it declined to make.
A clause is expressible when it can be stated as one action that must come before another. The rest are real guidance about intent, argument values or free text. They are kept with a reason.
That split is the honest denominator: of ABCD’s 499 clauses, 155 can be checked this way and 344 cannot.
Clone polyx-bench next to polyx; polyx finds ../polyx-bench/policies by default. Fetch the corpus, then evaluate the mined rule set against the clause set.
Refusal correctness measures whether polyx declined to propose rules the policy does not contain, which matters as much as finding the ones it does.
cd polyx node corpora/fetch-abcd.mjs node bin/polyx.mjs evaluate abcd
# the same figures across a grid: a curve, never a tuned point
node bin/polyx.mjs evaluate abcd --sweep "minInstances=5,10,30"Elsewhere on disk? POLYX_POLICIES=/path/to/polyx-bench/policies.
Alignment between a mined rule and a written clause is the least reliable step, so people rate it. An early rating pass was biased by showing the matcher’s verdict before the rater’s. Those verdicts were superseded, not deleted, and the reason is recorded.
node bin/polyx.mjs gold abcd serve
One pair per screen: is this mined rule and this written clause the same rule? Your verdicts change the headline figures, which then measure polyx instead of the matcher.
The files carry no dependency on polyx. Use them to score any rule-mining or agent-compliance method, and disagree with a specific clause rather than with a number.
Only decompositions of public benchmarks live here. Clause sets from private corpora are kept elsewhere and licensed to nobody, because a shipped answer key is a shipped evaluation.