Evaluation
Labelled scenarios scored against each prompt version. Every number comes from a saved run.
Root-cause accuracy
v1
Not run
v2
Not run
Retry-verdict accuracy
v1
Not run
v2
Not run
Hallucination rate
v1
Not run
v2
Not run
Message-rule breaches
v1
Not run
v2
Not run
v1 Not runv2 Not run
| ID | Input | Expected cause | v1 cause | Expected verdict | v1 verdict | Result | Failure type | |
|---|---|---|---|---|---|---|---|---|
| S01 | Card / 51 / recurring / 1st | Insufficient funds | — | yes | — | Not run | — | |
| S02 | Card / 54 / recurring / 1st | Card expired | — | conditional | — | Not run | — | |
| S03 | Card / 41 / one-off / 1st | Card lost stolen | — | no | — | Not run | — | |
| S04 | Card / 91 / one-off / 1st | Bank or network downtime | — | conditional | — | Not run | — | |
| S05 | Card / 82 / one-off / 1st | Payment details invalid | — | conditional | — | Not run | — | |
| S06 | Card / 1A / one-off / 1st | Auth not completed | — | conditional | — | Not run | — | |
| S07 | Card / card_disabled_for_online_payments / one-off / 1st | Instrument restricted | — | conditional | — | Not run | — | |
| S08 | Card / authentication_required / recurring / 1st | Auth not completed | — | conditional | — | Not run | — | |
| S09 | Card / 94 / one-off / 2nd | Duplicate or already collected | — | no | — | Not run | — | |
| S10 | Card / R1 / recurring / 1st | Mandate revoked or stopped | — | no | — | Not run | — |
Showing 1–10 of 50
By scenario type
| Scenario type | v1 root cause | v1 verdict | v2 root cause | v2 verdict |
|---|---|---|---|---|
| clear (30) | Not run | Not run | Not run | Not run |
| ambiguous (8) | Not run | Not run | Not run | Not run |
| unrecognised (5) | Not run | Not run | Not run | Not run |
| trap (7) | Not run | Not run | Not run | Not run |