Evaluation
Labelled scenarios scored against each prompt version. Every number comes from a saved run.
Loading results from the latest saved run. Please wait a few seconds before starting a new run.
Root-cause accuracy
↑ higher is better
v1
Not run
v2
Not run
Retry-decision accuracy
↑ higher is better
v1
Not run
v2
Not run
Hallucination rate (unknown codes)
↓ lower is better
v1
Not run
v2
Not run
Message-rule breaches
↓ lower is better
v1
Not run
v2
Not run
v1 Not runv2 Not run
| ID | Input | Expected reason | v1 reason | Expected retry decision | v1 retry decision | Result | Failure type | |
|---|---|---|---|---|---|---|---|---|
| S01 | Card / 51 / recurring / 1st | Not enough balance | — | Retry: Yes | — | Not run | — | |
| S02 | Card / 54 / recurring / 1st | Card expired | — | Retry: After an action | — | Not run | — | |
| S03 | Card / 41 / one-off / 1st | Card reported lost or stolen | — | Retry: No | — | Not run | — | |
| S04 | Card / 91 / one-off / 1st | Bank or network was down | — | Retry: After an action | — | Not run | — | |
| S05 | Card / 82 / one-off / 1st | Wrong or outdated payment details | — | Retry: After an action | — | Not run | — | |
| S06 | Card / 1A / one-off / 1st | Customer didn't complete verification | — | Retry: After an action | — | Not run | — | |
| S07 | Card / card_disabled_for_online_payments / one-off / 1st | Card or account is restricted | — | Retry: After an action | — | Not run | — | |
| S08 | Card / authentication_required / recurring / 1st | Customer didn't complete verification | — | Retry: After an action | — | Not run | — | |
| S09 | Card / 94 / one-off / 2nd | Already paid or a duplicate | — | Retry: No | — | Not run | — | |
| S10 | Card / R1 / recurring / 1st | Customer cancelled autopay | — | Retry: No | — | Not run | — |
Showing 1–10 of 50
By scenario type
| Scenario type | v1 reason | v1 retry decision | v2 reason | v2 retry decision |
|---|---|---|---|---|
| Straightforward (30) | Not run | Not run | Not run | Not run |
| Unclear (8) | Not run | Not run | Not run | Not run |
| Unknown code (5) | Not run | Not run | Not run | Not run |
| Trick case (7) | Not run | Not run | Not run | Not run |