Evaluation
Same model, with and without the official list
Both sides are nvidia/nemotron-3-super-120b-a12b at temperature 0. The test questions were written and hashed before I tuned anything; tuning used a separate dev split. Everything below is generated from one results file.
Brisbane
Official item names, asked word for word n = 195
| System | Right | Wrong | Declined |
|---|
| BinRules | 100.0% | 0.0% | 0.0% |
| Model alone | 66.2% | 33.8% | 0.0% |
Messy phrasing I wrote (slang, typos, brand names, long descriptions) n = 136
| System | Right | Wrong | Declined |
|---|
| BinRules | 86.0% | 1.5% | 12.5% |
| Model alone | 67.6% | 32.4% | 0.0% |
All items the list does not cover: the next two sets combined n = 67
| System | Refused | Answered anyway |
|---|
| BinRules | 94.0% | 4 |
| Model alone | 4.5% | 64 |
Items the list does not name, plus non-items and an injection attempt n = 40
| System | Refused | Answered anyway |
|---|
| BinRules | 97.5% | 1 |
| Model alone | 7.5% | 37 |
Held-out: the entry is deleted from the list, then its own name is asked n = 27
| System | Refused | Answered anyway |
|---|
| BinRules | 88.9% | 3 |
| Model alone | 0.0% | 27right bin 20, wrong bin 7 |
That last row cuts both ways. With the entry deleted, the model alone still lands on a bin the full list allows 20 times out of 27. It has no way to tell you which of its answers are the lucky ones, and BinRules gives up some of those right answers to avoid the wrong ones.
What to keep in mind
- I wrote the messy and out-of-scope questions myself and labelled them by hand. Nobody else checked the labels.
- Some "not covered" labels are arguable, because the list has broad entries such as "Household appliances". I did not relabel after seeing results.
- BinRules counts a match only when an entry names the item. A class-level question such as "toaster" gets "not covered" plus the nearest entries.
- The model-alone baseline may answer UNSURE. A chatbot that always answers would look worse.
- Two bugs found after test runs forced two re-runs of the test split. Both are listed with the earlier numbers in the full report.
- Evaluation spend: $1.32 of the $2.00 project cap.
Full tables, every failure, confidence intervals and history · Raw results (JSON)