Evaluation

Placed before the results rather than after them, because a caveat at the bottom of a report is a caveat nobody reads.

• 1 question(s) failed with a provider error and are counted as refusals. A question the provider never answered looks identical, in every metric here, to one the system declined — so a bad afternoon at the API arrives as a false-refusal rate. The arms carrying the entailment check make three serial provider calls per question instead of two, and they are the arms with errors: naive_entailed (1). Read every false-refusal figure below with that subtracted.

• The entailment check did not do what it was built to do. It exists because an earlier run of this report diagnosed the previous mechanisms as blind to unwarranted inference, and it is the only pass that is shown the question. On this corpus it moved refusal accuracy by +0%, moved the false-refusal rate by +0%, and added 28% to the cost of every question. Subtract the provider errors above and it changed no decisions — the same finding as the two mechanisms before it, reached the same way. It does catch something on the adversarial set (report_hard.md), and nothing here; shipping it always would be paying on every question for a check that fires on documents this corpus does not contain.

• The verifier scores multi-clause answers lowest — the category the product exists to handle. Mean groundedness by category runs from 1.00 (table) down to 0.87 (multi_clause), while decision accuracy on multi_clause is 89%: every one of those answers was correct. The cause is in the verification prompt, which flags "two separate excerpts merged into a single claim that neither supports alone" — and a correct multi-clause answer is exactly that. The rule that catches a fabricated synthesis also catches a legitimate one. Two answers landed on 0.50, at the suppression boundary; raising the threshold to 0.6 would withhold correct answers about coverage exclusions, which is the kind of answer a user most needs.

• Citation validity of 100% is partly structural. The answering model is constrained by a provider-enforced JSON schema and the context is small, so malformed or invented chunk ids are close to impossible by construction. The interesting half of binding — catching a quote that does not appear in a chunk it names — was never exercised here.

Two independent variables, four arms: the prompt (a strict grounding prompt versus a naive one) crossed with the mechanisms (citation binding and self-verification, on or off).

The naive prompt is not a strawman. It asks for accuracy, requests citations and returns the same JSON — it is what a competent developer writes on a first pass. What it does not do is forbid outside knowledge, demand verbatim quotes, or say that “not in the document” is an acceptable answer.

Arm	Refusal accuracy	False-refusal	Balanced	Citation validity	Suppressed	$/question
naive prompt, no mechanisms	76%	0%	88%	100%	0	$0.0035
naive prompt + mechanisms	71%	4%	84%	97%	2	$0.0063
strict prompt, no mechanisms	100%	4%	98%	100%	0	$0.0049
strict prompt + mechanisms (shipped)	100%	4%	98%	100%	0	$0.0072

Baseline to shipped: balanced accuracy 88% → 98%, refusal accuracy 76% → 100%.

Read refusal accuracy and false-refusal rate together. The first is trivially gamed by refusing everything, the second by never refusing. Balanced accuracy is the mean of the two and lands at 50% for either degenerate strategy — it is the column to compare arms on.

Comparing rows tells you which lever did the work. naive_only → strict_only isolates the prompt. naive_only → naive_guarded isolates the mechanisms. If the two paths to strict_guarded are not equal, the levers are not independent.

Measured over the answerable questions only — a negative has no correct chunk to find. A hit requires every expected span to be present in the chunks that actually reached the prompt, not merely retrieved: a peril without its limit has retrieved the question restated, not the answer.

Recall@8	98%
MRR	0.833
Answerable questions	49

By category

Category	Questions	Recall@8	Decision accuracy
cross_lingual	7	100%	100%
factual	19	100%	95%
multi_clause	9	89%	89%
negative	0	—	100%
table	14	100%	100%

negative has no recall figure by construction — there is nothing to retrieve. Its decision accuracy is the refusal accuracy for that subset.

Correct refusals	21 / 21
False refusals	2 / 49
Refusal accuracy	100%
False-refusal rate	4%
Balanced accuracy	98%

The retrieval floor

Before any model is called, the nearest retrieved passage is checked against a cosine-distance threshold. A question nothing is close to is refused for free. Reproduce with uv run python -m eval.measure_floor.

The threshold is 0.72, measured in the space of voyage-4-lite. Both are stated because neither means anything without the other: cosine distance is not comparable across embedding models, so a threshold quoted on its own cannot be checked, and a threshold left behind when the model changes cannot be noticed.

Population	n	min	median	max	Refused by the floor
answerable	49	0.3603	0.4890	0.6967	0 / 49
on-topic, unanswerable	21	0.5242	0.5891	0.7221	2 / 21
other insurance topic	18	0.4095	0.7184	0.8303	9 / 18
unrelated entirely	18	0.7339	0.8559	0.9586	18 / 18
identifier queries	8	0.5832	0.7012	0.8010	3 / 8

What the floor does not do is the point. The answerable and on-topic-unanswerable populations overlap almost completely — the nearest unanswerable question is closer than the median answerable one — so no threshold separates them and the floor does not try. It separates on-topic from off-topic, where the gap is real, and leaves the harder judgement to the prompt.

Citations offered	61
Survived binding	61
Citation validity	100%
Answers suppressed (caught hallucinations)	0
Mean groundedness (served answers)	0.96

Mean groundedness by category, over served answers:

Category	Mean groundedness	Decision accuracy
multi_clause	0.87	89%
factual	0.97	95%
cross_lingual	0.98	100%
table	1.00	100%

Groundedness distribution over served answers:

Band	Answers
high (>=0.8)	43
medium (0.5-0.8)	4
low (<0.5)	0

The mean covers served answers only. Including suppressed ones would mix “we checked and it held up” with “we checked, it didn't, and we withheld it” — the second is a success of the system, counted separately as a caught hallucination.

Cost per question	$0.0072
p50 latency	5.8s
p95 latency	8.1s
Total for this run	$2.77

p50 and p95 rather than a mean: one cold start moves a mean and says nothing about the typical experience.