Answer checker for RAG systems
Give it the retrieved source, the question and the generated answer. It returns the probability that every claim in the answer is backed by that source.
Check one answer against its source
Pick an example or paste a source and answer from your own system. The model server sleeps when nobody is using it, so the first check after a quiet spell can take up to two minutes.
Read the model card81.5 F1 at 1.9B parameters
On the 2,700 answers of the RAGTruth test set. The fine-tuned Llama-2-13B from the RAGTruth paper reaches 78.7, and the same d1-2B before fine-tuning 70.9.
- Precision
- 79.8
- Recall
- 83.2
- AUROC
- 0.936
- Training
- 29 min
- on one RTX 5090
Routes each answer by its score
Answers scoring 0.88 or more go to the user, answers at 0.12 or less are blocked, and the band between goes to a reviewer.
- Cut-offs chosen on 1,500 held-out development answers
- 61% of test answers decided without review
- 96.1% of those decisions match the annotators
Results on the RAGTruth test set
Response level, 2,700 answers. Rows other than the two d1 models are as their authors report them.1
1 Llama-2-13B: RAGTruth paper, table 5. RAG-HAT and LettuceDetect: LettuceDetect paper, table 2.
Does the model say which sentence is wrong?
No. It scores the whole answer; check sentences one by one to find it.
Which answers count as unsupported?
Any answer with a claim the source does not back.
Which languages can the model read?
English. Other languages run but are not measured.
Why is the model slow on the first check?
The model server shuts down when idle and starts again on the next check. Once it is up, a check takes one to three seconds.
Does the model server store my text?
No. Text goes to the model for one check and is not logged.
The checker runs on a GPU or a CPU
One forward pass per answer and no text generation. The weights ship as bfloat16 safetensors, a LoRA adapter and a Q8_0 GGUF for llama.cpp.
- 144 ms per answer on a 12 GB Radeon RX 6750 XT
- About 40 answers a second on one RTX 5090
- LFM Open License v1.0, from the base model
import transformers as tf
m = tf.AutoModel.from_pretrained(
REPO, trust_remote_code=True)
q = {"type": "noul",
"instructions": QUESTION}
out = m.system_one(
state, {"ok": q})
p = out["answers"]["ok"]
print(round(p["noul"], 3))