How to evaluate an AI summary
Five criteria for evaluating AI-generated summaries: verifiable citations, faithfulness, coverage, acknowledging missing data and human review.
Language models summarize quickly and with good wording. That is precisely the risk: a well-written summary conveys confidence even when it contains a fact that was not in the sources. In monitoring, where a summary may end up in a report for management, the useful question is not whether the text sounds right, but whether you can check what it says.
These five criteria can be used to evaluate any AI-generated summary, regardless of the tool that produces it.
1. Verifiable citations
Every relevant claim should point to the document it comes from. A citation is verifiable when it meets three conditions:
- It exists. The link opens a real, accessible document.
- It is specific. It leads to the specific document and, if possible, to the passage, not to a site’s home page.
- It supports the claim. When you read the cited passage, you find what the summary says.
The third condition is the easiest to overlook. A summary can cite an authentic source and attribute to it something that source does not state. Checking that the link works is not enough: you have to read.
2. Faithfulness
A faithful summary does not add, change or exaggerate. The typical problems are:
- Altered data: figures, dates, names or titles that do not match the original.
- Lost nuance: “is proposed” turned into “was approved”; “could” turned into “will”.
- Crossed attributions: what one source said appears attributed to another.
- Added causality: two facts that the sources present separately appear linked by a “due to”.
- Outside knowledge: the model fills in the text with information that is not in the documents it was given.
To check faithfulness, take each figure, date and proper name in the summary and look for it in the cited source. It is a mechanical check, and it is worth starting with it because errors in specific data do the most damage when they reach a report.
3. Coverage
A summary can be faithful in everything it says and still mislead through what it leaves out. Ask yourself:
- Does it capture the main facts from all the documents, or only from the first one?
- If the sources disagree, does it show the disagreement or pick one version without saying so?
- Does it include the data your reader needs to decide: who, what, when, whom it affects?
- Does it keep proportion? A secondary detail should not take up more space than the central fact.
A practical way to assess coverage is to write down, before reading the summary, the three or four points you would consider essential after reading the sources, and then check how many appear.
4. Ability to say “not enough data”
A good system recognizes when the sources fall short. If you ask about the impact of a regulation and the available documents only announce its publication, the correct answer is that there is not enough information, not a paragraph of plausible generalities.
You can test this deliberately:
- Ask a question whose answer you know is not in the documents.
- Ask for a specific piece of data, such as a date or an amount, that the sources do not mention.
- Ask about a topic close to, but different from, the one the sources cover.
If in those cases you get a confident, detailed answer, you have a fundamental problem: you will not be able to tell well-founded answers from unfounded ones. A “not stated in the sources” is a sign of quality, not a defect.
5. Human review
None of the previous criteria removes the need for a person to review. What changes is how much and where:
- Full review for everything that leaves the organization or reaches a decision-making body.
- Sample review for high-volume summaries for internal use: take a periodic sample, compare it with the sources and log errors by type.
- Enhanced review for sensitive content: regulations, financial figures, claims about people or organizations.
The reviewer should have the summary and the sources side by side, and be able to correct or discard easily. If reviewing costs more than writing from scratch, the review will stop being done.
Checklist
Before accepting a summary, confirm that:
- Every relevant claim has a citation, and the citation opens the correct document.
- Figures, dates and names match the original.
- Verbs keep the source’s degree of certainty.
- There are no conclusions that the sources do not draw.
- The essential points are present.
- Disagreements between sources are flagged.
- What is not known is stated explicitly.
Common evaluation mistakes
- Judging by fluency. The quality of the prose says nothing about accuracy.
- Evaluating without reading the sources. Without the original at hand, you can only comment on style.
- Testing only with easy cases. It is worth including contradictory sources, long documents, tables and texts in several languages, which are more demanding situations.
- Evaluating only once. Behavior can change when the model, the instructions or the type of source change; evaluation should be repeated.
A simple rule
If you cannot get from the summary to the source and find there what the summary claims, treat it as an unverified draft. On the role of citations in working with agents, see the AI agents page; and to include an already reviewed summary in a deliverable, how to put together a useful newsletter.