Calibration
How much we find
EvidenceLine measures retrieval coverage: how much of the indexed source record a search returns. This page explains that measurement and its limits. It does not rate, interpret, or draw conclusions from the material retrieved.
This is a ceiling, not a score
The number above is an upper bound. Real coverage is at or below it, never above. The method compares two independent searches, and two web searches are never fully independent: they read the same crawled web, so they agree partly for reasons that have nothing to do with how much exists. That inflates the estimate. We publish the inflated figure and tell you it is inflated, rather than publishing a flattering number quietly.
How it is measured
You cannot measure whether you found everything by counting what you found. You need the size of what you missed. So we borrow the method used to estimate a wild animal population from two separate surveys: run the same test query two independent ways, and look at the overlap.
If both searches keep landing on the same documents, few remain unseen. If each finds things the other missed, plenty more exist that neither reached. The overlap carries the information, not the total.
estimated total = (found by A x found by B) / (found by both) coverage = (what we saw) / (estimated total)
A higher number here is not always better
Coverage is a ratio, and the bottom of it is an estimate of how much exists. Search harder and you find more documents, but you also discover the pool is larger than you thought, so the ratio can fall while the product improves. That has already happened here once: adding search providers raised sources retrieved per test query by about three quarters and moved this figure DOWN, because the better we look, the more clearly we can see what we are still missing. Read the two numbers together, and treat any comparison across different measurement setups as meaningless.
Limits of this measurement
Measures the indexed web reachable through our search providers, plus X. It cannot see what no crawler reaches: most paywalled archives, records behind portals, closed platforms, anything offline. Findable is not the same as existing.
- Findable is not the same as existing. This measures the indexed web our providers reach. A document nobody crawled is invisible to the measurement and to us, and neither of us can tell you it is there.
- It is measured on one kind of input. FEVER contains short statements about people, works and places. A contested political topic, a breaking story, or something in a language our providers index thinly will behave differently, and probably worse.
- It says nothing about quality. Finding every source is not the same as finding a good one. Sources can be wrong, outdated, or all copies of each other. We show you what each one says so you can judge that yourself.
- The worst tenth is the useful number for a difficult search. If a topic is obscure, the typical figure is not the one that applies to you.
What we do not measure
Source reliability, meaning, agreement, or conclusions. EvidenceLine measures retrieval coverage only. It presents the source record for you to examine.