How to tell whether an evidence review has actually reviewed the evidence

A policy team receives a systematic review. It has a DOI, a university logo on the cover page, and a references section that runs to four pages. Someone marks it "evidence base confirmed" and the briefing moves forward. This happens dozens of times a year across government departments, quangos, and commissioning bodies — and it is one of the most quietly consequential shortcuts in public policy. When Aurora Pioneer spent several months quality-assessing seventy-one systematic reviews produced in a single intervention field, the finding that kept surfacing was not that the reviews were fraudulent or even careless. It was that the gap between what a review claimed to do and what it had actually done was almost never visible from the abstract alone. The five questions in this article are designed to close that gap, in plain language, before a review gets cited in a briefing, a funding decision, or a parliamentary submission.

Question 1 — Does the search strategy actually match the review's claim? ¶

The title says 'a systematic review of interventions to reduce X.' The methods section, if you read it, lists three databases searched, one of which was last indexed two years before the review was published. No grey literature. No trial registries. No hand-searching of reference lists. That is not a comprehensive search — it is a selective one — and the conclusions the review draws about what works and what does not are therefore conclusions about what was easy to find, not about what exists.

The practical check is mechanical: read the search strategy table, not the abstract. Count the databases. Look for CENTRAL, PsycINFO, and any field-specific registry relevant to the topic. Look for a date range on the search, and compare it to the publication date. A gap of more than eighteen months without explanation is a flag. In Aurora Pioneer's sample of seventy-one reviews, twenty-three listed fewer than four databases, yet all twenty-three described their methods as 'systematic.' That word is doing a lot of work it has not earned.

Grey literature matters most in fields where commercially or politically inconvenient findings are less likely to reach peer-reviewed journals. If the review is about a government programme, a pharmaceutical intervention, or a technology a large vendor is promoting, and it has searched only published academic journals, you are almost certainly looking at a biased evidence base, regardless of how rigorous everything downstream of the search turns out to be.

Question 2 — Were the included studies actually comparable to one another? ¶

Pooling studies in a meta-analysis feels authoritative. A forest plot with a diamond at the bottom communicates certainty in a way that a paragraph of caveats never quite manages. But a meta-analysis of studies that measured different outcomes, in different populations, over different time horizons, using instruments that were never designed to be compared, is not a synthesis of evidence. It is an average of apples, oranges, and one thing that might be a small melon.

The question to ask is whether the review reports heterogeneity — specifically, the I-squared statistic — and what it says about it. An I-squared above 75 percent signals that the studies are producing results so variable that averaging them may be actively misleading. A good review names this problem and explores it. It runs subgroup analyses. It asks whether the heterogeneity can be explained by differences in population age, intervention intensity, or follow-up length. A review that reports high heterogeneity and then presents its pooled estimate without comment has done the arithmetic and skipped the thinking.

In the seventy-one reviews Aurora Pioneer assessed, thirty-one contained a meta-analysis. Of those, fourteen reported I-squared values above 60 percent. Only six of those fourteen included any substantive discussion of what the heterogeneity meant for the validity of the pooled result. The other eight presented the estimate prominently in their abstract as though it were a stable finding.

A forest plot with a diamond at the bottom communicates certainty in a way that a paragraph of caveats never quite manages.

Question 3 — Who funded the primary studies, and did the review look? ¶

Funding bias in primary research is one of the most replicated findings in the methodology literature. Industry-funded trials of pharmaceuticals are several times more likely to report positive results than independently funded trials of the same compounds. The same pattern appears, with some variation, in nutrition research, environmental health, and educational technology. A systematic review that does not record and report the funding sources of its included studies cannot tell you whether the evidence base it is synthesising is structurally skewed.

This is not a cynical point about researchers behaving badly. It is a structural point about incentives and publication. When Aurora Pioneer coded funding information across the seventy-one reviews, only nineteen reported primary study funding at all. Of those nineteen, twelve noted funding as a potential limitation but did not conduct any sensitivity analysis to test whether industry-funded studies were driving the effect estimate. Three reviews did. Those three were also the three that had the most candid abstracts.

Questions 4 and 5 — How recent is the evidence, and what did the review leave out on purpose? ¶

Systematic reviews age. A review published in 2019 with a search cutoff of 2017 may have missed a cluster of large randomised trials that ran between 2017 and 2024. If the policy question is live today, that gap matters enormously. The check is simple: look at the search cutoff date in the methods, not the publication date on the cover. In fast-moving fields — infectious disease, digital health, early years intervention — a review with a search more than three years old should be treated as historical context, not current evidence.

The fifth question is about scope exclusions, and it requires reading the inclusion and exclusion criteria carefully. Every systematic review excludes something: studies below a certain sample size, studies conducted outside certain countries, studies that used a particular comparator. Sometimes these exclusions are methodologically justified. Sometimes they are convenient. A review that excludes all studies with follow-up shorter than twelve months will look more optimistic about long-term effects than the evidence as a whole, because the null results from short follow-up studies are gone. A review that includes only randomised controlled trials in a field where most real-world delivery has never been randomised will have a high-quality evidence base that describes almost nothing about how the intervention performs at scale.

When Aurora Pioneer mapped the stated exclusion criteria against the resulting study pools in all seventy-one reviews, the most common pattern was geographic restriction that was never flagged as a limitation. Forty-four reviews excluded non-English-language studies. Twenty-seven of those forty-four made no mention of this in their limitations section, even though the excluded-language literature in their fields was substantial and documented. The practical implication is straightforward: before citing any review in a Canadian policy context — where French-language research, Indigenous community-based evidence, and provincial programme evaluations all exist in forms that English-only searches routinely miss — checking what was deliberately left out is not optional.

Seventy-one reviews, five recurring failure points, and the same basic problem each time: the gap between a review's claim and its execution is almost never visible without opening the methods section. The real question for commissioners is not whether to trust systematic reviews — they remain the best tool available for synthesising a body of evidence — but whether anyone in the decision chain has the time, training, and institutional permission to actually read them. What happens to policy quality when the answer to that question is no?