Designing a programme evaluation that funders and practitioners will both actually read

Most evaluation reports land in inboxes and stay there. The funder finds the executive summary too thin, the practitioner finds the methodology section impenetrable, and the whole document quietly fails at both jobs it was hired to do. At Aurora Pioneer we have spent the past several years building evaluation frameworks for programmes ranging from early childhood literacy pilots to municipal employment schemes, and the pattern we keep finding is the same: the problem is not the quality of the findings. The problem is that the report was designed as a single document for a composite reader who does not actually exist. What follows is a behind-the-scenes account of how we restructure that process so the evaluation genuinely serves both audiences, without hiding the complexity from either.

The two audiences are not asking the same question ¶

Funders, whether a national government department, a philanthropic foundation, or a regional development agency, are primarily asking: did the money achieve what we said it would achieve, and should we continue, scale, or exit? Their accountability runs upward, toward boards or ministers or trustees, and they need findings they can defend in a governance meeting. They want confidence intervals, attribution logic, and clear statements about what the evidence does and does not support.

Practitioners, the coordinators, caseworkers, facilitators, and team leaders actually running the programme, are asking something structurally different: what should I do differently on Tuesday? They are not indifferent to evidence. Most practitioners we have worked with are genuinely hungry for it. But they need the findings translated into the operational reality they inhabit, which means specific, conditional statements rather than aggregate effect sizes. A finding that a youth employment programme showed a 14-point improvement in six-month job retention across all participants is useful to the funder. The practitioner needs to know that the improvement was concentrated among participants who attended at least eight sessions in the first ten weeks, because that changes how they structure onboarding.

When a single report tries to serve both questions from the same running text, it usually satisfies neither. The funder skips to the appendices looking for a logic model. The practitioner reads two pages and stops.

Start the design conversation at the same table ¶

The structural fix begins before a single data point is collected. Aurora Pioneer's standard practice is to convene what we call a findings-use workshop at the evaluation design stage, usually six to eight weeks before fieldwork begins. We bring the funder's programme manager, two or three frontline staff, and, where possible, one or two programme participants into the same room and ask each group to write down the three decisions they are hoping the evaluation will inform. Not the questions they want answered in the abstract, but the actual decisions sitting on their desks.

The outputs of that workshop are often surprising. In one recent evaluation of a community mental health navigation service, the funder's list included 'whether to fund a second cohort' and 'how to report outcomes to the national funding body.' The practitioners' list included 'whether to keep the drop-in Tuesday slot or move it' and 'which referral pathways are actually working.' These are not contradictory needs, but they require different evidence structures. Knowing both lists at the design stage means the data collection instruments can be built to genuinely serve both, rather than hoping the eventual report will somehow cover all of it.

Building the layered report architecture ¶

Once the decision lists exist, the report architecture becomes an engineering problem rather than a writing problem. Aurora Pioneer structures evaluations using a three-layer model. Layer one is a two-page findings brief oriented entirely around the funder's governance questions. It contains the headline outcome data, the confidence the evidence allows, and an explicit statement of limitations. It is written to be read in a meeting, cited in a board paper, or attached to a grant renewal application.

Layer two is the practitioner report, typically eight to twelve pages, structured around programme stages rather than research questions. Instead of opening with methodology, it opens with the part of the programme that showed the most variation in outcomes and explains what drove that variation in operational terms. The methodology is present but placed in a clearly labelled annex that practitioners can consult if they want to understand how a finding was reached, without having to read past it to get to something actionable.

Layer three is the full technical report, which includes the complete data tables, the sampling logic, any modelling decisions, and the evaluator's reasoning about alternative explanations for the findings. This layer exists for peer scrutiny, for other researchers building on the work, and for any funder who wants to go deeper. It is a public accountability document, not a communication tool. Keeping it separate means the communication tools can be genuinely communicative.

The report was designed as a single document for a composite reader who does not actually exist.

How the findings get stress-tested before publication ¶

Writing three separate layers sounds like three times the work. In practice it is closer to one and a half times, because the analytical work is done once and the layers draw from a shared findings database. But the stress-testing process is genuinely additional, and it is where the quality is actually determined.

Aurora Pioneer runs what we call a challenge session roughly two weeks before any report is finalised. The practitioner report draft goes back to two or three frontline staff who were not involved in the evaluation itself. Their brief is not to approve the findings but to identify any sentence that would cause them to stop reading, any finding that seems to describe a programme they do not recognise, and any recommendation they could not implement given their actual constraints. This is not a stakeholder consultation checkbox. It is a calibration exercise, and it has caught real problems. In one evaluation, a draft recommendation to 'streamline referral pathways' was flagged by a navigator who pointed out that two of the three pathways we were recommending they consolidate were mandated by separate funding streams with different compliance requirements. The finding was sound. The recommendation was not workable, and we rewrote it.

The funder layer goes through a parallel process, presented to the programme officer with a structured set of questions: does this accurately represent what the data shows, does it clearly state what we cannot conclude, and does it give you what you need to make the decisions you identified at the start? The last question is the most important. If the answer is no, the problem is usually that the design-stage decision list was not specific enough, and we go back to it.

What actually changes in the findings when you do this ¶

The most consistent effect of this architecture is not that findings become more optimistic or more critical. It is that they become more specific. When you know the practitioner report needs to be structured around programme stages, you collect data that can be disaggregated by stage. When you know the funder brief needs to address a specific renewal decision, you make sure the evaluation design includes the comparison data or baseline reference that makes that decision legible.

There is also a secondary effect on the honesty of the reporting. One of the quiet pressures on evaluation is the incentive to smooth findings into a coherent story, particularly when the programme being evaluated is politically important or when the relationship between evaluator and funder is ongoing. The three-layer model creates a structural counterincentive. The technical report is a public document, and its conclusions have to be consistent with the practitioner report, which has been tested against people who know what actually happened. A finding cannot survive in the practitioner report in diluted form if its full version sits in the technical annex three pages away. The architecture enforces a kind of coherence that a single-document report can more easily avoid.

A compact checklist for evaluation design ¶

The points below are not a methodology in themselves, but they represent the specific decisions that most often determine whether an evaluation report gets read and used or quietly archived.

• Run a findings-use workshop at design stage, not after fieldwork: document the actual decisions each audience needs to inform.
• Build three separate output layers — funder brief, practitioner report, technical report — from a single shared findings database.
• Structure the practitioner report around programme stages, not research questions; place methodology in a labelled annex.
• Run a challenge session with frontline staff who were not part of the evaluation itself, two weeks before publication.
• Check every recommendation against the operational and regulatory constraints practitioners actually face.
• Ensure the technical report is publicly accessible and consistent with both shorter layers — this is the coherence guarantee.

Evaluation reports are not neutral documents. The choices made about who they are written for, and in what sequence, and with what architecture, shape which findings get acted on and which get filed. Getting that architecture right is design work, and it happens before the data collection starts, not after it finishes. The three-layer model is one answer to that design problem, and it is one Aurora Pioneer returns to because it keeps producing reports that people actually open more than once.