Why and how to assess non-academic evidence so that it can inform improvements in policy and practice

Introduction

Why does education decision-making need a broader evidence base?

Despite consensus on the scale of the learning crisis—the global education sector continues to struggle to identify and scale context-sensitive solutions. Education decision-makers are actively seeking better evidence to inform these efforts, yet the conventional evidence base—dominated by peer-reviewed, academic research—fails to meet their need for practical, timely, and context-relevant guidance.

The omission of large volumes of non-academic work of great value – so-called ‘grey literature’ – severely limits the available evidence base. As described in the 2021 Education.org White Paper: Building an Education Knowledge Bridge, this also limits the perspectives represented in evidence, since much of this grey literature is produced outside of traditional academic publishing channels, including by NGOs, government agencies, civil society organisations, and practitioners—many of whom are embedded in the very contexts policies aim to serve.

What’s wrong with how we define and access evidence today?

Current definitions of 'good evidence' overly privilege academic research, often published in English-language journals and behind paywalls. This narrows access to insights as well as the scope of perspectives represented—particularly those from the Global South—and excludes the experiential insights of educators and local communities. Many practitioners and policymakers report difficulty accessing research or trusting guidance that lacks contextual relevance or practical orientation.

Grey literature is increasingly recognised for its potential to fill these gaps, yet many systems still exclude it from formal evidence reviews. According to a landscaping exercise to understand the use of grey literature, only 9 of 26 international initiatives reviewed included it in their evidence definitions. Even then, most imposed academic-style quality standards that many of these sources were never designed to meet.

In Education.org’s synthesis providing insights on improving and standardising the quality of accelerated education, nearly 80% of the useful sources were unpublished, non-academic literature. In our synthesis focused on successfully transitioning out-of-school children into the formal education system, we doubled the number of countries included by taking a broad approach to what counts as useful evidence.

The health sector has recognised the value of grey literature for its documentation of practitioner experiences and local context, which helps indicate how and why things work (Kothari et al. 2012; Lewin et al. 2024).

How do we define ‘grey literature’?

The most widely accepted definition of ‘grey literature’ is the 2010 Prague definition:

Grey literature stands for manifold document types produced on all levels of government, academics, business and industry in print and electronic formats that are protected by intellectual property rights, of sufficient quality to be collected and preserved by libraries and institutional repositories, but not controlled by commercial publishers; i.e. where publishing is not the primary activity of the producing body (Schöpfel 2010).

This Guidance include a slightly wider scope to include products from think tanks, political parties, funding agencies, international non-governmental organisations (INGOs), civil society organisations, and schools and other educational institutions, regardless of their governance. These sources may be research. But they may also not be research; they may be descriptive, analytic, or first-hand accounts providing contextual information relevant to decision-makers.

How does this Guidance aim to address the problem?

Education.org and its International Working Group (IWG)—a diverse network of 27 organisations across multiple continents—developed this Guidance to build on existing appraisal guidance and to develop a new, coherent, and intellectually rigorous system for making a wider range of evidence easier to identify, access, and use in educational decision-making (see Annex 1 for a list of members).

Through a year-long, iterative process, the IWG collectively developed the Guidance and held consultations in a variety of forums. The draft Guidance was tested in Kenya and Sierra Leone, then revised by Education.org to support analysts, researchers, and evidence users in three key

1. Framing analytical questions: Guidance on crafting relevant, policy-facing questions that reflect the concerns and priorities of decision-makers
2. Identifying and accessing evidence: A systematic approach to search for both academic and grey literature, emphasising inclusion of local and underrepresented perspectives.
3. Appraising evidence for use: A novel appraisal tool designed to assess relevance, inclusivity, credibility, and methodological rigour across diverse source types.

This Guidance also builds on the LEARRN partnership’s Evidence Classification Framework, offering more technical tools for those conducting research and analysis.

Who is this Guidance for?

While researchers and evidence synthesisers are its primary audience, the Guidance is also designed to inform actors within the broader education sector.

When should this guidance be used?

This Guidance is most useful in the early stage of designing research reviews or conducting a secondary analysis. It guides you in framing your questions from an education decision-maker’s perspective. It helps point you to evidence sources you may not have considered. This contributes to understanding what is known and helps you determine if more data needs to be collected. Once you accessed an evidence source, the Guidance helps you assess the quality and relevance of those individual sources. This Guidance does not assist in the analysis or in the design of a new piece of primary research.

What impact do we hope to achieve?

By promoting inclusive, systematic methods for using both academic and non-academic sources, this Guidance aims to:

• Elevate diverse, often-marginalised voices in evidence-informed education policy;
• Increase the use and visibility of locally generated insights;
• Promote transparency, trust, and relevance in policy-relevant research.

Ultimately, the goal is to strengthen the evidence-policy-practice bridge, creating a world where education outcomes are improved due to better decision-making, which is informed by the most relevant and useful evidence available.

For further information, editable tools, and training, contact:

Part 1. Framing the analytical question

Why framing a good analytical question is important

In the context of evidence-based policymaking, an analytical question is a question that asks you to examine data, compare information or results, and explain the reasons behind certain outcomes or decisions. Instead of just asking for facts, it challenges you to explore why something happened, how different factors are linked, or what solutions might be most effective. Framing an analytical question is a critical first step in the process of identifying, accessing, and appraising evidence. A well-crafted question is essential because:

• It guides subsequent steps, shaping what sources of information are most relevant, how to analyse them, and what conclusions can be drawn.
• It helps you stay focused, deciding which sources and what type of information to look for, ultimately ensuring your findings are useful and applicable in a given context.
• It ensures a solution-oriented approach, collecting evidence on a specific question, urgent challenge, or pressing policy topic to better inform education decision-makers.

This is especially important when trying to inform decision-making processes. Without a strong analytical question, research or analysis can become too broad, unfocused, or irrelevant, making it difficult to generate clear, practical conclusions or actionable ways forward for decision-makers.

What are the different types of analytical questions

An analytical question informs how information is scrutinised in order to generate insights for policy or practice. For example, a question can be descriptive, explanatory, exploratory, comparative, or evaluative. The table below provides a brief explanation and example questions for each of these types of analytical questions.

Analytical questions do not always fit neatly into just one of the types listed above. Many questions overlap, combining elements of multiple types of questions to examine a topic in a deeper, more nuanced way. Below are a few examples to illustrate this:

Example question #1: How do differences in school infrastructure across urban and rural areas influence primary student attendance in sub-Saharan Africa?

• The question is descriptive because it documents what school infrastructure looks like
• The question is comparative, because it examines differences between infrastructure in rural and urban areas
• The question is also explanatory, because it seeks to identify causal relationship between school infrastructure and its impact on student attendance.

Example question #2: What factors contribute to the success of non-formal community-based education programmes in fragile and conflict-affected countries, and how do they impact student learning outcomes?

• The question is exploratory because it seeks to identify success factors.
• The question is explanatory because it connects the cause-and-effect relationship between the success factors and student learning.
• The question is evaluative because it assesses the impacts of community-based education models.

Note that you may want to craft multiple analytical questions to capture the different purposes of your study. For example, you may have a main question, with various sub-questions, such as:

What factors contribute to the success of community-based education programmes?

• How do these factors impact student learning in literacy and mathematics?
• How do these factors impact student social-emotional learning?
• What barriers exist to scaling effective programmes or practices?

What a good analytical question looks like

A well-structured analytical question should be clear and focused. This means it should address: (1) the topic and scope of the analysis, (2) the target population or context, and (3) the purpose of the inquiry.

•Topic and scope: What issue is being addressed?

•Target population and/or context: Who or what is affected by this issue?

•Purpose of inquiry: What decision or action will the findings support?

Below we provide guidance on how to craft a strong analytical question.

Part 2. Identifying and accessing sources

What is the purpose of a systematic search plan

Our objective is to use the best available evidence for decision-making. There are many settings where evidence is in short supply. By broadening one’s search to include research and non-research, published and unpublished sources, you are likely to find sources you were not aware of. However, this requires looking in the right places for the right sources. Because searching for grey literature sources is more complex than searching for published research, it is important to prioritise what you are looking for in terms of your analytical question, geographic focus, and timeframe. This section provides guidance on identifying and accessing relevant grey literature sources with the aim of being systematic and reducing bias in that sourcing.

What is the process for developing and implementing a systematic search plan

The process for developing and implementing a systematic search plan involves first identifying the types of sources needed to answer your analytical question (Step 1), then defining your additional search parameters (Step 2), and finally recording or tracking your searches, search outcomes, and challenges faced (Step 3).

Step 1. Identify types of sources to answer your analytical question

After creating an analytical question (Part 1), it is important to identify the most appropriate sources of information to answer that question. For example:

Example: If your interest is in how a programme is expected to be implemented and how it is being implemented, your search should seek:

• Research sources: Evaluations; Dissertations and theses; Technical Reports; Systematic reviews;
• Government/Official sources: Research reports; Guidelines; Memoranda/circulars
• Organisational/Company sources: Project/programme implementation reports; Evaluations; Manuals; Technical specifications and standards

As a first step, identify the types of sources that serve your analytical questions, drawing from Table 1 below. Table 1 is organised using the five classification clusters presented in the Introduction. It suggests where to search for each classification cluster and provides numerous examples, however, it is not comprehensive. This is just a starting point, and we encourage users to consult local education experts, librarians and other information specialists to identify more comprehensive sources of evidence depending on their needs and contexts.

Step 2. Define key search parameters

The previous section helped you identify which sources are most relevant to answer your analytical question. Next you will need to define additional parameters to guide your search. It is important to think systematically about a search, recognising that it is not possible to be comprehensive in capturing all grey literature sources that may be available. Instead, your search parameters will determine what is included in your analysis, and what may be excluded, or out of scope. For example, ask yourself:

• What websites, repositories, databases, and/or physical places will I go to find sources? While Table 4 in the previous section provides an initial list to guide your planning, you may want to create a more comprehensive list, including local organisations, institutes, government offices, universities, or libraries.
• What publication dates will I include? You may determine that you want to look at studies from the past 5 or 10 years, or after a specific and relevant date (e.g. after a 2003 policy reform, since a new competency-based curriculum was launched in 2010, or after the Sustainable Development Goals were established in 2015).
• In what languages will I accept sources? You may decide to only include sources in one language, or to include multiple, local languages, including indigenous languages. Importantly, this decision should be determined based on the language capacities of your research team.

By being transparent and systematic about what you include or don’t, it helps you to mitigate biases in the search process. Bias can be introduced in any search process. Mapping a search strategy and documenting the process will help mitigate biases introduced by depending on sources that are not relevant, credible, or representative.

Step 3. Record all searches, outcomes, and challenges

To ensure the search is systematic, it needs to be organised, with an accurate record of places searched and details on how the search was carried out. An example of this can be seen in Mitchell and Rose (2018). A search checklist template created in a spreadsheet, or a document table, can help keep the search focused. See Annex 2 for a Search Checklist Template.

It is recommended that, at a minimum, the following be tracked:

• Date the search was conducted
• Name of where you searched: database, repository, search engine, organisational website, or physical office.
• The URL of online platforms searched or addresses for physical offices.

If you need to outline your search methodology for an external audience, more extensive information will need to be recorded, including the filters used.

• Geographic coverage
• Search terms used
• Date ranges searched for
• Indicate search outcome, e.g. ‘searched, nothing found’; ‘searched, not relevant’ ‘searched, results found’; or ‘searched, results may be of peripheral interest’

Keeping track of this information has several benefits. It will help maintain the focus of the search and ensure the necessary information is captured. Making it a transparent process, such as by adding search details in the write-up, helps to amplify the work of NGOs, CSOs, and researchers and highlights the value of such evidence. A comprehensive record of your searches and their results will also help better prepare you for subsequent follow-up studies on the same topic, saving yourself time in identifying relevant sources. Further, searching for grey literature comes with challenges (described in the box below). Documenting the challenges you encounter is also important, as it will help you later reflect on which places searched gave more relevant sources than others. Google may provide many, but how relevant were they compared to other types of searches? What sort of challenges did you encounter while searching and how can authors or publishers make their sources more easily accessible?

Part 3. Appraising sources

What is the purpose of appraising sources

The process of appraising sources serves as a quality control mechanism. After identifying and collecting sources, it is important to use strong appraisal methods to ensure high-quality evidence. Traditional methods for appraising research are robust[1] but too limited for appraising non-research sources, like much of the grey literature sources covered by this Guidance. Additionally, other tools to appraise grey literature sources[2] do a good job in establishing source credibility but often miss key elements necessary for education decision-making.

This Guidance focuses on using evidence to bridge the gap between research and education policy, aiming to improve educational outcomes through evidence-based decisions. Therefore, this tool focuses on how well sources answer questions relevant to education decision-makers. This approach ensures that diverse types of evidence are considered in educational decisions. This Guidance builds on existing tools to offer a new, inclusive method for appraising individual sources for use. This tool is tailored to meet the needs of education decision-makers, focusing on how well sources answer relevant questions.

Introduction to the Appraisal Tool criteria

 These four sets of criteria translate into the Appraisal Tool (Annex 3). Each criteria has sub-questions that are assessed as:

• ‘Yes’ the source meets the criterion.
• ‘No,’ the source does not meet the criterion.
• ‘Unclear,’ there is no information available to answer the question.
• ‘N/A,’ the question is not applicable for this source.

Users are asked to also include information describing why they selected their answer. This is to support their assessment of each criterion and is especially important if a ‘No’ or ‘Unclear’ assessment was determined. For the most part, a ‘No’ or ‘Unclear’ answer on a specific criterion is a red flag, meaning it may indicate limitations of a source’s use, credibility, or diversity of voices, but does not necessarily mean that a source is unusable or not credible overall. However, it may raise questions and considerations, or require further investigation before use.

How to use the Appraisal Tool

The figure below describes the step-by-step process for appraising an individual source. Some criteria will be used to assess all sources (e.g. Relevance and Inclusivity in Step 1, Limitations and Biases in Step 3). However, during Step 2 you will answer two Screening Questions to determine whether the source is research or non-research. The Screening Questions will determine the criteria you use to appraise either Methodological Rigour (for research) or Credibility (for non-research). The final Step 4 is the same for all sources and asks to do a complete Final Assessment based on all the criteria. These criteria are explained in the following pages, and a template is provided in Annex 3.

[1] Such as the Critical Appraisal Skills Programme (2018), Mixed Methods Appraisal Tool (Hong et al., 2018), and International Development Research Centre’s Research Quality Plus Assessment Instrument (IDRC, 2022).

[2] For example, Assessing unConventional Evidence (ACE) tool (Lewin et al., 2024), AACODS Checklist (Tyndall, 2010), and the Joanna Briggs Institute Critical Appraisal Checklist for Text and Opinion (McArthur et al., 2015).

As indicated above, the criteria you use to appraise a source will depend on whether the source is research or non-research. The Appraisal Tool includes a set of Screening Questions (Step 2) to determine whether the source is research or non-research.

If the Screening Questions determine that the source is research, then the user is directed to use the Mixed Methods Appraisal Tool (MMAT) for appraising the methodological rigour of grey literature. We use the MMAT because it is the most versatile appraisal tool for using sources with a wide range of research methods. The MMAT includes appraisal questions for five different types of research methods: qualitative research, randomised controlled trials, non-randomised studies, quantitative descriptive studies, and mixed methods research. Using an appraisal tool that covers a wide range of research methods is thus essential when using grey literature sources.

Step 1. Appraise for relevance and inclusivity

Step 1 of the Appraisal Tool requires assessing a source against the Relevance and Inclusivity criteria. The sub-questions used to assess these criteria are found below in Table 2.

Step 4. Conduct a final assessment

The Appraisal Tool should be used to evaluate each individual source of evidence. Step 4 of the Appraisal Tool asks for an overall Final Appraisal Evaluation. While it is up to each user (or team of users) to determine whether a source meets their unique needs, the aim of the Appraisal Tool is to support a consistent and structured assessment of sources. The criteria should help you in determining whether a source is relevant to your analytical question or country context, whether it draws on diverse perspectives, and whether it is credible or methodologically rigorous.

In the final assessment of the source overall, users will have to use their judgement to determine whether or not to include or exclude a source. To do this, look across all the criteria in the tool and indicate one of the final assessment options:

• Include: If there are few limitations identified (i.e. where the user selects ‘Yes’ for most criteria) then they may want to indicate ‘Include.’
• Include with reservations: If most of the criteria are met with a ‘Yes,’ but perhaps some were ‘No’ or ‘Unclear’ pointing to gaps, weaknesses, or limitations.
• Probably not include: If most of the criteria are met with ‘No’ or ‘Unclear’ and there are only a few ‘Positive’ responses.
• Exclude: If almost all of the criteria are met with ‘No.’ This likely means the source is either irrelevant to your context, does not provide insights to answer your analytical question, or is full of biases and limitations that it may not constitute credible evidence.

Your final assessment does not necessarily determine what final decision you make. Importantly, you and your team should meet to discuss the sources reviewed. To minimise bias and enhance reliability, each source should be scored independently by at least two reviewers, followed by a comparison and discussion of results to reach consensus. A blank Appraisal Tool can be found in Annex 3, with space to explain your final assessment. If two reviewers assess a source as ‘Not recommended for Inclusion’ or ‘Exclude,’ the source should likely be excluded.

However, how your final decision on whether or not to include a source will also depend on how much evidence (or how many sources) you have access to in the first place. For example, you may end up choosing to include sources marked ‘Include with reservations’ or ‘Probably not include’ if they are the only sources providing the perspective of a specific stakeholder group (e.g. voices of teachers or students) or if they are the only sources found covering a particular marginalised group (e.g. students with developmental disabilities). In other words, your decision on whether to include a source should be made while considering the entire evidence base, as this will be the information you use to answer your analytical question. It is important to note that using a wide range of types of sources can help complement and strengthen the analysis. As such, it is important to aim for a comprehensive analysis by appraising a range of sources.

Considerations for adapting the appraisal tool

The appraisal tool is designed to be adaptable, enabling diverse users (researchers and non-researchers) to assess the quality of a wide range of evidence, and different types of sources. Users should consider tailoring the application of the tool to their specific needs, evidence types, and thematic foci. Some practical ways to contextualise the tool are described below.

Develop clear indicators to apply each criterion: The Appraisal Tool should be applied consistently and objectively by each user within a given project or study. To help make this happen, consider adding specific indicators to guide users on what to look for when assessing each criterion on different source types. For example:

It is important that these indicators are clearly defined early on. You should test whether the tool is being applied consistently, by having different users assess the same source, and seeing if they come up with the same answer. Sample indicators for the MMAT tool can be found in Hong et al.’s (2018) MMAT user guide.

Adapt the scoring framework: To differentiate the quality of evidence more precisely, consider a scoring system that goes beyond binary options like "Yes" (1 point) and "No" or "Unclear" (0 points). For example:

Use the tool for prioritisation: When managing large volumes of evidence, such as in systematic reviews, the appraisal tool can also serve as a mechanism for prioritisation. High-scoring sources can be flagged for inclusion in deeper analysis or synthesis, while lower-scoring evidence can be deprioritised or used to highlight gaps.

Conclusion

This Guidance aims to make a methodological contribution to the education sector, serving as an approach that contributes to the use of the best available evidence in education decision-making. It is written for a wide range of actors who are producing and promoting the use of evidence in education policy and practice. It is hoped that it supports greater understanding of, and appreciation for, the wide range of education evidence sources beyond traditionally published research and elevate locally driven contextual insights that often are excluded through traditional sourcing approaches. The roadmap for identifying and accessing grey literature or non-academic sources, and the tool for appraising them, can lead to knowledge products that better serve education decision-makers and contribute to a ‘world where the education of all children and young people is transformed by the best evidence’ as envisaged by Education.org.

Users are encouraged to send their feedback on this Guidance to . This will help improve subsequent iterations of the Guidance.