Evidence-based COVID-19 policy making in schools
New research could help policymakers make evidence-based decisions about the risks and benefits of personalized education; strategic use of the available data will be the key to doing this right.
Policy questions such as how best to reopen schools for personal learning during a pandemic are incredibly important, but also incredibly difficult to answer in an evidence-based way. As with many other policy decisions related to COVID-19, school reopening strategies were initially devised with minimal direct evidence. Recently, however, an increasing number of empirical studies are building a scientific basis on the risks and benefits of personal education, including two studies in this issue of naturopathy.
Credit: Prostock studio/Alamy Stock Photo
First, Ertem et al.1 Use data from the United States to examine the effects of different education models on COVID-19 cases by comparing counties with face-to-face education versus those with hybrid or virtual education. In a similar vein, Fukumoto et al.2 compare the number of COVID-19 cases in Japanese municipalities where schools opened with comparable municipalities that kept schools closed. Neither study found a consistent link between school reopenings and the number of COVID-19 cases — but the findings varied by context, especially within the United States. Because school policymakers now have to weigh up evidence from different studies (often with results that seem contradictory), it’s important to keep three considerations in mind: the causal question being asked, the comparisons being made, and the context to which the findings relate.
For policy decisions, we are almost always interested in a causal question — that is, one that compares outcomes (e.g., case rates) in two different possible states of the world. In one state, a well-defined group experiences the intervention of interest (e.g. school reopening); in the other, the same group experiences a comparison condition (such as continuous virtual learning). This equation immediately raises the ‘fundamental problem of causal inference’3 — that for a particular school at any time we can only observe results in one state (eg school reopening), while the other state is either unobserved or ‘counterfactual’. So we are forced to use data from different groups – or in this case different schools – to estimate what would be seen in the same group under the intervention state versus the comparison state; in other words, a ‘causal contrast’. Appropriate causal inference therefore requires strong research designs such as randomization, longitudinal evaluation of communities with schools that did and did not reopen (as in Ertem et al.1), and/or well-selected comparison groups (as in Fukumoto et al.2). Robust designs allow a reasonable estimate of what would have happened in the communities with schools that had reopened, had they actually remained closed.
While others have rightly emphasized the importance of study design in answering pandemic-related causal policy questions4,5,6, we argue that policymakers should also ‘keep it simple’. In particular, most causally oriented studies can be evaluated in terms of their question, comparison and context. By asking whether these three components of a given study seem reasonable — and to what extent they apply to a current decision — policymakers without extensive methodological expertise can make a quick assessment of the relevance of a particular study.
For example, consider a district school board evaluating the results of the Ertem et al. study.1 to decide whether personal learning should be curtailed in the face of a new pandemic wave. This study is concerned with provincial-level decisions about opening or closing personal learning – not, for example, decisions at the state or national level. In addition, the specific comparisons made in this study would only apply directly to an “all-or-nothing” closing decision, as few counties took an approach to keep primary schools open, but middle and high schools closed. In context, different results were seen in the South than in other regions – and results in the United States are not necessarily generalizable to other countries. But by focusing on the question, comparison, and context, non-expert decision-makers could reasonably estimate the relevance of this research to their policy decision. We need to encourage – and make more accessible – this kind of thinking by emphasizing these three elements in any analysis that attempts to estimate a policy-relevant causal effect.
Unfortunately, there is often a discrepancy between the questions, comparisons and contexts addressed in research studies and those that policymakers need to consider. As for the question posed, Ertem et al.1 and Fukumoto et al.2 both take into account policy decisions at area level; other studies of personal education have focused on the behavior of individual households7. Some studies have compared ‘school reopening’ to ‘school closure’ in general, while others have attempted to estimate the effects of specific mitigation strategies. But these may not be the questions local policymakers need to answer; even randomized studies in schools are not always directly relevant for local decision-making if the study population is too different from the population of policy interest8 or the strategies studied differ substantially from the policy options on the table. For example, a recent study compared daily testing with isolation for close contacts of individuals with COVID-199 — but many school systems may be interested in less frequent testing or other strategies for children versus staff.
Analyzes can and should evaluate differences in estimated effects in different contexts, but these explorations are often limited by the available data. Although Ertem et al.1 While highlighting interesting variation in the estimated effects of school closures in US regions, they also note that they are unable to pinpoint accurate explanations for this variation, including inconsistent mitigation strategies, weather-related factors, or differences in underlying transmission rates of SARS-CoV by the community -2. It is also worth noting that research studies use retrospective data, while policy decisions must be made in the present. Together, these challenges emphasize the importance of conducting research that is as close as possible to the actual policy decisions being considered in terms of questioning, comparison and context. If these diverge substantially, policymakers will fail to make decisions for lack of evidence, negating the significant effort to put a science base into this process. Rarely, general conclusions — for example, that reopening schools does not promote SARS-CoV-2 transmission — are appropriate.
It is therefore critical, when informing evidence-based decision-making, that researchers clearly state the causal questions, comparisons, and contexts, while using the most appropriate data and research designs available. In the social sciences, the UTOSTi framework (units, treatment, outcomes, settings and times) has helped to articulate some of these considerations10; we now need an equally simple guide for scientists and decision-makers asking policy-relevant questions about the COVID-19 pandemic. No single study will be relevant to all policy questions; therefore, we urgently need to build a diverse scientific base that matches the most probable we encounter – and then communicate those results to decision-makers in real time, using language that can be widely understood.
Ertem, Z. et al. Nature Med. https://doi.org/10.1038/s41591-021-01543-y (2021).
Fukumoto, K., McClean, CT & Nakagawa, K. Nature Med. https://doi.org/10.1038/s41591-021-01571-8 (2021).
Netherlands, PW J Ben. stat. associate 81, 945-960 (1986).
Bonvini, M., Kennedy, E., Ventura, V. & Wasserman, L. Preprint on https://arxiv.org/abs/2103.04472 (2021).
Goodman-Bacon, A. & Marcus, J. Survey res. Methods: 14, 153-158 (2020).
Haber, N., Clarke-Deeder, E., Salomon, J., Feller, A. & Stuart, E.A. Ben. J. Epidemiol. (in the press).
Lessler, J. et al. Science 372, 1092-1097 (2021).
Orr, LL et al. J. Policy anal. To manage. 38, 978-1003 (2019).
Young, B.C. et al. the lancet 398, 1217-1229 (2021).
Cook, TD J. Policy anal. To manage. 33, 527-536 (2014).
DWD was supported by a Hopkins Business or Health Initiative pilot grant and a Johns Hopkins University Catalyst Award. EAS was supported by a Johns Hopkins University Discovery Award and by the National Institutes of Health award P50MH115842 (PI: Daumit).
The authors declare no competing interests.
About this article
Cite this article
Stuart, EA, Dowdy, DW Evidence-based COVID-19 policymaking in schools.
Nat Med (2021). https://doi.org/10.1038/s41591-021-01585-2