Did a Johns Hopkins Study 'Prove' Lockdowns Don't Work? What We Know So Far

A working paper published on the Johns Hopkins University (JHU) website on January 31, has raised a question of the effectiveness of lockdowns as measures to reduce COVID mortality, fueling anti-lockdown narratives and speculative claims on social media and multiple news outlets.

The paper, which consists of a systematic literature review and meta analysis, was written by Steve Hanke, a professor of applied economics at the Johns Hopkins University and senior fellow and director of the Troubled Currencies Project at the Cato Institute, a libertarian think tank. His co-authors were Jonas Herby, a special adviser at the Centre for Political Studies in Copenhagen, and Lars Jonung, a Lund University economist. Fortune magazine described the authors as "free marketers." It has yet to be peer-reviewed, and it has received criticism on several fronts from the scientific community.

Newsweek has reached out to the Johns Hopkins University press office for comment, but were told to direct all inquiries about the working paper to Steve Hanke. Hanke has not yet responded to a separate comment request.

The paper is yet to undergo peer-review and has not been shared or promoted by JHU, with the author noting in the document that "views expressed in each working paper are those of the authors and not necessarily those of the institutions that the authors are affiliated with." But it is nevertheless being widely hailed and promoted by lockdown-skeptics as evidence that the policy has been a failure, with many inaccurately attributing the paper to JHU, even as it was neither commissioned nor endorsed by the university.

Several right-leaning outlets, including The National Post, The Washington Times and The Wall Street Journal reported on the study, but it was largely ignored by many mainstream or left-leaning outlets. The author's methodology has been criticized by some members of the scientific community, including leading immunologists, health policy experts and mathematicians.

How Was the Study Conducted?

Titled A Literature Review and Meta-Analysis of the Effects of Lockdowns on COVID-19 Mortality, the research says it attempts to determine whether there is empirical evidence to support the notion that lockdowns reduce COVID-19 mortality, through what it calls a "systematic review and meta-analysis of existing literature and studies on the subject."

According to the abstract, it focuses on a narrow time frame—the first half of 2020 (authors note that "most of the studies (29) use data collected before September 1st, 2020"). It also covers a wide array of restrictions—from stay-at-home orders to social distancing and compulsory mask wearing—under the umbrella term "lockdowns," which are referred to as "compulsory nonpharmaceutical interventions" (NPIs).

The authors note that the study employed "a systematic search and screening procedure in which 18,590 studies are identified," which after "three levels of screening," was whittled down to just 34 studies that ultimately qualified. "Of those 34 eligible studies, 24 qualified for inclusion in the meta-analysis," they explain.

Meta-analysis is a type of statistical analysis that combines and compares the results of multiple scientific studies. While this type of study in medicine may provide consolidated and quantitative review of a large and often complex body of literature, it also has its drawbacks. "A failure to identify the majority of existing studies can lead to erroneous conclusions," author A. B. Haidich warned in a December 2010 paper assessing the pros and cons of this approach.

What Did the Authors Conclude?

According to the authors, an analysis of three separate types of studies, including lockdown stringency index studies, shelter-in-place order (SIPO) studies, and specific NPI studies, demonstrated that "lockdowns have had little to no effect on COVID-19 mortality."

"Stringency index studies find that lockdowns in Europe and the United States only reduced COVID-19 mortality by 0.2% on average. SIPOs were also ineffective, only reducing COVID-19 mortality by 2.9% on average. Specific NPI studies also find no broad-based evidence of noticeable effects on COVID-19 mortality."

The authors note that some measures, such as the closure of nonessential businesses seem to have had some effect (reducing COVID-19 mortality by 10.6 percent), which they say "is likely to be related to the closure of bars." But they also highlight detrimental unintended consequences of the policy, finding "some evidence" that limiting gatherings was counterproductive and increased COVID-19 mortality.

Additionally, Hanke and his colleagues found that mask mandates appeared to be a highly effective measure, though this discovery doesn't appear to affect the bottom line of the analysis.

"Mandating face masks—an intervention that was not widely used in the spring of 2020, and in many countries was even discouraged—seems to have a large effect (-21.2%), but this conclusion is based on only two studies," the research states.

They conclude that while no noticeable effect on public health was observed, these measures "imposed enormous economic and social costs where they have been adopted. In consequence, lockdown policies are ill-founded and should be rejected as a pandemic policy instrument."

What Do the Critics Say?

After the study was published, it hit the headlines and started trending on social media. But some members of the scientific community soon began voicing their skepticism about the methodology, integrity and credibility of the research.

Among their primary concerns was the fact that the paper had not been peer-reviewed, that it had picked a very limited time frame to focus on, and that the vast majority of literature was dismissed by the authors, who chose to focus on just 24 papers.

"This study is flawed in numerous ways, and its conclusions are wrong," Seth Flaxman, associate professor in the Department of Computer Science, University of Oxford, who recently worked with the London Imperial College Department of Mathematics and School of Public Health to model the spread of COVID-19, told Newsweek.

"There are important open questions about which public health measures are the most effective at controlling the spread of COVID-19, but the authors did not engage in good faith with the epidemiological literature."

Flaxman directed Newsweek to his critique, published on the Science Media Centre (SMC) website, where he notes that the authors "systematically excluded from consideration any study based on the science of disease transmission, meaning that the only studies looked at in the analysis are studies using the methods of economics."

He also points to the limitations of using a narrow time frame for the analysis, which means key elements of disease transmission are not priced into the overview, including that later lockdowns are less effective than earlier lockdowns, and that lives are not saved "immediately," because there is a lag from infection to death.

"It's as if we wanted to know whether smoking causes cancer and so we asked a bunch of new smokers: Did you have cancer the day before you started smoking? And what about the day after?" Flaxman writes, concluding, "This study intentionally excludes all studies rooted in epidemiology—the science of disease."

Others focused on the study's extremely broad—and at times inconsistent— interpretation of what constitutes a "lockdown," with Samir Bhatt, professor of statistics and Public Health, Imperial College London, writing that under the authors' definition, mask wearing policy amounts to a "lockdown." He also notes that different countries imposed very different restrictions at different points in the pandemic, and the levels of enforcement and adherence also varied widely.

"[The study] looks at a tiny slice of the pandemic, there have been many lockdowns since globally with far better data," Bhatt concluded in his critique published on SMC.

A health policy expert contacted by Newsweek chose not to go on record, but pointed out that virtually every country had some sort of NPIs in place in the period studied, so there is no good "no-lockdown" counterfactual to carry out effective analysis.

Neil Ferguson, director of the MRC Centre for Global Infectious Disease Analysis, Jameel Institute, Imperial College London, is another vocal critic of the paper, noting on SMC that the interventions the authors assessed were aimed at reducing virus transmission rates, meaning that "impacts on hospitalization and mortality are delayed," which the study does not account for.

Ferguson admitted that extricating the precise impact of individual NPIs from a vast array of social and economic measures used in combination, as well as additional factors such as increasing vaccination rates or immunity gained through prior infect, remains highly challenging.

What Do We Know About the Authors and Underlying Studies?

Some have also raised concerns about the purpose of the research, and the reliability of the authors' method of selecting the material for the meta-analysis.

The critics expressed doubts about the claim that the only studies selected were those using the "difference-in-difference method," quasi-experimental approach that compares the changes in outcomes over time between the treatment group and the control group. In fact, a number of the select studies did not appear to include it in their methodology.

Questions have also been raised about the weight allocation for some of the studies in the meta analysis compared to others. Two studies in particular, Chisadza et al. (2021) and Alderman & Hajoto (2020), which found the impact of lockdowns to be smaller, appear to be weighted higher than most; while numerous studies that conclude the opposite have been excluded entirely.

The former is a study titled Government Effectiveness and the COVID-19 Pandemic by Carolyn Chisadza, a senior lecturer in Economics at the University of Pretoria in South Africa, and two co-authors. Though weighted heavily in the meta analysis, Chisadza's research offers a very different conclusion.

"We find that the overall government response index has a non-linear association with the number of deaths—driven by the containment and health interventions—for the aggregated sample of countries. The number of deaths increases with partially relaxed lockdown restrictions, but decreases with severe restrictions. We observe similar non-linear outcomes when we disaggregate the sample by global regions," Chisadza wrote in the study.

Herby, one of the co-authors, tried to address the various concerns in a FAQ, published January 30. Commenting, for example, on the significant weight of the Chisadza study, he noted that, as per their research, "Excluding Chisadza et al. (2021) from the precision-weighted average changes the average to -3.5%." The economist claims the reason is that " there are still too few studies that have examined the effect of shutdowns, and therefore one study may gain relatively high weight."

Though the authors insist that the selection of certain studies over others was merely due to the very narrow criteria of the meta-analysis, which they admittedly outline in a transparent way, it has led some experts to question the authors' motives in undertaking this research and their affiliations, given their previously stated opposition to the idea of lockdowns.

"This is a highly political "push/opinion piece" masquerading as "sober analysis," Jeremy Kamil, associate professor of microbiology and immunology at Louisiana State University Health Shreveport told Newsweek in an email. "One of the authors mentions affiliation with the Cato Institute, which as far as I know is a right-wing pro-business organization that is against governments doing anything at all about anything. The methods seem deceptive."

Is This a Game Changer?

While the findings from the Hanke study add to the body of research assessing the relative effectiveness of restrictions introduced by governments to tackle the spread of COVID-19, there are several significant caveats in the way of drawing any broad or global conclusions, from the lack of peer review to objective limitations of the study's scope and concerns about its interpretations of the underlying studies.

"This report on the effect of 'lockdowns' does not significantly advance our understanding of the relative effectiveness of the plethora of public health measures adopted by different countries to limit COVID-19 transmission," professor Ferguson wrote in his analysis.

Numerous studies in the past, including those which were peer-reviewed and covered far larger time spans (and some of which were dropped from this particular piece of meta-analysis) have shown the so-called non-pharmaceutical interventions—including lockdowns, social distancing measures, face masks and basic hygiene—to be effective at slowing the spread of the virus and, as a result, reducing mortality.

Similarly, basic common sense dictates that closing down society for an extended period of time, stopping business activity and telling people to stay indoors will have significant economic ramifications, a trade off that many governments around the world begrudgingly accepted (and some rejected).

And while the research of the relative impact and effectiveness of those measures is ongoing, the existing scientific consensus is highly unlikely to be overturned by a single piece of analysis.

As Flaxman concluded, "Smoking causes cancer, the Earth is round, and ordering people to stay at home (the correct definition of lockdown) decreases disease transmission. None of this is controversial among scientists. A study purporting to prove the opposite is almost certain to be fundamentally flawed."

Government-mandated lockdowns
The study by three prominent economists, which concluded that lockdowns have been ineffective at preventing deaths, has been panned by immunologists for, among other things, failing to differentiate between different types of government mandated restrictions. Above, Manchester, along with the rest of the U.K., was on strict “stay at home” orders on March 26, 2020. Christopher Furlong/Getty Images