select search filters
briefings
roundups & rapid reactions
before the headlines
Fiona fox's blog
A report published by the Institute of Economic Affairs looks at whether COVID-19 lockdowns ‘worked’.
Prof John Edmunds, Professor in the Centre for the Mathematical Modelling of Infectious Diseases, London School of Hygiene & Tropical Medicine, said:
“Did lockdowns reduce deaths? It’s an important question which should be addressed. Indeed, many researchers have tried to look at this question. A “new” study seems to suggest that they had very little impact. Not only that, the study in question is a meta-analysis that summarises the evidence from many studies. Indeed, the authors claim that they are just using the evidence from (in their view) the most valid type of study (so-called difference in difference studies). They even have a “quality score” for the studies that they did include. All good, you might think. Except it isn’t. First, by picking only a very particular type of study they ignore the vast majority of the evidence. Indeed, the report devotes page after page to why they are disregarding this (type of) study or that. If you ignore all the data that you don’t like, you tend to get the answer you were looking for in the first place. Second, their quality score is anything but. One of its criteria is whether the study was done by social scientists or not. So studies done by epidemiologists, for instance, (who might be expected to be the experts in epidemics, after all) are downweighed. Then, there is the critical issue of delays. If you implement a lockdown, then you wouldn’t expect to have an impact on deaths until about 3 weeks has passed, as this is the average delay between infection and death. That is, deaths would continue to rise for about 3 weeks after lockdown is implemented. Studies that ignore this can come to some wacky conclusions, like lockdowns cause an increase in deaths (if the study period is short). Indeed, the study that is most heavily relied on in this analysis does, indeed, ignore this delay. I could go on, but I won’t. There are very many problems with this study which have been pointed out by many people before. That is why I used “new” in inverted commas in my third sentence. This is the same study as did the rounds over a year ago. It still hasn’t been peer reviewed and would still struggle to get over that hurdle. Of the very many criticisms of the paper last year, the summary by “Factcheck” is an authoritative one:
“It is such a shame. We really do need to learn from the pandemic. We really do need unbiased, comprehensive, high-quality, data-driven analyses of the impact of different policies, including analyses on the effects of lockdowns on deaths. Although this is data-driven, it is far from unbiased, comprehensive and high quality. Its results should be treated with extreme caution.”
Prof Adam Kucharski, Professor in Infectious Disease Epidemiology, London School of Hygiene & Tropical Medicine (LSHTM), said:
“Much of this ‘new’ paper seems identical to an earlier May 2022 pre-print, so all the sensible criticisms made of the analysis back then – from a lack of accounting for epidemic dynamics to performing a ‘meta-analysis’ on datasets that aren’t independent – still hold.
“Despite the focus on lockdowns, this paper at the same time concludes that mask mandates are most effective at reducing COVID-19 deaths, more so than any other measure considered, including ones such as limiting gatherings, cancelling public events and closing businesses, which halted interactions all together. This seems at odds with basic logic: having an interaction while wearing a mask cannot be less risky than having no interaction in the first place.
“Ultimately, it’s a shame this report adds so little insight, because we do need rigorous analysis of how effective different measures were during dynamic COVID-19 waves – lockdowns were a blunt, last-ditch tool and countries need to find a better alternative for future severe pandemics.”
Dr Nick Davies, Associate Professor, London School of Hygiene and Tropical Medicine, said:
“It is hugely important to address the question of what policies short of lockdown could be used in future outbreaks, and under what circumstances.
“However, the Institute for Economic Affairs meta-analysis does not help to address this question. The process used by the authors for selecting the 22 included studies from among thousands of candidates is too susceptible to bias and to low study quality to be reliable.
“One illustrative example from the Institute for Economic Affairs meta-analysis is the finding that mask mandates are nine times as effective (18.7% reduction in mortality) than shelter-in-place orders (2.0% reduction in mortality). In the absence of a plausible mechanism that could explain this and other results, the validity of the paper is called into question.”
Prof Rowland Kao, Professor of Veterinary Epidemiology and Data Science, Roslin Institute, University of Edinburgh, said:
“There are of course many lessons to be learned from our approach to control of COVID-19. Any study which retrospectively analyses the approach of different countries may certainly provide some valuable inputs however it is important to consider that they have the advantage of information that was not available at the time and so are not in themselves valuable for telling what should have been done, given that much sparser information. Further, they do not represent a counterfactual analysis – i.e. what might have happened should alternative actions be considered. In particular, in the first wave of lockdowns a critical point was the large, tragic number of deaths in care homes – roughly a third of all deaths directly due to COVID-19 occurred there. These deaths were linked to the release of individuals in care homes directly from hospitals and therefore unrelated to the lockdown measures themselves. Should care homes have been better protected from that source of infection, the importance of reducing circulation in the general population and thereby reducing infection pressure on care homes, may have been much more apparent.”
Quotes we issued in February 2022 in response to a preprint by the same group on the same subject:
https://iea.org.uk/wp-content/uploads/2023/06/Perspectives-_1_Did-lockdowns-work__June_web.pdf
Declared interests
Dr Nick Davies: “former member of SPI-M.”
For all other experts, no reply to our request for DOIs was received.