PANDA, the thinktank of actuaries, economists and mathematicians, has stirred up a heated debate about Covid-19 case rate and death toll forecasts. In a nutshell, they suggest that number-crunchers advising governments on health policies to safeguard people have got things wrong and their errors are to the detriment of the global economy. Scaremongering may have worked well with other epidemics, but inflating death projections is leading to severe hardship. It’s time to acknowledge the experts got things wrong, they argue, in another powerful, hard-hitting analysis of the disease outbreak models – Editor.
Reviewing the models used to justify lockdown
By Nick Hudson, Shayne Krige and Ian McGorian*Â
“Ordinary fortune-tellers tell you what you want to happen; witches tell you what’s going to happen whether you want it to or not. Strangely enough, witches tend to be more accurate but less popular.”
Terry Pratchett
At the beginning of the coronavirus pandemic, everyone with a pocket calculator had a model and a prediction for how many deaths would be caused. Epidemiologists claimed to have the best models. Modelling disease outbreaks was their speciality. They would therefore be taken more seriously by policymakers charged with formulating the response to the virus.
As the pandemic ripped through Europe, it became clear that epidemiologists did not have all the answers. Neil Ferguson of Imperial College London produced a model predicting 510 thousand deaths for the United Kingdom, reduced to 250 000 if lockdown was implemented. (and 2.2 million deaths for the USA) which prompted the UK government to abandon their “herd immunity” strategy and implement lockdown measures. By the time the model was disclosed to the public, it was clear that the UK was not on track for 510 thousand deaths. Experts who analysed the released code described it as “a buggy mess that looks more like a bowl of angel hair pasta than a finely tuned piece of programming.” The co-founder of WANdisco, a British data technology company said of the model, “In our commercial reality, we would fire anyone for developing code like this and any business that relied on it to produce software for sale would likely go bust.”
Towards the end of February, policymakers in South Africa looked to local epidemiologists for advice. Although government has refused to cite the information they relied upon to implement the National lockdown, we have scores of press reports that suggest that Government was presented with a model developed by Sacema, a unit housed at the University of Stellenbosch headed by Professor Juliet Pulliam, which predicted that at least 87,900, and up to 351,000, South Africans could die and that hospital admissions would exceed available beds by many multiples.
It seems to us that many epidemiologists see their role not as accurately predicting the future or even to present a range of outcomes, but to provide the requisite motivation to the government to allocate healthcare resources by literally putting the fear of death into them with big numbers. The process appears to involve choosing the biggest numbers that your model produces and presenting only those.
If this is the approach adopted it would explain why the initial Sacema model had predicted 20,300 – 81,300 deaths using an alternative scientific pre-print paper based on the Diamond Princess experience, only the highest figures produced based on the Hubei experience (87,900 – 351,000 deaths) found their way into the advice given to government. We have not encountered the lower range numbers anywhere in the print media and understand from a politician involved in the process that the lower numbers were not presented to him.
This scaremongering process may have worked well with other epidemics. Overstating the HIV problem, for example, might motivate a reluctant government to allocate more resources to the problem than they would otherwise and in the worst case this would result in more money being spent and a better healthcare system than government had intended. If anyone dared to find fault with that overspend, the accusation could be easily refuted by moralising about lives vs money and claiming success of the very measures implemented.
The key difference between the response to HIV and that to coronavirus is lockdown. Epidemiologists had no tools to measure the impact of the steps taken on the basis of their advice. The very concept of measuring the negative impacts was alien to them. Before lockdown was on the table, excessive numbers had never meant causing lives to be shortened and lost.
In the weeks after Panda produced its paper stating that the initial 21-day lockdown would cause at least 29 times more life years to be lost than Covid-19, Panda was criticised by healthcare professionals (including epidemiologists, actuaries and academics) who saw Panda as dangerous because government might use our work to justify inaction.
That criticism intensified as Panda was invited into a government modelling discussion. We used the opportunity to demand that all models be made public so that, like the Imperial College model, they could be reviewed. We wrote letters to all modelling institutions and brought access to information applications. Sacema was first to reply and you can read our correspondence here.
Analysis of the Sacema Model raised a number of questions.
Why was the model secret?
Sacema explained that their model had always been publicly accessible via the Internet; they had simply never disclosed the address publicly. Without any links being posted to the model from the Sacema website or referenced anywhere in the public domain, how was anyone to stumble across it? To the best of our knowledge, Prof. Pulliam only shared access to the model after we wrote to her and brought a PAIA application against Stellenbosch University to disclose it. By that stage, Prof Pulliam had no interest in keeping the model secret because, with the model having been revealed to be deeply flawed, Sacema had had to turn its back on it, evidence of this may be found on the landing page of the Sacema website.
Was the model used properly?
The Sacema model is stupefyingly simplistic weighing in at 25 lines of Python code and two Excel population data spreadsheets. It would be interesting to examine the methodology, the code and test the hypotheses. This opportunity hardly presents when it boils down to a few multiplications. The main issue with the Sacema model is that it made assumptions that were known to be false at the time. For example, the model assumed that less than 20% of infections were asymptomatic when the Diamond Princess study (referenced in the model) had already shown this figure to be 80%. In February, Fauci, Layne and Redfield had published a paper arguing it was higher than 85%. That change alone would have reduced the death prediction to a pessimistic scenario of 21,400 – 85,600 deaths. The top of this range would only have been struck under the implausible assumption that none of the non-infected passengers had been exposed to the virus. Under shipboard settings it is clearly more likely that most would have been, and furthermore, it is hard to believe that this kind of setting wouldn’t have led to much worse viral spreading than in a normal country. So anything in this range would still have been an obvious overestimation.
Crucially, the figures presented to government assumed that 100% of the population was susceptible and 40% of the population would be infected (the lower predictions that appear not to have been presented assumed 10% and 20% susceptibility). This despite the fact that other countries had shown peak cases at attack rates of 2.5 to 15% and the Diamond Princess petri dish, an attack rate of less than 20%. It seems that each of the four inputs into the model was adjusted beyond the maximum rate observed in the real world.
What was government told about the model?
Sacema stated that, although they provided their model to the National Institute for Communicable Diseases (“NICD”), an organ of State, they were not aware that the results of that model were being used to inform policy decisions.
Government’s reticence to say anything about what data they relied on in taking the decision to impose lockdown makes answering this question difficult. There is, however, a plethora of press articles suggesting that the Sacema model was the key piece of information that prompted the lockdown decision. The model used two sources of Chinese input date, the “Riou” data consistently produced scarier numbers (based on Hubei) than the “Russell” data (based on the Diamond Princess). With government and Sacema both refusing to provide further information, we can only go on the press reports which suggest that only the scariest numbers (Hubei) were presented to government.
Sacema further stated that the models were not designed to inform policy decisions (like lockdown) but were merely for “situational awareness”. This is a puzzling statement. The models were supposed to make government aware of a situation (that 351,000 South Africans supposedly might die), but Prof Pulliam did not expect government to react to that “situation”? It is hard to imagine why Prof Pulliam went to the effort of producing a model she did not think anyone would pay attention to; but it stretches the very limits of the plausible to suggest that she did not expect government to react to this staggering projection. For context, at the time, the globe’s attention was seized by a virus that had killed fewer people across the entire world than Prof Pulliam was predicting for South Africa alone.
Sacema would have read the press articles quoting their model and stating that it was the catalyst for imposition of the national lockdown, but it took no steps to clarify publicly that 351,000 was only the worst case scenario produced by its model and that the model was not intended as a tool for making policy. Sacema clearly wanted to remain part of the Corona inner circle and so toed, and continues to toe, the government line of non-disclosure.
How do they explain the inaccuracy of the model?
Prof Pulliam concedes that the data they used was inaccurate but says that it was the best data available at the time. As we pointed out in our letter to her, the Diamond Princess data was clearly superior to the Chinese data she relied upon.
Prof Pulliam also says that the model was not designed to be accurate, but was essentially a quick and dirty “situational awareness” model. This seems callous in the context of the shortening and loss of life that has flowed and will continue to flow from lockdown, quite aside from the financial hardship caused. This, coupled with the fact that Sacema so readily turned its back on the model and moved on to a different model without explanation or any apparent effort to warn the government or the public of its flaws, shows that it remained insouciant; stuck in an academic paradigm in which numbers had no context or consequence.
Was the best available data used?
In our article, “The Epidemiologists Who Missed the Boat”, we explained why the best available data at the time was the Diamond Princess data. It was what we used to develop our predictions that we have had to refine only marginally over all these months. This data was included in the model but apparently not used in the Sacema presentation to government. Prof Pulliam has explained this by saying that only a draft of the paper was available at the time, which is a thin argument, since the ultimate choice made referred to an article rather than a peer reviewed paper. Surely its drastic implications would have been cause for pause?
Did they adapt as quickly as they should have?
The Sacema model contained the Diamond Princess data but this output was seemingly not presented. Prof Pulliam’s statement that it came too late (implausible in any event as it is evidenced in the model) must be seen in that context. Rather than amend the model, Sacema abandoned it and began working on a new one. No public announcements were made so it is impossible to know exactly when Sacema and their model got divorced. As a result, the figures (up to 351,000 deaths) circulated for a long time after it seems Sacema knew it was unreliable.
Did anyone else do a better job?
Sacema’s explanation for their model’s inaccuracy is that it was based on the best information available at the time. This was false. Moreover, crucial inputs that Sacema chose for their model were demonstrably exaggerated. And a lower result (Russell – Diamond Princess) simply discarded.
In April, Panda noted that age-based mortality estimates from abroad suggested that SA deaths would not exceed 20,000. A month later it became clear that the rest of the world was suffering a much lighter epidemic than Western Europe and North America, causing us to formulate the “Panda Hypothesis” and challenge modellers to explain why they were forecasting more than 10,000 deaths.
The Actuarial Society of South Africa produced a model on 29 April that was consistent in its worst case total death estimates with the optimistic scenario painted by Sacema, forecasting 88,000 deaths. Its optimistic scenario, premised on an arbitrary assumption regarding the effectiveness of lockdown, involved 40,000 deaths. At Dr Mkhize’s modelling symposium on 21 May, all the modellers [list] stood up and presented strangely consistent forecasts in the 40,000 to 45,000 range.
Panda pointed out that pronounced lockdown benefits were nowhere in evidence, saying that “models that assume much higher herd immunity thresholds, much higher attack rates, much higher ultimate prevalence rates or equilibrium rates … all of those models would misread the approach of an actual lower threshold as a reduction in R0 … owing to … whatever intervention is being assessed. We think that’s a real problem at the moment … All the models … were asking South Africa to behave in a way that no other country in the world to date has behaved and to a level much more serious than the most serious countries that have been observed on an age-adjusted basis… We think that’s a big problem. The stakes are enormous in this game, as our paper earlier in the month showed. The consequences of lockdown are a vast humanitarian crisis in and of themselves. The causes of that are well understood, and we have to be careful not to use models that are inconsistent with the reality that has emerged elsewhere in the world, and that cause us to stay in a lockdown situation that has terrible consequences for the population of South Africa.”
It is hard to believe that the modellers present in that meeting did not hear that message or read it in the press thereafter. Yet none of them amended their approaches. None of them even accepted the invitation to engage regarding Panda’s findings. Perhaps they had decided to put the pandemic on hold and wait for them to be published. That would at least be consistent behaviour.
Where to from here?
In our second article, we will explore the models that were used to justify the continuation of the lockdown. As you will see, in most cases the institutions involved simply turned their backs on the prior work and moved on to new models. As those models proved to be inaccurate, they simply rejigged the input data to generate new numbers. The Panda prediction, simple as our model is, has been correct since May and has never been changed, recalibrated or abandoned. The time has come to recognise that the so-called experts have got it wrong numerous times and that we can no longer rely on their work to take decisions that impact the lives and livelihoods of South Africans.
- PANDA (Pandemics ~ Data & Analysis) is a multidisciplinary initiative seeking to inform policy choice in the face of COVID-19. Panda’s technical team brings to bear knowledge from the fields of actuarial mathematics, economics and medicine and is continually recruiting.
