PANDA, a group of experts such as actuaries and economists who use statistics in their fields, is focused on exposing the flaws in South Africa’s Covid-19 death forecasts. This is because these statistics are used by the government to justify the very strict lockdown which is “robbing” people of their livelihoods and doing little to nothing to curb the spread of the Covid-19 virus. South Africa’s death rate is far lower in relation to the number of reported cases than many other countries. On Sunday, President Cyril Ramaphosa reintroduced an alcohol ban and imposed a curfew from 9pm to 4am, effectively bringing to a halt business activity in the supply and distribution of alcoholic beverages and restaurants. The economic damage wrought by Covid-19 stretches across the economy, with Finance Minister Tito Mboweni warning last month that a sovereign debt crisis is a distinct possibility. In this article, PANDA lifts the lid on how the scientists behind South Africa’s official data have avoided responding to criticism about their work. – Editor
SACMC model continues to grossly overestimate Covid-19 deaths
By Piet Streicher, Shayne Krige and Nick Hudson*Â
In this second article in our series on the faulty science of South Africa’s modellers, we take up where our first article left off. When Sacema abandoned their own model (the catalyst for the initial lockdown), they joined the South African Coronavirus Modelling Consortium (SACMC) to work on a new model, the so-called “COVID-19 Epi Model”. The SACMC is a group of researchers from academic, non-profit, and government institutions across South Africa. The group is coordinated by the National Institute for Communicable Diseases (NICD), on behalf of the National Department of Health.
Panda had initiated correspondence with the NICD on 26 April when we expressed concern about both a failure to take into account the humanitarian consequences of lockdown and the overblown Covid forecasts.
At Dr Mkhize’s 21 May Modelling Symposium, Panda pointed out to the modelling consortium teams, all of whom presented there, that pronounced lockdown benefits were nowhere in evidence, saying that “models that assume much higher herd immunity thresholds, much higher attack rates, much higher ultimate prevalence rates or equilibrium rates … all of those models would misread the approach of an actual lower threshold as a reduction in Rt … owing to … whatever intervention is being assessed.
We think that’s a real problem at the moment … All the models … were asking South Africa to behave in a way that no other country in the world to date has behaved and to a level much more serious than the most serious countries that have been observed on an age-adjusted basis … We think that’s a big problem. The stakes are enormous in this game, as our paper earlier in the month showed.
The consequences of lockdown are a vast humanitarian crisis in and of themselves. The causes of that are well understood, and we have to be careful not to use models that are inconsistent with the reality that has emerged elsewhere in the world, and that cause us to stay in a lockdown situation that has terrible consequences for the population of South Africa.”
We invited the modellers present to engage with us. Barry Childs, who headed up the Actuarial Society of South Africa’s efforts did engage, but none of the consortium modellers did in any meaningful way.
Our message could not have been missed, especially given that it was accurately reported that same day in a national newspaper and that the next day we had published an article in these pages, called “The epidemiologists who missed the boat”. This article itemised the SACMC’s key hypotheses that were being breached. It ended with a call to the modellers to explain why anything more than 10,000 deaths should be expected in South Africa, instead of the lower-bound 40,000 to 45,000 that they had all mysteriously converged upon.
Covid-19 modelling flaws: A process repeated the world over
This phenomenon of private citizens having to point out atrocious science and being ignored is not unique to South Africa. As we later learned, South African-born Nobel laureate, Prof. Michael Levitt of Stanford University, had suffered the same frustration when he pointed similar facts out to Neil Ferguson, whose notorious Imperial College model produced forecasts with characteristics similar to those of the SACMC’s model (which was constructed in a similar manner to that of Imperial College).
Similarly, in the United States, Prof. John Ioannidis, an esteemed epidemiologist at Stanford, had early on pointed to the wildly exponentiating model structures and overwrought mortality rates that seeded them. He was not so much ignored as pilloried.
Ignored
None of the points Panda made seem to have landed with any of the SACMC’s many modellers, despite being quite stunningly clear in the international data — by which we mean the experience of every single country in the world that was sufficiently advanced in its epidemic at that time and at every location where seroprevalence studies had been conducted.
The upshot was that, when the SACMC published an updated report on 12 June (“Estimating cases for Covid-19 South Africa: Short term Projections: June 2020. Report Update: 12 June 2020”), it was destined to be invalidated quickly, notwithstanding their comically wide range of projected deaths (from 2,510 – 7,720 by 13 July for the Western Cape). Within days of their updated report, the actual numbers (orange triangles) were already deviating from the SACMC’s trend. As of yesterday, the actual numbers are below the optimistic SACMC prediction, meaning that we are now no longer on track for even the lowest number of deaths they projected.
Figure 1: SACMC cumulative deaths forecast for the Western Cape (June 12).
With the Western Cape’s cumulative deaths standing at 966, just a month out, the SACMC was forecasting that they would reach 2,510 to 7,720 on 13 July, with a best estimate of 4,750. Despite a range of predictions wide enough to drive a cruise ship through, the SACMC must contend with a 12 July Western Cape death toll that stands at 2,343. Just 30 days after their report, daily deaths are running at one fifth of the rate they projected. This means that the SACMC overestimated incremental deaths by 268%. The overall South Africa forecast is faring no better to date.
When we realised that we had once again been ignored, on 12 June, the same day as the SACMC update, we sent the SACMC a simple illustration applying the foreign experience to South Africa. In contrast to the SACMC’s comical range, we proposed a much narrower one (2,238 to 2,763 deaths by 13 July), which has withstood the test of time. In the graphic below, actual deaths are again marked by the orange triangles against our optimistic (blue line) and our pessimistic (gray line) projections :
Figure 2: PANDA projection (12 June).
Behind the scenes: Evidence against exponential growth
As can be seen in figure 1, the modellers expected exponential growth in deaths. We warned them that this was wrong in Dr Mkhize’s 21 May modelling symposium. In further correspondence with the SACMC’s contact person, Dr Harry Moultrie, between 26 and 31 May, we had pointed out that it was already clear that the Western Cape’s growth was slowing down. He argued that the apparent stop in exponential growth was caused by a testing capacity constraint and by the cessation of the community testing and screening programme. We retorted that daily deaths were also slowing, as were other indicators, such as hospitalisations, ICU bed occupancy and test positivity. The Western Cape Department of Health were copied on this correspondence.
Figure 5: Confirmed cases by test completion date stopped growing exponentially by 20 May.
Figure 6: Hospitalisations and ICU beds utilised stopped growing exponentially between 23 and 31 May.
Figure 7: Deaths stopped growing exponentially by 31 May.
At this stage, the SACMC terminated correspondence. This forced us into the approach described in our first article, which was met with a uniform lack of cooperation. Lockdown and its impact did not afford us the time to go to court to enforce our rights, so we resorted to undertaking a laborious and painstaking process of reverse-engineering the Epi Model. Through this work, we deduced that the SACMC were making incorrect assumptions regarding infection processes, and that incongruent susceptibility and mortality assumptions were also at play.
We are first to admit that it is unreasonable to expect precision from any model, but we believe that one can call a model “wrong” when its inaccuracy is predictable, as the SACMC’s inaccuracy clearly has been. We knew both that they were wrong and why, and, frustratingly, we had shared all of that with them, perhaps with decreasing politeness, and we had been consistently ignored.
Further patience disappointed
Upon establishing that there would be a review presentation on 29 June, we decided to let a few days pass, hoping that a modicum of conscience or embarrassment would lead to a retraction of the SACMC model, not just for the Western Cape, but for the country as a whole. This was naĂŻve.
We were astonished to be greeted with a presentation by Prof. Andrew Boulle of the Western Cape Government, since removed from the internet, in which the massive departure of the actual epidemic process from what the SACMC model had predicted was rationalised in a way that strained the very bounds of reason and language.
This led us to put out an angry Twitter volley on 29 June, subsequently reported in these pages. Some took umbrage at our tone. We suspect they will have ceased to, if they’ve read this far. In that volley, we made a further prediction—that the SACMC model would blow through its lowest 90% confidence interval today, just a month after release. That milestone was in fact achieved yesterday, as the Western Cape’s death rate has continued to slow from a 7-day rolling average of 52 to 42 today.
For the period June 22 to June 29 (only 2 weeks after the report was published), the SACMC daily deaths projected for the Western Cape were 100 per day. Panda’s predicted range was 44 to 58. The actual number was 51. The SACMC’s most optimistic projection was for 53 deaths per day. Actual numbers are now running below that optimistic projection and yet only last night, government was bandying about the figure of 50,000 deaths by the end of the year.
To show you how divorced from reality this projection is, we’ve added a line to ours that would get to government’s 50 000 deaths by the end of the year (grey line):
A particularly alarming aspect of the 29 June presentation was an attempt to make up for the modelling excess by attributing “excess mortality” from the SAMRC and UCT Centre for Actuarial Researh’s 7 July “Report on weekly deaths in South Africa”. The authors of this report included one Tom Moultrie, brother of Dr Harry, and Prof. Rob Dorrington, with whom we had unsuccessfully attempted to engage in April. Finding insufficient evidence for unreported Covid deaths, they set up the argument that a lower expected death number ought to be entertained. Excess deaths would then be high enough to justify Harry’s modelling excess. At this juncture, bear in mind that Panda started this journey by pointing out that lockdown would cause loss of life.
Deloitte had also produced a model which, as at 20 May, projected around 40,000 deaths. Deloitte declined to disclose their model to Panda on the basis that this was technical information that if disclosed would cause harm to the commercial or financial interests of Deloitte. Deloitte did not say what those interests were or why their financial interests trump the rights of South Africans to ensure that the process of influencing government policies that impact their lives and livelihoods in material ways is transparent. One can only surmise how disclosing their workings would cause them harm.
The consequences of faulty science
The SACMC’s modellers had been confronted with facts that falsified their hypotheses. Faced with a choice between changing their hypotheses and abandoning reason, they chose to abandon reason.
We know that the government relies on these faulty models to inform strategy about lockdown, and they continue to claim to follow science that is, instead, fault-ridden science. Covid-19 coronavirus projections have real world implications. They are required for planning ahead for the number of hospital and ICU beds, as well as to assess the potential benefits of mitigation strategies against their human costs. They divert resources from the many other pressing healthcare problems that South Africa contends with (such as HIV and TB treatment). Numerous peer reviewed publications attest to the efficacy of early detection for these and other communicable diseases. The outcome of this diversion must be obvious to all and will play out in the months and years to come. In response to the faulty projections, the Western Cape government had been planning on a second hospital of hope setup, but were alert enough to observe that reality was not matching the projections and dodged that bullet.
At this stage, while the government can still claim to rely on these exaggerated models, the modellers remain both causative and complicit, whether intentionally or otherwise, in the initiation and maintenance of an ineffective and wholly destructive lockdown that will, as Panda’s first report clearly demonstrated, result in more loss of life than it purports to prevent.
One can only hope that they will do the respectable thing, which would be to retract their models and replace them with ones that reflect the real world, or resign if they are unwilling or unable to do so. An apology to their fellow South Africans for their destructive roles in this farce would also be advised.
The potential for further damage to ensue if they do not select one of these paths is clear, and it centres on the Gauteng province.
In a report published on 10 July, the Gauteng Provincial Command Council referred to an SACMC prediction that 8,000 to 11,000 ICU beds and 25,000 to 30,000 hospital beds would be required. The report also referred to a Gauteng model which predicted that 55,000 to 71,000 hospital beds would be required.
Let us compare this to the Western Cape scenario, at an equivalent stage. On 19 May, the SACMC predicted that the Western Cape would need 3,000 to 6,000 ICU and 7,500 to 15,000 hospital beds. The actual numbers were 330 ICU beds and 1,900 hospital beds used. For Gauteng, Panda predicts that 600 to 1,200 ICU and 4,000 to 8,000 hospital beds will be needed. This is just a tenth of the requirement in the SACMC 10 July Gauteng report.
The people Panda stands up for
Those on the front lines are to be saluted for the dedication they have shown in getting the Western Cape through its peak. The burden now shifts to their peers in other provinces. Our message to them would be to emulate their Western Cape colleagues over the next few weeks as the remaining provinces peak and subside. They should take solace from the fact that the models that predict many months of mayhem are flat-out wrong. We wish them courage and safe passage.
Our next article will show how these fine people are not protected by lockdown. Again, we turn to the international evidence to report that lockdown neither flattens curves nor buys time. We will explain how the fear it creates kills people, and how the economic destruction it creates affects not the “1%”, but people in the entrepreneurial and worker classes, whose very labour pays for the upkeep of the modellers, bureaucrats, corporate fat cats and ivory tower academics who are colluding to rob them of their livelihoods. They have been sorely underrepresented in this saga. We stand for them.
Footnote: While we have focused on the medical model there is apparently an economic model used in the deliberations of the NCCC. Can you just imagine the construct of this model if it still regards lockdowns as being beneficial to South Africa?
References
Hudson, N., Krige, S., McGorian, I., 2020. Reviewing the models used to justify lockdown. BizNews
https://www.biznews.com/inside-covid-19/2020/07/10/panda-covid-19-lockdown
Hudson, N, Castleden, P, McGorian, I, 2020. The epidemiologists who missed the boat. https://www.pandata.org.za/uncategorized/the-epidemiologists-who-missed-the-boat/
PANDA, 2020. Quantifying Years of Lost Life in South Africa Due to COVID-19. Report update: Monday, 11 May 2020.
https://www.pandata.org.za/wp-content/uploads/2020/06/PANDA-Research-Report-Quantifying-Years-of-Lost-Life-PDF_.pdf
SACMC, 2020. Estimating cases for COVID-19 in South Africa Short term Projections: Report Update: 12 June 2020.
https://www.nicd.ac.za/wp-content/uploads/2020/06/SACovidModellingReport_ShortTermProjections_12062020_Final2.pdf
SACMC, 2020. Estimating cases for COVID-19 in South Africa: Report Update: 19 May 2020.
https://www.nicd.ac.za/wp-content/uploads/2020/05/SACMC_19052020_slides-for-MoH-media-briefing.pdf
Streicher, P.E. 2020. The rate of increase in Covid-19 cases in the Western Cape is starting to slow down. Blog article.
https://mymaskprotectsyou.com/2020/05/31/the-rate-of-increase-in-covid-19-cases-in-the-western-cape-is-starting-to-slow-down-v-4-31-5/
Streicher, P.E. 2020. The Western Cape is likely to reach peak mortality rate between June 23 and July 7.
https://mymaskprotectsyou.com/2020/06/12/the-western-cape-is-likely-to-reach-peak-mortality-rate-between-june-23-and-july-7/
- PANDA (Pandemics ~ Data & Analysis) is a multidisciplinary initiative seeking to inform policy choice in the face of Covid-19. Panda’s technical team brings to bear knowledge from the fields of actuarial mathematics, economics and medicine and is continually recruiting.
Also read: Mailbox: PANDA giving actuaries and economists a bad name
