Professor Graham Barr of the University of Cape Town (UCT) offers expert insight into how science can influence the decision-making of politicians in creating regulations around the lockdown. Barr has an extensive history in statistics, having studied econometrics, financial modelling, sports statistics; games of shance and problem gambling and its treatment. He holds the following degrees: BSc(Hons) (1976), MSc (1977) (Static and Dynamic modelling of the South African economy) and PhD (1981) (A Contribution to Adaptive Robust Estimation), which were all completed at UCT. Barr tackles the science behind the lockdown regulations and how politicians have relied heavily on data forecasting, which in some instances has been detrimental to the economy of those countries under a hard lockdown – particularly in SA’s case. According to Barr, societies have a role to play in contributing to decision making around lockdowns and they should not merely accept a top-down approach from government. – Bernice Maune.
Science has good tools available for balancing the different interests of role players, when determining lockdown regulations
By Professor GDI Barr*
The Covid-19 pandemic, arguably the most shocking event of the 21st century, has exposed the fragility of humanity and, in particular, made society face up to the difficult choices relating to the suffering and death caused by Covid-19, and the suffering and death caused by the various lockdown responses. From the onset of the pandemic, South Africa has been the subject of a harsh lockdown, arguably the harshest in the world. Certainly in the realm of restrictions on alcohol and the smoking of tobacco, it is the strictest in the world.
It is assumed that the lockdown restrictions endured in South Africa were put in place on the basis of the best scientific advice available. In this note I will examine the role of Science in approaching such a devastating event and how it can sensibly respond in the most appropriate way.
The notion of Science, and the awe in which society holds the scientific method, is mainly rooted in what science has established regarding deterministic phenomena. That is, Science can, for example, aid us in determining the outcomes of a range of phenomena governed by “laws of physics” and which can be precisely described by mathematical equations. So when somebody throws a stone with known characteristics off a cliff of known height, science can accurately predict when the stone will hit the sand at the bottom of the cliff.
Read also: Lockdown: How SA flattened its economy instead of the curve – Johannes Wessels
Science is less helpful in fields which are driven by human behaviour like Economics and the Study of Disease Transmission (like Covid-19). Hence scientific models applied to processes like the Covid-19 pandemic cannot tell us how the pandemic will unfold in the future. In fact the forecasting record of models applied to processes which are dominated by human interaction including economies, stockmarkets and election-outcomes have a history of being rather poor.
Scientists have devoted considerable time to building a “model” of the infection path associated with Covid-19, as well as the impact that interventions might have on the virus.
The limitations of scientific models need to be understood, and the degree of uncertainty in model predictions made explicit
Any scientific “model” is effectively a mathematical structure which attempts to capture the known behaviour of the system being modelled. Of course, in the Covid-19 case this mathematical structure could only be hypothesized using information about previous viral pandemics as so little was known about the true character of Covid-19 and how Covid-19 would pan out. Using some hypothesized mathematical structure of the virus, statisticians would observe past data on general viral spread, as well as observations on any associated factors which might affect the path of future Covid-19 spread in order to “estimate” the assumed Covid-19 model. The observations constitute the model data, and the estimation process fits the mathematical model to the known data. This means that on the basis of past behaviour, the factors which are assumed to impact on the virus, and its effect on society, are given appropriate model weights according to what has happened in the past, so that the hypothesized model best fits the data.
Such a statistical model of a virus’s spread effectively try and mimic the path of a virus’s infectious past behaviour and then, if shown to be reliable, are used to project the path of the virus into the future under various assumptions. Of course, in the case of Covid-19, very little “past data” was available at all, and so statisticians had to guess at which model might be most appropriate, and were, anyway, unable to test the reliability of the model against any real data. Hence, on the basis of an uncertain model, and with effectively no historical data, statisticians found it challenging to use the model to accurately forecast the progress of the virus into at all.
Read also: ‘Scientists, please fight to end SA lockdown, it’s a COMPLETE failure’: Save the nation
To capture and reflect the uncertainty of statistical models in general, any model forecast must be put in a probabilistic framework. Rather than release a simple point forecast, statisticians always give estimates of the error inherent in that forecast. This is done by including “Confidence Intervals” for the forecast which reflect the uncertainty of the forecast produced by the model. In the case of Covid-19 forecasts, and anything related to the future of those factors associated with CoVid-19, confidence intervals must be established to reflect the extreme level of certainty related to these forecasts. One therefore cannot take the forecasts from any statistical modelling exercise seriously which do not include confidence intervals around the forecasts.
When potential deaths are involved, it is natural for politicians to base decisions upon the “worst case” scenarios predicted by models, without cognisance of the implications of these decisions
However, to further compound the problems with trying to use statistical models to forecast Covid-19 outcomes, there will be an Intrinsic Bias towards Conservatism, because the forecasts centre on the loss of lives.
To explain; a statistical modelling team who is presenting to a group of politicians (say the National Command Council on Covid-19 Lockdown (NCCC)) might present the modelling forecasts on Covid-19 in the following way. They might say:
“We have an uncertain model structure and very little past data. The best we can do is to come up with a 95% confidence interval for the forecasts that says there is a 95% probability that the actual death rate could range from a low of 10 thousand deaths and a high of 1m Deaths.”
The NCCC may well Respond:
Are you telling us that the #Deaths could be 1m. That sounds alarming. Could it be even higher?
The Statisticians could Respond
It could be as high as 1m and if we calculate the 99% confidence interval, the estimate of the #Deaths could be as high as 2m.
The NCCC could further Respond
We cannot let this happen. We must do whatever it takes to arrest this pandemic. We should lockdown the economy immediately.
A rational objective person (with no experience with confidence intervals) struggles with a probabilistic framework. The focus on the extreme (high) death rate makes things seem WORSE than they probably will be. When LIVES are involved, the Response will often be:
How can we worry about the economy when we might see 2m people dying?
So there is always an underlying BIAS from politicians towards a stronger LOCKDOWN than is probably required.
Notwithstanding this issue, attempts have been made to apply Covid-19 models in several countries around the world, and, as might be expected, none have been much good at forecasting the future path of Covid-19 infections. To cloud the issues further, there have been very stark country differences in the Covid-19 rates of infection and resulting illness and death, implying a very different character and spread of the virus in different countries. So, for example, in the case of Spain, Italy, UK and USA infections spread quickly and peaked with high death rates and high total deaths. In the case of most third-world countries, as well as in Eastern Europe and Japan, Covid-19 induced illness rates took off much more slowly and the virus appears to be more benign.
Moreover, my experience in analysing and assessing the underlying official released data in the South African context suggests that all the data needs to be carefully checked and cross-verified where possible. In the time of Covid-19 particularly, data support services are unstaffed and this has led to “lumpy” data in several provinces. For example, the Eastern Cape recorded 400 deaths from Covid-19 on the day of the 22nd July.
This was clearly an error, being several times larger than any of the close-in-time daily-deaths data points, but was never corrected. It can only be assumed that a large number of previously unrecorded death cases in the Eastern Cape were lumped together, and reported as an “on that day” data point. So the statistical modelling of Covid-19 is fraught with problems and resulted in a range of decisions which do not reflect the uncertainty of the models used to forecasts the outcomes.
Broader stakeholder engagement is required to improve the quality of decision making. The current debate, particularly in the case of alcohol and tobacco is dominated by ideological positions which are not backed up by properly interrogated positions.
One of the most controversial restrictions imposed on the SA populace has been on the sale of alcohol and cigarettes. In the case of alcohol, the ban has been supported by the contention that during the Covid-19 pandemic, one needs to take pressure off hospitals, which deal with a large number of alcohol related problems. As discussed above, scientific modelling and forecasting of Covid-related phenomena are challenging at best, and even with very good data needs to be put in a structured probabilistic framework. Any release of Covid-19 related forecasts as point forecasts have tended to imply that they are scientific truths whereas in fact quite the reverse is true.
Read also: PANDA: More evidence lockdown is complete failure in fighting Covid-19 – the SA picture
In South Africa, in order to address the effects of the banning of alcohol, the Medical Research Council (MRC) has modelled and forecast the alcohol-induced trauma cases at hospitals under various levels of restricted alcohol sales. However, their published forecasts have no obvious regard to the levels of uncertainty implicit in their own forecasts. Current data on alcohol-induced trauma is not available, and thus they were forced to use data that was several years old.
The statistical model estimated does, therefore, not reflect the current situation, but rather the situation that occurred years ago. Moreover, there appears to be no existing hard record of the extent to which the trauma cases in hospitals were alcohol-induced, so this adds further uncertainty to any forecasts produced by the model. Hence the attempted quantification by the MRC of the reduced trauma cases in hospitals should be treated with caution.
Even though it is clear that banning alcohol will in all probability reduce the total number of trauma cases, it is almost impossible to reliably quantify the effect through a statistical model, as the data is several years old and was not reliably recorded.
At the very best, one can make a probabilistic statement of the type “a 95% confidence interval for the reduction in trauma cases after a ban on alcohol is …”. In contrast, the forecasts and estimates produced by the MRC have been released with no confidence intervals and hence do not reflect the uncertainty underlying them.
Moreover, the fact that the banning alcohol will have an effect on reducing the number of trauma causes is considered without regard to the ramifications of such a ban on society. In a parallel way, the existing curfew will also reduce the number of (car accident) trauma cases too, but there will be knock-on consequences for the economy. Reducing the speed limit drastically would also reduce the number of car accidents, and have similar negative knock-on effects for the economy.
So, underlying any form of ban is clearly the issue that restricting people’s behaviour will change the harm they do to themselves and others, but this must be weighed against the effect on the economy and livelihoods. In addition restrictions also constitute an imposition on people’s right to pursue any lifestyle they want, clearly as long as it doesn’t harm others.
In the case of the alcohol and tobacco ban, there is a further effect that has consequences for the fiscal’s ability to operate. Tax receipts for the sale of alcohol and tobacco are extremely high for the fiscus. So the bans directly affect the provision of government services like education and social grants, and, indeed, health services themselves.
In fact, the alcohol case is one case of several that poses many questions about the lockdown regulations, but for which the government appears to have simply taken an extremely restrictive position on the basis of very little well-constructed statistical evidence.
A scientifically based Cost-Benefit Analysis known as Multi-Criteria Decision Analysis provides an appropriate platform to tackle such problems.
In reality, good scientific tools DO already exist for approaching such problems and have a widely recognised record of success. In fact, a branch of Statistical Analysis known as Operation Research provides an appropriate approach to problems of the type considered above.
It provides a methodological structure for approaching the problem of intervening in a system which is threatened by a pandemic (say), in a structured way that takes into account the various outcomes and weighs them up appropriately. Effectively a “Cost Benefit” type approach, the methodology is known in the scientific community as Multi-Criteria Decision Analysis (MCDA). In essence, the approach considers the Benefits of a set of regulations (lives saved) against the Costs to society of any imposed raft of lockdown regulations.
The estimated benefits and costs are then assessed by the different groups and stakeholders in society in a way which encapsulates their conflicting goals. So any given set of lockdown regulations could be more objectively assessed in terms of its NET effect on society, by the affected parties within society.
A properly constructed Cost Benefit Analysis would highlight the fact that the poor have been particularly affected by inappropriate decision making.
The approach is detailed, but provides a best-practice framework to systematically approach the problem we have been presented with. For example, when assessing the impact of Covid-19, one can immediately see that different income groups will be affected in different ways. While a large section of the middle-class is relatively unaffected by lockdown, especially those employed by government, the poor are much more drastically affected.
A number of employed-poor hold down jobs in the informal sector, or are employed on an ad hoc basis doing work to which they have to travel and are paid at an hourly rate for work done. Restrictions on the time and mode of travel have impeded such people’s livelihoods. These issues need to be carefully considered, and debated. Older people will be affected by the virus in a different way to younger people; so although it is clear that the elderly need to be carefully protected, the young and more resistant might be treated differently. This has obvious implications for schooling. An alcohol ban is having a destructive effect on restaurants and wine farms and the people they employ, and the resulting contraction in these tourism-centred activities will have a long-term effect on jobs and the economy.
Ideally, one would consult with all societal stakeholders, and attempt to establish weightings to the factors and groupings whose lives and livelihoods are affected. In practice, this would be contentious and time-consuming. The determination of societal weights could involve considerable negotiation and dispute. However, by laying out the structure we can present a negotiation starting point which establishes that in the process of trying to reach an overall, all-things-considered best policy, policy-makers must account for the fact that different policies affect different categories of citizens differently. Consequently, policy-makers must decide how much comparative weight to accord to the interests of these different groups.
Scientists have been called upon to lead the way in predicting the impact of the Covid, but the decisions made on the basis of their scientific advice have not properly considered the overall effects of the lockdown. In particular, those who stood to be affected were never consulted.
Hence, although the government clearly needed to move quickly to protect the vulnerable, they also needed to consider a sensible structure like MCDA within which role-players (stake holders) in society could properly discuss and determine what was appropriate for society. From the outset, we needed to amass as much relevant information from as many sources as possible including the expertise of stakeholders with diverse interests. Such interests would include, for example, those of people who, on ideological, religious, or even aesthetic grounds, favour the complete prohibition of tobacco and alcohol.
Society should not simply accept a set of seemingly arbitrary regulations imposed by politicians on the basis of questionable scientific recommendation, without consultation with the very people these regulations are going to affect.
- Professor GDI Barr is an Emeritus Professor in the department of Statistical Sciences at UCT.
