Study that suggests BCG vaccine may protect against Covid-19 is flawed – TB experts

Who has had the BCG vaccine?

• National BCG vaccination policies vary across the world, with universal BCG vaccination at birth still occurring in high TB-burden countries such as India and Ethiopia, while countries with lower overall TB incidence have largely discontinued universal BCG vaccination in favour of targeted vaccination of specific TB risk groups (such as children from families immigrating from high TB burden countries), says the McGill International TB Centre.
• The BCG World Atlas, created by the McGill International TB Centre, provides data on BCG policies and practices around the world. Furthermore, their information suggests that BCG vaccination was implemented universally in South Africa from 1973, though others may have received it.

The study by Miller et al – a quick snapshot

The McGill International TB Centre students looked at the following study: Correlation between universal BCG vaccination policy and reduced morbidity and mortality for COVID-19: an epidemiological study. Authors: Aaron Miller, Mac Josh Reandelar, Kimberly Fasciglione, Violeta Roumenova, Yan Li, Gonzalo H Otazu. 

They highlight the following:

Data of Covid-19 cases and death per country were obtained on March 21, 2020, from https://google.org/crisisresponse/covid19-map. The ecological analysis found an association between Covid-19 death rates and universal BCG vaccination policies among middle and high income countries. The authors also observed an association between Covid-19 cases per capita and BCG vaccination policies, with countries with a universal BCG vaccination policy having lower cases per capita and death rates. Based on the analysis, the authors state “The correlation between the beginning of universal BCG vaccination and the protection against Covid-19 suggests that BCG might confer long-lasting protection against the current strain of coronavirus”.

Six reasons to question the findings

1. Flawed study design

“Ecological studies often conflate population level exposures, such as country-wide policies, with exposures at the individual level, such as the effect of the BCG vaccine on a human body. What may hold at the aggregate level, will not necessarily be true when the heterogeneity of individuals is considered,” say the McGill International TB Centre students.

“In this study, low quality evidence observed at the population level is used to make sweeping inferences about BCG’s effectiveness on an individual level. Ecological analysis techniques simply cannot directly inform etiology of exposure/disease relationships; rather, they serve us well as strictly hypothesis-generating queries and should not be stretched beyond this purpose,” they caution.

2. Rapidly changing data

Drawing premature conclusions from rapidly changing data in a pandemic is problematic, say the academics.

“Since the analysis (COVID-19 data were downloaded on March 21, 2020), the pandemic has escalated in many countries, including low and middle income countries (LMICs) where BCG is given at birth. For example, COVID-19 cases in India have increased from 195 on March 21  to 1071 on March 31. In South Africa, cases have increased from 205 on March 21 to 1326 on March 31. Therefore, the paper’s findings could look very different as countries progress in their epidemic timelines (and expand diagnostic capacity),” they note.

3. Coronavirus cases are underestimated worldwide

There are shortages of diagnostic tests and the number of tests performed per capita varies wildly by country, point out PhD students Faust, Huddart, MacLean and Svadzian.

“Underestimation of the number of cases in lower income countries could entirely explain the authors’ observed results. India, for example, has one of the lowest Covid-19 testing rates in the world. Additionally, countries have each applied their own algorithm to determine who gets tested. Thus, the number and composition of known cases is heavily influenced by testing strategy,” they argue.

Although Miller and associates acknowledge the concerns surrounding current estimates of Covid-19 cases and also use Covid-19 deaths as a measure of the severity of each countries’ outbreak, “unfortunately, even current Covid-19 death data are unreliable in some parts of the world”.

“In wealthy countries, it is true that most deaths due to Covid-19 will occur in a hospital after having received a diagnosis of Covid-19. However, we know that in LMICs, even before this current pandemic, most deaths occurred at home without medical assistance. People dying at home in LMICs are unlikely to receive a confirmed diagnosis of Covid-19 and may not have their death accounted for in government records. Coronavirus deaths may also be missed in high-income countries that have been overwhelmed by the virus, like Italy,” say the McGill International TB Centre students.

4. Correlation is not causation

A critical flaw in the study is reading too much into the pattern in the statistics. The “perceived relationship between an exposure and an outcome does not mean one causes the other – a concept often expressed through the well-known adage ‘correlation does not imply causation’.”

“Specifically, in the case of the relationship between universal BCG vaccination and the apparent lower Covid-19 case numbers and Covid-19-associated deaths, a multitude of underlying factors are at play. The crucial take-away here is that if these factors could be appropriately accounted for, we would very likely not see the association between BCG vaccination and protection against Covid-19 suggested by this paper,” say the Canadian university students.

“The concerns about data quality now become a potential source of confounding in the relationship between BCG vaccination and lower Covid-19 case numbers. It is likely that under-funded health systems in (LMICs) have low Covid-19 testing capacity, and at the same time, are more likely to have universal BCG vaccination policies still in place, given that TB burden is highest in LMICs. Therefore, the study actually does not demonstrate that populations in countries with universal BCG vaccination policies are somehow more protected against Covid-19, because lower case numbers and deaths may be more attributable to low testing capacity than to any supposed protectiveness of BCG.”

“Statements in the paper that imply causality, such as the statement that ‘BCG vaccination also reduced the number of reported Covid-19 cases in a country’ are dangerously misleading, particularly in a context where reported case numbers are in large part a function of testing capacity and testing strategy, rather than actual relative burden,” they emphasise.

5. Countries are too different to make them comparable

It is difficult to draw causal conclusions across a broad range of countries with varying underlying demographic characteristics, such as differences in the age distribution of the populations of interest – and this is particularly true of diseases for which severity differs by age, say the McGill International TB Centre students.

“As shown in a recent study from Wuhan, China, older individuals are at higher risk for severe Covid-19, so differences in the age distribution of populations may significantly influence death rates observed across countries (for example, the WHO regions of South-East Asia and Africa have a median age of 27.0 and 18.7 years, respectively, whilst the population in the European region is much older (median 38.6 years), which may in part contribute to the higher death rates observed in Europe).”

6. No clear evidence TB vaccine protects against anything other than TB

Lastly, the students turn to biological plausibility and warn that some of the “common sense” appeal of this paper’s claim is based on a bit of a false equivalence.

“The authors twice mention that BCG protects broadly against respiratory diseases. The idea that a vaccine for an infectious disease of the lungs could also protect against other infectious diseases of the lungs seems somewhat consistent and thus palatable. This hits a snag though, when we look at the type of TB against which BCG most effectively protects,” they point out.

“Due to an unfortunate natural experiment wherein there was a global shortage of the BCG vaccine from 2013 to 2015, retrospective data show there was a surge in cases of TB meningitis, that is, TB of the central nervous system. The tidy idea of a lung disease vaccine protecting against another lung disease, i.e. COVID-19, is not quite so straight-forward, since BCG seems to offer most significant protection against TB meningitis.

“Additionally, it is well-established in the TB field that the BCG vaccine only confers protection against TB in young children, and that protection almost completely wanes by age 12. Recent studies have even examined whether applying a booster vaccination to older adolescents or adults would help extend this anti-TB protection. The important take-away is that BCG’s protective effect fades over time. So it is not clear how a vaccine that offers protection early in an individual’s life would protect individuals against Covid-19 once they are of an advanced age,” the McGill International TB Centre students note.

But don’t give up hope; BCG trials may save the world

The students warn that giving people hope that BCG will protect against Covid-19 could lead to a false sense of security and lead to inaction, ultimately causing harm. Nevertheless, they say that ” with further research, it may emerge that the BCG vaccine does confer protection against Covid-19″.

“An Australian trial is set to start imminently, aiming to investigate whether BCG vaccination protects against Covid-19 or reduces severity of Covid-19 in Australian healthcare workers. If, in a well-performed trial, we do in fact see  a benefit in re-purposing this vaccine, the evidence would be much more convincing,” they add.