The Covid-19 pandemic turned a few things upside down. Society was turned on its head when normal, natural social interaction became unlawful with the imposition of unprecedented restrictions such as social distancing. Evidence-based medicine (EBM), “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients”, was cast aside, the most notorious example of which was global public health authorities’ denunciation of ivermectin, the safety of which was already compellingly evident prior to Covid-19. And, of course, the emergency use authorisation and rollout of experimental Covid-19 vaccines. In this article by several acclaimed healthcare scientists, including Professor Colleen Aldous, the principle of the Totality of Evidence-Based Medicine (T-EBM) wheel was applied in contrast to the quality/certainty hierarchy associated with the pyramid model. The authors concluded; “Never in the history of using EBM theory has it been that only a specific type of study design is considered adequate for informing practice.” By using the T-EBM wheel, the authors show that we will benefit dealing with the current pandemic as well as those in the future – a fundamental benefit that, during the Covid-19 pandemic, was sidelined as the science supporting the efficacy of ivermectin in treating COVID-19, which is visible throughout the entire T-EBM wheel, was ignored. – Nadya Swart
The Totality of EBM still points to the efficacy of ivermectin in COVID-19: The Totality of Evidence–Based Medicine Wheel
By Colleen Aldous1, Philip Oldfield2, Jerome Dancis3, Barry Dancis4
This week we published the Totality of Evidence-Based Medicine (T-EBM) wheel on TrialSite News for the first time. By totality, we mean both the totality of the types of reports and the totality of reports for a given type. The T-EBM wheel also shows outcomes and visualises the relative numbers of reports and outcomes without the quality/certainty hierarchy associated with the pyramid model. We have applied the principle of the T-EBM wheel to the repurposing of ivermectin and show that the totality of evidence for the efficacy of ivermectin in treating COVID-19 still weighs towards an advantage in using the drug, enough to risk a TYPE 1 error decision.
Current evidence for ivermectin
Healthcare practitioners regularly prescribe drugs outside the scope of their initial approved usage, a practice referred to as “off-label use”. Before making therapeutic decisions, practitioners should consult existing literature to establish the safety and possible efficacy of a drug for a disease.
Prior to the COVID-19 pandemic, there was already compelling evidence of the safety of ivermectin. In the first year of the pandemic, searching the WHOs VigiAccess database should have been a first call to check on existing safety data. RCTs and observational studies from the period of development of the drug by Merck would have shown that ivermectin had an established and exemplary safety profile for human use. The dosages proposed for the treatment of COVID-19 were adequately covered based upon the data from the Phase 1 ascending dose study, and subsequently published in a peer-reviewed publication in 2002. And, in humans, it was the standard dose for scabies and only half the dose for crusted scabies.
By May 2020, a few months into the pandemic, a laboratory study from Australia demonstrated that ivermectin effectively inhibited replication of the SARS-CoV-2 virus by preventing it from entering the nucleus. The authors reported that by 48 hours, there was a >5000-fold decrease of viral RNA, indicating that ivermectin treatment resulted in the effective loss of essentially all viral material by 48 hours. Although this in vitro study used a very high ivermectin dosage, it showed ivermectin killed the SARS-COV-2 virus, and physicians began using the standard human scabies dosage regimen with some success. However, since then, opinion as well as research on ivermectin efficacy in treating COVID-19 has diverged to a point where distasteful and disparaging public mudslinging has occurred.
We secured the services of an impartial actuary to carry out a literature search for published reports on the use of ivermectin in the treatment of COVID-19 from 2020. As proof of concept, we used only published reports found in the PubMed database, an internationally accepted medical literature source. We acknowledge that this search would not capture all the available literature. The in-depth literature trawl showed there is a large amount of evidence on ivermectin use in COVID-19 from disparate study methods. To get a sense of the scope of this evidence, we have developed the T-EBM wheel for ivermectin, which shows all report types, the relative numbers of publications for each type, and the proportions of positive, negative, and inconclusive outcomes. In addition, it shows outcomes, presents a visual representation of the relative numbers of reports and outcomes. In short, the wheel presents a super-set of the data in the pyramid while omitting the quality/certainty hierarchy associated with the pyramid model.
Figure 1. T-EBM wheel for the efficacy of ivermectin-based COVID-19 treatments.
The inner ring of the wheel catalogues reports by four types, namely: mechanisms of action, clinical studies, plus the reviews and structured analyses based on primary research. With the exception of mechanisms of action, the middle ring disaggregates the types of reports into levels of certainty similar to those in the pyramid model. The outer ring disaggregates each middle ring section by the outcomes of its reports. The legend in the lower right-hand corner of the figure shows the color-coding of the outcomes, the names of the outcomes and the sum of the percentages for each outcome. The size of each section of a ring is proportional to the number of reports in the literature for that section. The size of each section in this figure is for illustrative purposes only.
On 10 October 2022, the actuary retrieved 377 published and publicly available peer-reviewed reports on ivermectin-based COVID-19 treatments from Pubmed. After excluding reports not meeting inclusion criteria, namely ivermectin not being used to treat COVID-19, we analysed the balance of these reports for various attributes. When categorising ivermectin reports for our T-EBM wheel (Figure 1), we used only the authors’ overall statement about efficacy from the conclusion section of a report, not the findings sections, even in those few cases when the conclusions were contradicted or not supported by the findings. In this T-EBM wheel for ivermectin efficacy, 62,6% are inconclusive. However, on closer examination of each paper, most show signals for efficacy which in turn suggest both further research for confirmation as well as potential protocols for the front-line practitioner.
Collectively, all reports of all study types showed that efficacy slightly outnumbered those that showed a lack of efficacy. Even if the positive results were outnumbered, that is irrelevant. What needs to be done is to look at the positive results for a proof of concept, and both the positive and inconclusive results with positive signals for designing protocols. Negative results suggest protocols to avoid. In all cases, the known safety profile should limit what is tried. Looking at the wheel for ivermectin, what is clear is that there are signals from all types of research for the efficacy of ivermectin when treating COVID-19.
On close inspection of just the 27 RCTs included in the T-EBM wheel, it appears that trials showing non-efficacy outnumber those that show efficacy or that are inconclusive. On further scrutiny of their protocols, it is evident that most of the negative studies used inadequate dosages with inappropriate dosage regimens, for example, only giving a single dose or not stipulating that ivermectin should be taken with meals. The RCTs that rendered positive results mostly used higher dosages. When examining all RCTs, regardless of outcome, it appears that there has yet to be a single adequately designed trial with adequate participant numbers, participant oversight and monitoring, dosage and dosage regimens, and objectively obtainable clinical outcome measures. This does not mean these studies should all be rejected, but they do need to be scrutinised for what is valid and useful. Regardless of outcomes, what is clear from every one of these studies is that there have been no serious safety concerns when using ivermectin.
The T-EBM wheel shows that most reviews, whether narrative or systematic, alert researchers and decision makers to at least the considerable number of positive signals when using ivermectin in therapy. The science supporting the efficacy of ivermectin in treating COVID-19 is visible throughout the entire T-EBM wheel. Using the wheel will benefit dealing with the current pandemic as well as those in the future.
Governmental and medical authorities, by considering as valid evidence only those RCTs claiming negative results for repurposed drugs, have led to groupthink. They have dismissed the use of ivermectin as explained above and their message has been amplified by the media. A large body of evidence supporting the efficacy of ivermectin-based COVID-19 treatments has been ignored for decision making. This includes primary research from case studies, laboratory investigations, extensive population-based epidemiological studies, and the higher forms of secondary research of meta-analyses and systematic reviews. In addition, RCTs take time to design and execute. Performing a full RCT including safety assessment for drugs with a known safety profile is unnecessary, costly, and time-consuming, and may unnecessarily delay the introduction of effective treatments. Faster results could be had with protocols based on observational studies. Successful protocols could then be applied to treat patients during the pandemic, at the same time as higher certainty testing with RCTs are being conducted. Even quicker signals of efficacy might be gotten from in vitro, in vivo, and in silico testing to direct attention to possible effective therapeutics.
When selecting medications for repurposing, a retrospective review of existing data for similar diseases would be an appropriate first step in evaluating therapeutic potential in a pandemic. This could lead at worst to trying a drug protocol that later possibly proves to be ineffective, that is, a Type 1 error. Such an error, however, should not pose a safety risk. Case series and cohort studies could quickly follow. Dosing to effect for such drugs is especially useful and available to the front-line practitioner, but not safe with a new drug. When examining the repurposing of therapeutics, all efficacy evidence should be scrutinised and, the Totality of EBM wheel provides a useful visual summary as a starting point.
Globally, the most widely publicised RCTs that have been used to determine governmental policies regarding ivermectin (IVM) treatment of COVID-19 have all exhibited design flaws and/or issues of scientific integrity. The TOGETHER TRIAL was promoted across global media as the coup de gras for, hoping to end the debate on its efficacy. However, the study outcome measures were rejected by both the FDA and NIH as inadequate. The statistical design for mortality outcome was brought into question, and repeated requests for the underlying data have been consistently denied, a serious violation of scientific integrity.
The Lopez-Medina study in Cali, Colombia, was also internationally lauded as a trial to show ivermectin’s inefficacy. This study had several glaring design and execution flaws, including lapses in segregation and blinding of ivermectin and control doses and the appearance of ivermectin-characteristic adverse events in controls. The Vallejos et al. study from Argentina provided only two doses of ivermectin 24 hours apart, resulting in underdosing. The I-Tech Trial found a 28-day in-hospital death of three participants in the ivermectin arm vs 10 in the control arm (RR, 0.31; 95% CI, 0.09-1.11; P = .09) in high-risk patients with mild to moderate COVID-19. Yet they found they could not support the use of ivermectin for patients with COVID-19, not even mentioning the signal for mortality benefit.
On the other hand, the Ahmed et al. and Mahmud et al. studies, using successful five-day ivermectin regimens and published in recognised international academic journals with H-indices over 60, found little traction in international media. Both studies showed efficacy for ivermectin regimens. Other smaller RCTs that also demonstrated positive outcomes for ivermectin-based regimens have been ignored.
More importantly, while a single observational study may be of lower certainty than a single RCT, a collection of observational studies is often of similar certainty to a collection of RCT’s.
Governmental policies along with the media promoting the findings of the flawed studies and ignoring the positive findings of other studies have resulted in the suppression of successful ivermectin-based treatments.
The EBM pyramid has been wrongly interpreted
Evidence-based medicine (EBM) has a long history, but only since the 1990s has it been at the forefront of medical education spurred by publications from David Sacket and his colleague Gordon Guyatt. They pointed out the need for medical decision-making to be based on peer-reviewed science, eliminating the ‘hunch’ or the thoughtless following of trends in treatment decisions. In 2006, Glover et al. developed what has become the traditional hierarchy of evidence-based medicine pyramid model, disaggregating the totality of evidence into levels of ‘quality’ of research. While the pyramid model is commonly used to show levels of quality of evidence, in fact, it only shows levels of certainty of evidence.
This flawed usage has led to a serious misapplication of EBM over the last decades such that observational studies and background information/expert opinions, which constitute the levels lower than that of randomised control trials (RCTs) in the pyramid, are regularly considered inadequate for clinical decision making and policy recommendations. As a result, much useful data and important treatment signals are incorrectly dismissed. This is a significant problem during a pandemic, especially when considering known drugs for repurposing.
Evidence-based medicine’s central tenet is to consider all available evidence when making treatment recommendations and decisions. As the concept of EBM has been developed over the past 30 years, so has there been a change in its interpretation and its application. “Certainty of evidence” has come to mean “quality of evidence”, leading to a shift away from considering the totality of evidence to a focus exclusively on randomised controlled trials (RCTs). Embodied in the pyramid model since 2006, certainty of evidence, namely the power to make inferences from the data for a particular study design, is ordered by levels of increasing certainty the higher its position. The replacement of the word “certainty” with the word “quality” in the pyramid model, however, has led to the incorrect conclusion that reports in the RCT level and above are acceptable while all observational studies and background information/expert opinions which are found only in the lower levels are improperly dismissed. This also ignores the fact that high-quality case-controlled studies would be found in a lower level of the pyramid model, and poor-quality double-blinded RCTs (DBRCTs) higher up.
Never in the history of using EBM theory has it been that only a specific type of study design is considered adequate for informing practice. The incorrect, but albeit common practice of replacing certainty with quality has shifted focus away from considering the totality of evidence to focusing exclusively on RCTs and their meta-studies for much clinical and governmental decision-making.
This dismissal of observational studies has sometimes led to delayed and/or erroneous medical decision-making on therapeutics, especially for repurposed drugs with an already established safety profile. The insistence on RCTs for such drugs, without considering the totality of evidence, has resulted in overlooking effective treatments.
By replacing certainty with quality, medical training and media discourse have ingrained in physicians and the public that case series are of poor quality and DBRCTs are of high quality. This has been a serious misuse of EBM during the COVID-19 pandemic. For example, DBRCTs ‘proving’ a drug does not work, and was not well designed or executed, have been accepted as good quality because of their position in the pyramid model, but not because of their actual scientific quality. Conversely, well-run case series rendering more reliable results with more certainty than poorly executed DBRCTs have been ignored.
However, applying the principles of EBM correctly by looking at all types of research across several study designs may provide an increased level of certainty that would be ignored using the pyramid model.
Figure 2. Correspondence of traditional evidence-based medicine Pyramid with the two inner rings of the T-EBM wheel. Arrows show correspondences between levels of the pyramid and sections of the rings.
Comparing the traditional EBM pyramid with the newly proposed T-EBM wheel in Figure 2, it is seen that the T-EBM wheel shows outcomes and visualises the relative numbers of reports and outcomes without the quality/certainty hierarchy associated with the pyramid model. The wheel includes reports found in all levels of the traditional evidence-based medicine pyramid, as well as additional types of reports.
A proposed way forward to evaluate the use of repurposed drugs in an emergency
In a pandemic situation we must rely upon front-line healthcare practitioners for their training and experience. It is they who, after all, are face-to-face with patients. Once assured that the treatment is safe and likely to be effective, consenting patients may be treated, and should be followed up closely. The main objective is to cause no harm to the patient whilst anticipating positive outcomes and recording any and all negative ones. Healthcare practitioners should monitor patients closely, modifying treatments accordingly, and publically share their results through the publication of case studies and case series which, in turn, should lead to higher certainty studies.
Governmental and medical authorities monitoring such results, should recommend, as appropriate, experimental usage on a broader scale while knowing that the drug might not work but at least it should be safe. At an institutional level, a T-EBM wheel for efficacy should be drawn up at the beginning of a pandemic using all the existing data. The wheel organises the data visually to show the relative frequency of reports about efficacy outcomes and, analysing those reports, yields information about the other important aspects of therapeutic decision making, namely: dosage, safety, and mechanisms of action.
The wheel should be updated in real-time as further evidence becomes available. This will provide comprehensive summaries of the totality of evidence developing around the efficacy of the drug. Concurrently, Phase 1 ascending dose studies should be initiated, followed by Phase 2/3 pivotal studies in the clinical setting.
Summary and conclusions
In a pandemic situation, double-blinded randomised controlled clinical trials on already approved therapeutics are neither cost-effective, nor required since the safety profile and mechanisms of action are already known. Therefore, allowing health providers to prescribe such medications off-label should be the first response, and should be encouraged. As observational studies become available, they can provide early signals of efficacy and inform properly designed RCTs. These studies should not be ignored. The permutation of protocols using such drugs is vastly larger than that for RCTs and gives front-line practitioners far greater flexibility to test and customise as they search for increasingly positive results. Such flexibility, especially when dosing to effect, is not available in an RCT.
To prevent this from happening for repurposing of drugs in future pandemics, our new approach of considering the totality of evidence should be initiated early on. EBMs true spirit for safe and effective treatment for all is acknowledging the best available evidence, which is not exclusive to, but inclusive of RCTs. All primary research must be recognised, including that from observational and laboratory studies. RCTs will require time and funding, both resources often in short supply. Observational studies together with prior safety studies for known drugs can provide rigorous and reliable evidence in a much shorter time. There could be sufficient scientific evidence to make competent Type 1 error decisions about testing the scaling of protocols.
In addition, pharmaceutical companies have extensive financial disincentives to perform RCTs on out-of-patent drugs. Those same pharmaceutical companies may have large financial incentives to perform designed-to-fail RCTs on drugs for repurposing when they have a new drug or vaccine under development.
Whilst we used only PubMed to make the point here, the application of the T-EBM wheel in a real world situation should take in a wider trawl of reliable publications. One can start with Pubmed and expand from there.
Acknowledgements
- The services of an actuary for the collection of the data was sponsored by Mr Sean Flanagan.
- Several graphics were provided on request by Mr Joel Smalley, including the sunburst graphic on which the T-EBM wheel was further developed. He is an independent quantitative data analyst from the UK who has been doing comprehensive empirical analyses of a variety of COVID-related datasets.
References
- College of Health Sciences, University of KwaZulu-Natal, Durban 4041, South Africa
- Scientific and Regulatory Consultant, Rigaud, QC J0P 1P0, Canada
- Department of Mathematics, University of Maryland, College Park, MD 20742, USA
- Independent Researcher, Silver Spring, MD 20903, USA
Read Also:
- SAHPRA’s ivermectin decision shows wilful blindness to totality of evidence – Professor Aldous
- ‘Principle over profit’ – The Noakes Foundation COO Jayne Bullen on overcoming dietary disease, the greatest obstacle to human health
- Persecution of doctors under Covid has pushed medical science back into the Dark Ages
