text:
Conflicts of interest, financial or otherwise, clearly influence some people's actions some of the time. When most experts in an area of medicine share a similar conflict, it can be hard to know if you are getting an unbiased opinion, guideline or recommendation. Disclosure of conflicts is a good idea, but once disclosed you are still left wondering how an expert's opinion might have been shaped and altered by the conflict.
Conflicts, though, can be something on which to blame all manner of disliked opinions, errors in judgment, or just plain mistakes, out of all proportion to their actual role in a given circumstance.
Recently in BMJ, Jeanne Lenzer published a piece titled "Why we can't trust clinical guidelines". Ms. Lenzer, a journalist, has written about conflicts of interest frequently, and she and Shannon Brownlee published a list of industry-independent experts that journalists could contact for unconflicted opinions about new research and products. For my own disclosure, I am on this list and think it's a great idea, though journalists have only very rarely contacted me through this listing.
In her current piece, though, I think other problems are being mistaken for conflicts of interest, at least if we are talking the financial kind. Ms. Lenzer writes about guidelines recommending that steroids be given to patients with spinal cord injuries. (She uses this as a backdrop to a more current concern about recommendations around alteplase for stroke, but I'm not going to address the alteplase issue in this post). She makes it clear that she feels this is an example where conflicts of interest have led to actively harmful guidelines:
One expert estimated that more patients had been killed by the treatment in the past decade than died in the 9/11 World Trade Center attacks.
She sets this up by pointing out that the lead author on a study published in the NEJM supporting the use of steroids in spinal cord injury, "...declared he was an occasional consultant to steroid manufacturers Pharmacia and Upjohn...." He then went on to be the "sole reviewer" on a Cochrane Collaboration review of the issue.
So, in this view of the world, an author has occasionally consulted to Pharmacia and Upjohn; those companies make steroids, and so the opinion about whether to administer steroids to patients with spinal cord injuries is so tainted that more people are killed by this harmful therapy in a decade than died in the WTC attacks on 9/11.
Is this really believable? Could large pharmaceutical manufacturers and their occasional consultants actually care enough about an inexpensive, uncommon therapy to desire to promote it in the face of harm?
I've been following the overall issue of steroids in spinal cord injury because it's an intersection between evidence and values that causes real problems for anyone who wants to write a guideline. There are published trials suggesting neurologic benefit when steroids are administered early after spinal cord trauma. However, to find these benefits, the data must be interpreted with a forgiving eye. An early subgroup must be split off from the trial as a whole, and the definition of early does not seem to have been made a priori. As such, the benefit may just be a statistical blip -- in general, post hoc subgroup analyses should be viewed with suspicion. Additionally, the actual neurologic benefit seen, even in the "early" subgroups, were quite small and perhaps of marginal clinical importance.
Weighing against the use of steroids are their obvious harms, including an increased risk of infection in trauma patients who are already setups for developing such infections.
The problem, though, is that many people are willing to accept a significant increased risk of death for a small chance of benefit when the issue is spinal cord injury. I watched a group of physicians trained in medical evidence who had participated in a detailed review of the data around steroids for spinal cord injury vote overwhelmingly that they would want steroids if they suffered such trauma. (They were far more willing to take the steroids themselves than recommend it for their patients based on the data, but that should always give us pause about where real-world values and preferences lie.)
So, are the issues with guidelines around steroids for spinal cord injury really do to conflicts of interest?
I do think the author of the NEJM trial is conflicted, but not because he has some association with some large companies that happen to make steroids as one of their products. Instead, he has the universal intellectual conflict of having studied something, wanting to find an important result in his research, and wanting what he found to be true. I would not view him as the best person to judge whether the subgroup analyses in the trial were reasonable, or whether the neurologic benefits in the trial were clinically meaningful.
But the big problem with guidelines around steroids has relatively little to do with these author conflicts. Instead, the problem for guideline developers is that we are very uncertain that steroids help but have some evidence that they might. Though being administered steroids likely increases the chance of dying, this is apparently a chance that many people (including a roomful of fully-informed physicians) appears ready to take when the alternative is so grim (at least to those who are not currently quadriplegic; it's worth pointing out that another problem with values and preferences around this issue is that people living as quadriplegics typically view their quality of life as much better than the quality of life that healthy people anticipate for themselves were they to become quadriplegic.)
So to come back to where I started, conflicts of interest can be a huge problem for the development of trustworthy guidelines. But there are many other things that can be problems, and for a particular guideline these may loom much larger than conflicts. It's important to diagnose the problem correctly in each case, so that we don't view every flawed (or potentially flawed) guideline as due to a single cause.
We have known for many years now that patients having an acute coronary syndrome (ACS) do better if they are immediately treated with aspirin, and that they do even better if in addition to that single antiplatelet agent they are also treated with clopidogrel.
In 2009, the PLATO trial was published in the NEJM. It was a randomized trial in more than 18,000 people having an ACS, comparing the benefits of adding either clopidogrel or the newer agent ticagrelor to aspirin. Ticagrelor appeared to cause some increased serious bleeding, but overall it reduced a composite endpoint of vascular death, MI, and stroke (hazard ratio 0.84). AstraZeneca, the manufacturer of ticagrelor, has to have been pleased.
Many subgroups were examined in PLATO, and in a few of them there was a suggestion of differences in effects across the subgroups; that is the benefit seen with ticagrelor was not as great or non-existent in three subgroups: those who weighed less, those not taking statins at enrollment, and those living in North America.
It's common when a trial examines multiple subgroups to find spurious positive results due to chance alone, given the multiple statistical tests being performed. For this reason, there has to be a good reason to bother to pay attention to a subgroup, and overly trusting subgroup analyses is one of the most common errors in thinking about what a study has shown. Given this, and the positive result in PLATO, it would have been typical for the authors to make short shrift of the subgroup analysis and write the results off to chance.
However, one of those subgroups seemed to worry the investigators. They wrote in the discussion section:
The difference in results between patients enrolled in North America and those enrolled elsewhere raises the questions of whether geographic differences between populations of patients or practice patterns influenced the effects of the randomized treatments, although no apparent explanations have been found.
They were likely concerned because the point estimate of effect in North America was that clopidogrel was substantially superior to ticagrelor, the opposite result from that seen elsewhere. However, ticagrelor was apparently approved by the US FDA without concern that the benefits seen in the trial as a whole would not apply to patients in North America, and that might well have been where things were left.
Instead, apparently working closely with authors/investigators at Duke, AstraZeneca carried out an extensive investigation to figure out whether the subgroup effect was real and, if so, what might have caused it. The article detailing this investigation was published in Circulation in 2011.
If you consider the problem, it is unlikely that genetic differences between North American patients and European patients could explain the differences. That suggests that if the finding was not caused by chance it was either due to some way in which patients were managed differently in North America (most likely) or some way in which patients enrolled in North America were consistently different (less likely), such as being sicker or older. Once you start looking for such possibilities, though, you are faced with the high likelihood of even more spurious results caused by chance. All the exploratory analyses are now post-hoc subgroup analyses -- those that you had not thought of before the primary analysis -- and so highly susceptible to findings due to the play of chance.
As they began looking into the issues, the investigators first found that the real difference wasn't between North America and the rest of the world, but really between the US and the rest of the world, so they refocused on the results when patients in Canada were included with the rest of the world. Then they checked in with major study sites to see if the trial was conducted differently in different locations -- it wasn't. Additional checks found that US sites seemed to have gotten the correct study drugs.
They looked at the play of chance, and concluded that results like those seen in North America would have been expected 32% of the time, and results like those seen in the US would have been expected in at least one country 10% of the time, so chance was certainly a possible explanation. Still, because US practices can be so different from those in many other countries, the investigators must have remained concerned.
The investigators then looked into many baseline and management variables to see if they could explain results and started focusing on aspirin dose as their investigation unfolded. At the US sites, patients were typically treated with more than 300 mg of aspirin (likely 325 mg) daily, while in other locations patients typically received less than 100 mg daily. In statistical analyses, including aspirin dose appeared to explain 80-100% of the regional variation in outcomes.
Additional analyses that stratified by aspirin dose found a dose response effect that seemed similar across regions. Patients in the US who received low dose aspirin saw similar benefits with ticagrelor to patients in the rest of the world, while patients in the rest of the world who received higher dose aspirin saw similar benefits with clopidogrel to patients in the US; patients in the rest of the world who received 100 to 300 mg of aspirin had similar outcomes with either clopidogrel or ticagrelor:
Overall, though, patients receiving low dose aspirin plus ticagrelor had the best outcomes:
Still, this is a post-hoc subgroup analysis, and the results may have all been due to chance. The authors of the paper acknowledge this repeatedly while still suggesting that patients treated with ticagrelor only be given low dose aspirin. This seems like a completely appropriate conclusion. We can't know for sure whether aspirin or chance or some other difference in management in the US explains the results, and it is unlikely that another RCT will be conducted to answer the question, but the analysis from this paper really informs appropriate clinical decision making.
I am not routinely a fan of Pharma in general or of AstraZeneca in particular. One of my favorite posts from the old medical humor website Q Fever does a wonderful job poking fun at the way in which AstraZeneca promoted Nexium over omeprazole (Prilosec) once the latter drug went off patent. The trials used to claim superiority for Nexium are great examples of how Pharma can manipulate results by manipulating study design and interpretation.
But as far as I can tell, AstraZeneca deserves real credit for pursuing the subgroup analyses from the PLATO trial. I can't picture how they could have seen a marketing upside to this, since they had already found an overall benefit with ticagrelor. If the issue were never pursued, the regional variation in effect would likely have been ascribed to chance. Furthermore, it was possible that the in-depth investigation they participated in would have turned something up showing that ticagrelor was no better than clopidogrel or was even inferior.
Instead of resting on the positive result in PLATO, AstraZeneca seems to have moved forward on this and supported the authors/investigators from Duke who clearly also wanted to pursue the issue. In so doing, they found what is likely the right explanation for what really happened in the trial, which allows patients throughout the world who receive ticagrelor to be treated with the appropriate dose of aspirin to maximize the chances of a favorable outcome.
The study's authors and investigators deserve our thanks for pursuing this issue with hard work and logical thinking, but it seems likely that thanks are due to AstraZeneca and its employees as well.
I've written previously about how the plausibility of results might affect the interpretation of new evidence from a trial. In particular, I argued that results that cannot fit in with a web of prior evidence must be viewed with skepticism.
While being overly credulous of results that are unlikely to be true is a common problem in medicine, a less common but more severe problem is how unwilling doctors tend to be to believe a new result that upsets past dogma even when that dogma was built on low quality evidence.
Historical disbelief in startling results
I discussed in the past the unreasonable attacks that I watched academic physicians make on trials showing that H. pylori was causing ulcers. The classic example of this, though, in my medical lifetime is probably what occurred with hormone replacement therapy (HRT). Large, well-analyzed observational studies consistently found that in post-menopausal women, the use of HRT was associated with a reduced risk of coronary disease. Based on this, HRT became a central facet of recommended primary prevention in such women, despite the clear increased risk of breast cancer with HRT. The data showing about a 30% reduction in coronary events and reduced fractures with HRT were felt to more than offset the increased breast cancer risk.
Through the 1990s, more and more post-menopausal women were treated with estrogen. In 1998, the HERS trial was published in JAMA. It was a randomized trial of HRT, but in women who already had coronary disease. In HERS, HRT showed no benefit on coronary events and produced an increase in thromboembolic events and a nonsignificant trend toward higher overall mortality.
HERS was a startling result. If HRT was good for preventing coronary disease, you would have expected that it would be beneficial in preventing the progression of coronary disease. We knew that all the evidence we had for HRT was from observational studies that were at risk of confounding caused by healthier women choosing to take HRT. And yet the nearly universal reaction to HERS was to try to either figure out what was wrong with it or, more commonly , to explain how the observed effects of HRT for secondary prevention did not apply at all to HRT when used for primary prevention. These arguments tended to lose track of how much the opposite was predicted before the trial was conducted. Yet I watched as nearly all the academic primary care internists I knew continued to prescribe and recommend HRT.
When HERS was published, I stopped recommending HRT and started suggesting to women taking it that they might want to stop. I worked with one other physician who did the same and many who argued that HERS simply did not apply to their primary prevention patients. Then, in 2002, the results of the WHI trial were published after the trial was stopped for harm. This was a primary prevention trial of HRT and showed an increased risk of coronary disease. Word spread quickly, and HRT for primary prevention became a thing of the past, though for years after people still tried to explain away the results.
The present day
HRT and disbelief in an infectious cause of ulcer disease are really ancient history at this point, but the tendency of doctors to try to explain away randomized trial results that they should find startling and disturbing remains.
In May of 2011, the NEJM published a trial of giving IV fluids (either saline or albumin) to children in resource limited settings in Africa who appeared to have septic shock. This trial included 3100 very ill children, and I have to think that the IRB approving this trial must have worried about the ethics of having a control arm where such sick children with impaired perfusion were not given fluid boluses, since I would guess that very few clinicians would have thought there was equipoise between the likelihood of clinical benefit and harm to the intervention. The trial must have been intended as a prod to get local governments and international agencies to expend some resources to provide this minimal level of care for septic children.
Stunningly, the trial showed around a 45% increase in mortality in the children given fluid boluses. The researchers stopped the trial for harm and wrote a paper clear on the effects seen in resource limited settings, while pointing out that this raises some concerns for other settings. An accompanying editorial allowed for much greater concern about the generalizability of these results:
It seems clear that the results of this trial indicate that bolus-fluid resuscitation with either crystalloids or colloids in patients with compensated shock who do not have a clinical fluid deficit must be practiced with much greater caution than is now the case and with increased vigilance.
The editorialist goes on to argue for the urgent need for additional study around the questions raised by this trial.
The evidence we have for fluid resuscitation in children with compensated septic shock in resource rich settings is entirely observational. Mortality rates are fairly low (in the 1% to 2% range) and so an individual clinician would be unlikely to notice harm from a therapy that increased mortality by 45%.
And yet...
I have heard and read multiple experts try to explain away the generalizability of the trial. A common aspect of the arguments seems to be a belief that the burden of proof lies with those who want to generalize from these results. They seem to think that because giving fluids is standard practice that this means the practice is supported by high quality evidence, but this just is not the case.
I expect that nearly everyone who I have heard or read attack this trial would have never thought the intervention could cause a 45% increase in mortality in Africa. That is because they thought the benefits they attributed to fluid resuscitation were generalizable. As such, they should both have been completely shocked by this result, and have been extremely worried that the results generalize back from resource limited settings to resource rich settings. They should have examined the basic premises and evidence around fluid resuscitation, in just the way the editorialist did.
Instead, predictably, people tried to figure out how to defeat the trial and prove it could not possibly apply to resource rich settings. The main arguments made seem to be either that many of these children had malaria, or that many of them had severe anemia, and that in both settings there is a suspicion that fluids can be harmful.
Letters in response to the article make both these arguments. This is odd, since the data presented in the paper show them clearly to be unreasonable attacks. The increase in mortality with fluid boluses in children with malaria was 59% and in children without malaria was 43%, and these results were not significantly different from each other. The increase in mortality with fluid boluses in children with severe anemia was 71% and in children without severe anemia was 31% and again these were not significantly different from each other (and note that even if the real increase in mortality in children without severe anemia is lower than in those with severe anemia and the trial did not have the power to show this, the point estimate is still a one-third increase in mortality). One of the letter writers points out that the increase in mortality in those without severe anemia was not statistically significant, but this is a completely unfair argument. Once an increase in mortality is seen in the group as a whole, you can always find subgroups that by virtue of having fewer patients will have a similar mortality rate that does not pass a given level of statistical significance.
All who were startled by the results of this trial should be much less convinced of the benefits of fluid resuscitation in compensated sepsis and should be wary of its use. They should be pushing for trials in resource rich settings, and while awaiting those results should keep the possibility that administering fluids may increase mortality in the front of their minds however they decide to treat septic shock. Reasonable people may still choose to cautiously administer such fluids for compensated shock in resource rich settings, but they should be anxious every time they do so.
Personal notes
The authors of this trial wrote in the paper of how surprised they were by the results and at one point say:
Although fluid boluses adversely affected the outcome, important survival gains, across all groups, may have resulted from training and implementation of triage, basic life-support measures, and regular observation.
When I was reading the paper, it wasn't until I read those words that I realized the impact of this trial not just on the children but on the researchers. There were around 70 excess deaths in the fluid bolus arms of the trial; 70 children who likely died because of the intervention. We can imagine the affect on parents and families, but this must also be a researcher's worst nightmare. This was an incredibly difficult study to carry out, and those who did so must have thought they were heading down a path that would ultimately save innumerable children's lives in resource limited settings. Instead, perversely, the intervention killed around 70 children and those same researchers are likely having trouble sleeping and comforting themselves with the knowledge that the training from the study may have helped save some children overall. They deserve our honor for a trial well designed and executed and our sympathy for what must be the emotional toll.
For me, I remember a man in his 30s who was admitted to my service in septic shock near the end of my internship. We gave him innumerable fluid boluses over the course of 12 hours as he inexorably moved towards death. When I think of my hardest moments as a doctor, telling his 13-year-old daughter in the middle of those 12 hours that her father was not doing alright has pretty much been at the top ever since that day. Now, after reading this trial, I can add to that the worry that those repeated fluid boluses didn't merely swell him with edema past the point of recognizability, but may actually have contributed to his death.
I've written a bit previously in this blog about the evidence around lipid-lowering therapies, and in particular about the lack of evidence for combining other lipid-lowering therapies, including ezetimibe, with statins.
Tomorrow, the FDA will apparently be taking up a request to approve Vytorin (simvastatin/ezetimibe) for the treatment of patients with chronic kidney disease (CKD). As a result of this request for approval, we also have access to the FDA review of data from the SHARP trial.
SHARP was published in Lancet in June of 2011. SHARP was a trial comparing simvastatin/ezetimibe with placebo in nearly 9300 patients with chronic kidney disease. By way of background, it is important to know that two prior trials of statins in patients on dialysis, 4D and AURORA, had failed to find a statistically significant reduction in events with statin therapy. The non-significant point estimates were about an 8% reduction in events in 4D and a 4% reduction in AURORA.
That makes end-stage renal disease (ESRD; essentially, patients requiring chronic dialysis) an unusual condition when it comes to statin therapy. People with ESRD are at very high risk for cardiovascular events, but get little benefit from statins, unlike almost every other high risk group.
Into this comes simvastatin/ezetimibe, a drug without a clear use. Ezetimibe has never been shown to add clinical benefit to treatment with simvastatin, but based on surrogate outcomes the combination drug became widely prescribed until other surrogate outcomes made people concerned that ezetimibe might really be worthless or worse.
Perhaps that made CKD seem like fertile ground for exploring benefits with simvastatin/ezetimibe. The SHARP trial when it was published was promoted as showing unique benefits in CKD including, for the first time, in patients with ESRD.
The abstract says things like:
After weighting for subgroup-specific reductions in LDL cholesterol, there was no good evidence that the proportional effects on major atherosclerotic events differed from the summary rate ratio in any subgroup examined, and, in particular, they were similar in patients on dialysis and those who were not.
Reduction of LDL cholesterol with simvastatin 20 mg plus ezetimibe 10 mg daily safely reduced the incidence of major atherosclerotic events in a wide range of patients with advanced chronic kidney disease.
So, based on SHARP, the claim is being made that simvastatin/ezetimibe, unlike statin monotherapies before it, works in patients on dialysis because it reduced events in all patients with CKD, and the effect in the subgroup of patients on dialysis was not statistically significantly different from the effect in those not on dialysis.
Is this a fair argument?
In general, we look askance at claiming that effects in subgroups differ from the overall effect in a randomized trial. While people are always asking "does drug X really work in women? the elderly? blacks?", in reality most interventions work similarly across a broad patient population. So why object to such a standard, and seemingly reasonable argument for simvastatin plus ezetimibe?
There are two interrelated main problems:
The first is that we are not starting from a position of believing that the effect of a statin-based intervention should be the same in patients with and without ESRD. Based on what happened in prior trials of statins that included less severe CKD, where statins seemed to have their usual benefits, and the 4D and AURORA trials where they had much smaller benefits if any, we already knew that the ESRD subgroup was likely important and special.
The second is that the actual results are in no way reassuring that the subgroups were similar, even if they were not statistically significantly different. Among all patients in SHARP, simvastatin/ezetimibe reduced events by 17%. However, in the 6247 patients not on dialysis it reduced events by 22%, and in those on dialysis it reduced events by a non-significant 10%. Those numbers look rather far apart given our a priori concerns, and even if not statistically significantly different from each other, 10% seems reminiscent of 4D and AURORA.
Still, 10% is a larger reduction than the 8% in 4D or the 4% in AURORA, right? Well, only if you look at all the dialysis patients in SHARP. The 4D and AURORA trials only enrolled patients on hemodialysis, while SHARP enrolled patients both on peritoneal dialysis and hemodialysis. In the small number of peritoneal dialysis patients in SHARP there was a non-significant 30% reduction in events, which is what brought down that average number for all dialysis patients. If you looked just at the 2500 hemodialysis patients, the population studied in the earlier trials, the reduction in events was just a non-significant 5%, similar to AURORA and not even as good as 4D.
Given, this, the argument is completely unpersuasive. Failing to show a statistical difference between effects in the dialysis subgroup and the non-dialysis subgroup is just a way of hiding the reality that simvastatin/ezetimibe did not produce clinically significant or statistically significant benefits in patients on hemodialysis.
So what are we left with? I think we can confidently say after SHARP that simvastatin/ezetimibe reduces cardiovascular events in patients with CKD who do not have ESRD, and that this reduction is right in the ballpark of what we would have expected with simvastatin alone. Further, we can note that yet again a statin-based treatment showed little or no benefit in patients on hemodialysis and that there is absolutely nothing in the results from SHARP to suggest that ezetimibe provides some added clinical benefit to simvastatin alone in either patients with earlier stage CKD or patients with ESRD.
As was the case before SHARP, we have no convincing evidence that any lipid-lowering therapy added to a statin improves clinical outcomes in any patient population.
I won a bottle of wine this week. At least in theory. I haven't posted to this blog in a long while, but the opportunity to brag about this has overcome inertia.
A few years ago I was at a cookout with some friends and colleagues, and got into a discussion with one of them, an emergency physician and sepsis expert, about activated protein C.
Many modifiers of the immune response to sepsis have been studied in an attempt to lower the large mortality rate of severe sepsis, but almost none have shown any benefit. Then, in 2001, a study of activated protein C in severe sepsis was published in the NEJM, and showed a substantial reduction in mortality.
I was suspicious from the beginning, and commented that there had been enough trials in sepsis that we were bound to get a positive result by chance alone. My skepticism, not surprisingly, had little effect on the rapid rise of activated protein C and its manufacturer Lilly. It became a central part of the management of sepsis.
In the original trial there was some suggestion that activated protein C worked better in patients at high risk for death than in those at lower risk. This was somewhat counter-intuitive, though certainly possible. To answer the question and expand the indications for the drug, a study was conducted to look at patients at lower risk of death. This study, published again in the NEJM in 2005, found no suggestion of benefit in patients with severe sepsis at lower risk of death and was stopped early for futility. Additionally, patients treated with activated protein C were at increased risk for serious bleeding. My skepticism about the drug grew.
Then, in 2007, a study was published in the Lancet, showing no benefit of activated protein C in children with severe sepsis.This study, too, was stopped early for futility and an increase in CNS bleeding.
And so, in the setting of these trials, I found myself debating the merits of activated protein C with a sepsis expert. He took what was, up until last week, the pretty standard position of critical care physicians that what this sequence of studies showed was that activated protein C was only effective in adults with severe sepsis who were at high risk for mortality; it was ineffective or harmful in adults at lower risk for mortality from sepsis and for all children with sepsis.
In contrast, I took the position that the first trial had been a statistical blip and that, in reality, activated protein C was useless or harmful in treating sepsis. At the time of our debate, a very large international trial was getting under way to answer this question, and so we wagered a bottle of wine on the results.
These two ways of looking at trial results come up in all sorts of contexts. One option, when trials seem to disagree with each other, is to try to find the thread that resolves the conflict: the specific patient population that benefited in one trial, or the specific dose that made all the difference, or the specific type of location that explains how a drug could show benefits in one trial and be useless in another. As a web of trials is put together, people will often argue that the "evidence-based" approach is to look to the trials as trees and find, for instance, those medium height oaks that are sure to benefit from a particular therapy that is of no benefit for shorter or taller oaks, or for maples of any height.
Occasionally this approach does make sense, but more often, looking at the forest, it becomes clear that a single positive trial result amongst a network of negative trials (or vice-versa) may be better explained by chance or errors in the conduct of the study than by looking to the individual trees.
Researchers get things wrong, by being unlucky, by making errors, by being foolish, by fabricating data, and for any number of other reasons. When a study result feels unbelievable for good reason, it often should not be believed. In an earlier post I argued that XMRV would not turn out to be the cause of chronic fatigue syndrome. It now appears that lab error was responsible for the early results suggesting such a connection, but I certainly had no training or special knowledge to have predicted what might have gone wrong; I had only the clinical knowledge that I thought finding any single pathogenic cause of CFS was extremely unlikely. When neutrinos were found moving faster than the speed of light, scientists were appropriately even more skeptical. (The medical equivalent of this, to my mind, are the occasional studies suggesting benefits from homeopathy.)
With activated protein C, the best way to explain the data seemed to be to assume that the one positive trial was incorrect while awaiting the results of the large international trial. Those results have not been published in final form, but the trial was stopped early for futility this week and activated protein C was withdrawn from the market. My friend the sepsis expert owes me a bottle of wine. I'd certainly trade in that bottle of wine, though, if I could figure out a way to get expert clinicians and clinical epidemiologists to step back from apparently conflicting clinical trials and think about what the results as a whole are suggesting, rather than trying to believe that being fair to the evidence involves taking each individual trial result as true. This latter approach does not sufficiently consider the probability that when trials seem to conflict, some of them are likely to simply be wrong.
I was speaking with a patient recently who was explaining why he thought that colon cancer screening with fecal occult blood testing (FOBT) was equivalent to colonoscopy. He wished to avoid colonoscopy for all the usual reasons, but also believed logically that it was not a superior test.
I won't get into the actual evidence behind the two methods of screening; that is not an issue for this post. Instead, I want to discuss his argument. He assumed that colonoscopy had a sensitivity (probability of the test being positive in someone with a significant colon lesion) of 98%, and this figure is in the ballpark of believable numbers. Let's assume that the sensitivity of FOBT is around 35% (also a reasonable ballpark figure). At that rate, he said, since colonoscopy is performed once every 10 years, and FOBT is performed yearly, after 10 years the performance of the two tests is similar with FOBT having a cumulative sensitivity of 99%.
There are two important problems with this argument. The first, which I will just mention in passing, is also the more obvious. If a test has to be repeated for 10 years to detect a cancer, that cancer may grow and become incurable during the testing period. Had it been found at year 1, it might have had a better chance of being cured.
The second problem is the focus of this post, and to examine it, we need to understand the calculation the patient was performing to get to a cumulative sensitivity of 99%. This is the same type of calculation that is often presented when thinking about the probability of an abnormal lab test due to chance alone in a battery of lab tests. The argument goes as follows:
The actual calculation of the likelihood of an abnormal test result typically confuses medical students and early residents until they've heard it presented repeatedly. A common assumption is that if there is a 1 in 20 chance of an abnormal result on each test, then if 20 tests are run there will definitely be an abnormal result. This is not the correct calculation. Under the usual assumptions that people make in thinking about this, the calculation would be that the probability of all the tests being normal is the probability of a single test being normal raised to the power of the number of tests.
Thus, for 20 tests it would be 0.95^20, which is 0.36. The probability that at least one such test will be abnormal due to "chance" is 1-0.36 or 0.64. Or, about 2/3 of normal patients would be expected to have at least one abnormal test on a battery of 20 tests under these assumptions.
So the patient utilizing FOBT for colon cancer screening was saying that, with a sensitivity of about 35%, he could expect a false negative rate of 0.65 per test but that 0.65^10 (for the ten years of testing) yielded a miss rate of 1%. This cumulative sensitivity would then be similar to that of a single colonoscopy, so why should he get the invasive procedure?
And so we come to the second problem with the patient's argument: each round of testing is not an independent event.
The calculations I described above for cumulative probabilities make the assumption that the individual events are independent from each other. That is, the result of one test has no influence on the others.
If you flip a fair penny three times and get three heads, there is still a 1 in 2 chance that it will come up heads on the fourth flip. But if you perform FOBT once for colon cancer and it is negative it might be in the setting of your particular precancerous lesion that doesn't tend to bleed. If so, it's less likely to be bleeding on subsequent FOBT than an "average" lesion and so the cumulative sensitivity cannot assume independence of events. This is not just a theoretical issue: based on some research, repeated testing is thought to actually have a cumulative sensitivity of around 85%, not 99%.
So the patient was miscalculating in his decision about how to be screened for colon cancer. I actually briefly discussed this with him during the appointment, but since the real issue was that he did not want a colonoscopy he was singularly uninfluenced by the math.
This plays out in other areas as well, though:
What about that standard example above regarding batteries of lab tests that all medical students and residents are taught? The tests in the battery, too, are clearly not independent events. Normal and abnormal tests tend to cluster and so it is likely that the probability of the 20th test being abnormal in a healthy patient is affected by whether the prior 19 tests included any abnormal results.
To contrast with the assumption of probabilities in the face of independent events, here we are talking about "conditional" probabilities where we want to know the likelihood of an event given some other set of events. However, in the real world we have very few data about these situations. If, for instance, I wanted to know how likely, due to random variation, a patient with an abnormal serum sodium and chloride is to have a high serum potassium, it is extremely unlikely that I could get a high quality answer without doing my own primary research.
This problem of not knowing conditional probabilities when faced with non-independent events has an important effect on how diagnostic strategies might be misinterpreted if clinicians really started utilizing a test parameter that is a favorite in the EBM community but has not really permeated the clinical world. I'll address this in a future post.
I'm a little late reading the June 19th Lancet, but was intrigued to find letters in response to the meta-analysis by Sattar et al. looking at whether statin therapy increases the risk of diabetes.
I had previously written about this well-performed meta-analysis, and also written about some unfair ways that people use to try to attack randomized trials, and these letters provide an interesting (at least to me) intersection between these posts.
Letters in academic scientific journals are sociologically revealing. There's typically a polite veneer on even the most vicious attacks. Letters written to European medical journals have a somewhat different feel from those to American medical journals, and letters to the Lancet often seem to have a sneering tone that would be unusual to find in the NEJM or JAMA.
One letter about the meta-analysis objects that the results cease to be statistically significant when diabetes diagnosed only by physician report are excluded, and secondly that the results involved a post-hoc analysis of the data, with the warning that we might fall victim to the logical fallacy, "Post hoc ergo propter hoc".
Are these fair objections?
Diagnosing diabetes by physician report rather than blood glucose measurement is likely to lead to misclassification: some patients will be classified as having diabetes who don't, and some who have diabetes will be missed. In an RCT, though, misclassification like this will almost certainly be random as well, leading to random misclassification bias. Bias of this sort is toward the null hypothesis (no difference between the groups), as you can convince yourself of if you imagine that the classification is perfectly random such that there is no relation between the classification and diabetes. Under such perfect misclassification, the two groups would have equal numbers of patients classified as having diabetes and there would be no difference between treatment and control. In a meta-analysis that found higher rates of diabetes in patients receiving statins, misclassification bias can be expected to have somewhat reduced the true effect, not to have created an effect out of thin air.
The second objection might be called the "post-hoc-ergo-propter-hoc-fallacy fallacy". The actual fallacy is, of course, a way of saying that just because B follows A, you should not conclude that A caused B. This question of causality is central to epidemiologic research and one of the primary reasons for performing randomized trials, which have particular strengths when arguing for causality. The fallacy has nothing to do with performing post hoc analyses of trials. (To be fair, it's possible the letter writer understood this and was being humorous when writing of this fallacy.) The main problem with a post hoc analysis of a randomized trial is that it often involves multiple comparisons/data dredging, where statistical blips are likely to confuse the issue of what is a true effect. As discussed in my earlier post, a prime issue preceding this meta-analysis was whether JUPITER had found just such a random blip or detected a real problem. The meta-analysis' reason for being performed was primarily to answer this question, and in such a setting there is nothing at all concerning about going back to previously conducted RCTs and performing post hoc analyses looking for diabetes effects. No data dredging was involved, and the analysis should not be looked at askance simply for being post hoc. Revealingly, the meta-analysis found an increased risk of diabetes even when data from JUPITER were excluded.
A second letter complained that the analysis would have been better had it been carried out using hazard ratios rather than odds ratios. While this would likely be true, such an analysis was not possible given the information available to the authors, and it is hard to imagine why an OR analysis would have shown statins to be causing diabetes if it were not true. The same letter also re-raised the possibility that statins appeared to be causing people to have more diabetes by keeping them alive longer to develop diabetes. However, the authors had already addressed this in their meta-analysis and reiterated in their response to the letters that differences in survival were much too small to produce such an effect.
A third letter mis-states the definition of a type I error on its way to arguing that the meta-analysis should have used 99% confidence intervals (p-value cutoff of 0.01) for some reason that was not made terribly clear, but seemed related to concerns that a very large meta-analysis would be more likely to detect a spurious result. It is true that given the enormous N in the analysis, it was possible to find a statistically significant difference in diabetes rates that is likely of little clinical significance, but this has nothing to do with the truth or falsehood of the result itself. The letter also argues that the result is biologically implausible, though it does not seem implausible that a medication could increase diabetes rates during the time of a randomized trial, if only by raising blood sugars in patients near the margin between insulin resistance and diabetes.
A fourth letter suggests that the "diabetes" found in the study might be different in terms of patient-important outcomes than the clinical condition we think of as diabetes. That is, statins might be raising blood sugars in a way that is harmless. While this is possible, it's interesting that when a drug class raises blood sugar people are willing to argue it might be harmless, but when a drug class lowers blood sugar there's a tendency (at least for the manufacturer) to argue that blood sugar control is an excellent surrogate for clinical outcomes. The author of the letter suggests an analysis that might have been done to sort out this issue, which the authors of the meta-analysis correctly point out would not have answered the question.
There were a few other replies to the article, which I have not detailed. Overall, though, this is a fairly typical picture of what happens when someone publishes a trial that conflicts with conventional beliefs, such as "statins are good". This occurs even when the conflict is quite minor -- the meta-analysis merely shows a small increase in diabetes that would be heavily outweighed by cardiovascular benefit in anyone who would be appropriately treated with a statin.
There is no guarantee that the meta-analysis by Sattar et al. is correct about statins and diabetes, but none of the letters published by Lancet raise a sensible reason to think that the post-analysis state of knowledge should change: it is now far more likely than not that statins cause a small increase in diabetes risk. Our response to a meta-analysis like this should be to congratulate the authors on a job well done, while recognizing the possibilities for errors and chance to disrupt the conclusions. It should not be to search high and low for far-fetched flaws that would allow us to discard the inconvenient likelihood that a new statin side-effect has been detected.
In my most recent post, I commented on the DREAM trial's problems with composite endpoints, but also touched on the issue of using diabetes as an endpoint that was assessed while patients were on rosiglitazone.
Today the Lancet (which published DREAM) published online CANOE, a trial of relatively low dose rosiglitazone and metformin for the "prevention" of diabetes. Given the timing right after my discussion of DREAM I feel like I should comment on CANOE as well.
Although I want to mainly focus on the notion of diabetes as an outcome, I'll first point out a major difference between DREAM and CANOE -- the number of patients. While DREAM enrolled 5269 patients at high risk for diabetes, CANOE enrolled only 207. Normally small numbers would not be a concern in a "positive" trial, but where DREAM found evidence of cardiovascular harm, CANOE was so small that it would be unlikely to detect any differences in clinical events -- likely an advantage from the point of view of the sponsor (who manufactures rosiglitazone and rosiglitazone/metformin).
Like DREAM, however, CANOE assessed development of diabetes while patients were still taking the active hypoglycemic medications. As such, the trial was virtually certain to yield a positive result. "Diabetes" was diagnosed by either two blood sugars above 126 mg/dL or an OGTT at 2 hours above 198 mg/dL.
Although we often don't notice it since we think of diabetes as "real", when diagnosed this way, diabetes is a surrogate outcome. The notion of a surrogate outcome can feel quite confusing given how entrenched these sorts of diagnoses (hyperlipidemia, hypertension) are in out psyches, but they should be contrasted with patient-important outcomes like stroke or clinically apparent MI (as opposed to a troponin leak, which is also a surrogate outcome). You can usually tell the difference by asking whether the outcome would matter to a patient in and of itself if you could prevent the potential outcomes that the surrogate is felt to be a risk factor for. For instance, LDL lowering is a surrogate outcome. No patient feels ill with and LDL-C of 200 mg/dL, and if you had a drug that raised LDL higher but lowered cardiovascular risk it would be a helpful intervention. LDL lowering IS NOT a patient-important outcome. Reduction of cardiovascular risk IS a patient important outcome.
Is diabetes a patient-important outcome?
The tricky aspect to diabetes is that, as originally conceived, it is a real disease of great patient importance. Type I diabetes with weight loss, polys, DKA, and rapid death is obviously not a surrogate outcome. Type II diabetes, diagnosed by polys, nonketotic coma, infections, and fatigue is also not a surrogate outcome.
But that is no longer how "diabetes" is diagnosed. Based on diabetes as a risk factor for various bad events, and following the recommendations of endocrinology societies and international organizations, we now routinely accept a diagnosis of diabetes following the rules used in CANOE (repeated fasting glucose above 126 mg/dL; OGTT above 200 mg/dL at two hours) and have added A1c to the mix. While this method of diagnosis may have some justification in thinking about when to initiate therapy (though evidence for this is limited), it clearly makes "diabetes" a surrogate outcome in a trial like DREAM or CANOE: patients do not feel ill with a fasting glucose of 130 mg/dL, and if we could prevent the various long-term outcomes associated with "diabetes" it would not matter at all to patients whether their measured blood sugar stayed at 130 mg/dL or was reduced to 75 mg/dL.
Because diabetes has been made into a surrogate outcome, the prevention of diabetes ceases to be a compelling result in a trial designed like DREAM or CANOE. DREAM and CANOE may show that taking hypoglycemic agents postpones the diagnosis of "diabetes", but the major clinical consequence of that diagnosis is that the patient would be told to start taking hypoglycemic agents. Thus these trials at best show that initiating hypoglycemic agents delays the time until a physician would have told the patient to initiate hypoglycemic agents -- hardly an exciting outcome. Wouldn't it be easier for the patient to just wait until they actually needed the drugs? Instead, the outcomes of the trials should be clinical endpoints like vascular events or renal failure. (As mentioned in my prior post, DREAM created a false clinical endpoint with the composite of "development of diabetes or death".)
At the least, it would have been nice if either DREAM or CANOE had assessed diabetes while the patients were off medication since there was never any doubt that while on medication the surrogate outcome of "diabetes" based on lab results would be less common. Although this issue is never mentioned in the abstract of CANOE, the final paragraph finally pays it some lip service:
An important limitation of our study is that we cannot clearly differentiate whether the effect recorded represents true prevention of diabetes or early treatment of diabetes. This limitation applies to all pharmacological clinical trial interventions for diabetes prevention, and has been reviewed. Further clarification of this question will need a longer follow-up after drug washout, with careful assessment of changes in the underlying pathophysiological events that bring about type 2 diabetes.
The paragraph is an interesting one, since the authors appear to state in the second sentence that the problem of assessment is an inherent issue with "all" diabetes prevention trials, but then in the third sentence explain how a trial ought to be conducted to avoid the problem.
So, like DREAM before it, CANOE produced a completely predictable result that added little to our knowledge of whether diabetes prevention with hypoglycemic agents might actually be of clinical benefit. And unlike DREAM, CANOE was so small that it did not risk finding important harm from treatment with rosiglitzaone/metformin.
In recent years, a common practice in clinical trials is to make the primary endpoint a composite endpoint. That is, the primary endpoint is a combination of various clinical events that might happen, such as heart attack or death or stroke, where any one of those events would count as part of the composite endpoint.
There can be good reasons for composite endpoints. If a single therapy is likely to have similar effects on a number of different related endpoints (such as cardiovascular events) creating a composite can help achieve adequate power. For instance, if statins decrease nonfatal strokes, nonfatal MIs, and cardiac deaths by about 25%, then creating these endpoints as a composite may make it possible to run an adequately powered trial with fewer patients. Additionally, there may be situations were a therapy might decrease an outcome by killing patients off before they can reach that outcome, and so a composite may be a way of evaluating safety.
Composites can quickly get you into trouble, though, if you combine events of very different importance to patients. Sometimes this appears to have been done with the intention of obscuring the real outcome of a trial or to make a therapy look far better than it really is. The primary outcome of DREAM, for instance, was "the development of diabetes or death". DREAM was a randomized trial that looked at the effects of rosiglitazone in patients at high risk for developing diabetes. The article states in the Discussion:
This large, prospective, blinded international clinical trial shows that 8 mg of rosiglitazone daily, together with lifestyle recommendations, substantially reduces the risk of diabetes or death by 60% in individuals at high risk for diabetes.
A reasonable person reading this might conclude that rosiglitazone had had a rather dramatic and important effect on the risk of death in patients in the DREAM trial. In reality, however, the study found no effect of rosiglitazone on mortality (and was always much too small to find any difference), and the primary outcome was driven entirely by development of diabetes. Additionally, since development of diabetes was assessed while patients were taking rosiglitazone (a drug that reduces blood sugar), it was virtually certain that the arm taking rosiglitazone would appear to have a lower rate of "diabetes and death" simply because development of diabetes is based on blood sugar and A1c measurements. Even if rosiglitazone had no effect on the natural history of diabetes, it could be counted on to reduce blood sugar and A1c during the time patients were taking it, and thus appear to delay the onset of laboratory-diagnosed diabetes. Worse, rosiglitzaone has harmful cardiac effects that could be minimized by focusing on a dramatic reduction on a primary endpoint that included the word "death".
It hardly seems fair or true to conclude that rosiglitazone decreased "diabetes and death", but by making that the primary composite endpoint, the trial can claim that this is the only statistically fair way to report the results.
With this in mind, we are now faced with the CREST trial, published online in the New England Journal. CREST was a randomized trial that compared two methods of dealing with carotid vascular disease: carotid endarterectomy and carotid stenting. The primary endpoint was a composite of periprocedural stroke, myocardial infarction, or death, or ipsilateral stroke in the following four years.
CREST found no difference between the procedures in the primary endpoint, but this obscures what really happened. Unsurprisingly, patients treated with stenting, a less invasive procedure, had lower rates or periprocedural MI (1.1 versus 2.3 percent). But they also had higher rates of periprocedural stroke ( 4.1 versus 2.3 percent), as well as a trend toward higher mortality.
I suspect most people undergoing a procedure to avoid having a stroke (the rationale behind both endarterectomy and stenting) would hesitate to accept a 1.2 percent gain in the form of a lower risk of MI in return for a 1.8 percent loss in the form of a higher rate of stroke. As such, the apparent equivalence on the primary endpoint obscures the importance of the underlying components of the composite.
The article was much more upfront about these individual outcomes than the DREAM article, but this did not alter a general buzz in the press (presumably traceable back, in some fashion, to the way the results were promoted) that CREST had shown equivalence between stenting and endarterectomy, and that stenting had now proven itself in a randomized trial.
Hopefully, though, when doctors talk to patients about the choice between carotid endarterectomy and carotid stenting, they will focus on the comparative risks of stroke and MI, and not on the composite outcome, since I suspect very few patients would consider a stroke and an MI to be outcomes that could be fairly equated and thus lumped into a single basket.
ACCORD Lipid, performed in patients with type 2 diabetes being treated with simvastatin, compared adding fenofibrate or placebo. I realized as I read the title and saw the methods in the abstract that I had a rooting interest in the results.
In the past few years, I have written about lipid management, argued about it, and had my judgments vociferously called into question. I've posited that whether or not LDL is important in the pathogenesis of coronary heart disease (CHD), what we really know is that statins reduce adverse cardiovascular outcomes and mortality, and further that we have little evidence that any other lipid-lowering agent (with the possible exception of niacin) actually reduces mortality.
While I'm certainly not alone in questioning the efficacy of non-statin lipid agents, my position has been outside the mainstream. I'd made this point publicly and specifically about ezetimibe at a time when there were no data on clinical endpoints. That position became more mainstream after the ENHANCE trial was published (though it too looked only at a surrogate endpoint), but most experts seemed to feel that ENHANCE only cast doubt on ezetimibe and not on the larger group of non-statin agents. Three days before ACCORD Lipid was published, I was making this argument yet again in a resident journal club meeting, where the article being discussed was analyzing a statin-centric approach to lipids that the article credited, at least in part, to me.
So, within that context of being publicly identified with one position in the argument, and a minority position at that, I read the abstract of ACCORD hoping that it would show no benefit from adding a fibrate to statin therapy, since, if adding a fibrate worked, much of what I had been arguing was likely wrong.
It probably goes without saying -- but I'll say it anyway -- that this is no way to approach a scientific result. The best way to approach a result in a paper and fit it into prior evidence is from a position of disinterest. It allows you to realize that peptic ulcer disease is an infectious disease and estrogen supplementation causes heart attacks, even if, prior to the pivotal publications, you had been treating PUD as a psychological disorder or recommending that female friends and family members take estrogen supplementation.
Yet, when I got to the results of ACCORD Lipid, even though it might have been better for my patients had fenofibrate been of benefit, I was clearly pleased to see that the conclusion was negative:
The combination of fenofibrate and simvastatin did not reduce the rate of fatal cardiovascular events, nonfatal myocardial infarction, or nonfatal stroke, as compared with simvastatin alone. These results do not support the routine use of combination therapy with fenofibrate and simvastatin to reduce cardiovascular risk in the majority of high-risk patients with type 2 diabetes.
But, of course, nothing is ever that clear cut. The authors suggested that fenofibrate might have been of benefit in the subgroup of patients with both high triglycerides and a low HDL-C. In these patients, mortality was 5% lower with fenofibrate with an interaction p value that was borderline significant at 0.06. They even were allowed to make this point within the abstract.
To me, that this seemed like a stretch would be putting it mildly. The paper had looked at at least 10 subgroups, suggesting that even a statistically significant p value in a subgroup would be dubious. Furthermore, fenofibrate had shown no hint of additional benefit in patients with high triglycerides or low HDL-C, but only in patients with both.
Yet, the authors argued that this subgroup result was consistent with post-hoc analyses of a number of prior trials, and the editorial accompanying the paper took the argument quite seriously and stated that the value of fenofibrate in such patients remained unsettled.
So, was I right dismissing this subgroup analysis as likely no more than a statistical blip, or were these others right in thinking it pointed toward an important subgroup of patients who could benefit from fibrate therapy?
I'd like to know that in thinking about this, I had approached the data from a position of objective neutrality, but as I discussed above, I already knew I hadn't. I also knew that the authors of the paper and editorial all had their own intellectual agendas as well.
Conflicts of interest come in many shapes and sizes. But even without such conflicts influencing the interpretation of results, the desire to have been right in the past, and for their current research to have been of value, causes important, and perhaps overriding, biases in experts as they confront and interpret new evidence.
When hierarchies of evidence are listed for the EBM world, meta-analyses of randomized trials generally sit at the pinnacle.
And yet, the actual meta-analyses that you encounter when researching a clinical question can be far less enlightening. Even if we grant a pass to the many systematic reviews at The Cochrane Collaboration that conclude with the a priori obvious fact that no high quality RCTs addressing a question have been performed, and another pass to the reviews that find a single RCT and publish its results as the results of the systematic review, we are still left with the innumerable meta-analyses that seem to provide less of a window on truth than the underlying trials.
Frequently such meta-analyses are either driven by the single large RCT that everyone would have cited anyway or, worse, a number of small, poorly-performed RCTs are combined with a moderate-sized, well-performed RCT and alter the results away from what was likely the best estimate of reality: the results of the well-performed RCT.
Meta-analysts often seem to either be too removed from their subject area and thus lack the expertise to really understand what went clinically right and wrong in the underlying RCTs (or be unwilling to use that knowledge to discriminate among the trials), or be too cozy with a single trial (typically as an author) and thus too willing to ding trials that found conflicting results.
Ultimately, meta-analysis only rarely seems to importantly advance our knowledge of an issue beyond where we would have found ourselves by just reading through the RCTs.
So with that background it is always interesting to me when a meta-analysis comes along that really seems to shed new light on a subject such that we seem to know something that we somehow didn't know when we just had the underlying trials.
An example came along in The Lancet last week.
Despite the enormous number of patients participating in randomized trials of statins, it has been uncertain what effect statins have on the development of diabetes. Some biochemical and animal studies suggested that statins might prevent diabetes. Clinical trials have been conflicting with some showing protection and other showing increased risk. In reviewing the underlying trials, it has been hard to figure out what is going on:
The new analysis found that patients treated with statins had about a 9% higher risk of diabetes than those treated with placebo or other agents. When I started reading the analysis, I had the questions in the list above already in mind and so was prepared to challenge the meta-analysis on several fronts. The authors of the analysis had appropriately anticipated my concerns and, to the extent the data allowed, answered them:
1) Was this really a chance finding driven by JUPITER? Before JUPITER found an increased risk of diabetes, there had been little discussion of statins and diabetes risk. JUPITER's findings could have been due to chance, but the publicity around the result could have triggered the meta-analysis. JUPITER was large enough to sway the results in the meta-analysis and perhaps lead to a self-fulfilling conclusion based in random variation. The meta-analysis, though, did a secondary analysis that excluded JUPITER, and found that the results were essentially the same.
2) Were the varying results in the trials due to random variation or true differences? The meta-analysis found little need to invoke anything more than randomness (as measured by a statistic called the I2). What had seemed to be conflicting results was likely nearly entirely due to random variation around a likely single true effect of slightly increased risk of diabetes.
3) Are some statins protective while others cause diabetes? The finding of little heterogeneity suggests the answer is no, but ultimately this is a hard question to answer definitively because of the more limited data about each individual statin. The meta-analysis found that the confidence intervals of the effects for individual statins overlapped such that it seemed unlikely that there were important differences among the statins, but it's hard to be certain. Additionally, lipophilic and hydrophilic statins showed the same effects on diabetes. And beyond that, the meta-analysis found that one of the main trials that had suggested a protective effect of pravastatin on diabetes had used an unusual definition of diabetes, and the effect was not seen when they substituted a standard definition.
While no new trials were published, as a result of this meta-analysis we have a much better feel for the effect of statins on diabetes than we had a few weeks ago. So, if after hours of trying to answer clinical questions by reading Cochrane you find yourself wondering whether meta-analyses are ever worth the effort that seems to go into them, remember this one and how much we learned about diabetes and statins from a new analysis of existing data.
Recently a friend wrote to me and asked what I thought of a presentation she had heard about a product containing beta-glucan:
I'm not suggesting anyone spend the time to watch all of this, but it is a 30 minute talk at a meeting (of salespeople, I think) of a company planning to market a version of an existing product with a preparation of beta-glucan added to it.
The speaker is excellent, and his audience is clearly impressed to have such a knowledgeable scientist speaking to them, and excited at the chance to market a new healthful product that the speaker promises will be one of the major revolutions in health in this half century or even century.
A claim like that is a bit astounding, and would obviously require high quality evidence to back it up. As Carl Sagan used to say, "extraordinary claims require extraordinary evidence". That someone would make such claims at a sales meeting, such that the attendees were hearing about this revolution in health there first, rather than from the mainstream media reporting on a health breakthrough, might cause the incredulous to suspect they were about to learn about snake oil.
The credulous, on the other hand, might believe that meetings like this are exactly where health breakthroughs are presented because the mainstream media and mainstream doctors would never allow such information to be presented in a medium that they control.
The speaker talks of the astounding immune stimulating virtues of beta-glucan to repair the ravages that modern society has wreaked on our immune systems and points to a number of clinical trials. He goes on to describe the benefits shown in the clinical trials (mainly for upper respiratory tract infections -- URIs) in glowing terms.
Since I'd been asked by a friend for my opinion, I decide to see what I could learn. I searched for clinical trials of the preparation for URIs and found a small randomized trial showing no apparent benefit of a beta-glucan product on URI in athletes.
Not surprisingly, this RCT had not been highlighted in the video. However, a number of other trials were mentioned and were linked on the product's website. Two of the trials listed were apparently only presented in meetings, one was reported as published and I was unable to find the full text (perhaps others will have more luck), but one trial was published in a form I could access.
On the product website, the accessible study was described as showing:
While there were no significant differences in the incidence of symptomatic respiratory infections among the study groups, the duration and severity of symptoms were alleviated in subjects receiving Wellmune WGP.
In the study results, the Wellmune WGP group reported:
• No missed work or school due to colds, compared with 1.38 days of work/school missed for the placebo group. (p = 0.026)
• No incidence of fever, compared with 3.50 incidence in the placebo group. (p = 0.042)
• An increase in quality of life, including physical energy and emotional well-being, as measured by a clinically validated health survey questionnaire (SF-36v-2). (p = 0.042)
This actually tracks fairly closely with what the published study states in its abstract. However, if you actually read the paper it becomes quickly apparent that an enormous number of comparisons were made, nearly all of which showed no benefit to beta-glucan (the missed work or school due to colds was one isolated positive finding). The reported effect on fever does not match the data in the text of the article and, even there, no effect on fever was seen in the intention to treat analysis or in the per protocol analysis, though the latter got close to achieving that all important p<0.05 though it didn't make it.
• No adverse events were detected and no safety concerns were present.
I communicated back to my friend my doubts that anything very interesting had been shown about the product and inquired about possible financial incentives influencing the interpretations of those who were reporting the results to her.
Why is any of this even worth thinking about? After all, the claims made in the video had raised my doubts as soon as I'd heard them, and there are many such claims for many products that are at least as dubious.
The reality, though, is that this beta-glucan product will be marketed and may gain traction. The friend who asked me about it actually currently sells an acai berry product in which she is a true believer. Acai berry products are being marketed all over the Web and doctors are now fielding questions from patients about their value. In my role as a medical editor, I was asked recently by a reader (a skeptical physician) to have our resource include an article discussing the evidence for and against clinical benefits from acai berry supplements.
If we wrote such an article about acai berry, what should it contain? There is no high quality evidence about acai berry and clinical outcomes. Should an article take an agnostic view about its benefits? Should it point out that lots of antioxidant supplements have failed to improve health or have even caused harm? Should it be written at all, or is there some threshold of believability or widespread use that must first be overcome.
The scientific medicine world generally claims it expects some reasonable evidence before administering a new preparation or trying a new procedure (though is not always as good in practice as in theory), and the pharmaceutical industry, for all its problems, can't release a drug without impressive clinical trials.
But when random claims of snake oil salesmen selling some new product spring up, eventually, if the product really does become widespread in the marketplace, the expectation of many in the media and the public seems to be that the burden has shifted to scientific medicine to disprove the benefit or else not take a position about whether a product is good or bad.
At the extreme, you have the most ludicrous ideas of alternative medicine like homeopathy. But even more plausible-sounding ideas when untested do not deserve much credence. That someone who is selling a product can weave a good story about why it is beneficial should not require the resources of the rest of us to test those claims.
We'd all be better off starting out as skeptics, particularly when those promoting an untested product have a financial stake in its success. Snake oil salesmen have been a bane of appropriate medical care for centuries, and there's no reason to think this will change. Those who care about science and evidence, though, must remember that we need not approach entirely untested claims from a position of neutrality. Most things you could imagine doing to improve health do not work, and many are harmful. Though absence of evidence is not evidence of absence, we need not spend time and thought disproving the claims of every snake oil salesman who comes along. It already takes enough resources testing just those claims that have some shred of plausibility as well as the products that have become so widespread that letting them stay untested endangers health.
Lay of the Land
There have been some recent blog posts around yet again debating the concept of evidence-based medicine (EBM).
This blog is titled "Evidence in Medicine" not because "Evidence Based Medicine" was already taken as a URL and blog title, but rather because I didn't want some of the baggage and implications of the EBM label.
EBM is frequently defined as "the conscientious, explicit and judicious use of current best evidence in making decisions about the care or individual patients", and you would think it would be hard to object to that. And yet....
One blogger, Laika Spoetnik (Jacqueline), recently wrote a piece complaining about the attacks on EBM, and mentioned the humorous parachute example that I had referred to in an earlier post. She wrote about the parachute piece::
I found the article only mildly amusing. It is so unrealistic, that it becomes absurd. Not that I don't enjoy absurdities at times, but absurdities should not assume a live of their own. In this way it doesn't evoke a true discussion, but only worsens the prejudice some people already have.
(As I'd suggested in that same earlier post, EBM folks are not necessarily known for their great senses of humor.)
In response to Jacqueline's post, Kimball Atwood wrote a thoughtful piece explaining his problems with EBM, and linked to an earlier piece he'd written on the same subject. I would encourage people to read Dr. Atwood's posts.
Personal Systematic Reviews
So, how could any reasonable person possibly object to the use of best evidence in making medical decisions?
The answer is that they could not, but that EBM is not necessarily synonymous with using best evidence, despite the definition above. If I'd been writing a few years ago, I would have agreed with a number of the posters attacking EBM, since many of the people I have seen promulgate EBM do so in a way that misses some basic truths.
EBM,at least as it was promoted in academic internal medicine circles in the 1990s, was very focused on a specific hierarchy of evidence (RCTs > cohort studies > case control studies > case series) and on the belief that we should train primary care internists that the right way to answer a clinical question they encountered was to perform a literature search along with their own, on the fly, systematic review. I'll come back to the hierarchy of evidence in a bit.
The idea of teaching people to perform personal systematic reviews was being pushed at a time when I was co-directing the resident journal club at my hospital. This was couched as "teaching EBM", and the various residency course directors around the country seemed to be spending a lot of time telling each other what an important activity this would be for academic general internists to engage in. I and the other director of journal club pointed out repeatedly that real systematic reviews take months, and that it was a bizarre notion to think that the right way to answer a question that arose in a clinic would be to try to do this on the cheap. A quick review of randomized trials is fraught with the likelihood of missing important pieces of evidence. When residents try to do this, they often end up focusing on the most recently published trial rather than the entirety of the available knowledge.
We argued, instead, that for most such questions residents would be far more likely to find the right answer in a textbook than by trying to interpret a body of literature in between seeing patients. (These days I would make the same argument but replace "textbooks" with "electronic resources", however that raises a conflict of interest with one of my day jobs, so I won't push that particular argument any further.)
As a result of our resistance, this curriculum never quite got implemented for our journal club, but my sense was that put our hospital in the minority. I think many programs in the late 90s and early 2000s tried to convince residents that it was sensible, practical, and necessary to perform their own systematic reviews whenever confronted with a clinical question.
Hierarchies of Evidence
Coming back to the hierarchy of evidence issue, it's clear that a system that focuses too much on the hierarchy mentioned above will reach absurd conclusions. For instance, one might decide that we have higher quality evidence that an HIV vaccine is protective than that people who wish to avoid lung cancer should not smoke. We have essentially no evidence from RCTs in humans supporting that latter hypothesis and yet we should clearly believe it to be true as surely as we believe anything in medicine, while it remains quite uncertain that an HIV vaccine is of any benefit despite the published RCT with a "statistically significant" p value.
And when effect sizes are large enough, even a couple of clinical events may provide higher quality evidence than some RCTs that try to examine questions of minor benefits of therapies. How many people have to survive meningococcal meningitis after treatment with penicillin before you can be quite sure that penicillin is lifesaving?
Dr. Atwood goes to the specific issue of biologic plausibility in the hierarchy of evidence as it applies to homeopathy. I actually agree with those who are suspicious of biologic plausibility arguments, since smart people can make almost anything sound biologically plausible. Dr. Atwood focuses more on biologic implausibility and I tend to find these arguments more compelling on average. In the case of homeopathy, he argues, it would be pointless and a waste of resources to perform RCTs examining homeopathic remedies. This is a fair point, since if homeopathy is true than our understanding of chemistry and physics is incorrect; thus it would require extreme levels of proof before we should begin to entertain the notion that a homeopathic preparation could be of benefit.
(I plan to write a future post about the burdens placed on the scientific medicine community by people who promote unproven remedies; valuable resources must then often be spent to prove the lack of benefit of the now widely popular yet worthless preparations and interventions.)
EBM and GRADE
Given what I've written you might conclude that I lean toward the anti-EBM camp. As mentioned, that's likely where I would have placed myself a few years ago. In the interim, though, I've started working with the GRADE group and in particular with Gordon Guyatt.
Through these interactions, I've come to realize that when a really smart and thoughtful clinician like Dr. Guyatt discusses EBM, the above problems and concerns mostly go away. For instance, GRADE recognizes the importance of randomized trials, but also that high quality evidence can be found in other places. The group works to formalize what might lead rational clinicians to conclude they have high quality evidence, and while I don't agree with GRADE 100% of the time, my disagreements mostly involve some edge cases that really don't arise very often.
And since Dr. Guyatt is generally credited with inventing the term EBM, it's hard to dismiss his interpretation as being unrepresentative of true EBM. Instead, I've come to believe that most of the people who have interpreted and disseminated "EBM" lacked either the clinical understanding or the epidemiologic knowledge to communicate a fair representation of EBM to the clinical world.
I'd encourage those who find themselves troubled by aspects of EBM to spend some time looking at what the GRADE Working Group is doing. Even better, if you have the chance to interact with Gordon Guyatt at one of his courses or lectures, jump at the opportunity. You likely will feel differently about EBM for having done so.
Over at Effect Measure, Revere has posed a thought experiment (or, perhaps, a disguised real experiment) that provides a valuable jumping off point for discussing the worth of observational data.
In brief, Revere describes an uncontrolled study for a new application of an antiepileptic drug as a treatment for refractory hypertension. A study is conducted by administering the drug to 29 patients and checking their blood pressures at baseline and after a month of therapy.
I glanced at the comments on the blog, and interestingly a lot of people are spending time trying to apply a name to what exactly this study is (i.e. "case series", "observational", and the like). I often don't find naming things all that helpful unless everyone has a clear understanding of the naming convention, but this would most commonly either be called an observational study (if we view that the therapy was going to be administered independent of the interest in collecting data about what happened), or an uncontrolled clinical trial (if we view that the primary goal was to find out if the AED works for refractory hypertension).
In the thought experiment, the blood pressure readings a month later are substantially lower.
Revere poses the following question:
"...is this a sufficiently reliable study that a reasonable practitioner, committed to the use of scientific evidence in her practice, would consider?"
I'm going to move on quickly from that question, because I think the correct answer isn't that helpful: the study contains data about a potential new therapy, and a reasonable practitioner should always consider such evidence if she or he has the time and ability to do so.
To understand, instead, whether we should believe that the results in this study can be ascribed to the drug working, we might look first at how we would feel about the result if it had been obtained from a randomized trial performed in the same manner described as the uncontrolled trial, except now we have 29 patients randomly assigned to receive the AED, and 29 to not receive it. Notice that this RCT is still unblinded and, as such, has no placebo arm.
We'd likely note a few problems with this RCT. First, it has a relatively small number of patients, though if the drug is highly effective this might not be a problem. We'll come back to this point a bit later.
Second, the outcome is fairly indirect (in GRADE terminology) to an outcome of clinical importance. Revere points out that BP reduction is not a clinical outcome: it's a commonly used surrogate, but what we really care about is a reduction in cardiovascular events and this is not being measured. BP lowering is such a standard surrogate that it's easy to lose sight that it is a surrogate at all, but not everything that lowers BP is good for you (consider endotoxin, for instance). However the actual measurement has an additional level of indirectness: we are measuring BP reduction at one month as a surrogate for long-term blood pressure reduction. Long-term BP reduction is the marker that we actually typically use as a surrogate for a reduction in CV events.
Third, as noted above, the study is unblinded. Does this matter? We're told that the BP both at baseline and at one month is being taken with an automated cuff and all the clinical assistant has to do is write down the measurement. This sounds like a simple solution, but it is fraught with problems. The patients are unblinded, and so the ones taking the medication may believe in its efficacy, relax more, and so have lower BP readings in the office. The clinical assistants believe in the drug and want the study to succeed, and so may take several readings in patients on the AED ("why don't you sit here and relax, and I'll recheck your BP in a few minutes -- this drug looks like it's working in almost everyone, so I'm sure your BP will come down if we just wait"), but record the first BP reading in the control group patients. And, of course, those analyzing the results know who is and is not receiving the AED, and this might affect all sorts of things (not that we like to admit this).
So this small, unblinded, short-term study of a surrogate outcome might not be all that convincing, even as an RCT.
When we push it a step further and make it uncontrolled, we add an additional major problem of regression to the mean (as well as secular trends in BP over time). In the RCT, we had a control group, and so we at least knew what happened to the BP readings in patients not treated with the AED. in the uncontrolled study we have no such knowledge. We're not told exactly what the entry criteria are, but presumably having a well-controlled BP would prevent a patient from getting the AED as an additional anti-hypertensive. Thus, patients who are oscillating around a mean BP value will likely be excluded if they are at a BP minimum and enrolled if they are at a BP maximum. One month later, just due to the usual biologic variation plus this entry criterion, the BP can be expected to be lower on average.
Adding this problem to the ones we noted in the RCT version, we are unlikely to be terribly sanguine about believing the results of this small trial.
Are there any circumstances, though, where despite all this we might trust the results anyway? One of the real insights of the GRADE group, to my mind, is a focus on magnitude of effect in thinking about observational data, rather than focusing on minor differences in study design (such as cohort versus case control). With large enough effects, we might accept some pretty minimalist study designs.
For instance, rabies is historically a 100% fatal illness once clinical symptoms appear in someone who has never been vaccinated. In 2005, a 15-year-old girl survived rabies after treatment with a novel experimental regimen. You could imagine that anything can happen once, and it may have been coincidence that this girl survived and received this novel regimen. However, if one more person with rabies were to receive the regimen and survive, it would seem spectacularly unlikely that the explanation would be anything but that the regimen works. A rational clinician would treat any new patient with rabies with that regimen from that moment until a superior regimen was found. (As far as I know, no other patient has survived rabies on this regimen.)
Similarly, in response to the humorous attack on EBM, asking what the RCT data are supporting parachutes when skyjumping, the GRADE group would respond that the magnitude of effect of parachutes is sufficient to constitute high quality evidence for their use (okay, clinical epidemiologists can be somewhat humor-challenged). That is, we have all sorts of historical evidence about what happens when people fall from great heights (and that we might consider only slightly indirect to falling from an airplane), as well as lots of observational data about what happens when people fall from airplanes wearing parachutes. Not everyone who falls from a great height without a parachute dies, and not everyone wearing a parachute lives, but the effect size is so large that we have high quality evidence for parachutes in the absence of a clinical trial.
To give one example that might feel more real, I was doing a lot of AIDS care in the 1990s. In 1995 an abstract was published from one of the ID meetings about the effects of ritonavir in about 50 people with late AIDS. (I've tried in vain in the recent past to find this abstract -- if anyone can point me to it, I would be grateful.) The results were like nothing we had seen before -- patients' CD4 counts rose dramatically, opportunistic infections improved, and many patients who would have been expected to die improved instead. We did not know what would happen long-term, but it was obvious, without any RCT, that ritonavir was effective therapy for AIDS, at least in the short term. By 1996, we were treating people with triple therapy "cocktails" for HIV, again without any RCTs with clinical endpoints, and watching people who had been dying walk out of hospitals and hospice care as their OIs resolved. The magnitude of effect was such that we had high quality evidence for these cocktails based on observational data alone. (Not that this actually prevented researchers from proceeding with an RCT of triple therapy, but that's a post for another day.)
In the refractory hypertension thought experiment, the problem is that we do not have enough evidence about how unusual the BP lowering seen was to conclude that this was a very large (and thus very unexpected) result. The result could have been due to measurement biases and regression to the mean, and so have had nothing to do with the actual drug being studied.
Revere started his/her post talking about Cochrane and the focus on RCTs. I agree both with concerns about Cochrane reviews (which can be of quite variable quality), and about the general issue of ignoring high quality observational data (most often high quality because of very large magnitudes of effect) to focus exclusively on RCTs. However, I don't find the evidence from the study in Revere's thought experiment to be sufficiently high quality that I would be generally willing to administer the AED for refractory hypertension without further study. Having read such a study, I might take it into consideration sufficiently to administer the AED as a last resort -- a situation in which I might be willing to grasp at any straw of evidence.
One consequence of my writing this blog is that many of the residents I work with have started reading it -- a risk I clearly hadn't adequately considered. I received a request from one resident to write about some problem that I think residents commonly have with applying evidence to practice. That was quickly followed by my sitting in as discussant at a resident-led journal club, which led to my thinking about the many unfair and fair ways that people attack published randomized trials. I'll have to give additional thought in the future to the more general question of common ways residents misapply evidence in practice.
By way of background, it is standard practice in academic medical centers to present a trial at journal club and then rip it to shreds. After repeating this exercise over and over, it's easy for all involved to conclude that every trial is flawed and nothing can be learned from any of them. It can become almost a game to figure out why not to believe a given trial, rather than discover which trial results should and should not be applied to practice. (I should point out, here, that the resident who led the journal club I mentioned above made fair criticisms of the presented trial and concluded with how the trial would affect her clinical thinking and decisions.)
I think I fully recognized how far out there people could get in criticizing virtually any trial back in 1993 in the midst of the H. pylori debates. At that point there was still a lot of doubt in some quarters as to whether H. pylori caused peptic ulcers. There had been trials of eradicating H. pylori showing reduced ulcer recurrence, but the eradication regimens included bismuth which was believed to have some direct anti-ulcer activity, leaving a hole in the causal argument.
To deal with this a trial was performed looking at recurrence rates after an eradication regimen using amoxicillin plus metronidazole. Ulcer recurrence rates at one year were 8% in the treated arm and 86% in the placebo arm. To me, this seemed to give a final answer to the question of H. pylori causing ulcers, but I listened to a room full of internal medicine residents from one of the top programs in the country come up with outlandish criticisms of the results. Several residents argued, apparently convincingly to others in the group, that amoxicillin and/or metronidazole might have direct anti-ulcer effects and so the reduced recurrence rates might have had nothing to do with H. pylori. There were several problems with this argument, including the lack of any a priori belief that these antibiotics had such an effect, that an anti-ulcer effect of a 12-day regimen would be unlikely to have such an enormous impact a year later, and that when the trial was analyzed by H. pylori eradication rather than treatment arm, the effect size was even larger.
Another common and generally unfair way to attack a randomized trial is to look for unmeasured confounders or confounders that are somewhat unbalanced in the table 1 data.
Confounders are things associated with both the exposure and the outcome of interest in a study, that through these associations confound the assessment of a relationship between exposure and outcome. For instance, we might find in a study that coffee drinking is associated with lung cancer. Coffee drinkers may smoke more than other people, and so, if we fail to control for smoking, we might falsely conclude that coffee drinking causes lung cancer.
In a properly performed randomized trial of a reasonable size, unmeasured confounders will sort nearly equally among the treatment arms. Even if smoking status is unmeasured, if there are 800 patients in each of two arms, we can be pretty sure that the number of smokers in the two arms will be close enough not to disturb the trial results. Table 1 in a published RCT typically gives the baseline data on the groups, and if lots of potential confounders are measured and reported on at the 0.05 significance level, we shouldn't be surprised when 1 in 20 appears unbalanced at such a level. That said, if lots of measures appear unbalanced, with one group clearly sicker than the other, it's important to go back and see that allocation in the trial was properly concealed. It's rare, though, to see problems of unbalanced allocation in a major clinical trial, particularly a placebo-controlled trial of a medication.
As is clear from some of my earlier posts on this blog, I obviously believe that some attacks on randomized trials are fair and appropriate. More fruitful attacks often deal with issues of what the GRADE group calls "indirectness". Results of a randomized trial might be indirect to the question a clinician is interested in because the trial did not study the exact intervention of interest (for example, too high or low a dose of a comparator drug was used, as in a trial comparing esomeprazole with omeprazole), perform the study in the exact population of interest (for example, the study was performed in young healthy patients, and the target population is the sick elderly), or measure the exact outcome of interest (for example, a surrogate outcome like carotid intima medial thickness was measured, rather than a clinical outcome).
I recall an argument years ago on Usenet about whether aspartame really causes headaches in the many people who report headaches when they consume aspartame-flavored foods and drinks. At the time, I pointed out the randomized trial in the NEJM showing no difference in rates of headaches in subjects who believed they were sensitive to aspartame, whether given aspartame or placebo. Another poster on Usenet objected, saying that he did not think that aspartame in a pill caused headaches, but rather that the combination of sweet tasting food and lack of sugar led to headaches, perhaps by inducing reactive hypoglycemia. Testing aspartame without letting the subjects taste the preparation was completely unconvincing to him. This was essentially an argument about indirectness of intervention -- the intervention of interest was aspartame as commonly consumed in foods and drinks, not aspartame in pill form.
To me, on reflection, this seemed like a fair objection, though I didn't think it really held up to scrutiny when looking at the actual trial results. In the trial, which had a crossover design, the 40 subjects developed headaches 35% of the time when they were given aspartame, and 45% of the time when given placebo. To argue that encapsulated aspartame doesn't cause headaches in susceptible individuals is to miss that the trial found that placebo induced headaches in these individuals almost half the time.
I'll come back to the issue of indirectness of outcomes in the near future when I'll write more about surrogate outcomes, since this underlies some of the arguments that came up in the comments responding to my vitamin D posting, and I'll likely write about indirectness with regard to population down the road.
The general point, though, is the importance of being fair, open-minded, and keeping a big-picture view when evaluating a new randomized trial, and then interpreting the results in the context of existing evidence. Lee Goldman talks about dealing with randomized trials when one is performing the role of playwright rather than theater critic. But even if you're just the theater critic, if you hate all plays you see, your reviews won't prove terribly edifying to your readers.
My previous post, though not really intended to be focused on p values, led to a long and interesting discussion on the issue by Revere.
Revere commented that it wasn't so clear whether I was writing as a frequentist or a Bayesian. In reality I'm a consumer of biostatistics, not a statistician. For me, the issue is pretty simply to try to know whether a given result I'm seeing is true.
Although I wrote about this in my last post I didn't frame it in exactly this way: the problem with p values is that I want to know how likely it is that the result I'm seeing is correct, while the p value tells me how unlikely it is that I would be seeing a given result if the null hypothesis were true. This is similar to the problem with knowing the sensitivity and specificity of a test, when what you really want to know is the positive predictive value (how likely is it that the patient has the disease now that the test has come back positive?).
The academic medical world started hearing a lot about using confidence intervals in preference to p values 15 or 20 years ago, and I think that led to many doctors concluding that confidence intervals solve this basic p value problem. Confidence intervals have some real benefits compared with p values, but this is not one of them. To look at this, we need to try to understand what a CI means.
Again using the HIV vaccine example from my last post, we can look at the same parameter that had a p value of 0.04, but now examine the point estimate of efficacy (31.2%) and its 95% CI (1.1 to 52.1%). What is it that has a 95% chance of being true, given that CI of 1.1 to 52.1%?
However tricky p value interpretation is for physicians, understanding the meaning of this CI is much worse. I've almost never heard a physician able to correctly interpret the meaning of such a CI when put on the spot with the above question. When trying to teach the interpretation I repeatedly get told there must be a simpler way to communicate the idea. Given my years of failure at this, I have little hope that this post will adequately clarify things for most readers, so if others who teach this have found a way that works to explain it, please write!
All we can really say about that CI is that (excluding any problems with the design or performance of the trial) if we performed the same study 100 times and calculated 95% CIs each time, we would expect that 95 of the 100 CIs formed in this way would include the true point estimate of vaccine efficacy.
Note that just as with p values, this isn't what we want to know: we want to know how likely it is that the true value is inside the particular CI that we are looking at, but that isn't what the CI actually tells us.
However, despite recognizing what the CI really does and does not tell us, consumer of biostatistics that I am, I (and others) approach CIs operationally: we choose to interpret the CI as a range of values with which the data are reasonably compatible, and to interpret values outside the CI as reasonably incompatible with the data. So, other things being equal, I would say that a vaccine efficacy of 5% was compatible with the results of the NEJM study, while a vaccine efficacy of 60% was not. This does not mean that I think the study has excluded the possibility of the vaccine having 60% efficacy, just that this would be unusual under the plays of chance.
This operational definition works as I decide how to write recommendations about whether to administer such a vaccine. If the vaccine truly had an efficacy of 31% I would likely recommend wide use in high risk patients. If the high end of the CI were true (52% efficacy), I might recommend universal vaccination. If the low end were true (1% efficacy) I would probably recommend leaving it on the shelf. Looking at this, I can quickly realize that if this trial were the only information I had about vaccine efficacy then I have inadequately precise data to support whatever recommendation (or set of recommendations) I might want to make about administering HIV vaccine.
If I were using the GRADE scheme for grading such recommendations, I might have started with the assumption that I had high quality evidence from a large randomized trial. But when I realize that I would make different recommendations based on the reasonable values at each end of the CI, I know that I must downgrade the quality of the evidence for such imprecision. Recommendations for HIV vaccine based on this trial, using the GRADE scheme, would certainly be graded as having no better than moderate quality evidence because of this imprecision.
Instead of some arbitrary definition of whether a trial is large enough or precise enough, using the CI in this way allows me to communicate something important about the quality of the evidence as I grade recommendations. I downgrade for imprecision not because a CI crosses a null effect boundary (like 1.0 for a relative risk) but because the CI crosses a clinical boundary where the appropriate recommendation would change from one side of the boundary to the other.
In this way, the CI is far more useful than a p value. I keep in the back of my mind, though, that the CI doesn't really mean what I'm trying to use it to mean -- it's just that I usually don't have anything better.
I'll write more in the future about how GRADE looks at other types of limitations on the quality of evidence from randomized trials.
As mentioned in my last post, I want to continue the discussion of plausibility with the recent evidence suggesting benefit from an AIDS vaccine. This was a randomized, placebo-controlled trial in 16,000 adults, that, at least in the initial press releases, showed a statistically significant 31% decrease in HIV infection in vaccinated individuals. How confident should we be in the results?
The analysis that showed the statistically significant benefit (p = 0.04), was a reasonable one. There were 56 people who developed HIV in the vaccine arm, and 76 in the placebo arm, but seven patients were found to be infected with HIV at the time the vaccine was administered, and this analysis excluded those patients. Before we come back to that decision, we should keep in mind what that p value means -- what it is that has a 4% chance of having happened?
Although this causes untold confusion for physicians at all levels of training, that p value says that (excluding any problems with the design or performance of the trial), if vaccine were no better than placebo we would expect to see a difference as large or larger than the one seen in this trial only 4 in 100 times. This is distinctly different from saying that there is a 96% chance that this result is correct, which is how many people wrongly interpret such a p value.
One reason this is so is that many values of truth closer to the null hypothesis (but still where vaccine is superior to placebo) are possible and consistent with the result.
A more important issue of iterpretation in many cases including this one, though, is that we are not starting from a position where all results are equally likely. We had a lot of reason to be suspicious prior to this trial that the vaccine would fail to protect against HIV. Although this way of looking at results is often given important-sounding terms like "Bayesian reasoning", really all we're saying is that if something was unlikely to begin with, we need a higher level of statistical "proof" than if something were likely; if something is extremely likely to begin with, we need a much lower level of "proof".
Imagine a small randomized trial of statins in a population where they have haven't been clearly studied (say, hypertensive smokers without known CHD). If the trial found an insignificant relative risk of CV events in the treated group of 0.78 (22% relative risk reduction,p = 0.40), it would not be the case that there is a 40% likelihood that this result was due to chance. We have lots of prior evidence suggesting that just such a RR is likely, and our belief that this reduction in risk was due to chance should be much smaller than suggested by the p value in a single trial.
In the case of an HIV vaccine, the failure out multiple prior HIV vaccines gives us a baseline doubt about any new vaccine. Added to that is the apparent difficulty of even actual HIV infection to prevent subsequent infection: patients with HIV who have a normal CD4 count and are on HAART that is adequately controlling their infection can become superinfected with a strain of HIV that is resistant to the HAART regimen they are receiving. This raises real concerns about how to create a vaccine to prevent infection.
So, the modestly positive result found in the trial must be weighed against our prior belief that such a vaccine would fail. Had the vaccine been dramatically protective, giving us much stronger evidence of efficacy, our prior doubts would be more likely to give way in the face of high quality evidence of benefit.
And then, into the mix must enter the reality of the multiple analyses performed by the researchers. Although I said the analysis that gave this result was reasonable, it was not the analysis they apparently first planned to perform. The study specified an intention to treat analysis (that did not find a statistically significant benefit at the 0.05 level), and a per protocol analysis (that also did not find a statistically significant benefit).
While the actual analysis the investigators decided to make primary would be completely appropriate had it been specified up front, it now suffers under the concern of showing marginal significance after three bites at the statistical apple; these three bites have to adversely affect our belief in the importance of that p value. And, it's not so obvious why they would have reported this result rather than excluding those 7 patients from the per protocol analysis and making that the primary analysis; there might have been yet a fourth analysis that could have been reported had it shown that all important p value below 0.05.
The authors of the study tell us that the choice of analysis was made prior to unblinding of the results, but that probably isn't adequate. It had been used in interim analyses (and the final analysis) on the blinded data, but likely so had the intention to treat and per protocol analyses. If this analysis were the only one that achieved a p value below 0.05, it could have been chosen as the preferred analysis for that reason alone. The investigators wouldn't have known prior to unblinding whether the trial would show benefit or harm with vaccine, but either result would be more interesting than a statistically non-significant outcome. The vaccine manufacturer would be far better off with a statistical benefit, but probably no worse off if the vaccine showed harm than if it showed no effect.
I've been struck since the beginning of my medical training that biostatistics is an area of mathematics where people can have remarkably different opinions about what is or is not not resonable. I don't usually think of "opinion" and "mathematics" falling in the same sentence, but perhaps that just shows my lack of expertise in theoretical math.
In this case, while the investigators may have felt they were reasonable picking the most "appropriate" analysis, we who interpret the study have to worry about those three statistical bites. Combined with the prior likelihood that the vaccine would fail, I think we need to remain very wary that despite an apparent 31% vaccine efficacy in a trial involving 16,000 patients, the vaccine may in reality be no better than placebo.
Listening to NPR
There was a third incitement to this blog that I have not yet mentioned, as I am not currently in possession of an asbestos suit, but I feel the time has come to discuss it. I was driving to work listening to NPR's Morning Edition last month when I heard, "We have news, now, of a scientific breakthrough about a disease that affects over a million people in the U.S."
My ears perked up, wondering what major breakthrough I had missed. I learned from the interview that an article had been published in Science linking the retrovirus XMRV with chronic fatigue syndrome. On the usually reserved and balanced NPR, I was startled to hear the interview proceed in such a way as to assume that the study was accurate and that it proved wrong all those who had questioned CFS as a single disease entity. I had the sense listening that either the interviewer or the producer of the piece had a personal stake in the outcome of the story.
The article in Science reported that 67% of patients with CFS and 3.7% of healthy controls had XMRV DNA. A news story reported that, after the submission to Science, the authors of the study improved their assay for XMRV antibodies and found them in 95% of patients with CFS. Is it plausible that 95% (or 67%) of patients diagnosed with CFS really have a single disease caused by the retrovirus XMRV?
Plausibility in medicine is a tricky concept. Almost any result can be argued to be biologically plausible by someone with a modicum of creativity. And yet, biologic plausibility often does not pan out so well when it meets real life. When I was in training, beta blockers were considered absolutely contraindicated in patients with heart failure and the incipient notion that peptic ulcers were an infectious disease was considered absurd.
There's a different kind of plausibility, though, for which I have greater fondness. It has to do with prior probabilities developed from a fair-minded look at the epidemiology of diseases or from prior trials that test some similar notion to the current idea under scrutiny. For instance, while JUPITER found a 44% reduction in its composite primary endpoint, it seems highly implausible to me that rosuvastatin really lowers risk by this much. Nearly every statin seems to lower risk in primary and secondary prevention by 20-25%; why should rosuvstatin have such a marked effect in a fairly low risk population? It seems more likely that the effect was exaggerated when the trial was stopped early for benefit than that we can really expect these sorts of results from rosuvastatin in patients with low LDLs and mildly elevated CRPs.
Evaluating the Evidence
The article in Science is an odd read for a clinician. The same article in a medical journal would spend a lot of time describing who the patients and controls were, and how the patients were diagnosed with CFS, but none of that is in the article. Like most articles in Science, it spends its time on bench methodology (how the retrovirus was detected) rather than on the methods by which CFS was diagnosed and the blinding used when the samples were analyzed.
That raises an initial question of "are these CFS patients similar to those who walk into a primary care physician's office carrying a diagnosis of CFS?" That is, can the results be generalized to the broad group of people who are said to have CFS? Without a more detailed description in the article, I have no way to be sure.
But beyond that, what has happened with other CFS "breakthroughs"? Back in the 1990s, after an article showing much higher rates of neurally mediated hypotension in patients with CFS (22 of 23 patients) than in healthy controls (4 of 14 controls), I made the same argument on Usenet that I'm in the process of making here. That article led to patients with CFS being treated with fludrocortisone and similar agents. Eventually people concluded that the neurally mediated hypotension story didn't really make sense and clinicians and patients moved on.
From my point of view, given the patients I have seen carrying the label CFS, it seems extremely unlikely that any single etiology will explain 95% of cases. Patients carrying the label CFS in the real world (rather than in research studies) seem to be a mixed bag. Many seem to have atypical depression or another psychological cause of fatigue (thus my need for the asbestos suit). Others, though, likely have a missed diagnosis or have some condition that is, as yet, unknown to medical science (retroviral infection with XMRV would have previously fit in this category).
Interpretation and Proof
Why does it matter? From the clinical perspective, many people with depression (typical or atypical) can be helped by standard therapies. Avoiding the label of depression for those who are depressed is no kindness (though it is also of no help for doctors to insist that symptoms are due to depression when they are not; I do not believe that all patients with CFS really suffer from depression). And getting therapies for purported etiologies that don't pan out (such as fludrocortisone for neurally mediated hypotension) simply exposes patients to side effects, harms, and costs, without resulting benefits.
Beyond that, though, this issue is at the intersection between evidence, experience, and opinion. I have no expertise in retroviruses, know nothing about XMRV or the methods used to detect it, and am lacking detailed evidence about how the study in Science was performed. But evidence about diseases still must be interpreted through the lens of clinical experience.
Ultimately, many of the arguments between clinicians about CFS (let alone the arguments joined by patients) have to do with differing clinical interpretations of whether CFS feels like a believable single disease process. This is an area where a discipline like EBM doesn't provide much of a road map. Yet expert clinicians must make these sorts of judgments all the time when they decide whether to consider a cluster of signs and symptoms as an individual disease. Before HIV was discovered, I listened to expert clinicians reject the arguments for multiple immune system insults or inhaled nitrates as causes of AIDS, and instead point out that AIDS would turn out to be caused by a virus that was transmitted similarly to hep B given the similar epidemiology of the two diseases. There is no path from evidence to understanding that does not rely on expert interpretation, and, ultimately, no mechanical measure of sufficient evidence or proof outside of what counts as proof to those patients and providers who must make decisions.
XMRV may yet turn out to be the cause of CFS, but given the track record of such causes, I think we need better evidence. Before NPR describes a "scientific breakthrough" and gets the hopes up of so many, I wish they would spend more time thinking and interviewing and looking for some balance. I have to believe they could have found an expert or two who would have urged caution in interpreting these results.
If future studies confirm the XMRV hypothesis, those out there now who are infuriated by my comments can feel free to say "we told you so." But if XMRV turns out not to be a very good etiology for real world cases of CFS, perhaps there might be more patience for those of us who urge not trying to explain all the many chronically fatigued patients with a single disease entity. In either case, it's worth recognizing the role that expert interpretation plays in medical controversies.
This post is long enough, and the second virus from the title of this piece will need to wait for a future post: I'll discuss the likelihood that we've really found an effective vaccine for HIV.
(After I wrote this, Marilyn Mann pointed me to a blog posting with some similar concerns about CFS having a specific viral etiology.)
I'd mentioned in an earlier post that a relative was prescribed fairly high dose vitamin D by her primary care physician despite a blood test showing that her vitamin D level was in the normal range. Her PCP may not be much of an outlier. Among the residents I work with, I've been encountering a lot of desire to prescribe vitamin D to pretty much everyone.
Why the willingness to administer vitamins? Would these same doctors feel that a drug with similar levels of evidence for safety and efficacy should be administered to tens of millions of generally healthy people?
Somehow patients and providers alike seem willing to treat vitamins as articles of faith -- as substances that must be good for humans based on their very name and nature. Yet the actual track record of vitamins, and particularly the fat-soluble vitamins, should not lead us to be so sanguine.
Based on their antioxidant properties, vitamin A/beta carotene and vitamin E were believed to be good for health before any reasonable studies had been performed. Doctors recommended beta carotene and vitamin E at moderate or high doses, and health-conscious patients took even higher doses. Then the results of randomized trials started to come in:
Two large randomized trials of vitamin A/beta carotene found increased rates of lung cancer in smokers, pretty much putting an end to the notion of using beta carotene to prevent cancer. Then, a meta-analysis of trials of vitamin E found an apparent increase in mortality in patients treated with higher doses of vitamin E, and suddenly vitamin E no longer appeared so benign.
With this as a backdrop, we might expect physicians, at least, to show some skepticism before recommending other vitamins, and particularly other fat-soluble vitamins.
Yet, in their defense, vitamin D has been studied in randomized trials (mainly for prevention of fractures/osteoporosis), and these trials have reported other outcomes including cancer rates and mortality.
Although much larger trials have found no effects of vitamin D on cancer rates, one trial that I've heard people cite did indeed find a reduction in cancer rates in patients supplemented with calcium and vitamin D. The RR was 0.23 -- in other words, the supplementation appeared to reduce cancers by 77%. Since virtually nothing in medicine has an effect that large, and such an effect would be hard to have missed in other trials of calcium and vitamin D, this result seems implausible at best.
With regard to mortality, a meta-analysis of randomized trials found that vitamin D supplementation appeared to decrease mortality by 7%. A concern, though, is that the result was just barely statistically significant, and there could be reporting bias with regard to mortality: trials that found larger differences in mortality (or any other safety endpoint -- these were, after all, not trials designed to look at mortality) would be more likely to report on mortality than trials that found nothing interesting when they looked at their safety data. Given the risk that the results could be due to chance, even a small amount of reporting bias would be worrisome.
So, I've now objected to one trial of vitamin D for having too large an effect to be believable, and a meta-analysis for having too small an effect to convincingly rule out chance plus bias. Am I being fair and true to the evidence?
If we were talking about a therapy for a morbid illness where we were compelled to treat, I'd likely accept the evidence from the meta-analysis as being sufficiently hopeful to make vitamin D worth a try. After all if treatment is required, even if vitamin D eventually turned out to be harmful, we would know that we had needed to do something for the ill patient and so we made our best guess.
But that is not the case here. Instead, we are talking about administering a therapy to enormous numbers of healthy people with the hope of preventing disease, and where causing even small increases in harmful outcomes would have no justification. And remember the history of vitamin E and beta carotene. If that is not sufficient to make you concerned, read what happened to cancer rates and mortality in two trials of folate/B12 supplementation in Norway, published just last week in JAMA.
In such a situation we should wait for (and demand) more convincing evidence before we act. It may turn out that vitamin D really does prevent cancer, decrease mortality, and protect people from everything from influenza to the common cold. But I would rather be late jumping on this bandwagon than jump early and find that I hadn't learned a thing from past mistakes.
A number of the other blogs that have noted the existence of this blog seem to have "Pharma" in their titles in one way or another, and the implication is not that the blogger likes much about Big Pharma.
In the late 1990s, when I was spending much of my clinical time providing HIV care, I was a defender of Pharma on Usenet, at least as it related to HIV. We'd discovered the HIV virus in 1985, and by late 1995 we had a spectacularly successful treatment, due, at least in part, to the efforts of the drug manufacturers. HAART changed the lives of people with AIDS, and the lives of their caregivers, and costs of $10K to $15K per year to keep a young person alive and functional seemed perfectly reasonable to me. I thought the carping about the high prices of these medications was unfair -- it was a remarkable accomplishment to find an effective treatment for a viral infection in so short a time, and the billions spent on it deserved some reward.
By a few years later, though, I was finding myself as distrustful and angry at Pharma as anyone out there. I spend a lot of my time reading research papers and interpreting evidence, and realized one day that Pharma was employing people at least as smart and knowledgeable as I to come up with ways to trick us all about what studies showed. One example of this is running parallel studies of an agent and then only publishing the positive studies while burying the negative studies, as was done with trials of SSRIs in adolescents. The concept of a p value becomes essentially meaningless if researchers perform an unknown N of parallel trials and then publish the ones that show a positive result.
I'll write about other ways of manipulating results in future posts, but for today I want to comment on one pernicious effect of this behavior -- I have come to distrust and then dislike the companies that provide the (often useful) drugs I must use on patients. When the new HDL-raising drug torcetrapib went down in flames a few years ago, I was actually pleased: I was furious with Pfizer for having tried to get torcetrapib approved in a way that would have only marketed it in a combo pill with atorvastatin. Objectively, it makes no sense that I should be happy at the failure of a drug that would have helped my patients had it worked.
Pharma not only tries to manipulate the interpretation of results of trials, in the design, analysis, and publication phases, they apparently also try to sow doubt about existing competitors. Adriane Fugh-Berman wrote in 2005 about how she was recruited to be the author on a ghost-written manuscript intended to highlight the importance of herbal interactions in patients on warfarin (interactions that Dr. Fugh-Berman felt were overstated). The company recruiting her was presumably working for AstraZeneca, the manufacturer of ximelgatran. Ximelgatran, a direct thrombin inhibitor that aimed to replace warfarin, was eventually withdrawn from the market in Europe (never approved in the US) because of hepatotoxicity.
In the face of this comes the RE-LY trial comparing warfarin with dabigatran, another direct thrombin inhibitor. I didn't start blogging until well after the publication of RE-LY, but hopefully others noted that this 18,000 person trial may reflect one of the biggest advances in medical therapy of the decade. Dabigatran appears to be a superior drug to warfarin -- at lower doses it is safer than wafarin and as effective, and at higher doses it is equally safe to warfarin and more effective. Unlike warfarin it does not require a careful diet or routine monitoring of INRs.
By all rights, dabigatran should take over the oral anticoagulation market, at least for atrial fibrillation (we need additional trials in other clinical settings). The natural skeptic in me would always want to wait a bit on using a new drug. Warfarin has been around for decades and has been used in tens of millions of people and we have a good sense of its benefits and (myriad) risks and burdens, while dabigatran has not been used nearly so widely nor for so long. New drugs have a habit of showing up rare side effects once they are out of clinical trials and on the market.
But my greater fear is that somehow Pharma is pulling something over on me, the editors of the NEJM that published RE-LY, and everyone else, and that it will somehow turn out that dabigatran not only isn't as big an advance as it appears, but that its manufacturers already know this and aren't telling.
Ultimately, if I had to bet, I don't think this will be the case with dabigatran. Five to ten years from now, I'm guessing that the residents I teach will have no clue that we used to routinely treat patients with rat poison and monitor bleeding parameters every couple of weeks or so in order to prevent clots and emboli. But I wish I didn't have to fear that the manufacturers of the drugs I use might not actually be working toward the same goals that I am, and instead might be working to fool me yet again.
It's hard to have a blog about evidence and medicine without commenting on the changes to the USPSTF's breast cancer screening guidelines.
At a first pass, it's worth pointing out that a lot of academic primary care physicians have been wondering what the USPSTF was thinking when they published their 2002 guidelines recommending mammography for women in their 40s. Yes there may be some small benefits (and lots of harms), but this is just the sort of thing that the USPSTF can usually be relied on to not recommend -- look for instance at what they recommend for prostate cancer screening or lipid screening for women. As a result, there were lots of insinuations that the USPSTF had caved to political pressure in making the mammography recommendation. Perhaps others know more about that one way or the other.
Now they've backed away from that recommendation, going from a favorable to a negative recommendation for mammograms. The overall issues aren't actually that complex: if you imagine 1000 women in their 40's getting mammograms yearly for 10 years, the best estimates are that you will cause more than half of them to need repeat mammograms for concerning findings, and will cause nearly 200 to get breast biopsies, while only preventing 2 deaths from breast cancer.
What is remarkable is the firestorm that has been set off. When I present these data to women patients in their 40s (in a way that I think is fair to both sides of the argument), I've found that approximately half want mammography and half want to wait until they turn 50. Individual patient values and preferences heavily influence the decision. The USPSTF takes this into account by recommending against routine mammograms in the 40s, but states that a decision to screen should take consider an individual woman's values. All in all, not that unreasonable a position as far as I'm concerned.
I think, though, that the USPSTF's way of grading recommendations is awkward in general and may even have contributed to the confusion over their 2002 and 2009 recommendations. The evidence base didn't substantially change over that time frame, but the recommendations did, and that opens them to attack.
I think the USPSTF would be better off using a different grading system, such as GRADE, which they explicitly chose not to do recently.
(Here I need to make one of the only conflict of interest disclosures I should ever need to mention on this blog: one part of my work is as a medical editor, and in our publication we use GRADE.)
That out of the way, the USPSTF grading scheme is unusual in the way that it combines strength of recommendation and quality of evidence into a single value (a letter grade). To me, this overloading causes problems. For many of the recommendations that USPSTF gives a grade of I to (insufficient evidence), GRADE and a number of other schemes would note the low or moderate quality of the evidence and still allow a guideline committee or other recommending body to make a suggestion about what a clinician should do.
When there is enough evidence to make a suggestion (a "weak recommendation" in GRADE parlance), GRADE explicitly deals with the need for considering patient values and preferences. If, instead of being used to this single letter grade from USPSTF, we were used to seeing evidence and recommendation divided, I'm not sure the recommendation change would have caused quite so much trouble.
The USPSTF recommendation, cast into GRADE style, would likely have read:
For most women ages 40 to 50, we suggest not performing yearly mammography (Grade 2B). Women who are willing to accept a high risk of false positive mammograms and a high risk of unnecessary biopsies in return for a small mortality benefit may reasonably prefer annual mammography starting at age 40.
Would a recommendation written this way have caused the same firestorm? I'd be interested in what others think about this. Would it have caused a firestorm if the 2002 recommendation had said the opposite?:
For most women ages 40 to 50, we suggest performing yearly mammography (Grade 2B). Women who are unwilling to accept a high risk of false positive mammograms and a high risk of unnecessary biopsies in return for a small mortality benefit may reasonably prefer annual mammography starting at age 50.
You can see how easy it is to split the unchanged evidence base from the change in what you are suggesting for the majority of women (based on a guess about their values and preferences) when using GRADE. I think such an approach is more transparent than that used by the USPSTF, and keeps the focus where it belongs in a situation like this -- on the benefits, risks, and burdens that influence the decisions for individual patients and for society.
I'll write more about the methods of GRADE, including its strengths and weaknesses, down the road. I do want to say that while I don't like the grading scheme used by USPSTF, I think they mostly do a wonderful job of staying true to the evidence as they present their guidelines. We should all hesitate to recommend screening tests in the absence of solid evidence showing benefit, and the USPSTF can almost always be counted on in this regard.
The results from ARBITER 6 were released this week.
The interpretation is less clear.
I've been in a minority who has doubted that most drugs that lower LDL are actually beneficial in primary prevention. Statins clearly show benefit, but really no other agents have shown a reduction in all cause mortality or, in primary prevention, in cardiovascular mortality. As such, I'd been raising concerns about ezetimibe even before ENHANCE was published, and have been unconvinced that anyone should be treated with combination therapy to achieve LDL goals, since there is little to suggest added benefit when other lipid-lowering drugs are combined with a statin.
Recently, the Oxford Niaspan Study found that CIMT was improved when niacin was combined with a statin, but CIMT is a surrogate marker and it's not so clear that changes in CIMT will predict clinical benefits. However, of all the non-statin therapies, niacin has been the one that looks most promising for reducing cardiovascular mortality, at least in secondary prevention.
Now, soon after that study, we have ARBITER 6, showing that niacin plus a statin seemingly reduced cardiovascular events compared with ezetimibe plus a statin. There were 2 events in the niacin arm (1%) and 9 in the ezetimibe arm (5%) and this was statistically significant. So, what can we conclude from this?
One possibility is that niacin actually does add some clinical benefit to a statin in secondary prevention, while ezetimibe does not.
Another possibility is that ezetimibe is actively harmful, while niacin is neutral. There were slightly more events in ENHANCE in the ezetimibe arm, but the numbers were tiny.
A third possibility is that this was due to chance. The actual results are quite "fragile" in the sense that only a few events moving from one arm to the other would make the difference disappear. While, in theory, a statistical test captures this, there's been a sense in the EBM world that very small event numbers create a fragility of results not adequately captured by a p value. It seems extremely unlikely that niacin really reduces events by 80%, since this should have been obvious in prior trials of niacin. As such, either niacin combined with a statin is much superior to niacin alone, or these results are in part due to the play of chance. (It's not likely that ezetimibe raises events by this degree when used with a statin, since we would likely have noticed this in other trials.)
ARBITER 6 was stopped early for benefit, based on CIMT results. This seems like an odd decision to say the least, since change in CIMT is not a clinical outcome requiring an ethical decision to stop a trial. The early stopping was unfortunate since with more events we might be more certain about whether the reduction in events was real.
I'll be interested to see how others interpret ARBITER 6. So far I've communicated with one lipid expert (who has not shared my views about non-statin therapy) and he doesn't believe that ezetimibe is clearly to be avoided, while another expert (who does share my views about non-statin therapy) doesn't feel that ARBITER 6 means that niacin should be used. Apparently neither one found ARBITER 6 substantially reshaped their positions on this.
My best guess is that niacin reduces events a bit, though not as much as ARBITER 6 suggests. If so, for the highest risk patients, such as those who have CV events while on a statin, adding niacin might make sense. Given the side effects with niacin (36% of patients in ARBITER 6 had flushing, and many more in the niacin arm than the ezetimibe arm discontinued therapy), I'd hesitate to use niacin in lower risk patients even for secondary prevention, out of concern that they might discontinue both niacin and the statin.
I'd be quite hesitant to treat anyone with ezetimibe at this point. While I was a skeptic that ezetimibe did anything beneficial even before ARBITER 6, I'm now substantially more concerned that it could be causing real harm.
In the future, I'll write more about this subject, since the distinction between expert opinion and clinical experience seems to cause unending confusion to people who write and talk about EBM.
For now, though, I want to write about the second recent event that caused me to decide to blog on evidence and medicine. I caught a snippet on NPR's Science Friday where they were talking about concussions in athletes and heard the following: "And take a very careful step-wise return not just to the playing of the sport, but also to even, you know, text messaging and doing things intellectually that can be harmful to their brains while they're still recovering from a concussion." www.npr.org/templates/story/story.php?storyId=120174333
I was startled to hear that an expert thought that someone should not be texting after a concussion (it turns out this is referred to as "cognitive rest"), and I had a hard time believing there was any evidence for it. And though I have no expertise in concussion management, the idea sounded somewhat contrived or even goofy to me.
It turns out that this mainly comes from consensus guidelines for the management of concussion (PMID 19433429) where, if they have any evidence to back this up, I couldn't find it. I certainly didn't find any published studies showing that cognitive rest improves outcomes after a concussion. If anyone reading this knows of such evidence, I'd be interested to see it.
To make recommendations, experts always need to interpret what evidence there is (expert opinion), and in the absence of high or moderate quality evidence about cognitive rest, the consensus panel presumably had to make their best guess for management. This particular recommendation seems a bit odd in that it has potentially high burdens on the injured athlete (don't read; don't text) in the absence of good evidence, and rest is not always such a good idea after a physical injury -- bed rest for low back pain and immobilization for an injured shoulder are both clearly a bad idea. But ultimately doctors need to either recommend cognitive rest or not recommend it and if some people think it's a good idea an expert recommendation is required.
However, a certain degree of humility would seem to be required when transmitting a recommendation grounded in such low quality evidence. Perhaps this is not such an instance, and I am just missing the better evidence, but it would certainly be common to hear expert panels and individual doctors make forceful pronouncements based on little or no foundation. If there really is nothing much to support this, perhaps the experts describing the recommendation (both in the guideline and on the radio) might have said, "Well, we think cognitive rest is a good idea, but we don't really know. In fact, for all we know, it might do more harm than good."
So, while I'll write more about expert opinion and clinical experience down the road, this episode of seeing yet another consensus guideline treated as gospel was an added push to create this blog.
Although an early participant in the world of Usenet newsgroups, I'm a latecomer to the world of blogging. I don't read or participate in blogs often, and as a newbie will undoubtedly make mistakes of etiquette and function much as I did in the very early days of Usenet before I got a feel for the music of the newsgroups. I hope anyone reading this will take this into account and cut me some slack!
I'd been considering for a while now blogging about evidence and medicine, something I spend a fair amount of my time thinking about in any case, but a couple of events pushed me to take the plunge.
The first was when a relative, a woman in her mid-40's, called to tell me that she'd seen her new primary care doctor. The doctor had run tests on her including checking a lipid profile. The lipid profile showed a total cholesterol of 160, an HDL of 44, and an LDL of 105. Based on these results, the doctor informed her that she should be very concerned about her "low" HDL number and told her to start taking fish oil.
This was an interesting interpretation of the results and an unusual suggestion for management, and it gives some sense of my concerns about how evidence is used, ignored, or misused in medicine. This relative had a low normal blood pressure, no smoking history, did not have diabetes, and had no family history of coronary heart disease. As such, there is even some controversy as to whether she should have had a lipid profile checked at all.
Assuming a lipid profile was worth doing, she should certainly have been told that this was an excellent profile putting her at low risk for a cardiovascular event, and that there was little modern medicine could do to substantially affect her risk. Her 10-year risk of an event (using the 2008 calculator that overestimates vascular risk by including things that probably shouldn't be included -- a blog for another day) works out to about 2.2%. If we pretend that somehow fish oil could increase her HDL to 80 and that she would get all the same benefit as if her HDL were naturally 80, we could drop her calculated risk to about 1.4%. So, if she took medication daily for 10 years, she would reduce her risk of an event (under these outlandishly favorable assumptions) by 0.8%. Not something this relative was interested in doing.
But, do we even know that fish oil raises HDL? The main effect of fish oil on lipids is to lower triglycerides. Most studies suggest that fish oil only raises HDL levels by about 3%, so we might expect an increase from an HDL of 44 to an HDL of 45. It's hard to see how the doctor felt this would fix the woman's "low" HDL.
But, beyond that, we have no solid evidence that raising HDL with medication improves outcomes. Changing HDL with medication is a surrogate outcome for that medication -- that is, it is of no direct interest to the patient unless it also predicts better cardiovascular outcomes, and HDL appears to be a poor surrogate. Some medications that raise HDL appear to cause harm and worse vascular outcomes.
As for whether fish oil supplements are beneficial through some non-HDL mechanism, I think we can safely say that the jury remains out as to whether a woman in her 40s at low cardiac risk could expect any important benefit from fish oil supplementation.
Interestingly, my relative's doctor also checked her vitamin D level and told her that a level of 34 (in the normal range) was concerningly low and that she should start taking 1000 IU of vitamin D per day (she was already taking 400 IU). Vitamin D supplementation is controversial, with lots of people claiming big benefits, but we can safely say for now (again, perhaps a longer blog for another day) that there is no solid evidence that having a woman with a normal vitamin D level start taking additional vitamin D is likely to be of any benefit to her.
My relative asked what I thought these events said about her doctor, but I'm not really sure. I see decisions like this made all the time, where evidence is misunderstood, misapplied, or ignored. I have the sense that this particular doctor may believe she is getting out ahead of the evidence -- suggesting therapies that she believes will be proven down the road to be of benefit. She could, of course, turn out to be right, but we should be very hesitant to give preventive medications without a decent evidentiary basis. Other drugs that raise HDL have increased mortality. Supplementation with other vitamins that were thought beneficial has turned out to cause cancer or increase mortality.
I said there were a couple of events that prompted me to start this blog, but this post seems long enough. I'll talk about other events in future postings.