In my post of January 27, 2014, Concerns at FDA: Nissen’s Avandia Analysis, I discussed concerns expressed by Deputy Director Robert Temple, in a memo to Janet Woodcock, Director of the Food and Drug Administration’s Center for Drug Evaluation and Research: Nissen and Wolski’s2 peculiar and unblinded1 selective choices from those studies that qualified for their meta-analysis,3 their potential bias in doing the analysis,1 and their (improper) inclusion of DREAM (Diabetes REduction Assessment With ramipril And rosiglitazone Medication) after using it to develop their hypothesis.4 Dr. Temple concluded that “there was potential bias in its development (or at least the analysis was done with knowledge of what would emerge).” Then, in my January 29, 2014 post, More Concerns With Nissen’s Avandia Analysis, I discussed some of Nissen and Wolski’s2 improper use of statistics related to their multiple comparisons between those who were treated with and those who were treated without rosiglitazone (Avandia; GlaxoSmithKline).
Nissen and Wolski did not consider the combined endpoint of acute myocardial infarctions plus cardiovascular deaths, attributing this to uncertainty about the possibility of counting the same event twice.2 However, for any reasonable assumption about the occurrence of both acute myocardial infarction and cardiovascular death in the same patient, the effect would be much less than the problem of multiple comparisons, which they ignored.
Speaking of multiple comparisons, Steven Nissen’s “Flaws, Bias, Misinterpretation and Fraud in Randomized Clinical Trials” showed that he was aware of the problem, but, although he used his own meta-analysis2 as an example to impugn another paper, Dr. Nissen hypocritically omited mention that he and Kathy Wolski2 were guilty of the very same multiple comparisons that he decried in others.5
“Stroke is not mentioned at all” by Nissen and Wolski, “a strange omission as almost all CV [cardiovascular] event trials include it.”1 Nissen and Wolski might have lacked the data, but, if so, they “apparently, did not seek it.”1 Ironically, Nissen and Wolski2 claim to have used Peto’s method in their meta-analysis,2 but “Peto, for example, has emphasized the need to find critical missing data (strokes, for example, or time of events) . . . .”1 Then again, as the Food and Drug Administration’s own analyses have shown, “rosiglitazone treatment was in fact associated with fewer strokes.”1
Since Nissen and Wolski did not believe that they had the data to combine acute myocardial infarctions and did not evaluate strokes, it is no surprise that they did not address major adverse cardiac events1 (acute myocardial infarctions plus strokes plus cardiovascular deaths), either.
Of course, acute myocardial infarctions can cause cardiovascular deaths, although not all cardiovascular deaths are caused by acute myocardial infarctions (Some, for example, are caused by strokes, which were curiously not even considered by Nissen and Wolski.). Unless just one of these outcomes was counted per person, the number of independent observations (in statistical jargon, “n”) will also be exaggerated, giving us too much confidence in the apparent differences between groups. Nissen and Wolski mentioned this problem, with respect to the measure, acute-myocardial-infarctions-plus-cardiovascular-deaths,2 but the same problem would also have affected the techniques of multiple stage statistical testing (See More Concerns With Nissen’s Avandia Analysis, January 29, 2014.), had actually used such techniques for their multiple comparisons. Once again, however, for any reasonable assumptions, the potential magnitude of the error would have been far less than the errors introduced by not correcting for multiple comparisons.
The Food and Drug Administration’s own meta-analysis of fifty-two studies6 included all of the measures that Nissen and Wolski ought to have included. In the meta-analysis of fifty-two studies, acute myocardial infarctions were “nominally statistically significant,”7 strokes trended “favorably for rosiglitazone,” neither cardiovascular nor all-cause mortality was even close to being “nominally statistically significant,”7 and the major adverse cardiovascular events category was not statistically significant. Unlike Nissen and Wolski,2 The Food and Drug Administration at least acknowledged the impact of multiple comparisons: “Any sort of correction for multiplicity leaves the finding well short of nominal statistical significance.”1,7
Nissen and Wolski’s2 “strange omissions” would have been bad enough, but there was plenty more.
© 2014 Myron Shank, M.D., Ph.D.8
1 Temple Robert. Memorandum to Janet Woodcock: Data on Rosiglitazone, August 8, 2010. http://www.fda.gov/downloads/drugs/drugsafety/postmarketdrugsafetyinformationforpatientsandproviders/ucm226066.pdf
3 Schachtman Nathan A. Learning to embrace flawed evidence–the Avandia MDL’s Daubert opinion. Schachtman Law January 10th, 2011. http://schachtmanlaw.com/learning-to-embrace-flawed-evidence-the-avandia-mdls-daubert-opinion/.
4 To review, I observed that “It is never appropriate to test a hypothesis with the same data that was used to generate it.” For the statistically literate, I pointed out that excluding the statistical “degrees of freedom” that were improperly included from DREAM would, by itself, eliminate even the appearance of statistical significance. I also noted that this was inexcusable from Dr. Nissen’s co-author, Kathy Wolski, since she is a statistician.
5 Steven Nissen. Flaws, bias, misinterpretation and fraud in randomized clinical trials. http://spo.escardio.org/eslides/view.aspx?eevtid=48&fp=3132
6 In other words, twenty-four percent more than the Nissen and Wolski meta-analysis.2
7 The phrases, “nominally statistically significant” and “nominal statistical significance” may refer to a “p” value of less than 0.05 or to the fact that an apparently statistically significant “p” value does not take into consideration complicating factors, such as multiple comparisons. In the source memo, it is not always clear from the context whether the former, the later, or both meanings were intended.
8 I served: in the speaker program for Takeda Pharmaceuticals America, Inc. and Eli Lilly and Company, Chicago, Illinois, August 7, 1999; as a consultant for Takeda Pharmaceuticals America, Inc. and Eli Lilly and Company, “Current Update and Discussion of the Glitazone Class of Oral Antidiabetics,” January 3, 2001; on Avandia Regional Advisory Panels in Cleveland, Ohio (February 4, 1999), Toledo, Ohio (September 15, 1999), San Juan, Puerto Rico (February 2000), and Beverly Hills, California (April 2000); as a preceptor for SmithKline Beechum, Lima, Ohio (June 23-25, 1999); and in a “Meet-the-Specialist,” SmithKline Beecham, Lima, Ohio, (2000). In 2000, SmithKline Beechum merged with GlaxoWellcome, to form GlaxoSmithKline. Before it became impractical to continue prescribing rosiglitazone (Avandia, Avandamet, Avandaryl; GlaxoSmithKline), I prescribed both it and pioglitazone (ACTOS, ACTOSPlusMet, DuetAct; Takeda).