Relative risk is the ratio of a frequency in one group of individuals divided by the frequency in another group of individuals (or controls). It functions as a multiplier of the probability that an event will occur in one situation, relative to the probability that the same event will occur in another situation. If the probability, say, of a regular ticket winning a lottery is 0.001% and a premium ticket has a 0.002% probability of winning, the relative “risk” of winning with a premium ticket, compared to a regular ticket, is 0.002%/0.001%=2. Stated another way, if the relative “risk” of winning with a premium ticket is 2, as compared to a regular ticket, the “risk” of winning with a premium ticket, although still extremely small, is 2 times that with a regular ticket, or (2)(0.001%)=0.002%.1
Estimates of risk can be of any size. On the other hand, small estimates of risk are probably methodological noise, no matter what statistical “significance” is calculated for them. No matter how carefully investigators believe that they have considered all possible errors that could have occurred, actually demonstrating successful consideration is difficult, and questions always remain. The smaller the estimate of relative risk to more rigorously freedom from errors must be demonstrated. At some point, it becomes impossible to reduce the errors sufficiently, and the estimate of relative risk becomes meaningless. Viewed the other way around, the larger the relative risk, the less likely it is that factors such as bias or confounding could have overcome a true association. No matter how large the relative risk, however, cause and effect can never be proven by a relative risk, alone.1
Calculated confidence intervals can only take into consideration purely random variations in data. In observational studies, almost invariably, however, nonrandom errors in the data will be more important than random variations. The most important criteria for evaluating relative risk are a very high calculated risk and a biologic mechanism that is highly plausible. Most epidemiologists insist upon a relative risk of 3 or 42–not the puny 1.43 to 1.64 relative risks that Nissen and Wolski claimed3 (and which were inflated,4 at that). Even The New England Journal of Medicine‘s Marcia Angell is quoted as saying that, before accepting a paper to be published, “As a general rule of thumb, we are looking for a relative risk of three or more, particularly if it is biologically implausible or if it’s a brand-new finding.”3 So why did The New England Journal of Medicine accept Nissen and Wolski’s “brand-new finding” that had no biological plausibility and whose relative risks were only half of the Journal‘s minimum requirements? There is more. The Food and Drug Administration’s Director of Drug Evaluation, Robert Temple, is quoted as saying, “My basic rule is that if the relative risk isn’t at least three or four, forget it.” Why, then, did the Food and Drug Administration not do exactly that (forget it) with the Nissen and Wolski publication?
What was the point about brand-new findings, above? Avoiding the mathematics involved, from Bayes theorem, we know that the probability that a hypothesis is true after an observation (such as Nissen and Wolski’s meta-analysis) is strongly dependent upon the probability that it was true before that observation. Therefore, meta-analyses and studies that attempt to confirm hypotheses from prior studies are more likely to generate true findings than are studies that produce the hypotheses in the first place.1
The fraction of risk that can be attributed to a factor (such as rosiglitazone) can be calculated as the attributable risk: Attributable Risk=(100%)(Relative Risk-1)/(Relative Risk). If the relative risk is 2, then 50% of the risk is attributable to the factor. Relative risk must be greater than 2 for more than 50% of the risk to be attributable to the factor and, more likely than not, to have caused the event.1 That standard, used in litigation, is extremely low, because being wrong in either direction is considered to be equally bad. Even Nissen and Wolski’s inflated relative risks, however, mustered (at most) a 39% attributable risk–not even rising to the meager “greater-than-50%” standard, let alone being overwhelming evidence against rosiglitazone (Avandia, Avandamet, Avandaryl, GlaxoSmithKline). Ironically, Peto, whose methodology Nissen and Wolski improperly applied to their meta-analysis, stated, “. . . when relative risk lies between 1 and 2 . . . problems of interpretation may become acute, and it may be extremely difficult to disentangle the various contributions of biased information, confounding of two or more factors, and cause and effect.”5
To be sure, the studies on which Nissen and Wolski based their meta-analysis were not purely observational. The study subjects had been randomly assigned to groups. However, collection of data about cardiovascular events was not systematic, because they were not prespecified endpoints. In that sense, a study of the “observations” constitutes an observational study. For that case, maybe a relative risk of 3 or 4 is too high a standard, but a compromise of at least 2 might be reasonable.
Nissen and Wolski’s meta-analysis was only hypothesis-generating; the relative risks were inflated by their improper use of statistical techniques; even without systematic biases, the inflated relative risks were still within the ranges expected by chance; however, there were systematic biases against rosiglitazone; even if the relative risks could have been accepted at face value, they would still have been of dubious validity; and the results were not biologically plausible. As I have already pointed out, in my December 19, 2013 post, “Nissen, Wolski, and How Not to Do a Meta-Analysis,” Nissen and Wolski’s meta-analysis also suffered from multiple subgroup comparisons, seriously further inflating the probability of a statistically “significant” result.1 In sum, apart from being essentially meaningless, their meta-analyses was otherwise excellent.
© 2013 Myron Shank, M.D., Ph.D.
1 Nicolich Mark J., Gamble John F. What is the Minimum Risk that can be Estimated from an Epidemiology Study? In: Moldoveanu Anca Maria. Advanced Topics in Environmental Health and Air Pollution Case Studies InTech, 2011.
5 Doll Richard, Peto Richard. The Causes of Cancer, Oxford-New York: Oxford University Press, 1981, p. 1219. As quoted in Nicolich Mark J., Gamble John F. What is the Minimum Risk that can be Estimated from an Epidemiology Study? In: Moldoveanu Anca Maria. Advanced Topics in Environmental Health and Air Pollution Case Studies InTech, 2011.