[An] original study by Mark Hatzenbuehler and his co-authors claims that anti-gay social stigma stably accounts for an average of 12 years of diminished life span among sexual minorities. That claim now seems unfounded, since multiple attempts to replicate the imputed stigma variable yielded no scenarios in which it was significantly associated with subsequent mortality.
The study of social stigma’s influence on health has surged in recent years. Together with five co-authors, Columbia University School of Public Health professor Mark Hatzenbuehler assessed the effect of anti-gay stigma on the mortality of sexual minorities, using data from the 1988-2002 administrations of the General Social Survey (GSS) linked to mortality outcome data in the 2008 National Death Index. This enabled the original study’s authors to explore possible connections between anti-gay stigma and the subsequent deaths of research participants.
Stigma, however, is not simple to define or operationalize, prompting measurement challenges that make it difficult to assess just how influential stigma is on health outcomes. Moreover, few have been able to adequately construct and test a contextual measure of such stigma. Indeed, the original study’s authors noted “little or no variation to study” in previous attempts, given “the pervasiveness of structural stigma” in American communities. To construct the structural stigma variable, the researchers dichotomized and summed the responses to four survey items for each respondent, averaged this value for each primary sampling unit (PSU), and then constructed a dichotomous measure of “high structural stigma” based on this PSU-level average. The authors decided to use the top-quartile threshold (1.77, with a range of 0 to 4), indicating that respondents were considered as living in a community with high anti-gay stigma if its sampled residents offered an anti-gay answer to fewer than two of four questions.
The original study’s authors reported that, after controlling for individual and community-level risk factors, the structural stigma measure was still strongly associated with premature mortality among sexual minorities, displaying a hazard ratio of 3.03 (95% CI: 1.50, 6.13), which translates into an average life expectancy difference of 12 years. For purposes of comparison, 12 years of reduced life span is greater than that found by the CDC among regular smokers when contrasted with nonsmokers. This prompted concern about possible alternative explanations and pathways of influence, and hence an interest in replication.
The replication process was successful in everything except the creation of the structural stigma variable. The original study’s process of imputing missing data for its four key social stigma items bars the way to the successful recreation of the original key predictor variable—structural stigma. Each of the four measures that constitute the original study’s stigma variable exhibits around 40 percent missing values, the vast majority of which is intentional given the GSS design. This is not an inherent problem. However, the magnitude of missing data was not made plainly evident in the original study, save for a passing reference to a “sizable portion of our data.” After efforts to obtain more information from Professor Hatzenbuehler about his team’s decisions concerning the imputation of missing data on the stigma measures were unsuccessful, a variety of focused attempts at replication—including both tweaking and improving the missing data imputation process—were undertaken.
All such efforts, however, failed to generate any report of a strong and statistically significant effect of structural stigma on the premature mortality of sexual minorities. Attempts to replicate the structural stigma measure following what could be called a “best practice” approach, as well as one following the most straightforward reading of the original study’s description, each issued in results that indicated greater numbers of people living in “high” stigma PSUs as well as no effect of that stigma on the mortality of sexual minorities. Eight additional approaches to the imputation of missing data were attempted, including adjusting the threshold for what should be considered a high-stigma PSU. None generated anything like the results reported in the original study. Furthermore, replication estimates appear similar to those generated using “complete cases only,” that is, only those GSS-NDI cases that display no missing values. This further undermines confidence in the original study’s imputation of missing data, and hence its findings. The unavailability of the original study’s programming syntax and the insufficient description of its multiple imputation procedures leave unclear the reasons for the failed replication. Minimally, the findings of Hatzenbuehler et al.’s (2014) study of the effects of structural stigma seem to be very sensitive to subjective decisions about the imputation of missing data, decisions to which readers are not privy.
Measurement and analytic challenges, like those facing studies of social stigma, make it harder to develop broad confidence in conclusions across studies. In his widely disseminated and discussed 2005 study on the poor validity of most published research findings, John Ioannidis highlighted how diversity in research designs, definitions, outcome measures, and ways of analyzing data, as well as the surge in popularity of a particular research topic are all factors apt to elevate the risk of scientific missteps and weaken confidence in published research findings. This, together with the rapid expansion of publication outlets and pressure to publish, has contributed to a surge in scientific overstatements, errors, accusations of fabrications, and the issuing of errata or retractions, as well as a renewed call for greater transparency across the research process. While the original study is not unique in its lack of details about its analytic procedures, future efforts ought to include supplementary (online) material enabling scholars elsewhere to evaluate and replicate studies’ central findings. This would enhance the educational content of studies as well as improve disciplinary rigor across research domains.
In sum, the original study by Mark Hatzenbuehler and his co-authors claims that anti-gay social stigma stably accounts for an average of 12 years of diminished life span among sexual minorities. That claim now seems unfounded, since multiple attempts to replicate the imputed stigma variable yielded no scenarios in which it was significantly associated with subsequent mortality.