Post a Comment On: Vigilance

There has been quite a bit of hullaballoo in the past few days about a Heritage Foundation study that reanalyzed data from the National Longitudinal Study of Adolescent Health (Add Health), a comprehensive survey of 90,000 seventh through twelfth graders. An earlier report on the data set by Hannah Brückner at Yale and Peter Bearman at Columbia, published in the The Journal of Adolescent Health had found that teens who pledged to remain abstinent until marriage had the same rates of STDs as those who did not. Yesterday's New York Times:
Challenging earlier findings, two studies from the Heritage Foundation reported yesterday that young people who took virginity pledges had lower rates of acquiring sexually transmitted diseases and engaged in fewer risky sexual behaviors.

The new findings were based on the same national survey used by earlier studies and conducted by the Department of Health and Human Services. But the authors of the new study used different methods of statistical analysis from those in an earlier one that was widely publicized, making direct comparisons difficult.

Independent experts called the new findings provocative, but criticized the Heritage team's analysis as flawed and lacking the statistical evidence to back its conclusions. The new findings have not been submitted to a journal for publication, an author said. The independent experts who reviewed the study said the findings were unlikely to be published in their present form. Studies Rebut Earlier Report on Pledges of Virginity
The Times' reporting on this is a model of ... whatever. They are balanced, I guess you could say.

So the reader is left with two analyses of the same data set that come to opposite conclusions. The ordinary person is going to just shake their head, muttering something about statistics, and move on to the funnies.

I spent some time looking at this, and I admit, the key to it slipped past me. It was just a little thing, but the sleight-of-hand was good, and I was looking the wrong way.

Here -- read what this blog, "Big Monkey, Helpy Chalk," says about it: A big red flashing sign saying "This science is completely bogus". It's a good, readable discussion about "junk science" -- which the Heritage Foundation guys accuse the academics of -- and how to tell the difference.

There are a couple of keys. First of all: peer review. The first paper was sent to a legitimate scientific journal. The editor of that journal sent it around to some reviewers, generally experts in the field. They are competitive, and hate it when some other researcher finds something before they do, so they criticize the paper to death. The author gets the comments, addresses them through changes to the manuscript, re-submits it ... sometimes this goes on for several cycles. Well, usually not -- most papers are rejected in the first round. But this one survived, and finally reached publication.

The Heritage Foundation paper, on the other hand, went from the conservative think-tank directly to the press.

That's your first clue.

The NYT and Big Monkey agree that a significance level of 0.10 is too high, and it is, but that's quibbling.

The tricky part is in the way the two studies defined STDs. Here's where the data came from for the original Brückner and Bearman study:
Biomarker data (urine samples) on STD status were collected from 92% of wave 3 respondents. A total of 1183 individuals (8%) refused participation in the biospecimen collection. Urine samples were collected in the field and analyzed for the presence of three sexually transmitted diseases, Chlamydia (CH), Gonorrhea (GC), and Trichomoniasis (TR). In addition, 7000 female respondents who reported ever having had vaginal sex in wave 3 were randomly selected for Human Papilloma Virus (HPV) assays. The specific procedure and tests used are described in detail in [5]. Both males and females were tested for the three bacterial STDs regardless of their sexual activity status.

Collection procedures followed a strict protocol; samples that arrived in a condition not suitable for testing were discarded. Between 4% (CH), 5% (TR), and 10% (GC) of the samples were not tested for these reasons [5,6]. Pledgers do not differ from others with respect to whether their samples yielded results in the testing (p = 0.32 for CH, p = 0.17 for TR, p = 0.20 for CG). Pledgers did not differ from others in the extent to which they refused to provide urine
samples (p = 0.28). Data are weighted to adjust for oversampling of various groups and wave 3 nonresponse.

...

Unless otherwise noted, the analyses reported below are based on 11,471 respondents with valid data on STD status and grand sample weights. The majority of the results are derived from cross-tabulating pledge status with various outcome and behavioral measures.

...

HPV is analyzed separately because it is much more prevalent, may be transmitted by noncoital sexual behavior, and the testing was based on a different sample (n = 3317; sexually active females only). After the promise: the STD consequences of adolescent virginity pledges

I'll come right back to the STD measurement, but first...

Note that one criticism that the Heritage Foundation researchers had of the Brückner and Bearman study was, as they put it:
The centerpiece of their argument about pledgers and heightened sexual risk activity is a small group of pledgers who engaged in anal sex without vaginal sex. This "risk group" consists of 21 persons out of a sample of 14,116. Bearman and Bruckner focus on this microscopic group while deliberately failing to inform their audience of the obvious and critical fact that pledgers as a whole are substantially less likely to engage in anal sex when compared to non-pledgers. Virginity Pledgers Have Lower STD Rates and Engage in Fewer Risky Sexual Behaviors
First of all, it is not in any sense a "centerpiece" of Brückner and Bearman's paper. These sentences are tucked in the middle of a paragraph reporting a ton of statistical results:
... Similarly, 0.7% of nonpledgers report anal but no vaginal sex, compared with 1.2% for pledgers. Although too few females report anal but no vaginal sex, for males we find a significant difference between pledgers and nonpledgers (p = .021). Specifically, slightly more than 1% of male nonpledgers report anal sex but no vaginal sex, compared with almost 3% for inconsistent pledgers and 4% for consistent pledgers...
Let me reconstitute the totals here. 0.7 percent of 9,072 nonpledgers would be 63.5, call it 64 nonpledgers; 1.2 percent of 707 pledgers would be 9.3, call it nine of them. There is some ambiguity here -- if "pledgers" means all respondents who reported pledging, whether they were consistent over time or not, then that would be 0.012 percent of 2,399, or about 28.788 pledgers who had anal and not vaginal sex. So we're talking about 93 people, not 21. Whatever, the difference between 1 and 4 percent is significant in this sample, it doesn't matter that the percentages were small. They are certainly higher than the percentage of people who die from head injuries in automobile crashes, but we accept the conclusion that seatbelts significantly reduce that percentage.

They are counting on you not doing what I'm doing, that is, finding the original paper and doing the arithmetic.

We have read how Brückner and Bearman defined STDs in their paper. Now listen to the Heritage Foundation guys describe it:
One problem is that Bearman and Bruckner examined only one of several STD measures available in the Add Health data file. Analysis of the remaining measures reveals that adolescent virginity pledging is strongly associated with reduced STDs among young adults. These results are statistically significant in four of the five STD measures examined and are very near significance on the fifth measure. With all the STD measures, the allegedly ineffective virginity pledge is actually a better predictor of STD reduction than is condom use. Virginity Pledgers Have Lower STD Rates and Engage in Fewer Risky Sexual Behaviors
They make it sound like Brückner and Bearman ignored the other STD data. In reality, their Table 4 is just that. And it shows just what both groups say it shows: people who have pledged to be abstinent report having fewer STDs, and they report going to the doctor fewer times to be checked for STDs.

The Heritage Foundation reports cover up the differences between their data and the academic data, and as a smokescreen they assert that Brückner and Bearman looked at "only one of several STD measures available in the Add Health data file."

Brückner and Bearman analyzed real data from real urine tests. People peed in the cup and it went to a lab. It didn't matter what the respondent said, the laboratory measured whether there were signs of infection in the urine. The Heritage Foundation guys relied only on what the respondent said. According to Brückner and Bearman:
Add Health asked respondents also about their experiences with STDs. Specifically, all respondents were asked whether they had been diagnosed with various STDs in the past year; whether they had ever seen a doctor because they were worried about having a STD; and whether they had been tested for various STDs in the past year.
And those are the other STD variables that the Heritage guys like so much. No sensible person is going to think that someone's answers to those questions are more accurate than the lab results.

In sum, the Heritage Foundation papers are political, not scientific. They do not meet any of the standards of scientific literature. They are not peer-reviewed, they contain scurrilous ad hominem comments about other researchers, they conceal the nature of the data they used while implying that the other researchers' methods were inadequate. The Heritage Foundation wants abstinence pledges to succeed because it fits their ideology, and they will twist any data in whatever way is necessary to prove their point.

It's a sad statement, that science is under attack in this way, that any nut can put a paper on the Internet and call the newspapers, and people will accept it as if it were the real thing.