Caveat lector

Our current howler (part I): Playing the margin

Synopsis: Gore led Bradley in all three polls. At the Post, that meant they were even.

Bradley Pulls Even in N.H.
Dan Balz, The Washington Post, 9/19/99

Gore slips in New York (Inside Politics)
Greg Pierce, The Washington Times, 9/16/99

More proof (Inside Politics)
Greg Pierce, The Washington Times, 9/15/99

Republican road to White House suddenly bumpy
Jill Lawrence, USA Today, 9/16/99

Moynihan’s defection to Bradley is latest gore
Susan Page, USA Today, 9/24/99

Bradley Closes Gap With Gore
John Harwood, The Wall Street Journal, 9/16/99

Here at THE HOWLER, we have no idea who's going to win the Democratic nomination. We don't know who's going to win in New Hampshire. We don't maintain a polling unit. All we know is what we read in the newspapers.

But our analysts really jumped to attention at a page one headline in the September 19 Post. "Bradley pulls even in N.H.," said the headline, sitting at the top of the big fat Sunday paper.

Our analysts were surprised because, at the time, they knew of no poll that had shown the New Hampshire race even. And they couldn't help chuckling when they read the data contained in the Dan Balz report.

Balz reported on three separate polls of likely New Hampshire Democratic voters. All three polls showed Al Gore ahead of rival Bill Bradley, by margins of four, five and seven points.

The polls were closer than earlier Granite State polls, in which Gore had enjoyed a much healthier lead. But how had the Post said the race was even when Gore was ahead in all polls?

The answer? It's that persistent ol' debbil, "margin of error," the device routinely used to make races seem tighter—when the press corps wants a race nice and tight. The truth is, when three separate polls showed Gore ahead by four to seven points, the odds were extremely slim—very slim indeed—that the race was actually tied. But don't tell that to the Washington press corps, which never met a technical measure that it can't misapply. No, nothing much will ever turn on the error lodged in the Post headline that morning. But we thought it might be worth taking a day to enjoy a good chuckle at "margin of error," before going on to examine more significant aspects of recent poll/survey reporting.

The Post headline reflected an increasing trend among political reporters. When the difference between two hopefuls is less than a poll's "margin of error" (perhaps doubled), writers declare the race a "statistical tie." Before the Post headline on the 19th, the trend had been widely exhibited in reporting on New Hampshire/New York polling. Here's Greg Pierce on September 16:

PIERCE (9/16): In a match-up with Republican front-runner George W. Bush [in New York], Mr. Bradley led 47 percent to 37 percent. Mr. Gore, meanwhile, was in a statistical tie with the Texas governor; 46 percent for the vice president and 43 percent for Mr. Bush.

This formulation has been seen again and again in recent polling stories. It helps explain why others, besides the Post headline writer, had declared Bradley to be even in New Hampshire. Jill Lawrence, on September 16:

LAWRENCE: Bradley, the former New Jersey senator and former professional basketball player, is showing increasing strength in key primary states. Polls show him in statistical ties with Gore in New Hampshire and New York.

The New Hampshire polls to which Lawrence referred were the same polls described in the Post's later story. Susan Page seemed to be a bit more careful with her language, but seemed to imply the same thing:

PAGE: In three recent statewide polls, Bradley has pulled neck-and-neck with Gore in New Hampshire...A Marist Institute Poll conducted last week out the two men in a perfect 42%-42% tie in New York.

Apparently, Page discerns two kinds of "ties;" "perfect" ties, where the numbers are even, and other "ties," where they're not.

Again, without getting into the actual stats, when three separate polls showed Gore four to seven points ahead, it was very unlikely that the race was tied. (It's harder to say that from one single poll.) But let's focus on one particular poll—a WMUR-CNN poll released on September 13. Here's how Greg Pierce described it:

PIERCE (9/15): A WMUR-CNN poll released Monday showed Mr. Gore is favored by 46 percent of likely voters in the 2000 Democratic presidential primary, compared with 41 percent who support Mr. Bradley. That was within the poll's margin of error of 5 percentage points.

Was that poll a "statistical tie?" It's not clear what the term even means. It is certainly possible that, had all Dems been polled, the result would have been a tie. A single poll can't prove that a survey of all Dems would have broken down 46-41 for Gore. But, while it is possible that Gore and Bradley were actually tied, it is equally possible—repeat, equally possible—that Gore was actually in a ten-point lead. ("Margin of error" doesn't work just one way.) But we defy you to find any writer who reported these results as a "statistical ten-point lead" for Gore.

Nope—the rule is, make the story more thrilling. So the press corps routinely cites one possibility while failing to mention the other. In doing so, it misapplies the concept of "margin of error," and gins up a term—"statistical tie"—that would be hard to explain or defend in these contexts.

By the way, some writers have avoided these undesirable impulses, reporting these polls in simple, plain language that doesn't pretend to be technical. Here, for example, was Pierce's lead paragraph on that WMUR poll:

PIERCE (9/15): Bill Bradley, boosted by his strength among independent voters, is in a close race with Vice President Al Gore in the Democratic presidential race in New Hampshire, says a new poll that echoes the results of two recent surveys.

Is 46-41 a "close race?" That, of course, is a judgment call, but Pierce didn't pretend that the hopefuls were now somehow known to be "tied." And John Harwood, in the Journal, made the whole thing look easy. Here's how he described the New Hampshire polls, in a story concerning a new nationwide survey:

HARWOOD: The [nationwide] survey, on the heels of recent polls showing Mr. Bradley drawing close to Mr. Gore in the critical primary state of New Hampshire, suggests the ex-basketball star is moving into a strong position to capitalize on an early breakthrough in the coast-to-coast crush of primaries next month.

Indeed, that is exactly what the three polls most likely reflected—Bradley now was close to Gore (and improving his position). They did not show that Bradley was "even" or "statistically even," although excitable scribes wanted to say so.

One last point: despite the headline in the September 19 Post, Balz had not said that Bradley was even. Here is what he had actually said:

BALZ: Five months before the nation's first primary, former New Jersey senator Bill Bradley has virtually pulled even with Al Gore in New Hampshire, turning a lopsided advantage for the vice president into a fiercely competitive contest.

Later in his article, he referred to the "narrowing" of the New Hampshire race. The headline misstated his writing. Yep—as Balz's writing showed (overlooking the headline), it's possible to describe close poll results without goosing them up with references to "statistical ties." The WMUR-CNN poll had shown a "statistical" five-point lead. Scribes would be better off simply saying it.

Tomorrow: A recent Pew survey revealed a Big Fact that the press corps dare never utter.

Has anyone here seen Fred: We remain concerned about Fred Wertheimer, still missing from Ceci Connolly's finance stories. On Saturday, Connolly penned a story in the Post about Gov. Bush's successful fund-raising. Connolly reported that Bush "is on track to bring in more than $75 million for the primary campaign," and may even skip federal funds for the general election. Realistically, that would mean that Bush would have to raise around $150 million overall.

In our view, there's nothing wrong with that—more power to him, if the governor can swing it. But remember how upset Connolly was, back in April, at the thought that Al Gore might raise around $35 million? The good-guv scribe was worried sick that a hopeful would dream of raising such sums. So she called in Fred Wertheimer to fire off quotes about the problems with amassing such lucre.

In her Saturday piece, Connolly imagines Bush raising five times as much, and not a word is said about any troubling implications. Has Connolly lost Fred Wertheimer's number? THE HOWLER is sick with concern.