WHERE ARE STANDARDS? Uh-oh! The NAEP is setting its standards too high, Gerald Bracey intriguingly says: // link // print // previous // next //

WHERE ARE STANDARDS: On Friday, we discussed Ross Wiener’s op-ed in last Monday’s Post—a piece which explained why low-income kids are still being “left behind” in our schools (see THE DAILY HOWLER, 1/6/06). Principally, Wiener said that low-income kids aren’t getting their share of experienced teachers. But he also argued that various states have been setting their statewide standards too low. In this passage, Wiener refers to the performance of low-income kids on last year’s NAEP testing program:

WIENER (1/2/06): One thing put in stark relief is the low level of state standards. Students who demonstrate proficiency on their own state's tests often perform far below that level on NAEP, suggesting that the states have set standards too low to indicate adequate academic preparation.

As we said on Friday, we’ve never understood the logic of Wiener’s position as it applies to low-achieving kids. That is, we’ve never understood why raising standards would help a kid who can’t come close to meeting the bar which already exists. But in Saturday’s Post, educational researcher Gerald Bracey challenged Wiener in a different way. Are some states setting their standards too low? Actually, NAEP is setting its standards too high, Bracey says in a letter:

BRACEY (1/7/06): Ross Wiener writes that kids do well on state tests but not on the National Assessment of Educational Progress (NAEP). Therefore, he concludes, states have low standards. Actually, NAEP standards are too high. The Government Accountability Office, the National Academy of Sciences, the National Academy of Education and the National Center for Research on Evaluation, Standards and Student Testing have all studied NAEP achievement levels and found them wanting.

As he continues, Bracey quotes the National Academy of Sciences: "NAEP's current achievement level-setting procedures remain fundamentally flawed, appropriate validity evidence for the cut scores is lacking, and the process has produced unreasonable results.”

We suggest that you read the entire letter. (Below, we’ll link to a fuller presentation by Bracey.) But here’s an example of what Bracey means when he says that NAEP’s standards are “too high:”

BRACEY: What does the National Academy of Sciences consider an "unreasonable result"? Just this: In the Third International Mathematics and Science Study, American fourth-graders ranked third in science among the 26 participating nations. Yet only 30 percent scored proficient on the NAEP's science assessment. Only 30 percent of our world-beating kids are proficient? Unreasonable. Ludicrous.

For the record, the study to which Bracey refers took place in 1996.

It’s easy to see why Bracey would say that the NAEP standard was “too high” in this instance. We wouldn’t use that language ourselves; we’d be inclined to say that the standard was artificially high. (Or: Misleadingly high.) To explain why, we’ll now toss in a point of general logic.

When we set the standard for “passing” a test of this type, it always involves subjective judgments. What sorts of things must fifth-graders do to be judged “proficient” in reading (or math)? We can try to tie our answer to traditional notions of “grade level” work. But those traditional notions of “grade level” work weren’t taken from tablets which Moses delivered. Presumably, conventional notions of “fifth grade work” are based on observations of past fifth grade kids. But there has never been a time when fifth graders all functioned at the same level. To what level should a fifth grader be held? Inevitably, that’s a matter of judgment. And oh yes! Wherever you end up setting the bar, it will be far below where some kids are working—and far above the functioning level of other fifth-grade kids.

Remember, kids are different! Consider reading, for example. In America today, many fifth graders can handle books written on traditional sixth or seventh grade level (or beyond). And many fifth graders are functioning years below grade level—or are barely able to read at all. What must fifth graders do before we say that they are “proficient?” Inevitably, that’s a subjective judgment. We can set the bar as high (or as low) as we want. In the end, there’s no “right” answer.

In 1996, was the NAEP standard “right?” To that, there can be no ultimate answer. But it’s easy to see why someone might say that the standard was artificially high. More specifically, it’s easy to see a problem which may have resulted; citizens may well have been misled by the gloomy results of that year’s NAEP test. Today, are we being misled, in some similar way, by the standards set by the NAEP? Bracey raises an excellent point—a point which should be explored further.

BRACEY’S FULLER PRESENTATION: In his letter, Bracey cites a September article in which he questioned those high NAEP standards. To read that full piece, just click here.

BUT HOW DO WE MEET HIGHER STANDARDS: Will raising standards help struggling kids—kids who are reading far below level? As a general matter, we’ve never quite understood the logic of that supposition. When we taught fifth-grade kids reading on second-grade level (who might be sixth or seventh grade by age), we understood that they weren’t meeting standards. What we didn’t necessarily understand was how we could help them do better.

On Friday, we posted Wiener’s prompt response to a question we sent him about this matter. In retrospect, we thought his reply deserved more review. Let’s recall what he said:

WIENER E-MAIL: Expectations play a critical role in determining what students will be taught, and what they will learn. If the standards are pegged to low expectations, teachers (and the parents and students themselves) may be lulled into thinking they're doing well enough—or very close to it, which inevitably will be seen as good enough for some—even when skills/knowledge are not close to adequate. Ideally, systems for teacher training, curriculum design and evaluation, formative assessments, etc., should be built around the standards—that is, what students are expected to learn. If we start out aiming for mediocrity or worse, we might be giving ourselves a pat on the back when we reach abysmally low levels of achievement.

Research documents that students learn more when they are put into more academically rigorous courses; this is true even for students who had been at the lowest levels of achievement in the past. In fact, students who previously were in the lowest quartile for achievement failed less often when they were put in college prep courses, compared to previously similarly performing students who were put into remedial courses. Standards can help push teachers to teach to higher levels (or not).

Wiener’s specific example involves high school students. We wouldn’t doubt that the students described might be better served by that college prep class—as opposed to some “remedial course” which may be a glorified storage facility. But as a general matter, we don’t see why low-scoring fifth graders (or their teachers) will be helped by raising the bar on state tests. In our teaching days, we knew that we weren’t making the grade. We certainly didn’t “think we were doing well enough.” We just didn’t know how to do better.

Of course, Wiener’s example isn’t necessarily exhaustive; there may be other research which suggests that “raising the bar” helps low-achieving fifth graders. But here’s a shortcoming we often observe with theorists of the “standards movement.” Such analysts suggest this: If we put more pressure on teachers (through higher standards and public accountability), they will respond by teaching better. But what should these teachers actually do to help their students meet higher standards? On that, these theorists often have no suggestions. Quite often, they demand that schools produce better results, without ever telling them how.

It’s amazingly easy to “raise the bar” without explaining how to get over it. Mordantly grumbling, we sometimes say this: “Nice work—if you can get it.”