PILOT PROGRAM! Good news! The Virginian-Pilot is trying to learn what its states passing rates really meant: // link // print // previous // next //
WEDNESDAY, MARCH 15, 2006
TIERNEY TOMORROW: Tomorrow, well review John Tierneys recent column on charter schools. Today, though, were sticking to that continuing story. And good news! The Richmond Times-Dispatch and the (Norfolk) Virginian-Pilot have both been on the case. Good news! Virginians deserve to be fully informed about this important matter. For basic links to past work, see below.
Continuing story: Yes, Virginia!
TIMES-DISPATCH REPORTS: Good news! On Tuesday, the Richmond Times-Dispatch filed a short report about Virginias missing report cards. State report cards inaccessible on site, the headline said. Heres the complete dispatch:
RICHMOND TIMES-DISPATCH (3/14/06): The 2005 Virginia School Report Cards cannot be accessed on the Virginia Department of Education's Web site because of technical problems, a department spokesman said yesterday.The cards cant be accessed because of technical problems? Its hard to see how that could be accurate. As we noted yesterday, the Virginia report cards can be accessed—if you know what link to click. (Just click here, for example.) They just cant be accessed from the states web site, where people would normally go to find them. At the web site, the state continues to say theyre temporarily unavailable. We dont know why the link to this material has been disabled. But its rather hard to believe that the answer is technical problems.
As weve said, we dont know why this basic information has been removed from public view. But the explanation received by the Times-Dispatch does seem a bit hard to credit.
PILOT PROGRAM: In the past two days, weve heard from personnel at both the Times-Dispatch and the (Norfolk) Virginian-Pilot; each newspaper has a reporter or two trying to sort out this ongoing story (basic links below). For what its worth, each paper also seemed to know, in the past, that the passing rates on the states school report cards included some kids who had been retested. The papers had apparently decided not to report on this matter—and we havent found anyone yet who can really explain the states procedure in a definite way.
Example: What produced Maury Elementarys high passing rate in Reading/Language Arts last year? (From now on, well just call it reading.) Did 5 out of 19 Maury third-graders pass the states reading test last spring? Or was it really 17 out of 19? The schools report card presents the first passing rate at one point (27 percent); but at another, much more prominent place, it says the overall passing rate for grades 3 and 5 was 92 percent. According to Jay Matthews February 28 report, this high passing rate derived from these figures: 17 out of 19 third-graders passed, along with 22 out of 24 fifth-graders.
So what exactly was the procedure which turned that 5 out of 19 third-graders into that much higher score?
Yesterday, we spent a fair chunk of time on the phone with a reporter for the Virginian-Pilot. Her initial interpretation was this: The 12 third-graders who failed the reading test last spring were retested at some point this year, as fourth-graders. (Note: She wasnt sure about this.) That was not what we took Alexandria testing director Monte Dawson to have told us in his e-mails. Here, for example, is the language Jay quoted in that February 28 report:
DAWSON E-MAIL: Remediation Recovery, which has been around since 2001, means that fourth grade students who failed the third grade test in 2004, got to retake the third grade test in 2005. Up until this year (2005), if they passed the third grade test, then they were included in the numerator only of the calculation to determine the third grade passing score. As illustration, if 4 out of 5 third grade students passed and 1 out of 5 fourth grade Remediation Recovery students passed, the passing percentage would be 100 percent.In this e-mail, Dawson clearly refers to fourth grade students who failed the third grade test in 2004. We took this to mean that, when twelve of these students passed the test as fourth-graders in 2005, that number was added to the total of third grade students who passed the test that year. (As weve endlessly noted, this procedure would be completely bizarre.) The reporter from the Virginian-Pilot suggested something different. Her suggestion: When third-graders failed the test last year, they were restested this year as fourth-graders. That isnt the meaning we took from Dawsons e-mail, and we dont know if thats what actually happened. But well guess that this may turn out to be right, in part because so many schools have boosted their passing rates through Remediation Recovery. Its hard to believe that so many schools could engage in such an irrational process without someone complaining about it.
Is this what happened? When third-graders failed a test last spring (in the spring of 2005), were they retested at some point this year, as fourth-graders? If so, lets note the following about this procedure:
This procedure would make (somewhat) more sense than the one we thought Dawson described. But, alas, the same problem obtains. Do Virginias parents know what theyre getting when they check a schools passing rate? For example, when they see that 92 percent of Maurys kids passed last years Grade 3 and 5" reading tests, do they understand that many of these kids only passed the tests this year—a year late? Do they understand that the vast majority of students who passed the third-grade test only passed it this year, as fourth-graders?
Overview: Around 1970, public scrutiny began to put pressure on public school testing programs. Ever since, schools and school systems have searched for ways to inflate average scores and, now, passing rates. If this writers suggestion turns out to be right, Virginia found a way to do this through Remediation Recovery. Lets state the obvious—every school in America could improve passing rates if they kept retesting their third-grade students on into the fourth-grade year. Is there some educational reason to retest a child the next year? Well mark ourselves as skeptics on that. But one thing is certain; engaging in next-year retesting will drive up every schools passing rates. The school will look better to the public—and no, the public will not understand where those passing rates come from, or what they actually mean. (As weve said, we havent spoken to a single Virginia education reporter who seemed clear about the way this four-year-old process actually works.)
Yes, Virginia: Every school will improve passing rates if fourth-grade kids take the third-grade tests. And it can lead to absurd situations, as it did with the Post and Maury. Remember what happened: When these tests were given last spring, Maury recorded the second-lowest score in the state of Virginia on third-grade reading. Only one school in the entire state scored lower during this statewide test session. But thanks to the still-murky process known as Remediation Recovery, Maury was being hailed just nine months later, at the top of the Posts front page—and no, the Post didnt mention the schools remarkably low passing rates from the previous spring. What is really happening at Maury? We dont have the slightest idea. But its just strange when a school can score this low and still be hailed as a study in progress—without anyone being told about those very low scores.
Did third-grade students who failed last spring get tested again this year as fourth-graders? Were still not sure (for some doubts, see below). But can we make a simple suggestion? Here it is, and its really quite radical: Virginia should give third-graders the third-grade test! After that, the state should tell the public how many kids passed! Yes, this will have one big disadvantage—this will lower the states passing rates. But it will have one major advantage—it wont be a big giant scam.
REASON TO DOUBT: Is the writer from the V-Pilot right? Did some of last years third-grade students get retested this year as fourth-graders? (She wasnt sure.) That may be the case, but it doesnt quite jibe with the mathematical mumbo-jumbo included in the two e-mails we were sent by Monte Dawson, Alexandrias director of testing. (Link to the e-mails below.)
Example: Dawson sent us technical language explaining what the state would do if a schools passing rate ended up exceeding 100 percent! (Answer: Theyd cap the passing rate at 100!) Its hard to see how this could happen under the V-Pilot writers scenario. At Maury, for example, only 19 students would have been given the third-grade test; there would be no way that more than 19 students could end up passing it. This could only happen if the state retested fourth-graders who flunked as third-graders in 2004, then added their number onto the total of third-graders who passed in 2005. (In that situation, a given school could easily produce a passing rate which exceeded 100 percent.) This is one of the reasons why we interpreted Dawsons e-mails as we did.
Second problem: Note the part of Dawsons second e-mail which we highlight below:
DAWSON E-MAIL: Remediation Recovery, which has been around since 2001, means that fourth grade students who failed the third grade test in 2004, got to retake the third grade test in 2005. Up until this year (2005), if they passed the third grade test, then they were included in the numerator only of the calculation to determine the third grade passing score. As illustration, if 4 out 5 third grade students passed and 1 out of 5 fourth grade Remediation Recovery students passed, the passing percentage would be 100 percent. However, beginning July 12, 2005, passing Remediation Recovery students were added to the numerator and the denominator.Despite that e-mails semi-clear language, we assumed this meant that (forgive us) the number of passing students would be added to the denominator in school years which started after July 2005. We say that because what Dawson describes doesnt seem to be what happened with last years third-grade class at Maury. In that instance, 5 out of 19 third-graders passed the reading test—and, according to Jay Mathews, a 12" was subsequently added to the numerator only, transforming 5 out of 19" into 17 out of 19" and helping produce an overall passing rate of 92 percent. According to Dawson, this procedure will change—the 12" will be added to the denominator too. But this procedure will be absurd if the same kids are being retested the next year, as fourth-graders. Heres what will happen in a Maury-of-the-future: Nineteen kids will take the third-grade test in the spring of their third-grade year. Five of them will pass the test, creating this fraction: 5 out of 19. The next year, when theyre fourth-graders, the 14 failing students will be retested, and 12 will pass. Then, that 12" will be added to the numerator and the denominator, producing this new passing rate—17 out of 31 (55 percent). But to what would that fraction/passing rate correspond? Only 19 students will have been tested! And 17 of them will have passed. This new passing rate would be virtually impossible to explain. It would correspond, pretty much, to nothing. As weve noted before, this mumbo-jumbo doesnt seem to make sense if the same kids are just being retested. It would make a kind of sense if an entirely different group of kids were being added into the stew.
(By the way: What will happen to the two kids who fail the retest? Apparently, that 2" will just be thrown away. Under this bizarre statistical scenario, that number—2"—wont figure into the new passing rate.)
Yes, of course, its sadly true—the state of Virginia would be capable of a procedure this cock-eyed. And were still not sure what really produced that 92 percent passing rate at Maury. But heres what the state of Virginia should so, and once again, its extremely radical: The state should give third-graders the third-grade test, then tell the public how many kids passed! Yes, this would reduce the rate inflation which has made the state DOE look so good. But it would also produce the following boon—citizens would get to see real information about their childrens actual schools. They wouldnt be handed incomprehensible data which no one can quite explain.
Lets say it again: The state should test its third-grade kids. It should then report how many kids passed! This would produce straightforward data—and lead to straightforward public discussions. And oh yes—if Maury really does improve with its new principal, that will show up in future years testing! In future years, we hope this school will rock our socks—with impressive passing rates which are both straightforward and real.
FINAL NOTE: All hail the Times-Dispatch and the Virginian-Pilot! Lets hope they report this all the way out, informing us all in the process.
BASIC LINKS: On February 2, Maury hit the top of the Posts front page. You know what to do—just click here.
We questioned this story the following week. See THE DAILY HOWLER, 2/6/06, then click forward from there.
Post reporter Jay Mathews followed up on February 28. Click here and you can read every word.
We responded all last week—and the state of Virginia has removed its school report cards from view. See THE DAILY HOWLER, 3/7/06, for our first installment.
For the full text of Monte Dawsons two e-mails, see THE DAILY HOWLER, 3/8/06.
For a list of Alexandria schools, just click here. Keep clicking to see each schools report card. All schools in Virginia can be accessed from this basic link.