Rhees score gains seem large to us/The world of educational experts: On the one hand, your DAILY HOWLER keeps getting results!
On the other hand, its routinely stunning to review the work of our educational experts. This includes high-ranking experts who both support and oppose the ministry of Michelle Rhee.
At least two major experts seem to have bungled this week in commentaries on Rhees work. One of these experts is a leading Rhee critic. The other is a leading supporter.
Where has THE HOWLER been getting results? In last Sundays Washington Post, Diane Ravitch reviewed Richard Whitmires new book, The Bee Eater, a comically sycophantic book about Rhees vast stunning greatness. In part, the book was funded by the Broad Foundation, one of the billionaire-funded foundations which favor Rhee-style policies.
Early in her piece, Ravitch reviewed DCs test scores under Rhee on the National Assessment of Educational Progress, the widely-praised gold standard of educational testing. Note: There is no reason to think that cheating or excessive erasures occurred on these federally-administered tests. The widespread irregularities now under investigation occurred on the DC-CAS, the Districts own high-stakes test program. The federally-administered NAEP is a whole different critter.
Ravitch reviewed DCs test scores under Rhee. She covered a two-year span corresponding to Rhees first two years in DC. (The NAEP was given in 2007 and 2009.) At first glance, the analysts cheered, noting the apparent lack of cherry-picking by Ravitch:
RAVITCH (4/10/11): Did [Rhee] succeed? Whitmire insists that she did and points to the Districts test score gains on the National Assessment of Educational Progress from 2007 to 2009. But the gains on the federal reading tests under Rhee were no greater than those under her predecessor Superintendent Clifford Janey, which were achieved without the firings and angst of the Rhee era. From 2005-07, under Janey, black fourth-grade students made a five-point gain in reading, but only a three-point gain under Rhee; Hispanic students made a 13-point gain in reading during Janeys tenure, but only a one-point gain from 2007-09. Reading scores for eighth graders didnt budge from 2002-09, regardless of who was running the school system. On the federal math tests for fourth grade students, the gains recorded on Rhees watch outpaced Janeys, but the gains from 2003-05 were larger than those achieved under either Janey or Rhee. In eighth-grade math, D.C. students have made steady gains from 2003-09.
At first glance, the analysts cheered. Often, Ravitch looks at eighth-grade reading alone. That is the area where the nations schools have had the most problem improving student performance. Focusing on that one measure (out of four), Ravitch often asserts that our public schools have floundered and failed in the age of Bushor of Rhee.
In this passage from the Post, Rhee runs through all four NAEP measures; she discusses math and reading, in grade 4 and grade 8. Lustily, the analysts cheereduntil we made them stop.
Oof. After a tedious review, we would say that Ravitch is wrong in her assessments in all four subject areas. The problem tracks to the large number of charter schools in DCand to some procedural changes made by the NAEP.
A few unavoidable technical notes, offered as quickly as possible:
DC has a large, growing number of charter schools within its public school system. About 40 percent of DC kids now attend charters; the students in those charter schools were not under Rhees direction. But uh-oh! Until 2003 (2005 for eighth grade), the NAEP didnt record what type of school a tested student attended. Before those years, the NAEP cant break out separate scores for charter schools or for non-charters.
Adding to the potential confusion, the NAEP changed a basic reporting procedure in 2009. Before 2009, the average scores the NAEP displayed for urban school districts included results from charter and non-charter schools alike. (We refer here to the scores displayed in the NAEPs basic reportsin its basic charts and graphs.) But starting in 2009, the NAEP changed this basic reporting procedure; from 2009 forward, the scores in its basic charts and graphs will exclude a school districts charters. This creates an apples-to-oranges problem if youre comparing scores from an earlier year with scores from 2009. In 2007, scores in the NAEPs basic reports reflect work done in all a systems schools. Starting in 2009, those same scores represent work done in non-charter schools only.
This hubbub creates a genuine problem with the NAEPs basic score displays. NAEPs basic reports include bullet-points warning users about this switch in procedurebut the charts and graphs do not include appropriate warning signs. On the other hand, heres a bit of good news: For all years starting with 2003 (2005 for eighth grade), you can use The NAEP Data Explorer (click here) to get average scores for a systems charter schools (if thats what youre tracking) or for its non-charter schools. In Washington, Rhee was in charge of the non-charter schools, like other superintendents before her.
Another problem lurks in the NAEP reports: Do you consider DC to be a state or an urban district? (Yes, it can make a difference.) These issues create a lot of confusion, but they can be figured out. But if you arent hip to all that jive, you may run into trouble.
In just this past week, at least two major educational experts did not seem hip to that jive. One of the two was Ravitch. After a highly tedious analysis, we would say that Ravitch got all four of her basic assessments wrong in last Sundays Washington Post. Everything she said about gains under Rhee strike us as basically wrong.
Lets examine all four areas tested by the NAEP, starting with the area where Ravitchs misstatement seems most egregious. (For access to all relevant NAEP reports, just click this. From there, youre on your own.)
Eighth-grade math: According to Ravitch, D.C. students have made steady gains from 2003-09. That may seem to be true if you look at the NAEPs basic charts and graphs, where charter and non-charter schools were mixed together before 2009, at which point charters were excluded. But if you use the Data Explorer to look at scores from non-charter schools only, there was no gain at all from 2005 to 2007 under Janey, and a very large gain from 2007 to 2009 under Rhee. (Before 2005, the average score of non-charter schools isnt available.)
How big was the score gain under Rhee? According to the Data Explorer, DC kids in non-charter schools gained 7.21 points in that two-year span. By a very rough rule of thumb, ten points on the NAEP scale is often said to correspond to one academic year. As such, this would be a very large gain by the Districts eighth-graders. (Under Janey, the score for kids in non-charter schools went up by 0.08 points.)
Eighth-grade reading: According to Ravitch, average scores in this area didnt budge from 2002-09, regardless of who was running the school system. That may seem true if you look at the NAEPs basic charts and graphs, but it doesnt seem to be true if you look at non-charter schools only. (In fact, a bullet-point in the NAEPs basic report warns users that its graph is misleading for the 2007-2009 periodthat a three-point gain occurred in the non-charters alone.) In fact, kids in non-charter schools gained 3.63 points from 2007 to 2009 (after losing 0.23 points under Janey). That isnt as large as the gain in math, but it represents very strong growth over a two-year span.
(For the record, its just as that bullet-point notes, right out in broad daylight for all to see: The overall score in 2009 (240) was higher than in 2007 when the 2007 average score is recomputed to exclude charter schools (237) to account for the change in population definition for 2009. See the Technical Notes for more information.)
Fourth-grade math: According to Ravitch, the gains recorded on Rhees watch outpaced Janeys, but the gains from 2003-05 were larger than those achieved under either Janey or Rhee. This doesnt even seem to be true if we look at the NAEPs basic graph; that graph shows a six-point gain from 2003-05, with a matching six-point gain from 2007-09. (The same graph shows a three-point gain from 2005-07.) Reviewing non-charter schools only, what does the Data Explorer produce? According to the Data Explorer, the gains were largest under Rhee. Students in non-charter schools gained 6.38 points from 2007-09. The corresponding gains were 4.96 points from 2003-05 and 3.43 points from 2005-07.
For the record: If we accept the very rough 10 points equals one academic year rule of thumb, a gain of 6.38 points over two years would represent a very large gain. All the gains in this subject area were strongbut the gains under Rhee were largest.
Fourth-grade reading: In this one area, Ravitch cited black students only; this may be because the NAEPs basic graph for all students seems to show a large gain under Rhee. At any rate, Ravitch says this: From 2005-07, under Janey, black fourth-grade students made a five-point gain in reading, but only a three-point gain under Rhee. (We knowthat sentence doesnt parse, but you get the point. Ravitch also cites Hispanic students, plucking a figure which makes Rhee look bad. Warning: Sample sizes are very small for this group in DC. Standard errors are therefore quite large.)
Ravitch comes closest to being right in this one area. If we look at non-charter schools, the Data Explorer does show larger gains under Janeythough gains were also good under Rhee, and the differences are smaller than Ravitch reported. For black students only, the score gain was 5.70 points under Janey, 4.05 points under Rhee. For all students, the gain was 7.25 points under Janey, 5.76 points under Rhee. (In each case, the gains from 2003-05 were small just 0.97 points for black students, 1.49 points for all students.)
Wed be happy to be shown that something is wrong with our analysis. (Due in part to the large number of charter schools in DC, the NAEP has produced a fairly confusing field of play when it comes to the DC schools.) But these are the data which seem to emerge if we consider non-charter schools onlyif we exclude the large number of students in charter schools, students who werent under the direction of Janey or Rhee. The NAEP has made these data available to permit these sorts of basic distinctions. But for decades, we have observed a basic pattern: When the educational experts arrive on the scene, all heck tends to break loose.
Question: Do these experts ever get anything right? If someone shows us the answer is yes, well be happy to say so.
One again, Ravitch has produced a piece which says score gains werent that hot under Rhee. We arent fans of Rhee around here, but the data look quite different to us if you consider only the non-charter schools, the schools Rhee was directing. To be sure, a person should be careful in drawing conclusions based on a single two-year time span; in DC, student migration between charters and non-charters would seem to call for particular caution. But the score gains under Rhee seem large. And Ravitchs claims strike us as inaccurate and/or misleading, in all four subject areas.
We said two major experts had fumbled. The second such expert is Harvard political scientist Paul E. Peterson, who stepped up to challenge a rather plainly bungled report by a third educational expert. Peterson is a Rhee supporter. Jay Mathews discusses the whole shebang under this headline: Harvard education expert attacks Rhee critics.
Oh, those educational experts! As a general matter, wed be inclined to agree with much that Peterson says in his report, although he downplays the current excessive erasure problem is a ridiculous manner. But beyond that, Peterson makes a truly bewildering error, again concerning the NAEPs change in procedure regarding charter schools. To read Mathews piece, just click this. If youre intrigued by the ways of expert culture, we posted a comment quite late in the thread. (Ignore our prior half-comment. The Posts new comment system tends to fire on its own.)
Jays assignment, and we insist he accept it: Mathews should report this whole thing out. What were score gains like under Rhee? Ravitch has said the score gains werent all that. Mathews should telephone technical folk at the NAEP and report the whole affair, reviewing all four subject areas. Well be happy to hear that we are wrong in some way, although were working right from the Data Explorer. But the expert work has been confusing and contradictory.
Jay should report this whole thing out. Or is this all a silly gamea game which exists so liberals can yell at Bush and Rhee and conservatives can yell about those lazy teachers and their fiendish unions?
Is this all a tribal game? Do the actual interests of actual DC kids actually matter at all?
Special report: Mark Twains ineffectual mob!
INTERLUDEHOW DOES DAVID BROOKS KNOW THAT (permalink): David Brooks has special powers (click here). How else could David Brooks know this?
BROOKS (4/15/11): The president, meanwhile, hit the political sweet spot with his speech this week. He made a sincere call to reduce debt, which will please independents, but he did not specify any tough choices. He called for defense cuts and asked the Pentagon to find some. He called for a reduction in tax credits but didnt point to any that should actually go. He called for reductions in Medicare costs and asked his board of technocrats to come up with some.
These are exactly the sort of vague but well-intentioned policies that have sold well in election after election. The president is not being cynical about this. He genuinely does believe that seniors and the middle class can be spared from any shared sacrifice. He really does believe in calling together teams of experts to devise proper solutions. Obamas sincere preferences happen to be more popular.
How can David Brooks possibly know what the president genuinely does believe? How can he possibly know if he is being cynical in this, that or three other things? We have no idea, but Brooks can read Paul Ryans mind too! Like Obama, Ryan is one of the most admirable men in Washington, Brooks says at the start of his column. Along the way, he helps us know what Ryan sincerely believes.
How can David Brooks know these things? Simple! Brooks has been cast in the role of a major journalistand major journalists have adopted the novelists posture over the past many years. They know who the good and the bad people are, and theyre eager to help us rubes understand. They will even invent bogus tales to help us spot the Big Liars!
Alas! When it comes to politicians character, their judgments have turned out to be grotesquely wrong again and again and again. But so what? They never stop pimping their character talesnovels then tend to type and recite as a guild, as a clan, as a well-scripted group.
In modern times, this sort of thing dates at least to the late David Broders remarkable conduct in 1972, when he and some journalistic pals decided that Edmund Muskies bad temper disqualified him for the White House. (They had reached this judgment playing cards with Muskie. No, were not making this up!) Broder then concocted a story about the way the furious Muskie boo-hoo-hooed in public. This ballyhooed story badly damaged Candidate Muskies White House campaign.
Fifteen years later, Journalist Broder basically said that he pretty much made the tale up.
In 1987, Broder wrote this: In retrospect, though, there were a few problems with the Muskie story. First, it is unclear whether Muskie did cry. For a longer account of this truly astonishing conduct, see THE DAILY HOWLER, 11/28/07. (Note: A certain major liberal blogger has come a long way since then.)
Broder and his fellow journalists could see Ed Muskies soul. This morning, Brooks displays the same power. Did we mention the fact that these novelists have routinely been wrong in their supremely confident assessments of character?
Whatever! Brooks has massively clowned in the past two weeks. He has ceaselessly fawned over Ryans high character; today, he mind-reads (and loves) Obama too! But along the way, we think he got one thing quite right. On Tuesday, Brooks helped us see what happens when our store-bought liberal intellectual leaders keep lazily losing major debates. Long ago, we liberals lost the public debate about taxesabout the nature of community. Who lost taxes? The liberal world did! Last Friday, Brooks explained where that big defeat leaves us:
BROOKS (4/8/11): The Democrats are on defense because they are unwilling to ask voters to confront the implications of their choices. Democrats seem to believe that most Americans want to preserve the 20th-century welfare state programs. But they are unwilling to ask voters to pay for them, and they are unwilling to describe the tax increases that would be required to cover their exploding future costs.
Raising taxes on the rich will not do it. There arent enough rich people to generate the tens of trillions of dollars required to pay for Medicare, let alone all the other programs. Democrats, thus, face a fundamental choice. They can either reverse President Obamas no-new-middle-class-taxes pledge, or they can learn to live with Paul Ryans version of government.
Until they find a way to pay for the programs they support, they will not be serious players in this game. They will have no credible plans and will be in an angry but permanent retreat.
On Monday, well review the conduct of Mark Twains famous mob. For today, we think that passage by Brooks is still well worth considering.
This morning, Paul Krugman presents his own view about future taxes. (Over the longer run, I believe that well need modestly higher taxes on the middle class as well as the rich to pay for the kind of society we want.) What does modestly higher mean? We have no idea. But it will be very hard to campaign on such an outrageous idea. Our side lost the debate on taxes a very long time ago.
Remember when Mondale made such a fool of himself, telling the public the truth about taxes? In 2010, the Post compared Mondales statement to Mayor Barry saying Bitch set me up.
How have we managed to lose these debates? Frankly, our liberal leaders have often been store-bought. They have had very good jobs, at very good wages, working at very Serious news orgs.
Our intellectual leaders have often been bought, but we rubes refuse to see it. If they arent working at Fox News, we just know theyre on our side!
That is only part of the way our side has resembled Mark Twains hapless mob. Well review that mobs pitiful conduct next week. But alas! On the topic of taxes, Brooks pretty much got it right.
As the world turns: Is the world finally turning against no new taxes? Go aheadjust click here.