By Sol Stern
City JournalMay 19, 2008

The Institute for Education Sciences (IES) was established in 2002 as a quasi autonomous agency within the federal education department. One of its legislative mandates was to encourage the dissemination of scientific research to improve instructional practices in the nation's classrooms. Yet IES officials themselves undermined that worthy goal when they recently released a methodologically flawed and incomplete study of the federal Reading First program. The study found that students in a sample of schools receiving Reading First grants showed no greater improvement in reading comprehension than a similar group of schools that did not receive the federal grants. The combination of the poorly designed impact study plus sloppy media coverage of the findings is causing serious, and perhaps irreparable, damage to the only federal education program that has ever required schools receiving federal grants to adhere to instructional approaches backed by evidence and science.

Reading First was the Bush administration's landmark legislative initiative designed to raise reading achievement in grades K-3. The program, now in place at 6000 low performing, poverty schools in all 50 states, was funded at $1 billion per year. (See my City Journal piece, http://www.city-journal.org/html/17_1_reading_first.html) The separate federal Title I program, which ostensibly is also aimed at raising the academic achievement of disadvantaged children, is budgeted at about $14 billion this year. Since the passage of the 1965 Elementary and Secondary Education Act, close to $400 billion in taxpayer funds have been dispersed to Title I schools around the country. Most of this money has gone straight down the drain. Not only has the massively funded Title I program produced little in the way of school improvement, but the Congress never even asked for a research study to determine why Title I was not having the hoped-for impact on student learning.

By contrast, the Reading First legislation required the U.S. Department of Education to carry out a scientifically rigorous and comprehensive evaluation measuring whether the new program is fulfilling its promise of improving reading instruction in the early grades. Section 1205 of the law lists 10 specific components of Reading First (i.e., professional development for teachers, use of various diagnostic tools, plus effectiveness of various publishers' instructional materials) that must be assessed. These evaluation requirements were written by the chief architects of the new legislation, Dr. Reid Lyon, the head of reading research at the National Institutes of Health (NIH) and Robert Sweet, who served on the staff of the House education committee. With the benefit of thirty years of NIH sponsored scientific research in hand, Lyon and Sweet were confident that a national program of training classroom teachers in practices which put heavy emphasis on early mastery of decoding skills such as phonemic awareness and phonics would lead to improvement in reading achievement, particularly for disadvantaged children. But the law's authors also wanted to leave nothing to chance. Thus the Reading First legislation sets aside a huge pot of money -- up to $25 million per year, for six years -- to evaluate the program's impact on student learning through the most comprehensive study possible.

What the IES was able to produce after six years, however, was neither very rigorous nor comprehensive. We gave them money for a Cadillac and they came up with a Chevy, says Robert Sweet. (IES spent $30 million on the impact study.) Moreover, IES' process for developing an appropriate research design was both chaotic and late -- so much so that it precluded any evaluation of the first cohort of several thousand Reading First schools that started implementing the program either by September, 2002, or September, 2003. One outside expert who was involved in those early research design discussions with Abt Associates (the subcontractor selected by IES to perform the study) provided me with a letter from the study's project director requesting him/her to serve on one of the technical advisory groups. The letter's date was December 10, 2003, already very late in the day.

The letter also refers to IES' plan for a random assignment study that would involve a total of 30 Reading First schools and 30 control schools spread within 6 different school districts. As late as 2004, however, the study design was abruptly changed. Instead of 30 Reading First schools in 6 districts, 128 Reading First schools in 13 states would now be compared to a control group of schools that applied for Reading First grants but didn't qualify for the federal grants. And instead of the "gold standard," a random assignment design, the Reading First schools would be compared to the schools that weren't funded through a statistical technique known as a regression discontinuity model. In other words, the overall assessment was becoming even less rigorous and comprehensive.

IES received warnings from outside experts about the possible weaknesses of the latest study design. One reading scientist who has been willing to speak on the record about these concerns is University of Illinois Professor Timothy Shanahan, former president of the 85,000 member International Reading Association, the world's largest professional organization of reading teachers and scholars, and a recent inductee into the Reading Hall of Fame. Shanahan told me that he raised the lack of a randomized sample with IES officials and was told that it was too late to change the protocol. Along with other experts, Shanahan also pointed out that the eventual study design was contaminated, because "the control groups were often doing the same thing that the Reading First groups were doing."

IES may have missed the significance for its eventual study design of the fact that many states used up to 20% of their federal grant funds to encourage all low achieving schools to adopt the same instructional reforms that were mandated for the Reading First schools. Thus, says Shanahan, the comparisons were not Reading First with non-Reading First schools, but Reading First with less-Reading First.

Professor Shanahan's point about contamination of the control group has since been amplified with empirical preciseness by Dr. James A. Saltzman, co-director of Ohio's Reading First program. In a recent policy paper, Saltzman deconstructed IES' assumption that the schools in the study sample were either doing Reading First or not. He offered evidence from the Cleveland school district to prove his point:

While 20 schools were funded by Reading First in the district, the district spent their own funds to run a parallel program that infused scientifically-based reading research (SBRR) strategies and practices alongside a SBRR-based core reading program . . . Under this scenario, it's not unexpected that IES cannot tease out any differences among RF and non-RF schools and, in fact, the lack of differences may say more about districts in the study recognizing the importance of SBRR practices and use of a strong core program. If this is true, then it is further evidence of the positive impact of Reading First.

I raised the issue of the possible contamination of the study's control group directly with IES officials. In an email reply I was told merely that, "Spring 2007 surveys and district interviews (to be included in the study's final report) will provide more information about the extent to which Reading First-like practices diffused to non-RF schools in our study sample."

The final IES report is not due for release until 2009. In the meantime the New York Times (no friend of the Reading First program in any event) found no reason to withhold final judgment. "An Initiative on Reading is Rated Ineffective," trumpeted the headline on a story by Sam Dillon, the paper's long time national education reporter. Dillon didn't bother talking to any of the study's reputable critics, such as Timothy Shanahan or Dr. Saltzman. But his report did include this quote from Ted Kennedy, chairman of the Senate Education Committee: "The Bush administration has put cronyism first and the reading skills of our children last, and this report shows the disturbing consequences." Of course even Sam Dillon knew that there wasn't a word in the IES report about the Bush administration's "cronyism" or even anything explaining the null findings about reading improvement, but he left Kennedy's big lie out there unchallenged.

Even before the final IES report is out next year, Senator Kennedy and his anti-Reading First colleagues on the House side, Education Committee Chair George Miller and Appropriations Committee Chair Dave Obey, may gang up to finally put an end to Reading First. They have already cut the program by 60% and the flawed IES report could provide the bullets for the coup de grace. IES director Russ Whitehurst could at least go public and point out the unwarranted conclusions that the New York Times and Senator Kennedy have drawn from this very limited (and what many reputable reading scientists are saying is a very flawed) study. To do anything less, to hide behind the guise of scientific objectivity, would be a stunning display of irresponsibility.

Home | About Us | About Phonics | Resources
Research | Topics | Reading Reform | Links | Search

Unless otherwise noted, you may copy and distribute any information on this site as long as The National Right to Read Foundation at www.nrrf.org is given credit. The National Right to Read Foundation is a 501(c)(3) publicly supported organization.