Reading Recovery Research Project
SAN DIEGO UNIFIED SCHOOL DISTRICT
Office of the Board of Education
READING RECOVERY RESEARCH PROJECT
October 12, 1999
October 26, 1999 (Revised)
As directed, Board staff conducted research and analysis of the District’s Reading Recovery/ Descubriendo La Lectura (RR/DLL) Program. The research project compared the academic performance of participating and non-participating first grade students based on two established assessments: norm reference tests and literacy assessment portfolios. The purpose of the research project is to determine whether San Diego City Schools should continue primarily financing the RR/DLL Program through Title I funding.
History of Reading Recovery-
New Zealand educator and psychologist, Dr. Marie M. Clay, is the acknowledged founder of Reading Recovery. During the 1960’s, Dr. Clay conducted observational research, which laid the foundation for techniques in detecting early reading and writing difficulties of children. During the 1970’s, Dr. Clay worked with a team of experienced teachers in New Zealand to develop program procedures and conduct field trials. Based on test results of the pilot studies, in 1979, New Zealand adopted the RR Program nation-wide. In 1984, Reading Recovery was introduced in the United States. In 1990, Reading Recovery was introduced in California. In 1992, Descubriendo La Lectura, an application of Reading Recovery in Spanish, was constructed for Spanish-speaking students.
In San Diego City Schools, the RR Program was introduced during the 1991-92 school year. The impetus was provided by the San Diego County Reading Recovery Consortium that included three school districts: San Diego Unified, Poway Unified and Escondido Union. The Spanish version of Reading Recovery was included during the 1995-96 school year. Since then, the RR/DLL Program has expanded to 48 school sites with a staffing level of 117.5 teachers. Provided below is a table illustrating this expansion:
1Teacher leaders are those who train the RR/DLL Teachers.
2Figures include the DLL Program, which began during 1995-96.
Overview of Reading Recovery-
Reading Recovery is an early intervention program designed to assist children in first grade who are having difficulty learning to read and write. The RR/DLL Program entails the use of trained teachers providing one-to-one tutoring in 30-minute daily sessions to the lowest 20% of a first grade class who have the prerequisite skills.1 The primary goal of Reading Recovery is to bring the lowest 20% performing students up to the average reading proficiency level of their classmates by the end of first grade within 60 lessons, or approximately 12 weeks.
The first two weeks of the student’s participation are designed to promote engagement with literacy activities. This period is referred to as “roaming around the known” and is comprised of a variety of reading and writing activities. The student’s individualized lesson plans include the following seven components:
1. Reading familiar books
2. Taking a running record
3. Letter identification and word making/breaking (including graphophonic relationships)
4. Writing a story
5. Rearranging a cut-up story
6. Introducing a new book
7. Reading a new book.
Approximately 400 “little” books are included in the Reading Recovery book list, and are organized into 20 levels based on level of complexity and difficulty. In San Diego City Schools, over 90% of the time spent in the RR/DLL Program is devoted to reading books and writing stories which are then read.
Generally, when students successfully complete 60 RR/DLL lessons AND reach the average reading proficiency level of their classmates, they are discontinued,2 at which time the RR/DLL Teacher can take another student into the 30-minute slot. Each RR/DLL Teacher, working a half-day with Reading Recovery, is expected to be able to tutor eight students in one year. The other half a day is devoted to conducting “guided reading literacy groups.”
Professional development is an integral part of the RR/DLL Program. Training consists of an intense, year-long graduate course for teachers made up of weekly classes affiliated with a university-based Regional Training Center. Teachers from San Diego City Schools attend the Center located at California State University, San Bernardino. In August 1994, San Diego City Schools established its own training facility located at Valencia Park Elementary. As teachers learn how to implement the program, they work simultaneously with children in their home schools. Through clinical and peer-critiquing experiences guided by Teacher Leaders, RR/DLL Teachers learn to use diagnostic techniques and teaching procedures for conducting lessons.
The criteria for selecting classroom teachers to become RR/DLL teachers has been: 1) at least three years of teaching experience; 2) experience at the primary level; and 3) recommendation by principal, administrator and/or other teachers.
1An observation survey (designed by Dr. Clay) is used to assess and select participating students. This diagnostic tool is also used as a pre- and post-test instrument and includes the following six measures: letter identification, word test, concepts about print, writing vocabulary, hearing sounds in words, and text reading level.
2The 1996-97 Reading Recovery Site Report (page 17) further states: “Decisions concerning whether or not children could be discontinued were made by examining a variety of data for each dhild: 1) highest level of text reading at 90% accuracy or better; 2) scores on three Observation Survey assessments: Writing Vocabulary, Dictation and Text Reading Level; 3) reading behavior as shown in recent running records and text reading tests; and 4) achievement in the classroom instructional program.”
Demographic Profile of RR/DLL Participant-
The research project focuses on first grade students who participated in the RR/DLL Program during 1995-96 and 1996-97. Provided below is a table listing the number of students distinguished between those who successfully completed the 60 lessons AND then performed at the average level of their classmates (discontinued) and those who did not (not discontinued).
|Year of Participation||Total||Discontinued||Not Discontinued|
Provided below is a table reflecting the race/ethnicity of those first grade students who participated in the RR/DLL Program:
#Other includes Alaskan/Indian and Pacific Islander.
For first grade students participating during 1995-96, analysis was conducted to ascertain the reasons why certain participating students DID NOT successfully complete the RR/DLL Program. Similar analysis could not be conducted for 1996-97. A summary table3 is provided below:
|In RR/DLL Program
|Year of Participation||Total||Completed 60+ lessons#||Completed <60 lessons||Withdrew w/60+ lessons||Withdrew w/ <60 lessons||Withdrew – Special Education|
#These include first grade students who completed 60+ lessons but DID NOT attain the average reading level of their first-grade classmates on the Observational Survey at the end of the school year.
The research project compares the reading proficiency levels of first grade students who participated in the RR/DLL Program during 1995-96 and 1996-97 with a control group of non-participating students from each school site. In fact, the control group consists of the same students included in the Reading Recovery Site Reports4 authored by the District’s RR Program Teacher Leaders.
As previously stated, Reading Recovery asserts that, through targeted intervention, the reading level of the lowest performing 20% of students will improve to the average reading level of their non-participating classmates once the student has successfully completed the program. For example, the 1996-97 Site Report states at least 96% of its RR participants and 98% of its DLL participants accomplished this task (pages 19-20). The “sustained effect” of Reading Recovery into the third grade for participating, first grade students was ascertained by re-administering the Text Reading component of the Observation Survey. The 1996-97 Site Report (pages 42-43) states that 88% of RR discontinued students and 95% of DLL discontinued students were reading at or above grade level.
Thus, to further evaluate the efficacy of the RR/DLL Program the above-referenced tenets of Reading Recovery are examined by relying upon norm reference and reading assessment test results for participating vs. non-participating first grade students one and two years after participating in the internal program study. By confining the elapsed time to two years, the study focuses on the initial period of the student’s academic career when they are still “learning to read.”
3Source: 1996-97 Progress Report on the Reading Recovery Program, prepared by SDUSD Planning, Assessment, Accountability & Development Division.
4Prepared by SDUSD Educational Services Division, Humanities Department. The Reading Recovery Site Reports refer to these students as a “site random sample” established by compiling class lists of first grade students from sites with a RR/DLL Program. This control group was established to determine “a site average band.” At the end of each school year, participating first grade students along with the control group are tested with the same three parts of the Observation Survey. The three parts include writing vocabulary, dictation and text reading.
The research project assesses the reading proficiency levels of participating, first grade students distinguished between three groups that comprise the study cohorts:
1. Those who successfully completed the RR/DLL Program (discontinued),
2. Those who did not successfully complete the RR/DLL Program (non-discontinued), and
3. Those who did not participate (control group) from the same school site.
The summary table below outlines the academic indicators used to assess the reading proficiency levels of the study cohorts:
|First Grade Student||SAT 9 Total Reading||Literacy Portfolio|
|Grade 2||Grade 3||Grade 3|
|Year of Participation||1996-97||1995-96||1995-96|
The SAT 9 Total Reading test scores included sub-tests assessing Reading Vocabulary and Reading Comprehension. The Literacy Assessment Portfolio included sub-tests assessing Oral Accuracy, Oral Comprehension and Oral Fluency.
Using the data from SAT 9 and Literacy Portfolios, the performance levels of the first grade students included in this research project were assessed based on the following methodology:
Number and/or percentage of students who participated in the RR/DLL Program above and below the 50th percentile, compared to a control group of students who did not participate.
As previously suggested, the academic indicators and methodology used in this research project are substantively different than those used in previous studies of the District’s RR/DLL Program.
For example, over the past years, the District’s Reading Recovery Program Teacher Leaders have prepared annual Reading Recovery Site Reports.5 These Reports focused on participating, first grade students who completed the required 60 lessons. Academic performance was evaluated using three of the six measures included in the program-designed Observation Survey. The evaluation process included selected text materials in graded levels of difficulty, based on a scale of 1-30. The efficacy of the RR/DLL Program was based on comparative data using the Observation Survey. Participating students were tested at the beginning and end of the school year. Non-participating students randomly selected from each school site were tested at the end of the school year. Program success was based on the number of participating, first grade students who discontinued from the RR/DLL Program AND attained the average reading level of the site random sample.
Other district reports have been prepared as well. One particular report, entitled the “1996-97 Progress Report on the Reading Recovery Program6” also evaluated the sustained effect of Reading Recovery. The study analyzed second and third grade mid-year English Language Arts academic marks of first grade students who participated in Reading Recovery during 1994-95 and 1995-96. It is important to note that this report states that 1994-95 was the first year in which the District’s RR Program adhered to the criterion of selecting first grade students in the lowest 20% of their classes. In previous years, students were selected from an upper range within the lowest 30%. This study did not include a control group per se. Rather, participating students served as their own control group distinguished between discontinued vs. not discontinued. The report concluded:
“Thus, the program’s adherence to the selection criterion has been accompanied by three important outcomes: a noticeably low percentage of students who are discontinued; a noticeably low percentage who are at least “moderately experienced readers” in Grades 2 or 3; and a considerable number of students who are not discontinued because they have not completed sufficient RR lessons or they have not reached the average level of their first-grade classmates.” (Executive Summary)
The study cohort for the research project includes 678 participating first grade students, along with 106 first grade students who comprise the control group of non-participants. A summary table is provided below:
|RR Discontinued Students||91||206|
|RR Non-Discontinued Students||82||78|
|DLL Discontinued Students||52||84|
|DLL Non-Discontinued Students||41||44|
|RR/DLL Participant Total||266||412|
|Non-Participating Students (Control Group)||50||56|
|Study Cohort Total||316||468|
5Prepared by SDUSD Educational Services Division, Humanities Department.
6Prepared by SDUSD Planning, Assessment, Accountability & Development Division. This report was submitted to the former Senior Management Council, but was never presented to the Board of Education for review and comment.
7The 1996-97 Progress Report (page 6) indicated 237 RR participants, and 127 DLL participants for a total of 364 participating, first grade students. Based on the state 364 students, the study cohort reflects a difference of 98.
8The 1996-97 Site Report (page 8) indicated 303 RR participants, and 150 DLL participants, totaling 453 participating, first grade students. However, analysis was conducted on 361 students only. Thus, 92 students were unaccounted for in the study’s research findings. Based on the stated 453 students, the study cohort reflects a difference of only 41.
Attachment B details the norm reference and literacy assessment test scores between participating and non-participating first grade students who comprise the 1995-96 and 1996-97 study cohorts. A review of the test scores reveal the following:
(The above finding pertaining to Oral Accuracy vs. Oral Comprehension coincides with district-wide portfolio test scores [total and disaggregated by language proficiency] as indicated by Attachment C.)
3) Aprenda 2 test scores indicate first grade students who participated in the Spanish version of Reading Recovery, on average, scored significantly higher than those who participated in the English version of Reading Recovery – regardless if they successfully completed the program or not. Likewise, the percentage of first grade students scoring above the 50th percentile was higher for those who successfully completed the Spanish version vs. the English version of Reading Recovery.
(A plausible explanation may be attributed to the phonology systems of Spanish vs. English, which allow Spanish Speakers to decode sooner than English Speakers. The Aprenda 2 test results may reflect this since the Total Reading sub-tests were administered in the student’s primary language – Spanish.)
Discontinued vs. Control Group Students
1) Literacy Assessment test scores indicate first grade students who successfully completed the English version of Reading Recovery, on average, scored lower than the control group of non-participating students on all three sub-tests. First grade students who successfully completed the Spanish version of Reading Recovery, on average, scored lower, on two of the three sub-tests. The exception was Oral Accuracy.
2) SAT 9 test scores indicate first grade students who successfully completed the English version of Reading Recovery, on average, scored lower than the control group of non-participating students used in the internal studies conducted by the District’s RR/DLL Program.
The Aprenda 2 test scores indicate first grade students who successfully completed the Spanish version of Reading Recovery, on average, scored lower than the control group of non-participating students used in the internal studies conducted by the District’s RR/DLL Program.
Likewise, the percentage of first grade students scoring above the 50th percentile was higher for those comprising the control group than those students who successfully completed either version of Reading Recovery.
This finding is particularly significant as it contrasts with the findings from previous Site Reports regarding the initial and sustained effect of Reading Recovery. An array of plausible explanations can be posited on this reemerging gap in reading proficiency levels within a one and/or two year period. However, without corroborating research and analysis, only questions can be put forth at this juncture. For example, can this re-emerging gap be attributable to a:
- Subsequent disparity in regular classroom instruction provided to the first grade students as they entered the second grade and/or third grade in the same or different school site?
- Inability of RR/DLL participants to transfer strategies obtained through Reading Recovery into a regular classroom?
- Predicating successful completion of Reading Recovery upon attaining the average reading levels of non-participating classmates vs. a national norm?
- Operational issues affecting the RR/DLL Program? (Some of these issues were mentioned in a March 1998 report by Dr. Jacqueline Chaparro to ex-Superintendent Bertha Pendleton.) The Institute for Learning has recently implemented some reforms. These include: a) ensuring Reading Recovery is offered AFTER the 3-hour Literacy Block; b) supporting sites to implement early literacy groups that are consistent, well designed and offered as “push-in” models; and c) providing site leadership with training sessions designed to support and enhance quality RR/DLL Programs.
- Heretofore, relying upon research based on an evaluation system that aligns itself with the instructional program (ex: research design, performance instrument and measures, and data collection procedures) that creates a bias in the results of the evaluation? Additionally, past internal studies have been conducted by program practitioners who have a vested interest in the research results.
- Other external factors that may have come into play between the time the first grade students undertook the post-test and the SAT 9 and/or Literacy Assessment one and two years later?
RR/DLL Program Funding-
Since its inception, the primary source of funds used to finance the District’s RR/DLL Program has been Title I federal categorical funding. During 1991-92, Title I constituted 50% of Reading Recovery’s total funding, which increased to 87% in 1998-89. During 1998-99, 20.4% of the District’s total Title I funding was expended on the RR/DLL Program. Provided below is a table outlining the historical growth rate in funding dedicated to the District’s RR/DLL Program:
(Centrally Funded From Title I)
|Fiscal Year||Title I
|Amount||Percent of Total
Title I Budget
Source: SDUSD Finance Division
The accelerated growth rate since 1994-95 can be attributed, in part, to the inclusion of Reading Recovery as one of 17 Obey-Porter Model Programs approved for funding through Title I. During 1998-99, 101.25 of the 117.5 FTE allocations were financed through Title I funding.
An expressed concern of the District’s RR/DLL Program has been the high per capita cost of delivering this intervention strategy. Provided below is a summary table outlining the per capita cost, based on the number of participating students:
|Fiscal Year||Reading Recovery
This evaluation of the District’s RR/DLL Program using SAT 9 and Literacy Portfolio test scores of participating vs. non-participating first grade students indicates a dire need to conduct an independent, comprehensive analysis of the District’s Reading Recovery/Descubriendo La Lectura (RR/DLL) Program.
Beginning with the literacy framework that was introduced during the 1998-99 school year, the District’s Institute for Learning has prioritized reading as the “gatekeeper” core subject upon which to elevate the overall academic performance of all students. Consequently, new initiatives have been implemented to address student reading proficiency levels, including After-School Reading (Grade 3), and Two-Period Literacy Blocks (Grades 6, 7 and 9).
Consequently, it must be recognized the District needs to increase its capacity to systematically conduct comprehensive studies involving major qualitative and quantitative analyses of its instructional programs – especially those dedicated to increasing reading proficiency levels.
There are no facilities implications.
Since its introduction in 1991-92, the level of Title I funding devoted to Reading Recovery has increased by over $6.8 million. In 1998-99, Reading Recovery encumbered 20% of the District’s total Title I categorical funding.
The financing of support programs and intervention strategies to address social promotion/retention coupled with the Superintendent’s edict that Title I school site funding allotments will be based on 40% dedicated to literacy and 20% to academic achievement will create increasing competition for limited school site discretionary funding. The high per capita cost to deliver Reading Recovery to address the lowest 20% of performing first grade students precludes elementary school sites from enlisting alternative strategies that may (or may not) prove more cost-effective and cost-efficient. Thus, before expanding Reading Recovery critical comprehensive research and analysis should be conducted to validate the efficacy of this intervention strategy.
Public Support and Engagement Implications
There are no direct implications regarding public support and engagement as a result of the research findings from this independent study of Reading Recovery.
Board Policy Implications This Board Report complies with board policies F-8400 (Assessment and Training), F-8800 (Program Evaluation) and F-9400 (Research).
Based on the findings of this research project conducted on the District’s RR/DLL Program, it is recommended that your Board direct Board Staff to:
4) Submit the final research methodology design and timeline for Board review and approval by no later than July 31, 2000.
5) Initiate a Request for Proposal (RFP) to hire an external evaluator(s) to conduct the comprehensive study (upon completion of the above tasks).
It is further recommended that your Board direct the Superintendent to provide a supplemental report concerning operational improvements to enhance the District’s RR/DLL Program in conjunction with the 2000-2001 Title I/State Compensatory Education Participation Plan (tentatively scheduled for April 2000).
Report prepared by Jerome Torres, Board Policy Analyst. Technical assistance provided by Nick Nicholas.
IMPLEMENTATION OF READING RECOVERY WITHIN
SAN DIEGO CITY SCHOOLS
RR Program was
RESEARCH STUDY FINDINGS
|1995-96 COHORT – SAT 9/APRENDA 2, Spring 1998|
|Reading Comprehension||Reading Vocabulary|
|RR Discontinued Students||36.0||84||33.1||80|
|RR Non-Discontinued Students||24.3||65||18.1||63|
|DLL Discontinued Students||40.8||44||43.8||44|
|DLL Non-Discontinued Students||35.4||30||35.7||27|
|Control Group Students-SAT 91||40.1||38||34.6||37|
|Control Group Students-Aprenda 2||59.3||9||59.1||9|
|Reading Comprehension||Reading Vocabulary|
|N <50th %||N >50th %||N <50th %||N >50th %|
|RR Discontinued Students||73||11||65||15|
|RR Non-Discontinued Students||63||2||62||1|
|DLL Discontinued Students||32||12||29||15|
|DLL Non-Discontinued Students||27||3||21||6|
|Control Group Students-SAT 91||31||7||27||10|
|Control Group Students-Aprenda 2||2||7||1||8|
1Six of nine control group students who took the Aprenda 2 took the SAT 9 test as well. The SAT 9 test results are excluded from the mean score and percentile ranking to ensure a comparison between English Speaking students ONLY.
|1995-96 COHORT – Literacy Portfolio, Spring 1998|
|Mean Score||Total N1|
|Portfolio Oral Accuracy|
|RR Discontinued Students||4.47||83|
|RR Non-Discontinued Students||3.30||67|
|DLL Discontinued Students||4.51||51|
|DLL Non-Discontinued Students||3.81||37|
|Portfolio Oral Comprehension|
|RR Discontinued Students||3.33||83|
|RR Non-Discontinued Students||2.86||64|
|DLL Discontinued Students||3.33||51|
|DLL Non-Discontinued Students||2.91||35|
|Portfolio Oral Fluency|
|RR Discontinued Students||3.71||83|
|RR Non-Discontinued Students||2.97||65|
|DLL Discontinued Students||3.71||51|
|DLL Non-Discontinued Students||3.05||37|
|Non-Participating Students (Control Group)|
|Portfolio Oral Accuracy||4.49||47|
|Portfolio Oral Comprehension||3.65||46|
|Portfolio Oral Fluency||3.87||47|
1Not included are Ns and related scores of “0” for portfolios marked as NE (not enough evidence) and/or NS (portfolio not submitted).
Test scores of 2 and/or 3 indicate “below grade-level standards.”
Test scores of 4 and/or 5 indicate “meets or exceeds grade-level standards.”
|1996-97 Cohort — SAT 9/Aprenda 2, Spring 1998|
|Reading Comprehension||Reading Vocabulary|
|RR Discontinued Students||37.6||170||32.5||150|
|RR Non-Discontinued Students||27.4||38||19.7||33|
|DLL Discontinued Students||47.9||79||47.9||71|
|DLL Non-Discontinued Students||33.5||32||29.9||29|
|Control Group Students-SAT 9||44.6||45||42.8||45|
|Control Group Students-Aprenda 2||53.5||10||59.1||10|
|Reading Comprehension||Reading Vocabulary|
|N <50th %||N >50th %||N <50th %||N >50th %|
|RR Discontinued Students||141||27||121||29|
|RR Non-Discontinued Students||37||1||30||3|
|DLL Discontinued Students||44||35||39||32|
|DLL Non-Discontinued Students||29||3||27||2|
|Control Group Students-SAT 9||25||20||27||18|
|Control Group Students-Aprenda 2||5||5||4||6|
SAN DIEGO CITY SCHOOLS
SPRING 1998 LITERACY ASSESSMENT PORTFOLIO
DISTRICT-WIDE, GRADE 3
|Oral Reading Accuracy|
|Oral Reading Comprehension|
|Oral Reading Fluency|
|Overall Portfolio Score|
NOTE: Mean score does not reflect NE and NS scores, which are scored as “0.”
NS = Portfolio not submitted. NE = Not enough evidence.
“Reading Recovery Site Report -Year Three 1996-97,” prepared by Ann Fulmer, Billie Greene, Helena Sabala, SDUSD Educational Services Division, Humanities Department
“1996-97 Progress Report on the Reading Recovery Program,” prepared by Barry Fass-Holmes and Frank Ciriza, SDUSD Evaluation Unit, September 8, 1997
“Report on the Reading Recovery Program in San Diego Unified School District,” prepared by Barry Fass-Holmes and Donna Davis, SDUSD Evaluation Unit, August 21, 1995
“1995-96 Progress Report on the Reading Recovery Program,” prepared by Barry-Fass Holmes and Frank Ciriza, SDUSD Evaluation Unit, December 16, 1996
“Reading Recovery in Relation to Language Factors, Reading Self-Perceptions, Classroom Behavior Difficulties and Literacy Achievement: A Longitudinal Study,” prepared by James W. Chapman, William E. Tummer & Jane E. Prochnow, Massey University, New Zealand. (Paper presented at the annual meeting of the American Educational Research Association, San Diego, California – April 1998.)
‘Touching the Future: Executive Summary, 1991-97,” prepared by Judith Neal, Patricia Kelly, Adria Klein, and Barbara Schubert, California Reading Recovery
“Reading Recovery Review: Understandings, Outcomes & Implications,” prepared by Billie Askew, Irene Fountas, Carol Lyons, Gay Su Pinnell and Maribeth Schmitt, Reading Recovery Council of North America, Inc.
“Reading Recovery: An Evaluation of Benefits and Costs,” prepared by Bonnie Grossen, Gail Coulter and Barbara Ruggles, Effective School Practices, 15(3), Summer 1996
“Questions and Conclusions from a Discussion of Reading Recovery,” prepared by Patrick Groff, Effective School Practices 15(3), Summer 1996
“Reading Recovery in the San Diego Unified School District 1998: A Summary of Findings and Recommendations,” prepared by Jacqueline L. Chaparro, March 1998