With the recent release of this year’s results from “the nation’s report card,” supporters and critics of charter schools have renewed their debate over charter students’ relative performance, even while acknowledging serious limitations in the data’s reliability.
Charter school students’ scores appeared to be lower in most categories on the National Assessment of Educational Progress tests given this year to a national sample of 4th and 8th graders in reading and mathematics.
But average charter scores also closed most of the gap in 4th grade reading and narrowed it slightly in 4th grade math. That was enough for the National Alliance for Public Charter Schools, a Washington-based pro-charter advocacy group, to hail the scores as evidence of “real progress for charter schools.”
The American Federation of Teachers, which tangled with charter supporters repeatedly last year over earlier NAEP results, said the latest scores confirmed that “charter schools continue to lag behind regular public schools.”
Both sides agreed, though, that their conclusions were based, at least in part, on differences that the National Center for Education Statistics, the branch of the U.S. Department of Education that administers NAEP, does not regard as statistically significant. That is largely because of the relatively small number of charter students who were tested.
Yet researchers defended the effort to interpret scores from the congressionally mandated tests, which are seen as the most reliable national yardstick of student achievement in core subjects.
“Because the numbers are small, we’re probably not going to get statistical significance on charter gains,” said Bryan C. Hassel, a Chapel Hill, N.C.-based consultant who analyzed the NAEP results for the charter alliance. “So we have to look at the face-value numbers and then just be cautious about how we interpret them.”
Overall, the NAEP results, released Oct. 19, showed essentially flat scores or modest gains. (“NAEP Gains Are Elusive in Key Areas,” Oct. 26, 2005.)
The charter alliance cited 4th grade reading scores as particularly encouraging. Charter students’ average score on a 500-point scale rose from 212 in 2003 to 216 this year, a period when the average scale score for 4th graders in regular public schools stayed flat at 217.
When those figures were broken down by race and socioeconomic status, the average reading score for African-American 4th graders in charter schools rose 4 points to place them dead even with their black counterparts in noncharter schools, whose average rose only 2 points, to 199. The score for charter 4th graders eligible for federally subsidized lunches climbed by 8 points, to 203. That was the same as the average score for such students in regular public schools, which rose 2 points from 2003.
And the reading score for Hispanic 4th graders in charter schools jumped from 201 to 211, 10 points higher than for their noncharter counterparts, whose average rose just 2 points.
“Charter students and Ķvlog can be proud of these results,” Nelson Smith, the president of the charter-school alliance, said in a statement.
Reliability in Doubt
Both Mr. Hassel and Howard Nelson, an AFT researcher who analyzed the NAEP scores of charter students in 2003 and 2005, said just one of the apparent gains for charter students was considered statistically significant by the NCES: the 8-point rise in reading scores for 4th graders eligible for subsidized lunch.
Meanwhile, many of the differences in this year’s average scores between charter and noncharter students were not considered statistically significant, with some exceptions. Mr. Nelson said most of the statisically significant differences showed charter students scoring lower, including aggregate scores for grade 8 reading and math as well as grade 4 math.
And he defended the AFT’s use of the numbers. “We commented about charter and noncharter differences in 2003, and then again in 2005,” he said. “These differences are a little more likely to show statistical significance because there are so many more kids in the noncharter sample.”
One statistically significant gap in charter students’ favor was the 10-point spread in reading scores for Hispanic 4th graders, with charter students averaging 211 compared with 201 for their peers in other public schools.
The AFT touched off months of debate in August of last year with its analysis of NAEP scores from 2003, when the NCES tested an extra sample of charter school students. Some pro-charter researchers protested at the time that NAEP should not be used to draw conclusions about the performance of charter schools, which are public but largely independent of regular school districts. (“AFT Charter School Study Sparks Heated National Debate,” Sept. 1, 2004.)
One of those scholars, Paul T. Hill of the University of Washington, said last week that NAEP remains “a terrible tool” for comparing charter and noncharter public schools. “The data were never appropriate for the use they were put to,” said Mr. Hill, who directs the Seattle institution’s Center on Reinventing Public Education, “and they still aren’t.”
