Chester E. Finn, Jr.

Watchdogs are a welcome and necessary part of any community and everyone deserves to be watched, scholars and think tanks included. But to be effective, watchdogs must be impartial. And to be viewed as impartial, one must diligently conduct oneself accordingly.

The recently launched Think Tank Review Project (TTRP), based at Arizona State University, touts itself as such a watchdog, barking and nipping at think tank studies and reports that, according to project co-director Kevin Welner, are "little more than ideological argumentation dressed up as research."

We at the Thomas B. Fordham Foundation always thought it passing strange that so opinionated a group would set itself up as impartial arbiters. After all, TTRP is a collaborative effort of the Education and Public Interest Center at the University of Colorado , and Arizona State University 's Education Policy Research Unit. The latter is run by Alex Molnar, who has for years opposed charter schools, privatization, and half (or maybe three-quarters) of every other education reform and reform idea worthy of attention. How can a contestant also play umpire? How can a butcher be fair to vegetarians? How, for that matter, can a leopard change his spots?

We predicted that this group would climb right into the ring and start sparring, perhaps with gloves, perhaps bare-handed. Now we have evidence that this is the case. And guess whom they're sparring with? Us.

Either TTRP is not impartial, which is what we believe, or they can't read. The latter seems unlikely but in Arizona , home of whole-language guru Ken Goodman, one can never assume.

Whether bias or limited reading skills is at work, or possibly both, TTRP's May 19 reviews of two recent Fordham studies are disreputable.  Especially when one reads the two reviews and the press release dated May 19 by TTRP challenging the validity of Rebecca Gau's Trends in Charter School Authorizing, and Dick M. Carpenter II's Playing to Type? Mapping the Charter School Landscape.

Let's begin with Dick Carpenter's Playing to Type? Mapping the Charter School Landscape , which was released on May 2. (You can read it online at www.edexcellence.net/doc/Carpenter%20ProjectV2.pdf.)

This study had two goals: 1) To create a useable typology of charter schools (traditional, progressive, vocational, etc.), and 2) to see if there were discernable differences among types with respect to student achievement.

TTRP reviewer Gary Miron praised Carpenter for his "substantial and commendable effort" in developing this first-ever typology and placing schools in it. He turned attack dog thereafter, however, accusing Carpenter of exerting "less effort and rigor" when "analyzing the differences between the school types."

Miron bites, for example, at Carpenter's finding that low-achieving charter school types make greater gains over time. "It is important to note," Miron writes, that "schools with especially low test results can more easily show gains than schools whose results are already at or above state standards."

Duh. Carpenter says precisely the same thing. From page 10 of the report: "The gain score analysis is also problematic. The relatively small gain scores and the general lack of statistical significance between them undermine the ability to ascribe differences strictly to school type." Carpenter only points out what his data show and draws no sweeping generalizations. A pity, said Fordham's public-affairs team; his study would have garnered more attention if he had said more than his data permit.

Miron also knocks him for using performance data from one school year to create the typology, and from a different year to grade the results. "Different years are used for performance data than for demographic data," Miron writes in his review.

Carpenter explains why on page 7: "It was not possible to use 2001-2002 school year data, as we did when analyzing school demographics. This is because some states changed assessments ( Texas switched from TAAS to TAKS in 2003), while others did not test at every grade level, severely limiting sample sizes."

Why Miron doesn't bother to note this is unexplained. Could it be he was too eager to undermine what little descriptive data Carpenter had?

Miron also accuses Carpenter of being less rigorous in the second half of his study. We contend that Carpenter was simply careful not to read more into the data than was there. He would have loved to be more rigorous-but the available data would not allow it. And we worked overtime to avoid overhyping the results. Here's what the study's foreword said: "available data were less than ideal, and, as [Carpenter] pointed out to us, few of his findings withstand a rigorous test of statistical significance." We can imagine think tanks "dressing up ideological arguments as research," but in the history of think tankery, no one has ever been more careful to dress down their findings.

Now turning to TTRP's hatchet job on Rebecca Gau's Trends in Charter School Authorizing, also published by Fordham and released on May 2. (You can find it on-line )

Gau set out first to create a typology of charter authorizers, then to see if she could identify variations in how they perform their duties. A principal conclusion of her report is that some types of charter authorizers appear better able to practice quality authorizing than others. She points to nonprofit charter authorizers and independent chartering boards (ICBs) as examples.

TTRP reviewer Ernest R. House begs to differ. Using the charts in the report, House notes that "The ICBs rank poorly, and the nonprofits do about the same as SEA, LEA, and municipal [authorizers], all of which come in behind the higher education authorizers on this data analysis."

Unfortunately, he didn't bother to get the whole picture. House complains that both nonprofits and ICBs were rated "strong" in the report because they make good use of data-driven decisions when re-authorizing a school. He then deduces that Gau failed to grade higher education as "strong," even though in Appendix B, "Authorizers' Use of Data Models," higher education authorizers score as well as, or better than, ICBs and nonprofits.

House has part of a point. That chart has higher education authorizers scoring better. But he fails to consider the whole picture. "To receive a strong rating," Gau said in a phone interview, an authorizer scored well not only for using data models in its decision to reauthorize, but for "actually collecting school data themselves and for being rigorous in their analysis. When all three components are considered, the ratings stand."

Apparently, House was so eager to point out this contradiction that he failed to note Table 1 on the second page of the report, which identifies the seven separate questions used to determine a type's grade for data-driven decisionmaking. When considering this set of activities in its entirety, Gau is correct. (The entire data set of responses to all of the questions in the survey are available at www.edexcellence.net .)

House also takes her report to task for its low response rate: "The survey return rate was 33 percent. This is a low return, especially since the responses were not representative across the primary categories of analysis."

What he fails to cite is page 8 of Gau's report. "For six of the seven [authorizer] types, all or most of the authorizers responded. Together, these types authorize 69 percent of the schools represented in the survey."

Moreover, she continues, "the lowest response rate (24 percent) came from local school districts (or Local Education Agencies-LEAs).. Because districts make up an overwhelming majority of the nation's authorizers, even with their low response rate they still represent a majority of survey respondents (64 percent). Did House not read this? Did he not understand it?

Still more disturbing is House's comment on the utility of Gau's report: "The findings of this study would best be used as a starting point for other investigations into the important problem of how charter schools should be authorized. The study is not conclusive enough to guide policy, though it might point to future directions for consideration."

Gau's response? "We agree," she said by phone. As her Introduction makes clear: "The primary aims of this study are descriptive, i.e., to explain what charter authorizers look like, how they behave, and the kinds of resources they have at their disposal."

In this report, too, we went to considerable lengths-the heck with media attention-to explain that we scaled down our goals due to data problems encountered along the way. From our Foreword: "We hoped that we could. make a direct link between authorizers and their schools' student achievement. Alas, we quickly learned from some of the best researchers and savviest methodologists in the country that this linkage is not yet possible."

Such blatant misreadings of Carpenter's and Gau's research suggest that TTRP is more interested in shutting the lid on honest research into the charter movement than in offering serious analysis and critiques of Fordham Foundation and Institute publications. Impartial arbiters they surely are not. Furthermore, whoever is funding their endeavor should know that the reviewers added no value. All the problems and limitations they noted had already been mentioned by the researchers themselves. Which makes you wonder what this enterprise is really about.

Chester E. Finn, Jr. is president of the Thomas B. Fordham Foundation and a senior fellow at Stanford's Hoover Institution.