*************************
This is a response to my posting of "Study shows charter schools better" by George Archibald. Sent with the permission of Tim Pelton, who wrote it.
*************************
"Study shows charter schools better"

What do you mean by better?

Snapshot assessments of student achievements may be able (a very qualified
maybe) to describe the distribution of achievement level at various schools, but it is inappropriate to use that information to compare school performance.

Schools are complex places within even more complex contexts, and should not be compared against anything but their own prior performance in support of professional development and meeting the needs of their students. (NCLB notwithstanding)

In order to adequately examine student or school performance on any dimension you first have to have a valid measurement instrument that has at its foundation a) a theory of learning and growth, and b) a calibrated measurement scale (or reasonable approximation to such) that supports aggregation. Such a measurement scale allows for the comparison of achievement level estimates independent of location on the scale and supports the meaningful description of gain.

The NWEA put out a very interesting report this past April (McCall, Kingsbury & Olson, 2004) that effectively demonstrated that state snapshot tests are inappropriate for comparing schools - see especially figure 9. The minimal correlation between success on the snapshot indices (generally not measures) and the NWEA MAP RIT scale gain estimates provided strong evidence that academic growth in schools is not adequately estimated by the traditional snapshot results. I would argue that the widespread misuse of these snapshot indices by the media, government, parents and other groups (including the charter schools) as comparators for schools makes them invalid.

Charter schools and private schools may choose to emphasize performance on these state assessments - forsaking other curricular areas in order to maximize their performance on a limited subset of the curriculum so that their relative ranking will appear higher. The distribution of achievement levels for students attending charter schools (or any set of schools) cannot be assumed to be the same just because the location and racial composition - or even SES - is found to be similar. Because so many factors are left uncontrolled it is impossible to use quantitative data in isolation to assert that one school is performing "better" or is "better" than another.
Please examine the "parable of the Gyms" presented in McCall et al 2004.

When as a teacher you are put in a position of misusing assessments to support comparisons between your class or school and other classes or schools - rather than to support instruction - you are being asked to participate in a process that is unsound and rife with potential for misuse and abuse.

I would further argue, that whenever a system uses snapshot estimates of achievement (whether the results are indices of curricular coverage or measures of growth along well articulated theories of learning and growth) for anything other than professional development and instructional guidance and support then that system is misusing these assessments. Misuse of assessments invalidates them (see Messick). When teachers, students, parents, administrators and politicians are set at odds to one another through this misuse of assessments, then we truly have a dysfunctional system.

Tim Pelton

McCall, M.S., Kingsbury, G.G. & Olsen, A.. (2004). Individual Growth and School Success. Lake Oswego OR: Northwest Evaluation Association.
http://www.nwea.org/research/national.asp
**********************************
--
Jerry P. Becker
Curriculum & Instruction
Southern Illinois University
Carbondale, IL 62901-4610
Phone: (618) 453-4241 [O]
(618) 457-8903 [H]
Fax: (618) 453-4244
E-mail: jbecker@siu.edu