The New York City Department of Education produces an annual progress report for every school in the city for which there exists sufficient test data.  The DOE website gives a good description of the reports.  The DOE also makes available an Excel workbook with all of the results for the year 2007-08.

I compared the results for the 46 charter schools with the 1,307 traditional schools.  Here is the workbook with my additional calculations and results.

Some conclusions:

1. With respect to overall letter grades (i.e. A,B,C,D,F), charter schools performed slightly better than traditional schools, but the results are not statistically significant.

2. With respect to the overall numeric scores (which range from 9.6 to 106.5), charter schools performed significantly better than traditional schools.  The difference between this result and the letter grade result is explained by a disproportionate number of very high scores for charter schools.  This information gets “lost” in the large number of schools that get A’s.  (Almost 40% of schools received an overall grade of an A.)  The following graph gives a good picture of the situation by plotting the different frequencies of scores normalized by the mean and standard deviation of the overall school population.  Notice that very high scores are much more common for charter schools.  Notice also that there are a disproportionate number of charter schools with very low scores.  This is attributable to two charter schools that received amongst the worst overall scores in the city.
Charter Versus Traditional Graph

3. Charter schools performed significantly better than traditional public schools in all three components of the overall score: environment, performance, and progress.  The most impressive average score was actually the “environment” component which is based on attendance figures and surveys given to teachers, parents, and students.  In other words, charter schools performed particularly well on the metric that has nothing to do with test scores, but rather attempts to represent teacher, parent, and student impressions.

charter-versus-traditional-environment-graph-2

4. I looked at the performance of various “support networks” for schools.  In New York City, traditional public schools can select one of a number of different support options to help to manage their school.  Only two of the support organizations had statistically significant results: the Empowerment Schools showed small but statistically significant out-performance; the schools using the Knowledge Network Learning Support Organization (KLSO) showed larger and statistically significant under-performance.

UPDATE: I added the number of charter schools and traditional public schools in the text of the post based on a reader comment.