Utterly Meaningless

Can we safely assume that SAT and NAEP scores have not increased 10-fold in the last three decades? I firmly believe that whoever named the VA accountability tests the SOL had a perverse sense of humor.

This picture may be worth a thousand words, but they’re a thousand words of nothing. Without knowing the scale on which these variables are measured, comparing such disparate measures is completely useless. At best, the graph is unintentionally misleading, and at worst it’s willfully fraudulent.

An example tangentially related to the data presented in the graph: the US Census Bureau reports that the population of Virginia changed from 7,078,515 to 7,567,465 between 2000 and 2005 (quickf...0.html). This gives a total percent change of 6.9% over those years. But this is dependent on fact that the change is small relative to the large size of the population. The exact same change in absolute size (7078515 – 7567465 = 488,950) would appear to be massive if you drop the “7 million” part: going from 78,515 to 567,465 is a seemingly incredible jump of 634%!

So if the enrollment changed by some tens of thousands compared to millions of pupils, and funding changed by tens of millions compared to hundreds of millions, the percentage increase may seem to be vastly different and highly important. The information is completely useless, however.

This is why any statistician / economist / analyst who was even vaguely awake would standardize the results – for instance, depicting them as funding per student.

In fact, after I wrote that, I did a quick Google, which lead me to an NEA report (nea.or...gs.pdf). Table H-9 of that report give the expenditures for public K-12 schools per student in fall enrollment (2002-2003) as $8,666 per student, 17th in the country and only 7% above the national average. In 2003-2004, Virginia had slipped to 20th in the nation with expenditures of $8,705 per student, or only 5.4% above the national average. Thus, expenditure per student increased only 0.4% in that time, and that was outstripped by growth in other parts of the United States.

All in all, a diferent thousand words than the picture above…

None of this meant to be a defense of, or attack upon, the school system in Virginia, the quality of which I know nothing about. But as someone who analyzes data for a living, I’m appalled at how misleading and useless that graph is.