In ten years, will historians look back at the 30-year effort to get teachers to interpret test data as a failure? I'm impatient. I don't want to wait ten years. Although I'm no historian, here's my verdict. Yes, it was a failure. This long push to make teachers do "data-driven decision-making" has flopped. But of all those who share responsibility for this flop, teachers are the least culpable.
Evidence of its failure is everywhere. The tests most teachers favor are the ones they create. Those teacher-created tests often reveal the least about the degree to which students have mastered the subject matter they've studied. State tests are disliked, disrespected and misunderstood. Proof of how often misunderstanding occurs is featured in Jenny Rankin's research. She discovered in 2013 that when 211 California teachers interpreted state test results using conventional reports, they were wrong between 61 to 89 percent of the time. Shocked? Then take 15 minutes and read this summary of her research published in the newsletter of the California Council on Teacher Education (pages 14 to 21).
How did it get this bad? Many causes led to this sorry state of affairs. But one cause was district leaders transferring the responsibility of analysis and interpretation from assessment directors to teachers. That job belongs in the district office in the hands of a well prepared assessment director. Teachers should be assisted by district office analysts in the same way that doctors are assisted by x-ray and MRI techs. Teachers are practitioners, like doctors. And if they are to do their jobs best, they need to be served by technical teams that can build evidence and interpret it for their use.
In my view, assessment directors are like x-ray and MRI techs. They apply tests to patients, produce artifacts (both data and a digital image) which they interpret and return to the doctor. The quality of their interpretive judgment is a key part of the diagnosis. The doctor may take the tech's interpretation at face value, modify it or even disagree with it. In the end, it is the doctor's responsibility to diagnose the patient's ailment. The doctor needs to understand the test's limits, of course – its degree of imprecision (fuzzy x-rays are ambiguous), for example, or in the case of a diagnostic test for COVID-19, the test's rate of false negatives and false positives. But the techs build the evidence and interpret it. The doctors take it from there.
Their split roles are baked into the medical profession. Their respective specialties require different kinds of knowledge. As a result, their paths of professional preparation are vastly different. Their ways of analyzing evidence differ. If both the techs and the doctor are well prepared, the likelihood of a correct diagnosis is highest.
In K-12, this clarity of roles between analysts and practitioners has been blurred. So the specialized contribution of the assessment director is lost. I believe that the sad result is that assessment directors are understood by most superintendents to be people who administer tests – not interpret them. And most superintendents also expect teachers to interpret test results assisted mainly by reports that reduce test evidence to a red/yellow/green signal system. Learning is not that simple, and measuring evidence of learning is not reducible to traffic lights.
Imagine if the medical profession dealt with blood tests or x-rays or MRI scans the same way that K-12 deals with assessments. There would be no techs adding their interpretive judgment to the meaning of these tests. Instead, doctors would send their patients to a lab. The lab assistant would just connect the patient to the machine, and then send the patient home. Later the doctor and the patient would receive a lab report, with a simplified, automated interpretation of the meaning of the test. The doctor would face the diagnostic challenge without the benefit of a technician's judgment. Given how ambiguous test evidence can be, and given the rate at which hospital-based doctors misinterpret diagnostic evidence (about 15 percent of the time), the loss of a lab tech is no small matter.
I am championing the cause of assessment directors. It's time to put some pride in your stride and resume the defense of your profession. Your judgment of the meaning of assessments is desperately needed. You face many foes. You're up against the anti-testing crowd. You also face adversaries who favor tradition over evidence-based judgment. But you're also up against those who favor making teachers into analysts. They're wrong. You're the analysts. They're the teachers. Let's get those boundary lines clear, and see if the cause of instruction is better served.
About the Author: Steve Rees
Leader and founder of the company and its K12 Measures team. Author of curriculum for our courses and workshops. Analyst with a special talent for visual interpretation. Loves leading exploratory data analysis missions. Hates waste of time, money and human effort.