Standardized Testing Accuracy and Precision

An Evaluation of NWEA's MAP Testing

( 11/07/08 )

Introduction

Both state and federal (NCLB) regulations have placed great emphasis on standardized testing as a measure of the quality of schools and teachers. Many schools have adopted standardized testing to guide their improvement plans. But are the tests reliable? Are the tests providing meaningful information?

We understand from math, science, and technology that all measures must be checked for accuracy and precision. Both accuracy and precision must be high for a measure to be useful. In this study, we review problems with both the accuracy and precision of MAP testing as a guide for improving education. We will demonstrate that MAP testing does not provide data that is sufficiently accurate and precise to guide differentiation in the classroom.

Parts:

• Philosophical Errors
• Accuracy Errors
• Precision Errors
• Cost-Benefit Analysis

It will be left to the readers to generalize the problems reviewed in this study to other standardized tests.

a short simple version of this analysis

Side Notes: Supporting details

Test related research:

• Testing high stakes testing trivializes learning
• School administrators use data that they don't understand
• SAT only accurate 15% of the time
• Testing leads to narrowing of curriculum
• high scores correlate to superficial learning
• testing produces opposite outcome than desired

* *

how to chose a test

Part 1: Introduction: Philosophical Errors Intrinsic to MAP Testing

Any measure based on philosophical errors or scientifically inaccurate assumptions will produce erroneous results. MAP testing has intrinsic to its design at least two such errors.

• Linear Learning Hypothesis & the RIT Scale: The MAP test is sold on the idea of the linear learning hypothesis. This idea suggests that all students will learn material in the same order, and thus learning made be defined to be levels which are equal and the same for all students and can be measured by the RIT scale. This hypothesis is known to be false. The assumption is so blatantly false that anyone who has worked in education for more than a year should recognize the severity of the error from experience alone. For example, I have had students who were very strong in algebra who could not add fractions or perform long division if their lives depended on those lower skills. The low level skills had minimal bearing on their higher level successes.
• High Achievement vs. Accelerated Learning: Standardized testing fails to distinguish between high achievement and accelerated learning. Testing purports to determine the knowledge and skill levels of my students. But it can not tell me whether my students can achieve at high levels. For example, this very page was originally created for a person who had received high scores in an educational statistics class, but still could not evaluate the reliability of a standardized test from either data or statistical reports. This contrasted sharply to my middle school students who, using just basic chart and graph skills, could evaluate the precision problems in tests. The former demonstrates accelerated skills with low comprehension; the latter demonstrates high achievement with low skills. MAP testing focuses on basic skills, but does not, and can not, focus on high level achievement.

Achievement contrasted to acceleration

Accuracy is about hitting the mark that you intend to hit. In testing, that means translates to, "Did you actually measure what you intended to measure?" If your intention is to measure success in school, then you need to be sure you know how to test characteristics of academic or cognitive success. Here's a listing of significant places where all standardized testing, including MAP, fails to provide accurate information.

• Bloom's Taxonomy of cognitive levels: High level success involves the ability to integrate diverse information, and the ability to evaluate complex information. These two skills are nearly impossible to measure with multiple choice tests. Standardized tests focus almost entirely on knowledge and skills.
• Inquiry-based Learning: A real measure of academic success is the ability to ask big questions, then seek out understanding and knowledge that may answer those questions. This is how scientists, journalists, and engineers work. Since standardized tests focus on low level knowledge and skills, they never successfully test the ability to ask, and seek answers to, big questions. Real work usually starts with the big questions then seeks out the details. Test-based learning starts with the details and rarely ever finds the big picture.
• Reasoning, Problem Solving and Communication: Real success requires a person to reason through, and solve, non-routine problems, then to communicate the validity of the reasoning. Real problems are so complex that they may take hours to weeks to solve. This realization is summed up in the NCTM Standards. However, standardized testing focuses on simple concepts where each question may be answered in less than two minutes. Real cognitive success involves creatively finding solutions to problems that are so complex that a person may take days to solve each problem.
• Expeditionary Education: For a person to grow in skills, they must understand themselves. For those skills to be useful, they must integrate across a large cognitive spectrum, and create a final product must serve a real need. This is the underlying concept of the ELS Design Principles. Standardized tests separate skills into discreet units and discourage educators from integrating those skills into real projects.
• Self-awareness, Marzano's Taxonomy, & Holistic Education: Within just two years of teaching, educators will observe that most of the barriers that impede student learning are emotions, attitudes, or social issues. Even students who have missed many basic skills will perform at high levels if they are given a situation that supports the development of a good attitude, and good habits, for learning. Developing self-awareness then becomes the key element to learning. Standardized tests totally ignore these aspects of learning, even though these may be the most important factors.
• Needs of High Achievers vs. Needs of Low Achievers: Most teachers in test driven environments teach to the lowest third of their class. Most standardized tests, including MAP, are designed around the needs of the lowest third of the students. MAP testing has been praised for its ability to raise test scores of the lowest performing students. However, it is not praised for its ability to support the needs of high performing students. This results from the achievement needs of the high performers being structurally different than the knowledge and skill needs of the low performers. Testing tends to discourage schools from supporting the needs of the high achievers. Below, we will show strong evidence that MAP testing actually discourages schools from supporting the needs of high achievers.

In this list, we see that most measures and motivators of real academic success are either glossed over, or totally ignored, by standardized testing. A major cause of this problem is that real success can not be measured by multiple choice tests. An institution that depends too strongly on standardized tests can produce test scores that imply success. But those scores will not measure real achievement.

• Bloom's Taxonomy
  • levels and types
  • overview
• Inquiry Based Learning
• NCTM Standards
  • problem solving
  • reasoning
  • connections
  • communication
• Expeditionary Design Principles
• Holistic Education
• measures and accuracy

What would a precise growth measure look like?

What graph would precise strand scores produce?

more detail about standard deviation and test reliability

An important consideration in any program is whether the benefits justify the costs. To evaluate this, one must be able to list both the costs and the benefits. What did MAP testing draw from our little school's tight budget?

Costs:

• $3000 per year to test approximately 240 students.
• 6 weeks of schedule disruptions for both teachers and students interfering with projects and planning time.
• Many hours taken away from planning and professional development time for every teacher involved to train teachers on reading test results and planning based on test scores.
• Hours demanded of both the network administrator and school administrators.
• Hours taken from classroom teachers to deal with parent complaints regarding dropping scores. In almost every case, low testing precision was the cause.

Risks:

• Parent anger towards school and teachers because of stagnant or dropping test scores. The test gives high achievers dropping scores more often than low.. Yet the higher performing the student the more likely parents are to be angered by dropping scores.
• Incorrectly tracking students due to significant errors caused by low testing precision and low testing accuracy.
• Shifting emphasis of education from real accomplishments and high cognition, to test preparation and low level skills. This may result in lower interest and performance for all students, especially high achievers.
• Emphasis, resources, and focus drawn from average and high achieving students, to be given to address the needs of raising test scores for low scoring students.
• Distraction from the real needs of students including study skills, hearing & vision problems, emotional and social problems, and other factors that that significantly affect performance.
• Self-fulfilling low expectations for low scoring students.

The costs and risks of MAP Testing are very high, especially for the high achieving students. In just two years of testing, most of the risks identified here occurred within our school. Considering that the costs and risks of testing are so high, what were the identifiable benefits?

Benefits:

• teachers received help determining the knowledge and skill levels of the lowest performing students.

We were not able to identify any other clear benefits of testing. In fact, the only success stories any other school told us about MAP testing was that MAP helped teachers identify the academic needs of the lowest performing students. For a school who's population is average or higher, the costs and risks of MAP testing do not justify the benefit of the test.

Generalizing

We highly suspect that most of the problems discussed here apply to all standardized tests. MAP was the only test that we had time and resources to evaluate. We encourage others to perform similar evaluations on any tests that they are using. Please let us know of your results.