Choosing a Standardized Test

What Type of Standardized Test Should Your School Use?

( 11/29/08 )
 

The purpose of this page is to help schools determine what their testing needs really are. The intention for testing will determine what type of test must be used. Each reason for testing demands its own assessment methods, and correct focus, and specific reporting methods. Failure to use the correct assessment method, focus, or reporting method will lead to ineffective or counterproductive decisions. Thus, before choosing which test to use, school administrators, teachers, and parents, must clearly determine what their intentions are for testing.

There are three primary reasons for choosing to use standardized tests.
1. Evaluating Schools and Teachers
2. Determining Student Needs
3. Tracking Students

For each intention, we need to determine what characteristics a test must have to achieve its goals, and what characteristics the test can have to do so efficiently.

 
 
 

Reason 1: Evaluating Schools and Teachers
The Public School Regulatory Act (NCLB) requires that all schools test students and publish the results of their tests. The stated purpose of this regulatory act (NCLB) is to evaluate the quality of the schools and teachers. We need to be able to determine what types of testing can, and can't effectively measure the success of the teaching.
  • To evaluate a school, or its teachers, tests must be differential. Since each student has a different starting point, two tests must be administered, one at the beginning and one at the end. To evaluate schools, tests must measure growth, and to measure growth both a starting point and an ending point must be accurately measured. The success or failure of the school can not be measured by a single final number.
  • Tests to evaluate schools must be well aligned with the standards. The standards to be tested must be made very clear, but not so specific as to reward teaching to the test. If standards are too rigid, testing rewards the schools to dumb-down the curriculum, and teach to the test. If standards are not clear enough, then comparisons of results between schools will not be meaningful.
  • Tests to evaluate schools can be shorter than other tests. Twenty students, each taking tests of 20 questions, will result in 400 data points. For tests designed to test individual students, 50 to 100 questions is traditionally considered sufficient. 20 students, each taking a 20 question test, will provide 4 to 8 times as much data, as well as internal differentials within the data. Keeping the tests short will reduce costs, time lost, and stress on those involved.

NCLB Evaluation

precision reliability in growth

Accuracy error discussion

IQ vs. Multiple Intelligences

Reason 2: Tracking Students
Although many studies have shown that tracking is counterproductive for most students, still most schools track students, and many schools use standardized tests to determine how each student will be tracked. When tracking students, schools frequently fail to distinguish between accelerated learning and high level learning. What characteristics do we need tracking tests to have?
  • Tracking Tests must be more precise than the tracking distinctions being made by the school. For example if you want to divide your class into 3 tracked reading groups, the precision of the test results (2 sigma) should be tighter than the range of tightest group. If you want your middle group to include all students between 6th grade first month and 6th grade 8th month, then your test precision needs to be tighter than 4 months.
    • Parents who doubt the outcome of the testing should challenge the school to demonstrate that the results were precise enough for the tracking decision being made. For example, MAP testing is used by many schools to guide tracking of students. Yet MAP testing can not determine the performance of students middle school students precise to a single grade level.
  • Schools must determine whether they desire a high level program for high achievers, or an accelerated learning program for those who are ahead of the norm in their skills. High level learning and accelerated learning are not the same.
    • If the school wants to place students in a high level program, they must find a test that can identify high level learning styles. High level learning includes the ability to integrate complex information, infer from incomplete information, evaluate the reliability of information, and identify what information is missing that must be searched for. Traditional academic skills tests do not help identify these skills. Many high level learners have skill deficiencies, but still learn at high levels. So, skills tests will no correctly identify high level learners.
      o If the school wants to place students in an accelerated program, they must find a test that identifies the sequencing of skills within the curriculum. Students can then be tested for how far they have moved ahead of the class with these basic skills.
  • Although a traditional test can determine if a student is below grade level, the test can not determine why the student is below grade level. A precise test can tell schools who to put in the below level group, but few tests (none the author knows of) will identify why each student's learning is delayed. Rarely is delayed learning the result of failure to learn prerequisite skills. Delayed learning typically results from nonacademic issues, depression, poor eyesight or hearing, lack of study skills, not being familiar with the language or culture of the test, etc. Thus, although the test may identify who to put in the delayed group, it will not tell the teacher what type of help each student really needs.
We must understand that tracking, and identifying student needs, are not the same things. Tracking is used to facilitate more manageable classrooms, to make teaching easier for the teacher by grouping similar students. This is of value, since most teachers are overloaded. But the tests used to track the students do not identify the real needs of the students, the tests only identify their similarities.

Reason 3: Determining Student Needs
Students have many needs. Each student has an individual set of strengths. Each student has retained or forgotten different sets of knowledge and skills. Contrary to popular notions, high level learning can occur without being preceded by lower skills. With all these variations, identifying needs always constitutes creating individualized lists of assessed needs.
  • Needs Tests must report many data points. Any test that reports one single number (e.g.: RIT score) does not, and can not, determine student needs. Learning is nonlinear and needs can occur in any part of the learning. Take a fifth grade math class for example. One student might be strong at long division, but totally confused about fractions. The student who sits next to her might be strong at adding fractions, but hopeless at long division. Another student might be quite strong at the basic skills of both division and fractions, but have no idea why a person would ever use either long division or fractions. Each student has a different set of needs. Those needs are totally unrelated to the order in which the material is traditionally taught. The only way that a test may help teachers identify those needs would be for the test to generate a report on each specific skill that might matter to the teacher.
  • Needs Tests should determine the cause of the need. In an algebra class one student might get a problem wrong because he did not understand integers. Another student might get the same problem wrong because he does not understand the distributive property. Another student might get the same problem wrong because he does not understand like terms. A good test would determine which probable misunderstanding led to each student's wrong answer. Simply reporting a score, or level, will not identify the real need.
  • Needs Tests will be longer than the other types of tests because more information is needed. Enough questions must be asked for each specific skill to generate a precise report on each skill. With multiple-choice tests, a minimum of three questions must be asked for each skill being checked.
 
 
Above we discussed how identifying student needs has distinctly different requirements than either tracking students, or evaluating schools. We were also reminded that high achievement tracking is quite different from accelerated learning tracking. We should follow that up with a few reminders about testing. 		 

Test follow up and validation
After each round of testing, validation checks should be performed. Do the test scores match teacher observations? Did high achievers get high scores? Did slower students get lower scores? Did the change in test scores over time match the performance the teachers saw in the classroom? Some variation will always occur. But too much variation means that something is not working. The test might be the wrong type of test. The test may be testing insignificant, or inconsequential, skills. Changes in the curriculum might not match changes in the tests.
If a school is testing the students, the school should be able to demonstrate to the parents that they have done basic validation checks on their tests. If a government agency is testing the schools, the agency should be able to demonstrate to the schools that they have performed validation checks on their tests.
 
 
 
How do the reminders above affect you?

School Administrators and State Educational Agencies
Can you demonstrate to concerned citizens that your mandated standardized tests serve their intended purpose? Can you verify that the tests both accurately and precisely provide useful data? Have you acknowledged the concerns of your teaching staff and parents? Has your staff been trained to understand the limits of both accuracy and precision in testing? Can you demonstrate that you have performed validation checks on all mandated tests?

Teachers
Have you been trained to understand the limits to accuracy and precision for the standardized tests you are required to use? Have you observed a mismatch between the test scores and your students' real performance? It is your job to report that mismatch. Nobody else will be aware of the testing problems if you don't report them.

Parents and Activists
Your child will be required to take many standardized tests. Decisions about how to treat your child, the teacher, and the school, will be based on the results of those tests. But, in many cases, the tests can not provide information that is both sufficiently accurate and precise to justify the decisions. In many cases, the teachers know that the test results do not match the students' real performance. You have a right to challenge the decisions. You should be sure your challenge is addressed to the correct level of authority, not the teacher. Who mandated the test? What validation has been performed on the tests? What is the precision and accuracy of the test? Was the correct type of test even used? You have a right and responsibility to ask.
 
     
     
     
 

Return to