It takes a write a test!

It takes a write a test!

From Memberanda, Spring 2012 

This winter, The Enrollment Management Association convened two Item Writers’ Workshops, one in Dallas focused on the Elementary SSAT (see p. 15) and one in Sarasota focused on the Upper and Middle Level (formerly Lower Level) SSATs. A total of 111 teachers representing 73 independent schools and two post-secondary institutions came together to network and learn the art of test item writing. The Enrollment Management Association is extremely grateful to Dr. Steve Robinson, Executive Director of SAIS, who is serving as Chair of the Elementary Examination Review Panel. This panel is not only reviewing test items, but also making important recommendations about the scope and parameters of this new assessment.

This community-based test development approach helps ensure that the SSAT reflects what is being taught in classrooms today; it also involves teachers in the test writing process. As The Enrollment Management Association Chief Testing Officer Dick Rovinelli emphasized, "the chance to engage independent school teachers is a win-win." Perhaps Corie Fogg, Fay School (MA), said it best: "The workshop illuminated how standardized tests are created, the comprehensiveness of psychometrics, and what makes an appropriate, engaging question. In addition to these insights, our conversations have been helpful for me as an educator; we’re talking metacognitively about assessments, and it’s assisting me as I create diverse, quality questions for my students." Glenys Quick, Hockaday School (TX), echoed Fogg’s sentiments: "The energizing and enlightening experience not only sharpened my awareness of the worth of the multiple-choice test, but it influenced my effectiveness as a teacher in designing fair evaluations."

Although those participating in the conferences are experienced teachers and administrators, training was required to distinguish the difference between item writing for a standardized admission test and for assessments used in a typical classroom. In Sarasota, Donna Tuttle, The Enrollment Management Association Examination Editor, offered the nuts-and-bolts presentation about item writing, answering important questions such as, "What is a fair item?" (Answer: An item that doesn’t unfairly disadvantage a student due to circumstances beyond their control). Steve Cohen, Allen-Stevenson School (NY), said "Designing an admission test is much harder than I imagined. It’s a great responsibility to construct a test that gives us the information about students we need for admissions, and equitably allows all students the opportunity to give us that information."

Catherine McClellan, a psychometrician formerly with ETS, brought statistics to life, describing psychometrics as, "The science of measuring psychological attributes, such as intelligence or understanding. Tests are one way to assess these intangible qualities. A psychometrician’s work ensures that each test is reliable and that all test results are valid." Ivy Carroll, Westminster School (GA), said "I’ve always wondered about the psychology behind testing and am impressed with the extent of statistical research that goes into each and every test question."

McClellan reminded the groups that the SSAT is an admission test and, as such, one is looking to it to test (and measure) a specific set of knowledge and skills that is relevant to a variety of educational programs. Contrary to a typical test item that a teacher would write for a test, an ideal standardized test item is one that half of the kids get right. Students who can decipher distractors well, allowing them to analyze questions to get a higher test score, are referred to as being "test wise."

McClellan and other testing experts also provided a useful overview of common test vocabulary, along with definitions for important features like test validity and reliability. The groups learned that it is no small feat to write the item distractors (i.e. the wrong answers). In order for an item to be acceptable, all of the distractors must be functioning at the same rate; in other words, each distractor must "distract" about the same number of students. Christina Han, Packer Collegiate (NY), noted "Classroom teachers in general have a bias against standardized tests. This weekend, I realized that the process is child-centered. It has made me a convert."

Armed with a file packed full of assessment instructions, further explanation and sample questions, and a book titled, Writing Test Items to Evaluate Higher Order Thinking by Thomas M. Haladyna, participants were eager to get to work. Groups convened in two rooms, broken into smaller groups by subject and grade level. The Enrollment Management Association staffers circulated to be sure each table was well-equipped and ready for the task at hand. By the second day, the need for guidance was greatly diminished as the writers easily developed a rhythm and started churning out questions, keeping their review panels quite busy. Jessie Lytton, Packer Collegiate (NY) said of the experience, "I’ve never pulled apart or teased apart words or groups of words to the extent that we’ve done here."

As a result of this work, The Enrollment Management Association has established a group of informed and dedicated test item writers, who are exceptionally prepared. Many thanks go to the teachers and presenters, as well as the schools that so generously lent us their exceptional educators. As Susan Pirruccello of Providence Day School (RI) described, "I’ve made a 180. I came here as a non-test person, and now I realize the time and consideration that goes into a test and making the right question work." Tom Woodward, Westtown School (PA), said, "I, like many others I think, came not quite knowing what to expect or what to do, and left feeling well enough trained to be able to continue the work and eager to learn more so that I can be more fully involved… When we began looking at questions, we took a great deal of time on each one, but by the end of our sessions, our discussion had sharpened and required less time to achieve the same results… We are all looking forward to working together in the future."

The Enrollment Management Association begins pretesting for the Elementary, Middle, and Upper SSAT this month and, thanks to the generous commitment of member schools, this pretesting will extend from April through June. We anticipate that thousands of students enrolled in independent schools will complete the item writing process that began this winter by ensuring that the questions perform as intended. In addition, each Examination Committee will review pretested items and only those items which pass their review will be selected for inclusion in the The Enrollment Management Association examinations. The Enrollment Management Association plans to roll out the Elementary SSAT for third and fourth grade students (applying to grades four and five) this fall. In addition, questions written by item writers for the Middle and Upper Level SSATs will appear in tests this fall.

Elementary SSAT Item Writers' Schools:

Allen-Stevenson School, NY
Elmwood Franklin School, NY
Garrison Forest School, MD
Hockaday School, TX
Independent Day School, CT*
Katherine Delmar Burke School, CA
Landon School, MD
Meadowbrook School, MA*
Newtown Friends School, PA*
Packer Collegiate, NY
Punahou School, HI
Saint Andrew’s School, FL
St. Mark’s School, TX
St. Stephen’s and St. Agnes School, VA
Stevenson School (Carmel Campus), CA*
Town School for Boys, CA
Westminster School, GA*
Westtown School, PA*
* Members of the Elementary Exam Review Panel

Middle & Upper SSAT Item Writers' Schools:

Abington Friends School, PA
Adda Clevenger School, CA
All Saints Academy, FL
Allen-Stevenson School, NY*
The Barnesville School, MD
Beacon Academy, MA
Carrollwood Day School, FL
Chapin School, NJ
The Congressional Schools of Virginia, VA
Convent of the Sacred Heart, NY
The Country Day School, ON
The Country School, MD
Dwight-Englewood School, NJ
The Episcopal Academy, PA*
Far Hills Country Day School, NJ
Fay School, MA
Galloway School, GA
Garrison Forest School, MD*
George Stevens Academy, ME
Georgetown Preparatory School, MD
Gulliver Schools, FL
Harford Day School, MD*
Hawaii Preparatory Academy, HI
The Hockaday School, TX*
The Hotchkiss School, CT*
The Landon School, MD
MacDuffie School, MA*
Marin Country Day School, CA
Marvelwood School, CT
Milton Academy, MA
Morehouse College, GA
Newtown Friends School, PA
Norwalk Community College, CT
Norwood School, MD
Oakwood Friends School, NY
Oldfields School, MD
Pine Point School, CT
Providence Country Day School, RI
Punahou School, HI
Roeper School, MI*
Sanford School, DE*
Sonoma Country Day School, CA
St. Catherine of Siena, KY
St. John’s Preparatory School, MA
St. Michael’s College School, ON
Storm King School, NY*
Taft School, CT
Tampa Preparatory School, FL
Tilton School, NH*
Town School for Boys, CA
Trinity School, GA
The Webb School of Knoxville, TN*
Westtown School, PA*
Whitefield Academy, GA
Woodlawn School, NC
Wyoming Seminary, PA
* Members of the Middle & Upper Exam Review Panels


Previous Article
Portrait of an Admission Committee
Portrait of an Admission Committee

From Memberanda, Spring 2012 At the heart of any admission process is the selection of students, and many i...

Next Article
Early Lessons in Early Childhood Assessment
Early Lessons in Early Childhood Assessment

From Memberanda, Spring 2012  The Enrollment Management Association has embarked on a study to better under...