Rasch Analysis 1: A tool to evaluate efficacy and quality of exams

How do you know you've written a good exam (multiple choice)? Does it measure the student learning youdesire? Is it redundant? Does it balance difficulty of questions?Historically, academics have used various approaches to answer some or all of these questions, ranging fromprofessional experience, assessing the distribution of grades or more sophisticated approaches such asexploratory factor analysis. Although these approaches can provide information on the performance ofstudents, they seldom provide information on the performance of the exam.Rasch analysis (Rasch, 1960) is a statistical approach that allows examination of dichotomous (yes/no) andMCQ tests. It transforms the raw performance of students (termed 'persons') and exam questions (termed'items') into a common scale of measurement called 'logit'. It therefore allows the academic to compare theperformance of students versus that of the questions using a single understanding of 'performance'.The Rasch approach differs strongly from other approaches. As opposed to other statistical approaches Raschanalysis is suitable to measure a single 'construct' or constructs that are likely to be correlated. Whenmeasuring constructs that are not correlated, Factor Analysis is a more appropriate statistical approach(Tennant and Pallant, 2006). Furthermore, traditional statistical approached tend to fit a statistical model tothe data at hand. By contrast, Rasch fit the data to an expected probabilistic model. Unlike its counterparts,once validated, this approach is no longer sensitive to small student numbers, and is also insensitive to achange in student affordances (Bond and Fox, 2007).