Investigating the use of vector analysis to assess students’ understanding

CONTEXT Many methods have been proposed in the study of linguistics for the representation of words and sentences. Most classical methods are symbolic and consist in things like dictionaries, thesauri, ontologies and syntax trees. Another approach is to represent words and sentences via the use of high dimensional vectors, which capture the distributional statistics of words and sentences. One application of representing words as vectors is to automatically evaluate text, which can further be applied to the assessment of students’ text-based answers. PURPOSE This study investigated approaches to automatically analyse student responses to questions in the signal processing domain. APPROACH We investigated vector analysis approaches to capture various semantic and syntactic features of words, such that these representations can be compared and scored in a graded fashion, as distinct to simply true/false or same/different. The approaches used in this study can be trained in a semi- supervised fashion, where minimal human input is typically required. RESULTS The data investigated in this study consisted of student responses to short-answer questions in text form with associated metadata indicating the correctness for answers. Difficulties encountered when automatically assessing student short answers, either for correctness or knowledge gaps, were a) variations in vocabulary b) variations in grammatical structures c) precisely determining when specific concepts occur and don't occur, and d) relevant concept modifiers that may alter the assessment of the short answer. One element—important for addressing these difficulties— is how words and sentences are represented in short-answer question responses. CONCLUSIONS The study described in this paper focused on vector space representations for text. We recommend the development an agile methodology to be employed so that regular outputs be produced and sent for comment, which can then be used to inform further work. We suggest the best approach is to make use of a combination of methods including the many classical Natural Language Processing (NLP) techniques such as part of speech (POS) tagging, and phrase chunking.