Differences in two evaluations of answers to a conceptual physics question: a preliminary analysis

In their exploration of student understanding of gravity, Sharma et al. (2004 and 2005) discovered a discrepancy between phenomenographic analysis of student answers to one short examination question and the distribution of marks for the same question between two first-year university physics classes. We report on a preliminary investigation of factors which, we hypothesised, may have contributed to that discrepancy. Additional analysis and evaluation of the original set of answers included a detailed study of the use of physics terminology (PhysicsSpeak) and diagrams in the answers, with the aim of discovering how those features may have affected the marks. A selection of the answers was reviewed for evidence of other characteristics which may have influenced the marker. The views and recollections of the original marker were also recorded. There is no single explanation for the discrepancy, but we found that the use of diagrams has a significant effect on marks, whereas the influence of PhysicsSpeak was weaker than expected