The reliability of the holistic grading system for the evaluation of essays at the Preparatory School of Eastern Mediterranean University in North Cyprus

Ankara : Institute of Economics and Social Sciences, Bilkent University, the Teaching as a Foreign Language, 1999.Thesis (Master's) -- Bilkent University, 1999.Includes bibliographical references leaves 60-62.The rating of writing has been the subject of an enormous volume of research. One type of formal writing assessment, used to evaluate non-native writers is holistic scoring. In holistic scoring, raters award a single mark based on a student’s overall performance. This assessment is usually done with a reference to a series of descriptions of different levels of writing ability given on a rating scale which are called descriptors. However, this scoring method may sometimes pose special problems in writing assessment because a scoring system in modem assessment does not always lead to feasible scales. When scale descriptors are not precise enough, a rater’s standards may change during a single rating session or different raters of the same students may not agree on the meaning of scale descriptors so they give different scores to the same student performance. Therefore, the rating scales used should be analyzed in order to find out whether they are workable or not. The purpose of this research study was to investigate whether there are significant differences among teachers in their use of the holistic grading system which is used for scoring students’ essays at Eastern Mediterranean University Preparatory School (EMUEPS) in North Cyprus and to make some recommendations for the improvement of the holistic scoring system. Two types of data were used in this study: essay scores and raters’ opinions. The participants in the study were 10 intermediate teachers. First, the teachers marked chosen 36 intermediate essays 6 from each band by using the six-point holistic scale used at EMUEPS. The grades given to 36 Intermediate essays by 10 different teachers were correlated by using the Friedman Test on the SPSS Program. The result of the Friedman Test revealed that there are significant differences among teachers in their use of the holistic rating system. Then the teachers were interviewed about the holistic scale used at EMUEPS, specific papers they marked and training they had had on the use of the holistic rating scale. The results of the interviews indicated that the raters have common problems in assigning grades to students’ essays. The common problems faced by the raters are as follows: 1. Some essays do not fit into just one band, 2. Some terms used in the descriptors are not clear such as a few, a number and many, 3. Bands are very close to each other so it is difficult to differentiate between them especially bands C and D. After an analysis of the holistic rating scale used at EMUEPS, some recommendations were made for the improvement of both the holistic rating scale and the training the teachers receive on the use of the rating scale. Recommendation for the improvement of the rating scale includes revising and clarifying some of the terms used in descriptors such as a few and a number in order to make them clear for the raters. For the training, the teachers need to go through more scripts and meet certain standards before they take part in actual scoring procedure.Onurkan, GülenM.S