The appropriateness of professional judgement to determine performance rubrics in competency based assessments

© 2003 Andrea J. BatemanWhen competency based training and assessment was introduced into the vocational education and training sector (VET) in Australia in the early 1990s, there were no clear policy guidelines about whether learners' levels of performance should be assessed and reported or whether assessment should be conducted and reported using a dichotomous system of competent/not yet competent. Across the literature various terms are used that describe or equate to the practice of differentiating performance of competence included: 'performance levels' (Smith 2000), 'levels of performance' (Thomson, Mathers & Quirk 1996), 'levels of competency' (Dickson & Bloch 1999), 'merit recognition practices' (Quirk 1995) and 'levels of achievement' (Strong 1995). Scoring, percentages, performance levels, profiling, alpha-numerical identifiers and additional criteria are examples of such approaches to 'grading'. Standards referencing is considered a form of criterion referencing where levels of performance are defined along a continuum of increasing competence and used for interpretive purposes to infer a competency decision (Glaser 1963 & 1981). The development of these levels of performance poses difficulties in terms of determining the validity of judgements for assessment practitioners. Williams & Bateman (2002) considered that determination of the levels of performance is inherent in the competency and therefore the scoring rubric is best developed by the subject expert/assessment practitioner. Much has been written regarding standards setting methods (such as Angoff, Nedelsky, Jaeger, Ebel) however recent literature (Berk 1995 & 1996; Hambleton, Jaeger, Plake & Mills 2000; Hambleton & Plake 1995; Plake & Hambleton 2000; Putnam, Pence & Jaeger 1995) has queried the applicability of such standard setting methods in relation to complex performance assessments. The recent research paper by Bennett (1998b) summarised the key issues and practices associated with setting performance standards in examinations and their implications for standard setting in the NSW Higher School Certificate program. Bennett proposed a multi-stage rather than a single approach to standard setting procedures using technical/subject experts. A similar standard setting procedure was used in this study. This research used item response theory to investigate the appropriateness of professional judgement to predict or determine performance rubrics within a graded competency based assessment framework. This study also investigated the criterion and construct validity of the analytical scoring procedures developed. The context of the research was that of the Public Services Training Package (ANTA 1999)