Validity of portfolio assessment: which qualities determine ratings?

CONTEXT The portfolio is becoming increasingly accepted as a valuable tool for learning and assess-ment. The validity of portfolio assessment, however, may suffer from bias due to irrelevant qualities, such as lay-out and writing style. We examined the possible effects of such qualities in a portfolio programme aimed at stimulating Year 1 medical students to reflect on their professional and personal develop-ment. In later curricular years, this portfolio is also used to judge clinical competence. METHODS We developed an instrument, the Port-folio Analysis Scoring Inventory, to examine the im-pact of form and content aspects on portfolio assessment. The Inventory consists of 15 items derived from interviews with experienced mentors, the literature, and the criteria for reflective compet-ence used in the regular portfolio assessment proce-dure. Forty portfolios, selected from 231 portfolios for which ratings from the regular assessment pro-cedure were available, were rated by 2 researchers, independently, using the Inventory. Regression ana-lysis was used to estimate the correlation between the ratings from the regular assessment and those resulting from the Inventory items. RESULTS Inter-rater agreement ranged from 0.46 to 0.87. The strongest predictor of the variance in the regular ratings was quality of reflection (R 0.80; R2 66%). No further items accounted for a significant proportion of variance. Irrelevant items, such as wri-ting style and lay-out, had negligible effects. CONCLUSIONS The absence of an impact of irre-levant criteria appears to support the validity of the portfolio assessment procedure. Further studies should examine the portfolio’s validity for the assessment of clinical competence. KEYWORDS documentation ⁄*standards ⁄methods; education, medical, undergraduate ⁄*methods; teaching ⁄*methods; educational measurement ⁄*standards ⁄methods; observer variation