Development and psychometric evaluation of an analytic rubric

INTRODUCTION: Writing is a method to develop cognitive skills in nursing. Evaluation of cognitive development is critical to achieving nursing program outcomes. Our RN-to-BSN program evaluations identified existing grading rubrics did not support student writing development or consistent faculty evaluation. PURPOSE: To develop and empirically test a grading rubric to (1) clarify writing performance expectations throughout the RN-to-BSN program and (2) improve nursing faculty teaching and evaluation consistency. METHODOLOGY: An analytic rubric was developed from a synthesis of current literature and existing rubrics; content was validated by experts. Psychometric testing was conducted by grading six papers from two archived RN-to-BSN courses. Papers were randomly selected, sanitized, and independently evaluated. Three full-time expert faculty and three novice faculty graded using the rubric without instruction. Inter-rater reliability was assessed with Cohen’s Kappa. RESULTS: Overall reliability for the rubric was moderate (kappa=.51 and .42). Content validity was confirmed for purpose, core content dimensions, and external relevance to the population. Consistent with Benner’s (2011) theory, experienced faculty demonstrated inter-rater reliability (kappa=.56 and .33) over novice faculty (kappa=.33 and .33). The results indicated a need for further study. Limitations included small sample size and lack of training for application of the rubric. CONCLUSIONS: Effort, resources, development, and testing of an analytic rubric are significant. A reliable, valid rubric was developed for use by expert faculty. Results indicated novice faculty training is warranted. A valid, reliable rubric highlights variations in faculty level of expertise