The use of e-rubrics for competence assessment in university students. Study on reliability of the instrument.

[ES] La evaluación es un elemento curricular que ha experimentado una importante transformación en la Educación Superior como consecuencia de diferentes procesos de reforma educativa. La investigación educativa ha sido muy fructífera en este campo dando lugar a un cambio de enfoque: evaluación ‘para’ el aprendizaje, sostenible, orientada a la valoración de las competencias y con un papel activo del estudiante. En un contexto tecnológico, la e-rúbrica se ha establecido como un instrumento útil para la realización de un proceso de evaluación adecuado a las actuales necesidades y demandas educativas. Los resultados de investigación muestran sus potencialidades para la autorregulación y autoeficacia del estudiante, la calidad de la evaluación y la mejora de la docencia. Pero es imprescindible elaborar e-rúbricas que cumplan con criterios de fiabilidad y validez. Con tal fin se diseñaron dos rúbricas para su aplicación en una asignatura de Grado en un grupo de estudiantes y se analizó la fiabilidad de los instrumentos. La fiabilidad de la rúbrica se ha evaluado a través de su consistencia interna para lo que se ha recurrido al coeficiente Alpha de Cronbach. Se obtuvieron unos coeficientes de fiabilidad considerados aceptables, aunque mejorables. Para la correcta aplicación de la e-rúbrica se considera fundamental la formación de estudiantes para el uso del instrumento en la autoevaluación y la evaluación entre pares. El artículo incluye las dos rúbricas utilizadas en el estudio.[EN] Assessment is a curricular element that has undergone a major transformation in higher education as a result of different processes of educational reform. Educational research has been very fruitful in this field resulting in a shift of focus: assessment ‘for’ learning, sustainable, competency based assessment and an active role of the student. In a technological context, the e-rubrics have been established as a useful tool for conducting a proper evaluation process to current educational needs. The research results show its potential for self-regulation and self-efficacy of students, assessment quality and teaching improvement. But it is necessary to develop e-rubrics that meet criteria of reliability and validity. To this end two rubrics were designed for use in a subject’s degree in a group of students and the reliability of the instruments was analyzed. The reliability of the rubric has been assessed through internal consistency to what was used Cronbach Alpha. Reliability coefficients were obtained considered acceptable, but can be improved. Specific training in the use of the instrument for self-assessment and peer assessment, it is considered essential for successful implementation of e-rubrics. The article includes two rubrics used in the study.Valverde Berrocoso, J.; Ciudad Gómez, A. (2014). El uso de e-rúbricas para la evaluación de competencias en estudiantes universitarios. Estudio sobre fiabilidad del instrumento. REDU. Revista de Docencia Universitaria. 12(1):49-79. https://doi.org/10.4995/redu.2014.64154979121Accino Domínguez, J. A., & Lozano Rosch, E. (2011). Servicio federado de eRúbrica para evaluación formativa. RedIRIS: boletín de la Red Nacional de I+D RedIRIS, (90), 46-54.Allen, D., & Tanner, K. (2006). Rubrics: Tools for Making Learning Goals and Evaluation Criteria Explicit for Both Teachers and Learners. CBE- Life Sciences Education, 5(3), 197-203. doi:10.1187/cbe.06-06-0168.Andrade, H., & Du, Y. (2005). Student perspectives on rubric-referenced assessment. Practical Assessment, Research & Evaluation, 10(3), 1-11.Andrade, H. G. (2000). Using Rubrics To Promote Thinking and Learning. Educational Leadership, 57(5), 13-18.Andrade, H. G. (2005). Teaching with rubrics: the good, the bad and the ugly. College Teaching, 53(1), 27-30.Andrade, H. G., & Boulay, B. A. (2003). Role of Rubric-Referenced Self-Assessment in Learning to Write. Journal of Educational Research, 97(1), 21-34.Andrade, H. L., Du, Y., & Wang, X. (2008). Putting Rubrics to the Test: The Effect of a Model, Criteria Generation, and Rubric-Referenced Self-Assessment on Elementary School Students' Writing. Educational Measurement: Issues and Practice, 27(2), 3-13. doi:10.1111/j.1745-3992.2008.00118.xAndrade, H. L., Wang, X., Du, Y., & Akawi, R. L. (2009). Rubric-Referenced Self-Assessment and Self-Efficacy for Writing. Journal of Educational Research, 102(4), 287-302.Baartman, L. K. J., Bastiaens, T. J., Kirschner, P. A., & Van der Vleuten, C. P. M. (2006). The wheel of competency assessment: Presenting quality criteria for competency assessment programs. Studies in Educational Evaluation, 32(2), 153-170. doi:10.1016/j.stueduc.2006.04.006Baartman, L. K. J., Bastiaens, T. J., Kirschner, P. A., & Van der Vleuten, C. P. M. (2007a). Evaluating assessment quality in competence-based education: A qualitative comparison of two frameworks. Educational Research Review, 2(2), 114-129. doi:10.1016/j.edurev.2007.06.001Baartman, L. K. J., Bastiaens, T. J., Kirschner, P. A., & Van der Vleuten, C. P. M. (2007b). Teachers' opinions on quality criteria for Competency Assessment Programs. Teaching and Teacher Education, 23(6), 857-867. doi:10.1016/j.tate.2006.04.043Baartman, L. K. J., Prins, F. J., Kirschner, P. A., & van der Vleuten, C. P. M. (2007c). Determining the quality of competence assessment programs: a self-evaluation procedure. Studies in Educational Evaluation, 33(3-4), 258-281. doi:10.1016/j. stueduc.2007.07.004Baryla, E., Shelley, G., & Trainor, W. (2012). Transforming Rubrics Using Factor Analysis. Practical Assessment, Research & Evaluation, 17(4).Bose, K. S., & Sarma, R. H. (1975). Delineation of the intimate details of the backbone conformation of pyridine nucleotide coenzymes in aqueous solution. Biochemical and Biophysical Research Communications, 66(4), 1173-1179.Boud, D., & Falchikov, N. (2006). Aligning assessment with long‐term learning. As sessment & Evaluation in Higher Education, 31(4), 399-413. doi:10.1080/ 02602930600679050Bresciani, M. J., Oakleaf, M., Kolkhorst, F., Nebeker, C., Barlow, J., Duncan, K., & Hickmott, J. (2009). Examining Design and Inter-Rater Reliability of a Rubric Measuring Research Quality across Multiple Disciplines. Practical Assessment, Research & Eva luation, 14(12).Brown, G. (1997). Assessing student learning in higher education. London; New York: Routledge.Brown, G. T. L., Glasswell, K., & Harland, D. (2004). Accuracy in the scoring of writing: Studies of reliability and validity using a New Zealand writing assessment system. Assessing Writing, 9(2), 105-121. doi:10.1016/j.asw.2004.07.001Cebrián de la Serna, M. (2008). La evaluación formativa mediante e-rúbicas. Indivisa: Boletín de estudios e investigación, (10), 197-208.Cebrián De la Serna, M., Martínez Figueira, M. E., Gallego Arrufat, M. J., & Raposo Rivas, M. (2011). La e-rúbrica para la evaluación: una experiencia de colaboración interuniversitaria en materia TIC. En II Conbreso Internacional Usos y Buenas Prácticas TIC. Málaga. Recuperado a partir de http://erubrica.uma.es/wp-content /uploads/2011/06/Comunicaci%C3%B3n.pdfChow, Y. W., Pietranico, R., & Mukerji, A. (1975a). Studies of oxygen binding energy to hemoglobin molecule. Biochemical and Biophysical Research Communications, 66(4), 1424-1431.Chow, Y. W., Pietranico, R., & Mukerji, A. (1975b). Studies of oxygen binding energy to hemoglobin molecule. Biochemical and Biophysical Research Communications, 66(4), 1424-1431.Ciudad Gómez, A. (2010). Teaching Innovation and Use of the ICT in the Teaching-Learning Process within the New Framework of the EHEA, By Means Of Moodle Platform. American Journal of Business Education, 3(13), 13-20. Recuperado a partir de http://journals.cluteonline.com/index.php/AJBE/article/view/972/956. Ciudad Gómez, A. (2011). Design of the Accounting Course Focusing On the Development of Competences, Journal of International Education Research, 7(5),15-22. Recuperado a partir de http://journals.cluteonline.com/index.php/JIER/article/view/6112 /6190Ciudad Gómez, A. & Valverde Berrocoso, J. (2012). Design of a Competency-Based Assessment Model in the field of Accounting, Contemporary Issues In Education Research, 5(5), 343-348. Recuperado a partir de http://journals.cluteonline.com/index.php/CIER/article/view/7466/7532D'Antoni, A., Zipp, G., & Olson, V. (2009). Interrater reliability of the mind map as sessment rubric in a cohort of medical students. BMC Medical Education, 9(1), 1-8. doi:10.1186/1472-6920-9-19East, M. (2009). Evaluating the reliability of a detailed analytic scoring rubric for foreign language writing. Assessing Writing, 14(2), 88-115. doi:10.1016/j.asw.2009.04.001Fernández March, A. (2010). La evaluación orientada al aprendizaje en un modelo de formación por competencias en la educación universitaria. REDU: Revista de Docencia Universitaria, 8(1). Recuperado a partir de http://dialnet.unirioja.es/servlet/extart?codigo=3996629Gibbs, G. (2006). Why assessment is changing. En C. Bryan & K. Clegg (Eds.), Innovative assessment in higher education (pp. 11-22). New York: Routledge.Guzmán Cedillo, Y. I., Flores Macías, R. del C., & Tirado Segura, F. (2012). La evaluación de la competencia argumentativa en foros de discusión en línea a través de rúbricas. Revista Innovación Educativa, 12(60), 17-40.Halonen, J. S., Bosck, T., Clay, S., McCarthy, M., Dunn, D. S., Hill IV, G. W., … Whitlock, K. (2003). A Rubric for Learning, Teaching, and Assessing Scientific Inquiry in Psychology. Teaching of Psychology, 30(3), 196.Huber, E., & Harvey, M. (2013). Time to participate: Lessons from the literature for learning and teaching project evaluation in higher education. Studies in Educational Evaluation, 39(4), 240-249. doi:10.1016/j.stueduc.2013.10.004Ibarra Sáiz, M., Rodríguez Gómez, G., & Gómez Ruiz, M. A. (2012). La evaluación entre iguales: beneficios y estrategias para su práctica en la universidad. Revista de educación, (359), 206-231.Ibarra Saiz, M. S., & Rodríguez Gómez, G. (2010). Los procedimientos de evaluación como elementos de desarrollo de la función orientadora en la universidad. Revista Española de Orientación y Psicopedagogía, 21(2), 443-461.Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130-144. doi:10.1016/j. edurev.2007.05.002Lovorn, M. G., & Rezaei, A. R. (2011). Assessing the Assessment: Rubrics Training for Pre-Service and New In-Service Teachers. Practical Assessment, Research & Evaluation, 16(16), 1-18. Recuperado a partir de http://pareonline.net/getvn. asp?v=16&n=16Martínez Figueira, M. E., & Raposo Rivas, M. (2011). La evaluación del estudiante a través de la rúbrica. En Xornada de Innovación Educativa 2011. Vigo. Recuperado a partir de http://webs.uvigo.es/xie2011/Vigo/XIE2011-077.pdfMcGoldrick, K., & Peterson, B. (2013). Using rubrics in economics. International Review of Economics Education, 12(0), 33-47. doi:10.1016/j.iree.2013.04.009Messick, S. (1996). Validity of Performance Assessments. En Phillips, G.W. (Ed.), Technical Issues in Large-Scale Performance Assessment (pp. 11-28). Washington, DC: NCES - National Center for Education Statistics. Recuperado a partir de https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=96802Monreal Gimeno, M. del C., & Terrón Caro, M. T. (2010). Evaluación de competencias en la doble titulación de trabajo social y educación social: una experiencia interdisciplinar. REDU: Revista de Docencia Universitaria, 8(1). Recuperado a partir de http://dialnet.unirioja.es/servlet/extart?codigo=4015012Mora, J. J. (2010). Rubrics as an Evaluation Tool in Economics (SSRN Scholarly Paper No. ID 1578439). Rochester, NY: Social Science Research Network. Recuperado a partir de http://papers.ssrn.com/abstract=1578439Navarro Soria, I., & González Gómez, C. (2010). La autoevaluación y la evaluación entre iguales como estrategia para el desarrollo de competencias profesionales: una experiencia docente en el grado de maestro. REDU: Revista de Docencia Universitaria, 8(1). Recuperado a partir de http://dialnet.unirioja.es/servlet/extart? codi go=4015026Panadero, E., & Jonsson, A. (2013). The use of scoring rubrics for formative assessment purposes revisited: A review. Educational Research Review, 9(0), 129-144. doi:10.1016/j.edurev.2013.01.002Panadero, E., Romero, M., & Strijbos, J.-W. (s.f.). The impact of a rubric and friendship on peer assessment: Effects on construct validity, performance, and perceptions of fairness and comfort. Studies in Educational Evaluation, (0). doi:10.1016/j. stueduc.2013.10.005Panadero, E., Tapia, J. A., & Huertas, J. A. (2012). Rubrics and self-assessment scripts effects on self-regulation, learning and self-efficacy in secondary education. Learning and Individual Differences, 22(6), 806-813. doi:10.1016/j.lindif.2012.04.007.Penny, J., Johnson, R. L., & Gordon, B. (2000). The effect of rating augmentation on interrater reliability: An empirical study of a holistic rubric. Assessing Writing, 7(2), 143-164. doi:10.1016/S1075-2935(00)00012-X.Popham, W. J. (1997). What's Wrong--and What's Right--with Rubrics. Educational Leadership, 55(2), 72-75.Porto Currás, M. (2009). Evaluación de estudiantes en la universidad de Santiago de Compostela: percepciones de los propios alumnos. REDU: Revista de Docencia Universitaria, (3). Recuperado a partir de http://dialnet.unirioja.es/servlet/extart? codigo=3087590Quesada Serra, V., Rodríguez Gómez, G., & Ibarra Sáiz, M. (2013). ActEval: un instrumento para el análisis y reflexión sobre la actividad evaluadora del profesorado universitario. Revista de educación, (362), 69-104.Raposo, M., & Martínez, E. (2011). La Rúbrica en la Enseñanza Universitaria: Un Recurso Para la Tutoría de Grupos de Estudiantes. Formación universitaria, 4(4), 19-28. doi:10.4067/S0718-50062011000400004Reddy, M. Y. (2007). Rubrics and the enhancement of student learning. Educate~, 7(1), 3-17.Reddy, Y. M., & Andrade, H. (2010). A review of rubric use in higher education. Assessment & Evaluation in Higher Education, 35(4), 435-448. doi:10.1080/02602930902862859Reynolds-Keefer, L. (2010). Rubric-Referenced Assessment in Teacher Preparation: An Opportunity to Learn by Using. Practical Assessment, Research & Evaluation, 15(8), 1-9. Recuperado a partir de http://pareonline.net/getvn.asp?v=15&n=8Rezaei, A. R., & Lovorn, M. (2010). Reliability and validity of rubrics for assessment through writing. Assessing Writing, 15(1), 18-39. doi:10.1016/j.asw.2010.01.003Robin, T., & Marielle, S. (2004). What's still wrong with rubrics: Focusing on the consistency of performance criteria across scale levels. Practical Assessment, Research and Evaluation, 9(2). Recuperado a partir de http://pareonline.net/getvn.asp? v=9&n=2Rodríguez Gómez, G., & Ibarra Saiz, M. S. (2012). Reflexiones en torno a la competencia evaluadora del profesorado en la Educación Superior. REDU: Revista de Docencia Universitaria, 10(2). Recuperado a partir de http://dialnet.unirioja.es/servlet/extart?codigo=4021048Sadler, P. M., & Good, E. (2006). The Impact of Self- and Peer-Grading on Student Learning. Educational Assessment, 11(1), 1-31.Santos, J., & Lavin, A. M. (2004). Do as I Do, Not as I Say: Assessing Outcomes When Students Think Like Economists. The Journal of Economic Education, 35(2), 148- 161. doi:10.2307/30042586Schmoldt, A., Benthe, H. F., & Haberland, G. (1975). Digitoxin metabolism by rat liver microsomes. Biochemical Pharmacology, 24(17), 1639-1641.Vaccaro, A. M., Raschetti, R., Ricciardi, G., & Morpurgo, G. (1975). Temperature adaptation at the hemoglobin level in Carassius auratus. Comparative Biochemistry and Physiology. A, Comparative Physiology, 52(4), 627-634.Villalustre Martínez, L., & Moral Pérez, M. E. del. (2010). E-portafolios y rúbricas de evaluación en ruralnet. Pixel-Bit: Revista de medios y educación, (37), 93-105.Walstad, W. B. (2001). Improving Assessment in University Economics. The Journal of Economic Education, 32(3), 281-294. doi:10.2307/118338