The role of e-rubrics in the evaluation of digital materials for language teaching in virtual learning environments.

[ES] Presentamos en este artículo una investigación en la que analizamos la funcionalidad de las e-rúbricas en procesos de coevaluación, en la evaluación de materiales digitales enfocados a la enseñanza de lenguas y en la evaluación de destrezas lingüísticas desde el principio del trabajo colaborativo y cooperativo de 40 estudiantes que cursan el “Máster universitario en tecnologías de la información y la comunicación en la enseñanza y tratamiento de lenguas” de la Universidad Nacional de Educación a Distancia (UNED). A través de una concepción metodológica basada en la investigaciónacción colaborativa virtual, aplicamos un procedimiento de análisis con una doble vertiente cualitativa mediante el empleo del programa Atlas-Ti y del enfoque reticular-categorial del análisis de redes sociales con la aplicación de UCINET y el visor yED Graph Editor. Este procedimiento metodológico nos permite entender la forma y estructura de esas relaciones como un todo, lo que favorece el entendimiento de los mecanismos causales que subyacen en las consideraciones de los estudiantes a través de las opiniones no controladas ni moderadas en foros educativos, permitiéndonos descubrir pautas de interacción ocultas. Los resultados muestran que el empleo de e-rúbricas optimiza la autoevaluación, la pertinencia de los criterios de evaluación, la observación de la funcionalidad y utilidad del formato y conectividad de los elementos audiovisuales de los materiales digitales en red. Asimismo, resultan bastante eficaces en la observación de indicadores de la productividad y destrezas lingüísticas; especialmente, en aspectos como la producción oral y escrita de la lengua y en procesos gramaticales y léxicos.[EN] This paper shows a research project which analyses the usefulness of e-rubrics in terms of peer assessment, evaluation of digital materials for languages and also assessment of language skills from the perspective of collaborative and cooperative work, undertaken by 40 students from the Masters in Information and Communication Technology for the teaching and processing of languages at the Spanish National University for Distance Learning (UNED). Following a methodological framework based on virtual, collaborative action-research, a qualitative analysis is applied with a twofold approach: firstly through the use of Atlas-Ti and secondly through a reticular, category based social network analysis thanks to UCINET and yED Graph Editor. This methodological procedure allows understanding the shape and structure of relationships as a whole, enabling us to uncover hidden interaction patterns. Results show that the use of e-rubrics optimizes self-assessment, the adequacy of assessment criteria as well as the study of functionality and usefulness in terms of format and connectivity in the audio-visual elements included in digital online resources. Furthermore, e-rubrics are deemed as significantly effective for discerning indicators of productivity and language skills, especially in aspects such as oral and written production and lexico-grammatical processes.Vázquez-Cano, E.; Martín-Monje, E.; Fernández-Álvarez, M. (2014). El rol de las e-rúbricas en la evaluación de materiales digitales para la enseñanza de lenguas en entornos virtuales de aprendizaje. REDU. Revista de Docencia Universitaria. 12(1):135-157. https://doi.org/10.4995/redu.2014.6414135157121Al-Jarf, R. (2011). Creating and Sharing Writing iRubrics. Asian EFL Journal. Professional Teaching Articles, 51, 41-62.Andrade, H. G., Du, Y., y Wang, X. (2008). Putting rubrics to the test: The effect of a model, criteria, generation, and rubric referenced self assessment on elementary school students' writing. Educational Measurement: Issues and Practice, 27(2), 3-13.Attali, Y., Lewis, W., y Steier, M. (2012). Scoring with the computer: Alternative procedures for improving the reliability of holistic essay scoring. Language Testing, 30(1), 125-141.Barabási, A.L. (2002). Linked-The new science of networks. Cambridge, MA.: Perseus Publishing.Borgatti, S., Everett, M., and Freeman, L. (2002). Ucinet for Windows: Software for Social Network Analysis. Harvard, MA.: Analytic Technologies.Boud, D. y Falchikov,N. (Eds.) (2007). Rethinking Assessment in Higher Education: Learning for the Longer Term. Londres: Routledge.Castells, M. y Monge, P. (2011). Network Multidimensionality in the Digital Age. Prologue to the Special Section. International Journal of Communication, 5, 788-793.Caverlee, J., Liu, L. y Webb, S. (2010). The SocialTrust framework for trusted social information management: Architecture and algorithms. Information Sciences, ScienceDirect, 180, 95-112.Cebrián,M. (2009) Formative and peer-to-peer evaluation using a rubric tool. En A. Méndez-Vilas, A. Solano Martín, y J.A. Mesa González. Research, Reflections and Innovations in Integrating ICT in Education, págs.60-64. Badajoz: Formatex. Disponible en http://www.formatex.org/micte2009/book/Index.pdfCebrián, M. (2011). La evaluación formativa a través de las e-rúbricas y los e-portafolios. V Ciclo de conferencias: Enseñanza y aprendizaje en la Universidad. Universidad de Vigo. Disponible en http://vicadc.uvigo.es/opencms/export/sites/vicadc/vica dc_gl/documentos/ciclos_conferencias/Material.ePor_eRubric.pdfCebrián, M. y Accino, J. (2009). Del ePortafolio a las tecnologías de la federación: La experiencia de Ágora Virtual®. Jornadas Internacionales sobre docencia, investigación e innovación en la universidad. Santiago de Compostela.Coniam, D. (2009). Experimenting with a computer essay-scoring program based on ESL student writing scripts. ReCALL: The Journal of EUROCALL, 21(2), 259-279.Dornisch, M. M., y Sabatini McLoughlin, A. (2006). Limitations of web-based rubric resources: Addressing the challenges. Practical Assessment, Research & Evaluation, 11(3).Downie, M. (2011). Choosing and Developing Classroom Material. En S. House (coord.) Didáctica del Inglés. Classroom Practice. Barcelona: Graó.Dublin Descriptors (2005). Shared "Dublin" descriptors for the Bachelor's, Master's and Doctoral awards. (Draft 1.31 working document on JQI meeting in Dublin. 2004PC).Enright, M., y Quinlan, T. (2010). Complementing human judgment of essays written by English language learners with e-rater® scoring. Language Testing, 27(3), 317- 334.Erben, T., Ban, R., & Castañeda, M. (2009). Teaching English Language Learners. Through Technology. New York: Routledge.Falchikov, N. y Goldfinch, J. (2000). Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks. Review of Educational Research, 70(3), 287-322.Farr, B. P., y Trumbull, E. (1997). Alternative assessments for diverse classrooms. Norwood: Christopher-Gordon.Fernández March, A. (2010). La evaluación orientada al aprendizaje en un modelo de formación por competencias en la educación universitaria. Revista de Docencia Universitaria, 8(1), 11-34.Fetterman, D. (1984). Ethnography in educational evaluation. Beverly Hills: Sage.Genesee, F., y Upshur, J. A. (1996). Classroom-based evaluation in second language education. Cambridge: Cambridge University Press.Gómez Lucas, M. C. y Álvarez Teruel, J. D. (coords.) (2011). El trabajo colaborativo como indicador de calidad del Espacio Europeo de Educación Superior. Alcoy: Marfil.Gottlieb, M. (2006). Assessing English language learners: Bridges from language proficiency to academic achievement. Thousand Oaks: Corwin Press.Griffin, M. (2009). What is a rubric? Assessment Update, 21(6), 4-13.Han, N. R., Chodorow, M., & Leacock, C. (2006). Detecting errors in English article usage by non-native speakers. Natural Language Engineering, 12(2), 115-129.Hanrahan, S. e Isaacs, G. (2010). Assessing Self- and Peer-assessment: The students' views. Higher Education Research & Development, 20(1), 53-70.Hung, S.A. (2012). A Washback Study on E-Portfolio Assessment in an English as a Foreign Langauge Teacher Preparation Program. Computer Assisted Language Learning, 25(1), 21-36.Ibarra, Mª S. y Rodríguez Gómez, G. (2010). Aproximación al discurso dominante sobre la evaluación del aprendizaje en la universidad. Revista de Educación, 351, 385-407.Jonsson, A., y Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130-144.Kitsantas, A. y Dabbagh, N. (2010). Learning to learn with integrative learning technologies (ILT): A practical guide for academic success. Estados Unidos: Information Age Publishing.Knoke, D. y Yang S (2008). Social Network Analysis. United States of America: SAGE.Lim, H. y Kahng, (2012). Review of Criterion®. Language Learning & Technology, 16 (2), 38-45.Linnenbrink, E. y Pintrich, P. (2003). The role of self-efficacy beliefs in student engagement and learning in the classroom. Reading & Writing Quarterly, 19, 119-137.Mansilla, V. B., y Duraisingh, E. D. (2009). Targeted assessment rubric: An empirically grounded rubric for interdisciplinary writing. The Journal of Higher Education, 80(3), 334-353.Miles, M. y Huberman, M. (1994). Qualitative data analysis: an expanded sourcebook. Thousand Oaks: Sage.Millis, B. J. y Rhem, J. (2010). Cooperative Learning in Higher Education. Across the Disciplines, across the Academy. Virginia: Stylus Publishing.Moya, S. y O'Malley, M. (1994). A Portfolio Assessment Model for ESL. The Journal of Educational Issues of Language Minority Students, 13, 1-16.Navarrete, C., y Gustke, C. (1996). A guide to performance assessment for linguistically diverse students. Albuquerque: New Mexico Highlands University.O'Malley, J. M., y Pierce, L. V. (1996). Authentic assessment for English language learners: Practical approaches for teachers. Nueva York: Addison-Wesley.Orsmond, P., Merry, S. y Reiling, K. (1996). The Importance of Marking Criteria in the Use of Peer Assessment. Assessment & Evaluation in Higher Education, 21(3), 239-250.Pool-Cibrian, W. J. y Martínez-Guerrero J. I. (2013). Autoeficacia y uso de estrategias para el aprendizaje autorregulado en estudiantes universitarios. Revista Electrónica de Investigación Educativa, 15(3), 21-37. Recuperado de http://redie.uabc.mx/ vol15no3/contenido-pool-mtnez.htmlReddy, Y. M. (2007). Effect of rubrics on enhancement of student learning. Educate, 7(1), 3-17.Reddy, Y.M. y Andrade, H. (2010). A review of rubric use in higher education. Assessment & Evaluation in Higher Education, 35(4), 435-448.Ricoeur, P. (1988). Discurso de la acción. Madrid: Cátedra.Seidman, S.B. (1983). Network structure and minimum degree. Social Network, 5, 269-287.Spence, L. K. (2010). Discerning writing assessment: Insights into an analytical rubric. Language Arts, 87(5), 337-352.Spencer, J. W. (2003). Global gatekeeping, representation and network structure: a longitudinal analysis of regional and global knowledge-diffusion networks. Jour nal of international business studies, 34, 428-442.Trujillo, F. (2012). Evaluar para aprender. Disponible en http://www.educacontic.es/ blog/evaluar-para-aprenderUr, P. (1996). A Course in Language Teaching. Practice and Theory. Cambridge: Cambridridge University Press.Varela, R. (2003). All about Teaching English. A Course for Teachers of English. Madrid: Centro de Estudios Ramón Areces.Vickerman, P. (2009). Student perspectives on formative peer assessment: an attempt to deepen learning? Assessment & Evaluation in Higher Education, 34(2), 221-230.Wasserman, S. y Faust, K. (1994). Social network analysis: methods and applications. Nueva York: Cambridge University Press.Wilson, M. (2006). Rethinking Rubrics in Writing Assessment. Portsmouth, NH: Heinemann.Wolfe, E. W., Matthews, S., y Vickers, D. (2010). The effectiveness and efficiency of distributed online, regional online, and regional face-to-face training for writing as sessment raters. The Journal of Technology, Learning, and Assessment, 10(1), 1-21.Zimmerman, B. J. y Schunk, D. (2011). Self-regulated learning and performance: an in troduction and an overview. En B. J. Zimmerman y D. Schunk (Eds.), Handbook of Self-Regulation of Learning and Performance. Routledge