Methodology for the assessment of informal and non-accredited knowledge obtained from tertiary education courses

This thesis examines non-accredited experience and knowledge or generic competencies/skills, which are developed during attendance of a university course and are not accredited by the respective diploma. Typical generic competencies/skills include: problem solving, teamworking, leadership, communication, organization and management, creativity and innovation, critical thinking, professionalism, use of information technology, etc. In the last thirty years there has been growing interest in generic competencies/skills, in that they are supplementary components of university studies and prerequisites for employment. In many developed countries, tertiary education curricula are being redesigned using a competency-based approach, with emphasis placed on both curriculum-related competencies and generic competencies/skills.Curricula are not being redesigned on the basis of standardized models. Each institution aims at different competencies, according to its priorities, which are associated with different approaches, needs, strategies, social and national priorities, etc. Moreover, there are considerable differences between the policies, methods and practices adopted for the enhancement and assessment of competencies. Universities have been implementing policies whose target is either to enhance and assess or solely to enhance competencies. Such policies include: writing-intensive courses or „communication across the curriculum‟ in the USA; courses for the enhancement of generic competencies/skills (e.g. communication) within general education curricula of US colleges; e-portfolios with free access for any interested party; assessment surveys for competencies acquired during university studies, highlighting the nation-wide annual survey for graduates in Australia GCE; curricula combining theoretical teaching with work-based learning, implemented by certain universities such as in Canada and USA, etc. At the same time, good practices applied for enhancing knowledge and generic competencies/skills of students are being applied, such as teamworking through jointly implemented projects (work team or virtual work team), class work done in micro teams, conversation groups, field assignments, portfolios, computer-based learning, peer tutoring of students, etc. However, bibliography shows that there is no formal, commonly accepted method for the assessment of competencies/skills. Therefore, certain researchers claim that the tutor should assess the gradual progress of each student individually. In this case, of course, many questions have been raised on whether teachers can actually carry out this procedure, considering that they spend only few hours with the students. According to Ferguson (2001), even at universities which have expanded their goals, it is not certain that students with successful results in their courses have actually acquired the competencies claimed by the universities. For this reason, Bath et al. (2004) agree that it is advisable to use students‟ perceptions to capture the achievements of education compared to traditional quantitative measurements.ixEngineering faculties as well as other faculties provide for the training of students in the subjects of their courses. However, in traditionally organized universities such as those in Greece, emphasis is placed mainly on the content of the courses, as evidenced in the curriculum, and as a result the competencies developed by students cannot be identified. Therefore, the primary goal is to translate each course or curriculum into terms of competencies or learning outcomes and learning objectives, to adopt training policies and practices and finally to assess the achievements of university studies. International literature and practice has shown that this shift requires a different approach to teaching and learning as well as to student assessment. Namely, students‟ competencies can be enhanced through student-centred learning methods, particularly when teaching abandons the usual lecture practice and turns to the active involvement of students in the learning process, and when assessment goes beyond formal examinations (summative assessment) to include formative assessment which can help enhance learning.The main objective of the thesis is to develop, on the one part, an original methodology for the assessment of generic competencies/skills of students and, on the other part, an integrated framework for enhancing and assessing both knowledge and generic competencies/skills. Such framework was used to redesign a traditional undergraduate engineering course. The said course was taught for three years in both traditional and pilot form, at the same time. Three out of five stages proposed in this framework refer to learning objectives, teaching and assessment, and are based on the principles of „constructive alignment‟ of Biggs (2003). Moreover, two stages were added for designing purposes: feedback from/to the teacher and testing of course design. Therefore, redesigning is divided in the following stages: learning objectives, teaching, assessment, feedback and testing.Transit from a traditionally taught course to a redesigned course required the addition of a preliminary stage of preparation for the teacher, which is not mandatory for courses already designed with a focus on competency enhancement.In particular, it is proposed to link the teacher‟s learning objectives to the learning outcomes specified by each university by breaking down the course into three components: (a) disciplinary content; (b) disciplinary competencies/skills or practical competencies/skills; and (c) generic competencies/skills. This is an adaption of the course decomposition into five components proposed by Bennett et al. (2000), for the needs of adapting the present approach to the Greek tertiary education context. Our proposal consists in reformulating the teacher‟s goals on a competency basis in a flexible manner, to allow for linking between objectives and any learning outcomes (competencies) as may be determined by a faculty, school or university. At this stage, due to lack of a specific framework of learning outcomes, we borrowed those suggested by ABET – Accreditation Board for Engineering and Technology.xIn addition, with the aim to enhance knowledge and competencies, it was attempted to abandon the usual practice of lecturing and to considerably reduce the interference of the researcher (acting as teacher), by using techniques for supporting the involvement, practical training and interaction of students inside and outside the classroom. Finally, a series of (mandatory and supplementary) measures are proposed at the assessment and feedback stage for the needs of enhancing and assessing knowledge and competencies/skills. Many of the measures proposed are treated as multi-criteria or multivariate issues. This approach allows for flexibility and adaptation to the specific interests of each teacher, faculty or school.The mandatory measures proposed at the assessment stage are tests of the performance and improvement of generic competencies/skills. In particular, it is proposed to statistically test students‟ performance in disciplinary content and practical skills related to the course. It is also proposed to apply a multi-criteria methodology for indirect self-evaluation of students‟ generic competencies/skills at semester start and end, combined with statistical testing. Statistical and multi-criteria techniques aim at testing the initial level of students‟ generic competencies/skills, which is reexamined at semester end, namely once the students have had the experience of practical training during a course. Comparison of the two levels helps establish whether the generic competencies/skills of students have improved, remained invariable or deteriorated, according to their own view of themselves. At the same time, the initial level is tested in order to highlight the students‟ weaknesses or strengths. The methodology provides a number of quantitative indices for the dimensions-criteria incorporated in the course by the teacher, which can help determine the change in the level of generic competencies/skills, the change in the importance of generic competencies/skills and the contribution of a course in the learning outcomes of a faculty/school.Furthermore, the course designing process includes a series of additional measures at assessment and feedback stages. More specifically, it involves peer-assessment, an educationally-beneficial assessment process. Peer-assessment is used in written and oral assignments, where students assess the work of their fellow students. The analysis of assessments through the use of multi-criteria techniques and clustering techniques results in the grouping of differently or similarly thinking assessors/students. Such results reveal the attitudes of students – in terms of the aspects examined – which may support the teacher‟s goals, namely to enhance specific generic competencies/skills through the course. It is also proposed to carry out a survey on the preference of educational techniques employed, to be used as feedback to the teacher. This topic is structured as a service provided (course), where students act as recipients/customers of this service. Therefore, it is proposed to use an approach based on a multivariate analysis, in particular the conjoint analysis. The results of the analysis point to the educational techniques which are most preferred by students for enhancing their generic competencies/skills.xiThe success of redesigning depends on testing on the basis of three components: a. students‟ performance (disciplinary content, practical skills); b. improvement of their generic competencies/skills; and c. their satisfaction with participating in the course. If test results are positive, then the teacher has achieved his/her goals and designing is considered to have been successful.The proposed approach has two key advantages: firstly, the redesigning of all components of a course that aims at the enhancement of competencies, and secondly it can be adapted to any context