Construct validation of teacher portfolio assessment : Procedures for improving teacher competence assessment illustrated by teaching students research skills

The study aims to design and test procedures for teacher portfolio assessments. What are suitable procedures to assess teachers' competencies in developing students' research skills? We first searched into the tasks teachers have in teaching students research skills and the competencies needed to fulfill these tasks. For the development of assessment criteria, a delphi method based on stakeholders' judgments has been designed and tested. In three rounds, 21 stakeholders judged and revised criteria. The method proved to be useful and adequate, resulting in criteria with a high degree of support and consensus: 1. selecting goals for students (GOAL); 2. Choosing an appropriate assignment (ASSIGN); 3. Preparing students to work on their assignments (MANAGE); 4. Previously thinking of teaching strategies (THINK); 5. Teaching students research skills (TEACH); 6. Creating a positive pedagogical climate (CLIMATE); 7. Assessing research skills (ASSESS); 8. Reflecting on the program and on actions in teaching (REFL). Using a policy capturing method, in 2 rounds with discussion in between, 9 stakeholders developed performance standards (or cut-off scores). The stakeholders gave most weight to the assessment criteria GOAL and CLIMATE and least weight to MANAGE. Policy capturing proved to be a clear and useful method generating consistent judgments. Two building blocks in a valid design of teacher portfolios for assessment purposes are the nature of the conceptual constructs to be assessed and teachers' operational performances that reflect these constructs. We developed and tested a chain model for construct validation that guides the process of portfolio procedure development. We illustrate the structures that underlie the caching model with the development of a portfolio procedure (used later on in our study). We used expert judgments (n=6) and evaluation of the assessed teachers (n=18) as evidence for the relation between the portfolio and the constructs measured. We searched into the reliability and cognitive representations of raters' assessments, based on raters' verbal protocols and judgment forms. Six raters systematically assessed 18 portfolios. The interrater reliability of 12 portfolios was reasonable to good. Variance analysis showed hardly any rater effects. We used Associative Systems Theory (Carlston, 1992, 1994) to analyze retrospective verbal protocols and judgment forms. Certain representations of the raters (those on the dimensions 'abstract - concrete' and 'positive - negative') were significantly related to the judgments given and to their reliability. Based on the positive results of the application of the developed procedures, we conclude that the portfolio assessments in our study are valid to a sufficient degree. The reliability is reasonable. A next question is: what competencies do the teachers possess and how are their beliefs related to their behavior? We used the Theory of Planned Behavior (Fihbein & Ajzen, 1975; Ajzen, 2000) to qualitatively analyze teachers' original portfolio data. In line with our study the theory assumes that teachers' prior knowledge and beliefs about teaching and learning critically influence how they teach students in the context at hand. However, on the aggregate level, we did not found a relation between teachers' knowledge and beliefs and their behavior as documented in their portfolios. Also raters and students assessed the teachers. The assessment criteria formed a reliable scale. The mean of the scale was related to the student evaluation. In general the teachers received the lowest judgments on the criteria REFL: (2,5 on a five-point scale), and INSTR (2.8) and the highest judgments on CLIMATE (3.4) and MANAGE (3,4). Teachers differ in the degree they motivate their teaching. Teachers with high judgments on the criterion THINK have significant higher student evaluation, than those with a low rating on this criterion