Evaluating an Assessment Instrument for the Oregon Environmental Literacy Plan

This mixed methods study evaluated the validity, and reliability of an instrument designed to assess a middle school student\u27s proficiency in systems thinking as described in the 2010 Oregon Environmental Literacy Plan. In Stage 1, a forum of middle school students, formal, and non-formal educators used the Delphi technique to reach consensus regarding which skills were important to include in a scoring guide for systems thinking. In Stage 2, the scoring guide was field tested by formal and non-formal educators using a sample of students\u27 work. The two groups\u27 scores were compared using Cohen\u27s kappa to make inferences regarding inter-rater reliability. Concurrently, an autoethnographic narrative was written to explore issues of equity related to the assessment of environmental literacy. The commonalities between formal and non-formal educators revealed a high level of validity for the construct of proficiency with systems thinking, and a moderate level of reliability between the scores assigned by two groups of educators. In the words of the middle school students, formal, and non-formal educators, who volunteered to create the scoring guide, the ability to make responsible decisions with natural systems, community, and the future in mind involves: creating solutions for systems that are not in balance, presenting the complex inner workings of a system in a simple and succinct way, collaborating, exploring multiple solutions, and sharing ideas in a way that people will understand you