Identifying Factors Associated With Local Use of Large -Scale Evaluations: A Case Study

340 p.Thesis (Ph.D.)--University of Illinois at Urbana-Champaign, 2008.In the current political climate there is an increased use of large-scale evaluations as an accountability measure for educational programs. Although the audience for such a large-scale evaluation may be the program decision makers at the federal, state, or district level, one questions the utility of the evaluation if the recommendations are not communicated or implemented in a way that impacts school personnel and students at the local level. The main goal of conducting this developmental instrumental case study was to consider how large-scale evaluations can be rooted in the concerns, interests, and problems of the program's local practitioners. Data collection in this developmental instrumental case study of a large-scale evaluation included a combination of document review, interviews, and a stratified survey with stakeholders from a state-wide evaluation of a federal literacy initiative. This case study first identified variables from the literature previously shown to enhance evaluation utilization. Then these variables were identified in the case to illustrate how such variables from the literature can be implemented by large-scale evaluations. Three mini-case studies were used to further illustrate how these variables play out in the context of local programs. Finally, the stratified survey offered quantitative insight into how local program implementers experienced the evaluation and were able to utilize the evaluation. Main findings of the study were that (a) large-scale evaluations can be utilization focused at the local program implementation level and (b) regardless of the evaluation design efforts to be utilization focused at the local program implementation level, there are key elements of organizational structure at the local level that support use of evaluation methods, strategies, and tools. To encourage local use of large-scale evaluation processes, information, and findings, the evaluator must first implement methods, strategies, and tools to enhance use, keeping in mind that evaluation utilization is a person-to-person process. However, this case study found that the methods, strategies, and tools meant to enhance local use of evaluation processes, information, and findings identified within Evaluation R did not enhance evaluation utilization unilaterally across all local programs involved in the state-wide evaluation. The results from the ANOVA analysis consistently identified a positive correlation between local leadership support of evaluation activities and the overall impact the evaluation had on a school's local literacy program. These two findings together highlight the phenomenon of the interaction between evaluator intent and local context. Suggestions for further research into evaluation utilization include examining individuals' decisions to use information from a cognitive psychology perspective.U of I OnlyRestricted to the U of I community idenfinitely during batch ingest of legacy ETD