Evaluating the evalution of training at Canada Revenue Agency : a descriptive analysis of models

Evaluation shares many characteristics with other fields within the social sciences, but has struggled to be recognized as a stand alone academic discipline. Reasons include an overshadowing emphasis on practice rather than theory and modeling. Despite this challenge, much advancement has occurred in evaluation stemming from early works in program evaluation. These methods, theories and early practice have influenced adult education practitioners who borrowed and adapted concepts so to assess workplace training and human resources development (HRD) programs. HRD evaluation literature has seen little advancement since the highly noted inaugural work of Kirkpatrick in 1959. His famous four-level model of has remained largely unchanged, is often used as a benchmark for other HRD models and has influenced other field contributions. Notably, this includes the work Phillips and his message of the ultimate level of evaluation: return on investment (ROI). Many organizations are clamoring to adopt the Phillips process, yet there is little documented evidence that supports the application or the utility of the model. Although ROI seems to translate easiest in the corporate world, many public sector organizations are taking up the challenge as well in a bid to provide results-based evidence in support of accountability. The Canada Revenue Agency (CRA) chose to adopt the Phillips model for its HRD and training evaluation program. The experience and results were documented in the public review of its technical training by the Auditor General of Canada. Overall, CRA garnered positive reviews but were told to not have made adequate use of the Phillips model in providing evidence of effectiveness. The descriptive analysis uncovers other factors, such as staffing, funding and suitability as important in determining the extent in which a model should be applied. In the case of CRA, targets and methods were unrealistic to sustain.Education, Faculty ofEducational Studies (EDST), Department ofUnreviewedGraduat