Evaluating assessment tools in child protection : a conceptual framework of internal and ecological requirements

Despite substantial evidence on the higher predictive validity of empirically-derived instruments compared to clinical judgement, the controversy on the best direction in child protection assessment is far from over. We introduce a conceptual framework that may help explain why this controversy continues. The framework distinguishes between internal and ecological requirements of assessment tools. First, existing frameworks have primarily focused on internal requirements that refer to the psychometric qualities of a tool, which are theoretically independent of the organisational context. For these internal requirements, we suggest a distinction between three types of validity: construct validity, predictive validity, and indicative validity. Second, the degree of fit with the ecological requirements determines how well the tool works in a specific organisation: for example, whether a tool makes sense to practitioners, whether they readily adopt or tacitly adapt it, or how well it fits with the objectives of the organisation and the goals of individual workers. We define four such requirements: adequacy, organisational suitability, practicality, and utility. The framework is illustrated with data from an ethnographic study in Switzerland. The framework leads to questions that may guide managers and frontline workers in developing, implementing, and evaluating standardised risk assessment in child protection