Does ORTO-15 produce valid data for 'Orthorexia Nervosa'? A mixed-method examination of participants' interpretations of the fifteen test items

Purpose: Orthorexia Nervosa (ON) is defined as a pathological eating behaviour stemming from being “healthy” or “pure”. Survey-based studies typically rely on the ORTO-15 questionnaire or its variations to detect orthorexia. However, frequent post-hoc adjustments to the ORTO-15 suggest psychometric problems. In this study, we explored people’s cognitions about the ORTO-15 items to (1) identify problems specific to ORTO-15 items and (2) explore participants’ understanding of ON symptoms. Methods: Fifty adult participants (40% male, mean age = 34.0 ± 14.4 years) completed the ORTO-15, the Eating Attitudes Test (EAT-26) and the Obsessive–Compulsive Inventory–Revised edition (OCI-R). Qualitative data were collected using the modified “think aloud” protocol, which asked participants to ‘verbalise’ their responses to the ORTO-15 items. These qualitative responses were first analysed conjunctively with the quantitative responses; then subjected to thematic analysis. Results: ORTO-15 identified 64% of the participants for orthorexic tendencies. In most cases (76%), participants reported no issues completing the ORTO-15. However, in some cases, qualitative responses differed from quantitative ones. When people encountered problems, it was because of poor psychometric construction: lack of clarity, ambiguous wording and multiple statements in a single item. Elaborations around the ORTO-15 items formed four major themes: “preoccupation with physical appearance”, “control”, “food is fuel” and “alone, not isolated”. Conclusion: Even though in the majority of cases there were no issues with completing ORTO-15, thematic analysis revealed several discrepancies between our participants’ perceptions of the ORTO-15 items and the previously proposed diagnostic criteria for ON. The results suggest that ORTO-15 is, at best, a mediocre screening tool for ON, which is sensitive to diet but fails to have sufficient level of specificity to detect the pathological stage. More accurate instruments are needed to further research on ON. Level of evidence: V (cross-sectional descriptive study with qualitative analysis)