Testing out suicide risk prediction algorithms using phone measurements with patients in acute mental health settings: a feasibility study

Background: Digital phenotyping and machine learning are nowadays being used to augment or even replace traditional analytic procedures in many domains, including health care. Given the heavy reliance on smartphones and mobile devices around the world, this readily available source of data is an important and highly underutilized source that has the potential to improve mental health risk prediction and prevention and advance global mental health. Objective: To apply machine learning in an acute mental health setting for suicide risk prediction. This study is using a nascent approach, adding to existing knowledge by using data collected through a smartphone in place of clinical data which has typically been collected from health care records. Methods: We created a smartphone application called Strength Within Me (SWiM) that was linked to Fitbit, Apple Health kit, and Facebook, to collect salient clinical information such as sleep behavior and mood, step frequency and count, and engagement patterns with the phone from a cohort of acute mental health inpatients (n=66). In addition, clinical research interviews were used to assess mood, sleep, and suicide risk. Multiple machine learning algorithms were tested to determine best fit. Results: K-Nearest Neighbors (k=2) with uniform weighting and Euclidean distance metric emerged as the most promising algorithm, with 68% average accuracy (averaged over 10,000 simulations of splitting the training and testing data via 10-fold cross validation) and average AUC of 0.65. We applied a 5x2cv combined F test to test model performance of KNN against baseline classifier that guesses training majority, random forest, and others and achieved F statistics of 10.7 (p-value .0087), 17.6 respectively (p-value .0027), rejecting null of performance being the same. We have therefore taken the first steps in prototyping a system that could continuously and accurately assess risk of suicide via mobile devices. Conclusions: Sensing for suicidality is an under-addressed area of research to which this paper makes a useful contribution. This is part of the first generation of studies to suggest it is feasible to utilize smartphone generated user input and passive sensor data/digital phenotyping to generate a risk algorithm among inpatients at suicide risk. The model reveals fair concordance between phone-derived and research generated clinical data and with iterative development has potential for accurate discriminant risk prediction. However, while full automation and independence of clinical judgement or input would be a worthy development for those individuals who are less likely to access specialist mental health services, and for providing a timely response in a crisis situation, the ethical and legal implications of such advances in the field of psychiatry need to be acknowledged