Development and testing of a smartphone-based solid oral dosage form image recognition system by machine learning to support the identification of dispensing errors

Background and importance Misidentification of oral dosage forms contribute to medication errors and compromise patient safety. Especially in manual dose dispensing, identification and verification of medicinal products at point-of-care can be a challenge for healthcare professionals. Machine learning is a powerful tool for object detection and image classification. As mobile technology and smartphones have developed exponentially in terms of computing power and camera systems, handheld devices could serve as a convenient and cost-effective solution for real-time point-of-care tools for supporting the identification and verification of dispensed oral dosage forms for pharmacists, physicians and nurses in hospital settings. Aim and objectives We aimed to develop and test the realworld point-of-care applicability of a smartphone-based pill recognition system using machine learning. Material and methods Formularies and number of dispensed oral dosage forms of three hospitals were evaluated to select the 10 most commonly prescribed medications. A total of 8960 images were taken with a Sony IMX363 camera sensor with resolution of 12 megapixels under various conditions (lighting, distance, angle, dose container) and were used without augmentation to train the model. Microsoft Azure Custom Vision platform was utilised to develop our object detection and image classification model. An application was built using Android Developer Studio, and the model was exported in TensorFlow lite format and integrated in the application. A validation dataset of 200 test images were captured by two pharmacists at the Central Clinical Pharmacy, and precision, recall, mean average precision (mAP) and F1 score evaluation metrics were calculated. Results Our model reached 98.1% precision, 87.4% recall and 96.4% mAP after training, with probability and overlap thresholds set to 50% and 30%, respectively, under the reference condition. Confusion matrix of 200 real-world test images showed a lower overall mAP (73.04%), recall (72.35%) and F1 score (70.6%). Per-class (medication) precision and recall ranges were between 50% and 100% and\ud 20% and 100% respectively. Conclusion and relevance Our model’s performance indicates promising potential for application of smartphone-based identification and verification of dispensed medications at point-ofcare. Eventually, the robustness of the model must be improved by adding more images and extending the dataset with additional commonly used medications before such a system can be utilised in a healthcare setting