Multimodal convolutional neural networks for sperm motility and concentration predictions

Manual semen analysis is a conventional method to assess male infertility which includes laboratory technicians examining on parameters such as sperm motility and concentration. Manual evaluation is prone to human errors that causes precision and accuracy issues. The purpose of this research study is to adopt computer vision deep learning techniques and multimodal learning approach in sperm parameters prediction using video-based and image-based input. Convolutional neural network (CNN) has benefited technology industry in recent years, and it has been widely applied in computer vision research tasks as well. Most of the well-established model were designed and pretrained for image-based input, whereas temporal information of video-based input might not be extracted properly using these architectures. Three-dimensional CNN (3DCNN) would be an alternative as it was designed to extract motion and temporal features, which are vital for sperm motility prediction. For sperm concentration, since twodimensional CNN (2DCNN) is efficient in recognizing and extracting spatial features, Residual Network (ResNet) could be adopted for sperm concentration prediction with minimal modification on the original architecture. On the other hand, multimodal learning approach is a technique to aggregate learnt features from different deep learning architecture that adopted other forms of modalities, and provide deep learning model better insights on their tasks. Hence, multimodal learning has been introduced in this research study, where the finalized deep learning architecture received both image-based (frames extracted from video samples) and video-based (stacked frames pre-processed from video samples) input that could provide well-extracted spatial and temporal features for sperm parameters prediction. In this research study, VISEM dataset has been used because it is an open-source dataset which contains 85 sperm videos and biological analysis data from different patients. The video samples went through pre-processing stage to obtain the suitable modalities for training and validation. The developed system has been proven to be capable of improving performance which was as proposed, after the results had been compared to other similar research works. Average mean absolute error (MAE) for sperm motility was observed with high accuracy up to 8.05, and competent performance for sperm concentration with Pearson’s correlation coefficient (RP) of 0.853