Privacy-preserving time series medical images analysis using a hybrid deep learning framework

Time series medical images are an important type of medical data that contain rich temporal and spatialinformation. As a state of the art, computer-aided diagnosis (CAD) algorithms are usually used on these imagesequences to improve analysis accuracy. However, such CAD algorithms are often required to upload medicalimages to honest-but-curious servers, which introduces severe privacy concerns. To preserve privacy, theexisting CAD algorithms support analysis on each encrypted image but not on the whole encrypted imagesequences, which leads to the loss of important temporal information among frames. To meet this challenge,a convolutional-LSTM network, named HE-CLSTM, is proposed for analyzing time series medical imagesencrypted by a fully homomorphic encryption mechanism. Specifically, several convolutional blocks areconstructed to extract discriminative spatial features and LSTM-based sequence analysis layers (HE-LSTM)are leveraged to encode temporal information from the encrypted image sequences. Moreover, a weightedunit and a sequence voting layer are designed to incorporate both spatial and temporal features with differentweights to improve performance while reducing the missed diagnosis rate. The experimental results on twochallenging benchmarks (a Cervigram dataset and the BreaKHis public dataset) provide strong evidence thatour framework can encode visual representations and sequential dynamics from encrypted medical imagesequences; our method achieved AUCs above 0.94 both on the Cervigram and BreaKHis datasets, constitutinga significant margin of statistical improvement compared with several competing methods