Fusion of ANNs as decoder of retinal spike trains for scene reconstruction

The retina is one of the most developed sensing organs in the hu- man body. However, the knowledge on the coding and decoding of the retinal neurons are still rather limited. Compared with coding (i.e., transforming vi- sual scenes to retinal spike trains), the decoding (i.e., reconstructing visual scenes from spike trains, especially those of complex stimuli) is more complex and receives less attention. In this paper, we focus on the accurate reconstruc- tion of visual scenes from their spike trains by designing a retinal spike train decoder based on the combination of the Fully Connected Network (FCN), Capsule Network (CapsNet) and Convolutional Neural Network (CNN), and a loss function incorporating the structural similarity index measure (SSIM) and L1 loss. CapsNet is used to extract the features from the spike trains, that are fused with the original spike trains and used as the inputs to FCN and CNN to facilitate the scene reconstruction. The feasibility and superiority of our model are evaluated on five datasets (i.e., MNIST, Fashion-MNIST, Cifar- 10, Celeba-HQ andCOCO). The model is evaluated quantitatively with four image evaluation indices, i.e., SSIM, MSE, PSNR and Intra-SSIM. The results show that the model provides a new means for decoding visual scene stim- uli from retinal spike trains, and promotes the development of brain-machine interfaces