Comprehensive benchmark and architectural analysis of deep learning models for Nanopore sequencing basecalling

Nanopore-based DNA sequencing relies on basecalling the electric current signal. Basecalling1 requires neural networks to achieve competitive accuracies. To improve sequencing accuracy further,2 new models are continuously proposed. However, benchmarking is currently not standardized, and3 evaluation metrics and datasets used are defined on a per publication basis, impeding progress in4 the field. To standardize the process of benchmarking, we unified existing benchmarking datasets5 and defined a rigorous set of evaluation metrics. We benchmarked the latest seven basecaller models6 and analyzed their deep learning architectures. Our results show that overall Bonito has the best7 architecture for basecalling. We find, however, that species bias in training can have a large impact8 on performance. Our comprehensive evaluation of 90 novel architecture demonstrates that different9 models excel at reducing different types of errors and using RNNs (LSTM) and a CRF decoder are10 the main drivers of high performing models