On imbalanced classification of benthic macroinvertebrates: Metrics and loss-functions

Aquatic biomonitoring is an integral part of assessing the state and quality of freshwater systems. An important part of biomonitoring is the identification and classification of benthic macroinvertebrates, a species group containing several indicator species of high interest. Lately, automating the process of identifying these species using visual and chemical systems has gained interest. The methods presented for this often overlook the imbalanced nature of taxonomic data, where the size difference between largest and smallest classes is substantial. This thesis has two main themes: analyzing the suitability of different performance metrics used to evaluate imbalanced domain classification models, as well as testing methods that could be used to improve the performance of these models. Performance metrics are analyzed from the standpoint of experts with no machine learning expertise, focusing on understandability and visualizations of the metrics. Focus is given on metrics that can be derived from a multi-class confusion matrix, due to the intuitive derivation of these metrics. These metrics are used to produce both single-score and class-wise metrics, that describe the model performance either as whole, or separately for each class. As for classification improvement methods, experiments with different loss functions, rebalancing and augmentation methods are conducted. This thesis presents as results a comparison of different evaluation metrics with their pros and cons from the biomonitoring point of view. The main argument is that a single metric for describing model performance can be very ambiguous, and if it is possible, further assessment by class-wise metrics should be conducted when comparing models. The results of classification improvement methods did not yield better results than the reference model with the experiments conducted. This thesis also presents a modern reference model trained with a benthic macroinvertebrate benchmark dataset, outperforming most of the current flat classification models in the literature