CitySAT: a System for the Semantic Answer Type Prediction Task

This paper describes the CitySAT system that we developed for the DBpedia Answer Type (AT) prediction task of the SMART 2021 challenge. The challenge can be interpreted as a multi-class classification task that takes natural language questions and returns pairs of the predicted answer category and types. For training, we merged the SMART 2021 DBpedia dataset with the 2020 dataset given for the previous year's AT task. In this study, three local Machine Learning (ML) models are deployed to cover the three different types of task and question (category prediction, literal type prediction and resource type prediction). The best model obtains a 98.36% accuracy for the category prediction using a Logistic Regression (LR) classifier. Similarly, another LR model results in 97.90% accuracy for the literal type prediction task. Lastly we also built a Multi-Layer Perceptron (MLP) model to deal with several ontology classes (∼760 classes for DBpedia) in the resource type prediction task. The best MLP model achieves 79.34% on the merged training dataset. The final system output obtained a 98.4% accuracy, 84.2% NDCG@5, and 85.4% NDCG@10 on the (official) test dataset