Machine Translation datasets from the KAS corpus KAS-MT 1.0

The Machine Translation datasets KAS-MT 1.0 contain automatically sentence-aligned Slovene and English plain-text abstracts from KAS-Abs 2.0 (http://hdl.handle.net/11356/1449) and is meant for studies in machine translation. The setence alignment approach used requires an alignment reliability threshold that omits candidate pairs below a certain value. This value represents a trade-off between the quantity and quality of aligned pairs. We estimate that the default threshold value produces a good-quality dataset for most users. We release three such datasets (files) that reflect a trade-off between quality and quantity of the data. The Normal dataset uses the default reliability threshold and contains 496,102 sentence pairs, the Strict dataset 474,852 sentence pairs, and the Very Strict dataset 425,534 sentence pairs. A file with thesis metadata is also included. The first column in each of the three TSV files gives the confidence that the alignment is correct (higher is better), the second and third are the source and target Slovene and English sentences, while the fourth gives the “merged” state, i.e. whether sentences in the source or target language were merged (sentences do not always exhibit one-to-one mapping). The last column gives the thesis ID. Reference: Žagar, A., Kavaš, M., & Robnik Šikonja, M. (2021). Corpus KAS 2.0: cleaner and with new datasets. In Information Society - IS 2021: Proceedings of the 24th International Multiconference. https://doi.org/10.5281/zenodo.556222