Add to ORKGThe tokenizer is covering all languages that use Latin1, Laitn2, Latin3 and Cyrillic tables of Unicode. Can be extended to cover other tables in Unicode if necessary. The implementation is as a cascaded regular grammar in CLaRK. It recognizes over 60 token categories. It is easy to be adapted to new token categories
A tokeniser for the Maltese language. The tokeniser accepts UTF8 text and produces UTF8 text, so can...
Tokenization and segmentation are steps performed in the earlier stages of most text analysis. It is...
International audienceThis paper describes how a tokenizer can be trained from any dataset in the Un...
The tokenizer is covering all languages that use Latin1, Laitn2, Latin3 and Cyrillic tables of Unico...
Tokenization is the process of splitting running texts into minimal meaningful units. In writing sys...
It is used morphological lexicon of Bulgarian (100 000 lemmas) compiled as a finite-state automaton ...
Trainable Tokenizer is able to tokenize and segment most languages based on supplied configuration a...
Tokenize source code into integer vectors, symbols, or discrete tokens. The following languages are...
When comparing different tools in the field of natural language processing (NLP), the quality of the...
Written, synchronic, general, manually annotated, 1 000 000 tokens divided in three sets: 215 000 to...
This paper describes how a tokenizer can be trained from any dataset in the Universal Dependencies 2...
100 000 most frequent Cyrillic tokens in the BulTreeBank text archive, UTF-16 list of token-frequenc...
A python module to tokenise texts in the Alsatian dialects. See the module header for help on how to...
The CLaRK System incorporates several technologies: - XML technology - Unicode - Cascaded Regular Gr...
Tokenization is considered a solved problem when reduced to just word borders identification, punctu...
A tokeniser for the Maltese language. The tokeniser accepts UTF8 text and produces UTF8 text, so can...
Tokenization and segmentation are steps performed in the earlier stages of most text analysis. It is...
International audienceThis paper describes how a tokenizer can be trained from any dataset in the Un...
The tokenizer is covering all languages that use Latin1, Laitn2, Latin3 and Cyrillic tables of Unico...
Tokenization is the process of splitting running texts into minimal meaningful units. In writing sys...
It is used morphological lexicon of Bulgarian (100 000 lemmas) compiled as a finite-state automaton ...
Trainable Tokenizer is able to tokenize and segment most languages based on supplied configuration a...
Tokenize source code into integer vectors, symbols, or discrete tokens. The following languages are...
When comparing different tools in the field of natural language processing (NLP), the quality of the...
Written, synchronic, general, manually annotated, 1 000 000 tokens divided in three sets: 215 000 to...
This paper describes how a tokenizer can be trained from any dataset in the Universal Dependencies 2...
100 000 most frequent Cyrillic tokens in the BulTreeBank text archive, UTF-16 list of token-frequenc...
A python module to tokenise texts in the Alsatian dialects. See the module header for help on how to...
The CLaRK System incorporates several technologies: - XML technology - Unicode - Cascaded Regular Gr...
Tokenization is considered a solved problem when reduced to just word borders identification, punctu...
A tokeniser for the Maltese language. The tokeniser accepts UTF8 text and produces UTF8 text, so can...
Tokenization and segmentation are steps performed in the earlier stages of most text analysis. It is...
International audienceThis paper describes how a tokenizer can be trained from any dataset in the Un...