Japanese word segmentation using Bayesian inference

Abstract In studies of human cognition, Bayesian models are increasingly popular tools for understanding how children acquire language. Ideal learner Bayesian models investigate what learning strategies are optimal for solving the problem under consideration, while constrained learner models reduce memory and processing power in order to investigate what learning strategies might be optimal for humans. Both of these model types have been used to investigate an early problem in language acquisition: word segmentation. Use of constrained models for word segmentation with an English corpus shows a surprising “less is more ” effect, where simulated learners with fewer memory and processing resources out-perform the ideal learners. However, it remains to be seen if this effect persists in other languages. This study examines the results of Bayesian modeling for word segmentation on a Japanese corpus. The results of the modeling suggest that a robust “less is more ” effect is not present in Japanese, and thus does not hold across languages. Instead, the presence of this effect may depend on language-specific properties. 1. Introduction: Bayesia