Natural Belief-Critic: a reinforcement algorithm for parameter estimation in statistical spoken dialogue systems

This paper presents a novel algorithm for learning parameters in statistical dialogue systems which are modelled as Partially Observable Markov Decision Processes (POMDPs). The three main components of a POMDP dialogue manager are a dialogue model representing dialogue state information; a policy which selects the system’s responses based on the inferred state; and a reward function which specifies the desired behaviour of the system. Ideally both the model parameters and the policy would be designed to maximise the reward function. However, whilst there are many techniques available for learning the optimal policy, there are no good ways of learning the optimal model parameters that scale to real-world dialogue systems. The Natural Belief-Critic (NBC) algorithm presented in this paper is a policy gradient method which offers a solution to this problem. Based on observed rewards, the algorithm estimates the natural gradient of the expected reward. The resulting gra-dient is then used to adapt the prior distribution of the dialogue model parameters. The algorithm is evaluated on a spoken di-alogue system in the tourist information domain. The experi-ments show that model parameters estimated to maximise the reward function result in significantly improved performance compared to the baseline handcrafted parameters. Index Terms: spoken dialogue systems, reinforcement learn-ing, POMDP, dialogue managemen