Dynamic adjustment of hotword detection threshold

Many devices allow users to speak a hotword to activate the device, e.g., a virtual assistant application, which then responds to the user command. With user permission, incoming speech data is analyzed to determine whether a hotword was uttered. First, coarse hotword detection is performed. If coarse detection indicates that the hotword was spoken, fine hotword detection is performed to confirm that the hotword was spoken. Per techniques described herein, when fine hotword detection is unsuccessful, the threshold for fine hotword detection is reduced for a short time window. Such reduction improves the likelihood of recognition of the next utterance of the hotword, and can reduce consecutive false negatives. Further, the response from the device is adjusted to improve user experience