A virtual, intelligent, or computational assistant (e.g., also referred to simply as an “assistant”) is described that relies on supplemental data (e.g., contextual information, user information, etc.) to decipher, and determine actions to perform based on, otherwise ambiguous user commands. With explicit permission from a user, the assistant may access a user’s location history, calendar, e-mail, messages, past assistant interactions, contacts, photos, search history, sensor data, and other contextual or user information to interpret user statements. The supplemental data can be stored locally on a device that is executing the assistant or in a cloud computing environment that is accessible to the assistant from the device. This way, th...