Context based image analysis with application in dietary assessment and evaluation

Dietary assessment is essential for understanding the link between diet and health. Mobile devices with cameras provide a new way of collecting dietary information by acquiring images of foods and beverages. In this thesis we extend earlier work of a novel mobile-based food recording system for dietary assessment. The development of image analysis methods to automatically segment, identify and quantify food items from food images becomes imperative. In this thesis, we have investigated methods for image preprocessing, image segmentation, food identification and weight estimation. We describe a single image specular highlight removal method to detect and remove highlight areas in food images as a preprocessing step to improve image segmentation. We also evaluate several image segmentation methods for non-rigid objects. We define novel visual feature descriptors for food classification, including color, texture and local region descriptors. We further describe methods for food classification that can be extended to more general object classification tasks. Food classification decisions from multiple features are combined together to achieve a final decision. We integrate contextual dietary information that a user supplies to the system either explicitly or implicitly to correct potential misclassifications. Contextual dietary information is the data that is not directly produced by the visual appearance of an object in the image, but yields information about a user\u27s diet or can be used for diet planning. We also propose a post-classification single view weight estimation method based on the area of a food item. This is ultimately used to extract the nutrient content of food items using the USDA Food and Nutrient Database for Dietary Studies (FNDDS). We evaluate our models using food image datasets from both controlled studies and natural eating events