Enabling type differentiated explainable queries across modalities for different fashion items

One of the biggest differences between shopping online and in person is the limited scope and expressibility of the queries that current systems allow and can handle. In person, users often employ a combination of linguistic and visual tools at their disposal to create complex queries. Handling such queries requires modeling relationships between products of the same type, products of different types, products and outfits, and products and their attributes. In this paper, we propose a system that models these relationships by: (i) building a robust visual representation of items that captures notions of similarity and compatibility between products, (ii) learning to predict low-level (color, type) and high-level (style, brand) attributes of the items from their visual representations, and (iii) learning segment-wise maps of outfits to items. For each part, we evaluate the model by demonstrating its performance on relevant tasks like outfit completion, item retrieval, etc., and flexibility through example results for complex queries