SmartSketcher: Sketch-based Image Retrieval with Dynamic Semantic Re-ranking

We present a sketch-based image retrieval system, designed to answer arbitrary queries that may go beyond searching for prede ned object or scene categories. While sketching is fast and intuitive to formulate visual queries, pure sketch-based image retrieval often re- turns many outliers because it lacks a semantic understanding of the query. Our key idea is to combine sketch-based queries with interactive, semantic re-ranking of query results. We leverage progress in deep learning and use a feature representation learned for image classi cation for re-ranking. This allows us to cluster semantically similar images, re-rank based on the clusters, and present more meaningful query results to the user. We report on two large-scale benchmarks and demonstrate that our re-ranking approach leads to signi cant improvements over the state of the art. Finally, a user study designed to evaluate a practical use case con rms the bene ts of our approach