An investigation of query-by-drawing image search on mobile devices

The rapid growth of touchscreen mobile devices has opened up many new opportunities for exploring Content-based Image Retrieval (CBIR) techniques, e.g., it has enabled the development of intuitive and natural user interfaces to facilitate the Query-by-Drawing (QbD) image search paradigm. Although ubiquitous mobile computing holds the promise of drastically changing the ways users search for images from large image repositories, there is yet no comprehensive and systematic study that examines the factors that influence the usability, user behaviours and task performance of a QbD image search mobile application (App). No guidelines and principles exist for designing fundamental user interface components such as the colour picker and little is known how its different designs affect system usability and task performance. Little research exists that examines how different user interface designs affect the ways users draw the queries and how users go about the search process for different types of tasks. The questions of whether or not it is possible or how well users can perform QbD image search by drawing from memory of previously-seen images are unexplored. An understanding of these questions is very important and useful in the development of effective and user-friendly systems for a QbD image search App on touchscreen mobile devices. This PhD project attempts to answer these questions and more by examining the factors that influence the usability and task performance of a QbD painting search mobile application. Starting with a comprehensive literature review and current mobile App review of various related fields, we first designed and implemented a Client-Server painting search mobile App based on an existing CBIR algorithm as the research platform for collecting empirical data; we then conducted a focus group study from whose findings we re-designed the user interfaces of our QbD App. An online survey about art preference was carried out, and we designed four comprehensive user studies and recruited a total of 123 participants to take part in the experiments. Both qualitative and quantitative measures are collected and analysed to discover the various factors that influence the usability and task performance of the interface designs of a QbD image search mobile App and the memory drawing over time. Finally, we made recommendations and suggestions on the design and implementation of various interface components of the QbD image search mobile App based on our findings. This thesis presents the following contributions summarised as: 1) We build a flexible platform that can be used for research in drawing-related fields, such as a new QbD technique or interface tests and psychological study. 2)We present a comprehensive and systematic review of the methods and techniques related to the investigation of QbD image search mobile App. 3) We propose a new way of categorising colour picker: ‘1D-1D-1D’, ‘1D-2D’, and ‘3D’ colour picker. 4) The rationales of choosing stimuli are proposed and a real-world painting database is built. 5) We also develop a variety of novel methodologies for experimental design, data collection and data analysis, and we formulated a new protocol for assessing drawing accuracy and search result. 6) We propose some colour picker design guidelines through a series of experiments and the analysis of comprehensive experimental data. 7) Although indeed memory decays over time, we found the participants are able to draw from the memory of a painting with simple compositional structure (6-7 colour blobs), even for the paintings viewed a month ago. And finally, 8) the experiments also provide valuable insights into how general users draw and modify a query, and judge the result relevance for different tasks on mobile phones, as well as search pattern and memorisation strategy, which also extend a scientific understanding of using current Query-by-Drawing techniques for real world image search