Towards Developing Theoretical Foundations for Personalisation: Local Intrinsic Dimensionality Perspective

© 2020 Tahrima HashemPersonalised data analytics can make a significant improvement to the individuals' career, lifestyle, and health by providing them with an in-depth analysis of their profiles in contrast to other people. This thesis aims to obtain such personalised analytics by developing effective query-driven data-mining algorithms and strategies. However, existing research lacks in measures and strategies for the discovery of query aware patterns. State-of-the-art techniques mainly focus on mining patterns/clusters for illustration of underlying trends or association across the entire data rather than paying attention to an individual object of interest. The usefulness of feature space is typically evaluated in terms of its ability to preserve the global or/and local intrinsic structures of data without considering its relevance in relation to a specific object's interest, e.g., preserving its neighbourhood structure. Customisation to the general patterns according to the user given criteria is challenging, and often it is not possible to obtain the optimal or best solutions. In this thesis, we demonstrate the significance of personalised patterns over general patterns and identify the challenges of adapting existing research to the personalised case. We propose a novel query driven framework, \texttt{PRESS} that mines objects with subspace similarity for discrete data. Under this framework, we develop multiple algorithms by adopting a top-k pattern enumeration technique to mine personalised clusters, where the objects are homogeneous with respect to the query subspaces. We exploit the strictness and flexibility of query subspaces to effectively reduce the exponential search space. Our empirical case studies on real and artificial data showcase \texttt{PRESS}’s ability to identify cohesive groups, its scalability, and its interpretability while providing users with personalised feedback and recommendation. We conduct enhanced experimental and theoretical investigations on the query's local neighbourhood with respect to different size feature-set in order to get a better understanding of its latent behaviour in numerical data. The statistical modelling of real data enables us to identify the role of two key phenomena influencing the local complexity of neighbourhood surrounding the query object for different feature combinations: correlation and dominance. We develop theoretical foundations in the light of local intrinsic dimensionality (\texttt{LID}) model for assessing the usefulness of a feature space in relation to the query object’s interest. Our proposed LID oriented \texttt{dominance} measure (estimator) quantifies the degree of relevance of a feature space by its ability to preserve the local latent neighbouring structure of the given query object during the projection of data from higher to lower dimensional space. Such query aware local measure of feature relevance would facilitate the subspace learning process, i.e., feature selection, ranking and enumeration, of the query focused machine learning algorithms. It is worth mentioning that our latent analysis of individual characteristics not only benefits the personalised learning but could also enhance the general learning process