DuckPGQ: Bringing SQL/PGQ to DuckDB

In this research project, we investigate an alternative to the standard cloud-centralized data architecture. Specifically, we aim to leave part of application data under the control of the individual data owners in conceptually decentralized personal data stores. Our primary goal is to increase data minimization, i. e., enabling more sensitive personal data to be under the control of its owners while providing a straightforward and efficient framework for architects to design data architectures that allow applications to run and their data to be analyzed. To serve this purpose, the centralized part of the schema contains aggregating views over this decentralized data. We propose to design a declarative language that extends SQL, for architects to specify at the schema level different kinds of tables: decentralized, centralized, and replicated, as well as centralized materialized views, and in addition, the sensitivity of decentralized columns and their minimum granularity levels, when these end up in centralized views. When users modify their personal data stores, the changes need to be reflected in the centralized views while ensuring privacy; this calls for the integration of cryptography techniques in distributed materialized view maintenance. We finally aim to implement this system, where the personal data stores could either live in mobile devices or encrypted cloud storage, in order to evaluate its performance properties experimentally.We demonstrate the most important new feature of SQL:2023, namely SQL/PGQ, which eases querying graphs using SQL by introducing new syntax for pattern matching and (shortest) path-finding. We show how support for SQL/PGQ can be integrated into an RDBMS, specifically in the DuckDB system, using an extension module called DuckPGQ. As such, we also demonstrate the use of the DuckDB extensibility mechanism, which allows us to add new functions, data types, operators, optimizer rules, storage systems, and even parsers to DuckDB. We also describe the new data structures and algorithms that the DuckPGQ module is based on, and how they are injected into SQL plans. While the demonstrated DuckPGQ extension module is lean and efficient, we sketch a roadmap to (i) improve its performance through new algorithms (factorized and WCOJ) and better parallelism and (ii) extend its functionality to scenarios beyond SQL, e.g., building and analyzing Graph Neural Networks.