Genomic BigData Warehousing with Apache Spark and LakeHouse Architecture

Abstract In this repo/article, we explore and discuss Cloud-native BigData engineering approach for handling Genomics data warehousing. The technique underpin BigData processing and data warehousing with Apache Spark and Data Lakehouse architecture on Genomics data of "gold" label VCF files. We set up Genomic Table along with Metadata Table in Cloud-native fashion to bring both Genotype and Phenotype together within single or federated queryable interface. By leveraging Cloud-native setup, we bring Data & Compute closer together to avoid unnecessary data staging for further Genomic analysis or Cohort data building process. Genomic Table and, as such sourcing data warehouse directly from VCF could easily get out of hand. Typically, a "best practice" BioInformatics Pipeline should have finer, distilled, end-of-chain, "gold" label VCF product. This VCF file or metrics/fact files should contain a handful of records per patient that have already analysed & annotated well for a given use case. If sourcing Genomic Table from VCF is not appropriate then one could leverage more summarised tables output from MultiQC or some Cancer Reporter or some intermediary from pipeline output (such as bed file, stats file, metrics file, etc) as see fit. Hence, this kind of Genomic BigData warehouse would still rely on high quality output of BioInformatics Pipeline summary and, continuation of overall Data Pipeline as depict in the figure.DRAF