Belayer: Modeling Discrete and Continuous Spatial Variation in Gene Expression from Spatially Resolved Transcriptomics

Spatially resolved transcriptomics (SRT) technologies measure gene expression at known locations in a tissue slice, enabling the identification of spatially varying genes or cell types. Current approaches for these tasks assume either that gene expression varies continuously across a tissue or that a slice con- tains a small number of regions with distinct cellular composition. We propose a model for SRT data that includes both continuous and discrete spatial variation in expression, and an algorithm, Belayer, to esti- mate the parameters of this model from layered tissues. Belayer models gene expression as a piecewise linear function of the relative depth of a tissue layer with possible discontinuities at layer boundaries. We use conformal maps to model relative depth and derive a dynamic programming algorithm to infer layer boundaries and gene expression functions. Belayer accurately identifies tissue layers and infers biologically meaningful spatially varying genes in SRT data from brain and skin tissue samples