Differentiable Visual Computing: Challenges and Opportunities

Classical algorithms typically contain domain-specific insights. This makes them often more robust, interpretable, and efficient. On the other hand, deep-learning models must learn domain-specific insight from scratch from a large amount of data using gradient-based optimization techniques. To have the best of both worlds, we should make classical visual computing algorithms differentiable to enable gradient-based optimization. Computing derivatives of classical visual computing algorithms is challenging: there can be discontinuities, and the computation pattern is often irregular compared to high-arithmetic intensity neural networks. In this article, we discuss the benefits and challenges of combining classical visual computing algorithms and modern data-driven methods, with particular emphasis to my thesis, which took one of the first steps toward addressing these challenges