Online Learning of Deep Hybrid Architectures for Semi-Supervised Categorization

Abstract. A hybrid architecture is presented capable of online learning from both labeled and unlabeled samples. It combines both generative and discrimi-native objectives to derive a new variant of the Deep Belief Network, i.e., the Stacked Boltzmann Experts Network model. The model’s training algorithm is built on principles developed from hybrid discriminative Boltzmann machines and composes deep architectures in a greedy fashion. It makes use of its inher-ent “layer-wise ensemble ” nature to perform useful classification work. We (1) compare this architecture against a hybrid denoising autoencoder version of itself as well as several other models and (2) investigate training in the context of an incremental learning procedure. The best-performing hybrid model, the Stacked Boltzmann Experts Network, consistently outperforms all others