While perception tasks such as visual object recognition and text understanding play an important role in human intelligence, the subsequent tasks that involve inference, reasoning, and planning require an even higher level of intelligence. The past few years have seen major advances in many perception tasks using deep learning models. In terms of higher-level inference, however, probabilistic graphical models (PGM), with their ability to describe properties of variables and various probabilistic relations among variables, are still more powerful and flexible.

To achieve integrated intelligence that involves both perception and inference, we have been exploring a research direction we call Bayesian deep learning (BDL). BDL tightly integrates deep learning and Bayesian models (e...[ Read more ]

While perception tasks such as visual object recognition and text understanding play an important role in human intelligence, the subsequent tasks that involve inference, reasoning, and planning require an even higher level of intelligence. The past few years have seen major advances in many perception tasks using deep learning models. In terms of higher-level inference, however, probabilistic graphical models (PGM), with their ability to describe properties of variables and various probabilistic relations among variables, are still more powerful and flexible.

To achieve integrated intelligence that involves both perception and inference, we have been exploring a research direction we call Bayesian deep learning (BDL). BDL tightly integrates deep learning and Bayesian models (e.g., PGM) within a principled probabilistic framework. The aim of this thesis is to advance the fields of both deep learning and Bayesian learning by demonstrating BDL's power and flexibility in different real-world problems such as recommender systems and social network analysis. Its main contributions are as follows.

First, we propose a general framework, BDL, to combine the power of deep learning and PGM in a principled way to get the best of both worlds. Specifically, PGM formulations of deep learning models are first designed and then incorporated into the main PGM, after which joint learning is performed for the unified models.

Second, we devise several concrete models under the BDL framework: Collaborative Deep Learning (CDL) for recommender systems, Collaborative Recurrent Autoencoders (CRAE) for joint sequence generation and recommendation, Relational Stacked Denoising Autoencoders (RSDAE) for relational representation learning, and Relational Deep Learning (RDL) for link prediction and social network analysis.

Third, we propose Natural-Parameter Networks (NPN) as a backpropagation-compatible and sampling-free Bayesian treatment for deep neural networks. Such a treatment can then be naturally incorporated into BDL models to facilitate efficient Bayesian learning of parameters.