Machine learning, especially deep learning, has made remarkable progress in the past few years. However, the success of deep learning systems heavily relies on massive labeled data, while labeled data are usually scarce in real-world applications. Transfer learning, which leverages the knowledge in well-annotated source domain(s) and helps to learn in a low-resource target domain, can effectively reduce the dependency on labeled data.

In this thesis, we study the generalization ability of deep transfer learning models on clean and adversarial data and build deep transfer learning models that are effective and robust. We start with the transductive transfer learning on clean data, where the tasks of the two domains are the same and there is only domain discrepancy. We identify that the p...[ Read more ]

Machine learning, especially deep learning, has made remarkable progress in the past few years. However, the success of deep learning systems heavily relies on massive labeled data, while labeled data are usually scarce in real-world applications. Transfer learning, which leverages the knowledge in well-annotated source domain(s) and helps to learn in a low-resource target domain, can effectively reduce the dependency on labeled data.

In this thesis, we study the generalization ability of deep transfer learning models on clean and adversarial data and build deep transfer learning models that are effective and robust. We start with the transductive transfer learning on clean data, where the tasks of the two domains are the same and there is only domain discrepancy. We identify that the performance bottleneck of this setting lies in the large variance of the learned latent representations, and propose a Fisher loss to learn discriminative representations. We then consider the inductive transfer learning setting where both the tasks and the domain distributions are different. We propose a Parameter Transfer Unit (PTU) that learns a fine-grained parameter transfer strategy from data. The PTU is a general and flexible module that can be used in both Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). Our proposed methods effectively improve the transfer performance on clean data.

While most transfer learning research works focus on the generalization ability on clean data, transfer learning models are under the threat of adversarial attacks, and such risks have been less studied. To fill the gap, we systematically evaluate the robustness of transfer learning models under white-box and black-box Fast Gradient Sign Method (FGSM) attacks via empirical experiments. The empirical evaluations on the robustness of transfer learning models indicate that deep transfer learning models are vulnerable to adversarial attacks. We further propose Transferred Evolutionary Strategies (TES) that fail fine-tuned models effectively and efficiently in black-box attack settings. We introduce the research frontier of deep transfer learning in this thesis and identify several directions for future exploration.