Autonomous robots that can assist humans in the daily unstructured world have been a long standing vision of robotics and artificial intelligence (AI). Such autonomous intelligent robotic system requires two essential building blocks: perception and control. Meanwhile, the past few years have seen major advances in many perception and control tasks empowered by deep learning and reinforcement learning methods. Hence one natural question to ask is how AI techniques could help to accomplish those robotic tasks. In this thesis, we explore learning-based solutions to robotic tasks.

Our first attempt is constructing a unified benchmark for visual object tracking on the unmanned aerial vehicle (UAV) platform. We manually built a drone tracking dataset, consisting of a variety of vide...[ Read more ]

Autonomous robots that can assist humans in the daily unstructured world have been a long standing vision of robotics and artificial intelligence (AI). Such autonomous intelligent robotic system requires two essential building blocks: perception and control. Meanwhile, the past few years have seen major advances in many perception and control tasks empowered by deep learning and reinforcement learning methods. Hence one natural question to ask is how AI techniques could help to accomplish those robotic tasks. In this thesis, we explore learning-based solutions to robotic tasks.

Our first attempt is constructing a unified benchmark for visual object tracking on the unmanned aerial vehicle (UAV) platform. We manually built a drone tracking dataset, consisting of a variety of videos with high diversity captured by drone cameras. We performed an extensive empirical study of the state-of-the-art methods on the dataset and identified their major weakness in the motion model. We also devised new motion models by explicitly estimating the camera motion in the tracking phase, which are especially suitable and effective for the drone tracking scenario.

Collecting real-world data with robotic systems is generally expensive due to the hardware cost and the manual labeling effort. However, deep learning and reinforcement learning methods require a data-hungry training paradigm. We proposed to address this issue by learning from synthetic data while minimizing the gap from simulation to reality at the same time. For robotic perception task, we investigated instance segmentation for robot manipulation. We developed an automated rendering pipeline to generate a variety of photorealistic synthetic images with pixel-level labels. The synthetic dataset is then used to train an objectness deep neural network model which can successfully generalize to real-world manipulation scenarios. For robotic control task, we focused on the challenging problem of learning UAV control for actively tracking a moving target. We proposed a hierarchical approach that combines model-free reinforcement learning methods with conventional feedback controllers to enable efficient and safe exploration in the training phase. We showed that this hierarchical control scheme can learn a target following policy in a simulator efficiently and the learned behavior can be successfully transferred to real-world quadrotor control.