Deep reinforcement learning in action: innovative approaches to control a real self-driving vehicle

Abstract

Today, autonomous driving is one of the most fascinating challenging tasks of the past centuries, and thanks to technological progress it has become a worldwide topic of interest both for academia and industrial purposes. Recently, approaches such as Deep Reinforcement Learning and Imitation Learning have become increasingly popular also for the development of algorithms for autonomous driving. In this work, innovative methods for planning and control based primarily on these two learning approaches are proposed, with the ultimate goal of testing them on a real vehicle in an urban environment. Initially, we developed a multi-agent system based on Reinforcement Learning that allows lateral and longitudinal continuous control values that the vehicles involved exploit to correctly cross intersections respecting the precedences defined by traffic signs and the right of way rule. The agents learned not only to decide the correct time to cross the intersection, but also to negotiate and interact with each other to avoid collisions. In addition to the intersections used for training, the system is also able to generalize to other different types and, moreover, is even capable of dealing with real traffic and not just simulated one. However, it exhibits uncomfortable behavior when tested on board of a real autonomous vehicle, mainly caused by sudden changes in the choice of acceleration value by the trained model. To overcome the problem encountered in previous work, we developed a Deep Reinforcement Learning planner capable of driving safely and comfortably in an obstacle-free environment. This system predicts both the acceleration and steering angle of the vehicle using only data processed by localization and perception algorithms. The training takes place in a simulator based on HD Maps but, later, the system is also tested in a real-world urban area of the city of Parma, proving that the system features good generalization capabilities also driving in those parts outside the training scenarios. In order to reduce the gap between simulation and reality that allows testing the system in the real-world, a module represented by a tiny neural network able to reproduce the vehicle dynamic behavior was introduced in simulation. The last part of the project, on the other hand, consists of preliminary work to evaluate an additional architecture as an alternative to the previous one, to try to make the system more robust by splitting the neural network into several separately trainable components. This new architecture allows us to train a multi-agent system without it being too onerous and the training time prohibitive. Finally, it is also a model-based architecture which means that it can generate a model of the environment and exploit it to make future predictions that allow it to make better choices about what actions to take in the present.