Hierarchical behavior decision-making framework based on reinforcement learning for driverless cars

As the brain of driverless cars, behavioral decision-making module plays an important role. In recent years, reinforcement learning has been gradually introduced into this field. However, the algorithms proposed in many articles are only used to solve a certain scenario or a certain driving task, and cannot completely solve the unmanned driving in urban scenarios. Some articles try to use one framework to solve the whole urban scenario, which does not take into account the differences between individual scenarios in the city and has poor expansibility. In this paper we propose a hierarchical behavioral decision-making framework. In the top layer, we consider four typical scenarios in the city and adopt rule-based finite state machine (FSM) to realize reasonable switch between them, which makes the system have good expansibility. At the bottom layer, we train an agent for each scenario using reinforcement learning to realize deep traversal of the specified scenario to improve decision accuracy. First, FSM determines which scenario the car is in and then an according agent is selected to make decisions. Using SMARTS as the simulation environment, we test each agent in according scenario and the agent fused with our framework in urban scenario under free, stable and congested three traffic conditions. On the whole, the completion rate remains above 90% and the collision rate remains below 5% for every agent showing that our method is safe and effective.