Long short-term memory networks for vehicle sensor fusion

Jonah T. Gandy; John E. Ball

doi:10.1117/12.2618516

6 June 2022 Long short-term memory networks for vehicle sensor fusion

Jonah T. Gandy, John E. Ball

Proceedings Volume 12115, Autonomous Systems: Sensors, Processing and Security for Ground, Air, Sea and Space Vehicles and Infrastructure 2022; 121150M (2022) https://doi.org/10.1117/12.2618516
Event: SPIE Defense + Commercial Sensing, 2022, Orlando, Florida, United States

Abstract

Adaptive Cruise Control (ACC) is an important part of automotive autonomy. To have a robust Adaptive Cruise Control system, a well-defined sensor fusion system is required. This paper focuses on feature-level sensor fusion system where three fusion techniques are explored: independent Kalman filtering, interacting multiple model (IMM) filtering, and a novel hybrid technique using a combination of both an IMM Filter and deep neural network (DNN). Kalman Filtering is a popular approach for automotive sensor fusion, but heavily relies on the given kinematic models to make state estimations. We explore the IMM filter to capture multi-model motion as well as a DNN approach to build a data-driven model of the system. Our IMM filter is built with several Kalman Filters featuring different kinematic models. The proposed DNN used is a Long Short-Term Memory (LSTM) network trained on radar and camera data to forecast a vehicle’s state. To overcome the weaknesses in both IMM filtering and LSTM networks, we propose a hybrid technique of consisting of both the IMM filter and the LSTM network. The proposed hybrid system uses a single filter for each sensor. The filtered sensor data is synchronized and used as the input for a trained LSTM network. Our LSTM network was trained on over 100 simulated highway driving scenarios that might occur in an ACC application. Simulations and code were developed using MATLAB and the Autonomous Driving Toolbox. Our hybrid IMM-LSTM system outperformed the independent Kalman Filtering approach in longitudinal and lateral tracking accuracy (RMSE) by 23 percent and had promising results compared to the IMM filter where the tracking error was decreased by nearly 50 percent in certain driving scenarios.

Conference Presentation

Citation Download Citation

Jonah T. Gandy and John E. Ball "Long short-term memory networks for vehicle sensor fusion", Proc. SPIE 12115, Autonomous Systems: Sensors, Processing and Security for Ground, Air, Sea and Space Vehicles and Infrastructure 2022, 121150M (6 June 2022); https://doi.org/10.1117/12.2618516

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available