Deep learning and multimodal data fusion so that autonomous cars can see through fog

Beyond 5G: continuity of optical networks
February 22, 2023
ANEM: Models and techniques of data anonymisation with applications in the mobility sector
February 21, 2023


Autonomous vehicles are a reality that we are getting closer to gradually but inevitably. The Centre for Sensors, Instruments and Systems Development (CD6) of the Universitat Politècnica de Catalunya - BarcelonaTech (UPC) has developed technologies for this type of vehicles that improve safety on journeys. To meet the needs of perceiving the environment at long range, a very complete set of sensors has to be developed, including cameras, LIDAR (Light Detection and Ranging) and radar.

Although the hardware solution is well known (it requires combining different types of sensors as cameras to compensate for the failure modes of each one), a lot of work still needs to be done to improve the perception that this software can offer. The way in which the data are fused or the precision with which pedestrians or threats are perceived from far away are still fields of work with a long way to go before a definitive solution is reached. This project will improve the perception and fusion of image sensors, radar and high-density point clouds. This will improve vehicles’ perception in hostile environments, such as fog or smoke, or even sand or smog.


In the framework of the project, a software solution has been developed for detection in dispersive media. The solution includes a camera, LIDAR and radar, and works through the generation of datasets and deep learning algorithms. Specifically, sensor fusion algorithms have been developed that enable integration of image processing in real time from different image sensors, radar data and LIDAR respectively. This approach with many sensors covers the various failure modes of the sensors and therefore provides a solid working environment in all climatic conditions. The next step will be to incorporate different types of cameras (RGB, NIR, SWIR, thermal) so that these are perfectly fused with the LIDAR data at all distances.

This approach enables much safer detections in computer vision algorithms based on artificial intelligence and eliminates false alarms. The technology has been licenced and has led to a spin-off, which has sublicensed the technology to a TIER1 company.

Other collaborators in the project are the Intelligent Data Science and Artificial Intelligence Research Center (IDEAI) of the UPC, through the Image Processing Group (GPI). This project has received funding from the research and innovation programme Horizon 2020 of the European Union under the subsidy agreement Marie Skłodowska-Curie No 712949 (TECNIOspring PLUS), and from ACCIÓ, the Government of Catalonia’s Agency for Business Competitiveness.

Related Projects