PIA project’s perseverance

Author: Robert Moni

Sorry for the clickbait, but no, we aren’t the engineers that put together the Perseverance Mars rover, although we do possess the right amount of perseverance to achieve our goals.

This blog post attempts to provide a summary of our latest research achievements under the aegis of the PIA project.

PIA Project

The Professional Intelligence for Automotive Project was founded in 2019 as a cooperation project between Continental Deep Learning Competence Center Budapest and the SmartLabs of the Department of Telecommunications and Media Informatics (TMIT) and the Department of Control Engineering and Information Technology (IIT) at the Budapest University of Technology and Economics.

The motivation of the cooperation project is to establish a strong relationship between the industry and students. The goal is to support research and development activities for BSc, MSc and PhD students in the field of Artificial Intelligence and automotive. The participating students benefit from professional support offered by the Deep Learning engineers and expert of Continental, and financial support in the form of scholarship, also internship and future job opportunities are obtainable.

During 2020 there were 10 students conducting research and development activities. We published 6 conference papers and 6 of us competed in the 5th AI Driving Olympics competition where we gained four 1st, four 2nd and three 3rd placements in the self-driving challenges (simulation and real environments considered).

A brief explanation of our research field and our achievements presented in Medium blog posts is summarized in the following.

Research field

Our research focuses on the sphere we get by the conjunction of Deep Learning and Autonomous Driving. More precisely we attempt to elaborate on new approaches for Vision-based Environment Perception and Policy-based Vehicle Control. For environment perception, we applied Supervised Learning and Unsupervised Learning methods, while for vehicle control, Reinforcement Learning methods were elaborated.

Vision-based Environment Perception

Environment Perception refers to the active monitoring of the surroundings of a vehicle to extract meaningful task-related information for safety and control. The main sensors responsible for sensing are the Camera, Radar, Lidar and Communication devices (e.g. V2X via Wi-Fi or 5G) depicted by the following image.

Vehicle Environment Perception Source: www.continental.com

We focus on Camera-based Environment Perception and aim to provide state-of-the-art solutions for the following applications:

  1. Semantic and Instance segmentation

2. Depth estimation

3. Multi-task Learning

Policy-based Vehicle Control

Deep neural networks possess the ability to approximate complex functions. Self-driving based on camera input is indeed a complex function which we attempt to approximate with Deep Reinforcement Learning (DRL) methods. The aim is to reach an optimal or near-optimal self-driving policy represented by the weights in the deep neural network.

One of our working solution provided by András Kalapos

There are several ways to achieve such a self-driving policy. We developed multiple solutions considering end-to-end or modular architecture designs, on-policy or off-policy DRL algorithms and even Imitation Learning.

  1. End-to-end DRL-based self-driving agent

End-to-end design refers to the architectural design choice, where the agents learn directly from raw data.

2. DRL-based self-driving agent with a modular design

Modular design refers to the architectural design choice, where the agents learn from extracted observations, usually, low-dimensional representations provided by a separate deep neural network with the scope to extract meaningful task-related information from the raw data (e.g. Vision-based Environment Perception or Representation Learning).

3. Imitation Learning

Publication and conferences

Other than medium blog posts, we did contribute officially to science via articles:


Last but not least we are proud of our winning entries at AI Driving Olympics 5. A detailed blog post about our solutions can be found here: PIA project achievements at AIDO5.

Deep Learning and AI solutions from Budapest University of Technology and Economics. http://smartlab.tmit.bme.hu/