Identification of Critical Driving Scenarios from Raw Data via Large Language and Vision-Language Models

  • Type:Bachelor's/Master's Thesis
  • Date:Immediately
  • Supervisor:

    Lei Wan

Background:
Connected autonomous driv-ing systems rely on extensive multi-modal data from vehicles and infrastructure, such as RGB and thermal cameras, and LiDAR, to train robust per-ception models. Identifying rel-evant driving scenarios for col-laborative perception, particu-larly those involving severe visual occlusions and safety-critical situations (e.g., near-collisions), is essential for en-hancing system reliability and safety. However, the absence of ground truth annotations in raw data poses a significant challenge, as manual labeling is impractical for large-scale datasets. The goal of this thesis is to develop a novel method leveraging LLMs and VLMs to automatically identify inter-esting driving scenarios from raw, unannotated data. By harnessing their ability to process and rea-son over multi-modal inputs, this approach aims to enhance dataset creation for connected autono-mous driving, addressing limitations in generalization and scalability found in conventional methods.

Your Tasks:
  • Review state-of-the-art anomaly detection techniques in autonomous driving
  • Develop an approach that uses LLMs and VLMs to reason over multi-modal data, identifying interesting scenarios based on contextual understanding
  • Design an approach to rank and select scenarios, leveraging LLM/VLM outputs to prioritize interesting cases.
  • Assess the method’s effectiveness through synthetic dataset

Your Profile:
  • Strong background in machine learning and computer vision.
  • Experience with multi-modal sensor data processing (e.g., camera images, LiDAR point clouds, GPS time-series).
  • Knowledge of deep learning frameworks (e.g., PyTorch or Tensorflow) and practical experi-ence of LLM/VLM
  • Ability to work independently and tackle complex, open-ended research problems.
Start date: Immediately
Duration: As per the applicable examination regulations.

If you are interested or have any questions regarding this thesis position, feel free to contact.