Master thesis : Beyond All-or-Nothing Perception A Time-Aware Multi-Level Perception Dataset, Benchmark, and Method for Autonomous Vehicles

Abstract

Interest in autonomous vehicles has grown rapidly, driven by advances in AI-based perception that have moved driving automation from research prototypes onto public roads. Yet fatal incidents continue to occur even as higher levels of autonomy are deployed, which exposes a fundamental tension between safety and performance: the power of modern AI should be exploited as fully as possible, but never at the expense of safety, in keeping with a safety-first principle. Conventional perception is all-or-nothing: within the time available the model either produces a full, correct prediction or, if it overruns its deadline, produces nothing the planner can use. We think the safer approach is graded: knowing in time that an animal has run into the road is worth more than recognising it precisely as a black cat a moment too late. We introduce PRISM (Perception Refracted Into Staged Multilevel), a time-aware, budget-aware task for graded, multi-level perception. It sits at the Sense stage of the Sense–Plan–Act loop and rewards a system that degrades its answer gracefully as time runs short, instead of one that goes silent. Making PRISM measurable takes three contributions, and together they make up this thesis. First, we release PRISM-10K, a CARLA-generated dataset of about 10 000 sequences across five danger scenarios, twelve weather presets and seven map zones; it is the first driving dataset to carry dense per-frame time-to-collision, time-to-hazard and time-to-event labels. Second, we adapt the Performance-Based Ranking (PBR) framework into a graded score, so a single trained cascade can be re-ranked by priority across five application presets. Third, we design and benchmark three cascade variants: A_baseline, one independent model per level; E_shared, one image encoder shared across the cheap levels; and G_branchy, a dynamic early-exit network that answers sooner on the frames it is confident about. We train each on the PRISM-10K training split and evaluate it on the 1 796-clip test split, under a per-clip time-to-event budget and two execution strategies (Always First and Continued). Based on our experiments, we recommend G_branchy, the dynamic early-exit cascade, run under the Continued strategy: it is the strongest of the three variants and outperforms the per-level A_baseline on all five application presets.