Text2Scenario: Text-Driven Scenario Generation for Autonomous Driving Test

1State Key Laboratory of Intelligent Transportation Systems, School of Transportation Science and Technology, Beihang University, 100191, Beijing, P.R.China., 2Key Laboratory of Transport Industry of Comprehensive Transportation Theory (Nanjing Modern Multimodal Transportation Laboratory), Ministry of Transport, 21000, Nanjing, P.R.China., 3Shanghai Artificial Intelligence Laboratory, 200232, Shanghai, P.R.China. 4Zhongguancun Laboratory, 100191, Beijing, P.R.China.
First author: caixuan@buaa.edu.cn, xs_bai@buaa.edu.cn

*Corresponding author: zhiyongc@buaa.edu.cn, yilongren@buaa.edu.cn

Scenario generation case in CARLA for Dora-RS.

Abstract

Autonomous driving (AD) testing constitutes a critical methodology for assessing performance benchmarks prior to product deployment. The creation of segmented scenarios within a simulated environment is acknowledged as a robust and effective strategy; however, the process of tailoring these scenarios often necessitates laborious and time-consuming manual efforts, thereby hindering the development and implementation of AD technologies. In response to this challenge, we introduce Text2Scenario, a framework that leverages a Large Language Model (LLM) to autonomously generate simulation test scenarios that closely align with user specifications, derived from their natural language inputs. Specifically, an LLM, equipped with a meticulously engineered input prompt scheme functions as a text parser for test scenario descriptions, extracting from a hierarchically organized scenario repository the components that most accurately reflect the user's preferences. Subsequently, by exploiting the precedence of scenario components, the process involves sequentially matching and linking scenario representations within a Domain Specific Language corpus, ultimately fabricating executable test scenarios. The experimental results demonstrate that such prompt engineering can meticulously extract the nuanced details of scenario elements embedded within various descriptive formats, with the majority of generated scenarios aligning closely with the user's initial expectations, allowing for the efficient and precise evaluation of diverse AD stacks void of the labor-intensive need for manual scenario configuration.

Overview of the T2S testing scenario generation framework

Method Overview

Overview

The T2S system transmutes testing descriptive text into simulator-ready DSL scripts across five distinct stages, anchored by the ASAM OpenScenario textbook approach. The first stage necessitates listing the sextet of pivotal traffic scenario elements, in accordance with the Safety of The Intended Functionality (SOTIF) tenets, which will be exhaustively expounded. Notably, vehicle-to-everything communication level is exempted from this discourse, as it falls outside the scope of this study. The secondary stage witnesses the LLM empower the testing text components, utilizing a knowledge base of scenario descriptions to decode the testing text posited by users, identifying the relevant hierarchical pre-configured elements. The following stage proceeds to match the scenario primitive, employing a priority-based assembling methodology to concatenate scenario descriptors into a coherent DSL script. Ultimately, this scenario descriptor file is integrated into the simulation platform, which interprets and executes the scenario. This integration supports the validation and verification of various AD stacks during the execution phase to yield testing report results.

Prompt Engineering

Prmpt

This pipeline guides LLMs in generating structured outputs that align with DSL requirements. The Prompt Engineering process encompasses five stages: Role Setting, Few-Shot, Chain-of-Thought, Syntax Alignment Checking, and Self-Consistency. Each stage employs carefully engineered prompts to elicit stable output from the LLM. Within the first stage, Role Setting acts as the LLM’s prefix prompt, positioning the LLM as an AD testing expert to refine knowledge retrieval and enhance the quality of responses.

Generated Scenarios in CARLA

Control loss

Traffic negotiation

Right turn at an intersection with crossing traffic

Crossing negotiation at an unsignalized intersection

Crossing traffic running a red light at an intersection.

Crossing with oncoming bicycles

Highway

Highway cut-in from on-ramp

Static cut-in

Highway exit

Yield to emergency vehicle

Obstacle avoidance

Door obstacle

Slow moving hazard at lane edge

Slow moving hazard at lane edge

Vehicle invading lane on bend

Braking and lane changing

Obstacle avoidance without prior action

Pedestrian emerging from behind parked vehicle

Obstacle avoidance with prior action - pedestrian or bicycle

Obstacle avoidance with prior action - vehicle.

Parking Cut-in

Parking Exit

Night Cut-in

Safety-Crical Scenarios

Infinite parking

Rolling solid line

Collide with right car

Collide with fence

Infinite parking

Hit-and-run

Slipping and losing control

Collide pedestrian

Rolling solid line at roundabout

Control loss and collide

Collide pedestrian

Infinite parking

Collide pedestrian

Infinite parking

Infinite parking

Collide with parking car

Infinite parking

Collide with light

Rolling solid line

Collide with traffic light

BibTeX

BibTex Code Here