Lagrange: Open-Vocabulary Energy-Based Framework for Driving

Lagrange, a paper submitted to arXiv on 18 Jun 2026, proposes an open-vocabulary, energy-based sparse framework for end-to-end driving that encodes class-agnostic object proposals into continuous semantic tokens and solves decision-making as a Lagrangian action minimization problem. The method targets kinematic feasibility and collision avoidance while avoiding dense volumetric reconstructions.

What is Lagrange and how does it work?

Lagrange is an open-vocabulary, computationally sparse driving framework based on Masked Latent Fields, it uses Vision-Language Models to convert object proposals into continuous semantic visual tokens and decodes attended tokens into an implicit continuous energy field over spatial coordinates. The system applies an intent-driven masked cross-attention module to temporally filter irrelevant entities, then frames control as minimizing a Lagrangian action over that energy field so the resulting trajectories respect vehicle kinematics and avoid collisions.

The paper names its core pieces: Masked Latent Fields (MLF) for sparse, class-agnostic representation, Vision-Language Models (VLMs) to produce continuous semantic tokens, an intent-driven masked cross-attention layer for temporal filtering, and an implicit continuous energy field that the planner uses with Lagrangian action minimization to produce kinematically valid trajectories.

How does Lagrange differ from existing driving paradigms?

Lagrange positions itself between dense, geometry-first models and sparse, query-based planners: dense approaches such as occupancy networks are geometrically robust but impose heavy computational costs and limited high-level semantic reasoning, while sparse query planners are efficient but depend on closed-set definitions and are vulnerable to out-of-distribution events. The paper also contrasts Vision-Language-Action models, which offer open-vocabulary reasoning, with Lagrange, arguing that VLA models’ autoregressive discrete token generation conflicts with continuous, high-frequency vehicle control.

Instead of dense volumetric reconstructions or closed-set queries, Lagrange uses VLM-encoded, class-agnostic proposals turned into continuous tokens and decodes them into a space-continuous energy field. Decision-making is then a continuous optimization, not autoregressive token generation, which the authors present as better aligned with vehicle dynamics.

What evidence do the authors provide?

The paper reports extensive offline evaluations on both a standard benchmark and a long-tail benchmark, specifically nuScenes and CODA, and concludes that Lagrange establishes a promising framework for robust, interpretable, and kinematically feasible open-world autonomy. The submission lists four authors: Shihao Ji, HongXi Li, Zihui Song, and Mingyu Li, and is archived as arXiv:2606.20274 (cs.AI). The abstract summarizes the contribution as an "open-vocabulary, computationally sparse driving framework based on Masked Latent Fields (MLF)."

Why it matters

Lagrange attempts to reconcile two practical tensions in autonomous driving: computational cost versus semantic generalization, and discrete reasoning versus continuous vehicle control. By converting VLM outputs into continuous tokens and optimizing actions over an energy field subject to kinematic constraints, the approach could reduce failure modes tied to closed-set detectors and to discretized decision pipelines. For researchers, the paper provides a concrete architecture that bridges language-capable perception with motion-optimal planning.

What to watch

Look for code and reproducible evaluations linked from the paper and for published quantitative comparisons on nuScenes and CODA that break down failure modes in out-of-distribution scenarios. Also watch for follow-up work that evaluates online, closed-loop driving with the Lagrangian action minimizer and for demonstrations of how the intent-driven masked cross-attention performs in dynamic, multi-agent scenes.