Auto-DSM: Black-box evaluation of LLM-based DSM, metrics

Auto-DSM, a black-box evaluation framework for LLM-based Design Structure Matrix generation, was published on arXiv on 7 Jul 2026 by Niels Potters and Theo Hofman. The paper presents a reproducible methodology that benchmarks generated DSMs (GEN-DSMs) against manually validated ground-truth matrices (GT-DSMs) and proposes a Composite Quality Score (Q) to synthesise multiple evaluation axes.

How does the Auto-DSM framework evaluate LLMs?

The framework measures LLM-generated DSMs across structural, classification and stability dimensions, then aggregates them with a Composite Quality Score (Q). Structural metrics listed are Completeness, Correctness and Coupling Density; classification metrics are Selective Accuracy and Abstention Coverage; stability measures include Entropy and Fleiss' kappa. The paper describes both single-run and multi-run perspectives to capture one-off outputs and reproducibility.

The methodology explicitly benchmarks GEN-DSMs against GT-DSMs, treating the ground-truth matrices as manually validated references. The framework addresses prompt sensitivity and parameter variations by testing phrasing, parameter-dataset alignment and system complexity in controlled experiments.

What did the experiments reveal?

Controlled experiments on two datasets showed strengths and limits: LLMs can produce "structurally plausible DSMs" and achieve high reproducibility under well-structured inputs, but they remain sensitive to ambiguity, inconsistent dependency definitions and prompt formulation. The authors ran experiments on a fictive abstract system and a real-world refrigerator decomposition to cover variation in phrasing, alignment and complexity.

Results highlighted systematic sources of hallucination and failures in abstention behaviour. The paper ties these failure modes to the framework's metrics: for example, instability surfaced through elevated Entropy and lower Fleiss' kappa across multi-run trials, while classification problems appeared in Selective Accuracy and gaps in Abstention Coverage. To combine these signals the authors propose Composite Quality Score (Q) as a single synthesis of the measured aspects.

Why it matters

Auto-DSM supplies a transparent, reproducible benchmark for auditing Auto-DSM pipelines that are often closed-source. By formalising structural and classification criteria and adding stability measures, the framework creates a common vocabulary to compare LLM outputs to GT-DSMs and to surface consistent failure modes such as hallucination and poor abstention coverage. That matters for teams seeking to integrate LLM-driven decomposition into model-based systems engineering workflows because it signals where LLM outputs are presently reliable and where human validation remains necessary.

What to watch

Follow adoption of the Composite Quality Score (Q) and whether other researchers or practitioners extend the two-dataset evaluation to larger, domain-diverse corpora. A concrete next milestone will be published replications that apply the framework beyond the fictive system and refrigerator decomposition used in these experiments.

References and concrete facts drawn from the paper: submission date 7 Jul 2026; authors Niels Potters and Theo Hofman; the framework benchmarks GEN-DSMs against GT-DSMs; metrics named Completeness, Correctness, Coupling Density, Selective Accuracy, Abstention Coverage, Entropy and Fleiss' kappa; experiments on two datasets (a fictive abstract system and a real-world refrigerator decomposition); the proposed Composite Quality Score (Q).