Competency Questions in CQ Verification with OE-Assist: Study

When CQs Go Wrong: Challenges in CQ Verification with OE-Assist, submitted 23 Jun 2026 by Anna Sofia Lippolis and colleagues, presents an experiment showing how Competency Questions can derail ontology verification when they are ambiguous or overly complex. The paper documents a study where 19 participants performed CQ-verification across 20 tasks with an LLM assistant supporting ontology evaluation.

What did the study do and find?

The paper ran an experiment with data from 19 participants who completed 20 CQ-verification tasks while using an LLM assistant to support ontology evaluation; the results show a clear need for tools to refine CQs before publishing them. The authors report that CQ-verification remains time-consuming and error-prone because it requires careful interpretation of linguistic nuances and precise alignment with formal ontology constructs, and that ambiguities and complexity in CQs lead to inconsistent modelling decisions and verification outcomes.

The study appears in a submission titled "When CQs Go Wrong: Challenges in CQ Verification with OE-Assist" (arXiv:2606.24619), and the paper was accepted as a poster at the 23rd European Semantic Web Conference (Satellite Event).

How do CQs go wrong during verification?

CQs fail verification when natural language ambiguity and excessive complexity prevent consistent mapping to formal ontology constructs, producing inconsistent modelling decisions and verification outcomes. The paper notes that CQ-verification requires both interpretation of linguistic nuances and precise alignment with ontology constructs, and that these two demands make the process time-consuming and error-prone.

The authors highlight that ambiguities and excessive complexity in CQs are principal sources of trouble, and they connect those weaknesses to downstream inconsistencies in modelling and verification. The paper frames Competency Questions as "the central component of CQ-verification," stressing that flaws in CQs propagate into the overall ontology-engineering workflow.

Who wrote the paper and how was the experiment set up?

The study was authored by Anna Sofia Lippolis, Mohammad Javad Saeedizade, Robin Keskisärkkä, Aldo Gangemi, Eva Blomqvist, and Andrea Giovanni Nuzzolese. The paper was submitted to arXiv on 23 Jun 2026 (arXiv:2606.24619) and includes an experiment using an LLM assistant to support participants conducting CQ-verification tasks. The dataset for the experiment comprises 19 participants and 20 verification tasks; the authors used that data to evaluate what makes a CQ challenging and to explore possible solutions for improving user performance during verification.

Why it matters

Faulty or unclear Competency Questions undermine the very purpose of CQ-verification, which is to ensure an ontology models its intended purpose. If CQs are ambiguous or too complex, verification becomes unreliable and ontology modelling decisions can diverge, creating downstream costs during development and maintenance. The study points to a practical intervention: refining CQs before they are published or used in verification can reduce ambiguity and complexity and improve consistency.

What to watch

Look for follow-up work from the authors on tooling aimed at CQ refinement and for the poster presentation at the 23rd European Semantic Web Conference (Satellite Event). The arXiv entry includes a DOI link (https://doi.org/10.48550/arXiv.2606.24619) that tracks the submission and any future versions.

References

• Paper: "When CQs Go Wrong: Challenges in CQ Verification with OE-Assist," Anna Sofia Lippolis et al., arXiv:2606.24619, submitted 23 Jun 2026.
• Experiment details cited in the paper: 19 participants, 20 CQ-verification tasks, use of an LLM assistant.