LLM-as-Judge: Curriculum-Grounded Marking Pipeline for Exam Prep

LLM-as-Judge, a curriculum-grounded marking pipeline authored by Xiwei Xu, Chen Wang, Jacky Jiang, Phil Yang, Qian Fu, Mohan Dhall, Wenjie Zhang and Liming Zhu, was posted to arXiv on 16 Jun 2026 as arXiv:2606.17507. The paper describes a staged LLM workflow co-developed with an industrial partner that produces question-level marks and justifications for university-admission exam preparation.

What is the LLM-as-Judge pipeline and how does it work?

The pipeline is a configurable, curriculum-grounded system that first identifies the relevant topics, subtopics and cognitive demand of a question, then assembles verifiable and authorised context before using a staged LLM workflow to generate rubrics and allocate marks. It employs syllabus artefacts including prescribed verbs and outcomes, performance band descriptors, glossary definitions and marking-guideline principles to operationalise curriculum intent.

The workflow is staged: the LLM is used to generate question-specific rubrics that capture structured expectations of performance, and then to derive and evaluate marking criteria that are used to allocate marks to student responses. The design aims to improve consistency, transparency and alignment with official marking practices by making justifications traceable to authorised curriculum artefacts.

How does curriculum grounding change marking in this pipeline?

Curriculum grounding means the system assembles authorised context and syllabus artefacts to support each LLM judgement, rather than relying on prompt engineering alone. The paper lists concrete artefacts used: prescribed verbs and outcomes, performance band descriptors, glossary definitions and marking-guideline principles, which the pipeline uses to produce rubrics and marking criteria tied back to official curriculum documents.

Grounding operates at question level: the pipeline identifies the question's topic, subtopic and cognitive demand, then assembles the relevant authorised materials to inform rubric generation. That approach is presented as a way to make model outputs more verifiable and aligned with the expectations set by education authorities.

What did the authors evaluate and what were the findings?

The authors ran a preliminary evaluation and report that the proposed LLM-as-Judge pipeline "delivers marking outcomes comparable to human tutors" while providing justifications more traceable to authorised curriculum artefacts and marking standards. The pipeline has been integrated into an online study platform, where early deployment data provide initial insights into operational usage and manual overrides.

The paper emphasises design choices beyond prompt engineering: configurable pipelines, concrete syllabus artefacts and a staged LLM workflow. It notes the system was co-developed with an industrial partner to support exam preparation for university admission. The submission to arXiv lists eight authors and was submitted on 16 Jun 2026 as arXiv:2606.17507.

Why it matters

Grounding automated marking in authorised curriculum artefacts addresses two persistent problems in automated assessment: alignment with official learning goals and traceability of judgement. If models can produce rubrics and marks tied explicitly to syllabus verbs, band descriptors and glossary definitions, educators gain a clearer audit trail for automated decisions, and students receive feedback that maps to published standards.

This matters for high-stakes exam preparation because the paper explicitly frames the pipeline as a tool for university-admission preparation and reports early deployment with manual override data, signalling practical, not just experimental, use.

What to watch

Look for peer-reviewed evaluations or released deployment metrics that quantify agreement rates between the pipeline and human markers, and for details on the industrial partner’s deployment data, which the paper says provide initial insights into operational usage and manual overrides. The arXiv record is arXiv:2606.17507 and the submission date is 16 Jun 2026.