LLMs vs Bloom's Taxonomy: 20,700 generated educational questions

From Memorization to Creation, a paper by Xiaolong Wang and seven coauthors submitted 6 May 2026 evaluates six widely used large language models and their ability to generate higher-order educational questions. The study produces and analyzes 20,700 questions across computer science, K–12 math, and social-science domains, and its results were accepted by KDD 2026.

How did the researchers measure cognitive depth?

They used a hybrid human–AI evaluation protocol to generate and analyze 20,700 questions across computer science, K–12 math, and social-science, and they introduced quantitative metrics including cognitive shift intensity ("CogShift") and category drift to capture multi-level transitions. The authors also ran an interpretability analysis to examine metric-level correlations and the transparency of Chain-of-Thought prompting.

The protocol combines automated generation with human assessment to map outputs onto Bloom's Taxonomy, allowing the team to quantify whether model outputs remained at rote memorization or moved into higher-order thinking (application, analysis, synthesis, evaluation). The paper frames these measures as benchmarks for deploying LLMs in personalized learning systems.

Which models showed movement beyond memorization?

The paper reports model-specific gains from prompt engineering: a fine-grained prompting strategy reduced question repetitiveness by 24.45% for Qwen2.5-7B-Instruct and increased the proportion of higher-order cognitive level outputs by 11.53% for InternLM3-8B-Instruct; the authors also identify InternLM3 as superior in multi-level transitions.

Beyond those two named results, the study evaluates six LLMs in total and compares their behavior across domains. The fine-grained prompting approach is a central intervention: it both lowered redundancy in generated questions and nudged some models toward producing questions mapped to higher Bloom levels. The authors use category drift measures to track when a generated question shifts categories, and CogShift to quantify the intensity of those cognitive shifts.

What methodological details are important to know?

The study builds new, fine-grained prompts and pairs automated metrics with human judgments to assess cognitive level. The interpretability analysis examines correlations between the introduced metrics and Chain-of-Thought prompting, aiming to make the reasoning behind higher-order outputs more transparent. The dataset spans three curricular areas, allowing cross-domain comparisons rather than a single-subject focus.

The paper explicitly frames these contributions as: (1) a prompting strategy that reduces repetitiveness and raises higher-order outputs for specific models, (2) quantitative metrics (CogShift and category drift) for measuring cognitive transitions, and (3) an interpretability analysis linking metrics to Chain-of-Thought behavior.

Why it matters

If LLMs can be steered to produce questions that map to higher Bloom levels, they move from rote content generation toward tools that can prompt deeper student thinking. The study provides concrete, model-level evidence that prompt design materially affects output quality, and it supplies metrics that curriculum designers and platform engineers can use to assess cognitive depth. Educators and adaptive learning vendors are the most immediate stakeholders: the paper supplies both measurement tools and examples of prompt changes that altered outputs by measurable percentages.

What to watch

Look for follow-up work testing these metrics and prompts in classroom trials or production tutoring systems, and for broader benchmarking that reports full model-by-metric tables beyond the highlighted examples. The next concrete milestone will be replication of these gains (for example the 24.45% repetitiveness drop and the 11.53% increase in higher-order outputs) in user-facing educational deployments.

References and provenance: the above findings come from the arXiv paper "From Memorization to Creation: Evaluating the Cognitive Depth of LLM-Generated Educational Questions" by Xiaolong Wang, Zhe Zhao, Song Lai, Chaoli Zhang, Zijie Geng, Yu Tong, Ye Wei, and Qingsong Wen (submitted 6 May 2026; accepted by KDD 2026).