LLM distillation: scaling laws and FinHeadlineMix release

Lavinia Ghita, Dhruv Desai and Ioana Boier submitted an arXiv paper (arXiv:2606.24747) on 23 Jun 2026 that derives empirical scaling laws for task-specific LLM distillation and publishes the dataset FinHeadlineMix. The 24-page paper, which includes 13 figures, quantifies how in-domain and general-knowledge performance change with dataset size, compression ratio, supervision format and iterative pruning schedule in a quantitative finance setting.

What did the authors measure and find?

The paper measures how in-domain task quality and general-knowledge benchmarks respond to compression across multiple variables, and finds a consistent tradeoff: in-domain quality degrades predictably under compression while general-knowledge benchmarks collapse well before the same compression point. The authors quantify scaling behavior with respect to dataset size, compression ratio, supervision format and iterative pruning schedule, and report that supervision format is the key driver of the tradeoff between retained domain ability and loss of general knowledge.

They show that chain-of-thought style supervision can actively recover general knowledge that pruning erases. The paper introduces a blended chain-of-thought supervision loss that stabilizes KL-divergence distillation over reasoning traces, improving robustness of distilled models on reasoning tasks while compressing for a specific domain.

How did the paper test distillation methods?

The authors compared logit-based and LoRA-based distillation under iterative structural pruning inside a quantitative finance application, using those methods to explore how different supervision formats and pruning schedules affect both in-domain and general knowledge performance. Their experiments place the quantitative finance task as the domain-specific benchmark and use general-knowledge benchmark suites to probe broader capabilities.

The experimental setup explicitly contrasts logit-based distillation with LoRA-based approaches, and layers iterative structural pruning on top of those distillation techniques to trace performance as compression increases. To address instability in distillation over reasoning traces, the paper proposes a blended chain-of-thought supervision loss that stabilizes KL-divergence distillation over reasoning traces.

Why it matters

These scaling laws give practitioners concrete factors to weigh when compressing LLMs for a specific domain: dataset size, compression ratio, supervision format and pruning schedule interact in predictable ways. The finding that general-knowledge benchmarks collapse earlier than in-domain tasks warns that aggressive compression tuned solely for a narrow task can unintentionally erase useful general capabilities. The paper’s emphasis on supervision format, and the demonstrated benefit from chain-of-thought supervision, points to training-data and loss engineering as levers to recover or retain broader abilities even at high compression.

The authors also make practical resources available: they release the headline dataset FinHeadlineMix, the scaling law results and practical recommendations intended to help teams make domain-specific compression decisions. These releases provide testable artifacts for teams working on finance-focused deployments.

What to watch

Watch for replication of these scaling laws beyond the paper’s quantitative finance testbed and for open-source use of FinHeadlineMix in follow-up work. The paper’s submitted materials include dataset and results intended for reuse; whether those artifacts drive similar tradeoff curves in other domains will determine how broadly the recommendations apply.

Additional factual details: the paper appears on arXiv as arXiv:2606.24747, was submitted on 23 Jun 2026, and is 24 pages long with 13 figures. The subject classifications include Artificial Intelligence and Computational Engineering, Finance, and Science.