Nemotron-Labs-3-Puzzle-75B-A9B compressed MoE LLM, 2x throughput

Nemotron-Labs released Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super designed for interactive deployment, in an arXiv submission first posted 5 July 2026 and revised 7 July 2026. The team says the model targets high user throughput constraints and is publicly available on Hugging Face.

What is Puzzle-75B-A9B and how was it built?

Nemotron-Labs-3-Puzzle-75B-A9B is a compressed hybrid mixture-of-experts large language model derived from Nemotron-3-Super. The paper describes a multi-stage pipeline that combines the Iterative Puzzle compression framework with knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head. The compression process jointly optimizes heterogeneous MoE pruning, active parameter budget, and Mamba pruning to reduce inference cost while aiming to preserve model quality.

The authors list Akhiad Bercovich and 69 other authors on the submission. The repository and model files are published on Hugging Face, enabling others to inspect and deploy the compressed checkpoint.

How much faster and more concurrent is the compressed model?

Puzzle-75B-A9B achieves concrete deployment gains over its parent in the paper's reported settings: on a single 8xB200 node the compressed model achieves "approximately 2x higher server throughput" than Nemotron-3-Super at matched user throughput constraints. For ultra-long-context workloads on a single H100 GPU, the compressed model increases 1M-token concurrency from 1 request to 8 requests.

Those two data points are the clearest performance claims in the paper: a near doubling of server throughput in an interactive serving setup on 8xB200, and an eightfold increase in concurrent 1M-token streams on a single H100 for long-context deployment.

How does Puzzle-75B-A9B perform on tasks and benchmarks?

The paper evaluates Puzzle-75B-A9B across a broad suite of reasoning, coding, multilingual, long-context, and agentic benchmarks. Despite substantial compression, the authors report that the model "retains strong downstream accuracy relative to the parent model" across a wide range of tasks. The submission positions the work as evidence that large hybrid MoE models can be substantially optimized for deployment efficiency while maintaining strong downstream capability.

The paper does not publish task-by-task numeric scores in the arXiv abstract, but emphasizes that the combined techniques—Iterative Puzzle compression, distillation, reinforcement learning, quantization and Multi-Token Prediction—were used to preserve accuracy through the pruning and compression stages.

Why it matters

Compressing hybrid MoE models while keeping their capabilities addresses a key practical barrier to deploying large, expert-sparse architectures: inference inefficiency under heavy interactive loads. Puzzle-75B-A9B’s reported near 2x server throughput improvement and the jump from one to eight concurrent 1M-token requests on an H100 reduce the hardware pressure for long-context and high-concurrency use cases. Operators running large language models for interactive services and long-context tasks will be able to evaluate whether the model’s trade-offs meaningfully cut cost or raise latency headroom in their own stacks.

What to watch

Watch for independent benchmarks and the Hugging Face release artifacts for task-level scores and latency numbers in real deployments. Replication studies that run the model on different hardware profiles and examine per-task accuracy versus Nemotron-3-Super will be the next concrete signals of practical impact. The arXiv DOI is 10.48550/arXiv.2607.04371 for reference.