ReTeX: Recover Task Experts from a Merged Multi-Task Model

Jinwook Jung, Taegyu Kim, Kumju Jo and Sungyong Baik submitted a paper on 25 Jun 2026 that introduces Recover Task eXpert, or ReTeX. ReTeX predicts additive parameter offsets to undo parameter interference in merged multi-task checkpoints and, the authors report, recovers over 95% of individual-expert performance from a single merged checkpoint.

How does ReTeX recover task experts?

ReTeX treats parameter interference as a parameter perturbation and models that perturbation as an affine transformation, which the authors approximate with additive offsets. The framework trains an offset predictor that, given a merged model checkpoint, predicts per-expert additive offsets to restore each task expert’s parameters. To select the right expert when task identity is unknown, ReTeX uses a router-free task identifier based on SVD subspace signatures computed offline. At inference the identifier selects the task whose subspace yields the smallest projection residual for the input, then the offset predictor undoes the interference to recover the expert.

The paper highlights two concrete design moves: modeling interference as affine perturbation approximable by additive offsets, and replacing a runtime router with an SVD-based, offline signature matching step. Both components operate on a single merged checkpoint rather than storing and loading multiple redundant expert components at inference.

How well does ReTeX perform?

ReTeX recovers more than 95% of individual-expert performance in both vision and NLP domains, the authors state. The method also significantly improves generalization to unseen tasks, and the predicted parameter offsets produce emergent adaptive interpolation of expert knowledge for out-of-distribution tasks. In other words, ReTeX not only restores seen experts but the offset predictions adaptively interpolate knowledge from seen experts to handle unseen tasks.

The paper contrasts ReTeX with static merging, which consistently suffers from parameter interference, and with many dynamic merging approaches that the authors say rely on costly storage and loading of redundant expert components at inference. ReTeX is presented as a single-checkpoint alternative that undoes interference via learned additive offsets and selects experts with an SVD-based residual criterion.

Why it matters

ReTeX addresses a practical pain point in multi-task model merging: interference between task-specific parameters degrades per-task performance after consolidation. By framing interference as a recoverable perturbation and proposing a lightweight, router-free identifier, the approach promises to reduce the runtime cost of dynamic methods while restoring near-expert accuracy from one merged checkpoint. The reported recovery of over 95% of individual-expert performance in both vision and NLP domains suggests the technique could make consolidated multi-task models more viable where storage or runtime cost prevents keeping separate expert components.

What to watch

Watch for open-source code and replication: the authors note their code is available at a provided URL. Next milestones to validate the approach will be community reproductions across larger task pools, ablations of the affine-to-additive approximation, and quantitative comparisons against dynamic methods that keep per-expert components at inference.