MAG: Unsupervised Activation Geometry for LLM Features

Unsupervised Features Mining via Activation Geometry, submitted to arXiv on 5 July 2026 as arXiv:2607.04222, introduces Mining via Activation Geometry (MAG), an unsupervised framework for extracting reasoning features from large language model activations. The paper is authored by Amit LeVi, Elad David and Max Fomin and is listed with a journal reference to ICML 2026, workshop on Failure Modes in Agentic AI (FAGEN).

What is MAG and how does it extract features?

MAG extracts reasoning features by prepending the same natural-language instruction Q to every input p, then measuring the change in the model's internal representation using m(Q | p) − m(p) at a single readout point, a procedure the authors present as an unsupervised probe. The paper frames Q as an instruction that defines the reasoning feature of interest, for example “Can this object be found in the desert?” or “Is this prompt malicious?”, and compares activations with and without that instruction to reveal the feature signal.

MAG is described as a family of methods: the authors explore eight different MAGs. Each variant operationalizes the core idea — inspect how a fixed instruction shifts activations — and then uses those shifts to identify directions in activation space that correspond to reasoning features. The framework requires no labeled concept examples, distinguishing it from approaches that begin with human-provided concept labels.

How well does MAG work?

The authors report that the extracted reasoning features predict the model's own world understanding and judgment, can often be approximated by a single activation direction, and that linearity varies across features, with “some features more linearly represented and some less.” They also show that this linear representation, described as vector steering, can change the LLMs' decisions through activation steering by injecting reasoning features. The paper uses the same MAG-based procedure to select training datasets for prompt-injection classifier probes and finds that RFD-based similarity achieves 94.7% Top-1 and 100% Top-2 accuracy when choosing datasets.

Beyond the headline metrics, the authors emphasize two practical outcomes: first, that many reasoning features compress into a single direction in activation space; second, that those directions can be used both to probe what a model “knows” and to steer model behavior by manipulating activations at a readout point.

Why it matters

MAG offers an unsupervised alternative to concept probing that does not begin from human-labeled exemplars but instead uses instruction-conditioned activation geometry to surface features the model itself uses. That matters because the method sidesteps a dependence on predefined labels while producing concrete artifacts — activation directions — that are usable both for analysis and intervention. The reported 94.7% Top-1 and 100% Top-2 accuracy for RFD-based dataset selection points to an immediate application: choosing training data for classifier probes based on activation geometry rather than ordinary activation similarity.

These findings bear on two active areas: interpretability and robust probing. If many reasoning features are approximately linear and can be steered, researchers gain a controllable handle on model internals. If dataset selection can be reliably guided by RFD-based similarity, probe builders and evaluators may avoid mismatched training splits that degrade downstream performance.

What to watch

Watch the ICML 2026 workshop on Failure Modes in Agentic AI (FAGEN), where the paper is listed, for further discussion and for any accompanying materials presented there. Also follow whether subsequent work replicates the claim that many features reduce to a single activation direction and whether RFD-based similarity proves broadly useful beyond the prompt-injection classifier probes demonstrated in this paper.

Paper and metadata: arXiv:2607.04222, submitted 5 Jul 2026, authors Amit LeVi, Elad David and Max Fomin, journal reference ICML 2026 (FAGEN).