Abliterated models lose obedience before they lose knowledge

A technical post on dev.to by the developer behind Grunz, a service that runs abliterated open models, argues that abliterated checkpoints are usually evaluated for the wrong thing. Thousands of these models sit on Hugging Face, and what degrades first is not knowledge but obedience.

What abliteration does

Abliteration finds a refusal direction in a model's activation space and projects it out of the weights. No gradient steps, no training data: it is a weight edit, not a finetune. The usual assumption is a capability trade-off: compliance gained, general intelligence lost, tested by whether the model still knows facts and writes well. The author says the damage shows up elsewhere.

Obedience degrades first

Across variants and model families, what goes first is instruction-following and output-format adherence. The model still knows the material, but it becomes measurably worse at respecting the instruct template and prefill, honouring stop sequences, staying inside a structured-output contract such as a JSON schema or tool-call syntax, and holding a system-prompt constraint across a long context. A checkpoint that scores within noise of its base on MMLU will fail structured output at a materially higher rate.

Why chat testing misses it

The standard check is a conversation: confirm the model no longer refuses, confirm the prose reads well. That cannot detect the failure: the model looks fine, often better than fine, because the refusal behaviour is gone. The problem appears when the model is wired into something that parses its output and the failure rate climbs. Perplexity does not help either: on a generic corpus it barely moves while format compliance falls off a cliff.

The recommendation: score format compliance separately from quality and independently of correctness. Send requests that specify an exact output contract, score binary compliance with the contract, not the content, and report a comparable rate. A well-formed wrong answer scores 1.0; a correct answer wrapped in prose that breaks the schema scores 0.

The quantization hunch

The author flags the interaction with quantization as a hunch he has not measured properly: the damage appears to compound with quantization damage, so an abliterated model seems to degrade faster down a quant ladder than its base on format adherence. If so, abliteration has already flattened weight structure that quantization then rounds away. He reports better luck at q6_K and above on smaller models; a q8_0 build of a 4B model is only about 4.3GB. A proper test would quantize a base model and its abliterated twin to the same bits per weight and calibration set, then measure compliance rate rather than perplexity. In production, he adds, tool-call parsers should be more forgiving, and compliance rate belongs in the metrics.