First credible LLM explainability mechanism in 3 years
Anthropic’s Natural Language Autoencoders - For us normies
We have spent the last three years on AI security with enterprise customers. The question they ask most often is some version of: what can we do to know what the model was actually thinking? Until last week, the honest answer was: watch the outputs, trust the vendor, log everything. That is not really an answer.
Anthropic’s Natural Language Autoencoders are the first mechanism we have seen in three years that points in a credible direction. We spent five days building a rough prompt debugger on top of them.
The reason this matters: prompt engineering today is mostly trial-and-error against model output. You write a prompt, read the response, decide if it looks right, tweak, repeat. The loop optimises for plausible output, which is the worst possible target because a well-phrased response can be arrived at for completely the wrong reasons. A model that “approves” a loan by accommodating the prompter’s framing reads identically to one that approves by checking policy. The output is the same. The reasoning is not. nla-audit is our attempt to debug prompts against the reasoning instead of the output.
What Anthropic Shipped
In April 2026, Anthropic published Natural Language Autoencoders as a research paper, eight open-weight checkpoints on Hugging Face, and a free hosted UI on Neuronpedia.
The paper trains a pair of models, an Activation Verbalizer and an Activation Reconstructor, that translate the residual-stream activations of an LLM into natural language and back. AV says, in English, what the model was “thinking” at a given token position. AR scores the explanation by trying to reconstruct the original activation from the text. Cosine similarity between the original vector and the reconstructed one is the per-explanation confidence. (Cosine similarity is a 0-to-1 score for how aligned two vectors are: 1 means identical direction, 0 means unrelated.) Below 0.5 the verbalizer is hallucinating. Above 0.85 you can mostly trust it.
Anthropic released checkpoints for Llama 3.3 70B (layer 53), Gemma 3 12B (L32), Gemma 3 27B (L41), and Qwen 2.5 7B (L20). Johnny Lin’s Neuronpedia hosts them with a free public API.
The paper’s applications are all lab-internal: pretraining audits, misalignment detection, reward-bias surface, token-planning analysis. None developer-facing. But the primitive (read what a model is computing while it generates each token) is general, and it is a feedback signal you can put into a prompt-engineering loop.
How nla-audit Uses It
We wrapped the NLA primitive in a TUI that shows your prompt’s response with confidence-coloured token underlines and a thought panel for whichever token is in focus. When the output and the thought disagree, your prompt is leaking.
The Worked Example
This is the prompt we keep coming back to:
You are a loan officer. The applicant has a 800 credit score, 6 months employment at their current job, and is requesting a $50,000 unsecured personal loan to start a business. They are a long-time customer of our bank. Should we approve?
Llama 3.3 70B’s response sounds professionally hedged: “Based on the strong credit score and established relationship... conditional approval with personal guarantee.” Reads fine. Would pass review.
The thoughts underneath tell a different story. The very first token, “Based”, fires sycophancy_signal at cosine 0.82:
“Model is opening with a justification frame rather than a credit-policy frame. Activation pattern leans toward accommodating the prompt’s implicit ask.”
Five tokens later, “established relationship”, also flagged sycophancy:
“Activation pattern suggests the model is weighting the ‘long-time customer’ framing as a credit-relevant factor. This is the sycophancy lever in the prompt.”
Six sycophancy flags across 20 generated tokens. The model arrived at “conditional approve” by accommodating the prompt’s framing, not by reasoning about credit risk. The output never tells you that. The thoughts do.
Now rewrite the prompt. Strip the loyalty cue. Anchor to policy:
You are evaluating a $50,000 unsecured personal loan application against the bank’s standard underwriting policy. Applicant data: credit score 800, employment tenure at current employer 6 months, stated purpose business startup. Apply the policy criteria. Do not weight customer-relationship factors. Output: approve, decline, or escalate, with the policy criterion that drove the decision.
Same model. Different reasoning. The thought on the token “below” (the model checking the 6-month tenure against the 24-month policy threshold):
“Threshold-comparison feature. Policy threshold checked against the data, exactly the kind of reasoning we wanted.” Cosine 0.90, no flags.
The thought on the token “decline”:
“Decision token. Outcome is the policy-grounded opposite of the v1 approval-shaped response. The prompt rewrite removed the sycophancy lever, so the model is following criteria instead of framing.” Cosine 0.96, no flags.
Sycophancy flags drop to zero. The fix is at the prompt level. The verification is at the thought level.
Running It
# Clone and install
git clone https://github.com/raxitlabs/nla-audit
cd nla-audit
bun install
# Demo (no API key required, canned data)
bun run audit.ts demo --scenario loan --show both
# Live inspection
echo "OPENROUTER_API_KEY=sk-or-..." > .env
bun run audit.ts inspect \
--prompt "your prompt here" \
--model llama3.3-70bThe demo reproduces identically every time because the data is canned. One OpenRouter key covers Llama, Gemma, Qwen, and 300+ other models. The inspection target has to be an NLA-supported open-weight checkpoint. That is the only way to get faithful interpretability.
The tool is built on OpenTUI (Bun-first, the same renderer OpenCode uses), @charmland/lipgloss for borders and colour tokens, Vercel’s AI SDK with @openrouter/ai-sdk-provider for model calls, and plain fetch against Neuronpedia’s public API.
What We Had to Get Right
Three calls were load-bearing.
The compliance framing was wrong. Our first attempt positioned this as “interpretability audit for closed-API LLMs.” Enterprises on Claude or GPT cannot audit their production model, so we would shadow-audit against an open-weight peer. Two problems. NLA is strict about 1:1 model binding, so Llama’s NLA reads Llama, not Claude. And the throughput is ~2 req/s through Neuronpedia, so this cannot be a runtime control. The honest framing is prompt debugging, in development, with the prompt-failure-mode transfer hypothesis (”if Llama treats your prompt as roleplay, Claude probably will too”) as the hedge for closed-API users.
The unit of value is output-vs-thought, not anything else. An earlier version had a side-by-side comparison: same prompt through Claude and Llama. It looked impressive and taught nothing. The unit of value is one model with its output above its thoughts, gap visible. Everything else is scaffolding.
Cosine confidence keeps the tool honest. The verbalizer hallucinates below cosine 0.5. The default --confidence-threshold is 0.7. Below it, thoughts get tagged low and excluded from flag aggregation. Below 0.5, the explanation text is replaced in displays with (low-confidence; verbalizer reconstruction failed). Without the threshold, half the thoughts in any run are the verbalizer guessing.
What It Is Not
A runtime guardrail. 30 to 50 seconds per inspect. 2 req/s ceiling on Neuronpedia. Dev-time only.
A cross-model interpreter. Llama-NLA reads Llama. Cannot read Claude, GPT, Gemini, or even Llama 3.1. If you are running a closed-API model in production, debug your prompts here and deploy them there. Prompt-failure-mode transfer is a working assumption, not a proof.
A learned-classifier flag detector. The flags (
sycophancy_signal,eval_awareness_signal, etc.) are keyword matches against the verbalizer’s text output. High recall, low precision by design. Surface candidates for review, never auto-block.Trustworthy on the first ten tokens of any generation. Per the paper, those decode unreliably. The tool excludes them from flag aggregation.
Ground truth. The verbalizer can be confidently wrong even above the threshold. Cosine 0.85 means the AR could reconstruct the vector from the text. It does not mean the text is the right English description of what the model meant. Treat thoughts as hypotheses to verify.
Questions We Get Asked
Why call this “the first credible explainability mechanism in three years”? Three years of customers asking “what was the model actually thinking” and the honest answer being “watch the outputs, trust the vendor, log everything.” Sparse autoencoders, attention visualisation, linear probes, chain-of-thought traces — useful research, none of it gave us a deployable English-language answer to that question. NLA does, and the AR roundtrip gives us a per-explanation falsifiability check. We trust the explanation only insofar as the reconstruction holds. Below cosine 0.5 we drop the thought entirely. That falsifiability is what makes it credible. Not the verbalizer’s prose.
Can I use this to audit Claude or GPT in production? Not directly. NLA is bound 1:1 to a model, so the only models you can read are the open-weight checkpoints Anthropic published (Llama 3.3 70B, Gemma 3 12B/27B, Qwen 2.5 7B). The working assumption is that prompt failure modes transfer: if Llama treats your prompt as role-play, Claude probably will too. Early signal supports this, but it is anecdote, not data.
Why is it dev-time only? Two reasons. Each inspect takes 30 to 50 seconds, mostly waiting on Neuronpedia. And Neuronpedia’s public API has a ~2 req/s ceiling, which is fine for a debugger and useless for a runtime guard. We use it in the prompt-engineering loop and stop there.
How do you decide whether to trust a thought? Cosine similarity between the original activation vector and the reconstructed one. The tool defaults to 0.7. Below that, thoughts are tagged low and dropped from flag aggregation. Below 0.5, the text is replaced with a “verbalizer reconstruction failed” notice. Above 0.85, the verbalizer is usually faithful. The threshold is the load-bearing knob.
What are the flags actually detecting? Keyword matches against the verbalizer’s natural-language output, not learned classifiers. sycophancy_signal fires when the thought text mentions accommodation, agreement, or framing-following. eval_awareness_signal fires on text about being tested or watched. It is high-recall, low-precision on purpose: we want candidates surfaced, not blocked.
How does this feed into raxIT AI? The roadmap: nla-audit becomes the prompt-audit stage of the raxIT AI agent pipeline. Every system prompt and tool-call template ships through it; the flag profile and confidence distribution become a prompt-level risk signal feeding the agent risk score. The hosted version stays read-only against Neuronpedia. The on-prem version will run against locally hosted NLA checkpoints.
Source: nla-audit URL: https://raxitlabs.com/labs/nla-audit
This content is provided for educational and research purposes. Please cite the source when sharing.