arxiv:2512.20578

Can LLMs Predict Their Own Failures? Self-Awareness via Internal Circuits

Published on Dec 23, 2025

· Submitted by

Amirhosein Ghasemabadi on Jan 6

#2 Paper of the day

University of Alberta

Upvote

Authors:

Amirhosein Ghasemabadi ,

Abstract

Large language models (LLMs) generate fluent and complex outputs but often fail to recognize their own mistakes and hallucinations. Existing approaches typically rely on external judges, multi-sample consistency, or text-based self-critique, which incur additional compute or correlate weakly with true correctness. We ask: can LLMs predict their own failures by inspecting internal states during inference? We introduce Gnosis, a lightweight self-awareness mechanism that enables frozen LLMs to perform intrinsic self-verification by decoding signals from hidden states and attention patterns. Gnosis passively observes internal traces, compresses them into fixed-budget descriptors, and predicts correctness with negligible inference cost, adding only ~5M parameters and operating independently of sequence length. Across math reasoning, open-domain question answering, and academic knowledge benchmarks, and over frozen backbones ranging from 1.7B to 20B parameters, Gnosis consistently outperforms strong internal baselines and large external judges in both accuracy and calibration. Moreover, it generalizes zero-shot to partial generations, enabling early detection of failing trajectories and compute-aware control. These results show that reliable correctness cues are intrinsic to generation process and can be extracted efficiently without external supervision.

View arXiv page View PDF GitHub 10 auto Add to collection

Community

AmirhoseinGH

Paper author Paper submitter 1 day ago

Can Large Language Models predict their own failures? 🧠⚡

We all know the critical bottleneck in GenAI: LLMs are incredible, but they can confidently hallucinate and make mistakes.

Until now, most fixes have been computationally massive — relying on expensive external judges, huge reward models, or costly training to make the LLM itself more robust.

This brings us to two fundamental questions:
❓ Do LLMs recognize when they're making mistakes?
❓ Can we make them self-aware about their own failures?

🚀 Introducing Gnosis: A lightweight self-awareness mechanism for frozen LLMs.
Named after the Greek word for knowledge/insight, Gnosis gives LLMs a form of introspection.

We add only ~5M parameters to enable a frozen LLM to verify its own outputs by decoding internal hidden states + attention patterns during inference — with negligible overhead and no external judge.

The results challenge the classic efficiency–accuracy trade-off:

🏆 Superior performance across domains
Despite being orders of magnitude smaller, Gnosis can outperform strong 8B reward models and proprietary judges like Gemini 2.5 Pro on both multi-step reasoning and factual/parametric knowledge QA (e.g., TriviaQA), across multiple backbones.

⚡ Real-time early failure detection
Gnosis doesn’t need to wait for the final token. By monitoring the generation trajectory in real time, it can predict an error before the model finishes — enabling early stopping, preventing bad outputs from reaching users, and saving significant compute.

This suggests something important: the model often already contains signals of impending failure during generation — we just needed the right mechanism to read them.

👇 code + models:

💻 Code: https://github.com/Amirhosein-gh98/Gnosis