Feeling of Knowing in Large Language Models

NeurIPS 2026

Zichuan Fu¹ Xian Wu² Jingtong Gao¹ Wenlin Zhang¹ Jiaxuan Li¹ Binhao Wang¹ Yimin Deng^1,3 Guojing Li^1,4 Xiaopeng Li¹ Derong Xu^1,5 Yefeng Zheng⁶ Xiangyu Zhao¹

¹City University of Hong Kong ²Tencent Jarvis Lab ³Xi'an Jiaotong University
⁴Renmin University of China ⁵University of Science and Technology of China ⁶Westlake University

Paper (coming soon) Code

(a) Causal tracing of the FoK signal in a 7B LLM

FoK signal across normalized depth for Qwen2.5 sizes — (b) FoK signal across normalized depth

Figure 1. The FoK signal localizes late under causal restoration. Panel (a) shows a 7B example across tokens and layers; panel (b) aggregates FoK recovery across Qwen2.5 sizes over normalized depth.

Abstract

Post-training updates change what large language models (LLMs) can answer, but may leave their answerability judgments misaligned. We study this knowledge–judgment desynchronization through functional feeling of knowing (FoK), which supports selective generation, retrieval triggering, and model routing without implying subjective experience. We propose two trainable channels: a knowledge channel that changes answerability and a FoK channel that judges the current knowledge state. Three stages train them: on-policy supervised fine-tuning (OP-SFT) updates knowledge from corrected on-policy answers; knowledge-delta FoK training (KΔ-FT) adapts judgments across intermediate knowledge states; and FoK-guided policy optimization (FGPO) trains the subsequent answer or explanation. Factual-QA experiments, cross-dataset evaluation, and three sequential updates show that explicit FoK alignment improves judgment quality on updated knowledge. MMLU and ARC-Easy separately assess capability preservation.

Key Takeaways

🧠 FoK sits late in the network. Under causal tracing, the FoK signal peaks later in model depth than factual associations, so FoK acts as a judgment over the model's knowledge state rather than as factual recall itself.
🔀 Knowledge updates do not update FoK. Fine-tuning changes what the model can answer, but its judgment of whether it can answer does not follow. Unless FoK is updated explicitly, the mapping from knowledge to self-judgment breaks.
💭 Context sharpens the judgment. Reasoning before the judgment moves the FoK score away from uncertainty, so FoK should be judged from the model's current context, not from the question alone.
🛠️ Three stages re-synchronize the two channels. OP-SFT updates knowledge, KΔ-FT trains FoK over the knowledge trajectory, and FGPO lets the model answer or explain what it is missing within one generation.

Method

Starting from a base LLM M, we represent post-training changes as parameter deltas. The knowledge delta κ forms the knowledge-retrieval channel, which changes what the model knows. The FoK delta φ forms the FoK channel, which judges whether the current knowledge state supports a correct answer. The FoK score is read from the model's yes/no judgment:

J = P(yes) / (P(yes) + P(no))

Stage 1 · OP-SFTUpdates κ on the model's own sampled responses, supervising only the corrected answer span, and saves the trajectory of knowledge states κ₀, …, κ_T.

Stage 2 · KΔ-FTTrains φ over the OP-SFT checkpoint trajectory, so FoK follows the knowledge as it evolves instead of staying tied to a fixed knowledge boundary.

Stage 3 · FGPOTrains the model to reason, judge yes or no, and then answer or explain why it cannot, with one reward over the judgment and the response.

FGPO generates a short context trace, makes a FoK judgment, and produces an answer or explanation in one pass:

<think>...reasoning...</think> <answer>yes|no</answer>
<response>answer or explanation</response>

Results

We update Qwen2.5-7B-Instruct with TriviaQA questions and evaluate on the update set (Train), held-out TriviaQA questions (Test), and MMLU and ARC-Easy (General). Bias is the signed mean of illusion and omission errors; closer to 0 is better.

**Table 1.** Overall performance after one TriviaQA knowledge update on Qwen2.5-7B (selected rows; the paper reports all baselines).
Method	Train Acc. ↑	Train AUROC ↑	Test Acc. ↑	Test AUROC ↑	Test Bias →0	MMLU ↑	ARC ↑
Base	0.680	0.702	0.672	0.688	0.402	0.687	0.811
Base + EliCal	0.637	0.808	0.622	0.810	0.254	0.709	0.504
Vanilla K-SFT	0.874	0.458	0.546	0.519	0.500	0.713	0.840
Vanilla K-SFT + Prob	0.874	0.590	0.546	0.541	0.498	0.713	0.840
Vanilla K-SFT + SemEnt	0.874	0.538	0.546	0.526	0.504	0.713	0.840
Ours	0.961	0.817	0.692	0.809	0.308	0.711	0.830

Stage-wise ablation

**Table 2.** Held-out stage-wise ablation on Qwen2.5-7B.
Stage	Acc. ↑	AUROC ↑	Illu. ↓	Om. ↓	Bias →0
Base	0.672	0.688	0.683	0.122	0.402
Stage 1: + OP-SFT	0.679	0.603	0.601	0.283	0.442
Stage 2: + KΔ-FT	0.699	0.770	0.478	0.187	0.333
Stage 3: + FGPO	0.692	0.809	0.432	0.184	0.308

Where the remaining error comes from

Illusion and omission rates for each method — **Figure 2.** FoK error decomposition for Table 1. Red circles show illusion, the conditional rate of false “can answer” judgments; blue squares show omission, the conditional rate of false “cannot answer” judgments. The horizontal gap shows which error source determines the signed Bias value.

Coverage–accuracy trade-off

Coverage and accuracy as the FoK threshold varies on Qwen3-8B — **Figure 3.** Coverage–accuracy trade-off on Qwen3-8B (thinking mode). Each panel varies the FoK threshold and plots coverage (solid), conditional accuracy on the answered subset (dashed), and overall accuracy with unanswered examples counted as incorrect (dotted).

For Base and Vanilla K-SFT the curves are nearly flat: lowering coverage does not raise accuracy, so their FoK scores do not rank answerability. OP-SFT + KΔ-FT restores a monotonic trade-off. FGPO flattens the curves for a different reason: its scores collapse to 0 or 1, so the policy has already committed to answer or abstain, and it answers more selectively, with higher accuracy on the answered subset at the cost of lower coverage.

BibTeX

@inproceedings{fu2026feeling,
  title     = {Feeling of Knowing in Large Language Models},
  author    = {Fu, Zichuan and Wu, Xian and Gao, Jingtong and Zhang, Wenlin and Li, Jiaxuan and Wang, Binhao and Deng, Yimin and Li, Guojing and Li, Xiaopeng and Xu, Derong and Zheng, Yefeng and Zhao, Xiangyu},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}