On the Lexical Superstition of Large Language Models for Code Comprehension: Re-evaluation on Code of Low Lexical Quality

A semantics-preserving renaming framework reveals that LLMs disproportionately rely on identifier names, even when program structure alone should suffice.

Xin Shen; San-Zhuo Xi; Yali Du; Ming Li· 2026· DOI 10.48550/arXiv.2609.26388

The core problem

Large language models (LLMs) are now widely deployed for code-related tasks, including code comprehension, summarization, and bug detection. In naturally occurring code, identifier names are statistically informative: they often correlate with the function's purpose, data types, and control flow. However, this correlation is not a guarantee of reliability. The authors ask whether current LLMs assign disproportionate weight to lexical cues when renaming preserves program structure. They introduce **Face/Off**, a semantics-preserving identifier-renaming framework, and evaluate progressive naming conditions across multiple models and code-comprehension tasks. The central hypothesis is that LLMs exhibit a *lexical superstition*: they trust identifier names even when those names are misleading, and they degrade when names are removed or obfuscated. This matters because code comprehension should ideally be grounded in formal program semantics, not in surface-level lexical patterns that may be absent, misleading, or adversarial.

Across the evaluated models and primary tasks, lexical overemphasis is pervasive. Performance generally decreases as identifier information is removed or made misleading. More strikingly, model outputs are often directed toward the meanings suggested by misleading names, even when those meanings contradict the actual program structure. This pattern persists under representative prompt- and fine-tuning-based interventions, suggesting that lexical overemphasis is an entrenched problem rather than a superficial artifact. The type-inference control confirms a boundary: naming effects are smaller when the answer is locally recoverable without the target name. In other words, when the model can infer the correct answer from local structural cues alone, it relies less on the identifier. These results do not imply that identifiers are unhelpful; rather, they reveal a systematic vulnerability in how current LLMs balance lexical cues against program structure.

Large language models (LLMs) are now widely deployed for code-related tasks, including code comprehension, summarization, and bug detection. In naturally occurring code, identifier names are statistically informative: they often correlate with the function's purpose, data types, and control flow. However, this correlation is not a guarantee of reliability. The authors ask whether current LLMs assign disproportionate weight to lexical cues when renaming preserves program structure. They introduce **Face/Off**, a semantics-preserving identifier-renaming framework, and evaluate progressive naming conditions across multiple models and code-comprehension tasks. The central hypothesis is that LLMs exhibit a *lexical superstition*: they trust identifier names even when those names are misleading, and they degrade when names are removed or obfuscated. This matters because code comprehension should ideally be grounded in formal program semantics, not in surface-level lexical patterns that may be absent, misleading, or adversarial.

The authors design **Face/Off**, a framework that systematically renames identifiers while preserving program structure and semantics. The framework defines progressive naming conditions, ranging from natural, informative names to obfuscated or deliberately misleading names. For a given program P and a renaming function ρ, the transformed program P^{'} = ρ (P) is semantically equivalent to P; that is, for all inputs x, the observable behavior satisfies [[P]] (x) = [[P^{'}]] (x) . The evaluation spans multiple LLMs and code-comprehension tasks, including type inference, variable misuse detection, and program summarization. The authors also test representative prompt-based and fine-tuning-based interventions to see whether the lexical overemphasis can be mitigated. A type-inference control is included to probe a boundary condition: when the answer is locally recoverable without the target name, naming effects should be smaller. The experimental design thus isolates lexical influence from structural reasoning.

The findings motivate evaluations and modeling methods that preserve the benefits of natural code regularities while keeping conclusions grounded in accurate, formalized code semantics. The authors argue that lexical cues are not inherently bad—they are statistically informative in natural code—but LLMs must learn to weigh them appropriately against structural evidence. The persistence of the effect under interventions suggests that current training objectives and prompting strategies may inadvertently reinforce lexical shortcuts. A promising direction is to develop benchmarks and training signals that explicitly reward structural reasoning, for example by including semantics-preserving renaming as a data augmentation or adversarial evaluation. The type-inference control offers a useful diagnostic: tasks where the answer is locally recoverable are less susceptible, indicating that the vulnerability is tied to the need for global or non-local reasoning. Ultimately, the paper calls for a more principled treatment of identifiers in code LLMs, one that treats them as probabilistic hints rather than authoritative labels.

CS practitioners and researchers

Opening member content…