I Think, Therefore I Am Under-Qualified? A Benchmark for Evaluating Linguistic Shibboleth Detection in LLM Hiring Evaluations
amanchadha
Abstract
A benchmark evaluates Large Language Models' response to linguistic markers that reveal demographic attributes, demonstrating systematic penalization of hedging language despite equivalent content quality.
This paper introduces a comprehensive benchmark for evaluating how Large Language Models (LLMs) respond to linguistic shibboleths: subtle linguistic markers that can inadvertently reveal demographic attributes such as gender, social class, or regional background. Through carefully constructed interview simulations using 100 validated question-response pairs, we demonstrate how LLMs systematically penalize certain linguistic patterns, particularly hedging language, despite equivalent content quality. Our benchmark generates controlled linguistic variations that isolate specific phenomena while maintaining semantic equivalence, which enables the precise measurement of demographic bias in automated evaluation systems. We validate our approach along multiple linguistic dimensions, showing that hedged responses receive 25.6% lower ratings on average, and demonstrate the benchmark's effectiveness in identifying model-specific biases. This work establishes a foundational framework for detecting and measuring linguistic discrimination in AI systems, with broad applications to fairness in automated decision-making contexts.
Community
This paper proposes and validates a controlled benchmarking framework to detect and quantify linguistic shibboleth bias—subtle language cues like hedging—in LLM-driven hiring evaluations, revealing systematic penalization of certain linguistic styles despite equivalent content.
➡️ 𝐊𝐞𝐲 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬 𝐨𝐟 𝐨𝐮𝐫 𝐋𝐢𝐧𝐠𝐮𝐢𝐬𝐭𝐢𝐜 𝐒𝐡𝐢𝐛𝐛𝐨𝐥𝐞𝐭𝐡 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤:
🧪 𝑪𝒐𝒏𝒕𝒓𝒐𝒍𝒍𝒆𝒅 𝑳𝒊𝒏𝒈𝒖𝒊𝒔𝒕𝒊𝒄 𝑽𝒂𝒓𝒊𝒂𝒕𝒊𝒐𝒏 𝑭𝒓𝒂𝒎𝒆𝒘𝒐𝒓𝒌: Introduces a systematic methodology to generate semantically equivalent interview responses that differ only in specific sociolinguistic features (e.g., hedging), enabling attribution of score differences directly to linguistic style bias.
🧩 𝑯𝒆𝒅𝒈𝒊𝒏𝒈 𝑩𝒊𝒂𝒔 𝑪𝒂𝒔𝒆 𝑺𝒕𝒖𝒅𝒚: Constructs a 100-question hiring dataset with paired confident/hedged responses and tests across 7 LLMs, finding hedged answers receive 25.6% lower ratings on average and are more often rejected despite identical content.
🧠 𝑭𝒂𝒊𝒓𝒏𝒆𝒔𝒔 𝑨𝒖𝒅𝒊𝒕 𝑬𝒙𝒕𝒆𝒏𝒔𝒊𝒃𝒊𝒍𝒊𝒕𝒚: Framework generalizes to other shibboleths like accent markers and register variations, providing a reproducible, model-agnostic tool for systematic bias detection and informing debiasing strategies in high-stakes AI decision systems.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective (2025)
- Robustly Improving LLM Fairness in Realistic Settings via Interpretability (2025)
- Signal or Noise? Evaluating Large Language Models in Resume Screening Across Contextual Variations and Human Expert Benchmarks (2025)
- Fairness Is Not Enough: Auditing Competence and Intersectional Bias in AI-powered Resume Screening (2025)
- Evaluating the Promise and Pitfalls of LLMs in Hiring Decisions (2025)
- Fairness is Not Silence: Unmasking Vacuous Neutrality in Small Language Models (2025)
- When Meaning Stays the Same, but Models Drift: Evaluating Quality of Service under Token-Level Behavioral Instability in LLMs (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper