LLM Evaluators Exhibit Bias Across Languages
Research shows LLM evaluators are biased in multilingual contexts, favoring lower-resource languages.
LLM evaluators do not operate under the assumption of reliable, language-neutral scoring in multilingual settings. Experiments across 23 languages reveal significant score discrepancies, with lower-resource languages receiving more favorable evaluations. This bias is linked to model uncertainty and remains a significant predictor even after controlling for content difficulty.