Audio LLMs Make More Emotion Errors When Speech and Text Disagree — Artificial Intelligence

A new arXiv paper introduces a continuous incongruence score for speech emotion recognition and reports that three audio-capable language models are more likely to err when acoustic and textual emotional cues conflict across acted, conversational and spontaneous speech.

Voice-interactive large language models can process both the acoustic signal of an utterance and its verbal content. A new paper, arXiv:2610.40883v1, asks whether disagreement between those two channels degrades speech emotion recognition. The work evaluates, rather than tries to solve, cross-modal emotional incongruence, treating it as a factor separate from the noise in affective annotation.

The method maps text and audio into a shared valence-arousal-dominance space. Text-level VAD is drawn from the NRC VAD Lexicon, while audio-level VAD comes from a fine-tuned wav2vec2-large model. The Euclidean distance between the two vectors is converted into a continuous, threshold-free incongruence probability, p_incon, using a two-component Gaussian mixture. The score was validated against human judgement, with a Spearman correlation of 0.65, and compared with six simpler stratifiers; the paper says its advantage is consistency across corpora rather than strength in any single one.

The authors evaluated three audio-capable models on acted, conversational and spontaneous speech using eNTERFACE'05, MELD and CMU-MOSEI. With the model held constant across modalities, incongruence raised the odds of error whenever audio was present, in every corpus, after controlling for the ground-truth emotion and for distance in the feature space.

Supervised classifiers trained on the same VAD features behaved in the opposite way: incongruence, if anything, helped them. The paper says this pattern is consistent with the difficulty arising in how audio LLMs combine raw multimodal evidence, rather than in the VAD signal itself. The authors add a caveat that they cannot fully separate this from the supervised-versus-prompted contrast between the two model classes.

Text-only effects were unreliable. Swapping only the model that reads the transcript reversed the sign of the association on one corpus. Rebuilding the incongruence score without the emotion lexicon reproduced the central result, and among six simpler alternatives it was the only label-free score that predicted error in all three corpora. The paper also describes a paradigm dissociation on eNTERFACE, across three audio LLMs and twelve conditions.

The work releases an incongruence-aware benchmark, including per-sample VAD features, p_incon scores, validity-checked LLM predictions, prompts and analysis code.

Sources: arXiv cs.AI.