Study: AI Narrative Explanations Boost User Trust, Not Accuracy

A pre-registered large-scale behavioral study from DEVCOM Army Research Laboratory and the University of Texas at Dallas finds that LLM-generated narrative explanations paired with AI predictions do not improve human decision accuracy over predictions alone. More persuasive narratives actively degrade users' ability to distinguish correct from incorrect model outputs.

The study tested three conditions in a classification task: an AI prediction alone, a prediction paired with lower-persuasiveness narrative explanation, and a prediction paired with higher-persuasiveness narrative. The explanation framing draws from the Explingo architecture—a narrator LLM generates narratives explaining SHAP feature-importance outputs while a grader LLM scores each explanation on accuracy, completeness, fluency, and conciseness. Prior work (XAIStories by Martens et al., 2025) found that users judged LLM narratives more convincing than raw SHAP outputs in 93% of cases for SHAP and 90% for counterfactual explanations, confirming subjective appeal even when objective impact remains unclear.

On decision accuracy: narrative conditions produced no improvement over the prediction-alone baseline. This aligns with broader explainable AI research where feature-importance outputs—LIME, SHAP, attention maps—consistently fail to improve classification accuracy in human-in-the-loop settings. Narratives shift user confidence, not evaluation capability.

The critical finding emerged in reliance metrics. Narrative explanations increased human reliance on AI predictions regardless of whether the underlying prediction was correct or incorrect. A user receiving a persuasive narrative backing a wrong model output was more likely to follow that output than a user who saw only the raw prediction. The more persuasive the narrative, the lower the discrimination between correct and incorrect calls. Exploratory analyses showed the high-persuasiveness condition also degraded decision response times without accuracy benefit.

The paper does not disclose latency, throughput, or cost figures—this is behavioral research, not a deployment study. The specific LLM used for experimental narratives is not named in publicly accessible sections. Participant count is not surfaced in the abstract or introduction. Practitioners citing this work should confirm these details before internal review.

The integration risk is direct: auto-generated narrative explanations create a ratchet effect. As narrative quality improves, users trust wrong predictions more readily. Teams running high-stakes human-in-the-loop workflows—fraud review, medical triage, content moderation escalation—should treat narrative explanations as a system liability until they measure explanation impact on decision accuracy in their own deployment, not relying on user satisfaction scores alone.