'The AI is uncertain, so am I. What now?': Navigating Shortcomings of Uncertainty Representations in Human-AI Collaboration with Capability-focused Guidance

Abstract

As AI becomes increasingly relevant, especially in high-stakes domains such as healthcare, it is important to investigate which approaches can improve human-AI collaboration and, if so, why. Current research focuses primarily on technically available approaches, such as explainable AI (XAI), often overlooking human needs. This study bridges this gap by adopting a well-established technical approach - model uncertainty representations - by considering users' familiarity with the format and numeracy skills. Despite being provided with uncertainty representations, users may still struggle to handle uncertain decisions. Thus, we introduce an educational approach that communicates the capabilities of humans and the AI system to users, supplementing the uncertainty representations. We conducted a pre-registered, between-subjects user study to determine whether these approaches resulted in improved human-AI team performance, mediated by the user's mental model of the AI. Our findings indicate that solely providing uncertainty representations does not improve team performance or the user's mental model in comparison to only providing AI recommendations. However, incorporating capability-focused guidance alongside uncertainty representations significantly enhances correct self-reliance and, to some extent, overall team performance. Our additional exploratory analyses suggest that factors such as task uncertainty, case difficulty, and case type, rather than numeracy skills, the need for cognition or familiarity, can influence team performance. We discuss these factors in detail, provide practical implications, and suggest directions for further research. This work contributes to the CSCW discourse by demonstrating how technical approaches can be augmented with educational approaches to enhance human-AI collaboration in decision-making tasks.