New Audit Reveals Flaws in Shapley Value Benchmarks for Explainable AI

A recent study evaluates existing benchmarks for Shapley values, a key component in explainable AI (XAI). Researchers conducted a comprehensive audit of eight Shapley variants, examining their efficacy in high-stakes operational risk workflows. This work serves to isolate semantic differences among these variants while assessing their performance in a realistic fraud detection environment. Utilizing a large-scale empirical evaluation with professional analysts, the study reviewed 3,735 cases to determine how well these benchmarks align with human decision-making.

The findings reveal a troubling gap: traditional evaluation metrics, such as sparsity and faithfulness, do not reflect how human analysts perceive clarity and decision-making utility. Notably, while no specific Shapley formulation enhanced objective performance for analysts, the presence of explanations led to increased decision confidence. This raises concerns about potential automation bias in critical contexts, where human oversight is paramount.

This study underscores the inadequacy of current XAI evaluation proxies, which may misrepresent their effectiveness when applied in real-world scenarios. Stakeholders in AI development-particularly those focused on ensuring transparency and accountability in decision-making-must grapple with these findings. The disconnect between theoretical metrics and operational realities could hinder the broader adoption of XAI technologies, especially in sensitive high-stakes environments like finance and healthcare.

Looking ahead, the research suggests a reevaluation of the metrics and formulations used for assessing Shapley values. Developers and researchers should prioritize methods that promote better alignment with human-centric decision-making criteria. This shift could influence policies and product strategies in the AI industry, emphasizing the need for a more nuanced understanding of how AI explanations impact human users.