A simulation-based approach for measuring the performance of human-AI collaborative decisions

Despite the widespread adoption of artificial intelligence and machine learning for decision-making in organizations, a wealth of research shows that situations that involve open-ended decisions (novel contexts without predefined rules) will continue to require humans in the loop. However, such collaboration that blends formal machine and bounded human rationality also amplify the risk of what is known as local rationality, which is when rational decisions in a local setting lead to globally dysfunctional behavior. Therefore, it becomes crucial, especially in a data-abundant environment that characterizes algorithmic decision-making, to devise means to assess performance holistically, not just for decision fragments. There is currently a lack of quantitative models that address this issue