Artificial agents in operations management experiments

Abstract

Built on large language models (LLMs), silicon samples offer a promising approach for advancing behavioral research by simulating decision-making processes and replicating human decision-making in controlled experiments. This study evaluates LLMs’ potential to replicate behavioral hypotheses in operations management (OM) by conducting nine studies from the Management Science Replication Project. I develop an accessible framework where GPT-4o agents are assigned roles based on System 1 decision-making and participate in the experiments. Remarkably, the silicon samples replicated the core hypotheses in eight of the nine studies, with treatment effects closely matching the direction and magnitude of human decisions. The natural language explanations generated by GPT agents offer insight into their decision logic, revealing whether observed choices align with the behavioral theories being tested. These findings position silicon samples as a valuable tool for replication research, helping researchers distinguish between sampling limitations and theoretical divergence. More broadly, the results support using silicon samples to scale replication, design interventions, and conduct high-fidelity pretests before lab or field deployment.

Short bio

Sam Kirshner is an Associate Professor and Head of School of Information Systems and Technology Management at the UNSW Business School. Sam completed his PhD in Management Science at Queen’s University in Kingston, Canada. His primary research interests lie in analysing behavioural decision-making in operations and technology management and studying how algorithms and artificial intelligence impact and and make decisions. His research and commentaries have been published in prestigious academic journals, including Management Science, Manufacturing & Service Operations Management, Production and Operations Management, Decision Analysis, Decision Sciences, Service Science, and Science. Sam is a Regional Editor for the Journal of Supply Chain Management and an Associate Editor at Decision Sciences and Service Science.