Computational and Neural Mechanisms of Human Exploration-Exploitation Behavior

Abstract

For both living organisms and artificial agents, exploration-exploitation decisions are ubiquitous and vital. They (co-)determine human behavior in all areas of life, from the smallest everyday decisions like grocery shopping to life-changing choices such as choice of a partner or a career. Understanding how exploration-exploitation decisions are made is therefore crucial to understanding – and potentially being able to influence – how and why humans behave the way they do. This dissertation contributes to exploration-exploitation research by examining behavioral, computational, neural, and physiological mechanisms behind exploration-exploitation decision-making. After a brief introduction in which I outline open questions and how this dissertation addresses them (Chapter 1), I proceed to investigate behavioral and computational signatures of exploring and exploiting (Chapter 2). To this end, I use a newly designed task which captures naturally paced exploration-exploitation behavior, while allowing participants to directly indicate whether they explore or exploit in a model-independent way. Having tested participants both in the lab and online, I demonstrate that this task reliably captures key behavioral characteristics of exploration-exploitation behavior and is well suited to its further use in combination with neurophysiological methods. Using computational modeling, I further probe the underlying decision-making processes and their relationship to behavior. The best-fitting computational model highlights different roles that reward and uncertainty estimates play in exploration and exploitation, as well as differences in how quickly acquired information about reward and uncertainty becomes obsolete. I then use the task and computational model presented in Chapter 2 to investigate neural (Chapter 3) and eye tracking (Chapter 4) mechanisms behind exploration-exploitation decision-making. In Chapter 3, I show that uncertainty-driven BOLD signal variability could function as a neural mechanism that allows to adapt exploration-exploitation behavior to a rapidly changing environment. In Chapter 4, I demonstrate that gaze patterns during the decision-making period predict the trial type (exploration or exploitation) and that patterns with different number of dwell locations provide complementary insights into the decision-making behind exploration-exploitation choices. Lastly, Chapter 5 provides a summary of contributions this dissertation makes to the field of exploration-exploitation research, discusses limitations and presents suggestions for future studies. All in all, this dissertation presents a comprehensive investigation of the underlying mechanisms of human exploration-exploitation behavior.