An overview of decision making

Chapter 1: Introduction

1. An overview of decision making

In everyday life, there are numerous decisions waiting for us, from what food to eat, what clothes to wear, what hair style and what you are going to do in the

future…etc. All of these things need us to make decisions. In short, a decision is a process that weighs priors, evidence, and values of different options to generate a choice intended to achieve particular goals. And this is the main focus of the field of decision making. Recently, a cross disciplinary approach to study decision making process has come out to the mainstream: Neuroeconomics.

Neuroeconomics is a newly established field that integrates the confluence of economics, psychology and neuroscience to the study of decision making to try and create a better model about decisions, interactions, and risks and rewards. Accordingly, neuroeconomics combines the modeling from economics with psychological studies of social and emotional influences on decision making, and utilizes tools from neuroscience that permit the observation of valuation and decision-making

computations that take place in the brain. In the following section, a brief introduction of decision process and its corresponding brain areas are described.

1.1. Elements of a decision. As mentioned, a decision is a process that weighs

2

priors, evidence, and values of different options to generate a choice intended to achieve particular goals. It also can be regarded as a form of statistical inference (Kersten, Mamassian, & Yuille, 2004; Smith, 1961). According to Doya, the process of value-based decision making can be decomposed into four steps (Doya, 2008):

a. Subject identifies the existing situation (or state).

b. Subject evaluates possible options (or actions) according to the reward or punishment every potential choice could bring.

c. Subject makes the final decision after considering own needs.

d. Based on the outcome, subject revaluates the decision.

Although decisions are not always made through these four steps, a standardizing procedure of decision making process is useful in the understanding of how these steps are executed in the brain.

1.2. Brain areas related to value functions. Subject’s internal reward

expectancy represents value functions in decision process. Theoretically, neural signals related to reward expectancy can be divided into two categories: action value and state value (Lee, Seo, & Jung, 2012). Action value functions are useful in

choosing a particular action, especially if such signals are observed before the

execution of a motor response. However, based on the dimension in which choices are

3

made, brain areas related to the corresponding action value functions may vary substantially. In most previous studies, many brain areas are implicated in action value functions, including dorsolateral prefrontal cortex (Barraclough, Conroy, & Lee, 2004; Kim, Hwang, & Lee, 2008), posterior parietal cortex (Dorris & Glimcher, 2004;

Platt & Glimcher, 1999; Sugrue, Corrado, & Newsome, 2004), medial frontal cortex (Seo & Lee, 2009; So & Stuphorn, 2010; Sul, Kim, Huh, Lee, & Jung, 2010),

premotor cortex (Pastor-Bernier & Cisek, 2011), and striatum (Cai, Kim, & Lee, 2011;

Kim, Sul, Huh, Lee, & Jung, 2009; Lau & Glimcher, 2008; Samejima, Ueda, Doya, &

Kimura, 2005; Tai, Lee, Benavidez, Bonci, & Wilbrecht, 2012).

State value functions play a more evaluative role in the brain, and it can be further divided into two categories: pre-decision and post-decision. For the

pre-decision state value functions, researchers found that some of the related brain areas overlapped with the action value functions. Neural activity in the posterior parietal cortex and dorsal striatum showed both characteristics of pre-decision state value functions and action value functions (Cai et al., 2011; Seo, Barraclough, & Lee, 2009; Yang & Shadlen, 2007). Brain areas related to pre-decision state value functions are also found in the ventral striatum (Cai et al., 2011), anterior cingulate cortex (Seo

& Lee, 2007), and amygdala (Belova, Paton, & Salzman, 2008).

4

Post-decision state value functions are also called chosen values, and its related brain areas are also widespread, including orbitofrontal cortex (Padoa-Schioppa &

Assad, 2006; Sul et al., 2010), medial frontal cortex (Sul et al., 2010), ventromedial prefrontal cortex (Hare, Camerer, & Rangel, 2009), dorsolateral prefrontal cortex (Hare et al., 2009), and striatum (Cai et al., 2011; Kim et al., 2009; Lau & Glimcher, 2008). Since the revaluation happens after subjects made their decision, the chosen value may be utilized to revaluate (i.e. compute the difference between the outcome of a choice and the chosen value) and update value functions.

1.3. Brain areas related to action selection. In decision making process, the

action value must be transformed into specific action and corresponding motor structures. Hence, the brain areas involved in action value functions are likely to be related in action selection. Also, brain areas involved in motor control are likely to be related in action selection (Lee, Seo, & Jung, 2012). However, the character of a behavioral task may change the precise anatomical location involved in action selection. For instance, a well-trained motor sequence (fixed stimulus-response association) may rely more on the dorsolateral striatum (Hikosaka et al., 1999; Yin &

Knowlton, 2004, 2006; Yin, 2010), whereas the dorsomedial striatum may be rely

5

more on to perform flexible goal-directed behaviors (Yin, Knowlton, & Balleine, 2005; Yin, Ostlund, Knowlton, & Balleine, 2005). Moreover, recent study using transient optogenetic stimulation of dorsal striatal dopamine D1 and D2

receptor–expressing neurons during decision-making found that the striatal activity is involved in goal-directed action selection (Tai et al., 2012). There are cumulated evidence showing that the lateral intraparietal cortex (LIP) (Roitman & Shadlen, 2002;

Rorie, Gao, McClelland, & Newsome, 2010; Seo et al., 2009), frontal eye field (Ding

& Gold, 2012), and superior colliculus (Horwitz & Newsome, 2001) are involved in selecting a specific physical movement.

In addition, other brain areas may be related to more abstract action selection (Lee, Seo, & Jung, 2012). Action selections like making choices among different objects or goods may rely more on the orbitofrontal cortex (Padoa-Schioppa & Assad, 2006; Padoa-Schioppa, 2011). Compared to the orbitofrontal cortex, the medial frontal cortex may be involved more in action selection guided by endogenous cues (for example, memory) rather than external sensory stimuli. The medial frontal cortex, including the anterior cingulate cortex (Kennerley, Walton, Behrens, Buckley, &

Rushworth, 2006; Lee, Rushworth, Walton, Watanabe, & Sakagami, 2007; Shidara &

Richmond, 2002) and supplementary motor area (Okano & Tanji, 1987; Sohn & Lee,

6

2007; Soon, Brass, Heinze, & Haynes, 2008; Sul, Jo, Lee, & Jung, 2011), may integrate the information about the costs and benefits of particular behaviors and take action. Furthermore, it has been proposed that the anterior cingulate cortex might play a more important role in selecting an action voluntarily and monitoring its outcomes (Kennerley et al., 2006; Quilodran, Rothé, & Procyk, 2008; Rushworth, Walton, Kennerley, & Bannerman, 2004).

1.4. Neural mechanisms for updating value functions. Value updating

functions can be divided into two parts. First, subjects need to relate an action to its corresponding outcome correctly. Deficit of this function could interfere with the process of updating value functions suitably. Previous studies showed that subjects

with lesions in the orbitofrontal cortex are impaired in reversal learning tasks (Fellows

& Farah, 2003; Murray, O’Doherty, & Schoenbaum, 2007; Schoenbaum, Nugent,

Saddoris, & Setlow, 2002), and the deficits produced by the lesions were due to animals' choice behavior no longer reflected the history of precise conjoint

relationships between particular choices and particular rewards (Walton, Behrens, Buckley, Rudebeck, & Rushworth, 2010). Thus, orbitofrontal cortex may be a critical brain area to associate an action and its corresponding outcome correctly.

Second, subjects need to realize the difference between expected reward and

7

actual reward (i.e. the reward prediction error signal) and use this information to

update the value functions. Signals related to reward prediction error were first identified in the midbrain dopamine neurons (Schultz, 1997). Recent studies found

that it also exists in many brain areas, including the lateral habenula (Matsumoto &

Hikosaka, 2007), globus pallidus (Hong & Hikosaka, 2008), dorsolateral prefrontal cortex (Asaad & Eskandar, 2011), anterior cingulate cortex (Seo & Lee, 2007), orbitofrontal cortex (Sul et al., 2010), and striatum (Asaad & Eskandar, 2011; Kim et al., 2009; Oyama, Hernádi, Iijima, & Tsutsui, 2010). Thus, dopamine neurons may play an important role in relaying these error signals to update the value functions represented broadly in different brain areas. Brain areas related to chosen value are also widespread, including orbitofrontal cortex (Padoa-Schioppa & Assad, 2006; Sul et al., 2010), medial frontal cortex (Sul et al., 2010), ventromedial prefrontal cortex (Hare et al., 2009), dorsolateral prefrontal cortex (Hare et al., 2009), and striatum (Cai et al., 2011; Kim et al., 2009; Lau & Glimcher, 2008). Thus, brain areas related to the chosen value and reward prediction error overlapped, such as the orbitofrontal cortex, dorsolateral prefrontal cortex and striatum. These brain areas may therefore play an important role in updating the value functions.

8