2. Experimental Design
2.1 Design Details
In the first phase, subjects are independently assigned to either urn A or B with equal chance. Both urns contain one hundred digital balls, each ball is labeled by a number from 1 to 100. For each subject, the computer draws with replacement two balls randomly from the assigned urn. However, only one ball will be revealed to the subject. The ball number revealed to the subject is determined by one of two rules:
The computer either reveals the larger one (Maximum Rule) or the smaller one (Minimum Rule). In each round, urn A and urn B are independently randomized to either follow the Maximum Rule or the Minimum Rule with equal chance. Subjects are not told the realization of the states (which rules the urns follow), but those
1Alternative experimental designs that were considered, but not implemented are listed in AppendixC.
assigned to the same urn experience the same rule. After observing one ball, subjects have to predict the probability that the Maximum Rule is applied to urn A. Similarly, they also have to predict the probability that Maximum Rule is applied to urn B.
In the second phase, for each subject, the computer randomly chooses another subject, and reports his/her prediction of the urn they are assigned to. However, subjects do not know if this other subject was assigned to “Urn A” or “Urn B.” In other words, each subject observes a number, which is the prediction of another sub-ject in the first phase. After seeing the information from another subsub-ject, subsub-jects again predict the probability that the Maximum Rule is applied to each urn.
2.2 Belief Elicitation
We use a two-stage menu of lottery choices as the belief elicitation mechanism in the experiment. Essentially, it is the Becker-DeGroot-Marschak (BDM) pricing procedure but easier for subjects to understand. In the first stage, subjects choose from a list of lottery pairs, which are choices between a random lottery and an event lottery. The random lotteries have winning probabilities ranging from 0%, 10%, ..., to 100%. Thus, subjects compare the probability of each random lottery with their beliefs that the event would occur. We allow only one “switching point”
when completing the list of lottery pair. Based on the “switching point”, subjects decide a second digit of probability in the second stage. After the decision is done, the computer randomly draws one number from 0 to 100. The lottery is chosen according to the drawn number, and the payo↵ is determined by the corresponded lottery.
This method is incentive compatible. Without loss of generality, suppose one under-reports her beliefs from her real belief, 80%, to misreported belief, 60%. The results is the same when the drawn number is less than 60 and greater than 80.
However, it is disadvantage for her if the number falls into the interval between 60 and 80. Since the event lottery will be chosen if she truthfully report the belief, and in that case the probability of getting the prize is 80%. Conversely, in the case of misreporting, the random lottery will be chosen and its probability of getting the prize is between 60% and 80%. Therefore, truthfully report the belief is the best interest for subjects.
Holt and Smith (2016) compared three mechanisms of belief elicitation and dis-cussed the advantage of the two-stage menu of lottery choices. They find beliefs elicited from the two-stage menu to be more accurate and with lower average belief error in terms of Bayesian prediction.
2.3 Experimental Procedures
All sessions were conducted at Taiwan Social Sciences Experimental Laboratory (TASSEL), National Taiwan University (NTU). Six sessions were run during October 2019 and November 2019, for a total of 123 subjects. We recruited NTU students subjects using the TASSEL website powered by ORSEE (Greiner, 2015). Each session lasted approximately 100 minutes, and average earnings were 512 NT dollars (approx. $17). The experiment was programmed with z-Tree (Fischbacher, 2007) and conducted in Chinese. The experimental interfaces are shown in Figure 1a for the first stage and Figure1b for the second stage of elicitation processes.
Figure 1: Two-stage Menu of Lottery Choices: (a) 1st Stage, and (b) 2nd stage.
2.4 Bayesian Probability Predictions
For notation simplicity, we let urn A be the assigned urn and urn B be the irrelevant urn. We use ✓max and ✓min to denote the Maximum Rule and Minimum Rule of the assigned urn; the other urn also has two states, Maximum Rule and Minimum Rule, indicated by !maxand !min. The information s1 denotes the observed ball in the first phase, s2 is elicited probability of the assigned urn from another subject observed in the second phase.
2.4.1 The Structure of Two States
To calculate the Bayesian probability, we consider the structure of two possible states in advance. Consider the probability Pr(s1|✓max) of seeing s1 under Maximum Rule in the assigned urn. For two randomly drawn balls S11 and S12, there are two mutually exclusive events: Either the first drawn ball S11 is the observed ball and therefore the second drawn ball is smaller than the observed ball, or exactly the opposite, that is, the second drawn ball S12 is the observed ball and the first drawn ball is equal to or smaller than the observed ball. Therefore, the probability Pr(s1|✓max) is:
Pr(s1|✓max) = Pr S11 = s1\ S12 < s1 _ S11 s1 \ S12 = s1
Similarly, the other probability is Pr(s1|✓min) = (201 2s1)/10000. Therefore, the probability distribution of observing the ball S1 is linear under both the Maximum
Rule (increasing by 0.02% from 0.01% when observing 1 to 1.99% when observing 100) and Minimum Rule (decreasing by 0.02% from 1.99% when observing 1 to 0.01% when observing 100).
2.4.2 Phase 1
In the first phase, the processed information is the observed ball, which is only useful to infer the state of urn A. With the observed ball, the Bayesian probability prediction for urn A is as follows.
Pr(✓max|s1) = Pr(s1|✓max) Pr(✓max)
The Bayesian probability prediction for urn A shows that subjects should exactly predict at the percentage of their observed balls if they update the information by Bayesian Theorem. For example, suppose the observed ball s1 is 30, the Bayesian probability is Pr(✓max|s1 = 30) = 10030 2001 = 29.5%.
The intuition of this prediction is simple. Given the observed ball, the probability of Maximum Rule for urn A is only depends on the other “unobserved ball”. Thus, it is equivalent to the probability that the unobserved ball is smaller or equal to the observed ball, results in the term s1/100. The subtraction of 1/200 (0.5%) is representing the tie case, in which both drawn balls are the same as the observed ball; therefore, it could also be the state of Minimum Rule. Thus, the 1% is split to both cases. The Bayesian probability prediction for urn B is straightforward since there is no information about urn B. As a results, Pr(!max|s1) should be 0.5.
2.4.3 Phase 2
In the second phase, subjects see another ball s2, which is either from urn A or urn B. Because the actual source is unknown, it is useful to make inferences of both urns. Without loss of generality, we assume urn A is the assigned urn and urn B is the irrelevant urn, and their Bayesian probabilities in the second phase are:
Pr(✓max|s1, s2) = Pr(s1 \ s2|✓max)· Pr(✓max)
Equation 5 indicates the weightings that s2 is under Maximum Rule or Minimum Rule. Since it is given the state of A is Maximum Rule, ✓max, only the state of B remains uncertain. By the settings of experimental design, there is equal chance that s2 is either from urn A or urn B. It is the only possibility that s2 is drawn under Minimum Rule when s2 is from urn B and urn B is applied to Minimum Rule.
Therefore, s2 is drawn under Maximum Rule with 75% chance and Minimum Rule with 25% chance. With similar reason, we can also derive the probability in equation 6. The combination of probabilities (pA, pI) is the weights of the information source, indicating that the probability that new information is from the assigned urn or irrelevant urn. It is (0.5, 0.5) since the randomly drawn subject has equal chance to be assigned to urn A or B.
Equation7 also shows the weightings that s2 is under Maximum Rule or Minimum
Rule but given the state of urn B, !max, instead of the state of urn A, ✓max. We can divide the equation into two parts, the state of urn A is either Maximum Rule or Minimum Rule. First of all, when the state of urn A is Maximum Rule, with the probability derived in equation 3, it is for sure that s2 is drawn under Maximum Rule. Secondly, when the state of urn A is Minimum Rule, there is equal chance to draw s2 under Maximum Rule or Minimum Rule. Thus, the probability of observing s2given states of u two urns !maxand ✓minis the same as the probability of observing s2, 1%. Equation 8is derived by the same thoughts.
Hence, the Bayesian probability prediction for urn A (the assigned urn) is:
Pr(✓max|s1, s2)
= [3(2s2 1) + (201 2s2)] (2s1 1)
[3(2s2 1) + (201 2s2)] (2s1 1) + [(2s2 1) + 3(201 2s2)] (201 2s1) (9)
By substituting equation7and 8into4, the Bayesian probability prediction for urn B (the irrelevant urn) is as follows.
Pr(!max|s1, s2)
= (2s2 1)(2s1 1) + 100· (201 2s1)
(2s2 1)(2s1 1) + 100· (201 2s1) + 100· (2s1 1) + (201 2s2)(201 2s1) (10)
Alternatively, we can derive probabilities, Pr(s2|s1, ✓max) and Pr(s2|s1, !max) by the source of other’s information. It is beneficial for analyzing how subjects consider other’s information. Equation 11 and 12 show above concept. Exploiting the first
ball and consequent beliefs, subjects form probabilities that second ball comes from urn A and B. Depends on two balls, subjects may distort pA and pI, both of which are 0.5 and pA is equal to (1 pI) in theory.
In equation 5 and 7, it is assumed that subjects update posteriors of two urns together. In other words, they rationally assign probabilities pA and pI so that the sum of pA and pI is always equal to 1. Thus, if the information is considered very unlikely being drawn from urn A, subject should put higher weight on urn B.
Unfortunately, subjects may not be able to allocate probabilities pAand pI properly.
For example, even if they believe the information has 10% chance coming from their assigned urn, they might only assign 60% to the irrelevant urn. One possible and intuitive updating process is that they separately update two urns. Specifically, when they deem the information not from one urn, they do not attribute it to the other urn. In fact, it is useless to subjects when updating the belief. In this situation, it seems that the information is drawn from an urn in which each ball is drawn with equal probability. In other words, when subjects regard the information is from the ”useless urn”, it provide no further clue for updating. To derive the theoretical prediction, the di↵erences are caused by Pr(s2|s1, ✓max), Pr(s2|s1, ✓min),
Pr(s2|s1, !max), and Pr(s2|s1, !min). Therefore, the theoretical results are as follows.e
Pr(✓max|s1, s2)
= [(2s2 1)pA+ 100(1 pA)](2s1 1)
[(2s2 1)pA+ 100(1 pA)](2s1 1) + [(201 2s2)pA+ 100(1 pA)](201 2s1) (13) Pr(!max|s1, s2)
= [(2s2 1)pI + 100(1 pI)](2s1 1)
[(2s2 1)pI + 100(1 pI)](2s1 1) + [(201 2s2)pI + 100(1 pI)](201 2s1) (14)
3 Results
3.1 Adherence to Bayesian Updating
3.1.1 Compliance After Initial Draw
Figure 2a presents elicited probabilities of the assigned urn after drawing a ball in the first phase. Each data point represents the reported belief of a subject in a particular round. The majority of data are very close to the correct Bayesian posteriors, with nearly 90 percent of the data aligned with the theory if we allow for an errors margin of plus and minus 10 percentage points (±10%).2 The elicited probabilities of the irrelevant urn, in which they do not have any information, are shown on Figure2b, in which over 80% of the elicited probabilities are between 0.4 and 0.6 (50%± 10%). Table 1 shows that a majority of choices conform with the theoretical predictions as we reduce the margin of error allowed. Even under the strictest case allowing for only 1 percentage point error (±1%), 60% and 55% of the choices are considered Bayesian in the assigned and the irrelevant urn, respectively.
The squares in Figure2represent the mean elicited probabilities averaged across all subjects with the same initial draw. They closely adhere to the Bayesian pos-teriors, especially for the assigned urn. Notice that there is a cluster of elicited probabilities along the 45-degree line in Figure 2b, implying that some subjects also use the initial draw to update the irrelevant urn. We find that those choices come from one-time behavior of di↵erent subjects and not concentrated in
partic-2Alternatively, one could construct the upper and lower bounds relative to the initial draw. For example, allowing for a 10 percent error results in 50%± 5% for the ball 50, but 10% ± 1% for the ball 10. This criteria is harsh to those who draw a very small or large ball since they have stronger information. However, under it 76% of the data are still considered to be aligned with theory.
ular rounds, indicating that they are not caused by particular subjects or rounds.3 Although these choices consists of only 3% of the data, they inflate the correlation between the elicited probabilities of the assigned and irrelevant urn.4 Without these choices, the correlation is 0.003 (p > 0.1), indicating that the vast majority of proba-bilities are elicited with the knowledge that states of the two urns are independent.5 In conclusion, most of the choices are consistent with Bayesian updating derived in section2.4.2.
Figure 2: Elicited Beliefs in the First Phase of the (a) Assigned (b) Irrelevant Urn
Table 1: Percentage of Theory-consistent Choices Under Di↵erent Error Margins
Error Margin Assigned Urn Irrelevant Urn
±10 percentage points 89% 81%
±5 percentage points 81% 74%
±3 percentage points 75% 60%
±1 percentage points 66% 55%
3See Appendix Afor further details.
4A total of 37 choices lie exactly on the 45-degree line excluding initial draws between 40 and 60 where we cannot easily tell if they updated beliefs of the irrelevant urn or not.
5Similarly, the second phase correlation between the two urns is 0.006 (p > 0.1). Computing with all data, the first and second phase correlations are 0.067 and 0.029, respectively.
3.1.2 Failure After Observing New Information
There exists one intuitive di↵erence between the two possible states of the urn: When the true state is the Maximum Rule, the subject is more likely to observe a ball larger than 50, while under the Minimum Rule, the subject is more likely to observe a ball equal to or smaller than 50. This leads to a straightforward heuristic for subjects to determine whether new information in the second phase is more likely to come from an urn under the Maximum Rule or Minimum Rule. As a result, we classify the second-phase information coming from another subject, as either confirming or conflicting information. In particular, the new information is confirming if first and second phase information are both within 1–50 or both within 51–100, while it is conflicting when one is within 1–50 and the other one is within 51-100.6
Compared to the first phase, belief-updating in the second phase is much worse.7 Figure3summarizes the distribution of Bayesian posteriors and the average devia-tion from them on di↵erent intervals. When the new informadevia-tion is confirming, we find that subjects deviate less in the assigned urn, but deviate more in the irrelevant urn. This suggests that it is easier to correctly process new information regarding the assigned urn that aligns with what subjects already have. In contrast, updating behavior for the irrelevant urn is far from the Bayesian prediction as the overall deviations are larger than the assigned urn (Figure3b).
Furthermore, the R-squared predicting elicited probabilities using Bayesian pos-teriors shows that subjects perform updating well in the assigned urn when the
6Some information may be too close to 50 to be “confirming” or “conflicting” enough, such as initial draws or new information between 40 and 60. Excluding these cases, we expect to find stronger e↵ects.
7See Figure10of AppendixBfor the raw data plotted like Figure 2.
Figure 3: Elicited Beliefs Distribution in the Second Phase of (a) the Assigned, and (b) Irrelevant Urn
information is confirming (R2 = 0.82), but perform worse when it is conflicting (R2 = 0.51). In contrast, for the irrelevant urn, subjects perform worse when the new information is confirming (R2 = 0.33), but perform better when it is conflicting (R2 = 0.52). The di↵erences in R2 are statistically significant for both urns (vari-ance ratio test, p < 0.001). The results in AppendixBshow that the slopes between confirming and conflicting information are not significantly di↵erent in Figure 10a (p = 0.175) and Figure10b (p = 0.434).8
3.2 The Echo Chamber
In principle, subjects should update their beliefs of both urns regardless of the information received in the second phase because there is always a chance the new information could be from either urn. However, the irrelevant urn has the natural
8We test the coefficient 3 from the model: Beliefs 0 + 1Bayesian + 2Confirming +
3Interaction + ✏, where the dummy variable Confirming indicates the new information is con-firming (=1) or not (=0), Interaction is the interaction term of Bayesian and Concon-firming.
advantage that one should only update it according to the new information regarding the ball of the second phase, since the first ball only carries information about the assigned urn. Therefore, we can easily infer how subjects attribute new information to each urn in the second phase from their updating behavior.
Figure4 plots elicited probabilities against second-phase information.9 The red dots are elicited beliefs around 0.5, adhering to the Bayesian prediction of the first phase, indicating “fully dissociate” subjects who do not update irrelevant urn beliefs at all (and should completely attribute the new information to the assigned urn). On the other hand, the blue crosses along the 45-degree line indicate “fully attribute”
types who completely ignore the fact that there is some probability that the new information is from their assigned urn.10 These two types are strongly biased since they put extreme weight on the new information when updating the irrelevant urn.
However, they account for 76.7% of the choices when we allow 5 percentage points of error. The intermediate types with more reasonable weights are shown as green triangles in Figure4, but consist only 18.7% of the choices. This includes those who follow Bayesian updating. Lastly, the remaining 4.6% of choices in black are difficult to rationalize, and might reflect confusion or some other information processing method. We summarize the updating behavior in the Table2.
In Figure 5, we separate second-phase information into confirming and conflict-ing information as defined in section 3.1.2. To compare the di↵erence in behavior between receiving confirming and conflicting information, we use a dummy
indi-9We drop the choices if their first phase beliefs of the irrelevant urn are out of the range, [0.45, 0.55]. The remaining choices plotted in the Figure4contain 74% of the data.
10The purple dot-cross symbols are overlapping area of the two types, in which we cannot distinguish their types.
Figure 4: Types of Behavior (Irrelevant Urn)
Table 2: Types of Behavior (Irrelevant Urn)
Types of Choices Definition Percentage
Either Either fully dissociate or fully attribute type. 16.3 % Fully Dissociate Other subject’s information comes from the assigned urn. 25.4 % Fully Attribute Other subject’s information comes from the irrelevant urn. 35 % Intermediate Put reasonable weights on other subject’s information 18.7 % Others Choices cannot be classified into above four types. 4.6 %
cating confirming information to predict the occurrence of two distinct types of behavior, completely attribute the information to the assigned urn (Fully Disso-ciate) and the irrelevant urn (Fully Attribute). Table 3 report fixed-e↵ect panel regression results clustered at the subject level, predicting whether the inferred prior belief fully attributes the new information to the irrelevant urn using whether information is confirming or not. For confirming information, 33.7% of the choices completely attribute the new information to the assigned urn, while 31.1% of the choices completely attribute the new information to the irrelevant urn. However, when subjects receive conflicting information, only 16.5% of the choices attribute new information to the assigned urn, significantly lower than that under confirming information. Moreover, 39% of the choices completely attribute new information to the irrelevant urn, significantly higher than that under confirming information. This results demonstrates a confirmation bias where subjects overweight (underweight)
cating confirming information to predict the occurrence of two distinct types of behavior, completely attribute the information to the assigned urn (Fully Disso-ciate) and the irrelevant urn (Fully Attribute). Table 3 report fixed-e↵ect panel regression results clustered at the subject level, predicting whether the inferred prior belief fully attributes the new information to the irrelevant urn using whether information is confirming or not. For confirming information, 33.7% of the choices completely attribute the new information to the assigned urn, while 31.1% of the choices completely attribute the new information to the irrelevant urn. However, when subjects receive conflicting information, only 16.5% of the choices attribute new information to the assigned urn, significantly lower than that under confirming information. Moreover, 39% of the choices completely attribute new information to the irrelevant urn, significantly higher than that under confirming information. This results demonstrates a confirmation bias where subjects overweight (underweight)