Adherence to Bayesian Updating

3.1.1 Compliance After Initial Draw

Figure 2a presents elicited probabilities of the assigned urn after drawing a ball in the first phase. Each data point represents the reported belief of a subject in a particular round. The majority of data are very close to the correct Bayesian posteriors, with nearly 90 percent of the data aligned with the theory if we allow for an errors margin of plus and minus 10 percentage points (±10%).² The elicited probabilities of the irrelevant urn, in which they do not have any information, are shown on Figure2b, in which over 80% of the elicited probabilities are between 0.4 and 0.6 (50%± 10%). Table 1 shows that a majority of choices conform with the theoretical predictions as we reduce the margin of error allowed. Even under the strictest case allowing for only 1 percentage point error (±1%), 60% and 55% of the choices are considered Bayesian in the assigned and the irrelevant urn, respectively.

The squares in Figure2represent the mean elicited probabilities averaged across all subjects with the same initial draw. They closely adhere to the Bayesian pos-teriors, especially for the assigned urn. Notice that there is a cluster of elicited probabilities along the 45-degree line in Figure 2b, implying that some subjects also use the initial draw to update the irrelevant urn. We find that those choices come from one-time behavior of di↵erent subjects and not concentrated in

partic-2Alternatively, one could construct the upper and lower bounds relative to the initial draw. For example, allowing for a 10 percent error results in 50%± 5% for the ball 50, but 10% ± 1% for the ball 10. This criteria is harsh to those who draw a very small or large ball since they have stronger information. However, under it 76% of the data are still considered to be aligned with theory.

ular rounds, indicating that they are not caused by particular subjects or rounds.³ Although these choices consists of only 3% of the data, they inflate the correlation between the elicited probabilities of the assigned and irrelevant urn.⁴ Without these choices, the correlation is 0.003 (p > 0.1), indicating that the vast majority of proba-bilities are elicited with the knowledge that states of the two urns are independent.⁵ In conclusion, most of the choices are consistent with Bayesian updating derived in section2.4.2.

Figure 2: Elicited Beliefs in the First Phase of the (a) Assigned (b) Irrelevant Urn

Table 1: Percentage of Theory-consistent Choices Under Di↵erent Error Margins

Error Margin Assigned Urn Irrelevant Urn

±10 percentage points 89% 81%

±5 percentage points 81% 74%

±3 percentage points 75% 60%

±1 percentage points 66% 55%

3See Appendix Afor further details.

4A total of 37 choices lie exactly on the 45-degree line excluding initial draws between 40 and 60 where we cannot easily tell if they updated beliefs of the irrelevant urn or not.

5Similarly, the second phase correlation between the two urns is 0.006 (p > 0.1). Computing with all data, the first and second phase correlations are 0.067 and 0.029, respectively.

3.1.2 Failure After Observing New Information

There exists one intuitive di↵erence between the two possible states of the urn: When the true state is the Maximum Rule, the subject is more likely to observe a ball larger than 50, while under the Minimum Rule, the subject is more likely to observe a ball equal to or smaller than 50. This leads to a straightforward heuristic for subjects to determine whether new information in the second phase is more likely to come from an urn under the Maximum Rule or Minimum Rule. As a result, we classify the second-phase information coming from another subject, as either confirming or conflicting information. In particular, the new information is confirming if first and second phase information are both within 1–50 or both within 51–100, while it is conflicting when one is within 1–50 and the other one is within 51-100.⁶

Compared to the first phase, belief-updating in the second phase is much worse.⁷ Figure3summarizes the distribution of Bayesian posteriors and the average devia-tion from them on di↵erent intervals. When the new informadevia-tion is confirming, we find that subjects deviate less in the assigned urn, but deviate more in the irrelevant urn. This suggests that it is easier to correctly process new information regarding the assigned urn that aligns with what subjects already have. In contrast, updating behavior for the irrelevant urn is far from the Bayesian prediction as the overall deviations are larger than the assigned urn (Figure3b).

Furthermore, the R-squared predicting elicited probabilities using Bayesian pos-teriors shows that subjects perform updating well in the assigned urn when the

6Some information may be too close to 50 to be “confirming” or “conflicting” enough, such as initial draws or new information between 40 and 60. Excluding these cases, we expect to find stronger e↵ects.

7See Figure10of AppendixBfor the raw data plotted like Figure 2.

Figure 3: Elicited Beliefs Distribution in the Second Phase of (a) the Assigned, and (b) Irrelevant Urn

information is confirming (R² = 0.82), but perform worse when it is conflicting (R² = 0.51). In contrast, for the irrelevant urn, subjects perform worse when the new information is confirming (R² = 0.33), but perform better when it is conflicting (R² = 0.52). The di↵erences in R² are statistically significant for both urns (vari-ance ratio test, p < 0.001). The results in AppendixBshow that the slopes between confirming and conflicting information are not significantly di↵erent in Figure 10a (p = 0.175) and Figure10b (p = 0.434).⁸