ELICITING RISK PREFERENCES USING CHOICE LISTS
DAVID FREEMAN, YORAM HALEVY AND TERRI KNEELAND
Abstract. We experimentally study the eect of embedding pairwise choices be-
tween lotteries within a choice list on measured risk attitude. Using an experiment
with online workers, we nd that subjects choose the risky lottery rather than a sure
payment signicantly more often when responding to a choice list. This failure of
incentive compatibility can be rationalized by the interaction between non-expected
utility and the random incentive system, as suggested by Karni and Safra (1987).
Keywords: random incentive system, isolation, independence axiom, multiple price list, reduction of
compound lotteries, preference reversals.
1. Introduction
A preference relation is, by denition, a binary relation over alternatives. As such,
the gold standard for revealing preferences through choice is the observation of a
single pairwise choice. However, such an experiment provides very limited information
about preferences for example, it cannot reveal a subject's certainty equivalent
of a lottery.
To elicit ner information about preferences, a common experimental
practice presents to a subject a sequence of related pairwise choices arranged in a
list, known as a Choice List (or Multiple Price List).
A randomization device is
used to pick one decision to determine the subject's payment, a procedure known
as the Random Incentive System (RIS). In recent years, choice lists have become
the workhorse method in experimental economics to measure individual preferences.
Usually, each pairwise choice a subject makes in a list is interpreted as if she had faced
only a single binary choice (Bardsley, Cubitt, and Loomes, 2009, pages 272-273).
This paper investigates whether subjects' choices are inuenced by whether or not
they are embedded in a choice list.
In one group of treatments, subjects respond
Date : January 26, 2017 (rst version May 2012).
We are grateful for comments from audiences at CEA 2012 in Calgary, SITE Experimental Economics session 2012, ESA North American Winter Meetings 2012 and 2013 (special panel discussion on
incentives in experiments), FUR 2014 in Rotterdam, Decisions: Theory, Experiments and Applications (D-TEA) 2014 at HEC Paris, SEA 2014 in Atlanta, CEA 2015 in Toronto, and M-BEES 2015.
The research reported in this paper was conducted under UBC Behavioural Reserch Ethics Board
certicate of approval H11-01719.
Financial support from SSHRC through grants 410-2009-1062
and 435-2012-0577 is gratefully acknowledged.
1
ELICITING RISK PREFERENCES USING CHOICE LISTS
2
to one or two choice lists. In a second group of treatments, subjects make a single
(or two) pairwise choice(s); each such choice corresponds exactly to a single line
of the corresponding choice list. The main experiment employs online workers from
Amazon's Mechanical Turk (mTurk), and we replicate our main ndings on a standard
student population.
We nd that embedding a pairwise choice in a choice list increases the fraction of
subjects choosing the riskier lottery from 23% to 45% when the safer alternative is
certain (p
< .001), but does not signicantly aect choices when the safer alternative
(p = .17). This suggests that embedding choices in a list can aect subjects'
is risky
responses, especially when comparing sure outcomes to lotteries.
These ndings are consistent with Karni and Safra's (1987) impossibility result
concerning the experimental observability of non-expected utility preferences. They
show that in an experiment that uses the RIS to select a decision to be paid, if the
subject reduces compound lotteries, the experiment may fail to be incentive compatible if the independence axiom does not hold. We nd strong experimental evidence
to their claim when comparing risky lotteries to sure outcomes. We conjecture that
the list presentation invokes reduction-like evaluation of the experiment, and since in-
1
dependence is known to be prone to failure in the vicinity of certainty,
the certainty
eect can rationalize our experimental ndings.
Our work includes a practical recommendation for experimentalists who would like
to continue to use the more precise information contained in choice lists. We believe
that the between-subject design employed in the current study, in which a control
group of subjects who make a single pairwise choice is used to test for systematic bias
in the choice list, could and should be easily incorporated in future studies.
Related Literature.
Becker, DeGroot, and Marschak (BDM, 1964) proposed a con-
venient method for measuring the valuation of an alternative by asking a subject
to match an alternative to a value (a matching task). Under a standard assumption, this method is incentive compatible.
2
The psychology literature has suggested
that this response mode might lead to dierences in behavior as compared to choice
1Section
6 in Cerreia-Vioglio, Dillenberger, and Ortoleva (2015) provides a recent discussion of this
evidence.
2The
original version of the BDM asked subjects to report a monetary valuation. A version of the
BDM where subjects report by matching a probability was used by Grether (1981) and studied by
Karni (2009).
ELICITING RISK PREFERENCES USING CHOICE LISTS
tasks (Lichtenstein and Slovic, 1971).
3
The preference reversal literature demon-
strated that these valuations can be systematically inconsistent with subjects' pairwise choices (Grether and Plott, 1979).
Karni and Safra (1987) and Segal (1988)
identied that a failure of a version of the independence axiom may undermine the
incentive compatibility of the BDM and rationalize observed behavior.
3
Their expla-
nations are based on incentives and thus could similarly apply to implementations of
the RIS aside from the BDM.
Meanwhile, the experimental economics literature (confronted with the challenges
involving the BDM in other domains) opted to use choice lists (e.g. Cohen, Jaray,
and Said (1987); Andersen, Harrison, Lau, and Rutström (2006)), which are a discrete
implementation of the BDM through a sequence of related pairwise choices.
4
It has
been pointed out that such a design will be incentive compatible under Kahneman
and Tversky's (1979) isolation hypothesis. Starmer and Sugden (1991) and Cubitt,
Starmer, and Sugden (1998) studied experimentally incentive compatibility of RIS
when a subject makes a small number of pairwise choice, and did not nd evidence
that rejects the isolation hypothesis. More recently, Cox, Sadiraj, and Schmidt (2014;
2015) use comparable designs to Starmer and Sugden but nd evidence against the
incentive compatibility of RIS (we discuss these studies in greater detail in Section
4.3), as do Harrison and Swarthout (2014), who compare subjects who make a single
choice to subjects who make a large number of choices.
None of the papers cited above uses choice lists, which have become the workhorse
method of experimental economists studying individual preferences. Moreover, the
earlier experimental papers that support incentive compatibility when subjects make
a small number of pairwise choices have been taken to justify the usage of choice lists,
notwithstanding the existing theoretical critique (Karni and Safra, 1987; Segal, 1988)
, some of which apply equally to choice lists.
of the BDM mechanism
For example,
Bardsley, Cubitt, and Loomes (2009, pages 272-273) conclude that since a choice list
is basically a sequence a pairwise choices, individuals will approach them in this way
(e.g.
isolate), unlike BDM - in which the response mode is dierent.
Our results
point to the empirical relevance of these theoretical critiques for experiments that use
choice lists.
3Holt (1986) pointed out that the RIS relies on
4A choice list with varying probabilities, like
the independence axiom for incentive compatibility.
the one used in the present study, was rst used
(without incentives) by Davidson, Suppes, and Siegel (1957), revisited by McCord and De Neufville
(1986), and was revisited (with the RIS) by Sprenger (2015). The analogous version of the BDMin-probabilities was used by Grether (1981).
ELICITING RISK PREFERENCES USING CHOICE LISTS
4
Table 1. The list questions
Q1
Q2
Line
Option A
Option B
Option A
Option B
1
(3, 1)
(3, 1)
(4, 1)
(4, .98)
(3, .5)
(3, .5)
(4, .50)
(4, .49)
.
.
.
.
.
.
.
.
.
.
.
.
11
.
.
.
(3, 1)
(4, .80)
(3, .5)
(4, .40)
.
.
.
.
.
.
.
.
.
.
.
.
26
(3, 1)
(4, .50)
(3, .5)
(4, .25)
2
.
.
.
2. Main Experimental design and Findings
2.1.
Design.
Subjects were recruited by posting a task to the mTurk online labor
market in September 2011- July 2012.
5
The details of a robustness check using a
student sample, conducted a year later, are described in Appendix B.2. The mTurk
workplace permitted us to collect actual choices of real workers, hence increasing the
external validity of our ndings.
Recruiting workers from an online labor market
allowed us to use a much smaller participation payment (of $1) than is typically
permitted with student subjects in order to make the part of a subject's payment
that depends on her decision relatively large and salient. The size of mTurk's online
labor force allowed us to recruit more subjects than would be possible with the typical
subject pool of a university experimental economics lab.
The main experiment consisted of sixteen dierent treatments; for now, we describe
the main treatment variation. Throughout the study, we employed a between-subject
design in which each subject was randomly assigned to one of the treatments.
In choice list treatments, each question consisted of a list of pairwise choices
in which the left hand side lottery (Option A) was held constant throughout the
list while the probability of winning the higher prize in the right hand side lottery
(Option B) decreased as subjects proceeded down the list. In one list treatments
subjects responded to a single choice list while in the the two lists treatments subjects responded to two choice lists.
In pairwise choice treatments, each subject faced either a single or two pairwise
choice problems which corresponded to the choice between the lotteries on line 11
of the choice lists appearing in Table 1. In one choice treatments in Table 2, subjects made only a single pairwise (binary) choice and were paid based on their choice.
5The
full details of the experimental procedure are described in Appendix A.
ELICITING RISK PREFERENCES USING CHOICE LISTS
5
Table 2. Answers to line 11
Pairwise choice
Choice list
One choice
Two choices
One list
Two lists
Q1
23%
24%
40%
46%
Q2
27%
33%
39%
38%
n
39/41
42
95/94
260
Treatments
P1,P2
P12, P21
L1,L2,S1,S2
All LO, LA, SO, SA
Fraction choosing the riskier option, all subjects
In the two choices treatments subjects made two pairwise choices (in random order), displayed on separate screens, one of which was randomly selected to determine
payment. Table 2 summarizes the main treatment groupings and also gives the distribution of responses in the comparable choice problem for each question; these results
will be discussed shortly.
In all of the list treatments, subjects were informed that one line from that list
- which corresponds to a single pairwise choice, would be randomly selected to be
played out to determine the subject's bonus payment. We allowed subjects to switch
from Option A to Option B at any number of points on the list, but used a pop-up to
warn subjects who switched from Option B to Option A and then back to Option B.
Table 1 presents the list of pairwise choices employed in all list treatments. Our full
design includes a total of sixteen treatments, which are described in detail in section
3.
Given that the experiment took at most 15 minutes to complete, the bonus payments of $3 or $4 provided very high incentives for this subject pool. In Appendix C
we discuss mTurk as a subject pool for economic experiments.
2.2.
Main results: choice lists versus pairwise choice.
Table 2 presents the
distribution of choices for line 11 of the list, grouped by the incentives provided.
The most obvious dierence in Table 2 is that in Q1 only 23% of subjects responding
in the pairwise choice treatments chose the risky (B) option, but 45% of subjects
responding to the choice list chose this option, a signicant dierence (p
Fisher's exact test).
6
< .001,
In Q2, 30% of subjects chose lottery B in pairwise choice, and
38% chose this lottery in the choice list, a dierence that is not statistically signicant
(p
= .17,
6We
Fisher's exact test).
note that 19% of subjects in the pairwise choice treatments who made two choices exhibited
the certainty eect the same order of magnitude as the dierence in proportions of risky choices
in Q1 between pairwise choice treatments and list treatments. We discuss within-subject tests in
Appendix B.1.
ELICITING RISK PREFERENCES USING CHOICE LISTS
6
Table 3. Treatments and treatment labels
One Q
Pairwise choice (P)
Two Qs, Order
Q1 (1)
Q2 (2)
Q1, Q2 (12)
Q2, Q1 (21)
39
41
20
22
36 (27)
35 (29)
47 (43)
45 (41)
36 (31)
33 (26)
37 (36)
32 (29)
26 (25)
25 (25)
Pay One
Standard
List (L)
Lists
List (O)
Pay Both
Lists (A)
Separate
Pay One
Screens
List (O)
then List
Pay Both
(S)
Lists (A)
48 (46)
49 (48)
Entries indicate the number of subjects and, for L and S treatments, the number of
subjects who exhibit single-switching in brackets (where applicable)
3. Full Design and Analysis of Treatment effects
3.1.
Detailed explanation of treatments.
Our design included four types of vari-
ations among list treatments that we conjectured might aect subject behavior (Table
3).
Among subjects in pairwise choice treatments (denoted by P), we varied whether
subjects responded to one pairwise choice problem or two.
Subjects in the single
pairwise choice, P1, respond to the problem corresponding to line 11 of the Q1 list in
Table 1, while subjects in P2 respond to the pairwise choice problem corresponding
analogously to the Q2 list. Subjects in P12 made a choice in Q1 and then in Q2, while
the order was reversed in the P21 group. Subjects who responded to two pairwise
choice problems faced these problems on separate screens, and one of their two choices
was randomly selected and played out for real to determine their payment at the end
of the experiment.
Among subjects in list treatments, we varied whether subjects responded only to
a single list (as in L1, L2, S1, and S2) or to two lists (as in all treatments ending
with 12 or 21). In the standard list treatments (denoted by L), a subject faced the
list(s) shown in Table 1 immediately after the instructions. In the separate screens
treatments (denoted by S), before completing each list a subject responded to a
sequence of (non-incentivized) pairwise choices that appeared on separate screens.
These hypothetical pairwise choice tasks were presented so as to converge towards
the switching point in the list for a subject with monotone preferences. Subjects then
responded to an incentivized list that was already lled in using their responses to
ELICITING RISK PREFERENCES USING CHOICE LISTS
7
Table 4. Display, payment mechanism and order eects in choice list
Q1
Q2
Separate screens eect (L=S)
.94
.81
Order eect (12=21)
.15
.05
Payment mechanism eect (O=A)
.38
.77
p-values reported for a Wilcoxon rank-sum test of equality of distribution
Tests use only subjects who exhibit single-switching
the pairwise choice tasks but was otherwise identical to that in the corresponding L
treatment. Crucially, subjects in the S treatment were free to change their answers
in the (incentivized) list.
7
For subjects who responded to two lists, we varied which
list came rst (denoted by 12 vs. 21). Also, for subjects who responded to two lists,
we varied whether a subject's bonus payment was determined by one pairwise choice
randomly picked from the two lists (O treatments), or two pairwise choices with one
randomly picked from each list (A treatments).
3.2.
Dierences across treatments.
There are no signicant dierences in re-
sponses to one or two pairwise choices shown on separate screens (p
and
p = .53
= .92
for Q1
for Q2, Fisher's exact tests), a nding consistent with the literature sup-
porting the incentive compatibility of the RIS in which subjects respond to a small
number of pairwise choices (Starmer and Sugden, 1991; Cubitt, Starmer, and Sugden,
1998).
By employing a choice list, the possibility of within-list contamination is equally
present in all L and S treatments. Our treatments allow us to test whether any of
the varied factors in the subset of two-list treatments induces dierences in behavior.
The 2x2x2 design embedded in the treatments ({L,S}×{O,A}×{12,21}) allows us to
separately test for the presence of display (separate screen), payment mechanism,
and order eects in each question. We nd that only one of these eects (Table 4) is
marginally signicant at 5% before, but not after, correcting for multiple hypothesis
testing by the Holm-Bonferroni method.
One treatment eect is that the proportion of subjects who exhibit single-switching
behavior is higher in the S (separate screens) treatments than in the L (list) treatments
(96% vs. 85%;
p < .001,
Fisher's exact test see Table 3). As one might expect, there
are relatively more (94% vs.
88%;
p = .02,
Fisher's exact test) single-switching
subjects in the treatments in which subjects faced only a single list (as opposed to
7In
principle, one could consider an experiment without a list (which is not our focus and is rarely
used in practice), with the RIS applied over pairwise choices made on separate screens.
ELICITING RISK PREFERENCES USING CHOICE LISTS
two).
8
Neither order nor payment mechanism signicantly aect the proportion of
single-switching subjects (p
= .85, .57
respectively, Fisher's exact tests).
We introduced the S treatment later in our experimental investigation, hoping it
would bridge the dierence between standard choice list and pairwise choice.
We
conjectured that a combination of isolation in pairwise choices made on dierent
screens, a lack of inuence of hypothetical versus real incentives, and a creation of a
default when the actual list was displayed, would eliminate the observed dierences
in responses made in choice list versus pairwise choice. Table 4 shows that the incentivized choice data do not support this view. Moreover, comparing the responses
to line 11 in the S treatments to pairwise choices, we nd signicant dierence in Q1
(p
< 0.001,
Fisher's exact test) and insignicant dierence in Q2 (p
= 0.21,
Fisher's
8
exact test), similar to the L treatment.
4. Theory: Pairwise choice versus choice lists
Faced with the robust experimental ndings documented so far, this section demonstrates they can be rationalized within existing theoretical models of non-expected
utility preferences. The theory provides the tools to evaluate the robustness of inference from existing studies to the behavior documented here, and enables one to
improve future experimental designs.
The two main facts that we want to account for by all models considered are that
subjects are more risk averse when making a pairwise choice involving certainty than
when making choices in a choice list, although their choices in the latter tend to
satisfy the independence axiom (on average).
n
Consider a simple lottery p = (xi , pi )i=1 paying xi with probability pi , where xi >
m
xi+1 for each i. A compound lottery π = [pi , πi ]i=1 pays the simple one-stage lottery
pi with probability πi .
i in list Q1 faces a pairwise choice between two lotteries ($3, 1)
(option A) and ($4, 1 − 0.02 (i − 1) ; $0, 0.02 (i − 1)) (option B). The subject's choices
A subject at line
in each of the lines combined with the RIS creates a two-stage compound lottery. A
monotone subject who chooses option B for the last time at line
i
receives the two-
stage compound lottery:
8We nd that 29% of single-switching subjects who faced one list and 43% of single-switching subjects
who faced two lists in the S treatments amended at least one of their choices.
They switched in
both directions, with 65% of switches involving a move from riskier-preliminary to safer-incentivized
choices. Looking only at line 11, the aggregate distribution of preliminary choices made in Q2 is
identical to the incentivized choices, and the preliminary choices in Q1 are slightly more risk-taking
(by 5 subjects) than the incentivized choices.
ELICITING RISK PREFERENCES USING CHOICE LISTS
(4.1)
9
1
1
26 − i
($4, 1) , ; . . . ; ($4, 1.02 − .02i; $0, 0.02i − 0.02) , ; ($3, 1) ,
26
26
26
In experiments that employ the RIS, Segal's (1990) compound independence axiom
(recursivity) applied to (4.1) is exactly the condition that species whether a subject
with preferences over two-stage lotteries would make the same choice in each problem
as she would if that were the only choice problem she faced. A subject with certainty
equivalent function
c (·)
over one-stage lotteries and preferences over compound lot-
teries that satisfy Segal's (1990) compound independence axiom is indierent between
the compound lottery in (4.1) and the single-stage lottery:
(4.2)
1
1
26 − i
c (($4, 1)) , ; . . . ; c (($4, 1.02 − .02i; $0, 0.02i − 0.02)) , ; $3,
26
26
26
A subject whose preferences are monotone with respect to rst order stochastic dominance and satisfy compound independence (she evaluates the choice of switching line
in a choice list according to (4.2)), will choose the risky option on line
B) whenever
c (($4, 1.02 − .02i; $0, 0.02i − 0.02)) > $3.
i
(Option
Thus, she will make identical
choices in each line of the choice list as she would have made in an experiment in
which she only faced the single pairwise choice. This prediction is inconsistent with
the main nding of our experiment.
We believe that the key observation in understanding our experimental ndings is
that a subject who chooses the risky option (B) on the rst few lines of the list, does
not
face a choice that involves certainty on line 11 of the choice list. In other words,
choosing the sure outcome (A) on line 11 does not lead to payment with certainty
since the RIS may select one of the rst lines, reducing the attractiveness of the safe
option for a subject sensitive to certainty.
We suggest to model this behavior by assuming that the subject chooses her switching line in the choice list as if she reduces the compound lottery in (4.1) according to
the laws of probability (Reduction of Compound Lotteries Axiom, ROCL; Samuelson
1952). Therefore, a subject who chooses option B for the last time at line
the one-stage lottery:
(4.3)
26 − i
.01i2 − .01i
1.01i − .01i2
; $3,
; $0,
$4,
26
26
26
i
induces
ELICITING RISK PREFERENCES USING CHOICE LISTS
10
4
0.8
B1
0.75
List1
B2
0.4
0.375
Line 11
List2
Line 11
A1
3
A2
0
0.5
The decision problem in a Marschak-Machina triangle under
reduction of compound lotteries: the pairwise choice in Q1 is between A1
and B1. The choice of a switching point in Q1 corresponds to a choice on
the curve List1, such that an earlier switching point corresponds to a point
on List1 that is closer to A1. The slope of List1 at line 11 equals the
slope of A1B1: consider an expected utility subject (satisfying compound
independence in addition to ROCL) who is indierent between A1 and B1.
She will be indierent between Option A and Option B on line 11 of the
choice list. Since her indierence curves are parallel straight lines, it follows
that her indierence curves' tangency point to List1 occurs at the point
corresponding to Line 11.
Figure 4.1.
Our assumption of ROCL follows the modeling approach used by Karni and Safra
(1987) to explain preference reversals.
In our view, the presentation of the choice
list and instructions describing the RIS make the incentive structure particularly
transparent to subjects, leading them to choose the switching point
as if
they reduce
9
the compound lottery formed by their choices in the list and the RIS.
9There
exist evidence against ROCL as a descriptive axiom in other contexts. ROCL is a convenient
modeling simplication that is sucient but not necessary.
ELICITING RISK PREFERENCES USING CHOICE LISTS
11
We show by way of examples that plausible specications of preferences from the
non-expected utility literature can generate the type of behavior we observe in the experiment. We study the cautious expected utility (Cerreia-Vioglio, Dillenberger, and
Ortoleva, 2015) and rank dependent utility (Quiggin, 1982; Yaari, 1987) with the neoadditive weighting function (Chateauneuf, Eichberger, and Grant, 2007; Webb and
Zank, 2011). The former model generalizes expected utility to accommodate failure
of independence around certainty, while the latter weighting function transparently
captures the certainty eect
10
within the widely-used class of rank dependent utility
preferences.
4.1.
Cautious Expected Utility.
Suppose a subject's preferences are represented
by cautious expected utility; that is, she ranks any (single-stage) lottery according to
n
P
n
−1
pi u(xi ) , where U is a set of expected utility functions
U ((xi , pi )i=1 ) = min u
u∈U
i=1
from R+ → R. Cerreia-Vioglio, Dillenberger, and Ortoleva show that the Cautious
Expected Utility is characterized by the Negative Certainty Independence (NCI; Dillenberger, 2010) axiom (in addition to continuity and weak payo monotonicity). In
our setting, the NCI axiom implies that
(4.4)
($4, p; $0, 1 − p) % ($3, 1) =⇒
λ ($4, p; $0, 1 − p) (1 − λ) ($4, `1 ; $3, `2 ; $0, 1 − `1 − `2 )
% λ ($3, 1) (1 − λ) ($4, `1 ; $3, `2 ; $0, 1 − `1 − `2 )
where
is the linear mixture operator,
λ ∈ (0, 1). This is equivalent to:
for any
`1 , `2 ≥ 0
with
`1 + `2 ≤ 1
and any
($4, λp + (1 − λ) `1 ; $3, (1 − λ) `2 ; $0, λ (1 − p) + (1 − λ) (1 − `1 − `2 ))
% ($4, (1 − λ) `1 ; $3, λ + (1 − λ) `2 ; $0, (1 − λ) (1 − `1 − `2 ))
Consider a subject who chooses the risky alternative in a single pairwise choice,
hence she ranks
($4, .8; $0, .2) % ($3, 1).
Monotonicity with respect to rst order
stochastic dominance and transitivity imply that:
($4, 1 − .02 (i − 1) ; $0, .02 (i − 1)) % ($3, 1)
(4.5)
for
i ≤ 11
which means that the subject prefers the risky option (B) to the sure outcome (A)
on lines
10Most
1, . . . 11.
Note that for each line
i of the Q1 choice list, the reduced one-stage
other weighting functions accommodate the certainty eect, but also try to match other
nuances of choice patterns over risky prospects, which are of lesser interest here.
ELICITING RISK PREFERENCES USING CHOICE LISTS
12
lottery corresponding to arbitrary choices on all other lines, and the risky or safe
alternative on line
i
i
some `1 , `2
i,
can be written as a mixture of
($4, `i1 ; $3, `i2 , $0, 1 − `i1 − `i2 )
for
i
i
for which `1 + `2
≤ 1 (corresponding to the lottery induced by choices
on lines other than i), and ($4, 1 − .02 (i − 1) ; $0, .02 (i − 1)) or ($3, 1) (corresponding
to the choice on line i). By (4.5), NCI (4.4) implies that for each i ≤ 11 she would
≥0
also rank
1
25
($4, 1 − .02 (i − 1) ; $0, .02 (i − 1)) ⊕
$4, `i1 ; $3, `i2 , $0, 1 − `i1 − `i2
26
26
1
25
%
($3, 1) ⊕
$4, `i1 ; $3, `i2 , $0, 1 − `i1 − `i2
26
26
and strictly so for i < 11. Thus it follows that the subject will choose the risky
option on line 11 and all preceding lines. Thus we have shown that a subject who
($4, .8; $0, .2) over ($3, 1) in a single
($4, .8; $0, .2) over ($3, 1) in line 11 of the Q1 list.
satises NCI and chooses
also choose
pairwise choice would
An alternative way to view this result is through Figure 4.1. Recall that by Lemma
3 in Dillenberger (2010), indierence curves of preferences that satisfy the NCI axiom
A1; this indierence curve must also be linear.
Therefore, if the subject chose lottery B1 ($4, 0.8; $0, 0.2) in the pairwise choice, it
must be that the steepest indierence curve is atter than the line A1B1. It follows
are steepest through
($3, 1)
- point
that the optimal switching line in the Q1 choice list must be to the right of the point
List1, since the
the line A1B1.
representing line 11 on the curve
line 11 is equal to the slope of
slope at the point corresponding to
Preferences that satisfy NCI can be consistent with the reverse pattern of choosing
($3, 1)
over
($4, 0.8; $0, 0.2)
11 of the Q1 choice list.
in the pairwise choice and
List1,
on line
Visually, this corresponds to the fact that under NCI, a
subject's steepest indierence curve must run through
line intersects
($4, 0.8; $0, 0.2)
A1
and be linear. Since this
the indierence curve that determines her switching point in
the Q1 list must be atter than her indierence curve through A1. This creates the
possibility that this subject would switch after line 11 in the list, corresponding to a
point on
List1
to the right of the point representing line 11. Given information on
a subject pairwise choice in Q1, NCI does not have any implication for Q2, except
that it is inconsistent with the combination of choosing
($4, 0.8; $0, 0.2)
($4, 0.8; $0, 0.2)
($3, 0.5; $0, 0.5)
in Q2 and
in Q1 in the pairwise choice problems. This is because a choice of
over
($3, 1)
in a pairwise choice would imply that the slope of the
subject's indierence curve through
A1
is atter than the line
A1B1.
NCI would
ELICITING RISK PREFERENCES USING CHOICE LISTS
then require that the slope of her indierence curve through
line
A2B2.
A2
13
be atter than the
Cautious expected utility preferences can therefore accommodate our
main ndings, and exclude the opposite of the main behavioral pattern observed in
the experiment.
Rank Dependent Utility.
4.2.
Suppose a subject has rank-dependent utility prefn
erences; that is, she ranks any (single-stage) lottery according to U ((xi , pi )i=1 ) =
"
!
!#
n
P
P
P
f
pj − f
pj
u (xi ). Suppose further the subject evaluates her choices
i=1
j<i
j≤i
in our experiment according to (4.1). Take the neo-additive weighting function
Wakker, 2010, pages 208-210), of the form:

bp
f (p) =
1
(4.6)
(see
if
0≤p<1
if
p=1
u(3) ≥ .75
(Chew, Karni, and Safra, 1987). Assume the subject is indierent between ($3, 1) and
($4, 0.8; $0, 0.2) in pairwise choice. Then u (3) = f (0.8) = 0.8b. If b < 1 it follows
12
that ($3, 0.5; $0, 0.5) ≺ ($4, 0.4; $0, 0.6).
where
0 ≤ b ≤ 1.
f
11
Normalize
u(0) = 0, u(4) = 1.
Risk aversion implies that
Consider a continuous approximation to the choice lists in which the subject chooses
a switching point from B to A,
q1
and
q2
in Q1 and Q2 respectively, each of which
13
denote the probability of not winning on a particular line of the list.
Let each line
[0, 0.5] in Q1 and on [0.5, .75]
Q1 (q1 ) = ($4, 2q1 − q12 ; $3, 1 − 2q1 ; $0, q12 ) so
be selected for payment with a uniform probability on
in Q2. Choosing
q1
induces the lottery
the subject would choose
q1∗
to maximize:
U (Q1 (q1 )) = f 2q1 − q12 + u (3) f 1 − q12 − f 2q1 − q12
Substituting the neo-additive weighting function (4.6) and assuming an indierence
in the pairwise choice version of Q1 results in an optimal switching point of
1 − f (0.8) = 1 − 0.8b.
q1∗ =
Therefore the subject will switch to the safe outcome between
the lines corresponding to losing probabilities of 0.2 and 0.25 in the choice list. The
11It
is trivial to add a constant
a≥0
to the weighting function in order to capture the possibility
eect, and the same results as below hold as long as
a+b<1
and
a + 0.8b < 0.8.
12Since U ($4, 0.4; $0, 0.6) = 0.4b > 0.4b2 = (0.5b) (0.8b) = U ($3, 0.5; $0, 0.5).
13So the probability of winning $4 at the switching point is 1 − q . We use the
i
continuous approxi-
mation for simplicity and have obtained comparable results without this simplication.
ELICITING RISK PREFERENCES USING CHOICE LISTS
subject's payo at
if
b<1
q1∗
is higher than her payo of always choosing
14
($3, 1).14
That is,
($4, 0.8; $0, 0.2) to ($3, 1) in the choice list.
+ 4q2 − 23 ; $3, 32 − 2q2 ; $0, 2q22 − 2q2 + 1 . So the sub-
the subject will strictly prefer
$4, −2q22
Q2 (q2 ) =
ject chooses q2 to maximize:
3
3
2
2
2
U (Q2 (q2 )) = f −2q2 + 4q2 −
+ u (3) f −2q2 + 2q2 − f −2q2 + 4q2 −
2
2
Similarly,
Substituting the neo-additive weighting function (4.6) and calculating the optimal
1−q ∗
u(3)
∗
switching point yields 1 − q2 =
= 2 1 . Therefore, the subject will switch at
2
the same line in Q1 and Q2. That is, although she exhibited the certainty eect in
pairwise choice, the choices made in the two lists will be consistent with expected
utility.
These results can be visualized using Figure 4.1. The indierence curves of the neoadditive weighting function are parallel straight lines in the interior of the triangle, but
are discontinuous on its boundary in which the probability of earning $0 equals 0 (if
u(3)
b < 1). The indierence curves are atter (their slope equals 1−u(3)
) than the dashed
line A1B1, so the indierence curve passing through B1 approaches the vertical axis
15
above A1.
A subject who is indierent between
therefore choose a switching point in
List1
($3, 1)
and
($4, 0.8; $0, 0.2)
that is to the right of Line 11.
will
Since
the indierence curves in the interior of the triangle are parallel straight lines, this
subject will strictly prefer B2 to A2 in pairwise choice (exhibit the certainty eect)
but will have the same switching point in
List2
as in
List1
(just like an expected
utility subject).
4.3.
Alternative Explanations and Relation to Existing Findings.
Our lead-
ing explanation for the results reported in this study was initially posited by Karni
and Safra (1987) as a potential explanation for the preference reversal phenomenon.
But in contrast to experiments in the preference reversal literature, we compare behavior in pairwise choice treatments to behavior in identically-framed pairwise choices
contained in a list, thus procedure-based explanations of the preference reversal phenomenon do not obviously apply to our study.
14U (Q1 (q ∗ )) = f
1
Nonetheless, for completeness, we
i
h
2
2
=
1 − (1 − q1∗ ) + f (0.8) f 1 − q1∗2 − f 1 − (1 − q1∗ )
i
h
i
h
2
2
2
=
= b 1 − (0.8b) + 0.8b2 2 ∗ 0.8b − (0.8b) − 1 − (0.8b)
= b + b (0.8b) ((0.8b) − 1).
U (Q1 (q1∗ )) − U ($3, 1) = b [0.2 − (0.8b) (1 − 0.8b)] ≥ 0
since 0.75 ≤ 0.8b ≤ 0.8.
Thus:
15The
limit on the vertical axis equals
0.8(1−b)
1−0.8b .
ELICITING RISK PREFERENCES USING CHOICE LISTS
15
review a leading explanation for the preference reversal phenomenon as well as two
potential sources of bias in choice lists and argue that they cannot explain our ndings.
Contingent Weighting and Scale Compatibility.
One explanation for preference rever-
sals between pairwise choice and the BDM is that subjects' preferences depend on
how they are elicited, in violation of the principle of procedure invariance. In particular, Tversky, Slovic, and Kahneman (1990) posit that subjects facing a matching
task as in the BDM put more weight on the attribute being matched their scale
compatibility hypothesis. Unlike in earlier studies of preference reversals, our paper
compares behavior in pairwise choice treatments to behavior in an identically-framed
pairwise choice contained in a list (not a matching task), so this explanation does not
obviously apply. But if we did treat the list as akin to a BDM-matching task where
subjects match an indierence probability, then the scale compatibility hypothesis
would predict that subjects would put more weight on the probability dimension
when facing the list. This would generate more risk averse choices in the lists, which
is the opposite of what we nd.
Middle-Switching Heuristic.
It has been conjectured that when choices are presented
in a list, subjects are biased towards switching at the middle of the list (Andersen,
Harrison, Lau, and Rutström, 2006; Beauchamp, Benjamin, Chabris, and Laibson,
2015).
Any subject who switches at the middle of the list would choose the risky
option on line 11, which can qualitatively provide a possible explanation for our
results.
To see whether this can quantitatively explain our results, we re-run our
main analysis after discarding the 32 monotone subjects who switched at the middle
of the Q1 list (immediately after line 13), and we nd that 39.0% of the remaining 323
subjects picked the risky option on line 11, which is still signicantly dierent from
the 23.5% who chose the risky option in the Q1 pairwise choice treatments (p=.01,
Fisher's exact test).
16
We thus conclude that a middle-switching heuristic cannot
quantitatively explain our results.
Fixed side of the list as a reference point.
An alternative explanation is that subjects
facing a list treat the xed side of the list as a reference point (Sprenger, 2015).
However, if subjects were loss averse relative to the lottery on the xed side of the list
16This
is an extremely conservative correction, since it also discards subjects who intentionally
switched at line 13.
ELICITING RISK PREFERENCES USING CHOICE LISTS
16
as the reference point, they would demonstrate more risk aversion in the Q1 list than
in the corresponding pairwise choice treatments the opposite of what we observe.
Relation to other existing ndings.
The renewed interest in empirically evaluating
the incentive compatibility of various payment mechanism when responding to risk
tasks should be credited to Cox, Sadiraj, and Schmidt (2015), who use a betweensubjects design to compare a large number of payment schemes (including RIS), to a
single pairwise choice. In some comparisons, they nd signicant dierences between
behavior in the RIS to behavior in a single pairwise choice. They do not, however,
study choice lists, which are a crucial intermediate case between their design and the
BDM mechanism.
17
In another study (Cox, Sadiraj, and Schmidt, 2014) demonstrate
that using RIS while including additional comparisons among similar lotteries that
either dominate or are dominated by the original lotteries may aect subjects' choices.
In the choice list used in the current investigation, we vary only one alternative
(Option B), and include both better and worse lotteries than the lottery included in
line 11. Naturally, this framing may have an eect on choices made in line 11, but we
believe that this is similar in nature to the eect discussed in the middle-switching
above which we showed could not account for our ndings.
5. Conclusion
Our between-subjects study documents a signicant violation of incentive compatibility in choice lists for some pairwise choices that involve certain payment.
We
demonstrate how these ndings could be understood in light of the interaction between non-expected utility preferences and the RIS (Karni and Safra, 1987).
A typical experiment whose primary goal is to measure preferences at the individual
level must present a subject a sequence of decision problems. Usually (but not always)
one of them is selected for payment. A core feature of our design is a between-subject
comparison with a group of subjects who make a single pairwise choice. We believe
that this design feature can and should be incorporated into future studies to evaluate
existing or proposed methods for eliciting preferences.
17Related
work by Castillo and Eil (2014) also explores behavior in choice lists by proposing and
testing a model inspired by status-quo bias; however they do not compare behavior in choice lists
to behavior in pairwise choice.
Recent work by Brown and Healy (2014) follows up the current
study by considering behavior in choice lists that do not involve sure payments, and indeed cannot
reject the incentive compatibility of RIS in a treatment similar to ours. Work in progress by Loomes,
Maadi, and Pogrebna (2015) also compares choices made in choice lists to pairwise choices. In recent
theoretical work, Azrieli, Chambers, and Healy (2015) generalize Karni and Safra's results and study
the conditions for the existence of incentive compatible mechanisms for eliciting preferences.
ELICITING RISK PREFERENCES USING CHOICE LISTS
17
References
and E. Rutström (2006): Elicitation
Experimental Economics, 9(4), 383405.
Andersen, S., G. Harrison, M. Lau,
using multiple price list formats,
Azrieli, Y., C. Chambers,
and
P. Healy (2015): Incentives in Experiments: A
Theoretical Analysis, Working Paper.
Bardsley, N., R. Cubitt,
and
G. Loomes (2009):
Experimental Econoimics:
Rethinking the Rules. Princeton University Press.
Beauchamp, J., D. Benjamin, C. Chabris, and D. Laibson (2015):
Controlling
for the Compromise Eect Debiases Estimates of Risk Preference Parameters,
Working Paper.
Becker, G., M. DeGroot,
and
J. Marschak (1964): Measuring utility by a
single-response sequential method,
Behavioral science, 9(3), 226232.
Berinsky, A., G. Huber, G. Lenz, et al. (2012):
Markets for Experimental Research:
Analysis.
Brown, A.,
and
P. Healy (2014):
Evaluating Online Labor
Amazon.com's Mechanical Turk,
Political
Monotonicity failure versus framing eects
in list elicitation procedures, presented at the Southern Economic Association
Meeting in Atlanta.
Buhrmester, M., T. Kwang,
and
S. Gosling (2011):
Amazon's Mechanical
Turk: A New Source of Inexpensive, Yet High-Quality, Data?,
Psychological Science, 6(1), 35.
Castillo, M., and D. Eil (2014):
Perspectives on
Taring the Multiple Price List: Imperceptive
Preferences and the Reversing of the Common Ratio Eect, presented at the
Southern Economic Association Meeting in Atlanta.
and P. Ortoleva (2015): Cautious
Expected Utility and the Certainty Eect, Econometrica, 83(2), 693728.
Chateauneuf, A., J. Eichberger, and S. Grant (2007): Choice under uncertainty with the best and worst in mind: Neo-additive capacities, Journal of
Economic Theory, 137(1), 538567.
Chew, S. H., E. Karni, and Z. Safra (1987): Risk Aversion in the Theory of Expected Utility with Rank Dependent Probabilities, Journal of Economic Theory,
Cerreia-Vioglio, S., D. Dillenberger,
42, 370381.
Cohen, M., J.-Y. Jaffray,
and
T. Said (1987):
Experimental comparison of
individual behavior under risk and under uncertainty for gains and for losses,
Organizational Behavior and Human Decision Processes, 39(1), 122.
ELICITING RISK PREFERENCES USING CHOICE LISTS
Cox, J., V. Sadiraj,
and
18
U. Schmidt (2014): Asymmetrically dominated choice
problems, the isolation hypothesis and random incentive mechanisms,
PLoS One,
9(3), 13.
(2015): Paradoxes and mechanisms for choice under risk,
Experimental
Economics, 18, 215250.
and R. Sugden (1998): On the validity of the random
lottery incentive system, Experimental Economics, 1(2), 115131.
Davidson, D., P. Suppes, and S. Siegel (1957): Decision making: an experimental approach. Stanford University Press.
Cubitt, R., C. Starmer,
Dillenberger, D. (2010): Preferences for one-shot resolution of uncertainty and
Allais-type behavior,
Econometrica, 78(6), 19732004.
Greiner, B. (2015): Subject pool recruitment procedures: organizing experiments
with ORSEE,
Journal of the Economic Science Association, 1(1), 114125.
Grether, D. (1981): Financial incentive eects and individual decision-making, .
and C. Plott (1979): Economic theory of choice and the preference
reversal phenomenon, American Economic Review, 69(4), 623638.
Harrison, G., and J. Swarthout (2014): The independence axiom and the
bipolar behaviorist, Theory and Decision, 77, 423438.
Holt, C. (1986): Preference reversals and the independence axiom, American Economic Review, 76(3), 508515.
Horton, J., D. Rand, and R. Zeckhauser (2011): The online laboratory: conducting experiments in a real labor market, Experimental Economics, 14(3), 399
Grether, D.,
425.
and A. Tversky (1979):
Econometrica, 47, 263291.
Kahneman, D.,
under risk,
Prospect theory: an analysis of decision
Karni, E. (2009): A mechanism for eliciting probabilities,
Econometrica,
77(2),
603606.
Karni, E.,
and
Z. Safra (1987):
`Preference reversal' and the observability of
Econometrica, pp. 675685.
(2005): Testing Statistical Hypotheses,
preferences by experimental methods,
Lehmann, E.,
and
J. Romano
Springer
Texts in Statistics. Springer, 3rd edn.
and P. Slovic (1971): Reversal of preference between bids and
choices in gambling decisions, Journal of Experimental Psychology, 89(1), 4655.
Loomes, G., H. Maadi, and G. Pogrebna (2015): Multiple Price Lists and the
Lichtenstein, S.,
Elicitation of Risk Attitudes, Working Paper.
ELICITING RISK PREFERENCES USING CHOICE LISTS
Mason, W.,
and
S. Suri (2011):
Mechanical Turk,
McCord, M.,
and
19
Conducting behavioral research on Amazon's
Behavior Research Methods, pp. 123.
R. De Neufville (1986): " Lottery Equivalents": Reduction
of the Certainty Eect Problem in Utility Assessment,
Management Science,
pp.
5660.
and P. Ipeirotis (2010): Running experiments on
amazon mechanical turk, Judgment and Decision Making, 5(5), 411419.
Quiggin, J. (1982): A theory of anticipated utility, Journal of Economic Behavior
& Organization, 3(4), 323343.
Samuelson, P. (1952): Probability, Utility, and the Independence Axiom, Econometrica, 20(4), 670678.
Paolacci, G., J. Chandler,
Segal, U. (1988): Does the preference reversal phenomenon necessarily contradict
the independence axiom?,
(1990): Two-stage
American Economic Review, 78(1), 233236.
lotteries without the reduction axiom, Econometrica,
58(2), 349377.
Sprenger, C. (2015): An Endowment Eect for Risk: Experimental Tests of Sto-
chastic Reference Points,
Starmer, C.,
and
Journal of Political Economy, 123(6), 14561499.
R. Sugden (1991):
Does the random-lottery incentive sys-
tem elicit true preferences? An experimental investigation,
American Economic
Review, 81(4), 971978.
and D. Kahneman (1990): The Causes of Preference
Reversal, American Economic Review, 80(1), 204217.
Wakker, P. (2010): Prospect Theory: For Risk and Ambiguity. Cambridge UniverTversky, A., P. Slovic,
sity Press, Cambridge, UK.
Webb, C.,
and
H. Zank (2011): Accounting for optimism and pessimism in ex-
Journal of Mathematical Economics, 47(6), 706717.
(1987): The dual theory of choice under risk, Econometrica,
pected utility,
Yaari, M.
55(1),
95115.
Appendix A. Experimental details
Our online experiment was composed of two components: the mTurk interface used
to recruit and pay subjects and an external experiment website where subjects made
their choice decisions.
Using the mTurk interface, we (as the recruiter) released an ad for a task (HIT
for Human Intelligence Task) which could be viewed by online workers (turkers). All
ELICITING RISK PREFERENCES USING CHOICE LISTS
20
Figure A.1. Mechanical Turk HIT description
Description:
This HIT asks you to make a series of choices among alternatives that involve monetary prizes. The HIT
should take between 5-10 minutes to complete. Your answers will be used in an academic study on
decision-making.
Please click the link below to begin the HIT. Please enter your mTurk Worker ID and the following mTurk
HIT ID where prompted in order to begin the survey.
mTurk HIT ID: ${pw}
When you are finished, you will receive a Completion Code that you must enter in the box below to
receive credit for participation.
Completion Code:
Please do not take this HIT if you are not willing to commit 10 minutes of your full concentration to the HIT.
The data we collect is being used for scientific research. We greatly appreciate your full attention and
careful consideration of each question.
Note: Any versions of this HIT can only be taken once by each worker. If you complete
this HIT more than once, you will only be paid for the first time. Click here for list of workers
who have completed a version of this HIT.
*Note: Javascript is required for this HIT
Please accept the HIT before you begin!
CLICK HERE TO BEGIN HIT
turkers that satisfy the required criteria can view a description of the HIT (in our
case, we required turkers to have a US based account and a completion record of
95% or greater). Our HIT description was a short description of the experiment that
included a unique HIT passcode along with a link to our experiment webpage, which
was hosted on a private server. Turkers could either accept or decline the HIT once
they read the description. If a turker accepted the HIT, he would click on the link
to the external experiment website and enter his unique mTurk identier and the
HIT passcode. The passcode was unique per HIT and one-time use. The passcode
would expire after the turker completed the experiment.
This prevented a turker
from completing the HIT multiple times. Figure A.1 provides an example of one of
our HIT descriptions.
ELICITING RISK PREFERENCES USING CHOICE LISTS
21
Once subjects logged into the external experiment website, they consented to the
experiment, read the instructions, answered a short quiz to indicate understanding,
made their choices, and were then informed of their bonus payment (determined by
a random number computer generator) and received a unique completion code. Subjects then entered the completion code back in the HIT page in the mTurk interface
to complete the HIT. Figures A.2 and A.3 provide a set of example instructions and
questions from treatment L1. Instructions and questions from other treatments were
similar.
Subjects were linked in our dataset (that contained the choice and payment data)
to the mTurk site by both their mTurk identier and their completion code.
This
allowed us to match a turker's account with his payment information recorded in our
dataset, and pay the turker accordingly.
Subjects were paid a at rate payment for completing the HIT and earned a `bonus'
based on their choices. In our experiment, the payment corresponded to the show-up
fee and the bonus corresponded to the incentivized payment. Payments must be set
equal for all turkers who complete a HIT in the same batch, but bonuses may dier.
Both payments and bonuses are at the recruiters discretion, thus turkers do not need
to be paid unless they complete the task. We oered a payment of $1 for completing
the HIT, and a bonus of $0, $3, or $4 corresponding to the risky outcomes in our
lotteries.
Bonuses depended upon the element of chance described in the RIS, the
lotteries and the subject's choices. All payments were in American dollars. Subjects
completed the experiment (HIT) by submitting a completion code generated by our
website to the mTurk interface. A random number generator was used to resolve all
risks automatically, and subjects were informed of how much of a bonus would be
paid after completing the study. Payments were credited to subjects' mTurk accounts
within 30 minutes of completing the experiment.
A recruiter can recruit
n
subjects for an experiment by releasing a 'batch' with
n
HITS. These tasks can be identical or individualized by the inputs in a csv le. We
used individualized tasks dierentiated by unique HIT passcodes.
HITs recruits
n
A batch with
n
dierent subjects. However, dierent batches of HITs could poten-
tially be completed by the same subjects who completed HITs in dierent batches.
The mTurk interface has no method to block turkers who have completed a HIT in
a previous batch from completing future HITs. Our external experimental interface
prevented this by blocking such subjects by matching entered mTurk identiers with
ELICITING RISK PREFERENCES USING CHOICE LISTS
Figure A.2. Instructions for treatment L1
22
ELICITING RISK PREFERENCES USING CHOICE LISTS
23
Figure A.3. List for treatment L1
YƵĞƐƚŝŽŶϭ
&ŽƌĞĂĐŚůŝŶĞďĞůŽǁ͕ƉůĞĂƐĞĐŚŽŽƐĞǁŚĞƚŚĞƌLJŽƵƉƌĞĨĞƌKƉƚŝŽŶŽƌKƉƚŝŽŶ͘
zŽƵƌďŽŶƵƐǁŝůůďĞĚĞƚĞƌŵŝŶĞĚďLJLJŽƵƌĐŚŽŝĐĞ;KƉƚŝŽŶŽƌKƉƚŝŽŶͿĨƌŽŵĂƌĂŶĚŽŵůLJƐĞůĞĐƚĞĚůŝŶĞ͘ĂĐŚĐŚŽŝĐĞĐŽƵůĚ
ďĞƚŚĞŽŶĞƚŚĂƚĐŽƵŶƚƐ͕ƐŽLJŽƵƐŚŽƵůĚƚƌĞĂƚĞĂĐŚĂŶĚĞǀĞƌLJůŝŶĞĂƐŝĨƚŚĂƚĐŚŽŝĐĞǁŝůůĚĞƚĞƌŵŝŶĞLJŽƵƌďŽŶƵƐƉĂLJŵĞŶƚ͘
ŶƵŵďĞƌǁŝůůďĞĚƌĂǁŶĨƌŽŵĂďŽdžĐŽŶƚĂŝŶŝŶŐϭϬϬďĂůůƐŶƵŵďĞƌĞĚϭͲϭϬϬĂƐĚĞƐĐƌŝďĞĚŝŶƚŚĞŝŶƐƚƌƵĐƚŝŽŶƐ͘/ĨƚŚĞŶƵŵďĞƌ
ŽŶƚŚĞďĂůůĚƌĂǁŶŝƐůĞƐƐƚŚĂŶŽƌĞƋƵĂůƚŽƚŚĞŶƵŵďĞƌŝŶĚŝĐĂƚĞĚŝŶƚŚĞƋƵĞƐƚŝŽŶ͕ƚŚĞŶLJŽƵǁŝůůďĞƉĂŝĚĂĐĐŽƌĚŝŶŐƚŽLJŽƵƌ
ĐŚŽŝĐĞŝŶƚŚĞƐĞůĞĐƚĞĚůŝŶĞ͘
&ŽƌĞdžĂŵƉůĞ͕ŝŶƚŚĞƐĞĐŽŶĚůŝŶĞďĞůŽǁ͗KƉƚŝŽŶŝƐΨϯŝĨƚŚĞŶƵŵďĞƌŽŶƚŚĞďĂůůĚƌĂǁŶŝƐůĞƐƐƚŚĂŶŽƌĞƋƵĂůƚŽϭϬϬ͘KƉƚŝŽŶ
ŝƐΨϰŝĨƚŚĞŶƵŵďĞƌŽŶƚŚĞďĂůůĚƌĂǁŶŝƐůĞƐƐƚŚĂŶŽƌĞƋƵĂůƚŽϵϴ͘/ĨLJŽƵĐŚŽŽƐĞKƉƚŝŽŶ͕ƚŚĞŶLJŽƵǁŽƵůĚďĞƉĂŝĚΨϯ͘/Ĩ
LJŽƵĐŚŽŽƐĞKƉƚŝŽŶ͕ƚŚĞŶLJŽƵǁŽƵůĚďĞƉĂŝĚΨϰŝĨƚŚĞŶƵŵďĞƌŽŶƚŚĞďĂůůĚƌĂǁŶŝƐůĞƐƐƚŚĂŶŽƌĞƋƵĂůƚŽϵϴ͘
^ĐƌŽůůLJŽƵƌŵŽƵƐĞŽǀĞƌƚŚĞďŽdžĞƐďĞůŽǁƚŽĐŚŽŽƐĞLJŽƵƌƉƌĞĨĞƌƌĞĚŽƉƚŝŽŶ͘zŽƵƌƐĞůĞĐƚĞĚĐŚŽŝĐĞǁŝůůďĞŚŝŐŚůŝŐŚƚĞĚŝŶ
LJĞůůŽǁ͘WůĞĂƐĞƐĞůĞĐƚĞŝƚŚĞƌKƉƚŝŽŶŽƌKƉƚŝŽŶŝŶĞĂĐŚůŝŶĞ͘
KƉƚŝŽŶ
Line #
$3 if the numberon the
ball drawnis less than or
equal to:
KƉƚŝŽŶ
$4 if the numberon the
ball drawnis less than or
equal to:
1
100
100
2
100
98
3
100
96
4
100
94
5
100
92
6
100
90
7
100
88
8
100
86
9
100
84
10
100
82
11
100
80
12
100
78
13
100
76
14
100
74
15
100
72
16
17
100
100
70
68
18
19
20
21
22
23
24
25
26
100
100
100
100
100
100
100
100
100
66
64
62
60
58
56
54
52
50
When you have made all of your choices please press 'Continue'
Continue
ELICITING RISK PREFERENCES USING CHOICE LISTS
24
a list of those who had previously completed a HIT (this list was automatically updated each time a turker completed a HIT). We also built in a secondary feature to
ensure subject uniqueness by recording a subject's IP address when he completed a
HIT. We could then cross-check the list of IP addresses to ensure that the same IP
address did not appear for multiple subjects.
Appendix B. Robustness checks and Additional Analysis
B.1.
Tests of the independence axiom.
Under both pairwise choice and choice
list, aggregate responses are close to expected utility. Choices exhibit a slight common ratio eect with pairwise choice, and a slight reverse common ratio eect when
using choice lists. In pairwise choice, the violations of the independence axiom are
not signicant (p
= .80
for a Fisher's exact test for P1 vs P2,
all of the P treatments).
p = .38
when pooling
However, since these two questions only look in a very
particular region of the Marschak-Machina triangle, we do not view this as providing
strong evidence in favor of the independence axiom. Pooling all the list treatments
and ignoring the within-subject nature of part of the treatments, the median switching points (for single-switching subjects) in the choice lists are consistent with the
following ranking:
($4, .86) ($3, 1) ($4, .84)
and
($4, .44) ($3, .5) ($4, .43),
which
is inconsistent with the independence axiom in the reverse direction of the standard
common-ratio eect. This violation of the independence axiom is quantitatively small
and is statistically signicant at 5% (p
= .02,
rank-sum test).
The aggregate analysis of behavior masks substantial heterogeneity of individual
decisions that within-subject analysis of choice list data picks up. In choice lists, we
detect violations of the independence axiom for 78.5% of single-switching subjects,
split between standard common ratio and reverse common ratio violations with the
latter being slightly more frequent (Table 5), and is statistically insignicant at 5%
(p
= .052,
Sign test).
18
Pairwise choice data only detects violations of the indepen-
dence axiom (AB and BA choice patterns) for 29% of subjects. Using only data from
line 11 would detect a similar fraction of EU violations for the L and S treatments; in
both cases, the deviations from the independence axiom are statistically signicant
at 5% (p
B.2.
= .388
and
p = .019
Sign tests for pairwise and list data, respectively).
Comparison to a student subject pool.
A reader may have a legitimate
concern that our main nding, which is the substantial and signicant dierence in
18In
136.
a Sign test one omits the zeroes to achieve UMP test. See Lehmann and Romano (2005) page
ELICITING RISK PREFERENCES USING CHOICE LISTS
25
Table 5. Behavior relative to the independence axiom: within-subject
Common Ratio
Independence
Reverse Common
Ratio
Choice List (L and
a
33.3%
21.5%
45.2%
11.4%
68.0%
20.6%
19.1%
71.4%
9.5%
S)
Choice List (L and
b
S), line 11
c
Pairwise Choice (P)
a percentage of single-switching subjects who switched earlier to lottery A in Q1 than in Q2
(common ratio), switched on the same line in Q1 and Q2 (independence), or switched on an
earlier line to lottery A in Q2 than in Q1(reverse common ratio).
b percentage of single-switching subjects who answered both Q1 and Q2 and: chose lottery A in
Q1 and lottery B in Q2 on line 11 (common ratio), made similar choices on line 11 (AA or BB,
independence), chose B in Q1 and A in Q2 on line 11.
c percentage of subjects in the P12 and P21 treatments, who chose A in Q1 and B in Q2 (common
ratio), made similar choice in Q1 and Q2 (independence), chose B in Q1 and A in Q2 (reverse
common ratio).
responses to Q1 when it is made in a pairwise choice (P1) and when it is embedded in
a choice list (line 11 in L1), can be attributed to the subject pool of mTurk workers.
We therefore decided to check whether there exists a signicant dierence between
behavior of students (which comprise the standard subject pool used in economics
experiments) and mTurk workers.
Subjects were recruited through the UBC's Economics department subject pool
using ORSEE (Greiner, 2015) and oered the opportunity to participate in an online
experiment in which they would be paid by online money transfer.
19
There was no
show-up fee, which was disclosed to subjects in advance according UBC's BREB
requirement.
Our expectation was that students who are willing to sign up for an
experiment with no show-up fee may be less sensitive to the certainty of a reward
compared to a typical student subject. This possible selection bias could attenuate
our results. Since student subjects tend to be paid much higher amounts per hour
than mTurk workers typically earn, we scaled up the payos to $13, $10, and $0
from $4, $3, and $0. Table 6 presents our ndings in the two treatments: Q1 under
pairwise choice (P1), and Q1 in a choice list (L1).
The students' behavior demonstrates exactly the same pattern as the data from
mTurk workers - students are more likely to choose the sure payment in a pairwise
choice task than when the choice is embedded in a choice list. Students' responses to
19All major Canadian banks have a $10 minimum transfer.
ELICITING RISK PREFERENCES USING CHOICE LISTS
26
Table 6. mTurk workers vs. student subjects
Pairwise Choice
Choice List
mTurk
Students
mTurk
Students
23%
33%
45%
44%
n
81
27
355
27
Treatments
All P
Q1
All L,S,O,A
Fraction choosing the riskier option
Q1 in pairwise choice were not signicantly dierent from mTurk workers' choices in
Q1 under pairwise choice (p
= .32,
exact test), nor were students' responses to Q1 in
the list signicantly dierent from turkers' responses to Q1 under choice list (p
> .99,
exact test).
Appendix C. Review of experiments on Mechanical Turk
As a large online labor market, mTurk provides a convenient way to recruit and pay
subjects over the internet. It allows researchers to economize on costs and experiment
on a dierent population from undergraduates, and has been advocated as a platform
for recruiting subjects by psychologists studying judgement and decision-making (Mason and Suri (2011), Paolacci, Chandler, and Ipeirotis (2010), Buhrmester, Kwang,
and Gosling (2011)), political scientists (Berinsky, Huber, Lenz, et al., 2012), and
economists (Horton, Rand, and Zeckhauser, 2011). A potential downside of running
experiments on mTurk is that subjects complete the experiment from their home
computer, and not in a controlled lab environment, making it dicult to know for
sure who the subjects really are and how much attention they are paying to the tasks.
Paolacci, Chandler, and Ipeirotis (2010) nd that the population of US-based turkers
who participate in experiments is heterogeneous and is more representative of the
US population than typical undergraduate samples, and that turkers pay as much
attention to experimental tasks as undergraduates in a lab. Paolacci, Chandler, and
Ipeirotis (2010) and Horton, Rand, and Zeckhauser (2011) show that some standard
experimental results in the judgement and decision-making literature can be qualitatively and quantitatively replicated using turkers.
Our paper also replicates our
result with a set of student subjects.
Freeman:
Department
of
david_freeman@sfu.ca. Web:
Economics,
Simon
Fraser
University.
e-mail:
http://www.sfu.ca/~dfa19/
Halevy: Vancouver School of Economics, University of British Columbia. e-mail:
yoram.halevy@ubc.ca. Web:
http://economics.ubc.ca/yhalevy/
ELICITING RISK PREFERENCES USING CHOICE LISTS
Kneeland:
Department
of
t.kneeland@ucl.ac.uk. Web:
Economics,
University
College
http://terri.microeconomics.ca/
London.
27
e-mail: