{'fraction_non_alphanumeric': 0.023475024015369838, 'fraction_numerical': 0.009185878962536023, 'mean_word_length': 5.183961388527938, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Workshop Synthesis: Stated Preference Surveys and Experimental Design, an Audit of the Journey so far and Future Research Perspectives

This paper is a synthesis of the discussions and ideas that were generated during the workshop on “Stated preference surveys and experimental design” at the 2014 Travel Survey Methods Conference in Leura (Australia). The workshop addressed the challenges related to the design and implementation of stated preference surveys as a way to capture richer behavioural information on the preferences of individuals and groups. The discussion began by reviewing the current state of stated preference surveys and whether and what we have been doing correctly. We then analysed the areas where improvements are still needed, how we can achieve them, and some pros and cons of each improvement. © Elsevier B.V. Peer-review of International Steering Committee for Transport Survey Conferences ISCTSC.


Introduction
Stated preference (SP) or stated choice (SC) surveys have been extensively used in the last decades in many different fields such as marketing, transport, health economics, agricultural and environmental economics. There now exists a very rich literature showing the ability of SP studies to elicit behavioural responses and to allow identification through estimation of the preferences of a single individual or a group. The ability of SP surveys to support identification of individual preferences in a context that efficiently reduces the cognitive effort and fatigue of the respondent has made them a dominant data paradigm to study individuals' behavioural market decisions.
One the foremost problems generally recognised in SP surveys is that although they have the capability through careful design to mimic the real world, this does not ensure that such experiments represent it (lack of realism). A similar comment could be made about revealed preference (RP) surveys which have a number of challenges associated with non-chosen alternatives and measurement error more generally. The issue of realism has been predominantly related to the use of SP data in prediction, where the alternative specific constants and scale need to be adjusted to the real market shares or some assumption needs to be made to forecast new alternatives, not currently available in the market (Cherchi and Ortúzar, 2006;Glerum et al., 2013. When the focus of SP studies is on the derivation of marginal rates of substitution between attributes (i.e., willingness to pay measures) their validity is generally supported although the matter of hypothetical bias still remains; the concern exists however when researchers use SP data almost unconditionally. The use of such data in deriving (direct and cross) demand elasticities of the alternatives is somewhat more problematic since it is calculated using knowledge of predicted choice probabilities, and unless the SP model has been calibrated to reproduce true market shares, the elasticities may be unreliable.
However, the question to what extent the preferences elicited in SP surveys reflect real market observed preferences of the respondent, or to what extent are we manipulating these results, has been raised on a number of occasions, igniting renewed interest in the role of RP data. Problems of reality (or far from reality) are indeed present in both RP and SP data, for different reasons (at least in transport and some other contexts). Some shrewdness to improve realism applies equally to both types of data. For example, in both types of data, realism can be jeopardised by contacting respondents who are not the right persons to interview (a very real issue in, for example, freight studies where the driver of a truck may have some ownership of the travel time but not the travel cost), by not precisely defining the context or by using attributes not measured correctly, such as for example disregarding trip frequency when measuring travel costs. These are all well-known problems, and there are several studies where these issues have been carefully accounted for (e.g., Hensher and Bradley, 1993;Hensher and Raimond, 1995;Cherchi and Ortúzar, 2002;Iragüen and Ortúzar, 2004;Ehreke et al. 2014).
The issue of realism (linked to hypothetical bias, i.e. "the potential error induced by not confronting the individual with an actual situation" Schulze et al. (1981); see also Hensher (2010)) is especially relevant because the SP survey is based on constructed hypothetical scenarios designed to elicit individuals' preferences for specific attributes. The scenarios consist of a set of alternatives defined as a combination of a number of attributes at specific levels. Alternatives, attributes and attributes levels, as well as their combinations presented to the respondents, are defined in advance by the analyst. Keeping the balance between realism (i.e., relevance) and complexity represents probably the major challenge in building a stated preference survey. This is not a new problem. A "to do" list to achieve realism containing complexity is reported in Ortúzar and Willumsen (2011, page 114). However, this is a serious problem that still affects many SP experiments. During the workshop, several points included in that list and new points were discussed in the light of the most recent experiences and advances in the field.

Tasks complexity and respondents engagement
In order to ensure realism and reduce potential hypothetical bias, analysts may need to build rather complex survey tasks which respondents are asked to process in a short time, potentially exerting high burden and risking damaging the quality of response. Individuals have limitations in their capacity to process information, and are not always willing to invest the required degree of effort in evaluating alternatives. When presented with a complex task, it is then likely that they show disengagement, adopting simplifying strategies to reduce the mental effort required to solve the problem. On the other hand, simplifying the survey tasks to reduce the cognitive burden for respondents is also risky. Simplified survey tasks are often too simplistic and they can be seemingly perceived as unrealistic by the respondents, leading to problems with respondents' engagement, or respondents choosing based on other attributes not included in the design. The risk is that a design with too little alternatives or attributes or levels might produce more errors than more "complex" designs.
There are many different experiences and recommendations from the literature on how complex an SP survey should be (Arentze et al., 2003;Hensher, 2004Hensher, , 2006Caussade et al., 2005). Hensher (2004) promotes a position that 'relevance' is what matters and this is likely to result in a large range (i.e., heterogeneous) of 'complexity' as perceived by a survey analyst. The risk that the analyst will impose their own (biased) views on what is complex is a concern, resulting in a study bias that is artificially created and has nothing to do with preference revelation (see also the discussion in Hensher, 2015). The acceptable level of complexity varies among contexts and can also depend on cultural aspects (Rose et al., 2009). Some authors have reported a consistently positive experience with complex designs that include at least 15 attributes, and in some cases more than 40 choice sets (Brazell and Louviere, 1997;Stopher and Hensher, 2000;Hensher, 2001), whereas some studies recommend a number not higher than six (Ortúzar and Willumsen (2011, page 101), because there are some evidence that the number of choice situations to be evaluated led to an increase in the error term variance because the fatigue (Bradley and Daly, 1994;Ortúzar et al., 2000). Hess et al. (2012) suggest otherwise and conclude that "Using a comprehensive testing framework employing both multinomial logit and mixed logit structures, we provide strong evidence that the concerns about fatigue in the literature are possibly overstated, with no clear decreasing trend in scale across choice tasks in any of our studies." The appropriate answer has to be context dependent, and it is something that should always be investigated and carefully tested prior to starting any new SP study. Swait and Adamowicz (1996), for example, found that choice task complexity and cumulative burden affect mainly reliability. More recently, Petrik et al. (2014) used a control scenario to detect potentially disengaged respondents. The control scenario, which consists of replicating, at the end of the survey, one of the scenarios previously presented to each individual, increases the complexity of the survey and can be used only for relatively simple surveys. Moreover, other effects, such as learning, can explain differences in the control scenario. Mazaheri, et al. (2014) grouped the alternatives according to related attributes and select one alternative as a representative alternative for each group.  implement an alternative stated choice survey strategy where choice tasks are constructed such that the first alternative presented in the previous task is removed from the set shown, with the remaining alternatives shifted to fill the resulting gap, and a replacement alternative drawn from the candidate set inserted. This technique allows for the inclusion of a large number of attributes, whilst at the same time potentially reducing the cognitive load placed on respondents undertaking the experiment.
A typical way to minimise the cognitive effort required by respondents is the use of blocks to reduce the number of choice scenarios. However, recently  questioned the use of blocking techniques because the dispersion of survey questions will be perfectly confounded with any detected response heterogeneity, hence making it impossible to determine whether respondents truly have different preferences or error variances, or whether the heterogeneity is an artefact of the survey questions asked. They propose an alternative method of constructing experimental designs that involves repeating a set of questions across individuals, which allow testing for possible demand artefacts in stated choice experiments.

Unfeasible, implausible or dominated alternatives
The potential lack of realism is one of the most challenging elements of SP experiments, induced in part by poor experimental design, such as designs that include unfeasible, implausible or dominated alternatives. Using designs where the attributes are pivoted around the real attribute levels experienced by respondents (as typical, for example, in studies where joint models with revealed and stated preference data are estimated; see e.g., Ben-Akiva and Morikawa, 1990;Swait et al., 1994;Bradley and Daly, 1997;Louviere et al., 2000, chapter 8;Brownstone et al., 2000;Cherchi and Ortúzar. 2002;Bhat and Castelar, 2002;Börjesson, 2008) can reduce the risk of alternatives as packages of attributes that are confusing and lacking meaning.
It is also important to ensure that the combination of attribute levels presented is realistic. With the traditional orthogonal designs, this task was mainly undertaken a posterior, checking the tasks after the design was built. In the class of efficient designs, it is possible to incorporate behavioural constraints a priori, for example, imposing a condition that an attribute can take a given level only if another attribute is greater than a certain value (these are sometimes referred to as nested designs). Research by Collins et al. (2014) has developed ways to account for constraints when building experimental designs, which involve some amount of trading against statistical efficiency. But this is a small price to pay for increased behavioural relevance. They analysed various rule structures for specifying the constraints and proposed two new algorithms that can handle these constraints effectively. Which algorithm performs best depends on the type of constraints specified; however a convergence solution is not guaranteed. The general message is that it is better to lose some statistical efficiency if it results in greater experimental realism.
Realism is also improved by avoiding dominant alternatives. Since most experimental designs are computergenerated, it is easy to overlook choice tasks that contain a dominant alternative ). There are currently no techniques that allow avoiding a priori for dominant alternatives, with the solution methods usually involving eliminating problem design features a posterior, after the design is built. Any a posterior variation of the SP design violates the design's statistical properties, but this is appropriate if it increases realism. Importantly we know that the more attributes and levels there are in a choice experiment design, the less likely that dominant alternatives will exist, something that does tend to pervade relatively simple designs. Using an illustrative simple case with two attributes,  show that data sets that contain strictly dominant alternatives (i.e., alternatives always being chosen by a respondent in every offered choice scenario) are problematic, as they may lead to substantially biased parameter estimates. They proposed two different ways to mitigate the problem, namely (i) automatic detection and removal of problematic choice tasks, and (ii) adapting the discrete choice model to ensure that the parameter estimates are less sensitive to scaling issues.
Another potential problem related to the definition of attributes in the SP design is the inability to differentiate between the impacts of the attributes per se (i.e., the average impact of an attribute in the evaluation of an alternative), and the impact of the attribute levels shown in the experiment (i.e., the variation in impact due to the range of the attribute in an experimental setting). Recent literature has suggested that the best-worst experiment form nicely allows for this differentiation (e.g., Finn and Louviere, 1992;Marley and Flynn, 2015;Balbontìn et al., 2015). Best-Worst experiments can be also used to avoid redundancy in the offered set of 'statistically designed' alternatives. For example, Hensher and Ho (2014) found empirical evidence to support a simple best-worst binary form, in contrast to other ways of preserving the (constructed) full rank choice set, concluding that this appears to represent the most appropriate processing rule in choice making which can, ex post, be used to jointly estimate process and outcome choices.

Use of pictures
The use of pictures has also been discussed as a way to improve realism in SP experiments. "Image theory'' suggests that using images instead of words allows a study to better capture the psychological or distorted representation of objective reality residing and existing in the mind of the individual (Myers, 1968). From a theoretical point of view, the use of images is then recommended, but its use in practice can be risky because pictures can convey too much information that, if not controlled through some very explicit conditions, might generate responses that cannot be confidently associated with the attributes that are being assessed. Confounding outcomes are thus quite likely with a risk of ambiguity in the interpretation of the choice process. Hence if images are to be used, they need to be controlled; for example, real images modified on the computer in order to leave only the elements of relevance for the SP experiment, eliminating all the elements that can distract from the main focus avoiding confounding the attributes of interest with features we do not want to measure. Good examples of this can be found in Hensher and Mulley (2015), Galilea and Ortúzar (2005), Sillano and Ortúzar (2005), Strazzera et al. (2010), Iglesias et al. (2012), Kamargianni and Polydoropoulou (2013), Sottile et al. (2014). The majority of these studies report a positive experience. However, there is also evidence where no differences were found between using images or textual information or even worse where using images induced mistakes in the responses (Stopher et al., 2003;Alsnih et al., 2004).
The use of images requires more research since there is an intuitive sense that visualisation is value adding in respect to behavioural relevance and reducing hypothetical bias. A context where (a sequence of) images (or possibly videos) can be very useful is that involving public transport and the response to crowding. The perception of crowding is a tricky notion to capture, since it has both a subjective and an objective dimension. One can show respondents pictures of crowding on buses and trains that depict how many people are standing and how many are sitting; however this may only represent a single time point in the journey (typically defined as when one enters the train or bus), and yet this can vary along the trip and will influence a respondent's perception and assessment of the relevance of this attribute in preference revelation and consequent choice making. We also do not know why some people prefer to stand (possibly to alight quicker). What would be interesting is to show how the crowding (however defined) varies as the trip progresses so one gets a better sense of its true influence, unless it can be shown that decisions are only based on crowding levels upon entering a vehicle or carriage. Results from images can also be used to infer whether attributes are attended to or not. i.e., which attributes people are evaluating, though it is important in this case to avoid confirmation bias.
It has been suggested by a number of studies that the use of the available technology, such as eyes tracking, virtual reality and simulators, represents an important direction to improve the use of images in SP experiments (e.g., Meißner and Decker, 2010;Campbell et al., 2013).

Qualitative work
There was substantial discussion on ways in which the realism of SP surveys could be improved using qualitative research as an ex ante approach to teasing out relevant attributes, processing capabilities, and strategies of candidate respondents. Traditionally, qualitative work was carried out before designing an SP experiment, in the form of focus groups or other forms of group or in-depth interviews (Louviere et al., 2000, page 257). Such methods can be used at the preliminary design stage using (excel) mock ups of an SP instrument as a way to gather information from a small sample of individuals on matters such as the comprehensive and comprehendable nature of the survey instrument, including the relevance of attributes and their levels as well as the length and overall complexity of the tasks.
Qualitative work is also of importance to understand how people evaluate images. This work can be carried out during the focus group to help defining the best way to present the images or during (or immediately after) the SP experiment in the form of supplementary questions to better understand a posterior how people have used and evaluated the information conveyed in the images. If asking qualitative questions during the SP experiment might affect the evaluation of the subsequent tasks, it is then preferable to ask questions at the end of the SP survey.
Supplementary questions have often been used to better understand how respondents process the information in choice experiments. The goal of the SP survey is to elicit the behavioural responses; however, a growing number of studies and authors have recognised the need to understand and account for the underling processing rules (or heuristics) that respondents use in assessing choice experiments and making a choice (including ranking the offered alternatives). A growing body of literature in different fields, including transport, (see Hensher, 2014 for a broader review) has now consistently argued and demonstrated that individuals use different rules (or heuristics) to assist them in making choices, and it is important to identify the right choice response mechanism.
The great majority of choice modelling research has taken, as a maintained assumption, the behavioural position that individuals are fully compensatory in the way that they assess and trade-off attributes in choice making and that in circumstances where the analyst imposes a set of attributes to evaluate, as is common in stated choice experiments, it is commonly assumed that all attributes are relevant in choice making. One consequence of these assumptions is the resulting view that studies that require individuals to assess an increasing number of attributes impose growing complexity and cognitive burden that risks the loss of identifiable comprehendible settings of choice making.
One criticism of this position is that individual choosers are very heterogeneous in the way that they make choices in real and hypothetical markets, and that they draw on rules (or heuristics) to assist them in making choices. These rules may vary by contexts typified, for example, by habit or variety seeking behaviour. They may also reflect the fact that individuals bring to the choice making setting their views on what attributes are the key drivers of specific choice outcomes, and that these attributes may or may not be included in the set defined by an analyst. There is a real risk that the analyst may self-impose their own views (or prejudices) or even those of a client funding a study, on the number of attributes and alternatives that are deemed comprehendible to a sample of respondents in a survey.
These presumptions have been questioned in the broader literature on heuristics and decision making that has evolved in a number of literatures, notably, psychology, economics and marketing; however the migration of ideas from this literature, which we refer to as process heuristics, has been slow to influence the way that discrete choice modelling has been represented. This is changing now, with a growing number of studies questioning the standard fully compensatory choice paradigm.
An example of a process heuristic that have gained a lot of traction in a number of disciplines (especially transport and environmental economics) is attribute non-attendance (ANA), which recognises that individuals often ignore one or more attributes of the alternatives when making their choices (see Collins and Hensher, 2015 for an updated review). The reasons for non-attending to an attribute are many including the absence of relevancy of the attributes and disengagement, as discussed previously, to true indifference to the attribute. Alemu et al. (2013) provide an interesting example of supplementary questions used to detect the nature of attributes non-attendance. It was suggested that supplementary qualitative questions can also be used to reveal if people are telling the 'truth', and hence test the level of engagement and believability of the answers (Morrison et al., 2002).
Whatever is the scope of supplementary qualitative questions, particular attention needs to be pay to the way questions are asked in order to be able to correctly elicit what you would like to measure.

Interaction within groups
In the majority of stated choice surveys, the respondent (a single agent) is asked to evaluate the scenarios presented. However, for many type of choices the preferences underlying decisions may be affected/depend upon the preferences of other people (typically, but not only, other members of the household). It is then important to understand to what extent the preferences elicited in SP surveys reflect the real preferences of the respondent. This is particularly important for choices, such as vehicle ownership or residential choice, but might be true for simpler everyday mode choices (e.g., the use of the family car is often the result of a bargaining process among the activity schedules of different household members). Some studies have included a simple question to measure if the choices stated in the experiment were taken alone or involved a level of bargaining with other people. Although this is useful information that supports maintaining the simplicity of a design, in order to identify individual preferences correctly, group decisions may need to be measured explicitly. There are few examples in the literature where the interaction within groups is taken into account when eliciting individual's preferences. Some studies have attempted to do this by asking individuals to answer the SP experiment twice: first alone and then with the group (usually the partner) (Beharry-Borg et al., 2009), or by building a two-step SP experiment, where in the first step the scenarios imply a change only for one member of the family, while in the second they imply a change for several members of the family (Swärdh and Algers, 2009), or by simulating a bargaining situation (Dosman and Adamowitz, 2005;Hensher et al., 2008). Other studies have included an attribute in the design to measure the effect of friends, relatives or peers' choices on individual preferences (Rasouli and Timmermans, 2013). The problem in this case is to understand if the market share is endogenous or exogenous, and then modelling this properly.
Most advanced studies in this line of research use interactive SP experiments, where each individual provides the evaluation of a set of alternatives several times, every time after being informed about the choice of others (e.g., Beck et al. 2013Beck et al. , 2014. The process continues until both parties agree (or not). These approaches are important to measure individual preferences correctly, but they are also extremely time and cost consuming. Research along these lines is still rare, though crucial. Simulators can be particularly useful to increase realism in testing competition among respondents.

Adaptive/dynamic SP
Another important direction where major research is needed is the use of adaptive and dynamic SP surveys. Adaptive Stated Preference consists in adjusting the levels of the attributes presented in each choice task based on the answers provided in the previous tasks. The adjustment can be across respondents, if the attributes levels are adjusted based on the preferences elicited in previous respondents, or within respondent, if the attributes levels are adjusted based on the preferences elicited in the previous tasks presented to the same respondent. Adaptive designs are particularly suitable for small samples, but might generate endogeneity issues.
The lack of knowledge of the product presented represents another problem that affects the ability of the SP experiment to reflect the real preferences of the respondent. Some SP surveys are used to identify individual preferences for products that do not currently exist in the market. However, individuals might have a misconception about the relevance of the new product and its characteristics and this might affect the measure of the individual preference that we obtain from SP data. Even if the product is not completely new, individuals are adaptive decision makers who learn through trial and error. Their preferences are then affected by (and hence might change with) the experience obtained from using or consuming a certain product. It is important, then, to study the dynamic evolution of individual preferences in order to try and measure correctly individual preferences. Jensen et al. (2013 and2014) set up a "long panel" survey, where an SP experiment was repeated twice, before and after individuals experienced an electric vehicle in real life during a three-month period. They found that individual preferences indeed change significantly after a real experience and that willingness to pay and demand elasticity for some characteristics change significantly, whether measured before or after the real experience. The use of panel SP before and after experience or real interventions represent an important direction of research to properly account dynamic evolution of individual preferences. This reinforces the findings of Hensher et al. (1997) that overt experience changes preferences. Hensher (2105) indicates that when the analyst assumes that all attributes offered in a choice experiment are relevant, there is an implicit sense that the respondent has complete familiarity or awareness of both the alternatives on offer and the context within which a choice is being made. A classic example where this fails is the debate on road pricing reform, which is substantively not understood by the majority of respondents, certainly in Australia. The value of identifying respondent awareness and/or familiarity seems appealing to behavioural study as a way of establishing the extent to which better information will improve awareness and familiarity in a way that might engender greater support for a reform package. This is a sensible position for studies interested in stakeholder buy in.

Improving the efficiency of an experimental design
Based on the work of Rose and Bliemer 2006;Rose and Bliemer, 2008;, in the last few years there has been a substantial amount of discussion in the literature on strategies to generate experimental designs. Although, in principle, any strategy can be used to build a design, some strategies are statistically more efficient than others, allowing for smaller choice sets and hence smaller sample sizes and/or less cognitive effort and possible fatigue for the respondents. This motivated the use of efficient design versus orthogonal fractional factorial designs, which were originally the most used strategy within the transport literature. As highlighted in recent literature (Louviere et al., 2008;Bliemer and Rose, 2011), statistical efficiency has an impact not only on the standard errors of the parameter estimates but it can also influence the scale of the model. Some concerns about the use of efficiency criteria are related with the consistency between the phenomenon we are able to measure with SP data (i.e., the prior assumptions required about model specification and parameters) and the phenomenon we are able to estimate using such data. Since, in SP experiments, the behaviour underlying the data is controlled by the modeller via the experimental design, an additional challenge is to build the experimental design such that the behaviour underlying the data is consistent with the phenomenon we attempt to estimate with the discrete choice model. Concerns related to the statistical efficiency measure adopted do not matter.  show that although the design principles in different methods may seem very different, all methods follow exactly the same theory, but merely make different assumptions on the efficiency criterion, model specification, assumed parameter priors, and attribute level constraints.

Conclusions
The workshop on "Stated preference surveys and experimental design" generated a lively discussion and many fruitful ideas. As highlighted in this synthesis, researchers in the field have made significant advances in key directions such as a better understanding of behavioural response and improved design efficiency. The availability of modern technology (like the internet or programs that optimise the design of an experiment) have facilitated the use of stated preferences, with more complex experiments, although at the potential risk to the quality of the survey in terms of behavioural representation. At the same time, the advances in econometric models have enabled us to account a posteriori for behaviour that cannot be elicited directly from the survey. The research needs that were identified during the workshop, and are discussed in this paper, can be summarised as follows: • Tasks complexity and respondents engagement • Unfeasible, implausible or dominated alternatives • Use of pictures • Qualitative work • Interaction within groups • Adaptive/dynamic SP • Improving the efficiency of an experimental design A key recommendation is that future research should aim at improving realism in surveys, revisiting known techniques (such as for example, qualitative surveys), exploring the adoption of the new technologies available (such as eye tracking, simulators, smart phones) and exploiting the knowledge available from other disciplines (such as psychology, neuro-science, and marketing). The message from the workshop was that "better to have excellent data with simple models, rather than poor data with excellent econometric models". Finally, it was also pointed out that it is not always necessary to collect new data; it might be relevant to revisit existing data sets and to re-analyse them as we gain better insights into how data might be further re-analysed and interpreted.

Online license transfer
All authors are required to complete the Procedia exclusive license transfer agreement before the article can be published, which they can do online. This transfer agreement enables Elsevier to protect the copyrighted material for the authors, but does not relinquish the authors' proprietary rights. The copyright transfer covers the exclusive rights to reproduce and distribute the article, including reprints, photographic reproductions, microfilm or any other reproductions of similar nature and translations. Authors are responsible for obtaining from the copyright holder, the permission to reproduce any figures for which copyright exists.