The questionnaire method is a widely-used tool for researchers in many fields to collect information. It is used especially in marketing or management studies.
There are two kinds of questions: single response questions and multiple responses questions. The analyses of multiple responses questions are not as established in depth as those for single response questions. Approaches of analyzing multiple re-sponses questions have been lacking until recently. Umesh (1995) first discussed the problem of analyzing multiple responses questions. Loughin and Scherer (1998), Decady and Thomas (1999) and Bilder, Loughin and Nettleton (2000) propose sev-eral methods for testing marginal independence between a single response question and a multiple responses question. Agresti and Liu (1999,2001) discuss the mod-eling of multiple responses questions. These studies mainly focus on the analysis of the dependence between a single response question and a multiple responses question. However, most researchers are also interested in ranking the responses in a question according to the probabilities of responses being chosen. In fact, the ranking responses problem may be the primary issue in the study of a survey.
Wang (2008a) proposed several approaches to solve this problem. However, these methodologies are derived under the frequestist setup, which cannot be adopted in the Bayesian framework. In real applications, empirical information may exist for the probabilities of responses being chosen. Related applications can refer to Pammer, Fong and Arnold (2000), etc. An appropriate methodology which combines the current data with the past information can provide a more ob-jective ranking strategy than an approach based only on current data. Thus, this study proposes several methods for ranking the responses in a multiple responses question under the Bayesian framework. The methodologies are an extension of the methods proposed in Muller, Parmigiani, Robert and Rousseau (2004). More details about Bayesian multiple testing and applications are discussed by Gopalan and Berry (1998), Do et al. (2005), Gonen, Westfall, Johnson (2003), Scott and
Berger (2006), Muller, Parmigiani and Rice (2007) and Scott (2009).
A related study about Bayesian ranking was carried out by Berger and Deely (2008). Their approach is to rank the items based on the posterior probability of the null hypothesis or Bayes factor. Although the methodology provides a rule for ranking, it does not set up the error tolerance. In the methods used in this study, the statistic used for ranking is similar to the one proposed by Berger and Deely (2008). Furthermore, we also propose the FDR criterion to measure the testing error. In the Bayesian framework, the conventional approach does not associate a criterion to set up the error tolerance. Based on Muller’s approach, we can control the testing error within a tolerance level. From this viewpoint, using the Bayesian FDR approach to rank responses is more informative and useful than the conventional approach.
In addition, the Berger and Deelys’ approach cannot directly be applied to analyze multiple responses questions. In this study, we clearly illustrate the use of the Bayesian model for analyzing multiple responses questions and derive the exact and approximate Bayes estimator forms. The proposed method can provide a convenient way for researchers to directly adopt the formulas for ranking the responses for multiple responses questions.
First, we use the example described in Wang (2008a) to illustrate the problem.
A company is designing a marketing survey to help develop an insect killer. The researchers list several factors, including high quality, price, packaging and smell which could affect the sales market. Thus, the researchers want to know the rank of significance of these factors such that they can design a product with lower cost and higher profit. To obtain the data, a group of individuals are surveyed about purchasing an insect killer. They are asked to fill out questionnaires which list all the questions that addressed to each respondent. The following is the multiple responses question in the questionnaire:
Question 1: Which factors are important to you when considering the purchase of an indoor insect killer ? (1) price (2) high quality (3) packaging (4) smell
(5) others.
In this multiple responses question, there are a total of 25 − 1 = 31 kinds of possible answers because we exclude the case which respondents do not select any response. The 31 random variables constitute a multinomial distribution with multinomial proportions p ∈ P = {pi1i2i3i4i5, ij = 0 or 1 and 0 <
5
P
j=1
ij ≤ 5 }, where ij cannot be simultaneously equal to 0. Note that the requirement of a multiple responses question is that at least one response is selected. This is not equivalent to a true-false question with five items. If we allow respondents not to select any item or to select all items, it would be equivalent to the five true-false items question. The method developed in this study can extend to this situation.
If we consider the parameter space under the frequentist framework instead of the Bayesian framework. Wang (2008a) provides examples showing that the conventional testing approaches do not posses the property of ranking consistency.
This property is a reasonable criterion to reflect the validity of the testing approach.
Under the frequentist framework, it is still unknown if a satisfactory approach exists to ranking responses with the property of ranking consistency. In this study, in addition to proposing a ranking approach under the Bayesian framework, a Bayesian ranking consistency property is introduced and the proposed method is shown to be Bayesian ranking consistent.
In the Bayesian framework, assume that we have prior information on the pa-rameter space P and we rank the responses based on a survey study under this prior information. This problem is related to the usual Bayesian multiple testing problem if we consider a single response question. However, the application is more complicated when analyzing multiple responses questions. Muller et al. (2004) pro-posed several criteria for the Bayesian multiple testing. Miranda-Moreno, Labbe and Fu (2007) applied the methods to hotspot identification in an engineering study. Wang (2008b) carried out a related study estimating the proportions in a multinomial distribution. In this paper, we investigate these Bayesian multiple testing procedures and extend the approaches to rank the responses for multiple
responses questions.
The paper is organized as follows. In Section 2, we describe a Bayesian model for multiple responses responses. Section 3 proposes several Bayesian multiple testing procedures for testing an order of the responses are proposed in Section 3.
In Section 4, a ranking criterion is proposed to rank the responses. In addition, the Bayesian multiple testing procedures discussed in Section 3 are shown to be con-sistent. In Section 5, we present simulation studies to compare the rejection rates of the methodologies and appropriate false discovery rate tolerances for different testing procedures. Finally, Section 6 provides a data example which is ranking inconsistent under the frequentist framework, but is ranking consistent under the Bayesian framework. Finally, a conclusion is given in Section 7.