Pretesting refers to the testing of the questionnaire on a small sample of respondents to identify and eliminate potential problems. Even the best questionnaire can be improved by pretesting. As a general rule, a questionnaire should not be used in the field su vey without adequate pretesting. A pretest should be extensive, as illustrated by the Census 2000 questionnaire in the opening example. All aspects of the questionnaire should be tested, including question content, wording, sequence, form and layout, question difi1culty, and instructions. The respondents in the pretest should be similar to those who will be included in the actual survey in terms of background characteristics, familiarity with the topic, and attitudes and behaviors of interest.38 In other words, respondents for the pretest and for the actual survey should be drawn from the same population.
Pretests are best done by personal interviews, even if the actual survey is to be conducted by mail, telephone, or electronic means, because interviewers can observe respondents’ reactions and attitudes. After the necessary changes have been made, another pretest could be conducted by mail, telephone, or electronic means if those methods are to be used in the actual survey. The latter pretests should reveal problems peculiar to the interviewing method. To the extent possible, a pretest should. involve administering the questionnaire in an environment and context similar to that of the actual survey.
A variety of interviewers should be used for pretests. The project director, the researcher who developed the questionnaire, and other key members of the research team should conduct some pretest interviews. This will give them a good feel for potential problems and the nature of the .:. expected data. Regular interviewers, however, should conduct most of the pretest interviews. It is good practice to employ both experienced and new interviewers. The experienced interviewers can easily perceive uneasiness, confusion, and resistance in the respondents. New interviewers can help the researcher identify interviewer-related problems. Ordinarily, the pretest sample size is small, varying from 15 to 30 respondents for the initial testing, depending on the heterogeneity of the target population. The sample size can increase substantially if the pretesting involves several stages or waves.
Protocol analysis and debriefing are two commonly used procedures in pretesting. In protocol analysis, the respondent is asked to “think aloud” while answering the questionnaire. Typically, the respondent’s remarks are tape-recorded and analyzed to determine the reactions invoked by different parts of the questionnaire. Debriefing occurs after the questionnaire has been completed. Respondents are told that the questionnaire they just completed was a pretest and the objectives of pretesting are described to them. They are then asked to describe the meaning of each question, to explain their answers, and to state any problems they encountered while answering the questionnaire.
Computer and Internet Questionnaire Construction
Software is available for designing questionnaires administered over the Internet or other modes (e.g., telephone. personal interviews, or mail). Although we describe the use of the software for constructing Internet questionnaires, the functions are essentially similar for questionnaires constructed by other modes. The software will help develop and disseminate the questionnaire, and, in many cases, retrieve and analyze the collected data, and prepare a report. The software can automatically perform a variety of tasks such as:
such as open ended. multiple choice. scales. dichotomous questions. and so forth. Moreover. one can use buttons. drop-down boxes (closed position or open position). check boxes. or open-ended scrolling text boxes. Question Libraries. The user can select predefined questions or save questions used often In the question library. For example. the question library may contain predefined questions for measuring satisfaction. purchase intention. and other commonly used _~/
constructs in marketing. -Questionnaire Appearance. The user can select the background color and graph~~ questionnaire from a range of available templates or create a customized template using
the template manager. Preview. You can preview the questionnaire as it is being-developed to examine the content. interactivity, type of questions. and baekground design and make any changes that may be needed Publish. This user can create the HT~L questionnaire. post-it to a unique Web page. create a database to collecrthe-data on the hosting server. and obtaine a unique URL to which respondents can be directed.
Notification. The user can create. personalize. send. and track e-mail-based invitations to participate in the survey.
As each respondent completes the survey. the data are transferred over the Web to the data file on the host server. The data can be downloaded and analyzed at any time. even when the survey is running. Thus. results can be examined in real-time. Some commonly used questionnaire With this book you have access to which will enable you to electronically design questionnaires.
Several Web sites allow users to create and file their own questionnaires for free. CreareSurvey allows anyone to create and administer online surveys to whomever they want. It distributes the survey. monitors participation and participants. and then collects and analyzes the data. all for free. because it is sponsored by Web advertising. However. CreateSurvey docs not provide respondents. Users do this at their discretion. For instance. they can create a Web page and have the survey as a link from the Web page or send out an e-mail with the link asking people to participate in the survey. Another Web-based service is.
A number of Web site~used in universities offer valuableresources, such as scales and question libraries. for constructing questionnaires. Some helpful sites include the Interuniversity Consortium for Political and Social Research at the University of Michigan ); the Roper Center at the University of Connecticut ); the Survey Research Library at Florida State University and the Odum Institute. which houses the Louis Harris Data Center at the University of North Carolina. Chapel Hill
The questionnaire or research instrument should be adapted to the specific cultural environment and should not be biased in terms of anyone culture. This requires careful attention to each step of the questionnaire design process. The information needed should be clearly specified. It is important to take into account any differences in underlying consumer behavior, decisionmaking process, psychographic, lifestyle, and demographic variables. In the context of demographic characteristics, information on marital status, education, household size, occupation, income, and dwelling unit may have to be specified differently for different countries, as these variables may not be directly comparable across countries. For example, household definition and size vary greatly, given the extended family structure in some countries and the practice of two or even three families living under the same roof
The use of unstructured or open-ended questions may be desirable if the researcher lacks knowledge about the determinants of response in other countries. Unstructured questions also reduce cultural bias, because they do not impose any response alternatives. However, unstructured questions are more affected by differences in educational levels than structured questions. They should be used with caution in countries with high illiteracy rates. Unstructured and structured questions can be employed in a complementary way to provide rich insights, as in the following example.
Ethics in Marketing Research
Several ethical issues related to the researcher-respondent relationship and the researcher-client relationship may have to be addressed in questionnaire design. Of particular concern are the use of overly long questionnaires, asking sensitive questions, combining questions df more than one client in the same questionnaire or survey (piggybacking), and deliberately biasing the questionnaire
An important researcher-client issue is piggybacking, which occurs when a questionnaire contains questions pertaining to more than one client. This is often done in omnibus panels (see Chapters 3 and 4) that different clients can use to field their questions. Piggybacking can substantially reduce costs and can be a good way for clients to collect primary da a they would not be able to afford otherwise. In these cases, all clients must be aware of and consent to the arrangement. Unfortunately, piggybacking is sometimes used without the client’s knowledge for the sole purpose of increasing the research firm’s profit. This is unethicalFinally, the researcher has the ethical responsibility of designing the questionnaire so as to obtain the required information in an unbiased manner. Deliberately biasing the questionnaire in – a desired direction-for example, by asking leading questions-cannot be condoned. In deciding the question structure, the most appropriate rather than the most convenient option should be adopted, as illustrated by the next example. Also, the questionnaire should be thoroughly pretested before fieldwork begins, or an ethical breach has occurred