Instrumentation (I) refers to changes in the measuring instrument, in the observers, or ir I the scores themselves. Sometimes, measuring instruments are modified during the course of an ex periment. In the advertising experiment, ifa newly designed questionnaire was used to measure the ‘posttreatment attitudes, this could lead to variations in the re ponses obtained. Consider an experirnsnr in which dollar sales arc being measured before nnd after exposure to an in-store display (trea’lment). If there is a nonexperimental price change between 01 and °2, this results in a change in instrumentation because dollar ales will be measured using different unit prices. In this case, th’~ treatment effect (02 – 0t) could be attributed to Iichange in instrumentation.
Statistical regression (SR) effects occur when test units with extreme ~ scores move closer to the average score during the course of the experiment. In the advertising experiment, suppose that some respondents had either very favorable or very unfavorable attitudes. On post treatment measurement, their attitudes might have moved toward the ave’,rage, People’s attitudes change continuously. People with extreme attitudes have more room for change, so variation is more likely. This has a confounding effect on the experiment results, because the observed effect (change in attitude) may be attributable to statistical rf!regression rather than to the treatment (test commercial
Mortality (MO) refers to the loss units while the experiment is in progress. This happens for many reasons, such as test un;.ts refusing to continue in the experiment. Mortality confounds results because it is difficult to Determine if the lost test units would respond in the same manner to the treatments as those that remain. Consider again the merchandising display experiment. Suppose that during the course of the experiment, three stores in the new display treatment condition drop out. The researcher could not determine whether the average sales for the new display stores would have been ~Nigher or lower if these three stores had continued in the experiment.
The various categories of extraneous variables are not mutually exclusive. The)’ can occur jointly and also interact with each other. To illustrate, testing-e maturation-c rnortallty refers to a situation where, because of pretreatment measurement, the respondents’ beliefs and attitudes change over time and there is a differential loss of respondents from the various treatment groups.
Controlling Extraneous Variables
Extraneous variables represent alternative explanations of experimental results. They pose a serious threat to the internal and external validity of an experiment. Unless they are controlled for, they affect the dependent variable and thus confound the results. For this reason. they are also called confounding variables. There are four ways of controlling extraneous variables: randomization, matching, statistical control, and design control
Statistical control involves measuring the extraneous variables and adjusting for their effects through statistical analysis. This was illustrated in Table 7.2. which examined the relationship (association) between purchase of fashion clothing and education, controlling for the effect of income. More advanced statistical procedures. such as analysis of variance·(ANCOVA). ilre also available. In ANCOVA. the effects of the extraneous variable on the dependent variable are removed by an adjustment of the dependent variable’s mean value within each treatment condition. (ANCOVA is discussed in more detail
Experimenting with New Products
Controlled-distribution electronic te~t markets are used increasingly to conduct experimental research on new products. This method makes it possible to control for several extraneous factors mat affect new product performance and manipulate the variables of interest. It is possible 0ensure that a new product: (I,>obtains the right level of store acceptance and all commodity volume distribution, (2) is positioned in the correct aisle in each store, (3) receives the right number of facings on the shelf, (4) has the correct everyday price, (S) never has out-of-stock problems, and (6) obtains the planned level of trade promotion, display, and price features on the desired time schedule. Thus, a high degree of internal validity can be obtained.
A Classification of Experimental Designs
Experimental designs may be classified as experimental, true experimental, quasi-experimental, or statistical (Figure 7.1). Experimental designs do not employ randomization procedures to control for extraneous factors. Examples of these designs include the one-shot case study, the one-group pretest-posttest design, and the static group. In true experimental designs, the researcher can randomly assign test units and treatments to experimental groups. Included in this category are the pretest-posttest control group design, the posttest-only control group design, and the Solomon four-group design. Quasi-experimental designs result when the researcher is unable – to achieve full manipulation of scheduling or allocation of treatments to test units but can still apply part of the apparatus of true experimentation. Two such designs are time series and multiple time series designs. A statistical design is a series of basic experiments that allows for statistical control and analysis of external variables. The basic designs used in statistical designs include preexperimental, true experimental, and quasi-experimental. Statistical designs are classified on the basis of their characteristics and use. The important statistical designs include randomized block, Latin square, and factorial. These designs arc illustrated in the context of measuring the effectiveness of a test commercial for a department store.
One-Shot Case Study
A one-shot case study to measure the effectiveness of a test commercial for a department store. for example. Sears. would be conducted as follows. Telephone interviews are conducted with a national sample of respondents who report watching a particular TV program the previous night. The program selected is the one that contains the test (Sears) commercial (X). The dependent variables (Os) are unaided and aided recall. First. unaided recall is measured by asking the respondents whether they recall seeing a commercial for a department store. for example. “Do you recall seeing a commercial for a department store last night?” If they recall the test commercial. details about commercial content and execution are solicited. Respondents who do not recall the test commercial are asked about it specifically. for example. “00 you recall seeing a commercial for Sears last night?” (aided recall). The results of aided and unaided recall are compared to norm scores to develop an index for interpreting the scores
One-Group Pretest-Post test Design
A one-group pretest-post test design to measure the effectiveness of a test commercial for a department store. for example. Sears. would be implemented as follows. Respondents are recruited to central theater locations in different test cities. At the central location. respondents are first administered a personal interview to measure. among other things. attitudes toward the store. Sears (01)’ Then they watch a TV program containing the test (Sears) commercial (X). After viewing the TV program. the respondents are again administered a personal interview to measure attitudes toward the store. Sears (02)’ The effectiveness of the test commercial is measured
Static Group Design
The static group is a two-group experimental design. One group, called the experimental group (EG). is exposed to the treatment, and the other, called the control group (eG). is not. Measurements on both groups are made only after the treatment, and test units are not assigned at random. This design may be symbolically described as
A static group comparison to measure the effectiveness of a test commercial for a department store would be conducted as follows. Two groups of respondents would be recruited on the basis of convenience. Only the experimental group would be exposed to the TV program containing the test (Sears) commercial. Then. attitudes toward the department store (Sears) of both the experimental and control group respondents would be measured. The effectiveness of the test commercial would he measured as
Post test-Only Control Group
To measure the effectiveness of a test commercial for a department store. the post test-only control group design would be implemented as follows. A sample of respondents would.~$elected at random. The sample would be randomly split. with half the subjects forming the experimental group and the other half constituting the control group. Only the respondents in the experimental group wouldbe exposed to the TV program containing the test (Sears) commercial. Then. a questionnaire would be administered to both groups to obtain posttest measures on attitudes toward the department store (Sears), The difference in the altitudes of the experimental group and the control group would be used as a measure of the effectiveness of the test commercial
In this example. the researcher is not concerned with examining the changes in the-altitudes of individual respondents. When this information is desired. the Solomon four-group design should be considered. The Solomon four-group design overcomes the limitations of the pretest post test control group and posttest-only control group designs in that it explicitly controls for interactive testing effect. in addition to controlling for all the other extraneous variables (EV). However. this design has practical limitations: It is expensive and time-consuming to implement. Hence. it is not considered further
A quasi-experimental design results under the following conditions. First. the researcher can control when measurements are taken and on whom they are taken. Second. the researcher lacks control over the scheduling of the treatments and also is unable to expose test units to the treatments randomly.18 Quasi-experimental designs are useful because they can be used in cases when true experimentation cannot, and because they are quicker and less expensive. However. because full experimental control is lacking. the researcher must take into account the specific variables that are not controlled. Popular forms of quasi-experimental designs are time series and multiple time series designs
The major weakness of the time series design is the failure to control history. Another limitation is that the experiment may be affected by the interactive testing effect. because multiple measurements are being made on the test units. Nevertheless, time series designs are useful. The effectiveness of a test commercial (X) ‘may be examined by broadcasting the commercial a predetermined number of times and examining the data from a preexisting test panel. Although the marketer can control the scheduling of the test commercial, it is uncertain when or whether the panel members are exposed to it. The panel members’ purchases before. during, and after the campaign are examined to determine whether the test commercial has a short-term effect, a long-term effect. or no effect
Multiple Time Series Design
The multiple time series design is similar to the time series design except that another group of test units is added to serve as a control group. Symbolically, this design may be described as
If the control group is carefully selected. this design can be an improvement over the .simple time series experiment. The improvement lies in the ability to test the treatment effect twice: against the presentment measurements in the experimental group and against the control group ..To use the multiple time series design to assess the effectiveness of a commercial, the test panel example would be modified as follows. The test commercial would be shown in only a few of the test cities. Panel members in these cities would comprise the experimental group. Panel members in cities where the commercial was not shown would constitute the control group