Description
The purpose of this study is to estimate the extent (mean and range) of non-response bias in
online travel advertising conversion studies for 24 destinations located throughout the USA.
International Journal of Culture, Tourism and Hospitality Research
Non-response bias in internet-based advertising conversion studies
Sangwon Park Daniel R. Fesenmaier
Article information:
To cite this document:
Sangwon Park Daniel R. Fesenmaier, (2012),"Non-response bias in internet-based advertising conversion studies", International J ournal of
Culture, Tourism and Hospitality Research, Vol. 6 Iss 4 pp. 340 - 355
Permanent link to this document:http://dx.doi.org/10.1108/17506181211265077
Downloaded on: 24 January 2016, At: 22:20 (PT)
References: this document contains references to 71 other documents.
To copy this document: [email protected]
The fulltext of this document has been downloaded 394 times since 2012*
Users who downloaded this article also downloaded:
Toyin A. Clottey, Scott J . Grawe, (2014),"Non-response bias assessment in logistics survey research: use fewer tests?", International
J ournal of Physical Distribution & Logistics Management, Vol. 44 Iss 5 pp. 412-426http://dx.doi.org/10.1108/IJ PDLM-10-2012-0314
Ana Maria Soares, J osé Carlos Pinho, (2014),"Advertising in online social networks: the role of perceived enjoyment and social influence",
J ournal of Research in Interactive Marketing, Vol. 8 Iss 3 pp. 245-263http://dx.doi.org/10.1108/J RIM-08-2014-001
Swatee Sarangi, Shreya Shah, (2015),"Individuals, teams and organizations score with gamification: Tool can help to motivate employees
and boost performance", Human Resource Management International Digest, Vol. 23 Iss 4 pp. 24-27http://dx.doi.org/10.1108/
HRMID-05-2015-0074
Access to this document was granted through an Emerald subscription provided by emerald-srm:115632 []
For Authors
If you would like to write for this, or any other Emerald publication, then please use our Emerald for Authors service information about
how to choose which publication to write for and submission guidelines are available for all. Please visit www.emeraldinsight.com/
authors for more information.
About Emerald www.emeraldinsight.com
Emerald is a global publisher linking research and practice to the benefit of society. The company manages a portfolio of more than
290 journals and over 2,350 books and book series volumes, as well as providing an extensive range of online products and additional
customer resources and services.
Emerald is both COUNTER 4 and TRANSFER compliant. The organization is a partner of the Committee on Publication Ethics (COPE) and
also works with Portico and the LOCKSS initiative for digital archive preservation.
*Related content and download information correct at time of download.
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
Non-response bias in internet-based
advertising conversion studies
Sangwon Park and Daniel R. Fesenmaier
Abstract
Purpose – The purpose of this study is to estimate the extent (mean and range) of non-response bias in
online travel advertising conversion studies for 24 destinations located throughout the USA.
Design/methodology/approach – The method uses two weighting procedures (i.e. post strati?cation
and propensity score weighting) to estimate the extent of non-response bias by adjusting the estimates
provided by respondents to more closely represent the total target sample.
Findings – The results of this analysis clearly indicate that the use of unweighted data to estimate
advertising effectiveness may lead to substantial over estimation of conversion rates, but there is limited
‘‘bias’’ in the estimates of median visitor expenditures. The analyses also indicate that weighting
systems have substantially different impact on the estimates of conversion rates.
Research limitations/implications – First, the likelihood to answer a survey varies substantially
depending on the degree of the familiarity with the mode (i.e. paper, telephone versus internet). Second,
the competition-related variables (i.e. the number and competitiveness of alternative nearby
destinations) and various aspects of the campaign (i.e. amount of investment in a location) should be
considered.
Originality/value – This study of 24 different American tourism campaigns provides a useful
understanding in the nature (mean and range) of impact of non-response bias in tourism advertising
conversion studies. Additionally, where there is dif?culty obtaining a reference survey in the advertising
study, the two weighting methods used in this study are shown to be useful for assessing the errors in
response data, especially in the case of propensity score weighting, where the means to develop
multivariate-based weights is straightforward.
Keywords Tourism advertising, Advertising effectiveness, Conversion analysis, Non-response bias,
Surveys, Research methods
Paper type Research paper
Introduction
Tourism researchers have developed a variety of methods to assess the effectiveness of
advertising campaigns (Kim et al., 2005; McWilliams and Crompton, 1997; Siegel and
Ziff-Levine, 1990; Woodside and Ronkainen, 1984). The accuracy and reliability of these
approaches have been challenged in terms of sampling strategies and non-response bias
(Mok, 1990; Woodside and Ronkainen, 1984). Today, many tourism advertising studies use
online surveys instead of traditional mailing surveys due to the bene?ts associated with the
internet, including accessibility (Tierney, 2000), low cost (Tse, 1998), fast response (Weible
and Wallace, 1998), and veri?able delivery (James et al., 1995). With online surveys, one
source of sampling bias has largely been eliminated as online-based surveys can be sent to
the entire population rather than a sample of persons requesting travel information (Hwang
and Fesenmaier, 2004). Ellerbrock (1981) and Burke and Gitelson (1990) argue that
non-response bias may result in substantial overestimates of the effectiveness of the tourism
advertising (e.g. conversion rate and trip expenditure) because people who visit a
destination are more likely to respond to a travel survey, as compared to non-visitors. Thus,
PAGE 340
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012, pp. 340-355, Q Emerald Group Publishing Limited, ISSN 1750-6182 DOI 10.1108/17506181211265077
Sangwon Park is a Lecturer
in the School of Hospitality
& Tourism Management,
University of Surrey,
Guildford, UK. Daniel R.
Fesenmaier is a Professor in
the School of Tourism
& Hospitality Management,
Temple University,
Philadelphia, Pennsylvania,
USA.
Received March 2011
Revised June 2011
Accepted September 2011
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
some scholars suggest that the most effective approach to minimize non-response bias is by
increasing response rate. Others, however, argue that while increasing response rate is
important, this is not really practical due to the general patterns of low response rate in web
surveys (Best et al., 2001; Kwak and Radler, 2002). Also, despite high response rates,
studies indicate that there still is the potential for signi?cant differences between total sample
and respondents (Bandilla et al., 2003).
A number of studies published in the political, educational, and medicine literatures suggest
that various weighting procedures can be used to estimate the extent of non-response bias
by adjusting the estimates provided by respondents to more closely represent the total
target sample (Biemer and Lyberg, 2003). That is, because the true response of the
population cannot be known, adjusting the sample to look like the population provides a
viable way of guessing at the underlying behavior of the population. Advocated by Biemer
and Lyberg (2003) and others (Lee, 2006; Rosenbaum and Rubin, 1983a, b), this approach
is of interest for providing a reasonable estimate of a population’s behavior when true
behavior (i.e. travel decisions, visitor expenditures, etc.) is not known.
Using such an approach, the goal of this study is to estimate the extent (mean and range) of
non-response bias in online travel advertising conversion studies for 24 destinations located
throughout the USA. This goal was accomplished through a three-step process. First,
logistic regression was used to assess differences in respondents and non-respondents in
terms of geographic and demographic characteristics. Next, the respondent data for each
of the 24 American destinations was post hoc weighted based on the geographic and
demographic variables using two different weighting methods (post-strati?cation and
propensity score weighting). Last, the estimates of conversion rates and travel expenditures
for each of the 24 destinations were compared between unweighted data and two weighted
data.
Literature review
Survey errors in tourism advertising study
For several decades, tourismresearchers have sought to assess the effectiveness of tourism
advertising by estimating the proportion of people responding to advertisements who
actually visit a destination, and the amount of travel expenditure generated from these visits
(Faulkner, 1997). The conversion study, a common approach used by tourism organizations
to evaluate tourism advertising effectiveness, is most often based upon a direct response
from those who requested information from the tourism organization. The advantages of the
conversion approach include easy access to potential visitors, the straightforward
implementation of estimation procedures, and the low cost of collecting data (Lankford
et al., 1995; McWilliams and Crompton, 1997; Woodside and Sakai, 2003). However, tourism
researchers have identi?ed several methodological de?ciencies in the use of the conversion
approach (e.g. Burke and Gitelson, 1990; Hunt and Dalton, 1983; Mok, 1990; Siegel and
Ziff-Levine, 1990; Woodside, 1981). The two main methodological problems that affect the
validity and reliability of the advertising conversion estimates are sampling error
(i.e. sampling precision and size) (Perdue, 1986; Perdue and Botkin, 1988), and
non-response error (Rylander et al., 1995; Woodside and Ronkainen, 1984). However, low
cost, fast response, and wide accessibility of the internet enable tourism advertising
researchers to send surveys to the population of persons who requested travel information,
and therefore have largely eliminated the use of samples (Fricker and Schonlau, 2002;
Hwang and Fesenmaier, 2004). In addition, today’s survey tracking systems provide
extremely detailed information regarding the delivery and completion of online surveys,
thereby removing many of the problems associated with sampling error (Pan, 2009).
Non-response bias, on the other hand, has become an even greater concern as response
rates have declined substantially over the last decade and are often extremely low when
using the internet (Dolnicar et al., 2009; Sheehan, 2001). Non-response bias occurs when ‘‘a
signi?cant number of people in the survey sample do not respond to the questionnaire and
have different characteristics from those who do respond, when those characteristics are
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 341
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
important to the study’’ (Dillman, 2007, p 10). Many researchers consider the implications of
non-response bias in tourism advertising evaluation (e.g. Butter?eld et al., 1998; Kanuk and
Berenson, 1975; Houston and Ford, 1976; Heberlein and Baumgartner, 1978; Hunt and
Dalton, 1983; Lynn, 1996; Silberman and Klock, 1986; Woodside and Ronkainen, 1984).
Most prominently, Burke and Gitelson (1990) ?nd that people who visit a destination are
signi?cantly more likely to respond to a survey than those people who do not visit.
Additionally, research by Breen et al. (2001), Perdue and Botkin (1988) and Woodside and
Ronkainen (1984) indicates that travelers who spend more money during a trip are more
likely to complete the survey than those who spend less; they suggest that this response
bias can be explained by the level of involvement in the trip where visitation and trip
expenditures are good surrogates of involvement (Fesenmaier and Johnson, 1989; Van
Kenhove et al., 2002). Therefore, non-response bias should lead to a signi?cant in?ation in
the estimate of visitor conversion and expenditures.
In other research, Woodside and Ronkainen (1984) argue that wave analysis (comparing the
differences in demographic, attitudinal or behavioral variables across mail waves) is useful
when there is no information available about non-respondents. Many studies use wave
analysis to identify non-response error including Lankford et al. (1995), McCool (1991), and
Woodside and Ronkainen (1984) and suggest this as a general approach to reduce the error.
However, challenging this approach, Rylander et al. (1995) and Crompton and Cole (2001)
argue that late respondents are not reliable substitutes for non-respondents. This study
considers an alternative approach to assessing the impact of non-response bias in
travel-related conversion studies, arguing that conversion rate and visitor expenditure can be
estimated by post hoc weighting of the response data (Makela, 2003). Through this approach,
non-response bias can be calculated by comparing conversion rates and visitor expenditure
estimates using the ‘‘raw’’ response data and similar estimates made using weighted data. An
important limitation, however, is the dif?culty of knowing true behavior based upon any
sampling methodology. The post hoc weighting is a relatively ef?cient approach for balancing
the survey responses so that they are similar to the overall population in terms of key attributes
and, therefore, the estimates of behavior are closer to being re?ective of the overall population.
The following section provides a brief overview of this approach.
Weighting methods: post-strati?cation and propensity scoring adjustment
Many studies in the social sciences suffer fromnon-response error (Dillman, 1991; Sheehan,
2001; Sheehan and McMillan, 1999). In order to identify non-response error, weighting
adjustment techniques have been developed including post-strati?cation and propensity
score weighting (Glynn et al., 2006; Lee and Valliant, 2008; Holt and Elliot, 1991) and used to
compensate for unbalanced selection probabilities, non-coverage error, non-response error
and sampling error from known factors in the population (Brick and Kalton, 1996).
Post-strati?cation is, perhaps, the most widely used weighting method (Lessler and
Kalsbeek, 1992) and has been successfully applied in national government surveys
(e.g. Glassman, 2006; Hanson, 1978; Waterton and Lievesley, 1987) and was recently
applied to an internet panel survey focusing on health in the USA (Loosveldt and Sonck,
2008). Post-strati?cation weighting enables researchers to ‘‘rebalance’’ the distribution of
responses within the sample so as to correspond with the distribution of the overall
population. Speci?cally, the individual strata weight, SW
i
is the ratio of the population
proportion to sample proportion in terms of a certain strata shown to be relevant to the study
(Holt and Smith, 1979; Little, 1993):
SW
i
¼
P
i
p
i
;
where P
i
is the population proportion for strata i and p
i
is the sample proportion for strata i.
When SW
i
is larger than 1.0, the number of observations in the speci?ed strata in sample
data is fewer than in the population; if the weight value is less than 1.0, on the other hand, the
sample data included in the particular strata are over selected as compared to the
population data. Post-strati?cation weighting can be applied to non-response adjustment by
interpreting ‘‘population data’’ into ‘‘data for the total sample’’ and ‘‘sample data’’ into
PAGE 342
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
‘‘response data’’(Kalton and Flores-Cervantes, 2003). This adjustment signi?cantly
improves the accuracy of survey estimates by reducing bias (i.e. sampling and response
errors) and increasing precision, especially for survey outcomes highly correlated with the
post-stratifying variables (Little, 1993).
Propensity score adjustment is another approach to post hoc weighting that is shown to
alleviate the confounding effects of the response mechanism by achieving a balance of
covariates between the population and the sample (Rosenbaum and Rubin, 1983a). This
approach has been extensively used within the statistical community where the fundamental
idea underlying the propensity score adjustment is that the probability of responding to a
web survey for each individual is based upon a potentially large set of confounding variables
(Rosenbaum and Rubin, 1983b). Suppose, for example, that a population has a series of
explanatory variables denoted by X
1
, X
2
, . . . , X
p
and a response variable Z, where Z ¼ 1 if
people answered the survey, otherwise Z ¼ 0. The propensity score denoted as
eðX
1
; X
2
; . . . ; X
p
Þ is expressed as:
eðX
#
1; X
#
2; . . . ; X
#
pÞ ¼ PrðZ ¼ 1jX
#
1; X
#
2; . . . ; X
#
pÞ:
The propensity score can be estimated by logistic regression in the following form:
LRðxÞ ¼ logPrðZ ¼ 1jX
#
1; X
#
2; . . . ; X
#
pÞ=½1 2PrðZ ¼ 1jX
#
1; X
#
2; . . . ; X
#
pÞ?
¼ b
0
þ b
1
X
1
þ b
2
X
2
þ . . . þ b
p
X
p
:
Given the equation above, one can obtain the estimated probability derived from measured
explanatory variables. That is:
eðX
#
1; X
#
2; . . . ; X
#
pÞ ¼ PrðZ ¼ 1jX
#
1; X
#
2; . . . ; X
#
pÞ
¼
expðb
0
þ b
1
X
1
þ b
2
X
2
þ . . . þ b
p
X
p
Þ
1 þ expðb
0
þ b
1
X
1
þ b
2
X
2
þ . . . þ b
p
X
p
Þ
:
The key strength of the propensity score weighting method is the ability to reduce a large set
of confounding (or auxiliary) variables into a single propensity score (i.e. weight). This is
important for addressing the challenge of integrating a series of independent variables such
as age, gender, location and market investment into a single weighting scheme. Several
studies have con?rmed the usefulness of the propensity score weighting in online surveys
(Doagostino, 1998; Jenkins et al., 2007; Lee, 2006; Taylor, 2000). For example, Lee (2006)
analyzes the usability of the propensity score adjustment for panel web surveys to reduce
the biases taking place from non-coverage, non-probability sampling, and non-response
and ?nds that propensity score adjustment signi?cantly decreases bias. Similarly, Schonlau
et al. (2009) examine propensity scoring and post strati?cation matching within the context
of an internet-based study of health and retirement and ?nd that this approach signi?cantly
reduces differences between sample and population means.
Methodology
Data collection
Two types of data set were used in this study. First, the population data includes all American
travelers (158,705) who requested travel information via the internet from 24 tourism
destination marketing organizations located in the USA during the calendar year 2009. Each
inquirer was required to provide a home address, which was used as the basis for
developing alternative weighting schemes:
B IN-STATE was calculated to identify (1 ¼ yes; 0 ¼ no) those inquirers that live in the same
state sponsoring the advertising campaign. A second variable, ADJACENT STATE, was
calculated to identify (1 ¼ yes; 0 ¼ no) those inquirers living in bordering states. Last,
OUT-STATES was calculated to identify those inquirers that live in states further away
(1 ¼ yes; 0 ¼ no).
B TARGET MARKET – de?ned as whether or not (1 ¼ yes; 0 ¼ no) the inquirer resides
within the markets targeted in the advertising campaign.
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 343
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
B PRIZM demographic segment – a segmentation tool developed by Claritas, Inc.,
whereby each respondent is categorized (1 ¼ yes; 0 ¼ no) into one of 66 demographic
groups based on a ?ve-digit zipcode (Claritas, Inc., 2003; Schiffman et al., 2008)
(descriptions of each segment are provided in the Appendix). The use of PRIZM in
advertising studies is based upon the argument that individuals who live near others (i.e.,
within the ?ve-digit zip code) are likely to have similar demographic and life style
characteristics (Bowen, 1998).
The second data set is based upon an online survey of all persons included in the population
database. That is, the online survey was distributed to the 158,705 persons requesting travel
information fromthe 24 tourismorganizations. All surveys were conducted within two months
of the request for travel information and during 2009 (i.e. from January to December in 2009)
and the same web survey methodology was used throughout, including the same survey
design, operation system, number of reminders, and amount of incentives. In particular, the
survey used a three-step process:
1. the initial invitation was sent out along with the URL of the survey;
2. four days later, a reminder was delivered to those who had not completed the survey; and
3. the ?nal request for participation was sent out to those who had not completed the survey
one week later.
An Amazon.com gift card valued at $100 was provided to one winner for each of the 24
campaigns as an incentive for encouraging survey participation. The survey effort resulted in
a total of 14,700 responses, where the average response rate across the 24 campaigns was
9.6 percent (see Table I). The survey included three sections, where the ?rst section asked
Table I Summary of 24 tourism advertising surveys
Top 5 Prizm segmentation
Resident states (%)
Campaigns
Samples
(N)
Responses
Response
rate
(%)
Target
market
(%) Number of segments
a
(%)
In
State
Adjacent
State
Out
State
1 8,453 874 10.3 18.6 1 (4.5), 9 (3.9), 5 (3.5), 29 (3.0), 3 (2.9) 5.4 27.8 66.8
2 9,908 1,057 10.7 51.8 9 (4.6), 13 (4.3), 1 (3.8), 5 (3.6), 15 (3.5) 6.1 21.9 71.9
3 1,931 132 6.8 N/A 5 (6.0), 37 (5.2), 56 (5.2), 32 (4.3), 3 (3.4) 11.4 54.5 34.1
4 5,193 466 9.0 70.6 5 (5.3), 9 (4.6), 13 (4.6), 18 (4.2), 2 (3.5) 10.9 10.5 78.5
5 5,374 373 6.9 30.6 9 (6.1), 1 (4.9), 15 (4.6), 5 (4.3), 3 (4.0) 19.3 27.1 53.6
6 11,403 975 8.6 66.5 9 (4.2), 5 (3.4), 11 (3.4), 15 (3.2), 19 (3.2) 7.0 7.9 85.1
7 6,054 404 6.7 26.2 27 (3.6), 20 (3.4), 33 (3.4), 1 (3.1), 11 (2.9) 8.7 19.3 72.0
8 7,574 839 11.1 36.4 5 (4.6), 25 (3.8), 9 (3.7), 3 (3.6), 37 (3.2) 26.7 30.8 42.6
9 7,849 686 8.7 67.3 37 (5.1), 9 (4.9), 5 (3.7), 20 (3.1), 39 (2.8) 25.2 36.7 38.0
10 1,721 317 18.4 54.9 58 (6.6), 33 (5.3), 56 (5.3), 37 (4.6), 51 (4.6) 84.9 5.7 9.5
11 1,858 221 11.9 95.9 58 (7.0), 37 (6.0), 48 (6.0), 56 (4.2), 25 (3.7) 95.5 3.6 0.9
12 5,513 471 8.5 47.1 37 (5.2), 28 (4.6), 9 (4.3), 19 (3.7), 15 (3.3) 19.5 47.3 33.1
13 4,744 411 8.7 24.6 5 (5.8), 19 (4.4), 9 (3.8), 11 (3.2), 15 (3.2) 8.3 25.1 66.7
14 5,209 488 9.4 31.1 28 (4.3), 32 (3.9), 38 (3.9), 56 (3.6), 58 (3.6) 27.9 37.1 35.0
15 4,031 429 10.6 30.8 13 (3.9), 27 (3.6), 33 (3.6), 5 (3.1), 28 (3.1) 26.8 44.1 29.1
16 9,888 853 8.6 65.3 33 (4.0), 20 (3.4), 58 (3.0), 27 (2.9), 37 (2.8) 19.6 39.0 41.4
17 5,339 447 8.4 34.0 5 (5.6), 20 (3.7), 33 (3.7), 2 (3.5), 7 (3.2) 1.8 17.7 80.5
18 3,257 325 10.0 75.4 11 (4.4), 13 (3.8), 5 (3.5), 20 (3.5), 27 (3.5) 4.0 60.3 35.7
19 8,458 927 11.0 21.5 5 (4.7), 18 (3.8), 28 (3.8), 9 (3.6), 20 (3.3) 1.8 29.8 68.4
20 9,044 752 8.3 53.2 13 (4.8), 5 (4.0), 20 (3.3), 9 (3.2), 18 (3.2) 1.7 41.6 56.6
21 7,843 764 9.7 45.4 5 (5.1), 18 (4.9), 13 (4.8), 9 (3.9), 6 (3.3) 43.6 13.0 43.5
22 14,766 1,201 8.1 42.8 13 (4.9), 5 (4.1), 20 (4.0), 18 (3.9), 28 (3.1) 42.8 10.2 47.0
23 3,744 376 10.0 40.7 5 (4.5), 43 (3.9), 9 (3.6), 58 (3.4), 20 (3.1) 18.6 39.1 42.3
24 9,551 912 9.5 83.9 20 (4.9), 5 (3.9), 13 (3.7), 28 (3.7), 11 (3.6) 0.9 20.8 78.3
Total 158,705 14,700
Minimum 1,721 132 6.7 18.6 0.9 3.6 0.9
Maximum 14,766 1,201 18.4 95.9 95.5 60.3 85.1
Mean 6,613 613 9.6 48.5 21.6 28.0 50.4
SD 2.3 22.0 24.4 15.6 22.4
Note:
a
See Appendix for details
PAGE 344
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
respondents about the nature and timing of the travel information seen/obtained and its
impact on the trip. The second section included a series questions about travel behaviors at
the destination including length of trip, travel expenditures, and travel activities. The last
section of the survey included questions regarding demographic characteristics of the
traveler and household (i.e. age and annual income).
As can be seen in Table I, 21.6 percent of respondents live in state while 28.0 percent reside
in adjunct states, and the remaining 50.4 percent live in more distant states; 48.5 percent of
the respondents reside inside the target market. With regard to Prizm segmentation, the
majority of the respondents are described as:
B Upper Crust (1);
B Country Squires (5);
B Big Fish Small Pond (9);
B God’s Country (11);
B Upward Bound (13);
B Pools & Patios (15);
B Home Sweet Home (19);
B Fast-Track Families (20);
B Middleburg Managers (27);
B Traditional Times (28);
B Big Sky Families (33);
B Mayberry-ville (37);
B Blue Highways (45);
B Kid Country USA (50);
B Crossroads Villagers (56); and
B Back Country Folks (58).
Survey weights for the advertising survey
The ?rst step in the research process used logistic regression to evaluate the extent to which
there are systematic differences between those that answered the conversion survey and
those that did not. Logistic regression was deemed appropriate as the dependent variable is
dichotomous, indicating whether or not the respondent completed the survey (0 ¼ no,
1 ¼ yes), and the independent variables are also indicators of geographic location and
demographic make-up (i.e. in-state, adjacent state, target market, and Prizmsegmentation).
These variables are important behavioral factors affecting conversion rates and visitor
expenditures (McWilliams and Crompton, 1997; Messmer and Johnson, 1993). The speci?c
form of the model is as follows:
PrðResponse ¼1jX
#
1; X
#
2; . . . ; X
#
69Þ ¼ b
0
þ b
1
In State þ b
2
Adjacent State
þ b
2
PS1 þ . . . þ b
68
PS66 þ b
69
In Market;
where In State indicates whether or not (0 ¼ no, 1 ¼ yes) the respondent resides within the
destination state; Adjacent State indicates whether or not (0 ¼ no, 1 ¼ yes) the respondent
resides within a state adjacent to the destination state; PS
x
refers to one of 66 Prizm
segments; and In Market refers to whether or not (0 ¼ no, 1 ¼ yes) the respondent resides
within a zip code targeted by the advertising campaign.
Logistic regression was conducted for each of the 24 destinations. As can be seen in Table II,
the results for these campaigns are consistently statistically signi?cant, suggesting that
these variables can be used to explain differences between those who responded to the
survey and those who did not. The values of the Cox and Snell R
2
measure indicate that
these variables ‘‘explain’’ 1 to 6 percent of the difference between respondents and
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 345
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
non-respondents (the overall mean is 2.6 percent). The x
2
analysis reveals signi?cant
differences in the three types of variables (i.e. In State, Adjacent State, Target Market, and
Prizm segmentation) between respondents and non-respondents. In particular, the
signi?cant results for In State of campaigns indicate that in-state residents are more likely
to respond the survey. In Campaign 3, for example, 14 percent of in-state travelers
responded to the survey whereas only 5 percent of out-state travelers took the survey.
Similarly in Campaign 14, 18 percent of in-state residents and 6 percent of out-state
residents answered the advertising survey. Interestingly, the parameter estimates for
Adjacent State showed inconclusive results, where inquirers to some campaigns showed a
relatively high tendency to respond to the survey, while in other campaigns response rates
were much lower; also, several campaigns showed no signi?cant differences in the survey
response behavior (p . 0:05).
The results of the logistic regression analyses also indicate that those travelers who live in the
target market (In Market) are signi?cantly more likely to respond to the survey than those who
live outside the target market (p , 0:05 in 21 campaigns). For example in Campaign 5, 12
percent of those residing within the target market responded to the survey, whereas only 6
percent of those persons living outside the market answered the online survey; and in
Campaign 21, 16 percent of people living inside the market responded, while 7 percent who
live outside the market answered the conversion survey.
As expected, the results of the Prizm segmentation comparisons also show statistically
signi?cant results. In Campaign 13, for example, people classi?ed as Country Squires
(segment 5) and City Startups (segment 47) are signi?cantly (p , 0:05) more likely to
respond to the survey, while those described as Young In?uentials (segment 22) and
Mayberry-ville (segment 37) are signi?cantly (p , 0:05) less likely to respond to the web
survey, as indicated by negative parameter estimates. In Campaign 4, Country Squires are
more likely to respond to the survey, followed by Executive Suites (08) and American
Classics (49), while those described as Traditional Times (segment 28), Back Country Folks
(58), and Urban Elders (segment 59), are signi?cantly (p , 0:05) less likely to participate in
the advertising conversion survey. Finally in Campaign 16, those travelers described as Blue
Table II The result of 24 logistic regression analyses
Goodness-of-?t
Campaigns x
2
df Cox and Snell R
2
Probability (%) Signi?cance
1 145.88 69 0.017 89.7 0.00
2 126.68 69 0.013 89.3 0.00
3 71.259 68 0.036 93.2 0.37
4 105.94 69 0.020 91.0 0.00
5 150.81 69 0.028 93.1 0.00
6 120.02 69 0.010 91.4 0.00
7 144.60 69 0.024 93.3 0.00
8 193.59 69 0.025 88.9 0.00
9 188.46 69 0.024 91.3 0.00
10 79.229 66 0.045 81.7 0.13
11 86.950 67 0.046 88.2 0.05
12 130.11 69 0.023 91.5 0.00
13 125.51 69 0.026 91.3 0.00
14 171.21 69 0.032 90.6 0.00
15 137.10 69 0.033 89.3 0.00
16 125.94 69 0.013 91.4 0.00
17 113.94 69 0.021 91.6 0.00
18 210.94 69 0.063 90.0 0.00
19 183.35 69 0.021 89.0 0.00
20 134.20 69 0.015 91.7 0.00
21 260.71 69 0.033 90.3 0.00
22 258.76 68 0.017 91.9 0.00
23 104.75 69 0.028 90.0 0.00
24 204.08 69 0.021 90.5 0.00
PAGE 346
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
Blood Estates (segment 2), Money & Brains (segment 7), Big Sky Families (segment 33),
and Family Thrifts (segment 63) are signi?cantly (p , 0:05) more likely to answer the online
survey.
As discussed previously, two post hoc approaches (i.e. post-strati?cation and propensity
score weighting) using the respective variables (i.e. In State, Adjacent State, Target Market,
and Prizm segmentation) were employed to adjust for the differences between respondents
and non-respondents. For the post-strati?cation, weights for each of the separate variables
(W
s
for In State, Adjacent State, W
p
for Prizm segmentation, and W
t
for Target Market) were
calculated separately based upon the proportion of the total population (i.e. both
respondents and non-respondents) and the proportion of the respondents. A single weight,
W
i
, for each individual respondent was calculated (Cordell et al., 2002) by multiplying the
respective weights. Please note that even though two studies (i.e. Campaigns 3 and 10)
showed no signi?cant results, these campaigns were included in this analysis so that the
range of potential bias across all campaigns could be estimated.
The same variables (i.e. In State, Adjacent State, Target Market, and Prizm Segmentation)
were also used to estimate weight values for propensity score adjustment following
Rosenbaum and Rubin (1984). Speci?cally, logistic regression models were developed for
each campaign and used to obtain the conditional probability (i.e. propensity score) of
responding to the online survey. Respondents were strati?ed into quintiles based upon the
propensity score following Cochran (1968) and Rosenbaum and Rubin (1984), who
demonstrated that propensity scores based on ?ve strata remove approximately 90 percent
of the removable bias.
Main ?ndings
Conversion rates estimated using the unweighted data and the two weighting schemes
(post-strati?cation and propensity score weighting) are presented in Table III. The results
show a substantial overestimation of the conversion rates as compared to the unweighted
data. Speci?cally, the average conversion rate using the unweighted data is 41.3 percent,
while the average conversion rates for the weighted data are much lower at 34.8 percent (for
the post-strati?cation approach) and 37.0 percent (for the propensity weighting approach),
respectively. Interestingly, there appears to be little difference in the estimates for at least 13
of the 24 campaigns; in Campaign 11, for example, the conversion rate using the propensity
score weighting approach (91.3 percent) is only slightly higher than the estimates for the
unweighted data (91.0 percent), and the post-strati?cation data (90.2 percent). However,
there are substantial differences in the estimates for at least ?ve campaigns (i.e. Campaigns
5, 9, 14, 21 and 22). For example, the estimated conversion rate for Campaign 22 using the
unweighted data is 57.1 percent. This estimate contrasts sharply with the 35.5 percent
conversion rate based upon the weighted data using the post strati?cation approach (a
difference of 21.6 percent), or the 44.8 percent using the propensity score weighting (a
difference of 12.5). The analyses also indicate that the weighting systems have substantially
different impact on the estimates of conversion rates where propensity score weighting
differs substantially less from the unweighted estimates than the post-strati?cation
weighting. As can be seen, the difference in mean conversion rates using
post-strati?cation is 6.5 percent whereas the mean difference for propensity score
weighting is 4.3 percent; also, the range (i.e. minimum and maximum values) of the
respective conversion rates differ substantially.
A second set of analyses focused on evaluating the impact of the response bias on visitor
expenditure, arguing that they are often used as an important indicator of advertising
effectiveness. Due to the survey structure of the expenditure question (i.e. trip expenditure
was provided in 12 categories of differing ranges), the median value was used as an
estimate of average trip expenditure (see Table IV). The results show that the unweighted
median travel expenditure across all campaigns is $486; somewhat lower than the estimates
based upon the weighted data: $517 using the post-strati?cation approach and $503 for the
propensity score method. Comparison across campaigns shows that the median trip
expenditure estimates for Campaigns 1, 12, 16, 21, 22, and 23 are substantially
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 347
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
underestimated, whereas the estimated median trip expenditure for those responding to
Campaign 7 is overestimated by $20; however, the remaining 17 campaigns appear to have
essentially the same median trip expenditure. When using the propensity score weighting
approach, Campaigns 1, 7, 9, 12, and 23 are somewhat lower, while Campaigns 8 and 19
are overestimated by $100 and $130, respectively.
As part of this analysis, total expenditure values were calculated for all inquirers of the
campaign (i.e. number of inquirers £ conversion rate £ trip expenditure) with the intention of
understanding how much non-response bias affects the estimates of total ?nancial impact
for each campaign. As can be seen in Table IV, the mean total revenue from visitors across
the 24 campaigns is $136.3m(based upon the unweighted data) and ranges from$22.2mto
$504.2m. This estimate compares to a mean of $121.9musing a post-strati?cation approach
($14.4m of mean difference) and $124.8m using the propensity score approach ($11.5m of
mean difference). Comparison of the campaigns shows that four campaigns (i.e. Campaign
1, 12, 16, and 23 using post-strati?cation) and six campaigns (i.e. 1, 9, 11, 12, 17, and 23
using propensity score weighting) are substantially underestimated (ranging from
differences of $140,000 to $37.2m). Also, the estimates for a number of campaigns (i.e. 20
campaigns from post-strati?cation and 18 campaigns from propensity score weighting) are
substantially overestimated; for example, Campaign 22 is overestimated between $82.4m
(using propensity score weighting) and $91.1m when using post-strati?cation weighting.
Discussion
Destination marketing organizations need to understand non-response bias in order to
accurately evaluate the effectiveness of advertising campaigns. While many tourism
advertising researchers have discussed the effects of non-response bias, few have actually
been able to identify the extent of the bias caused. The results of an analysis of 24
Table III The estimated conversion rates between unweighted and weighted data sets
Campaigns Respondents Unweighted Post-strati?cation Propensity score weighting
n n
Response rate
(%)
Conversion rate
(%)
Conversion rate
(%)
Difference
(%)
Conversion rate
(%)
Difference
(%)
1 8,453 791 9.4 21.2 19.2 2.0 20.2 1.0
2 9,908 933 9.4 52.2 48.9 3.3 50.4 1.8
3 1,931 131 6.8 32.8 32.4 0.4 32.2 0.6
4 5,193 410 7.9 33.2 29.8 3.4 29.3 3.9
5 5,374 326 6.1 37.7 25.2 12.5 29.8 7.9
6 11,403 819 7.2 40.2 37.5 2.7 37.3 2.9
7 6,054 328 5.4 19.8 12.2 7.6 15.4 4.4
8 7,574 769 10.2 43.0 33.7 9.3 36.9 6.1
9 7,849 565 7.2 60.2 44.3 15.9 50.7 9.5
10 1,721 291 16.9 70.1 68.2 1.9 68.9 1.2
11 1,858 189 10.2 91.0 90.2 0.8 91.3 20.3
12 5,513 420 7.6 53.6 50.6 3.0 50.2 3.4
13 4,744 360 7.6 36.4 34.8 1.6 32.7 3.7
14 5,209 441 8.5 42.2 31.4 10.8 34.3 7.9
15 4,031 392 9.7 41.6 37.0 4.6 39.9 1.7
16 9,888 773 7.8 53.7 45.0 8.7 48.3 5.4
17 5,339 389 7.3 19.5 17.5 2.0 19.9 20.4
18 3,257 319 9.8 33.5 28.2 5.3 29.3 4.2
19 8,458 825 9.8 17.8 14.3 3.5 15.1 2.7
20 9,044 652 7.2 25.9 22.4 3.5 24.2 1.7
21 7,843 696 8.9 43.8 22.7 21.1 30.5 13.3
22 14,766 1,036 7.0 57.1 35.5 21.6 44.7 12.4
23 3,744 333 8.9 37.5 29.7 7.8 31.7 5.8
24 9,551 820 8.6 26.8 24.8 2.0 24.5 2.3
Minimum 1,721 131 5.4 17.8 12.2 0.4 15.1 20.3
Maximum 14,766 1,036 16.9 91.0 90.2 21.6 91.3 13.3
Mean 6,613 542 8.5 41.3 34.8 6.5 37.0 4.3
Standard deviation 2.2 17.5 17.6 6.1 17.2 3.7
PAGE 348
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
T
a
b
l
e
I
V
E
s
t
i
m
a
t
e
d
t
r
a
v
e
l
e
x
p
e
n
d
i
t
u
r
e
u
s
i
n
g
u
n
w
e
i
g
h
t
e
d
a
n
d
w
e
i
g
h
t
e
d
d
a
t
a
s
e
t
s
U
n
w
e
i
g
h
t
e
d
P
o
s
t
-
s
t
r
a
t
i
?
c
a
t
i
o
n
P
r
o
p
e
n
s
i
t
y
s
c
o
r
e
w
e
i
g
h
t
i
n
g
C
a
m
p
a
i
g
n
s
N
C
o
n
v
e
r
s
i
o
n
r
a
t
e
(
%
)
M
e
d
i
a
n
(
$
)
T
o
t
a
l
e
x
p
e
n
d
i
t
u
r
e
(
$
,
m
i
l
l
i
o
n
s
)
C
o
n
v
e
r
s
i
o
n
r
a
t
e
(
%
)
M
e
d
i
a
n
(
$
)
T
o
t
a
l
e
x
p
e
n
d
i
t
u
r
e
(
$
,
m
i
l
l
i
o
n
s
)
D
i
f
f
e
r
e
n
c
e
(
$
,
m
i
l
l
i
o
n
s
)
C
o
n
v
e
r
s
i
o
n
r
a
t
e
(
%
)
M
e
d
i
a
n
(
$
)
T
o
t
a
l
e
x
p
e
n
d
i
t
u
r
e
s
(
$
,
m
i
l
l
i
o
n
s
)
D
i
f
f
e
r
e
n
c
e
(
$
,
m
i
l
l
i
o
n
s
)
1
8
,
4
5
3
2
1
.
2
6
5
0
1
1
6
.
5
1
9
.
2
9
0
0
1
4
6
.
1
2
2
9
.
6
2
0
.
2
9
0
0
1
5
3
.
7
2
3
7
.
2
2
9
,
9
0
8
5
2
.
2
9
0
0
4
6
5
.
5
4
8
.
9
9
0
0
4
3
6
.
1
2
9
.
4
5
0
.
4
9
0
0
4
4
9
.
4
1
6
.
1
3
1
,
9
3
1
3
2
.
8
3
5
0
2
2
.
2
3
2
.
4
3
5
0
2
1
.
9
0
.
2
7
3
2
.
2
3
5
0
2
1
.
8
0
.
4
1
4
5
,
1
9
3
3
3
.
2
9
0
0
1
5
5
.
2
2
9
.
8
9
0
0
1
3
9
.
3
1
5
.
9
2
9
.
3
9
0
0
1
3
7
.
0
1
8
.
2
5
5
,
3
7
4
3
7
.
7
2
5
0
5
0
.
7
2
5
.
2
2
5
0
3
3
.
9
1
6
.
8
2
9
.
8
2
5
0
4
0
.
0
1
0
.
6
6
1
1
,
4
0
3
4
0
.
2
1
,
1
0
0
5
0
4
.
2
3
7
.
5
1
,
1
0
0
4
7
0
.
4
3
3
.
9
3
7
.
3
1
,
1
0
0
4
6
7
.
9
3
6
.
4
7
6
,
0
5
4
1
9
.
8
3
7
0
4
4
.
4
1
2
.
2
3
5
0
2
5
.
9
1
8
.
5
1
5
.
4
4
5
0
4
2
.
0
2
.
4
8
7
,
5
7
4
4
3
.
0
3
5
0
1
1
4
.
0
3
3
.
7
3
5
0
8
9
.
3
2
4
.
7
3
6
.
9
2
5
0
6
9
.
9
4
4
.
1
9
7
,
8
4
9
6
0
.
2
2
5
0
1
1
8
.
1
4
4
.
3
2
5
0
8
6
.
9
3
1
.
2
5
0
.
7
3
5
0
1
3
9
.
3
2
2
1
.
2
1
0
1
,
7
2
1
7
0
.
1
2
5
0
3
0
.
2
6
8
.
2
2
5
0
2
9
.
3
0
.
8
2
6
8
.
9
2
5
0
2
9
.
6
0
.
5
2
1
1
1
,
8
5
8
9
1
.
0
2
5
0
4
2
.
3
9
0
.
2
2
5
0
4
2
.
0
0
.
3
7
9
1
.
3
2
5
0
4
2
.
4
2
0
.
1
4
1
2
5
,
5
1
3
5
3
.
6
2
5
0
7
3
.
9
5
0
.
6
3
5
0
9
7
.
6
2
2
3
.
8
5
0
.
2
3
5
0
9
6
.
9
2
2
3
.
0
1
3
4
,
7
4
4
3
6
.
4
5
5
0
9
5
.
0
3
4
.
8
5
5
0
9
0
.
8
4
.
2
3
2
.
7
5
5
0
8
5
.
3
9
.
7
1
4
5
,
2
0
9
4
2
.
2
3
5
0
7
6
.
9
3
1
.
4
3
5
0
5
7
.
3
1
9
.
7
3
4
.
3
3
5
0
6
2
.
5
1
4
.
4
1
5
4
,
0
3
1
4
1
.
6
3
5
0
5
8
.
7
3
7
.
0
3
5
0
5
2
.
2
6
.
5
3
9
.
9
3
5
0
5
6
.
3
2
.
4
1
6
9
,
8
8
8
5
3
.
7
3
5
0
1
8
5
.
8
4
5
.
0
4
5
0
2
0
0
.
2
2
1
4
.
4
4
8
.
3
3
5
0
1
6
7
.
2
1
8
.
7
1
7
5
,
3
3
9
1
9
.
5
7
0
0
7
2
.
9
1
7
.
5
7
0
0
6
5
.
4
7
.
5
1
9
.
9
7
0
0
7
4
.
4
2
1
.
5
1
8
3
,
2
5
7
3
3
.
5
2
5
0
2
7
.
3
2
8
.
2
2
5
0
2
3
.
0
4
.
3
2
9
.
3
2
5
0
2
3
.
9
3
.
4
1
9
8
,
4
5
8
1
7
.
8
7
0
0
1
0
5
.
4
1
4
.
3
7
0
0
8
4
.
7
2
0
.
7
1
5
.
1
5
7
0
7
2
.
8
3
2
.
6
2
0
9
,
0
4
4
2
5
.
9
7
0
0
1
6
4
.
0
2
2
.
4
7
0
0
1
4
1
.
8
2
2
.
2
2
4
.
2
7
0
0
1
5
3
.
2
1
0
.
8
2
1
7
,
8
4
3
4
3
.
8
4
5
0
1
5
4
.
6
2
2
.
7
5
5
0
9
7
.
9
5
6
.
7
3
0
.
5
4
5
0
1
0
7
.
7
4
6
.
9
2
2
1
4
,
7
6
6
5
7
.
1
4
5
0
3
7
9
.
4
3
5
.
5
5
5
0
2
8
8
.
3
9
1
.
1
4
4
.
7
4
5
0
2
9
7
.
0
8
2
.
4
2
3
3
,
7
4
4
3
7
.
5
2
5
0
3
5
.
1
2
9
.
7
3
5
0
3
8
.
9
2
3
.
8
3
1
.
7
3
5
0
4
1
.
5
2
6
.
4
2
4
9
,
5
5
1
2
6
.
8
7
0
0
1
7
9
.
2
2
4
.
8
7
0
0
1
6
5
.
8
1
3
.
4
2
4
.
5
7
0
0
1
6
3
.
8
1
5
.
4
M
i
n
i
m
u
m
1
,
7
2
1
1
7
.
8
2
5
0
2
2
.
2
1
2
.
2
2
5
0
2
1
.
9
2
2
9
.
6
1
5
.
1
2
5
0
2
1
.
8
2
3
7
.
2
M
a
x
i
m
u
m
1
4
,
7
6
6
9
1
.
0
1
,
1
0
0
5
0
4
.
2
9
0
.
2
1
,
1
0
0
4
7
0
.
4
9
1
.
1
9
1
.
3
1
1
0
0
4
6
7
.
9
8
2
.
4
M
e
a
n
6
,
6
1
3
4
1
.
3
4
8
6
1
3
6
.
3
3
4
.
8
5
1
7
1
2
1
.
9
1
4
.
4
3
7
.
0
5
0
3
1
2
4
.
8
1
1
.
5
S
D
1
7
.
5
2
4
9
.
6
1
3
2
.
0
1
7
.
6
2
5
3
.
1
1
2
0
.
5
2
4
.
9
1
7
.
2
2
5
0
.
6
1
2
0
.
7
2
4
.
9
N
o
t
e
:
T
o
t
a
l
e
x
p
e
n
d
i
t
u
r
e
w
a
s
c
a
l
c
u
l
a
t
e
d
b
y
(
n
u
m
b
e
r
o
f
i
n
q
u
i
r
e
r
s
Þ
£
ð
c
o
n
v
e
r
s
i
o
n
r
a
t
e
s
Þ
£
ð
m
e
d
i
a
n
Þ
;
d
i
f
f
e
r
e
n
c
e
w
a
s
c
a
l
c
u
l
a
t
e
d
b
y
(
t
h
e
t
o
t
a
l
e
x
p
e
n
d
i
t
u
r
e
f
r
o
m
u
n
w
e
i
g
h
t
e
d
Þ
2
ð
t
h
e
t
o
t
a
l
e
x
p
e
n
d
i
t
u
r
e
f
r
o
m
e
a
c
h
w
e
i
g
h
t
i
n
g
m
e
t
h
o
d
)
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 349
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
advertising campaigns for US destinations clearly indicate that the use of unweighted data
to estimate advertising effectiveness leads to substantial and consistent overestimation of
conversion rates although there is limited bias in the estimates of median visitor
expenditures. Overall, due to the bias in conversion rates, the bias in estimates of visitor
expenditures is approximately 10 percent. Of course, there is substantial variation in sign
(e.g. over- or under-estimates) and extent of bias. As expected, based upon the
distributional assumptions underlying this approach, the effect of non-response bias seems
to be quite limited for many of the campaigns. In this study the conversion rate estimates of
approximately half of the campaigns were within ^3 percent of that based upon the
unweighted data; concomitantly, there were ?ve to six campaigns with very large errors
(ranging from10 percent to 22 percent). In the latter case, the 22 percent error is presumably
the result of a failure to include appropriate variables in the weighting scheme.
There are a number of limitations in this study that may in?uence the research results
including mode effect and the use of auxiliary variables used to calculate weighting. First,
studies have shown that the likelihood to answer a survey varies substantially depending on
the degree of familiarity with the mode (i.e. paper, telephone, internet). Thus, the online
survey approach used in this study to collect the data may have lead to higher response
rates among certain segments of society (i.e. younger and more urban segments). Second,
this study considered three auxiliary variables (i.e. residence state, target market, and Prizm
segmentation) which were believed to affect response rate, and which were available in both
non-response and response data sets. However, the relatively low level of explained
variance in the logistic regression analyses (on average, 2.6 percent of the variation in
response behavior) clearly indicates that additional variables should be included in the
weighting procedures. Indeed, Schonlau et al. (2007) suggest that competition-related
variables (i.e. the number and competitiveness of alternative nearby destinations) and
various aspects of the campaign (i.e. amount of investment in a location) should be
considered.
Although there are limitations in this study, the examination of 24 different American tourism
campaigns provides a useful understanding of the impact (mean and range) of
non-response bias in tourism advertising conversion studies. Additionally, where there is
dif?culty obtaining a reference survey in the advertising study, the two weighting methods
used in this study can be useful in assessing the errors in response data, especially in the
case of propensity score weighting, where the means to develop multivariate-based weights
is straightforward (Fitzmaurice, 2006). Finally, this study should encourage further
consideration of alternative approaches for evaluating the impact of advertising campaigns.
References
Bandilla, W., Bosnjak, M. and Altdorfer, P. (2003), ‘‘Survey administration effects? A comparison of
web-based and traditional written self-administered surveys using the ISSP environmental module’’,
Social Science Computer Review, Vol. 21, pp. 235-43.
Best, S.J., Krueger, B., Hubbard, C. and Smith, A. (2001), ‘‘An assessment of the generalizability of
internet surveys’’, Social Science Computer Review, Vol. 19 No. 2, pp. 131-45.
Biemer, P.P. and Lyberg, L.E. (2003), Introduction to Survey Quality, Wiley, New York, NY.
Bowen, J.T. (1998), ‘‘Market segmentation in hospitality research: no longer a sequential process’’,
International Journal of Contemporary Hospitality Management, Vol. 10 No. 7, pp. 289-96.
Breen, H., Bull, A. and Walo, M. (2001), ‘‘A comparison of survey methods to estimate visitor expenditure
at a local event’’, Tourism Management, Vol. 22, pp. 473-9.
Brick, J.M. and Kalton, G. (1996), ‘‘Handling missing data in survey research’’, Statistical Methods in
Medical Research, Vol. 5 No. 3, pp. 215-38.
Burke, J.L. and Gitelson, R. (1990), ‘‘Conversion studies: assumptions, applications, accuracy and
abuse’’, Journal of Travel Research, Vol. 28, pp. 46-51.
Butter?eld, D.W., Deal, K.R. and Kubursi, A.A. (1998), ‘‘Measuring the returns to tourism advertising’’,
Journal of Travel Research, Vol. 37 No. 1, pp. 12-20.
PAGE 350
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
Claritas, Inc. (2003), ‘‘PRIZM NE: the new evolution segment snapshots’’, available at: www.tetrad.com/
pub/prices/PRIZMNE_Clusters.pdf
Cochran, W.G. (1968), ‘‘The effectiveness of adjustment by subclassi?cation in removing bias in
observational studies’’, Biometrics, Vol. 24, pp. 205-13.
Cordell, H.K., Betz, C.J. and Green, G.T. (2002), ‘‘Recreation and the environment as cultural
dimensions in contemporary American society’’, Leisure Sciences, Vol. 24 No. 1, pp. 13-42.
Crompton, J.L. and Cole, S.-T. (2001), ‘‘An analysis of 13 tourism surveys: are three waves of data
collection necessary?’’, Journal of Travel Research, Vol. 39, pp. 356-68.
Dillman, D.A. (1991), ‘‘The design and administration of mail surveys’’, Annual Review of Sociology,
Vol. 17, pp. 225-49.
Dillman, D.A. (2007), Mail and Internet Surveys: The TailoredDesign Method, 2nd ed., Wiley, NewYork, NY.
Doagostino, R.B. (1998), ‘‘Propensity score methods for bias reduction in the comparison of a treatment
to a non-randomized control group’’, Statistics in Medicine, Vol. 17, pp. 2265-81.
Dolnicar, S., Laesser, C. and Matus, K. (2009), ‘‘Online versus paper: format effects in tourism surveys’’,
Journal of Travel Research, Vol. 47 No. 24, pp. 295-316.
Ellerbrock, M. (1981), ‘‘Improving coupon conversion studies: a comment’’, Journal of Travel Research,
Vol. 19, pp. 37-8.
Faulkner, B. (1997), ‘‘A model for the evaluation of national tourism destination marketing programs’’,
Journal of Travel Research, Vol. 35, pp. 23-32.
Fesenmaier, D.R. and Johnson, B. (1989), ‘‘Involvement-based segmentation: implications for travel
marketing in Texas’’, Tourism Management, Vol. 10 No. 4, pp. 293-300.
Fitzmaurice, G. (2006), ‘‘Confounding: propensity score adjustment’’, Nutrition, Vol. 22 Nos 11/12,
pp. 1214-6.
Fricker, R.D. and Schonlau, M. (2002), ‘‘Advantages and disadvantages of internet research surveys:
evidence from the literature’’, Field Methods, Vol. 14 No. 4, pp. 347-67.
Glassman, C.A. (2006), ‘‘The current population survey: design and methodology’’, Technical Paper 66,
US Bureau of the Census, Washington, DC.
Glynn, R., Schneeweiss, S. and Stu¨ rmer, T. (2006), ‘‘Indications for propensity scores and review of their
use in pharmacoepidemiology’’, Basic & Clinical Pharmacology & Toxicology, Vol. 98 No. 3, pp. 253-9.
Hanson, R.H. (1978), ‘‘The current population survey: design and methodology’’, Technical Paper 40,
US Bureau of the Census, Washington, DC.
Heberlein, T.A. and Baumgartner, R.A. (1978), ‘ ‘Factors affecting response rates to mailed
questionnaires: a quantitative analysis of the published literature’’, American Sociological Review,
Vol. 43, pp. 447-62.
Holt, D. and Elliot, D. (1991), ‘‘Methods of weighting for unit non-response’’, The Statistician, Vol. 40,
pp. 333-42.
Holt, D. and Smith, T.M.F. (1979), ‘‘Post strati?cation’’, Journal of the Royal Statistical Society, Vol. 142
No. 1, pp. 33-46.
Houston, M.J. and Ford, N.M. (1976), ‘‘Broadening the scope of methodological research in mail
surveys’’, Journal of Marketing Research, Vol. 13, pp. 397-403.
Hunt, J.D. and Dalton, M.J. (1983), ‘‘Comparing mail and telephone for conducting coupon conversion
studies’’, Journal of Travel Research, Vol. 21 No. 3, pp. 16-18.
Hwang, Y.H. and Fesenmaier, D.R. (2004), ‘‘Coverage error embedded in self-selected internet-based
samples: a case study of Northern Indiana’’, Journal of Travel Research, Vol. 42 No. 3, pp. 297-304.
James, M.L., Wotring, C.E. and Forrest, E.J. (1995), ‘‘An exploratory study of the perceived bene?ts of
electronic bulletin board use and their impact on other communication activities’’, Journal of
Broadcasting and Electronic Media, Vol. 39 No. 1, pp. 30-50.
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 351
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
Jenkins, P., Earle-Richardson, G., Burdick, P. and May, J. (2007), ‘‘Handling nonresponse in surveys:
analytic corrections compared with converting nonresponders’’, American Journal of Epidemiology,
Vol. 167 No. 3, pp. 369-74.
Kalton, G. and Flores-Cervantes, I. (2003), ‘‘Weighting methods’’, Journal of Of?cial Statistics, Vol. 19
No. 2, pp. 81-97.
Kanuk, L. and Berenson, C. (1975), ‘‘Mail surveys and response rates: a literature review’’, Journal of
Marketing Research, Vol. 12, pp. 440-53.
Kim, D.-Y., Hwang, Y.-H. and Fesenmaier, D.R. (2005), ‘‘Modeling tourism advertising effectiveness’’,
Journal of Travel Research, Vol. 44 No. 1, pp. 42-9.
Kwak, N. and Radler, B. (2002), ‘‘A comparison between mail and web surveys: response pattern,
respondent pro?le, and data quality’’, Journal of Of?cial Statistics, Vol. 18 No. 2, pp. 257-73.
Lankford, S.V., Buxton, B.P., Hetzler, R. and Little, J.R. (1995), ‘‘Response bias and wave analysis of
mailed questionnaires in tourism impact assessments’’, Journal of Travel Research, Vol. 33, pp. 8-13.
Lee, S. (2006), ‘‘Propensity score adjustment as a weighting scheme for volunteer panel web surveys’’,
Journal of Of?cial Statistics, Vol. 22 No. 2, pp. 329-49.
Lee, S. and Valliant, R. (2008), ‘‘Post-survey weighting methods using propensity scores: a review’’,
in Lepkowski, J.M., Tucker, C., Brick, J.M., De Leeuw, E.D., Japec, L., Lavrakas, P.J., Link, M.W. and
Sangster, R.L. (Eds), Advances in Telephone Survey Methodology, Wiley Series in Survey Methodology,
Wiley, New York, NY, chapter 8.
Lessler, J.T. and Kalsbeek, W.D. (1992), Nonsampling Error in Surveys, Wiley, New York, NY.
Little, R.J.A. (1993), ‘‘Post-strati?cation: a modeler’s perspective’’, Journal of the American Statistical
Association, Vol. 88 No. 423, pp. 1001-12.
Loosveldt, G. and Sonck, N. (2008), ‘‘An evaluation of the weighting procedures for an online access
panel survey’’, Survey Research Methods, Vol. 2 No. 2, pp. 93-105.
Lynn, P. (1996), ‘‘Weighting for non-response’’, in Banks, R., Fairgrieve, J., Gerrard, L., Orchard, T.,
Payne, C. and Westlake, A. (Eds), Survey and Statistical Computing 1996: Proceedings of the Second
ASC International Conference, Association for Survey Computing, Chesham.
McCool, S.F. (1991), ‘‘Using probabilistic incentives to increase response rates to mail-return highway
intercept diaries’’, Journal of Travel Research, Vol. 30 No. 2, pp. 17-19.
McWilliams, E.G. and Crompton, J.L. (1997), ‘‘An expanded framework for measuring the effectiveness
of destination advertising’’, Tourism Management, Vol. 18 No. 3, pp. 127-37.
Makela, P. (2003), ‘‘Impact of correcting for nonresponse by weighting on estimates of alcohol
consumption’’, Journal of Studies on Alcohol, Vol. 64 No. 4, pp. 589-96.
Messmer, D.J. and Johnson, R.R. (1993), ‘‘Inquiry conversion and travel advertising effectiveness’’,
Journal of Travel Research, Vol. 31 No. 4, pp. 14-21.
Mok, H.M.K. (1990), ‘‘A quasi-experimental measure of the effectiveness of destinational advertising:
some evidence from Hawaii’’, Journal of Travel Research, Vol. 29 No. 1, pp. 30-4.
Pan, B. (2009), ‘‘Online travel surveys and response patterns’’, Journal of Travel Research, Vol. 49 No. 1,
pp. 121-35.
Perdue, R.R. (1986), ‘‘Duplicate listing sampling bias in visitor surveys’’, Annals of Tourism Research,
Vol. 13, pp. 261-78.
Perdue, R.R. and Botkin, M.R. (1988), ‘‘Visitor survey versus conversion study’’, Annals of Tourism
Research, Vol. 15 No. 1, pp. 76-87.
Rosenbaum, P.R. and Rubin, D.B. (1983a), ‘‘The central role of the propensity score in observational
studies for casual effects’’, Biometrika, Vol. 70, pp. 41-55.
Rosenbaum, P.R. and Rubin, D.B. (1983b), ‘‘Assessing sensitivity to an unobserved binary covariate in
an observational study with binary outcome’’, Journal of the Royal Statistical Society, Vol. 45, pp. 212-8.
Rosenbaum, P.R. and Rubin, D.B. (1984), ‘ ‘Reducing bias in observational studies using
subclassi?cation on the propensity score’’, Journal of the American Statistical Association, Vol. 79
No. 387, pp. 516-24.
PAGE 352
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
Rylander, R.G., Propst, D.B. and McMurtry, T.R. (1995), ‘‘Nonresponse and recall biases in a survey of
traveler spending’’, Journal of Travel Research, Vol. 33 No. 4, pp. 39-45.
Schiffman, L., Bednall, D., O’Cass, A., Paladino, A., Ward, S. and Kanuk, L. (2008), Consumer
Behaviour, Pearson Education Australia, Frenchs Forest.
Schonlau, M., van Soest, A. and Kapteyn, A. (2007), ‘‘Are ‘webographic’ or attitudinal questions useful
for adjusting estimates from web surveys using propensity scoring?’’, Survey Research Methods, Vol. 1
No. 3, pp. 155-63.
Schonlau, M., van Soest, A., Kapteyn, A. and Couper, M. (2009), ‘‘Selection bias in web surveys and the
use of propensity scores’’, Sociological Methods & Research, Vol. 37 No. 3, pp. 291-318.
Sheehan, K. (2001), ‘‘E-mail survey response rates: a review’’, Journal of Computer-Mediated
Communication, Vol. 6 No. 2, available at: www.ascuse.org/jcmc/vol6/issue2/sheehan.html
Sheehan, K.B. and McMillan, S.J. (1999), ‘‘Response variation in e-mail surveys: an exploration’’,
Journal of Advertising Research, Vol. 39 No. 4, pp. 45-54.
Siegel, W. and Ziff-Levine, W. (1990), ‘‘Evaluating tourism advertising campaigns: conversion vs
advertising tracking studies’’, Journal of Travel Research, Vol. 28 No. 3, pp. 51-5.
Silberman, J. and Klock, M. (1986), ‘‘An alternative to conversion studies for measuring the impact of
travel ads’’, Journal of Travel Research, Vol. 24 No. 4, pp. 12-16.
Taylor, H. (2000), ‘‘Does internet research ‘work’? Comparing on-line survey results with telephone
surveys’’, International Journal of Market Research, Vol. 42 No. 1, pp. 51-63.
Tierney, P. (2000), ‘‘Internet-based evaluation of tourism web site effectiveness: methodological issues
and survey results’’, Journal of Travel Research, Vol. 39, pp. 212-9.
Tse, A.C.B. (1998), ‘‘Comparing the response rate, response speed and response quality of two
methods of sending questionnaire: e-mail vs mail’’, Journal of the Marketing Research Society, Vol. 40
No. 4, pp. 353-61.
Van Kenhove, P., Wijnen, K. and De Wulf, K. (2002), ‘‘The in?uence of topic involvement on mail-survey
response behavior’’, Psychology and Marketing, Vol. 19, pp. 293-301.
Waterton, J. and Lievesley, D. (1987), ‘‘Attention in a panel study of attitudes’’, Journal of Of?cial
Statistics, Vol. 3, pp. 267-82.
Weible, R. and Wallace, J. (1998), ‘‘Cyber research: the impact of the internet on data collection’’,
Marketing Research, Vol. 10 No. 3, pp. 19-26.
Woodside, A.G. (1981), ‘‘Measuring the conversion of advertising coupon inquiries into visitors’’, Journal
of Travel Research, Vol. 19, pp. 38-41.
Woodside, A.G. and Ronkainen, I.A. (1984), ‘‘Howserious is nonresponse bias in advertising conversion
research?’’, Journal of Travel Research, Vol. 22, pp. 34-7.
Woodside, A.G. and Sakai, M.Y. (2003), ‘‘Meta-evaluation: assessing alternative methods of
performance evaluation and audits of planned and implemented marketing strategies’’, Advances in
Business Marketing and Purchasing, Vol. 12, pp. 549-66.
Appendix. Prizm segmentation descriptions
01 Upper Crust
02 Blue Blood Estates
03 Movers & Shakers
04 Young Digerati
05 Country Squires
06 Winners Circle
07 Money & Brains
08 Executive Suites
09 Big Fish Small Pond
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 353
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
10 Second City Elite
11 Gods Country
12 Brite Lites Lil City
13 Upward Bound
14 New Empty Nests
15 Pools & Patios
16 Bohemian Mix
17 Beltway Boomers
18 Kids & Cul-de-Sacs
19 Home Sweet Home
20 Fast-Track Families
21 Gray Power
22 Young In?uentials
23 Greenbelt Sports
24 Up-and-Comers
25 Country Casuals
26 The Cosmopolitans
27 Middleburg Managers
28 Traditional Times
29 American Dreams
30 Suburban Sprawl
31 Urban Achievers
32 New Homesteaders
33 Big Sky Families
34 White Picket Fences
35 Boomtown Singles
36 Blue-Chip Blues
37 Mayberry-ville
38 Simple Pleasures
39 Domestic Duos
40 Close-In Couples
41 Sunset City Blues
42 Red White & Blues
43 Heartlanders
44 New Beginnings
45 Blue Highways
46 Old Glories
47 City Startups
48 Young & Rustic
49 American Classics
50 Kid Country USA
PAGE 354
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
51 Shotguns & Pickups
52 Suburban Pioneers
53 Mobility Blues
54 Multi-Culti Mosaic
55 Golden Ponds
56 Crossroads Villagers
57 Old Milltowns
58 Back Country Folks
59 Urban Elders
60 Park Bench Seniors
61 City Roots
62 Hometown Retired
63 Family Thrifts
64 Bedrock America
65 Big City Blues
66 Low-Rise Living
67 Unclassi?ed
About the authors
Sangwon Park is a Lecturer in the School of Hospitality & Tourism Management, University of
Surrey. Sangwon Park is the corresponding author and can be contacted at:
[email protected]
Daniel R. Fesenmaier is a Professor in the School of Tourism and Hospitality Management at
Temple University and Director of the National Laboratory for Tourism & eCommerce at
Temple University. He is also a Visiting Principal Research Fellow at the University of
Wollongong, Australia. His main research and teaching interests focus on the use of
information and the internet in travel decisions, the use of information technology for tourism
marketing and the development of knowledge-based systems for tourism marketing
organizations.
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 355
To purchase reprints of this article please e-mail: [email protected]
Or visit our web site for further details: www.emeraldinsight.com/reprints
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
This article has been cited by:
1. Xiong Lina, King Ceridwyn, Hu Clark. 2014. Where is the love?. International Journal of Contemporary Hospitality Management
26:4, 572-592. [Abstract] [Full Text] [PDF]
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
doc_182489521.pdf
The purpose of this study is to estimate the extent (mean and range) of non-response bias in
online travel advertising conversion studies for 24 destinations located throughout the USA.
International Journal of Culture, Tourism and Hospitality Research
Non-response bias in internet-based advertising conversion studies
Sangwon Park Daniel R. Fesenmaier
Article information:
To cite this document:
Sangwon Park Daniel R. Fesenmaier, (2012),"Non-response bias in internet-based advertising conversion studies", International J ournal of
Culture, Tourism and Hospitality Research, Vol. 6 Iss 4 pp. 340 - 355
Permanent link to this document:http://dx.doi.org/10.1108/17506181211265077
Downloaded on: 24 January 2016, At: 22:20 (PT)
References: this document contains references to 71 other documents.
To copy this document: [email protected]
The fulltext of this document has been downloaded 394 times since 2012*
Users who downloaded this article also downloaded:
Toyin A. Clottey, Scott J . Grawe, (2014),"Non-response bias assessment in logistics survey research: use fewer tests?", International
J ournal of Physical Distribution & Logistics Management, Vol. 44 Iss 5 pp. 412-426http://dx.doi.org/10.1108/IJ PDLM-10-2012-0314
Ana Maria Soares, J osé Carlos Pinho, (2014),"Advertising in online social networks: the role of perceived enjoyment and social influence",
J ournal of Research in Interactive Marketing, Vol. 8 Iss 3 pp. 245-263http://dx.doi.org/10.1108/J RIM-08-2014-001
Swatee Sarangi, Shreya Shah, (2015),"Individuals, teams and organizations score with gamification: Tool can help to motivate employees
and boost performance", Human Resource Management International Digest, Vol. 23 Iss 4 pp. 24-27http://dx.doi.org/10.1108/
HRMID-05-2015-0074
Access to this document was granted through an Emerald subscription provided by emerald-srm:115632 []
For Authors
If you would like to write for this, or any other Emerald publication, then please use our Emerald for Authors service information about
how to choose which publication to write for and submission guidelines are available for all. Please visit www.emeraldinsight.com/
authors for more information.
About Emerald www.emeraldinsight.com
Emerald is a global publisher linking research and practice to the benefit of society. The company manages a portfolio of more than
290 journals and over 2,350 books and book series volumes, as well as providing an extensive range of online products and additional
customer resources and services.
Emerald is both COUNTER 4 and TRANSFER compliant. The organization is a partner of the Committee on Publication Ethics (COPE) and
also works with Portico and the LOCKSS initiative for digital archive preservation.
*Related content and download information correct at time of download.
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
Non-response bias in internet-based
advertising conversion studies
Sangwon Park and Daniel R. Fesenmaier
Abstract
Purpose – The purpose of this study is to estimate the extent (mean and range) of non-response bias in
online travel advertising conversion studies for 24 destinations located throughout the USA.
Design/methodology/approach – The method uses two weighting procedures (i.e. post strati?cation
and propensity score weighting) to estimate the extent of non-response bias by adjusting the estimates
provided by respondents to more closely represent the total target sample.
Findings – The results of this analysis clearly indicate that the use of unweighted data to estimate
advertising effectiveness may lead to substantial over estimation of conversion rates, but there is limited
‘‘bias’’ in the estimates of median visitor expenditures. The analyses also indicate that weighting
systems have substantially different impact on the estimates of conversion rates.
Research limitations/implications – First, the likelihood to answer a survey varies substantially
depending on the degree of the familiarity with the mode (i.e. paper, telephone versus internet). Second,
the competition-related variables (i.e. the number and competitiveness of alternative nearby
destinations) and various aspects of the campaign (i.e. amount of investment in a location) should be
considered.
Originality/value – This study of 24 different American tourism campaigns provides a useful
understanding in the nature (mean and range) of impact of non-response bias in tourism advertising
conversion studies. Additionally, where there is dif?culty obtaining a reference survey in the advertising
study, the two weighting methods used in this study are shown to be useful for assessing the errors in
response data, especially in the case of propensity score weighting, where the means to develop
multivariate-based weights is straightforward.
Keywords Tourism advertising, Advertising effectiveness, Conversion analysis, Non-response bias,
Surveys, Research methods
Paper type Research paper
Introduction
Tourism researchers have developed a variety of methods to assess the effectiveness of
advertising campaigns (Kim et al., 2005; McWilliams and Crompton, 1997; Siegel and
Ziff-Levine, 1990; Woodside and Ronkainen, 1984). The accuracy and reliability of these
approaches have been challenged in terms of sampling strategies and non-response bias
(Mok, 1990; Woodside and Ronkainen, 1984). Today, many tourism advertising studies use
online surveys instead of traditional mailing surveys due to the bene?ts associated with the
internet, including accessibility (Tierney, 2000), low cost (Tse, 1998), fast response (Weible
and Wallace, 1998), and veri?able delivery (James et al., 1995). With online surveys, one
source of sampling bias has largely been eliminated as online-based surveys can be sent to
the entire population rather than a sample of persons requesting travel information (Hwang
and Fesenmaier, 2004). Ellerbrock (1981) and Burke and Gitelson (1990) argue that
non-response bias may result in substantial overestimates of the effectiveness of the tourism
advertising (e.g. conversion rate and trip expenditure) because people who visit a
destination are more likely to respond to a travel survey, as compared to non-visitors. Thus,
PAGE 340
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012, pp. 340-355, Q Emerald Group Publishing Limited, ISSN 1750-6182 DOI 10.1108/17506181211265077
Sangwon Park is a Lecturer
in the School of Hospitality
& Tourism Management,
University of Surrey,
Guildford, UK. Daniel R.
Fesenmaier is a Professor in
the School of Tourism
& Hospitality Management,
Temple University,
Philadelphia, Pennsylvania,
USA.
Received March 2011
Revised June 2011
Accepted September 2011
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
some scholars suggest that the most effective approach to minimize non-response bias is by
increasing response rate. Others, however, argue that while increasing response rate is
important, this is not really practical due to the general patterns of low response rate in web
surveys (Best et al., 2001; Kwak and Radler, 2002). Also, despite high response rates,
studies indicate that there still is the potential for signi?cant differences between total sample
and respondents (Bandilla et al., 2003).
A number of studies published in the political, educational, and medicine literatures suggest
that various weighting procedures can be used to estimate the extent of non-response bias
by adjusting the estimates provided by respondents to more closely represent the total
target sample (Biemer and Lyberg, 2003). That is, because the true response of the
population cannot be known, adjusting the sample to look like the population provides a
viable way of guessing at the underlying behavior of the population. Advocated by Biemer
and Lyberg (2003) and others (Lee, 2006; Rosenbaum and Rubin, 1983a, b), this approach
is of interest for providing a reasonable estimate of a population’s behavior when true
behavior (i.e. travel decisions, visitor expenditures, etc.) is not known.
Using such an approach, the goal of this study is to estimate the extent (mean and range) of
non-response bias in online travel advertising conversion studies for 24 destinations located
throughout the USA. This goal was accomplished through a three-step process. First,
logistic regression was used to assess differences in respondents and non-respondents in
terms of geographic and demographic characteristics. Next, the respondent data for each
of the 24 American destinations was post hoc weighted based on the geographic and
demographic variables using two different weighting methods (post-strati?cation and
propensity score weighting). Last, the estimates of conversion rates and travel expenditures
for each of the 24 destinations were compared between unweighted data and two weighted
data.
Literature review
Survey errors in tourism advertising study
For several decades, tourismresearchers have sought to assess the effectiveness of tourism
advertising by estimating the proportion of people responding to advertisements who
actually visit a destination, and the amount of travel expenditure generated from these visits
(Faulkner, 1997). The conversion study, a common approach used by tourism organizations
to evaluate tourism advertising effectiveness, is most often based upon a direct response
from those who requested information from the tourism organization. The advantages of the
conversion approach include easy access to potential visitors, the straightforward
implementation of estimation procedures, and the low cost of collecting data (Lankford
et al., 1995; McWilliams and Crompton, 1997; Woodside and Sakai, 2003). However, tourism
researchers have identi?ed several methodological de?ciencies in the use of the conversion
approach (e.g. Burke and Gitelson, 1990; Hunt and Dalton, 1983; Mok, 1990; Siegel and
Ziff-Levine, 1990; Woodside, 1981). The two main methodological problems that affect the
validity and reliability of the advertising conversion estimates are sampling error
(i.e. sampling precision and size) (Perdue, 1986; Perdue and Botkin, 1988), and
non-response error (Rylander et al., 1995; Woodside and Ronkainen, 1984). However, low
cost, fast response, and wide accessibility of the internet enable tourism advertising
researchers to send surveys to the population of persons who requested travel information,
and therefore have largely eliminated the use of samples (Fricker and Schonlau, 2002;
Hwang and Fesenmaier, 2004). In addition, today’s survey tracking systems provide
extremely detailed information regarding the delivery and completion of online surveys,
thereby removing many of the problems associated with sampling error (Pan, 2009).
Non-response bias, on the other hand, has become an even greater concern as response
rates have declined substantially over the last decade and are often extremely low when
using the internet (Dolnicar et al., 2009; Sheehan, 2001). Non-response bias occurs when ‘‘a
signi?cant number of people in the survey sample do not respond to the questionnaire and
have different characteristics from those who do respond, when those characteristics are
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 341
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
important to the study’’ (Dillman, 2007, p 10). Many researchers consider the implications of
non-response bias in tourism advertising evaluation (e.g. Butter?eld et al., 1998; Kanuk and
Berenson, 1975; Houston and Ford, 1976; Heberlein and Baumgartner, 1978; Hunt and
Dalton, 1983; Lynn, 1996; Silberman and Klock, 1986; Woodside and Ronkainen, 1984).
Most prominently, Burke and Gitelson (1990) ?nd that people who visit a destination are
signi?cantly more likely to respond to a survey than those people who do not visit.
Additionally, research by Breen et al. (2001), Perdue and Botkin (1988) and Woodside and
Ronkainen (1984) indicates that travelers who spend more money during a trip are more
likely to complete the survey than those who spend less; they suggest that this response
bias can be explained by the level of involvement in the trip where visitation and trip
expenditures are good surrogates of involvement (Fesenmaier and Johnson, 1989; Van
Kenhove et al., 2002). Therefore, non-response bias should lead to a signi?cant in?ation in
the estimate of visitor conversion and expenditures.
In other research, Woodside and Ronkainen (1984) argue that wave analysis (comparing the
differences in demographic, attitudinal or behavioral variables across mail waves) is useful
when there is no information available about non-respondents. Many studies use wave
analysis to identify non-response error including Lankford et al. (1995), McCool (1991), and
Woodside and Ronkainen (1984) and suggest this as a general approach to reduce the error.
However, challenging this approach, Rylander et al. (1995) and Crompton and Cole (2001)
argue that late respondents are not reliable substitutes for non-respondents. This study
considers an alternative approach to assessing the impact of non-response bias in
travel-related conversion studies, arguing that conversion rate and visitor expenditure can be
estimated by post hoc weighting of the response data (Makela, 2003). Through this approach,
non-response bias can be calculated by comparing conversion rates and visitor expenditure
estimates using the ‘‘raw’’ response data and similar estimates made using weighted data. An
important limitation, however, is the dif?culty of knowing true behavior based upon any
sampling methodology. The post hoc weighting is a relatively ef?cient approach for balancing
the survey responses so that they are similar to the overall population in terms of key attributes
and, therefore, the estimates of behavior are closer to being re?ective of the overall population.
The following section provides a brief overview of this approach.
Weighting methods: post-strati?cation and propensity scoring adjustment
Many studies in the social sciences suffer fromnon-response error (Dillman, 1991; Sheehan,
2001; Sheehan and McMillan, 1999). In order to identify non-response error, weighting
adjustment techniques have been developed including post-strati?cation and propensity
score weighting (Glynn et al., 2006; Lee and Valliant, 2008; Holt and Elliot, 1991) and used to
compensate for unbalanced selection probabilities, non-coverage error, non-response error
and sampling error from known factors in the population (Brick and Kalton, 1996).
Post-strati?cation is, perhaps, the most widely used weighting method (Lessler and
Kalsbeek, 1992) and has been successfully applied in national government surveys
(e.g. Glassman, 2006; Hanson, 1978; Waterton and Lievesley, 1987) and was recently
applied to an internet panel survey focusing on health in the USA (Loosveldt and Sonck,
2008). Post-strati?cation weighting enables researchers to ‘‘rebalance’’ the distribution of
responses within the sample so as to correspond with the distribution of the overall
population. Speci?cally, the individual strata weight, SW
i
is the ratio of the population
proportion to sample proportion in terms of a certain strata shown to be relevant to the study
(Holt and Smith, 1979; Little, 1993):
SW
i
¼
P
i
p
i
;
where P
i
is the population proportion for strata i and p
i
is the sample proportion for strata i.
When SW
i
is larger than 1.0, the number of observations in the speci?ed strata in sample
data is fewer than in the population; if the weight value is less than 1.0, on the other hand, the
sample data included in the particular strata are over selected as compared to the
population data. Post-strati?cation weighting can be applied to non-response adjustment by
interpreting ‘‘population data’’ into ‘‘data for the total sample’’ and ‘‘sample data’’ into
PAGE 342
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
‘‘response data’’(Kalton and Flores-Cervantes, 2003). This adjustment signi?cantly
improves the accuracy of survey estimates by reducing bias (i.e. sampling and response
errors) and increasing precision, especially for survey outcomes highly correlated with the
post-stratifying variables (Little, 1993).
Propensity score adjustment is another approach to post hoc weighting that is shown to
alleviate the confounding effects of the response mechanism by achieving a balance of
covariates between the population and the sample (Rosenbaum and Rubin, 1983a). This
approach has been extensively used within the statistical community where the fundamental
idea underlying the propensity score adjustment is that the probability of responding to a
web survey for each individual is based upon a potentially large set of confounding variables
(Rosenbaum and Rubin, 1983b). Suppose, for example, that a population has a series of
explanatory variables denoted by X
1
, X
2
, . . . , X
p
and a response variable Z, where Z ¼ 1 if
people answered the survey, otherwise Z ¼ 0. The propensity score denoted as
eðX
1
; X
2
; . . . ; X
p
Þ is expressed as:
eðX
#
1; X
#
2; . . . ; X
#
pÞ ¼ PrðZ ¼ 1jX
#
1; X
#
2; . . . ; X
#
pÞ:
The propensity score can be estimated by logistic regression in the following form:
LRðxÞ ¼ logPrðZ ¼ 1jX
#
1; X
#
2; . . . ; X
#
pÞ=½1 2PrðZ ¼ 1jX
#
1; X
#
2; . . . ; X
#
pÞ?
¼ b
0
þ b
1
X
1
þ b
2
X
2
þ . . . þ b
p
X
p
:
Given the equation above, one can obtain the estimated probability derived from measured
explanatory variables. That is:
eðX
#
1; X
#
2; . . . ; X
#
pÞ ¼ PrðZ ¼ 1jX
#
1; X
#
2; . . . ; X
#
pÞ
¼
expðb
0
þ b
1
X
1
þ b
2
X
2
þ . . . þ b
p
X
p
Þ
1 þ expðb
0
þ b
1
X
1
þ b
2
X
2
þ . . . þ b
p
X
p
Þ
:
The key strength of the propensity score weighting method is the ability to reduce a large set
of confounding (or auxiliary) variables into a single propensity score (i.e. weight). This is
important for addressing the challenge of integrating a series of independent variables such
as age, gender, location and market investment into a single weighting scheme. Several
studies have con?rmed the usefulness of the propensity score weighting in online surveys
(Doagostino, 1998; Jenkins et al., 2007; Lee, 2006; Taylor, 2000). For example, Lee (2006)
analyzes the usability of the propensity score adjustment for panel web surveys to reduce
the biases taking place from non-coverage, non-probability sampling, and non-response
and ?nds that propensity score adjustment signi?cantly decreases bias. Similarly, Schonlau
et al. (2009) examine propensity scoring and post strati?cation matching within the context
of an internet-based study of health and retirement and ?nd that this approach signi?cantly
reduces differences between sample and population means.
Methodology
Data collection
Two types of data set were used in this study. First, the population data includes all American
travelers (158,705) who requested travel information via the internet from 24 tourism
destination marketing organizations located in the USA during the calendar year 2009. Each
inquirer was required to provide a home address, which was used as the basis for
developing alternative weighting schemes:
B IN-STATE was calculated to identify (1 ¼ yes; 0 ¼ no) those inquirers that live in the same
state sponsoring the advertising campaign. A second variable, ADJACENT STATE, was
calculated to identify (1 ¼ yes; 0 ¼ no) those inquirers living in bordering states. Last,
OUT-STATES was calculated to identify those inquirers that live in states further away
(1 ¼ yes; 0 ¼ no).
B TARGET MARKET – de?ned as whether or not (1 ¼ yes; 0 ¼ no) the inquirer resides
within the markets targeted in the advertising campaign.
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 343
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
B PRIZM demographic segment – a segmentation tool developed by Claritas, Inc.,
whereby each respondent is categorized (1 ¼ yes; 0 ¼ no) into one of 66 demographic
groups based on a ?ve-digit zipcode (Claritas, Inc., 2003; Schiffman et al., 2008)
(descriptions of each segment are provided in the Appendix). The use of PRIZM in
advertising studies is based upon the argument that individuals who live near others (i.e.,
within the ?ve-digit zip code) are likely to have similar demographic and life style
characteristics (Bowen, 1998).
The second data set is based upon an online survey of all persons included in the population
database. That is, the online survey was distributed to the 158,705 persons requesting travel
information fromthe 24 tourismorganizations. All surveys were conducted within two months
of the request for travel information and during 2009 (i.e. from January to December in 2009)
and the same web survey methodology was used throughout, including the same survey
design, operation system, number of reminders, and amount of incentives. In particular, the
survey used a three-step process:
1. the initial invitation was sent out along with the URL of the survey;
2. four days later, a reminder was delivered to those who had not completed the survey; and
3. the ?nal request for participation was sent out to those who had not completed the survey
one week later.
An Amazon.com gift card valued at $100 was provided to one winner for each of the 24
campaigns as an incentive for encouraging survey participation. The survey effort resulted in
a total of 14,700 responses, where the average response rate across the 24 campaigns was
9.6 percent (see Table I). The survey included three sections, where the ?rst section asked
Table I Summary of 24 tourism advertising surveys
Top 5 Prizm segmentation
Resident states (%)
Campaigns
Samples
(N)
Responses
Response
rate
(%)
Target
market
(%) Number of segments
a
(%)
In
State
Adjacent
State
Out
State
1 8,453 874 10.3 18.6 1 (4.5), 9 (3.9), 5 (3.5), 29 (3.0), 3 (2.9) 5.4 27.8 66.8
2 9,908 1,057 10.7 51.8 9 (4.6), 13 (4.3), 1 (3.8), 5 (3.6), 15 (3.5) 6.1 21.9 71.9
3 1,931 132 6.8 N/A 5 (6.0), 37 (5.2), 56 (5.2), 32 (4.3), 3 (3.4) 11.4 54.5 34.1
4 5,193 466 9.0 70.6 5 (5.3), 9 (4.6), 13 (4.6), 18 (4.2), 2 (3.5) 10.9 10.5 78.5
5 5,374 373 6.9 30.6 9 (6.1), 1 (4.9), 15 (4.6), 5 (4.3), 3 (4.0) 19.3 27.1 53.6
6 11,403 975 8.6 66.5 9 (4.2), 5 (3.4), 11 (3.4), 15 (3.2), 19 (3.2) 7.0 7.9 85.1
7 6,054 404 6.7 26.2 27 (3.6), 20 (3.4), 33 (3.4), 1 (3.1), 11 (2.9) 8.7 19.3 72.0
8 7,574 839 11.1 36.4 5 (4.6), 25 (3.8), 9 (3.7), 3 (3.6), 37 (3.2) 26.7 30.8 42.6
9 7,849 686 8.7 67.3 37 (5.1), 9 (4.9), 5 (3.7), 20 (3.1), 39 (2.8) 25.2 36.7 38.0
10 1,721 317 18.4 54.9 58 (6.6), 33 (5.3), 56 (5.3), 37 (4.6), 51 (4.6) 84.9 5.7 9.5
11 1,858 221 11.9 95.9 58 (7.0), 37 (6.0), 48 (6.0), 56 (4.2), 25 (3.7) 95.5 3.6 0.9
12 5,513 471 8.5 47.1 37 (5.2), 28 (4.6), 9 (4.3), 19 (3.7), 15 (3.3) 19.5 47.3 33.1
13 4,744 411 8.7 24.6 5 (5.8), 19 (4.4), 9 (3.8), 11 (3.2), 15 (3.2) 8.3 25.1 66.7
14 5,209 488 9.4 31.1 28 (4.3), 32 (3.9), 38 (3.9), 56 (3.6), 58 (3.6) 27.9 37.1 35.0
15 4,031 429 10.6 30.8 13 (3.9), 27 (3.6), 33 (3.6), 5 (3.1), 28 (3.1) 26.8 44.1 29.1
16 9,888 853 8.6 65.3 33 (4.0), 20 (3.4), 58 (3.0), 27 (2.9), 37 (2.8) 19.6 39.0 41.4
17 5,339 447 8.4 34.0 5 (5.6), 20 (3.7), 33 (3.7), 2 (3.5), 7 (3.2) 1.8 17.7 80.5
18 3,257 325 10.0 75.4 11 (4.4), 13 (3.8), 5 (3.5), 20 (3.5), 27 (3.5) 4.0 60.3 35.7
19 8,458 927 11.0 21.5 5 (4.7), 18 (3.8), 28 (3.8), 9 (3.6), 20 (3.3) 1.8 29.8 68.4
20 9,044 752 8.3 53.2 13 (4.8), 5 (4.0), 20 (3.3), 9 (3.2), 18 (3.2) 1.7 41.6 56.6
21 7,843 764 9.7 45.4 5 (5.1), 18 (4.9), 13 (4.8), 9 (3.9), 6 (3.3) 43.6 13.0 43.5
22 14,766 1,201 8.1 42.8 13 (4.9), 5 (4.1), 20 (4.0), 18 (3.9), 28 (3.1) 42.8 10.2 47.0
23 3,744 376 10.0 40.7 5 (4.5), 43 (3.9), 9 (3.6), 58 (3.4), 20 (3.1) 18.6 39.1 42.3
24 9,551 912 9.5 83.9 20 (4.9), 5 (3.9), 13 (3.7), 28 (3.7), 11 (3.6) 0.9 20.8 78.3
Total 158,705 14,700
Minimum 1,721 132 6.7 18.6 0.9 3.6 0.9
Maximum 14,766 1,201 18.4 95.9 95.5 60.3 85.1
Mean 6,613 613 9.6 48.5 21.6 28.0 50.4
SD 2.3 22.0 24.4 15.6 22.4
Note:
a
See Appendix for details
PAGE 344
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
respondents about the nature and timing of the travel information seen/obtained and its
impact on the trip. The second section included a series questions about travel behaviors at
the destination including length of trip, travel expenditures, and travel activities. The last
section of the survey included questions regarding demographic characteristics of the
traveler and household (i.e. age and annual income).
As can be seen in Table I, 21.6 percent of respondents live in state while 28.0 percent reside
in adjunct states, and the remaining 50.4 percent live in more distant states; 48.5 percent of
the respondents reside inside the target market. With regard to Prizm segmentation, the
majority of the respondents are described as:
B Upper Crust (1);
B Country Squires (5);
B Big Fish Small Pond (9);
B God’s Country (11);
B Upward Bound (13);
B Pools & Patios (15);
B Home Sweet Home (19);
B Fast-Track Families (20);
B Middleburg Managers (27);
B Traditional Times (28);
B Big Sky Families (33);
B Mayberry-ville (37);
B Blue Highways (45);
B Kid Country USA (50);
B Crossroads Villagers (56); and
B Back Country Folks (58).
Survey weights for the advertising survey
The ?rst step in the research process used logistic regression to evaluate the extent to which
there are systematic differences between those that answered the conversion survey and
those that did not. Logistic regression was deemed appropriate as the dependent variable is
dichotomous, indicating whether or not the respondent completed the survey (0 ¼ no,
1 ¼ yes), and the independent variables are also indicators of geographic location and
demographic make-up (i.e. in-state, adjacent state, target market, and Prizmsegmentation).
These variables are important behavioral factors affecting conversion rates and visitor
expenditures (McWilliams and Crompton, 1997; Messmer and Johnson, 1993). The speci?c
form of the model is as follows:
PrðResponse ¼1jX
#
1; X
#
2; . . . ; X
#
69Þ ¼ b
0
þ b
1
In State þ b
2
Adjacent State
þ b
2
PS1 þ . . . þ b
68
PS66 þ b
69
In Market;
where In State indicates whether or not (0 ¼ no, 1 ¼ yes) the respondent resides within the
destination state; Adjacent State indicates whether or not (0 ¼ no, 1 ¼ yes) the respondent
resides within a state adjacent to the destination state; PS
x
refers to one of 66 Prizm
segments; and In Market refers to whether or not (0 ¼ no, 1 ¼ yes) the respondent resides
within a zip code targeted by the advertising campaign.
Logistic regression was conducted for each of the 24 destinations. As can be seen in Table II,
the results for these campaigns are consistently statistically signi?cant, suggesting that
these variables can be used to explain differences between those who responded to the
survey and those who did not. The values of the Cox and Snell R
2
measure indicate that
these variables ‘‘explain’’ 1 to 6 percent of the difference between respondents and
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 345
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
non-respondents (the overall mean is 2.6 percent). The x
2
analysis reveals signi?cant
differences in the three types of variables (i.e. In State, Adjacent State, Target Market, and
Prizm segmentation) between respondents and non-respondents. In particular, the
signi?cant results for In State of campaigns indicate that in-state residents are more likely
to respond the survey. In Campaign 3, for example, 14 percent of in-state travelers
responded to the survey whereas only 5 percent of out-state travelers took the survey.
Similarly in Campaign 14, 18 percent of in-state residents and 6 percent of out-state
residents answered the advertising survey. Interestingly, the parameter estimates for
Adjacent State showed inconclusive results, where inquirers to some campaigns showed a
relatively high tendency to respond to the survey, while in other campaigns response rates
were much lower; also, several campaigns showed no signi?cant differences in the survey
response behavior (p . 0:05).
The results of the logistic regression analyses also indicate that those travelers who live in the
target market (In Market) are signi?cantly more likely to respond to the survey than those who
live outside the target market (p , 0:05 in 21 campaigns). For example in Campaign 5, 12
percent of those residing within the target market responded to the survey, whereas only 6
percent of those persons living outside the market answered the online survey; and in
Campaign 21, 16 percent of people living inside the market responded, while 7 percent who
live outside the market answered the conversion survey.
As expected, the results of the Prizm segmentation comparisons also show statistically
signi?cant results. In Campaign 13, for example, people classi?ed as Country Squires
(segment 5) and City Startups (segment 47) are signi?cantly (p , 0:05) more likely to
respond to the survey, while those described as Young In?uentials (segment 22) and
Mayberry-ville (segment 37) are signi?cantly (p , 0:05) less likely to respond to the web
survey, as indicated by negative parameter estimates. In Campaign 4, Country Squires are
more likely to respond to the survey, followed by Executive Suites (08) and American
Classics (49), while those described as Traditional Times (segment 28), Back Country Folks
(58), and Urban Elders (segment 59), are signi?cantly (p , 0:05) less likely to participate in
the advertising conversion survey. Finally in Campaign 16, those travelers described as Blue
Table II The result of 24 logistic regression analyses
Goodness-of-?t
Campaigns x
2
df Cox and Snell R
2
Probability (%) Signi?cance
1 145.88 69 0.017 89.7 0.00
2 126.68 69 0.013 89.3 0.00
3 71.259 68 0.036 93.2 0.37
4 105.94 69 0.020 91.0 0.00
5 150.81 69 0.028 93.1 0.00
6 120.02 69 0.010 91.4 0.00
7 144.60 69 0.024 93.3 0.00
8 193.59 69 0.025 88.9 0.00
9 188.46 69 0.024 91.3 0.00
10 79.229 66 0.045 81.7 0.13
11 86.950 67 0.046 88.2 0.05
12 130.11 69 0.023 91.5 0.00
13 125.51 69 0.026 91.3 0.00
14 171.21 69 0.032 90.6 0.00
15 137.10 69 0.033 89.3 0.00
16 125.94 69 0.013 91.4 0.00
17 113.94 69 0.021 91.6 0.00
18 210.94 69 0.063 90.0 0.00
19 183.35 69 0.021 89.0 0.00
20 134.20 69 0.015 91.7 0.00
21 260.71 69 0.033 90.3 0.00
22 258.76 68 0.017 91.9 0.00
23 104.75 69 0.028 90.0 0.00
24 204.08 69 0.021 90.5 0.00
PAGE 346
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
Blood Estates (segment 2), Money & Brains (segment 7), Big Sky Families (segment 33),
and Family Thrifts (segment 63) are signi?cantly (p , 0:05) more likely to answer the online
survey.
As discussed previously, two post hoc approaches (i.e. post-strati?cation and propensity
score weighting) using the respective variables (i.e. In State, Adjacent State, Target Market,
and Prizm segmentation) were employed to adjust for the differences between respondents
and non-respondents. For the post-strati?cation, weights for each of the separate variables
(W
s
for In State, Adjacent State, W
p
for Prizm segmentation, and W
t
for Target Market) were
calculated separately based upon the proportion of the total population (i.e. both
respondents and non-respondents) and the proportion of the respondents. A single weight,
W
i
, for each individual respondent was calculated (Cordell et al., 2002) by multiplying the
respective weights. Please note that even though two studies (i.e. Campaigns 3 and 10)
showed no signi?cant results, these campaigns were included in this analysis so that the
range of potential bias across all campaigns could be estimated.
The same variables (i.e. In State, Adjacent State, Target Market, and Prizm Segmentation)
were also used to estimate weight values for propensity score adjustment following
Rosenbaum and Rubin (1984). Speci?cally, logistic regression models were developed for
each campaign and used to obtain the conditional probability (i.e. propensity score) of
responding to the online survey. Respondents were strati?ed into quintiles based upon the
propensity score following Cochran (1968) and Rosenbaum and Rubin (1984), who
demonstrated that propensity scores based on ?ve strata remove approximately 90 percent
of the removable bias.
Main ?ndings
Conversion rates estimated using the unweighted data and the two weighting schemes
(post-strati?cation and propensity score weighting) are presented in Table III. The results
show a substantial overestimation of the conversion rates as compared to the unweighted
data. Speci?cally, the average conversion rate using the unweighted data is 41.3 percent,
while the average conversion rates for the weighted data are much lower at 34.8 percent (for
the post-strati?cation approach) and 37.0 percent (for the propensity weighting approach),
respectively. Interestingly, there appears to be little difference in the estimates for at least 13
of the 24 campaigns; in Campaign 11, for example, the conversion rate using the propensity
score weighting approach (91.3 percent) is only slightly higher than the estimates for the
unweighted data (91.0 percent), and the post-strati?cation data (90.2 percent). However,
there are substantial differences in the estimates for at least ?ve campaigns (i.e. Campaigns
5, 9, 14, 21 and 22). For example, the estimated conversion rate for Campaign 22 using the
unweighted data is 57.1 percent. This estimate contrasts sharply with the 35.5 percent
conversion rate based upon the weighted data using the post strati?cation approach (a
difference of 21.6 percent), or the 44.8 percent using the propensity score weighting (a
difference of 12.5). The analyses also indicate that the weighting systems have substantially
different impact on the estimates of conversion rates where propensity score weighting
differs substantially less from the unweighted estimates than the post-strati?cation
weighting. As can be seen, the difference in mean conversion rates using
post-strati?cation is 6.5 percent whereas the mean difference for propensity score
weighting is 4.3 percent; also, the range (i.e. minimum and maximum values) of the
respective conversion rates differ substantially.
A second set of analyses focused on evaluating the impact of the response bias on visitor
expenditure, arguing that they are often used as an important indicator of advertising
effectiveness. Due to the survey structure of the expenditure question (i.e. trip expenditure
was provided in 12 categories of differing ranges), the median value was used as an
estimate of average trip expenditure (see Table IV). The results show that the unweighted
median travel expenditure across all campaigns is $486; somewhat lower than the estimates
based upon the weighted data: $517 using the post-strati?cation approach and $503 for the
propensity score method. Comparison across campaigns shows that the median trip
expenditure estimates for Campaigns 1, 12, 16, 21, 22, and 23 are substantially
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 347
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
underestimated, whereas the estimated median trip expenditure for those responding to
Campaign 7 is overestimated by $20; however, the remaining 17 campaigns appear to have
essentially the same median trip expenditure. When using the propensity score weighting
approach, Campaigns 1, 7, 9, 12, and 23 are somewhat lower, while Campaigns 8 and 19
are overestimated by $100 and $130, respectively.
As part of this analysis, total expenditure values were calculated for all inquirers of the
campaign (i.e. number of inquirers £ conversion rate £ trip expenditure) with the intention of
understanding how much non-response bias affects the estimates of total ?nancial impact
for each campaign. As can be seen in Table IV, the mean total revenue from visitors across
the 24 campaigns is $136.3m(based upon the unweighted data) and ranges from$22.2mto
$504.2m. This estimate compares to a mean of $121.9musing a post-strati?cation approach
($14.4m of mean difference) and $124.8m using the propensity score approach ($11.5m of
mean difference). Comparison of the campaigns shows that four campaigns (i.e. Campaign
1, 12, 16, and 23 using post-strati?cation) and six campaigns (i.e. 1, 9, 11, 12, 17, and 23
using propensity score weighting) are substantially underestimated (ranging from
differences of $140,000 to $37.2m). Also, the estimates for a number of campaigns (i.e. 20
campaigns from post-strati?cation and 18 campaigns from propensity score weighting) are
substantially overestimated; for example, Campaign 22 is overestimated between $82.4m
(using propensity score weighting) and $91.1m when using post-strati?cation weighting.
Discussion
Destination marketing organizations need to understand non-response bias in order to
accurately evaluate the effectiveness of advertising campaigns. While many tourism
advertising researchers have discussed the effects of non-response bias, few have actually
been able to identify the extent of the bias caused. The results of an analysis of 24
Table III The estimated conversion rates between unweighted and weighted data sets
Campaigns Respondents Unweighted Post-strati?cation Propensity score weighting
n n
Response rate
(%)
Conversion rate
(%)
Conversion rate
(%)
Difference
(%)
Conversion rate
(%)
Difference
(%)
1 8,453 791 9.4 21.2 19.2 2.0 20.2 1.0
2 9,908 933 9.4 52.2 48.9 3.3 50.4 1.8
3 1,931 131 6.8 32.8 32.4 0.4 32.2 0.6
4 5,193 410 7.9 33.2 29.8 3.4 29.3 3.9
5 5,374 326 6.1 37.7 25.2 12.5 29.8 7.9
6 11,403 819 7.2 40.2 37.5 2.7 37.3 2.9
7 6,054 328 5.4 19.8 12.2 7.6 15.4 4.4
8 7,574 769 10.2 43.0 33.7 9.3 36.9 6.1
9 7,849 565 7.2 60.2 44.3 15.9 50.7 9.5
10 1,721 291 16.9 70.1 68.2 1.9 68.9 1.2
11 1,858 189 10.2 91.0 90.2 0.8 91.3 20.3
12 5,513 420 7.6 53.6 50.6 3.0 50.2 3.4
13 4,744 360 7.6 36.4 34.8 1.6 32.7 3.7
14 5,209 441 8.5 42.2 31.4 10.8 34.3 7.9
15 4,031 392 9.7 41.6 37.0 4.6 39.9 1.7
16 9,888 773 7.8 53.7 45.0 8.7 48.3 5.4
17 5,339 389 7.3 19.5 17.5 2.0 19.9 20.4
18 3,257 319 9.8 33.5 28.2 5.3 29.3 4.2
19 8,458 825 9.8 17.8 14.3 3.5 15.1 2.7
20 9,044 652 7.2 25.9 22.4 3.5 24.2 1.7
21 7,843 696 8.9 43.8 22.7 21.1 30.5 13.3
22 14,766 1,036 7.0 57.1 35.5 21.6 44.7 12.4
23 3,744 333 8.9 37.5 29.7 7.8 31.7 5.8
24 9,551 820 8.6 26.8 24.8 2.0 24.5 2.3
Minimum 1,721 131 5.4 17.8 12.2 0.4 15.1 20.3
Maximum 14,766 1,036 16.9 91.0 90.2 21.6 91.3 13.3
Mean 6,613 542 8.5 41.3 34.8 6.5 37.0 4.3
Standard deviation 2.2 17.5 17.6 6.1 17.2 3.7
PAGE 348
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
T
a
b
l
e
I
V
E
s
t
i
m
a
t
e
d
t
r
a
v
e
l
e
x
p
e
n
d
i
t
u
r
e
u
s
i
n
g
u
n
w
e
i
g
h
t
e
d
a
n
d
w
e
i
g
h
t
e
d
d
a
t
a
s
e
t
s
U
n
w
e
i
g
h
t
e
d
P
o
s
t
-
s
t
r
a
t
i
?
c
a
t
i
o
n
P
r
o
p
e
n
s
i
t
y
s
c
o
r
e
w
e
i
g
h
t
i
n
g
C
a
m
p
a
i
g
n
s
N
C
o
n
v
e
r
s
i
o
n
r
a
t
e
(
%
)
M
e
d
i
a
n
(
$
)
T
o
t
a
l
e
x
p
e
n
d
i
t
u
r
e
(
$
,
m
i
l
l
i
o
n
s
)
C
o
n
v
e
r
s
i
o
n
r
a
t
e
(
%
)
M
e
d
i
a
n
(
$
)
T
o
t
a
l
e
x
p
e
n
d
i
t
u
r
e
(
$
,
m
i
l
l
i
o
n
s
)
D
i
f
f
e
r
e
n
c
e
(
$
,
m
i
l
l
i
o
n
s
)
C
o
n
v
e
r
s
i
o
n
r
a
t
e
(
%
)
M
e
d
i
a
n
(
$
)
T
o
t
a
l
e
x
p
e
n
d
i
t
u
r
e
s
(
$
,
m
i
l
l
i
o
n
s
)
D
i
f
f
e
r
e
n
c
e
(
$
,
m
i
l
l
i
o
n
s
)
1
8
,
4
5
3
2
1
.
2
6
5
0
1
1
6
.
5
1
9
.
2
9
0
0
1
4
6
.
1
2
2
9
.
6
2
0
.
2
9
0
0
1
5
3
.
7
2
3
7
.
2
2
9
,
9
0
8
5
2
.
2
9
0
0
4
6
5
.
5
4
8
.
9
9
0
0
4
3
6
.
1
2
9
.
4
5
0
.
4
9
0
0
4
4
9
.
4
1
6
.
1
3
1
,
9
3
1
3
2
.
8
3
5
0
2
2
.
2
3
2
.
4
3
5
0
2
1
.
9
0
.
2
7
3
2
.
2
3
5
0
2
1
.
8
0
.
4
1
4
5
,
1
9
3
3
3
.
2
9
0
0
1
5
5
.
2
2
9
.
8
9
0
0
1
3
9
.
3
1
5
.
9
2
9
.
3
9
0
0
1
3
7
.
0
1
8
.
2
5
5
,
3
7
4
3
7
.
7
2
5
0
5
0
.
7
2
5
.
2
2
5
0
3
3
.
9
1
6
.
8
2
9
.
8
2
5
0
4
0
.
0
1
0
.
6
6
1
1
,
4
0
3
4
0
.
2
1
,
1
0
0
5
0
4
.
2
3
7
.
5
1
,
1
0
0
4
7
0
.
4
3
3
.
9
3
7
.
3
1
,
1
0
0
4
6
7
.
9
3
6
.
4
7
6
,
0
5
4
1
9
.
8
3
7
0
4
4
.
4
1
2
.
2
3
5
0
2
5
.
9
1
8
.
5
1
5
.
4
4
5
0
4
2
.
0
2
.
4
8
7
,
5
7
4
4
3
.
0
3
5
0
1
1
4
.
0
3
3
.
7
3
5
0
8
9
.
3
2
4
.
7
3
6
.
9
2
5
0
6
9
.
9
4
4
.
1
9
7
,
8
4
9
6
0
.
2
2
5
0
1
1
8
.
1
4
4
.
3
2
5
0
8
6
.
9
3
1
.
2
5
0
.
7
3
5
0
1
3
9
.
3
2
2
1
.
2
1
0
1
,
7
2
1
7
0
.
1
2
5
0
3
0
.
2
6
8
.
2
2
5
0
2
9
.
3
0
.
8
2
6
8
.
9
2
5
0
2
9
.
6
0
.
5
2
1
1
1
,
8
5
8
9
1
.
0
2
5
0
4
2
.
3
9
0
.
2
2
5
0
4
2
.
0
0
.
3
7
9
1
.
3
2
5
0
4
2
.
4
2
0
.
1
4
1
2
5
,
5
1
3
5
3
.
6
2
5
0
7
3
.
9
5
0
.
6
3
5
0
9
7
.
6
2
2
3
.
8
5
0
.
2
3
5
0
9
6
.
9
2
2
3
.
0
1
3
4
,
7
4
4
3
6
.
4
5
5
0
9
5
.
0
3
4
.
8
5
5
0
9
0
.
8
4
.
2
3
2
.
7
5
5
0
8
5
.
3
9
.
7
1
4
5
,
2
0
9
4
2
.
2
3
5
0
7
6
.
9
3
1
.
4
3
5
0
5
7
.
3
1
9
.
7
3
4
.
3
3
5
0
6
2
.
5
1
4
.
4
1
5
4
,
0
3
1
4
1
.
6
3
5
0
5
8
.
7
3
7
.
0
3
5
0
5
2
.
2
6
.
5
3
9
.
9
3
5
0
5
6
.
3
2
.
4
1
6
9
,
8
8
8
5
3
.
7
3
5
0
1
8
5
.
8
4
5
.
0
4
5
0
2
0
0
.
2
2
1
4
.
4
4
8
.
3
3
5
0
1
6
7
.
2
1
8
.
7
1
7
5
,
3
3
9
1
9
.
5
7
0
0
7
2
.
9
1
7
.
5
7
0
0
6
5
.
4
7
.
5
1
9
.
9
7
0
0
7
4
.
4
2
1
.
5
1
8
3
,
2
5
7
3
3
.
5
2
5
0
2
7
.
3
2
8
.
2
2
5
0
2
3
.
0
4
.
3
2
9
.
3
2
5
0
2
3
.
9
3
.
4
1
9
8
,
4
5
8
1
7
.
8
7
0
0
1
0
5
.
4
1
4
.
3
7
0
0
8
4
.
7
2
0
.
7
1
5
.
1
5
7
0
7
2
.
8
3
2
.
6
2
0
9
,
0
4
4
2
5
.
9
7
0
0
1
6
4
.
0
2
2
.
4
7
0
0
1
4
1
.
8
2
2
.
2
2
4
.
2
7
0
0
1
5
3
.
2
1
0
.
8
2
1
7
,
8
4
3
4
3
.
8
4
5
0
1
5
4
.
6
2
2
.
7
5
5
0
9
7
.
9
5
6
.
7
3
0
.
5
4
5
0
1
0
7
.
7
4
6
.
9
2
2
1
4
,
7
6
6
5
7
.
1
4
5
0
3
7
9
.
4
3
5
.
5
5
5
0
2
8
8
.
3
9
1
.
1
4
4
.
7
4
5
0
2
9
7
.
0
8
2
.
4
2
3
3
,
7
4
4
3
7
.
5
2
5
0
3
5
.
1
2
9
.
7
3
5
0
3
8
.
9
2
3
.
8
3
1
.
7
3
5
0
4
1
.
5
2
6
.
4
2
4
9
,
5
5
1
2
6
.
8
7
0
0
1
7
9
.
2
2
4
.
8
7
0
0
1
6
5
.
8
1
3
.
4
2
4
.
5
7
0
0
1
6
3
.
8
1
5
.
4
M
i
n
i
m
u
m
1
,
7
2
1
1
7
.
8
2
5
0
2
2
.
2
1
2
.
2
2
5
0
2
1
.
9
2
2
9
.
6
1
5
.
1
2
5
0
2
1
.
8
2
3
7
.
2
M
a
x
i
m
u
m
1
4
,
7
6
6
9
1
.
0
1
,
1
0
0
5
0
4
.
2
9
0
.
2
1
,
1
0
0
4
7
0
.
4
9
1
.
1
9
1
.
3
1
1
0
0
4
6
7
.
9
8
2
.
4
M
e
a
n
6
,
6
1
3
4
1
.
3
4
8
6
1
3
6
.
3
3
4
.
8
5
1
7
1
2
1
.
9
1
4
.
4
3
7
.
0
5
0
3
1
2
4
.
8
1
1
.
5
S
D
1
7
.
5
2
4
9
.
6
1
3
2
.
0
1
7
.
6
2
5
3
.
1
1
2
0
.
5
2
4
.
9
1
7
.
2
2
5
0
.
6
1
2
0
.
7
2
4
.
9
N
o
t
e
:
T
o
t
a
l
e
x
p
e
n
d
i
t
u
r
e
w
a
s
c
a
l
c
u
l
a
t
e
d
b
y
(
n
u
m
b
e
r
o
f
i
n
q
u
i
r
e
r
s
Þ
£
ð
c
o
n
v
e
r
s
i
o
n
r
a
t
e
s
Þ
£
ð
m
e
d
i
a
n
Þ
;
d
i
f
f
e
r
e
n
c
e
w
a
s
c
a
l
c
u
l
a
t
e
d
b
y
(
t
h
e
t
o
t
a
l
e
x
p
e
n
d
i
t
u
r
e
f
r
o
m
u
n
w
e
i
g
h
t
e
d
Þ
2
ð
t
h
e
t
o
t
a
l
e
x
p
e
n
d
i
t
u
r
e
f
r
o
m
e
a
c
h
w
e
i
g
h
t
i
n
g
m
e
t
h
o
d
)
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 349
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
advertising campaigns for US destinations clearly indicate that the use of unweighted data
to estimate advertising effectiveness leads to substantial and consistent overestimation of
conversion rates although there is limited bias in the estimates of median visitor
expenditures. Overall, due to the bias in conversion rates, the bias in estimates of visitor
expenditures is approximately 10 percent. Of course, there is substantial variation in sign
(e.g. over- or under-estimates) and extent of bias. As expected, based upon the
distributional assumptions underlying this approach, the effect of non-response bias seems
to be quite limited for many of the campaigns. In this study the conversion rate estimates of
approximately half of the campaigns were within ^3 percent of that based upon the
unweighted data; concomitantly, there were ?ve to six campaigns with very large errors
(ranging from10 percent to 22 percent). In the latter case, the 22 percent error is presumably
the result of a failure to include appropriate variables in the weighting scheme.
There are a number of limitations in this study that may in?uence the research results
including mode effect and the use of auxiliary variables used to calculate weighting. First,
studies have shown that the likelihood to answer a survey varies substantially depending on
the degree of familiarity with the mode (i.e. paper, telephone, internet). Thus, the online
survey approach used in this study to collect the data may have lead to higher response
rates among certain segments of society (i.e. younger and more urban segments). Second,
this study considered three auxiliary variables (i.e. residence state, target market, and Prizm
segmentation) which were believed to affect response rate, and which were available in both
non-response and response data sets. However, the relatively low level of explained
variance in the logistic regression analyses (on average, 2.6 percent of the variation in
response behavior) clearly indicates that additional variables should be included in the
weighting procedures. Indeed, Schonlau et al. (2007) suggest that competition-related
variables (i.e. the number and competitiveness of alternative nearby destinations) and
various aspects of the campaign (i.e. amount of investment in a location) should be
considered.
Although there are limitations in this study, the examination of 24 different American tourism
campaigns provides a useful understanding of the impact (mean and range) of
non-response bias in tourism advertising conversion studies. Additionally, where there is
dif?culty obtaining a reference survey in the advertising study, the two weighting methods
used in this study can be useful in assessing the errors in response data, especially in the
case of propensity score weighting, where the means to develop multivariate-based weights
is straightforward (Fitzmaurice, 2006). Finally, this study should encourage further
consideration of alternative approaches for evaluating the impact of advertising campaigns.
References
Bandilla, W., Bosnjak, M. and Altdorfer, P. (2003), ‘‘Survey administration effects? A comparison of
web-based and traditional written self-administered surveys using the ISSP environmental module’’,
Social Science Computer Review, Vol. 21, pp. 235-43.
Best, S.J., Krueger, B., Hubbard, C. and Smith, A. (2001), ‘‘An assessment of the generalizability of
internet surveys’’, Social Science Computer Review, Vol. 19 No. 2, pp. 131-45.
Biemer, P.P. and Lyberg, L.E. (2003), Introduction to Survey Quality, Wiley, New York, NY.
Bowen, J.T. (1998), ‘‘Market segmentation in hospitality research: no longer a sequential process’’,
International Journal of Contemporary Hospitality Management, Vol. 10 No. 7, pp. 289-96.
Breen, H., Bull, A. and Walo, M. (2001), ‘‘A comparison of survey methods to estimate visitor expenditure
at a local event’’, Tourism Management, Vol. 22, pp. 473-9.
Brick, J.M. and Kalton, G. (1996), ‘‘Handling missing data in survey research’’, Statistical Methods in
Medical Research, Vol. 5 No. 3, pp. 215-38.
Burke, J.L. and Gitelson, R. (1990), ‘‘Conversion studies: assumptions, applications, accuracy and
abuse’’, Journal of Travel Research, Vol. 28, pp. 46-51.
Butter?eld, D.W., Deal, K.R. and Kubursi, A.A. (1998), ‘‘Measuring the returns to tourism advertising’’,
Journal of Travel Research, Vol. 37 No. 1, pp. 12-20.
PAGE 350
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
Claritas, Inc. (2003), ‘‘PRIZM NE: the new evolution segment snapshots’’, available at: www.tetrad.com/
pub/prices/PRIZMNE_Clusters.pdf
Cochran, W.G. (1968), ‘‘The effectiveness of adjustment by subclassi?cation in removing bias in
observational studies’’, Biometrics, Vol. 24, pp. 205-13.
Cordell, H.K., Betz, C.J. and Green, G.T. (2002), ‘‘Recreation and the environment as cultural
dimensions in contemporary American society’’, Leisure Sciences, Vol. 24 No. 1, pp. 13-42.
Crompton, J.L. and Cole, S.-T. (2001), ‘‘An analysis of 13 tourism surveys: are three waves of data
collection necessary?’’, Journal of Travel Research, Vol. 39, pp. 356-68.
Dillman, D.A. (1991), ‘‘The design and administration of mail surveys’’, Annual Review of Sociology,
Vol. 17, pp. 225-49.
Dillman, D.A. (2007), Mail and Internet Surveys: The TailoredDesign Method, 2nd ed., Wiley, NewYork, NY.
Doagostino, R.B. (1998), ‘‘Propensity score methods for bias reduction in the comparison of a treatment
to a non-randomized control group’’, Statistics in Medicine, Vol. 17, pp. 2265-81.
Dolnicar, S., Laesser, C. and Matus, K. (2009), ‘‘Online versus paper: format effects in tourism surveys’’,
Journal of Travel Research, Vol. 47 No. 24, pp. 295-316.
Ellerbrock, M. (1981), ‘‘Improving coupon conversion studies: a comment’’, Journal of Travel Research,
Vol. 19, pp. 37-8.
Faulkner, B. (1997), ‘‘A model for the evaluation of national tourism destination marketing programs’’,
Journal of Travel Research, Vol. 35, pp. 23-32.
Fesenmaier, D.R. and Johnson, B. (1989), ‘‘Involvement-based segmentation: implications for travel
marketing in Texas’’, Tourism Management, Vol. 10 No. 4, pp. 293-300.
Fitzmaurice, G. (2006), ‘‘Confounding: propensity score adjustment’’, Nutrition, Vol. 22 Nos 11/12,
pp. 1214-6.
Fricker, R.D. and Schonlau, M. (2002), ‘‘Advantages and disadvantages of internet research surveys:
evidence from the literature’’, Field Methods, Vol. 14 No. 4, pp. 347-67.
Glassman, C.A. (2006), ‘‘The current population survey: design and methodology’’, Technical Paper 66,
US Bureau of the Census, Washington, DC.
Glynn, R., Schneeweiss, S. and Stu¨ rmer, T. (2006), ‘‘Indications for propensity scores and review of their
use in pharmacoepidemiology’’, Basic & Clinical Pharmacology & Toxicology, Vol. 98 No. 3, pp. 253-9.
Hanson, R.H. (1978), ‘‘The current population survey: design and methodology’’, Technical Paper 40,
US Bureau of the Census, Washington, DC.
Heberlein, T.A. and Baumgartner, R.A. (1978), ‘ ‘Factors affecting response rates to mailed
questionnaires: a quantitative analysis of the published literature’’, American Sociological Review,
Vol. 43, pp. 447-62.
Holt, D. and Elliot, D. (1991), ‘‘Methods of weighting for unit non-response’’, The Statistician, Vol. 40,
pp. 333-42.
Holt, D. and Smith, T.M.F. (1979), ‘‘Post strati?cation’’, Journal of the Royal Statistical Society, Vol. 142
No. 1, pp. 33-46.
Houston, M.J. and Ford, N.M. (1976), ‘‘Broadening the scope of methodological research in mail
surveys’’, Journal of Marketing Research, Vol. 13, pp. 397-403.
Hunt, J.D. and Dalton, M.J. (1983), ‘‘Comparing mail and telephone for conducting coupon conversion
studies’’, Journal of Travel Research, Vol. 21 No. 3, pp. 16-18.
Hwang, Y.H. and Fesenmaier, D.R. (2004), ‘‘Coverage error embedded in self-selected internet-based
samples: a case study of Northern Indiana’’, Journal of Travel Research, Vol. 42 No. 3, pp. 297-304.
James, M.L., Wotring, C.E. and Forrest, E.J. (1995), ‘‘An exploratory study of the perceived bene?ts of
electronic bulletin board use and their impact on other communication activities’’, Journal of
Broadcasting and Electronic Media, Vol. 39 No. 1, pp. 30-50.
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 351
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
Jenkins, P., Earle-Richardson, G., Burdick, P. and May, J. (2007), ‘‘Handling nonresponse in surveys:
analytic corrections compared with converting nonresponders’’, American Journal of Epidemiology,
Vol. 167 No. 3, pp. 369-74.
Kalton, G. and Flores-Cervantes, I. (2003), ‘‘Weighting methods’’, Journal of Of?cial Statistics, Vol. 19
No. 2, pp. 81-97.
Kanuk, L. and Berenson, C. (1975), ‘‘Mail surveys and response rates: a literature review’’, Journal of
Marketing Research, Vol. 12, pp. 440-53.
Kim, D.-Y., Hwang, Y.-H. and Fesenmaier, D.R. (2005), ‘‘Modeling tourism advertising effectiveness’’,
Journal of Travel Research, Vol. 44 No. 1, pp. 42-9.
Kwak, N. and Radler, B. (2002), ‘‘A comparison between mail and web surveys: response pattern,
respondent pro?le, and data quality’’, Journal of Of?cial Statistics, Vol. 18 No. 2, pp. 257-73.
Lankford, S.V., Buxton, B.P., Hetzler, R. and Little, J.R. (1995), ‘‘Response bias and wave analysis of
mailed questionnaires in tourism impact assessments’’, Journal of Travel Research, Vol. 33, pp. 8-13.
Lee, S. (2006), ‘‘Propensity score adjustment as a weighting scheme for volunteer panel web surveys’’,
Journal of Of?cial Statistics, Vol. 22 No. 2, pp. 329-49.
Lee, S. and Valliant, R. (2008), ‘‘Post-survey weighting methods using propensity scores: a review’’,
in Lepkowski, J.M., Tucker, C., Brick, J.M., De Leeuw, E.D., Japec, L., Lavrakas, P.J., Link, M.W. and
Sangster, R.L. (Eds), Advances in Telephone Survey Methodology, Wiley Series in Survey Methodology,
Wiley, New York, NY, chapter 8.
Lessler, J.T. and Kalsbeek, W.D. (1992), Nonsampling Error in Surveys, Wiley, New York, NY.
Little, R.J.A. (1993), ‘‘Post-strati?cation: a modeler’s perspective’’, Journal of the American Statistical
Association, Vol. 88 No. 423, pp. 1001-12.
Loosveldt, G. and Sonck, N. (2008), ‘‘An evaluation of the weighting procedures for an online access
panel survey’’, Survey Research Methods, Vol. 2 No. 2, pp. 93-105.
Lynn, P. (1996), ‘‘Weighting for non-response’’, in Banks, R., Fairgrieve, J., Gerrard, L., Orchard, T.,
Payne, C. and Westlake, A. (Eds), Survey and Statistical Computing 1996: Proceedings of the Second
ASC International Conference, Association for Survey Computing, Chesham.
McCool, S.F. (1991), ‘‘Using probabilistic incentives to increase response rates to mail-return highway
intercept diaries’’, Journal of Travel Research, Vol. 30 No. 2, pp. 17-19.
McWilliams, E.G. and Crompton, J.L. (1997), ‘‘An expanded framework for measuring the effectiveness
of destination advertising’’, Tourism Management, Vol. 18 No. 3, pp. 127-37.
Makela, P. (2003), ‘‘Impact of correcting for nonresponse by weighting on estimates of alcohol
consumption’’, Journal of Studies on Alcohol, Vol. 64 No. 4, pp. 589-96.
Messmer, D.J. and Johnson, R.R. (1993), ‘‘Inquiry conversion and travel advertising effectiveness’’,
Journal of Travel Research, Vol. 31 No. 4, pp. 14-21.
Mok, H.M.K. (1990), ‘‘A quasi-experimental measure of the effectiveness of destinational advertising:
some evidence from Hawaii’’, Journal of Travel Research, Vol. 29 No. 1, pp. 30-4.
Pan, B. (2009), ‘‘Online travel surveys and response patterns’’, Journal of Travel Research, Vol. 49 No. 1,
pp. 121-35.
Perdue, R.R. (1986), ‘‘Duplicate listing sampling bias in visitor surveys’’, Annals of Tourism Research,
Vol. 13, pp. 261-78.
Perdue, R.R. and Botkin, M.R. (1988), ‘‘Visitor survey versus conversion study’’, Annals of Tourism
Research, Vol. 15 No. 1, pp. 76-87.
Rosenbaum, P.R. and Rubin, D.B. (1983a), ‘‘The central role of the propensity score in observational
studies for casual effects’’, Biometrika, Vol. 70, pp. 41-55.
Rosenbaum, P.R. and Rubin, D.B. (1983b), ‘‘Assessing sensitivity to an unobserved binary covariate in
an observational study with binary outcome’’, Journal of the Royal Statistical Society, Vol. 45, pp. 212-8.
Rosenbaum, P.R. and Rubin, D.B. (1984), ‘ ‘Reducing bias in observational studies using
subclassi?cation on the propensity score’’, Journal of the American Statistical Association, Vol. 79
No. 387, pp. 516-24.
PAGE 352
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
Rylander, R.G., Propst, D.B. and McMurtry, T.R. (1995), ‘‘Nonresponse and recall biases in a survey of
traveler spending’’, Journal of Travel Research, Vol. 33 No. 4, pp. 39-45.
Schiffman, L., Bednall, D., O’Cass, A., Paladino, A., Ward, S. and Kanuk, L. (2008), Consumer
Behaviour, Pearson Education Australia, Frenchs Forest.
Schonlau, M., van Soest, A. and Kapteyn, A. (2007), ‘‘Are ‘webographic’ or attitudinal questions useful
for adjusting estimates from web surveys using propensity scoring?’’, Survey Research Methods, Vol. 1
No. 3, pp. 155-63.
Schonlau, M., van Soest, A., Kapteyn, A. and Couper, M. (2009), ‘‘Selection bias in web surveys and the
use of propensity scores’’, Sociological Methods & Research, Vol. 37 No. 3, pp. 291-318.
Sheehan, K. (2001), ‘‘E-mail survey response rates: a review’’, Journal of Computer-Mediated
Communication, Vol. 6 No. 2, available at: www.ascuse.org/jcmc/vol6/issue2/sheehan.html
Sheehan, K.B. and McMillan, S.J. (1999), ‘‘Response variation in e-mail surveys: an exploration’’,
Journal of Advertising Research, Vol. 39 No. 4, pp. 45-54.
Siegel, W. and Ziff-Levine, W. (1990), ‘‘Evaluating tourism advertising campaigns: conversion vs
advertising tracking studies’’, Journal of Travel Research, Vol. 28 No. 3, pp. 51-5.
Silberman, J. and Klock, M. (1986), ‘‘An alternative to conversion studies for measuring the impact of
travel ads’’, Journal of Travel Research, Vol. 24 No. 4, pp. 12-16.
Taylor, H. (2000), ‘‘Does internet research ‘work’? Comparing on-line survey results with telephone
surveys’’, International Journal of Market Research, Vol. 42 No. 1, pp. 51-63.
Tierney, P. (2000), ‘‘Internet-based evaluation of tourism web site effectiveness: methodological issues
and survey results’’, Journal of Travel Research, Vol. 39, pp. 212-9.
Tse, A.C.B. (1998), ‘‘Comparing the response rate, response speed and response quality of two
methods of sending questionnaire: e-mail vs mail’’, Journal of the Marketing Research Society, Vol. 40
No. 4, pp. 353-61.
Van Kenhove, P., Wijnen, K. and De Wulf, K. (2002), ‘‘The in?uence of topic involvement on mail-survey
response behavior’’, Psychology and Marketing, Vol. 19, pp. 293-301.
Waterton, J. and Lievesley, D. (1987), ‘‘Attention in a panel study of attitudes’’, Journal of Of?cial
Statistics, Vol. 3, pp. 267-82.
Weible, R. and Wallace, J. (1998), ‘‘Cyber research: the impact of the internet on data collection’’,
Marketing Research, Vol. 10 No. 3, pp. 19-26.
Woodside, A.G. (1981), ‘‘Measuring the conversion of advertising coupon inquiries into visitors’’, Journal
of Travel Research, Vol. 19, pp. 38-41.
Woodside, A.G. and Ronkainen, I.A. (1984), ‘‘Howserious is nonresponse bias in advertising conversion
research?’’, Journal of Travel Research, Vol. 22, pp. 34-7.
Woodside, A.G. and Sakai, M.Y. (2003), ‘‘Meta-evaluation: assessing alternative methods of
performance evaluation and audits of planned and implemented marketing strategies’’, Advances in
Business Marketing and Purchasing, Vol. 12, pp. 549-66.
Appendix. Prizm segmentation descriptions
01 Upper Crust
02 Blue Blood Estates
03 Movers & Shakers
04 Young Digerati
05 Country Squires
06 Winners Circle
07 Money & Brains
08 Executive Suites
09 Big Fish Small Pond
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 353
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
10 Second City Elite
11 Gods Country
12 Brite Lites Lil City
13 Upward Bound
14 New Empty Nests
15 Pools & Patios
16 Bohemian Mix
17 Beltway Boomers
18 Kids & Cul-de-Sacs
19 Home Sweet Home
20 Fast-Track Families
21 Gray Power
22 Young In?uentials
23 Greenbelt Sports
24 Up-and-Comers
25 Country Casuals
26 The Cosmopolitans
27 Middleburg Managers
28 Traditional Times
29 American Dreams
30 Suburban Sprawl
31 Urban Achievers
32 New Homesteaders
33 Big Sky Families
34 White Picket Fences
35 Boomtown Singles
36 Blue-Chip Blues
37 Mayberry-ville
38 Simple Pleasures
39 Domestic Duos
40 Close-In Couples
41 Sunset City Blues
42 Red White & Blues
43 Heartlanders
44 New Beginnings
45 Blue Highways
46 Old Glories
47 City Startups
48 Young & Rustic
49 American Classics
50 Kid Country USA
PAGE 354
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
VOL. 6 NO. 4 2012
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
51 Shotguns & Pickups
52 Suburban Pioneers
53 Mobility Blues
54 Multi-Culti Mosaic
55 Golden Ponds
56 Crossroads Villagers
57 Old Milltowns
58 Back Country Folks
59 Urban Elders
60 Park Bench Seniors
61 City Roots
62 Hometown Retired
63 Family Thrifts
64 Bedrock America
65 Big City Blues
66 Low-Rise Living
67 Unclassi?ed
About the authors
Sangwon Park is a Lecturer in the School of Hospitality & Tourism Management, University of
Surrey. Sangwon Park is the corresponding author and can be contacted at:
[email protected]
Daniel R. Fesenmaier is a Professor in the School of Tourism and Hospitality Management at
Temple University and Director of the National Laboratory for Tourism & eCommerce at
Temple University. He is also a Visiting Principal Research Fellow at the University of
Wollongong, Australia. His main research and teaching interests focus on the use of
information and the internet in travel decisions, the use of information technology for tourism
marketing and the development of knowledge-based systems for tourism marketing
organizations.
VOL. 6 NO. 4 2012
j
INTERNATIONAL JOURNAL OF CULTURE, TOURISM AND HOSPITALITY RESEARCH
j
PAGE 355
To purchase reprints of this article please e-mail: [email protected]
Or visit our web site for further details: www.emeraldinsight.com/reprints
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
This article has been cited by:
1. Xiong Lina, King Ceridwyn, Hu Clark. 2014. Where is the love?. International Journal of Contemporary Hospitality Management
26:4, 572-592. [Abstract] [Full Text] [PDF]
D
o
w
n
l
o
a
d
e
d
b
y
P
O
N
D
I
C
H
E
R
R
Y
U
N
I
V
E
R
S
I
T
Y
A
t
2
2
:
2
0
2
4
J
a
n
u
a
r
y
2
0
1
6
(
P
T
)
doc_182489521.pdf