Description
Business Intelligence can bring critical capabilities to an organization, but the implementation of such capabilities is often plagued with problems.
Key Success Factors in Business Intelligence
Szymon Adamala * and Linus Cidrin **
* Business Intelligence and data warehousing consultant, Poland,
[email protected]
** Management consultant, Sweden,
[email protected]
Received 25 May 2011; received in revised form 3 June 2011; accepted 25 November2011
ABSTRACT: Business Intelligence can bring critical capabilities to an organization, but the implementation of
such capabilities is often plagued with problems. Why is it that certain projects fail, while others succeed? The
aim of this article is to identify the factors that are present in successful Business Intelligence projects and to
organize them into a framework of critical success factors. A survey was conducted during the spring of 2011 to
collect primary data on Business Intelligence projects. Findings confirm that Business Intelligence projects are
wrestling with both technological and non-technological problems, but the non-technological problems are
found to be harder to solve as well as more time consuming than their counterparts. The study also shows that
critical success factors for Business Intelligence projects are different from success factors for Information
Systems projects in general. Business Intelligences projects have critical success factors that are unique to the
subject matter. Major differences can be found primarily among non-technological factors, such as the presence
of a specific business need and a clear vision to guide the project. Success depends on types of project funding,
the business value provided by each iteration in the project and the alignment of the project to a strategic vision
for Business Intelligence at large. Furthermore, the study provides a framework for critical success factors that,
explains sixty-one percent of variability of success for projects. Areas which should be given special attention
include making sure that the Business Intelligence solution is built with the end users in mind, that the Business
Intelligence solution is closely tied to the company’s strategic vision and that the project is properly scoped and
prioritized to concentrate on the best opportunities first.
Keywords: Business Intelligence, Data Warehouse, Critical Success Factors, Enterprise Data Warehouse,
Success Factors Framework, project risk management
1. Introduction
This article originated from a Master’s Thesis that
the authors wrote as part of their MBA studies at
Blekinge Institute of Technology, supervised by
Dr. Klaus Solberg Søilen.
The aim of the article is to develop the tools
necessary to analyze, predict and manage the
success of Business Intelligence, data warehousing
and competitive intelligence initiatives in
contemporary organizations. “There is no reason
that each organization, as it begins and continues to
develop data warehouse projects, must wrestle with
many of the very difficult situations that have
confounded other organizations. The same
impossible situations continue to raise their ugly
heads, often with surprisingly little relation to the
Available for free online athttps://ojs.hh.se/
Journal of Intelligence Studies in Business 1 (2011) 107-127
108
industry, the size of the organization, or the
organizational structure” (Adelman, 2003). If
there is indeed little correlation between success
and the environmental factors of those initiatives,
the cause of success must be related to something
else. It is worth investigating whether the difficulty,
complexity of initiatives and the high probability of
failure are related to some factors inherent to
Business Intelligence itself.
In fact, most of the Business Intelligence and
data warehouse projects fail and the failure rate is
estimated to be between 50-80% (Beal, 2005;
Meehan, 2011; Laskowski, 2001; Legodi & Barry,
2010). Why do so many of these projects fail? “
The warehouse is used differently than other data
and systems built by information systems”
(Inmon, 2002). Therefore, the available knowledge
on how to manage ordinary projects in the
information systems domain should not be
expected to be readily applicable to the area of
Business Intelligence.
Our initial information search revealed that
Business Intelligence has been covered in a large
number of academic articles and books. However,
Business Intelligence success is not widely treated.
Few articles attempt to analyze what the factors are
which influencing BI. Some of them have
developed frameworks, but there is no proven
framework that can be relied upon for this
particular problem.
The success of BI initiatives undertaken by
companies depends on various factors. Because BI
implementation depends on the successful use of IT
resources, some of those factors are undoubtedly
technological. However, Information Technology
is merely a tool in implementing a BI solution. BI
success depends chiefly on organizational and
process factors that business administration focuses
on. Despite differences in BI systems between
various industries, those factors should be
identified and classified. The existence of a link
between individual factors or their combinations
and the success of BI initiatives should be verified.
Moreover, success factors could be classified and a
method of measurement could be devised for each
individual factor. As the management thinker Peter
Drucker once said, “what gets measured, gets
managed”. Currently, BI initiatives have a
relatively low success ratio. The successful ones
achieve ROI in excess of 400% (according to some
vendors). This figure is even more impressive
when taking into account the (traditionally capital)
investments in the infrastructure and equally high
(traditionally operational) expenses on
development projects, operations, consulting
services etc. An ability to assess the chances of
success prior to engaging in the initiative would
have two practical benefits:
1. Preventing some of the initiatives from
commencing if they score low in the
framework of success factors or allowing
them to focus on critical issues that otherwise
would have caused a failure - as a result, the
success ratio of all BI initiatives improves.
2. Facilitating management of the investment
in Business Intelligence and lowering the
entry barriers (given the high level of
investment, 50% probability of failure might
be too high for smaller companies if the loss
resulting from failure push the company into
bankruptcy).
The article focus on the following hypotheses:
Hypothesis 1. It is possible to identify critical
success factors (CSFs) of BI initiatives and
they are different from the CSFs of
information systems (IS) in general.
Hypothesis 2. Non-technical and non
technological CSFs play a dominant role in BI
initiatives’ success.
Hypothesis 3. Technology plays a secondary
role and is less critical for BI initiatives’
success.
Hypothesis 4. It is possible to develop
objective measures for each of CSFs of BI
initiatives.
If the abovementioned hypotheses can be proved,
CSFs of BI initiatives can create a framework that
can be used to predict the success in early stages of
BI initiatives.
It should be noted that, even though this study
is primarily focused on vendors in Poland, the
conclusions from this study may have relevance
and be applicable to other countries as well,
especially since many of the companies studied
were global enterprises and other aspects of
projects and IT are global: Outsourcing, offshoring,
nearshoring and other forms of international
operations. the differences in survey results
between Poland and other countries were not
significant and drawing conclusions from the
sample we gathered, we claim that our findings are
also applicable to Business Intelligence projects in
larger enterprises worldwide. The reason for
choosing Poland as the primary country is that one
of the authors has been working there for many
years and had good access to empirical data.
There is some ambiguity around the use of the
terms data warehouse and Business Intelligence
among practitioners. Historically, data warehousing
meant “the overall process of providing
information to support business decision making”
(Kimball, Ross, Thornthwaite, Mondy & Becker,
2008, p. 10). Following these authors, “delivering
the entire end-to-end solution, from the source
extracts to the queries and applications that the
109
business users interact with” (Kimball et
al.,,2008, p. 10) has always been a fundamental
principle. However, the term Business Intelligence
coined in the 1990s meant only reporting and
analysis of data stored in the warehouse, effectively
separating the two terms. Even now, there is no
agreement in the industry – some “refer to data
warehousing as the overall umbrella term, with the
data warehouse databases and BI layers as subset
deliverables within that context” (Kimball, et al.,
2008, p. 10), while others “refer to business
intelligence as the overarching term, with the data
warehouse relegated to describe the central data
store foundation of the overall business intelligence
environment” (Kimball et al., 2008, p. 10).
Because the aim of this article is not to define
the term of Business Intelligence, but rather to
investigate its success factors, both conventions are
accommodated. The literature referring both to data
warehouse systems/projects and Business
Intelligence systems have been reviewed. There is
no difference in their relevance to our study,
especially when taken into account the industry
practice that even “though some would agree that
you can theoretically deliver BI without a data
warehouse, and vice versa, that is ill-advised /…/.
Linking the two together in the DW/BI acronym
further reinforces their dependency” (Kimball et
al., 2008, p. 10). Despite this argumentation,
Business Intelligence is used in this article as an
overall term.
The research is concentrated mainly on the
factors that lead to the success of BI initiatives,
cause its success (or are able to prevent it), or are
otherwise correlated with the success. However,
little attention is paid to the definition of the
success itself or its measurement. According to
Delone and McLean (1992; 2003), success in
business entails net benefits. To properly define
success, those net benefits need to be measured.
The authors’ goal of obtaining objective
measurement criteria would call for a method to
compute the NPV of a BI initiative. A similar
subject (without the emphasis on BI, but rather for
software in general) has recently been a topic of a
PhD thesis at Blekinge Institute of Technology
(Numminen, 2010). Clearly, calculating the NPV
of BI initiatives will be out of scope for this article.
Therefore, the success itself will not be defined as
clearly as the authors should want to at first. This
would instead be a suggestion for further research
in this area.
Several of the models proposed in the
literature (Delone & McLean, 1992; 2003; Hwang
& Xu, 2008; Wixom & Watson, 2001) analyze the
interdependency between independent variables.
Furthermore, none of the abovementioned authors
propose a direct correlation with success for those
variables that were used to explain variability of
other independent variables. The goal of this study
is to enumerate all the CSFs of BI projects, not to
analyze how they depend on each other. For each
of the independent variables analyzed, a direct
dependency between success and that variable is
analyzed.
2. Literature review
Dr Ralph Kimball is credited with developing
arguably the most successful data warehouse
architecture, the dimensional modeling. The
success criteria as described by Kimball are named
“readiness factors”. The author suggests giving
up the initiative, if some of the criteria are missing
or insufficient. “Readiness shortfalls represent
project risks; the shortfalls will not correct
themselves over time. It is far better to pull the plug
on the project before significant investments have
been made than it is to continue marching down a
path filled with hazards and obstacles” (Kimball
R., Ross, Thornthwaite, Mondy, & Becker, 2008, p.
16).
Let us examine the existing frameworks that
might be applicable for explaining Business
Intelligence success.
2.1 Information Systems Success
The most obvious first choice when trying to
discover BI success factors is to look at
information systems (IS) in general. There exists an
excellent framework proposed by Delone and
McLean (Delone & McLean, 1992; 2003) that has
been widely cited, revisited, criticized, validated or
extended in hundreds of articles (Delone &
McLean, 2003). The original framework proposed
by Delone and McLean (1992) has been
reexamined in the light of that subsequent research,
resulting in a slightly updated version (Delone &
McLean, 2003) that will be discussed here. The
proposed framework defines IS success in terms of
system use, user satisfaction and net benefits
whereas the factors leading to the success
encompass information quality, system quality and
service quality. The interdependences between
those variables are depicted in Figure 1.
110
Figure 1: Delone&McLean model for IS success (Delone & McLean, 2003)
Our study did not measure system quality and
service quality at all, as they are inapplicable to
Business Intelligence. As Solberg Søilen and
Hasslinger (2009) found, vendors of BI tools do not
differentiate their products and tools (other than
adopting different definitions of Business
Intelligence). Therefore, system quality should not
be a factor in the Business Intelligence domain.
Because Business Intelligence systems are built to
present data and facilitate its analysis, data quality
is probably the most important factor. From the “
second tier” variables, intention to use and user
satisfaction should play a small role in Business
Intelligence. If BI is tied to strategic vision for the
entire company, users are obliged to use the
solution and not their own data sources. Therefore,
use alone as a variable plays a role.
One major drawback with this framework that
we identified is the concentration on technology.
The findings of our research dictate that the success
of BI initiatives relies on numerous factors that are
within the management domain rather than on
technological factors. Therefore, although the
Delone and McLean model of IS success has been
applicable to many specific information systems,
even those that have been unforeseen when the
framework was originally constructed (Delone &
McLean, 2003), we think it cannot be applied to
Business Intelligence systems without due changes.
Delone and McLean’s framework not only
does not propose any specific measurement
methods, but also avoids specific variable
definitions, leaving them to be concretized at the
time of framework application, depending on the
specific context (Delone & McLean, 2003).
2.2 Critical Success Factors for BI Systems
“The implementation of a BI system is not a
conventional application-based IT project (such as
an operational or transactional system), which has
been the focus of many CSF studies” (Yeoh &
Koronios, 2010). Having noted that, the authors
proceeded to propose a framework that included
some of the critical success factors or success
variables from Delone & McLean (1992) as a part.
System quality, information quality and system use
were grouped together and labeled “infrastructure
performance”. As an equal factors group, process
performance was proposed, encompassing classical
project management variables like budgets and
time schedules. The authors emphasized a different
set of factors, divided into three broad categories –
organization (vision and business case related
factors, management and championship related
factors), process (team related factors, project
management and methodology related factors,
change management related factors) and
technology (data related factors, infrastructure
related factors). All those factors cause business
111
Figure 2: Yeoh & Koronios model of success in BI (Yeoh & Koronios, 2010)
orientation, which in turn, together with before
mentioned infrastructure and process performance
factors lead to implementation success and,
subsequently, to perceived business benefit. This
framework is illustrated in Figure 2.
A summary of success factors across all the
dimensions with the highlight of the most
important points:
Organizational dimension
? Committed management support and
sponsorship
? Clear vision and well-established business
case
Process dimension
? Business-centric championship and
balanced team composition
? Business-driven and iterative development
approach
? User-oriented change management
Technological dimension
? Business-driven, scalable and flexible
technical framework
? Sustainable data quality and integrity
This framework is considerably better suited for
Business Intelligence systems than the one
presented by Delone and McLean (2003).
However, it has a few drawbacks. First, there are
no specific measurement criteria proposed. Since
several of the variables are defined in a way so that
very different measurement criteria can be defined
or it would be hard to devise measures other than
the Likert scale, the framework would be
impractical to apply and the results might depend
on the subjective opinions of those that provide or
calculate variable’s values, resulting in the false
prediction of BI initiative success. Another
drawback is an artificial reliance on the Delone and
McLean (2003) model of IS success. Some
infrastructure performance factors (system quality,
information quality) belong under the technology
category (infrastructure related factors or data
related factors) and should not be repeated at all or
the causal dependency should be included in the
framework.
Further criticism of this article (Yeoh &
Koronios, 2010) can be based on the way inclusion
of technology related factors was performed. “
Business-driven, scalable and flexible technical
framework” and particular variables that fall into
this category are too close to the way the vendor
who delivered the solution gives advice on his or
her product. Also, the fact that this article (Yeoh &
Koronios, 2010) was based on a single project
112
plays down the objectivity of its findings. As our
survey later revealed, technical issues are of
secondary importance.
2.3 Impossible Data Warehouse Situations.
Why are the success criteria so important? As we
have already mentioned in the introduction, some
data warehouse implementations fail. It is possible
that many of these ‘failures’ would be
considered successes (or at least to be positioned
someplace between failure and success). The
criteria for success are often determined afterwards.
This is a dangerous practice since those involved
with the project don’t really know their targets
and will make poor decisions about where to put
their energies and resources (Inmon, 2002).
According to Adelman (2003), the example
measures of success include the following
elements:
? The data warehouse usage.
? Usefulness of the data warehouse, including
end user satisfaction.
? Acceptable performance as perceived and
experienced by end users.
? Acceptable ROI - benefits justifying the
costs.
? Relevance of data warehouse application.
? Proper leverage of a business opportunity
(i.e. establishing competitive advantage
thank to the application of data warehouse).
? Timely answers to business questions.
? Higher quality / cleanliness of data (than in
the source systems).
? Initiation of changes in the business
processes that can be improved as a result of
the use of data warehouse (Adelman, 2003,
p. 5 & 40).
Douglas Hackney emphasized the critical
importance of establishing success measures – “
The ‘build it and they will come’ approach (…)
does not work in data warehousing” (Adelman,
2003, p. 41). He further suggests starting with
business “pain” with the available data. The
words “life-threatening” seem to accurately
describe the problem that need to be chosen to
solve (Inmon, 2002, p. 41). An example of a
business “pain” might be the “loss of revenue
or excessive operating costs” (Inmon, 2002, p.
42).
The main theme of the book of Adelman
(2003) is not the success factors of Business
Intelligence initiatives. Instead, the author
concentrates on various difficult situations and
proposes methods to overcome those adversities.
Although he does not name those factors as
influencing the success itself, some of them should
at least be evaluated for possible inclusion in the
final framework of this study. According to the
author, all difficulties can be organized in one of
the following categories (the first seven represent
management issues, the rest is concerned with the
technical side):
1. Management issues
2. Changing requirements and objectives
3. Justification and budget
4. Organization and staffing
5. User issues
6. Team issues
7. Project planning and scheduling
8. Data warehouse standards
9. Tools and vendors
10. Security
11. Data quality
12. Integration
13. Data warehouse architecture
14. Performance
From the factors identified in the book of Adelman
(2003), we found a few that were less important
than others, while some were not considered at all
in our survey as separate variables. For example,
architecture, performance, security, tools and
standards were considered as one broad variable
technical issue in our research and were deemed of
secondary importance. Data quality, on the other
hand, was measured in our survey separately for
source systems and target solution, both being
found to have importance.
Adelman ( 2003) devote at least equal
attention to technical issues in general as to all non-
technical issues, the latter being mentioned in less
detailed discussions. Broad and general categories
of “user issues”, “team issues”, “
organization and staffing” etc. make detailed
comparison with our study difficult. Also, a
different terminology plays a role here. What the
author considers measures, our study call variables.
For example, “data warehouse usage” was a
measure in the book (Adelman, 2003) while as we
propose three distinct measurement methods,
namely number of queries per period, number of
logons per period and number of users per period.
2.4 Competing Value Model
Since different stakeholders will look at different
criteria to determine if the initiative was a
successful one, different perspectives may need to
be taken into account. This issue of potentially
conflicting criteria has been discussed by Walton
and Dawson in their Competing Value Model
which organizes criteria in different dimensions
(Walton & Dawson, 2001). This could be present
in BI initiatives as well, where different
113
stakeholders can have different non-overlapping
criteria for success; for example a system integrator
can have different objectives for engaging in a BI
initiative, different from the end user organization
(the customer).
This study did not attempt to define BI
initiative success, but considered using this kind of
classification of criteria for different dimensions,
such as technical and non-technical. The definition
of success of BI initiatives is not part of this study.
One consequence of the findings of the article by
Walton & Dawson (2001) is that the development
of a framework allows for the interest of different
stakeholders to be organized. This problem is left
for future studies.
2.5 A Critical Success Factor Framework for
Implementing Business Intelligence Systems
Qualitative studies have been used in the area of
critical success factors in Business Intelligence
many times. One of these is a Critical Success
Factor Framework for Implementing Business
Intelligence Systems (Yeoh, Gao, & Koronios,
2007) where the authors performed a Delphi study
covering 15 BI systems experts to define a critical
success factor framework for implementing
business intelligence systems. The authors propose
a framework that is organized into seven
dimensions covering 22 factors. The seven
dimensions of this article (Yeoh, Gao, & Koronios,
2007) are:
? Commitment management support and
championship
? User-oriented change management
? Business vision
? Project planning
? Team skills and composition
? Infrastructure-related dimensions
? Data related issues
The Delphi panel rates the factors in those
dimensions and a mean is calculated for each
dimension. The above list of dimensions is ordered
in a falling mean ranking, starting with the
dimension with the highest rating. We see that
more technical dimensions can be found at the very
bottom of the list, while as non-technical
dimensions are found higher up in the list.
The paper by Yeoh et al. (2007) demonstrates,
in line with the result from this study, that
management championship and the presence of a
clear vision for Business Intelligence is important
when executing a Business Intelligence initiative.
While the result from this study seems to indicate
that the technical factors are more important than
the non-technical factors, the authors of the paper
include two “technical” dimensions out of seven.
It should be said that the ratings of the two
technical dimensions are the lowest out of the
seven, which is well in line with the findings of this
study.
2.6 Risk Management in Enterprise Data
Warehouse Projects in South Africa
The paper starts with the claim that data
warehousing implementation projects have high
estimated failure rates, up to about 50% (Legodi &
Barry, 2010). The objective of the paper and the
study was to investigate the main areas of risk for
these projects, and a Delphi method was used for
the data collection. The study was performed in
South Africa. Results from the study are in the
form of the main problems and the success factors
for this type of project. The identified factors that
have the greatest impact on success from the study
(Legodi & Barry, 2010) are (listed in priority
order):
1. Scope Creep
2. Uncontrolled Finances
3. Poor Communication
4. Stake Holder Non-Involvement
5. Skills Shortage
6. Unavailability of Tools and Technology
7. Uncontrolled Quality of Deliverables
8. Poor, Wrong or No Leader
9. Technical Difficulties
10. Legal Difficulties
We note that the first technical difficulties can be
found in the ninth item.
(Legodi & Barry, 2010) show, like the results
from this study, that non-technical factors affect the
success of a project the most. Out of the ten factors
Legodi and Barry (2010) identified, only one were
of technical nature and placed as the ninth factor
(technical difficulties). The fourth item,
stakeholder non-involvement, can be comparable to
the importance of vision and addressing a specific
business need of the sponsor. While the paper
surveyed a more narrowly defined geography
(South Africa), the finding that non-technological
factors dominate is shared between that paper and
this study.
2.7 An Exploratory Investigation of System Success
Factors in Data Warehousing
The variables used in a study by Shin (2003) are
system throughput, ease of use, ability to locate
data, access authorization, data quality (subdivided
into 4 more detailed categories of
recency/currency, level of detail, accuracy and
consistency), information utility, user training and
user satisfaction, with the latter is being used as
dependent variable. The data was gathered from a
single large US enterprise, based on a single
114
project, therefore even the author agrees that this
study must be treated as a case study (Shin, 2003,
p. 157). The authors found that 70% of end user
satisfaction could be explained by the independent
variables that were measured.
Shin (2003) approaches data warehousing
success in a different way than in the above study,
treating end user satisfaction as a proxy for project
success. This is also inconsistent with Delone and
McLean approach (Delone & McLean, 1992; 2003)
which Shin cited and claimed to be “the most
influential model in conducting research on
information systems success factors” (Shin, 2003,
p. 144). The variables used are mainly different
from the inspected ones, with only data quality
being common with this study. Variables used by
Shin (2003) describe predominantly technological
factors. As in this study actual use is only one of
the variables, the results are hardly comparable.
Also, a case study approach concentrated on
success within just one company makes this article
(Shin, 2003) less valuable for creating a larger data
warehousing or Business Intelligence success
criteria framework.
A Structural Model of Data Warehousing
Success by Hwang & Xu (2008) analyzes several
factors influencing data warehousing success.
Success is not measured as a separate variable.
Instead, following Delone and McLean (1992;
2003), benefits and quality are used as proxy, with
a distinction between individual and organizational
benefits as well as system and information quality.
The independent variables are; operational factor,
technical factor, schedule factor and economic
factor. The authors propose at least two measures
for each variable, dependent and independent. The
model (Figure 3) is able to explain between 27%
(system quality) and 41% (organizational benefits)
of the dependent variables variability.
Figure 3: Hwang & Xu structural model of data warehousing success (Hwang & Xu, 2008)
The most prominent difference in approaches is the
definition of what a variable is and what a measure
is. Many measures from this article (Hwang & Xu,
2008) are variables in our study, for example
business benefits definition, business benefits
measurability or scoping of a project. The aim for
objective measurement methods were not one of
the goals of Hwang & Xu (2008) and it is unclear
how measures proposed in their work should be
performed in practice. The authors were overly
reliant on the model of IS success (Delone &
McLean, 1992; 2003), obtaining little explanatory
power for their model. From the way the model
was constructed, it is also hard to compare whether
technical or non-technical factors play major role in
data warehousing success.
An Empirical Investigation of the Factors Affecting
Data Warehousing Success (Wixom & Watson,
2001) examined the factors influencing data
warehousing success. Authors verified several
hypotheses. Findings can be summarized as
follows:
Hypothesis 1a. A high level of data quality
will be associated with a high level of
perceived net benefits (supported).
Hypothesis 1b. A high level of system quality
will be associated with a high level of
perceived net benefits (supported).
Hypothesis 2a. A high level of organizational
implementation success is associated with a
high level of data quality (not supported).
115
Hypothesis 2b. A high level of organizational
implementation success is associated with a
high level of system quality (supported).
Hypothesis 3a. A high level of project
implementation success is associated with a
high level of data quality (not supported).
Hypothesis 3b. A high level of project
implementation success is associated with a
high level of system quality (supported).
Hypothesis 4a. A high level of technical
implementation success is associated with a
high level of data quality (not supported).
Hypothesis 4b. A high level of technical
implementation success is associated with a
high level of system quality (not supported).
Hypothesis 5. A high level of management
support is associated with a high level of
organizational implementation success
(supported).
Hypothesis 6a. A strong champion presence is
associated with a high level of organizational
implementation success (not supported).
Hypothesis 6b. A strong champion presence is
associated with a high level of project
implementation success (not supported).
Hypothesis 7a. A high level of resources is
associated with a high level of organizational
implementation success (supported).
Hypothesis 7b. A high level of resources is
associated with a high level of project
implementation success (supported).
Hypothesis 8a. A high level of user
participation is associated with organizational
implementation success (supported).
Hypothesis 8b. A high level of user
participation is associated with project
implementation success (supported).
Hypothesis 9a. A high level of team skills is
associated with project implementation success
(supported).
Hypothesis 9b. A high level of team skills is
associated with technical implementation
success (not supported).
Hypothesis 10. High-quality source systems
are associated with technical implementation
success (supported).
Hypothesis 11. Better development technology
is associated with technical implementation
success (supported).
A valuable part of in this article (Wixom &
Watson, 2001) that even led to verifying the
hypotheses proposed in this paper, was the
statistical model using partial least squares
regression. The resulting model can be seen in
Figure 4.
Figure 4: Wixom&Watson statistical model of DW success (Wixom & Watson, 2001)
R
2
values obtained by Wixom and Watson (2001),
(Figure 4) ranging between 0.016 to 0.435, means
that independent variables used in the model
provide limited explanatory power for dependent
variables. This makes it impossible to compare
findings with our study. The model relies on the IS
success model by Delone & McLean (1992) which
defines system success using a structure of proxy
variables – data and system quality, and benefits.
Variables used covered both technical and non-
technical factors influencing data warehouse
implementation success. The way the model was
116
constructed makes it impossible to verify whether
technical or non-technical factors influence the
success more strongly. The article (Wixom &
Watson, 2001) does not attempt to propose specific
measurement criteria for the variables used.
2.8 Current Practices in Data Warehousing
The study conducted in the article of Watson,
Annino, Wixom, Avery, & Rutherford (2001)
concentrated on some of the factors influencing
data warehousing projects success. Survey
respondents were asked to provide answers to
questions about who sponsored the data warehouse,
which organization unit was the driving force
behind the initiative, about solution architecture
and end users, about implementation costs,
operational costs, solution approval process, after
implementation assessment, realization of expected
benefits (together with expectations). To describe
success, two questions were used, one about ROI
and the other about perceived success of
implementation.
Although the authors gathered enough data to
analyze the dependency of individual factors with
success or to build a statistical model of several
variables, they chose not to do so. Instead, they
analyzed individual questions, summarizing
contemporary practices. Findings say that most
data warehouse sponsors come from business units
and no single architecture dominates all
implementations (Watson, Annino, Wixom, Avery,
& Rutherford, 2001). Because of the lack of a
proposed framework, it is impossible to compare
this article with our findings. Also, the article did
not attempt to propose any objective measurement
methods for variables included in the study.
2.9 Measuring User Satisfaction with Data
Warehouses: An Exploratory Study
The study of Chen, Soliman, Mao, & Frolick,
(2000) uses end user satisfaction as a proxy for
success and examines factors influencing the
aforementioned user satisfaction. Study results
identify three factors comprising several questions
each. Those factors are: support provided to end
users, accuracy, format and preciseness of data, and
fulfillment of end user needs. Authors did not
propose a framework for success and did not
attempt to build a statistical model. They also did
not attempt to develop objective and universal
measurement methods for the variables identified.
Treatment of end user satisfaction as a proxy
for project success and the goal to identify only the
factors that influence this variable makes it difficult
to compare the results with the findings. The
authors (Chen, Soliman, Mao, & Frolick, 2000)
claim to have relied on the model of IS success
(Delone & McLean, 1992), but included only one
of the “use” variables presented in that study.
The study treated use as a whole and positioned it
as one of the factors influencing success. It is
impossible to tell whether technical or non-
technical factors are more important to project
success following this article (Chen, Soliman, Mao,
& Frolick, 2000) and no objective measurement
methods are proposed.
3. Empirical Data
For primary research, surveys were used and 68
fully completed surveys obtained. The results were
analyzed using quantitative methods - simple
statistics (mean, mode, median) performed on
individual questions, correlation analysis of
individual variables with the dependent variable of
success and Partial Least Squares Regression (PLS-
R) used to build the target framework. Internet-
based e-surveys were used.
Each single data point / measurement was a
distinct Business Intelligence project. This implies
that several measurements could have been
obtained from a single vendor, customer or
individual, provided that each of those
measurements described distinct projects. Each
survey contained optional questions about customer
and vendor company names, project names, place
where the project was performed as well as
whether the survey participants were involved in a
project on the customer or vendor side, and a
precise project role of the participants.
Initial secondary research revealed that some
of independent variables chosen by authors are
correlated. Because of this and the fact that we
aimed at analyzing direct dependency of all
independent variables with success, there was a
very high probability of multicollinearity in the
data. For this reason, a method of data analysis
insensitive for multicollinearity had to be chosen.
Also, because of high number of independent
variables chosen based on phase 1 findings and
relatively low number of observations obtained
through the survey, an appropriate method had to
be chosen. Both requirements combined were
satisfied by Partial Least Squares regression, a
method that “was designed to deal with (…)
regression when data has small sample, missing
values, or multicollinearity” (Pirouz, 2006).
After completing the primary research, we
propose our own framework for BI initiatives
success, using prior research as a starting point.
The framework was separated from the
measurement methods to maintain the flexibility
and applicability of the framework in various
contexts. However, measurement methods were
also proposed. Keeping those two separated will
allow any future studies to propose different
measurement for the variables. Also, a fact of
117
importance is that it is much easier to combine a
theoretical framework with specific measurement
methods if they are kept separated, than to split
them if they are merged in a single framework.
Therefore, we decided to separately propose a
theoretical framework and a specific measurement
method for each of the variables.
4. Analysis and Implications
As mentioned, this study excludes trying to define
what BI project success looks like. Instead, to
determine whether a project is successful or not,
the following interpretation from the survey results
have been made throughout the article.
? Successful project/initiative - Project with
answers “definitely yes” and “rather yes
” to the question Q19:1 - “Was the
project you are describing successful?”
? Non-successful project/initiative - Project
with answers “neutral”, “rather no”
and “definitely no” to the question Q19:1
- “Was the project you are describing
successful?”. That is, non-successful
means unsuccessful or neutral.
The reason behind such a distinction is that this
article aims for developing tools that managers
could use to achieve success. Merely avoiding
failure is not good enough. If the project failed to
achieve success, it cannot be treated as a mild
success or neutral. Instead, it should then be
considered as a non-successful project.
Results from the survey indicate that the
funding of successful initiatives and non-successful
ones differs. Successful initiatives are more likely
to be funded in a phased approach, with separate
budgets for each iteration, while the non-successful
tend to be funded in a traditional way as other IT
projects, i.e. one budget for the entire project.
Iterative development is used by both
successful and non-successful initiatives, but
results from the survey show that for an
overwhelming majority (89%) of the successful
initiatives, each of the iterations provided business
value by themselves. This can be compared to the
non-successful initiatives that used an iterative
approach, where every other initiative had
iterations that provided value by themselves.
A BI solution designed with end users in mind
will naturally have a greater usage than BI
solutions that don’t consider the end users. A
whopping 96% of the successful initiatives in this
study have users that actually use the Business
Intelligence solution. The results for the non-
successful ones vary.
The presence of an overarching, clear
strategic vision for Business Intelligence within the
organization is important for the success of an
initiative. Results show that successful initiatives
exist in an environment with a clear strategic vision
to guide the initiative as compared to non-
successful ones. A total of 81% of the successful
initiatives agrees to this compared to 46% of the
non-successful.
Not only is the existence of a strategic vision
for Business Intelligence important, the business
need that the initiative tries to solve should be
aligned to that vision. Like the results for the
existence of vision, the successful initiatives have
business needs that tend to be more aligned with
the strategic vision than for non-successful ones.
73% of the successful initiatives agree to that.
Corresponding figure for the non-successful
initiatives is 39%.
The survey shows that successful initiatives
often concentrate on choosing the best
opportunities for the project scope. 57% of the
surveyed successful initiatives do this, compared to
23% of the non-successful ones.
Another parameter measured by the study is
whether an expert assigned to a team or made
available to a team, always was the best expert
available. The results from the study reveal,
somewhat surprisingly, that non-successful
initiatives to a larger extent assigns the best expert
available, with 85%. Corresponding number for the
successful ones are 54%.
Issues will occur during the duration of an
initiative, which can be both technological and
non-technological in nature. Both successful and
non-successful initiatives do, to some extent,
encounter non-technological issues, with non-
successful encountering them slightly more often
according to this study. The technological issues
are commonly found in both successful and non-
successful initiatives, but they are somewhat more
common in the non-successful initiatives (77%
compared to 63% reports presence of technological
issues). For all initiatives in the study only slightly
more than 20% found technology related problems
to be the hardest.
Consultants spend time on solving different
kinds of problems, technological as well as non-
technological and for a majority of the initiatives a
larger part of the time is spent on resolving non-
technological problems. This is especially true
when looking at the non-successful initiatives,
which show that almost 40% of the surveyed
initiatives indicated that non-technological issues
took the most time.
4.1 A Statistical model of Business Intelligence
initiatives’ success.
The survey results were analyzed using the Partial
Least Squares regression method. Two separate
series of models were built, one using original,
118
untransformed data and the other, using the data
transformed by centering on mean, dividing by a
factor and applying basic mathematical functions.
More details, and the rationale for performing those
transformations, are provided later in the text.
The models built for original data analysis
used 25 variables defined by answers to survey
question nr 19. For the clarity of information
presentation, variables were named after their
respective survey questions – variable A19_1
corresponded to question 19:1 answers, A19_2
corresponded to question 19:2 answers and so on.
Because question 19:1 asked users whether the
project they described was successful, this variable
was chosen as dependent for all the models, while
every other variable was treated as independent.
Unlike other linear regression methods,
adding more variables does not necessarily imply
an increase of model’s R
2
when PLS is being
used. The method cannot be used to eliminate
insignificant variables, which is easily achieved in
other linear regression methods. Therefore, the
model that used all the available variables was not
necessarily the best. Because the method cannot
eliminate insignificant variables by setting the
linear combination coefficients to 0 (or even close
to 0), models that used only some of the variables
were built.
Choice of variables that explain success
should be made based on the individual
independent variables’ correlation with the
dependent variable. After some experimentation, 5
variables with correlation of success stronger than
0.4 were chosen (absolute value of correlation
coefficient greater than 0.4).
Figure 5: PLS regression model with 5 significant variables
As can be seen the model achieved a R
2
of 0.559,
with changes in independent variables explaining
55.9% of changes in dependent variable. The
variables that enabled these results were obtained
as answers to the following questions:
Question 19_3. Did the end users use the
data warehouse / BI solution provided?
Question 19_6. Were the business needs that
project tried to solve aligned to the
abovementioned strategic vision?
Question 19_10. Did project scope
definition concentrate on choosing the best
opportunities ("low-hanging fruit")?
Question 19_19. If technological issues
were present, were all of them resolved?
Question 19_20. Were non-technological
issues encountered during the project?
PLS is a linear regression method, but it is most
likely that a dependency of success from the factors
is not linear. The possibility of other dependencies
had to be explored. While other, non-linear
regressions could possibly be applied, a lack of any
knowledge of the nature of the dependency would
imply the need to try all of them, which was clearly
beyond the scope of this study. Instead, linear
regression could be applied to the transformed
factors. For example, if linear regression could be
done on squared factors, the outcome would be a
case of square regression for the original factors; if
various powers could be applied to these factors,
the outcome would be polynomial regression and
so on.
It is always possible to choose a function that
will fit the given finite, discrete data set. To avoid
such a pitfall, the transformations had to be limited
to basic mathematical functions – power (including
square root), logarithm, exponential functions,
reciprocal or simple composition of
aforementioned functions. Similar selection
criterion as for original data applied, but the
transformation had to improve the data. The
transformed data was considered better than
original if for a given function f and variable x, f(x)
had a stronger correlation with success than
variable x itself.
119
Before the abovementioned functions could
be applied, initial data transformations were
necessary. The data was centered on mean and
divided by a factor to produce more distinct values
instead of the same 5 values for all questions. None
of those operations changed the correlation of data
prior to applying the functions.
Based on the experience from model 1, the
variables were sorted by correlation. To facilitate
experimenting with the data model, a relative rank
of correlation value was included in variable name
while keeping the question number embedded. For
example, variable V05_N19_20 had the fifth
strongest correlation of all variables (05 in the
name) and was a result of applying transformation
(N in the name) to question Q19:20 answers
(19_20 in the name). For some variables, an
alternative version of the same variable was also
used. For example V18b_A19_15 was an
alternative variable to V18a_N19_15, containing
answers to question Q19:15 without applying the
function, with only the transformations described in
data preparation section above. The decision on
whether to include an alternative variable or not
was made based on how much the correlation has
improved as a result of applying a given function,
with 0.05 as a threshold. However, models
containing alternative versions of the variables
were not any different from those using
transformed variables and will not be presented
here. The summary of functions applied to
variables is presented below:
? V01a_N19_3: q19_3 transformed using
log(2, 1-X/4)
? V02_N19_6: q19_6 transformed using
power(X, 3)
? V03_N19_6: q19_19 transformed using
power(X, 3)
? V04_A19_3: q19_10 transformed using X
? V05_N19_20: q19_20 transformed using
1/power(2,X)
? V06a_N19_5: q19_5 transformed using
exp(X)
? V07_N19_18: q19_18 transformed using
1/power(2,X)
? V08_N19_22: q19_22 transformed using
exp(X)
? V09_N19_25: q19_25 transformed using
sqrt(abs(X))
? V10N19_17: q19_17 transformed using
1/power(X,3)
? V11_N19_8: q19_8 transformed using
sqrt(abs(X))
? V12a_N19_16: q19_16 transformed using
1/power(2,X)
? V13_N19_14: q19_14 transformed using
1/X
? V14_N19_2: q19_2 transformed using
sqrt(abs(X))
? V15a_N19_12: q19_12 transformed using
1/power(X,3)
? V16_N19_7: q19_7 transformed using
power(X, 2)
? V17_N19_11: q19_11 transformed using
1/power(X,3)
? V18a_N19_15: q19_15 transformed using
power(X, 3)
? V19_N19_24: q19_24 transformed using
1/X
? V20_N19_23: q19_23 transformed using
1/X
? V21_N19_9: q19_9 transformed using
sqrt(abs(X))
Figure 6: PLS regression model 2 - equivalent of model 1 with transformed variables
120
In case of untransformed variables, the regression
model with the best fit used only 5 independent
variables. Therefore, the analogical model was
tried using transformed data. This time, the
improvement in R
2
was smaller and the model was
able to explain 0.584 of the success. The resulting
regression model is presented in Figure 6 above.
Figure 7: PLS regression model with the best fit
After some experimentation with various different
models, R
2
=0.611 was achieved by one of them. It
is presented in Figure 7. This was the best model
found using either transformed or untransformed
data. The variables used in this model are
summarized as follows:
? V14_N19_2: Overall, was it possible to
define factors that were critical to how
successful the project was?
? V01a_N19_3: Did the end users use the data
warehouse / BI solution provided?
? V06a_N19_5: Was there a clear strategic
vision for Business Intelligence or data
warehouse?
? V02_N19_6: Were the business needs that
project tried to solve aligned to the
abovementioned strategic vision?
? V16_N19_7: Was project's underlying
business case preceded by a business needs
analysis? This does not mean your party
(vendor or customer) performed this
analysis as part of the project
? V11_N19_8: Did the final solution evolve
beyond initial scope? In case a program was
established, answer on a program level
? V04_A19_10: Did project scope definition
concentrate on choosing the best
opportunities ("low-hanging fruit")?
? V17_N19_11: When an expert in a
particular field was needed, was project
team always granted access to an expert or
enhanced by including one?
? V15a_N19_12: When an expert was
assigned to a team or made available to a
team, was it always the best expert
available?
? V13_N19_14: If there were data quality
issues at source systems, were they known
before the start of BI initiative?
? V12a_N19_16: Were there data quality
issues in BI system or data warehouse?
? V10N19_17: Were data quality issues in BI
system or data warehouse eventually
resolved?
? V07_N19_18: Were technological issues
encountered during the project?
? V03_N19_19: If technological issues were
present, were all of them resolved?
? V05_N19_20: Were non-technological
issues encountered during the project?
? V08_N19_22: Data architecture was
performed at the beginning of BI initiative
(note - data architecture is different from
121
systems architecture or solution
architecture)
? V09_N19_25: Was there a process in place
to measure business benefits?
4.2 Measures Analysis
From the three measures of data warehouse use
proposed in the survey, none were found bad or
even rather bad. Conversely, none of them were
found definitely good. The best from the three was
a weekly number of queries. The fact that it was
positively evaluated in most responses and that
mean value was close to being rather good makes
this measurement method adequate and fairly
universal. It is objective and can be automatically
computed.
Out of the two proposed measurement
methods of access to subject matter experts,
respondents found both “rather good”, with over
two thirds of respondents agreeing with this view.
Despite the fact that the mean response time turned
out to score slightly better, we propose to use both
as measurement for the “access to subject matter
experts”- variable. To compute any of them, a
formal process of request tracking should be made,
but with this process in place, both measures are
objective and easy to compute.
Only one of the three proposed measurement
methods for subject matter expert quality was
found applicable. It turned out to be the same
measurement method as for the access to subject
matter experts. Because of that, the same
arguments about measurement method objectivity
and computability apply.
From the three measures proposed in the
survey for the question of whether there was
enough business staff on the project, we found that
both percentages of business issues resolved within
a business week. The mean time to resolve a
business issue qualify as an adequate measure for
the variable was there enough business staff on a
project. We found little difference in those
measures as they are equally good and equally
objective. They are both calculated in a similar way
therefore either one of them could be used, or even
both at the same time.
Two measures of deliverable size in iterative
development were proposed and we found that only
a percentage of total project time for each iteration
is a good one. However, this measure is only
objective after the project ends. It can be calculated
during the planning phases and project schedule
can be used for that purpose. Unfortunately, this
makes the measure less objective and easy to
manipulate. This diminishes its applicability to
predict initiative success. Despite these drawbacks,
we postulate to use this measure in absence of
better alternatives. Perhaps other measures could be
examined in the future to replace this one, for
instance percentage of total budget allocated or a
similar measure.
Out of three measures proposed for data
quality issues at source systems, it was found that
both counting the total number of issues and the
number of unresolved issues are adequate and
almost identically rated. Therefore no distinction
can be made as to which one of them to use and we
recommend using both. They should be easily
computable if only the formal issue tracking
process is in place. However, both measurement
methods can only be used after the project end as
the number of issues is unpredictable (following
our earlier findings, a number of issues were
discovered during the BI project). For this reason it
would be hard to predict initiative success based
only on the upfront knowledge about source
systems data issues.
From the three measurement methods
proposed for data quality issues in BI system or
data warehouse, no one better than the others were
found. Therefore, it is conclude that this variable
should be measured using the number of
unresolved issues. The fact that this variable could
be objectively measured only after the project ends
makes it unsuitable for use in predicting project
success.
All three measurement methods proposed for
benefits quantification were found adequate for
quantification. Both ROI and IRR calculations
were objective yet difficult to perform at project
start, while relating capabilities to company
strategy was less objective but easier to apply early
in the project. It would be best if financial
indicators could be calculated at project start;
therefore those two methods would be preferred
over the one called strategy, however all three are
worth using and will probably work in practice.
Both measurement methods proposed for
business benefits definition were evaluated
positively by survey respondents with percentage
of business benefits objectively measureable being
significantly better than just a written definition of
the benefits. Both were objective and computable
in the early stages of the project. Although,
percentages were recommended as a better
measurement, we also note that both are likely to
be used in practice as it is unlikely that a company
will have the benefits that are measureable without
first defining them. Therefore, in practice, when the
better measurement method can be applied, then so
can the one that is worse off.
From the two measurement methods proposed
for business benefit measurements, only one was
approved by survey respondents - the presence or
absence of business benefits process owner. This
measurement process can and should be used in
predicting the project success.
122
Figure 8: Analysis of measurement methods of variables (Q22)
For each of the variables analyzed in question 22,
there was at least one measure that was deemed
good enough to be generally used for all projects.
However, almost no measure was found definitely
good. There are two possible explanations for this
phenomenon. Either there is no universally
acceptable measurement method for those variables
or such a variable exists but was not discovered in
this study. Neither of the above possibilities can be
verified using data gathered in this study.
5. Hypotheses Analysis
Hypothesis 1 (first part). It is possible to
identify critical success factors (CSFs) of BI
initiatives in large enterprises.
To determine if factors exist that, if present, are
more likely to lead to initiative success, an analysis
of the correlation between the success and different
factors surveyed was performed. To demonstrate
causality, the statistical evidence of correlation
between success and different key success criteria
are complemented with an argumentation on what
the casual relationship is. To show correlation
between different criteria and the success of the
initiatives, Spearman’s rank correlation
coefficient (Spearman’s ?) was chosen.
Motivation for choosing Spearman’s rank
correlation coefficient includes that it does not
require the relationship to be linear, but also the
fact that it is often used for Likert scales data sets
in e.g. medicine, biochemistry and other sciences
with success (Jamieson, 2004). The
questions/factors with correlation coefficients with
an absolute value of 0.25 or higher are (correlation
coefficient in parenthesis):
? Did project scope definition concentrate on
choosing the best opportunities ("low-
hanging fruit")? (0,43)
? Were non-technological issues encountered
during the project? (-0,42)
? Were the business needs that project tried to
solve aligned to the abovementioned
strategic vision? (0,41)
? Was the project meant to address specific
business needs of the sponsor? (0,38)
? Data architecture was performed at the
beginning of BI initiative (0,37)
? Was there a clear strategic vision for
Business Intelligence or data warehouse?
(0,35)
? Were there data quality issues at source
systems? (-0,30)
? If iterative development was chosen, did
each iteration provide business value by
itself or in connection with previous
deliverables? (0,29)
? Were technological issues encountered
during the project? (-0,28)
? Were data quality issues in BI system or
data warehouse eventually resolved? (0,27)
? Have business benefits been quantified?
(0,26)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Q
2
2
:
1
Q
2
2
:
2
Q
2
2
:
3
Q
2
2
:
4
Q
2
2
:
5
Q
2
2
:
6
Q
2
2
:
7
Q
2
2
:
8
Q
2
2
:
9
Q
2
2
:
1
0
Q
2
2
:
1
1
Q
2
2
:
1
2
Q
2
2
:
1
3
Q
2
2
:
1
4
Q
2
2
:
1
5
Q
2
2
:
1
6
Q
2
2
:
1
7
Q
2
2
:
1
8
Q
2
2
:
1
9
Q
2
2
:
2
0
Q
2
2
:
2
1
Q
2
2
:
2
2
Q
2
2
:
2
3
Q
2
2
:
2
4
Q
2
2
:
2
5
Q
2
2
:
2
6
Analysis of measurement methods of
variables
no answer 1 definitely yes 2 rather yes 3 neutral 4 rather not 5 definitely not
123
From the above it can be conclude that there are in
fact criteria (those factors with a large correlation
with success of the initiative) that, when present,
are associated with a higher likelihood of initiative
success.
Hypothesis 1 (second part). CSFs of BI
initiatives are not the same as for other IS in
large enterprises.
Looking at the top criteria for successful initiatives
with the highest ranking from the surveyed
participants shows that success factors related to
the project actually addressing a specific business
need is most important. Building a solution that is
tailored to user requirements and therefore enables
usage of the Business Intelligence solution is also
important. In the third place comes; documenting
success criteria for the initiative early on in the
initiative.
Resolving technological issues, having a clear
vision to guide the initiative and including a
business need analysis early on are other important
criteria. The survey clearly shows that non-
technological factors rank high amongst the
answers. To compare this to success factors for IS
in general, a framework proposed by Delone and
McLean (Delone & McLean, 1992; 2003) was
used. The original framework proposed by these
authors has been updated with an updated version
(Delone & McLean, 2003). As discussed
previously the factors leading to the success of IS
project in general include information quality,
system quality and service quality. The first thing
that strikes one in the framework proposed by
Delone and McLean is the heavy focus on technical
issues and factors, and little to none concentration
on the softer factors that pertain to management
issues.
One difference from the general IS framework
proposed by Delone and McLean and the findings
from this study is the addition of more non-
technical factors such as the presence of a specific
business need to be addressed by the initiative, a
clear vision to guide the initiative and a pre-
initiative definition of success factors.
Hypothesis 2. Non-technical and non-
technological CSFs play a dominant role in BI
initiatives’ success in large enterprises.
From what has been seen earlier, most of the CSFs
with high correlation with success are in fact non-
technical. Examples are:
1. Successful initiatives do focus, to a larger
extent, on choosing the best opportunities
i.e. “low-hanging fruit” or quick wins.
2. Successful initiatives are more likely to
have a clear strategic vision for Business
Intelligence or data warehouse.
3. Successful initiatives are more often meant
to address a specific business need of the
sponsor.
4. The business needs addressed by successful
initiatives are more often aligned with the
abovementioned strategic vision.
Looking at individual CSFs and the responses from
the survey, one can see that for non-technical
issues, there is a noteworthy difference between the
results for successful initiatives and the non-
successful ones.
One example of this is the area of iterative
development. For those initiatives that used an
iterative approach, each iteration was more likely
to provide business value by itself for the
successful ones. A total of 89% agreed to this,
compared to 50% for the non-successful initiatives.
Another example concerns the presence of a
strategic vision for Business Intelligence. A total of
81% of the successful initiatives had a clear
strategic vision compared to only 46% of the non-
successful ones.
In conjunction with the presence of a strategic
vision for Business Intelligence as a whole, the
successful initiatives are more likely to address a
specific business need that is aligned to this
strategic vision. Some 73% of the successful
initiatives were addressing a specific business need
aligned with the strategic vision. This is almost
twice as common as the 39% of the non-successful
ones.
When looking at the scope definition, the
successful initiatives are more often concentrating
on choosing the best opportunities as depicted by
the chart below. 57% of the successful initiatives
do this compared to the non-successful ones 23%.
All of the above examples are non-technical
factors that are (to a varying degree) more
commonly found in successful initiatives than
those initiatives that fail. This in turn, indicates that
non-technical and non-technological CSFs play a
dominant role in BI initiatives’ success in large
enterprises.
Hypothesis 3. Technology plays a secondary
role and is less critical for BI initiatives’
success in large enterprises
The results from the survey show clearly that non-
technical factors were the hardest to solve. Results
show that for all initiatives in the survey only
slightly more than 20% found the technology
related problems to be the hardest to solve. The fact
that the non-technology related problems were
harder to solve is even more evident when looking
at the non-successful initiatives only, where almost
124
as many as 70% of the initiatives found the non-
technology related problem to be the hardest.
Another view on this is the time spent on the
different types of problems. Again, as we can see a
larger part of the time is spent on resolving non-
technological problems. This is especially true
when looking at the non-successful initiatives, out
of which almost 40% indicate that non-
technological issues took the most time.
When comparing the successful initiatives
with the non-successful ones and the presence of
non-technological issues, it is clear that the
initiatives that fail more often encounter non-
technological problems. One interpretation of this
could be that the presence of non-technological
issues will have a detrimental effect on the
likelihood of initiative success.
Hypothesis 4. It is possible to develop
objective measures for each of CSFs of BI
initiatives in large enterprises.
The result of this study shows that all the CSFs
clearly have one or more measures that are ranked
higher than the rest. The following summarizes the
proposed measurements methods resulting from the
survey.
Area/Variable Proposed method(s)
Data warehouse
use
? Weekly number of queries submitted
Access to subject
matter experts
? Percentage of requests responded within one business day
? Mean response time for a request
Subject matter
expert quality
? Average number of questions needed to be asked per subject
matter issue
? Average number of available documents per subject matter issue
? Percentage of questions answered within one business day
Adequate level of
business staff on
the project
? Percentage of business issues resolved within a business week
? Mean time to resolve a business issue
Size of
deliverable in
iterative
development
? Percentage of total project time for each iteration
Data quality
issues at source
systems
? Total number of issues
? Number of unresolved issues
Quality issues in
BI system or data
warehouse
? Number of unresolved issues
Benefits
quantification
? Calculated ROI
? Proposition on how the business intelligence capabilities enables
the realization of business strategy
? Contribution to the internal rate of return
Business benefits
definition
? Percentage of business benefits that are objectively measureable
Business benefits
measurements
? Presence or absence of business benefits measurements process
owner
Table 1: Summary of variables analysis
6. Conclusions
The hypotheses of this study are focused on the
factors that make Business Intelligence initiatives
successful.
A quantitative approach was used to gather
data to support or reject the above hypotheses and a
primary research survey was used for gathering
data on the above hypotheses. The survey was
conducted online during Q2 of 2011. The findings
can be summarized as follows.
6.1 Non-technological problems dominate in
Business Intelligence projects
Business Intelligence projects wrestle with both
technological and non-technological problems, but
the non-technological problems are found to be
125
both harder to solve sand more time consuming
than their technological counterparts. For all
initiatives in the survey, only slightly more than
20% found the technology related problems to be
the hardest. When looking at the non-successful
initiatives only, almost as much as 70% found the
non-technology related problem to be the hardest.
The other view on this, the time spent on the
different types of problems, also shows that a larger
part of the time is spent on resolving non-
technological problems. This is especially true
when looking at the non-successful initiatives,
almost 40% of which indicate that non-
technological issues drove the most time.
6.2 Successful projects share common factors
Certain factors or attributes of Business
Intelligence projects are correlated with the success
of the initiative. The correlation has been
calculated using Spearman’s rank and shown in
the model that was developed using Partial Least
Squares. By using rational argumentation, it can be
shown that those factors do in fact have a causal
relationship with the likelihood of success. The
highest correlated factors are:
? A project scope definition that concentrate
on choosing the best opportunities ("low-
hanging fruit")
? The alignment of the business needs that
project tries to solve aligned with the
strategic vision of business intelligence
? A project that is meant to address specific
business needs of the sponsor
? Performing data architecture at the
beginning of BI initiative
? Having a clear strategic vision for Business
Intelligence or data warehouse
6.3 Successful and non-successful projects show
differences in certain factors
For a number of different factors, clear differences
can be observed for successful and non-successful
initiatives. This shows that successful initiatives are
doing specific things more frequently than the non-
successful. Factors with great differences are:
? The type of funding of the initiative
? Business value provided by each iteration
? An alignment of the business needs that
project tried to solve aligned to the strategic
vision of Business Intelligence
? A project that is meant to address specific
business needs of the sponsor
? A project scope definition that concentrate
on choosing the best opportunities ("low-
hanging fruit")
6.4 Business Intelligence project have success
factors that differ from IS projects in general
Comparing the critical factors for BI projects that
are correlated with success, or are otherwise more
commonly found in successful initiatives, with the
success factors for general IS projects found in
literature shows that they do in fact differ. One of
the major differences between the results found in
this study is the focus on non-technological factors.
One example of this are the frequently quoted
papers by Delone and McLean (1992; 2003), that
predominately discussed technological factors
while the results of this study point to the non-
technological factors as being more important.
6.5 Clear objective measures for CSFs can be
identified
Clear objective measures that are commonly agreed
upon according to the survey exist for each of the
CSFs analyzed in the survey. For some of the
CSFs, the results are clearly pointing towards one
of the suggested measures, while others can have
multiple measures that claim to be equally good
measures. However, due to practical reasons, to the
limit of the survey size, measurements for only 10
variables were proposed. For the remaining
variables, it is impossible to verify whether the
remaining variables do or do not have objective
measurement methods.
6.6 A statistical model can be constructed that, to a
certain extent, predicts the success of BI projects
A critical success factors framework for Business
Intelligence initiatives have been developed as part
of this study. The results from the survey show that
61% of variability of success can be explained by
changes in independent variables. Limiting the
model to 5 independent variables (out of 17 that
offer the highest explanatory power of the model)
makes the model only slightly less powerful,
allowing it to explain 58% of success variability. A
conclusion from this would be that managers need
only to monitor properly and manage 5 factors to
achieve better probability of success, than the 20%-
50% currently achievable in Business Intelligence.
For a Business Intelligence project to be successful:
1. Business Intelligence solution must be built
with end users in mind, as they need to use
it
2. The Business Intelligence system needs to
be closely tied to a company’s strategic
vision
3. Project needs to be properly scoped and
prioritized to concentrate on best
opportunities first
126
4. Although technological issues are
encountered, all of them need to be solved
5. Non-technological issues should be avoided
as they can hinder the success of the BI
initiative
Below is a summary of the support for the four
hypotheses of this study.
Hypothesis 1. It is possible to identify critical
success factors (CSFs) of BI initiatives and
they are different from the CSFs of
information systems in general. (supported).
Hypothesis 2. Non-technical and non-
technological CSFs play a dominant role in BI
initiatives’ success for initiatives (supported).
Hypothesis 3. Technology plays a secondary
role and is less critical for BI initiatives’
success for initiatives (supported).
Hypothesis 4. It is possible to develop
objective measures for each of CSFs of BI
initiatives primarily (partially supported - the
survey did not cover measurement analysis all
CSFs).
7. Future research
The survey data received describes project
primarily from Poland and primarily from a vendor
perspective. Conducting a similar research on a
more generalized sample spanning projects
worldwide with equal emphasis on vendors and
customers would be an obvious next step. It would
verify if the relationship between success and its
factors is a phenomenon local to Poland or rather,
as we suspect, general in nature.
Several other frameworks were reviewed in
this study. A carefully constructed study with
properly chosen variables would allow comparing
each framework's validity with regards to
contemporary Business Intelligence projects seeing
if one is significantly stronger than the rest of them.
This is not a trivial task, because of different
treatment of success and its factors, using proxy
variables for success or even modeling
dependencies between various parts of a success.
The word critical in its strict mathematical
meaning would suggest that if even one of critical
success factors is missing, success cannot happen.
Such a relationship is impossible to model using
linear regression. Even the application of data
transformation must be taken into account, similar
to the achieved linear combinations of some
functions of variables. Therefore, another study
could be performed to search for a model that
would better explain the success using the variables
used by us. It is possible that the variables used in
this study would be able to predict more of the
Business Intelligence success when organized in a
different fashion
Because almost 40% of success variability
remains unexplained in this model, it is possible
that it can be attributed to the weakness of the
linear relationship assumed in the model, to the
missing variable that remains unidentified or even
to a random factor that is influenced only by
chance and which cannot be controlled by
management. Before a model with higher
explanatory power is constructed, it is appropriate
to keep looking for a missing variable that might
potentially account for a significant portion of
unexplained success variability.
Although this work proposes several objective
measurement criteria for independent variables,
most of them cannot be considered universally
applicable in all (or even most) Business
Intelligence projects. Therefore, another study
could be conducted to verify whether such
universal and objective measures exist and attempt
to define them. In case such measures do not exist,
perhaps there is a set of measures for each variable
such that at least one of the quantification methods
would be universally applicable.
One of the delimitations in this work was lack
the treatment of NPV, ROI and other financial
measures of Business Intelligence projects. Since
success factors are different for BI than for IS in
general, perhaps Business Intelligence managers
could also employ different methods for calculating
financial measures than in the case of other IS. In
case such differences are identified, appropriate
methods should be proposed and aforementioned
differences should be uncovered.
Refrences
Adelman, S. 2003. Impossible Data Warehouse
Situations. Solutions from the Experts. Boston,
MA: Pearson Education.
Adèr, & Mellenbergh, G. 2011. Advising on
Research Methods: A consultant's companion.
3
rd
ed. Huizen, The Netherlands: Johannes van
Kessel Publishing.
Ariyachandra, T., & Watson, H. 2006. Which Data
Warehouse Architecture Is Most Successful?
Business Intelligence Journal , 11 (1).
Babbie, E. 2009. The Practice of Social Research.
Wadsworth Publishing.
Beal, B. 2005. Report: Half of data warehouse
projects to fail. Retrieved May 22, 2011, from
Search CRM:http://searchcrm.techtarget.com/news/1066086/
Report-Half-of-data-warehouse-projects-to-fail
Chen, L., Soliman, K., Mao, E., & Frolick, M.
2000. Measuring user satisfaction with data
warehouses: An exploratory study. Information
& Management , 37 (3), 103-110.
Daniel, D. 1961. Management Information Crisis.
Harvard Business Review.
127
Delone, W., & McLean, E. 1992. Information
Systems Success: The Quest for the Dependent
Variable. Journal of Information System
Research, 3 (1), 60-95.
Delone, W., & McLean, E. 2003. The DeLone and
McLean Model of Information Systems
Success: A Ten-Year Update. Journal of
Management Information Systems , 19 (4), 9-
30.
Duncan, W. 1996. A Guide to the Project
Management Body of Knowledge. Sylva, North
Carolina: PMI Publishing.
Ellison, R., & Moore, A. 2003. Trustworthy
Refinement Through Intrusion-Aware Design.
Carnegie Mellon Software Engineering
Institute.
Garthwaite, P. 1994. An Interpretation of Partial
Least Squares. Journal of the American
Statistical Association , 89 (425), 122-127.
Hwang, M., & Xu, H. 2008. A Structural Model of
Data Warehousing Success. Journal of
Computer Information Systems , 48-56.
Inmon, W. H. 2002. Building the Data Warehouse,
3rd ed. New York, NY: John Wiley & Sons.
Jamieson, S. 2004. Likert scales: how to (ab)use
them. Medical Education, 1217-1218.
Kimball, R., & Ross, M. 2002. The data warehouse
Toolkit, 2 ed. New York, NY: John Wiley and
Sons, Inc.
Kimball, R., Ross, M., Thornthwaite, W., Mondy,
J., & Becker, B. 2008. The Data Warehouse
Lifecycle Toolkit, 2nd ed. Indianapolis, IN:
Wiley Publishing.
Laskowski, N. 2001. Gartner BI Summit: Business
intelligence benefits lie in orchestration.
Retrieved May 22, 2011, fromhttp://searchbusinessanalytics.techtarget.com/ne
ws/2240035533/Gartner-BI-Summit-Business-
intelligence-benefits-lie-in-
orchestration?asrc=EM_NLN_13851353&track
=NL-544&ad=832107
Legodi, I., & Barry, M.-L. 2010. The current
challenges and status of risk management in
enterprise data warehouse projects in South
Africa. PICMET 2010 Technology Management
for Global Economic Growth.
Linstone, H. A., & Turoff, M. 2002. The Delphi
Method Techniques and Applications.
Markarian, J., Brobst, S., & Bedell, J. 2007.
Critical Success Factors Deploying Pervasive
BI.
Meehan, P. 2011. Patrick Meehan speech during
Gartner Business Intelligence Summit 2011.
London, UK. Retrieved fromhttp://link.brightcove.com/services/player/bcpid
1156010110?bctid=741282639001
Merriam-Webster Dictionary. 2011. Retrieved May
24, 2011, from Merriam-Webster.com
Numminen, E. (2010). On the Economic Return of
a Software Investment omManaging Cost, PhD
Dissertation.
Pirouz, D. 2006. An Overview of Partial Least
Squares. Retrieved May 27, 2011, from Social
Science Research Network:http://ssrn.com/abstract=1631359
Pisello, T. 2001. IT Value Chain Management rom
Social Science Research Network: htt Alinean,
LLC.
Popovic, A., & Jaklic, J. 2010. Benefits of Business
Intelligence System Implementation: An
Empirical Analysis of the Impact of Business
Intelligence System Maturity on Information
Quality.
Ringle, C., Wende, S., & Will, S. 2005. SmartPLS
2.0 (M3) Beta. Hamburg.
Shin, B. 2003. An Exploratory Investigation of
System Success Factors in Data Warehousing.
Communications of the Association for
Information Systems , 6 (4), 141-170.
Solberg Søilen, K., & Hasslinger, A. 2009. How
application integration, security issues and
pricing strategies in business intelligence shape
vendor differentiation. ECIS 2009: THIRD
EUROPEAN COMPETITIVE INTELLIGENCE
SYMPOSIUM, (pp. 252-260). Stockholm.
Walton, E. J., & Dawson, S. 2001. Managers J., &
Dawson, S. MPETITIVE INTELLIGENCE
SYMPOSIUMtrategiesJournal of Management
Studies, 173-179.
Watson, H., Annino, D., Wixom, B., Avery, L., &
Rutherford, M. 2001. Current Practices in Data
Warehousing. Information Systems
Management, 18 (1), 47-55.
Wixom, B., & Watson, H. 2001. An Empirical
Investigation of the Factors Affecting Data
Warehousing Success. MIS Quarterly , 25 (1),
17-41.
Yeoh, W., & Koronios, A. 2010. Critical Success
Factors for Business Intelligence Systems.
Journal of Computer Information Systems, 50
(3), 23-32.
Yeoh, W., Gao, J., & Koronios, A. 2007. Towards
a CSF Framework for Implementation of
Business Intelligence Systems: A Delphi Study
in Engineering Asset Management
Organisations. Research and Practical Issues of
Enterprise Information Systems II. Beijing,
PRC.
doc_281253399.pdf
Business Intelligence can bring critical capabilities to an organization, but the implementation of such capabilities is often plagued with problems.
Key Success Factors in Business Intelligence
Szymon Adamala * and Linus Cidrin **
* Business Intelligence and data warehousing consultant, Poland,
[email protected]
** Management consultant, Sweden,
[email protected]
Received 25 May 2011; received in revised form 3 June 2011; accepted 25 November2011
ABSTRACT: Business Intelligence can bring critical capabilities to an organization, but the implementation of
such capabilities is often plagued with problems. Why is it that certain projects fail, while others succeed? The
aim of this article is to identify the factors that are present in successful Business Intelligence projects and to
organize them into a framework of critical success factors. A survey was conducted during the spring of 2011 to
collect primary data on Business Intelligence projects. Findings confirm that Business Intelligence projects are
wrestling with both technological and non-technological problems, but the non-technological problems are
found to be harder to solve as well as more time consuming than their counterparts. The study also shows that
critical success factors for Business Intelligence projects are different from success factors for Information
Systems projects in general. Business Intelligences projects have critical success factors that are unique to the
subject matter. Major differences can be found primarily among non-technological factors, such as the presence
of a specific business need and a clear vision to guide the project. Success depends on types of project funding,
the business value provided by each iteration in the project and the alignment of the project to a strategic vision
for Business Intelligence at large. Furthermore, the study provides a framework for critical success factors that,
explains sixty-one percent of variability of success for projects. Areas which should be given special attention
include making sure that the Business Intelligence solution is built with the end users in mind, that the Business
Intelligence solution is closely tied to the company’s strategic vision and that the project is properly scoped and
prioritized to concentrate on the best opportunities first.
Keywords: Business Intelligence, Data Warehouse, Critical Success Factors, Enterprise Data Warehouse,
Success Factors Framework, project risk management
1. Introduction
This article originated from a Master’s Thesis that
the authors wrote as part of their MBA studies at
Blekinge Institute of Technology, supervised by
Dr. Klaus Solberg Søilen.
The aim of the article is to develop the tools
necessary to analyze, predict and manage the
success of Business Intelligence, data warehousing
and competitive intelligence initiatives in
contemporary organizations. “There is no reason
that each organization, as it begins and continues to
develop data warehouse projects, must wrestle with
many of the very difficult situations that have
confounded other organizations. The same
impossible situations continue to raise their ugly
heads, often with surprisingly little relation to the
Available for free online athttps://ojs.hh.se/
Journal of Intelligence Studies in Business 1 (2011) 107-127
108
industry, the size of the organization, or the
organizational structure” (Adelman, 2003). If
there is indeed little correlation between success
and the environmental factors of those initiatives,
the cause of success must be related to something
else. It is worth investigating whether the difficulty,
complexity of initiatives and the high probability of
failure are related to some factors inherent to
Business Intelligence itself.
In fact, most of the Business Intelligence and
data warehouse projects fail and the failure rate is
estimated to be between 50-80% (Beal, 2005;
Meehan, 2011; Laskowski, 2001; Legodi & Barry,
2010). Why do so many of these projects fail? “
The warehouse is used differently than other data
and systems built by information systems”
(Inmon, 2002). Therefore, the available knowledge
on how to manage ordinary projects in the
information systems domain should not be
expected to be readily applicable to the area of
Business Intelligence.
Our initial information search revealed that
Business Intelligence has been covered in a large
number of academic articles and books. However,
Business Intelligence success is not widely treated.
Few articles attempt to analyze what the factors are
which influencing BI. Some of them have
developed frameworks, but there is no proven
framework that can be relied upon for this
particular problem.
The success of BI initiatives undertaken by
companies depends on various factors. Because BI
implementation depends on the successful use of IT
resources, some of those factors are undoubtedly
technological. However, Information Technology
is merely a tool in implementing a BI solution. BI
success depends chiefly on organizational and
process factors that business administration focuses
on. Despite differences in BI systems between
various industries, those factors should be
identified and classified. The existence of a link
between individual factors or their combinations
and the success of BI initiatives should be verified.
Moreover, success factors could be classified and a
method of measurement could be devised for each
individual factor. As the management thinker Peter
Drucker once said, “what gets measured, gets
managed”. Currently, BI initiatives have a
relatively low success ratio. The successful ones
achieve ROI in excess of 400% (according to some
vendors). This figure is even more impressive
when taking into account the (traditionally capital)
investments in the infrastructure and equally high
(traditionally operational) expenses on
development projects, operations, consulting
services etc. An ability to assess the chances of
success prior to engaging in the initiative would
have two practical benefits:
1. Preventing some of the initiatives from
commencing if they score low in the
framework of success factors or allowing
them to focus on critical issues that otherwise
would have caused a failure - as a result, the
success ratio of all BI initiatives improves.
2. Facilitating management of the investment
in Business Intelligence and lowering the
entry barriers (given the high level of
investment, 50% probability of failure might
be too high for smaller companies if the loss
resulting from failure push the company into
bankruptcy).
The article focus on the following hypotheses:
Hypothesis 1. It is possible to identify critical
success factors (CSFs) of BI initiatives and
they are different from the CSFs of
information systems (IS) in general.
Hypothesis 2. Non-technical and non
technological CSFs play a dominant role in BI
initiatives’ success.
Hypothesis 3. Technology plays a secondary
role and is less critical for BI initiatives’
success.
Hypothesis 4. It is possible to develop
objective measures for each of CSFs of BI
initiatives.
If the abovementioned hypotheses can be proved,
CSFs of BI initiatives can create a framework that
can be used to predict the success in early stages of
BI initiatives.
It should be noted that, even though this study
is primarily focused on vendors in Poland, the
conclusions from this study may have relevance
and be applicable to other countries as well,
especially since many of the companies studied
were global enterprises and other aspects of
projects and IT are global: Outsourcing, offshoring,
nearshoring and other forms of international
operations. the differences in survey results
between Poland and other countries were not
significant and drawing conclusions from the
sample we gathered, we claim that our findings are
also applicable to Business Intelligence projects in
larger enterprises worldwide. The reason for
choosing Poland as the primary country is that one
of the authors has been working there for many
years and had good access to empirical data.
There is some ambiguity around the use of the
terms data warehouse and Business Intelligence
among practitioners. Historically, data warehousing
meant “the overall process of providing
information to support business decision making”
(Kimball, Ross, Thornthwaite, Mondy & Becker,
2008, p. 10). Following these authors, “delivering
the entire end-to-end solution, from the source
extracts to the queries and applications that the
109
business users interact with” (Kimball et
al.,,2008, p. 10) has always been a fundamental
principle. However, the term Business Intelligence
coined in the 1990s meant only reporting and
analysis of data stored in the warehouse, effectively
separating the two terms. Even now, there is no
agreement in the industry – some “refer to data
warehousing as the overall umbrella term, with the
data warehouse databases and BI layers as subset
deliverables within that context” (Kimball, et al.,
2008, p. 10), while others “refer to business
intelligence as the overarching term, with the data
warehouse relegated to describe the central data
store foundation of the overall business intelligence
environment” (Kimball et al., 2008, p. 10).
Because the aim of this article is not to define
the term of Business Intelligence, but rather to
investigate its success factors, both conventions are
accommodated. The literature referring both to data
warehouse systems/projects and Business
Intelligence systems have been reviewed. There is
no difference in their relevance to our study,
especially when taken into account the industry
practice that even “though some would agree that
you can theoretically deliver BI without a data
warehouse, and vice versa, that is ill-advised /…/.
Linking the two together in the DW/BI acronym
further reinforces their dependency” (Kimball et
al., 2008, p. 10). Despite this argumentation,
Business Intelligence is used in this article as an
overall term.
The research is concentrated mainly on the
factors that lead to the success of BI initiatives,
cause its success (or are able to prevent it), or are
otherwise correlated with the success. However,
little attention is paid to the definition of the
success itself or its measurement. According to
Delone and McLean (1992; 2003), success in
business entails net benefits. To properly define
success, those net benefits need to be measured.
The authors’ goal of obtaining objective
measurement criteria would call for a method to
compute the NPV of a BI initiative. A similar
subject (without the emphasis on BI, but rather for
software in general) has recently been a topic of a
PhD thesis at Blekinge Institute of Technology
(Numminen, 2010). Clearly, calculating the NPV
of BI initiatives will be out of scope for this article.
Therefore, the success itself will not be defined as
clearly as the authors should want to at first. This
would instead be a suggestion for further research
in this area.
Several of the models proposed in the
literature (Delone & McLean, 1992; 2003; Hwang
& Xu, 2008; Wixom & Watson, 2001) analyze the
interdependency between independent variables.
Furthermore, none of the abovementioned authors
propose a direct correlation with success for those
variables that were used to explain variability of
other independent variables. The goal of this study
is to enumerate all the CSFs of BI projects, not to
analyze how they depend on each other. For each
of the independent variables analyzed, a direct
dependency between success and that variable is
analyzed.
2. Literature review
Dr Ralph Kimball is credited with developing
arguably the most successful data warehouse
architecture, the dimensional modeling. The
success criteria as described by Kimball are named
“readiness factors”. The author suggests giving
up the initiative, if some of the criteria are missing
or insufficient. “Readiness shortfalls represent
project risks; the shortfalls will not correct
themselves over time. It is far better to pull the plug
on the project before significant investments have
been made than it is to continue marching down a
path filled with hazards and obstacles” (Kimball
R., Ross, Thornthwaite, Mondy, & Becker, 2008, p.
16).
Let us examine the existing frameworks that
might be applicable for explaining Business
Intelligence success.
2.1 Information Systems Success
The most obvious first choice when trying to
discover BI success factors is to look at
information systems (IS) in general. There exists an
excellent framework proposed by Delone and
McLean (Delone & McLean, 1992; 2003) that has
been widely cited, revisited, criticized, validated or
extended in hundreds of articles (Delone &
McLean, 2003). The original framework proposed
by Delone and McLean (1992) has been
reexamined in the light of that subsequent research,
resulting in a slightly updated version (Delone &
McLean, 2003) that will be discussed here. The
proposed framework defines IS success in terms of
system use, user satisfaction and net benefits
whereas the factors leading to the success
encompass information quality, system quality and
service quality. The interdependences between
those variables are depicted in Figure 1.
110
Figure 1: Delone&McLean model for IS success (Delone & McLean, 2003)
Our study did not measure system quality and
service quality at all, as they are inapplicable to
Business Intelligence. As Solberg Søilen and
Hasslinger (2009) found, vendors of BI tools do not
differentiate their products and tools (other than
adopting different definitions of Business
Intelligence). Therefore, system quality should not
be a factor in the Business Intelligence domain.
Because Business Intelligence systems are built to
present data and facilitate its analysis, data quality
is probably the most important factor. From the “
second tier” variables, intention to use and user
satisfaction should play a small role in Business
Intelligence. If BI is tied to strategic vision for the
entire company, users are obliged to use the
solution and not their own data sources. Therefore,
use alone as a variable plays a role.
One major drawback with this framework that
we identified is the concentration on technology.
The findings of our research dictate that the success
of BI initiatives relies on numerous factors that are
within the management domain rather than on
technological factors. Therefore, although the
Delone and McLean model of IS success has been
applicable to many specific information systems,
even those that have been unforeseen when the
framework was originally constructed (Delone &
McLean, 2003), we think it cannot be applied to
Business Intelligence systems without due changes.
Delone and McLean’s framework not only
does not propose any specific measurement
methods, but also avoids specific variable
definitions, leaving them to be concretized at the
time of framework application, depending on the
specific context (Delone & McLean, 2003).
2.2 Critical Success Factors for BI Systems
“The implementation of a BI system is not a
conventional application-based IT project (such as
an operational or transactional system), which has
been the focus of many CSF studies” (Yeoh &
Koronios, 2010). Having noted that, the authors
proceeded to propose a framework that included
some of the critical success factors or success
variables from Delone & McLean (1992) as a part.
System quality, information quality and system use
were grouped together and labeled “infrastructure
performance”. As an equal factors group, process
performance was proposed, encompassing classical
project management variables like budgets and
time schedules. The authors emphasized a different
set of factors, divided into three broad categories –
organization (vision and business case related
factors, management and championship related
factors), process (team related factors, project
management and methodology related factors,
change management related factors) and
technology (data related factors, infrastructure
related factors). All those factors cause business
111
Figure 2: Yeoh & Koronios model of success in BI (Yeoh & Koronios, 2010)
orientation, which in turn, together with before
mentioned infrastructure and process performance
factors lead to implementation success and,
subsequently, to perceived business benefit. This
framework is illustrated in Figure 2.
A summary of success factors across all the
dimensions with the highlight of the most
important points:
Organizational dimension
? Committed management support and
sponsorship
? Clear vision and well-established business
case
Process dimension
? Business-centric championship and
balanced team composition
? Business-driven and iterative development
approach
? User-oriented change management
Technological dimension
? Business-driven, scalable and flexible
technical framework
? Sustainable data quality and integrity
This framework is considerably better suited for
Business Intelligence systems than the one
presented by Delone and McLean (2003).
However, it has a few drawbacks. First, there are
no specific measurement criteria proposed. Since
several of the variables are defined in a way so that
very different measurement criteria can be defined
or it would be hard to devise measures other than
the Likert scale, the framework would be
impractical to apply and the results might depend
on the subjective opinions of those that provide or
calculate variable’s values, resulting in the false
prediction of BI initiative success. Another
drawback is an artificial reliance on the Delone and
McLean (2003) model of IS success. Some
infrastructure performance factors (system quality,
information quality) belong under the technology
category (infrastructure related factors or data
related factors) and should not be repeated at all or
the causal dependency should be included in the
framework.
Further criticism of this article (Yeoh &
Koronios, 2010) can be based on the way inclusion
of technology related factors was performed. “
Business-driven, scalable and flexible technical
framework” and particular variables that fall into
this category are too close to the way the vendor
who delivered the solution gives advice on his or
her product. Also, the fact that this article (Yeoh &
Koronios, 2010) was based on a single project
112
plays down the objectivity of its findings. As our
survey later revealed, technical issues are of
secondary importance.
2.3 Impossible Data Warehouse Situations.
Why are the success criteria so important? As we
have already mentioned in the introduction, some
data warehouse implementations fail. It is possible
that many of these ‘failures’ would be
considered successes (or at least to be positioned
someplace between failure and success). The
criteria for success are often determined afterwards.
This is a dangerous practice since those involved
with the project don’t really know their targets
and will make poor decisions about where to put
their energies and resources (Inmon, 2002).
According to Adelman (2003), the example
measures of success include the following
elements:
? The data warehouse usage.
? Usefulness of the data warehouse, including
end user satisfaction.
? Acceptable performance as perceived and
experienced by end users.
? Acceptable ROI - benefits justifying the
costs.
? Relevance of data warehouse application.
? Proper leverage of a business opportunity
(i.e. establishing competitive advantage
thank to the application of data warehouse).
? Timely answers to business questions.
? Higher quality / cleanliness of data (than in
the source systems).
? Initiation of changes in the business
processes that can be improved as a result of
the use of data warehouse (Adelman, 2003,
p. 5 & 40).
Douglas Hackney emphasized the critical
importance of establishing success measures – “
The ‘build it and they will come’ approach (…)
does not work in data warehousing” (Adelman,
2003, p. 41). He further suggests starting with
business “pain” with the available data. The
words “life-threatening” seem to accurately
describe the problem that need to be chosen to
solve (Inmon, 2002, p. 41). An example of a
business “pain” might be the “loss of revenue
or excessive operating costs” (Inmon, 2002, p.
42).
The main theme of the book of Adelman
(2003) is not the success factors of Business
Intelligence initiatives. Instead, the author
concentrates on various difficult situations and
proposes methods to overcome those adversities.
Although he does not name those factors as
influencing the success itself, some of them should
at least be evaluated for possible inclusion in the
final framework of this study. According to the
author, all difficulties can be organized in one of
the following categories (the first seven represent
management issues, the rest is concerned with the
technical side):
1. Management issues
2. Changing requirements and objectives
3. Justification and budget
4. Organization and staffing
5. User issues
6. Team issues
7. Project planning and scheduling
8. Data warehouse standards
9. Tools and vendors
10. Security
11. Data quality
12. Integration
13. Data warehouse architecture
14. Performance
From the factors identified in the book of Adelman
(2003), we found a few that were less important
than others, while some were not considered at all
in our survey as separate variables. For example,
architecture, performance, security, tools and
standards were considered as one broad variable
technical issue in our research and were deemed of
secondary importance. Data quality, on the other
hand, was measured in our survey separately for
source systems and target solution, both being
found to have importance.
Adelman ( 2003) devote at least equal
attention to technical issues in general as to all non-
technical issues, the latter being mentioned in less
detailed discussions. Broad and general categories
of “user issues”, “team issues”, “
organization and staffing” etc. make detailed
comparison with our study difficult. Also, a
different terminology plays a role here. What the
author considers measures, our study call variables.
For example, “data warehouse usage” was a
measure in the book (Adelman, 2003) while as we
propose three distinct measurement methods,
namely number of queries per period, number of
logons per period and number of users per period.
2.4 Competing Value Model
Since different stakeholders will look at different
criteria to determine if the initiative was a
successful one, different perspectives may need to
be taken into account. This issue of potentially
conflicting criteria has been discussed by Walton
and Dawson in their Competing Value Model
which organizes criteria in different dimensions
(Walton & Dawson, 2001). This could be present
in BI initiatives as well, where different
113
stakeholders can have different non-overlapping
criteria for success; for example a system integrator
can have different objectives for engaging in a BI
initiative, different from the end user organization
(the customer).
This study did not attempt to define BI
initiative success, but considered using this kind of
classification of criteria for different dimensions,
such as technical and non-technical. The definition
of success of BI initiatives is not part of this study.
One consequence of the findings of the article by
Walton & Dawson (2001) is that the development
of a framework allows for the interest of different
stakeholders to be organized. This problem is left
for future studies.
2.5 A Critical Success Factor Framework for
Implementing Business Intelligence Systems
Qualitative studies have been used in the area of
critical success factors in Business Intelligence
many times. One of these is a Critical Success
Factor Framework for Implementing Business
Intelligence Systems (Yeoh, Gao, & Koronios,
2007) where the authors performed a Delphi study
covering 15 BI systems experts to define a critical
success factor framework for implementing
business intelligence systems. The authors propose
a framework that is organized into seven
dimensions covering 22 factors. The seven
dimensions of this article (Yeoh, Gao, & Koronios,
2007) are:
? Commitment management support and
championship
? User-oriented change management
? Business vision
? Project planning
? Team skills and composition
? Infrastructure-related dimensions
? Data related issues
The Delphi panel rates the factors in those
dimensions and a mean is calculated for each
dimension. The above list of dimensions is ordered
in a falling mean ranking, starting with the
dimension with the highest rating. We see that
more technical dimensions can be found at the very
bottom of the list, while as non-technical
dimensions are found higher up in the list.
The paper by Yeoh et al. (2007) demonstrates,
in line with the result from this study, that
management championship and the presence of a
clear vision for Business Intelligence is important
when executing a Business Intelligence initiative.
While the result from this study seems to indicate
that the technical factors are more important than
the non-technical factors, the authors of the paper
include two “technical” dimensions out of seven.
It should be said that the ratings of the two
technical dimensions are the lowest out of the
seven, which is well in line with the findings of this
study.
2.6 Risk Management in Enterprise Data
Warehouse Projects in South Africa
The paper starts with the claim that data
warehousing implementation projects have high
estimated failure rates, up to about 50% (Legodi &
Barry, 2010). The objective of the paper and the
study was to investigate the main areas of risk for
these projects, and a Delphi method was used for
the data collection. The study was performed in
South Africa. Results from the study are in the
form of the main problems and the success factors
for this type of project. The identified factors that
have the greatest impact on success from the study
(Legodi & Barry, 2010) are (listed in priority
order):
1. Scope Creep
2. Uncontrolled Finances
3. Poor Communication
4. Stake Holder Non-Involvement
5. Skills Shortage
6. Unavailability of Tools and Technology
7. Uncontrolled Quality of Deliverables
8. Poor, Wrong or No Leader
9. Technical Difficulties
10. Legal Difficulties
We note that the first technical difficulties can be
found in the ninth item.
(Legodi & Barry, 2010) show, like the results
from this study, that non-technical factors affect the
success of a project the most. Out of the ten factors
Legodi and Barry (2010) identified, only one were
of technical nature and placed as the ninth factor
(technical difficulties). The fourth item,
stakeholder non-involvement, can be comparable to
the importance of vision and addressing a specific
business need of the sponsor. While the paper
surveyed a more narrowly defined geography
(South Africa), the finding that non-technological
factors dominate is shared between that paper and
this study.
2.7 An Exploratory Investigation of System Success
Factors in Data Warehousing
The variables used in a study by Shin (2003) are
system throughput, ease of use, ability to locate
data, access authorization, data quality (subdivided
into 4 more detailed categories of
recency/currency, level of detail, accuracy and
consistency), information utility, user training and
user satisfaction, with the latter is being used as
dependent variable. The data was gathered from a
single large US enterprise, based on a single
114
project, therefore even the author agrees that this
study must be treated as a case study (Shin, 2003,
p. 157). The authors found that 70% of end user
satisfaction could be explained by the independent
variables that were measured.
Shin (2003) approaches data warehousing
success in a different way than in the above study,
treating end user satisfaction as a proxy for project
success. This is also inconsistent with Delone and
McLean approach (Delone & McLean, 1992; 2003)
which Shin cited and claimed to be “the most
influential model in conducting research on
information systems success factors” (Shin, 2003,
p. 144). The variables used are mainly different
from the inspected ones, with only data quality
being common with this study. Variables used by
Shin (2003) describe predominantly technological
factors. As in this study actual use is only one of
the variables, the results are hardly comparable.
Also, a case study approach concentrated on
success within just one company makes this article
(Shin, 2003) less valuable for creating a larger data
warehousing or Business Intelligence success
criteria framework.
A Structural Model of Data Warehousing
Success by Hwang & Xu (2008) analyzes several
factors influencing data warehousing success.
Success is not measured as a separate variable.
Instead, following Delone and McLean (1992;
2003), benefits and quality are used as proxy, with
a distinction between individual and organizational
benefits as well as system and information quality.
The independent variables are; operational factor,
technical factor, schedule factor and economic
factor. The authors propose at least two measures
for each variable, dependent and independent. The
model (Figure 3) is able to explain between 27%
(system quality) and 41% (organizational benefits)
of the dependent variables variability.
Figure 3: Hwang & Xu structural model of data warehousing success (Hwang & Xu, 2008)
The most prominent difference in approaches is the
definition of what a variable is and what a measure
is. Many measures from this article (Hwang & Xu,
2008) are variables in our study, for example
business benefits definition, business benefits
measurability or scoping of a project. The aim for
objective measurement methods were not one of
the goals of Hwang & Xu (2008) and it is unclear
how measures proposed in their work should be
performed in practice. The authors were overly
reliant on the model of IS success (Delone &
McLean, 1992; 2003), obtaining little explanatory
power for their model. From the way the model
was constructed, it is also hard to compare whether
technical or non-technical factors play major role in
data warehousing success.
An Empirical Investigation of the Factors Affecting
Data Warehousing Success (Wixom & Watson,
2001) examined the factors influencing data
warehousing success. Authors verified several
hypotheses. Findings can be summarized as
follows:
Hypothesis 1a. A high level of data quality
will be associated with a high level of
perceived net benefits (supported).
Hypothesis 1b. A high level of system quality
will be associated with a high level of
perceived net benefits (supported).
Hypothesis 2a. A high level of organizational
implementation success is associated with a
high level of data quality (not supported).
115
Hypothesis 2b. A high level of organizational
implementation success is associated with a
high level of system quality (supported).
Hypothesis 3a. A high level of project
implementation success is associated with a
high level of data quality (not supported).
Hypothesis 3b. A high level of project
implementation success is associated with a
high level of system quality (supported).
Hypothesis 4a. A high level of technical
implementation success is associated with a
high level of data quality (not supported).
Hypothesis 4b. A high level of technical
implementation success is associated with a
high level of system quality (not supported).
Hypothesis 5. A high level of management
support is associated with a high level of
organizational implementation success
(supported).
Hypothesis 6a. A strong champion presence is
associated with a high level of organizational
implementation success (not supported).
Hypothesis 6b. A strong champion presence is
associated with a high level of project
implementation success (not supported).
Hypothesis 7a. A high level of resources is
associated with a high level of organizational
implementation success (supported).
Hypothesis 7b. A high level of resources is
associated with a high level of project
implementation success (supported).
Hypothesis 8a. A high level of user
participation is associated with organizational
implementation success (supported).
Hypothesis 8b. A high level of user
participation is associated with project
implementation success (supported).
Hypothesis 9a. A high level of team skills is
associated with project implementation success
(supported).
Hypothesis 9b. A high level of team skills is
associated with technical implementation
success (not supported).
Hypothesis 10. High-quality source systems
are associated with technical implementation
success (supported).
Hypothesis 11. Better development technology
is associated with technical implementation
success (supported).
A valuable part of in this article (Wixom &
Watson, 2001) that even led to verifying the
hypotheses proposed in this paper, was the
statistical model using partial least squares
regression. The resulting model can be seen in
Figure 4.
Figure 4: Wixom&Watson statistical model of DW success (Wixom & Watson, 2001)
R
2
values obtained by Wixom and Watson (2001),
(Figure 4) ranging between 0.016 to 0.435, means
that independent variables used in the model
provide limited explanatory power for dependent
variables. This makes it impossible to compare
findings with our study. The model relies on the IS
success model by Delone & McLean (1992) which
defines system success using a structure of proxy
variables – data and system quality, and benefits.
Variables used covered both technical and non-
technical factors influencing data warehouse
implementation success. The way the model was
116
constructed makes it impossible to verify whether
technical or non-technical factors influence the
success more strongly. The article (Wixom &
Watson, 2001) does not attempt to propose specific
measurement criteria for the variables used.
2.8 Current Practices in Data Warehousing
The study conducted in the article of Watson,
Annino, Wixom, Avery, & Rutherford (2001)
concentrated on some of the factors influencing
data warehousing projects success. Survey
respondents were asked to provide answers to
questions about who sponsored the data warehouse,
which organization unit was the driving force
behind the initiative, about solution architecture
and end users, about implementation costs,
operational costs, solution approval process, after
implementation assessment, realization of expected
benefits (together with expectations). To describe
success, two questions were used, one about ROI
and the other about perceived success of
implementation.
Although the authors gathered enough data to
analyze the dependency of individual factors with
success or to build a statistical model of several
variables, they chose not to do so. Instead, they
analyzed individual questions, summarizing
contemporary practices. Findings say that most
data warehouse sponsors come from business units
and no single architecture dominates all
implementations (Watson, Annino, Wixom, Avery,
& Rutherford, 2001). Because of the lack of a
proposed framework, it is impossible to compare
this article with our findings. Also, the article did
not attempt to propose any objective measurement
methods for variables included in the study.
2.9 Measuring User Satisfaction with Data
Warehouses: An Exploratory Study
The study of Chen, Soliman, Mao, & Frolick,
(2000) uses end user satisfaction as a proxy for
success and examines factors influencing the
aforementioned user satisfaction. Study results
identify three factors comprising several questions
each. Those factors are: support provided to end
users, accuracy, format and preciseness of data, and
fulfillment of end user needs. Authors did not
propose a framework for success and did not
attempt to build a statistical model. They also did
not attempt to develop objective and universal
measurement methods for the variables identified.
Treatment of end user satisfaction as a proxy
for project success and the goal to identify only the
factors that influence this variable makes it difficult
to compare the results with the findings. The
authors (Chen, Soliman, Mao, & Frolick, 2000)
claim to have relied on the model of IS success
(Delone & McLean, 1992), but included only one
of the “use” variables presented in that study.
The study treated use as a whole and positioned it
as one of the factors influencing success. It is
impossible to tell whether technical or non-
technical factors are more important to project
success following this article (Chen, Soliman, Mao,
& Frolick, 2000) and no objective measurement
methods are proposed.
3. Empirical Data
For primary research, surveys were used and 68
fully completed surveys obtained. The results were
analyzed using quantitative methods - simple
statistics (mean, mode, median) performed on
individual questions, correlation analysis of
individual variables with the dependent variable of
success and Partial Least Squares Regression (PLS-
R) used to build the target framework. Internet-
based e-surveys were used.
Each single data point / measurement was a
distinct Business Intelligence project. This implies
that several measurements could have been
obtained from a single vendor, customer or
individual, provided that each of those
measurements described distinct projects. Each
survey contained optional questions about customer
and vendor company names, project names, place
where the project was performed as well as
whether the survey participants were involved in a
project on the customer or vendor side, and a
precise project role of the participants.
Initial secondary research revealed that some
of independent variables chosen by authors are
correlated. Because of this and the fact that we
aimed at analyzing direct dependency of all
independent variables with success, there was a
very high probability of multicollinearity in the
data. For this reason, a method of data analysis
insensitive for multicollinearity had to be chosen.
Also, because of high number of independent
variables chosen based on phase 1 findings and
relatively low number of observations obtained
through the survey, an appropriate method had to
be chosen. Both requirements combined were
satisfied by Partial Least Squares regression, a
method that “was designed to deal with (…)
regression when data has small sample, missing
values, or multicollinearity” (Pirouz, 2006).
After completing the primary research, we
propose our own framework for BI initiatives
success, using prior research as a starting point.
The framework was separated from the
measurement methods to maintain the flexibility
and applicability of the framework in various
contexts. However, measurement methods were
also proposed. Keeping those two separated will
allow any future studies to propose different
measurement for the variables. Also, a fact of
117
importance is that it is much easier to combine a
theoretical framework with specific measurement
methods if they are kept separated, than to split
them if they are merged in a single framework.
Therefore, we decided to separately propose a
theoretical framework and a specific measurement
method for each of the variables.
4. Analysis and Implications
As mentioned, this study excludes trying to define
what BI project success looks like. Instead, to
determine whether a project is successful or not,
the following interpretation from the survey results
have been made throughout the article.
? Successful project/initiative - Project with
answers “definitely yes” and “rather yes
” to the question Q19:1 - “Was the
project you are describing successful?”
? Non-successful project/initiative - Project
with answers “neutral”, “rather no”
and “definitely no” to the question Q19:1
- “Was the project you are describing
successful?”. That is, non-successful
means unsuccessful or neutral.
The reason behind such a distinction is that this
article aims for developing tools that managers
could use to achieve success. Merely avoiding
failure is not good enough. If the project failed to
achieve success, it cannot be treated as a mild
success or neutral. Instead, it should then be
considered as a non-successful project.
Results from the survey indicate that the
funding of successful initiatives and non-successful
ones differs. Successful initiatives are more likely
to be funded in a phased approach, with separate
budgets for each iteration, while the non-successful
tend to be funded in a traditional way as other IT
projects, i.e. one budget for the entire project.
Iterative development is used by both
successful and non-successful initiatives, but
results from the survey show that for an
overwhelming majority (89%) of the successful
initiatives, each of the iterations provided business
value by themselves. This can be compared to the
non-successful initiatives that used an iterative
approach, where every other initiative had
iterations that provided value by themselves.
A BI solution designed with end users in mind
will naturally have a greater usage than BI
solutions that don’t consider the end users. A
whopping 96% of the successful initiatives in this
study have users that actually use the Business
Intelligence solution. The results for the non-
successful ones vary.
The presence of an overarching, clear
strategic vision for Business Intelligence within the
organization is important for the success of an
initiative. Results show that successful initiatives
exist in an environment with a clear strategic vision
to guide the initiative as compared to non-
successful ones. A total of 81% of the successful
initiatives agrees to this compared to 46% of the
non-successful.
Not only is the existence of a strategic vision
for Business Intelligence important, the business
need that the initiative tries to solve should be
aligned to that vision. Like the results for the
existence of vision, the successful initiatives have
business needs that tend to be more aligned with
the strategic vision than for non-successful ones.
73% of the successful initiatives agree to that.
Corresponding figure for the non-successful
initiatives is 39%.
The survey shows that successful initiatives
often concentrate on choosing the best
opportunities for the project scope. 57% of the
surveyed successful initiatives do this, compared to
23% of the non-successful ones.
Another parameter measured by the study is
whether an expert assigned to a team or made
available to a team, always was the best expert
available. The results from the study reveal,
somewhat surprisingly, that non-successful
initiatives to a larger extent assigns the best expert
available, with 85%. Corresponding number for the
successful ones are 54%.
Issues will occur during the duration of an
initiative, which can be both technological and
non-technological in nature. Both successful and
non-successful initiatives do, to some extent,
encounter non-technological issues, with non-
successful encountering them slightly more often
according to this study. The technological issues
are commonly found in both successful and non-
successful initiatives, but they are somewhat more
common in the non-successful initiatives (77%
compared to 63% reports presence of technological
issues). For all initiatives in the study only slightly
more than 20% found technology related problems
to be the hardest.
Consultants spend time on solving different
kinds of problems, technological as well as non-
technological and for a majority of the initiatives a
larger part of the time is spent on resolving non-
technological problems. This is especially true
when looking at the non-successful initiatives,
which show that almost 40% of the surveyed
initiatives indicated that non-technological issues
took the most time.
4.1 A Statistical model of Business Intelligence
initiatives’ success.
The survey results were analyzed using the Partial
Least Squares regression method. Two separate
series of models were built, one using original,
118
untransformed data and the other, using the data
transformed by centering on mean, dividing by a
factor and applying basic mathematical functions.
More details, and the rationale for performing those
transformations, are provided later in the text.
The models built for original data analysis
used 25 variables defined by answers to survey
question nr 19. For the clarity of information
presentation, variables were named after their
respective survey questions – variable A19_1
corresponded to question 19:1 answers, A19_2
corresponded to question 19:2 answers and so on.
Because question 19:1 asked users whether the
project they described was successful, this variable
was chosen as dependent for all the models, while
every other variable was treated as independent.
Unlike other linear regression methods,
adding more variables does not necessarily imply
an increase of model’s R
2
when PLS is being
used. The method cannot be used to eliminate
insignificant variables, which is easily achieved in
other linear regression methods. Therefore, the
model that used all the available variables was not
necessarily the best. Because the method cannot
eliminate insignificant variables by setting the
linear combination coefficients to 0 (or even close
to 0), models that used only some of the variables
were built.
Choice of variables that explain success
should be made based on the individual
independent variables’ correlation with the
dependent variable. After some experimentation, 5
variables with correlation of success stronger than
0.4 were chosen (absolute value of correlation
coefficient greater than 0.4).
Figure 5: PLS regression model with 5 significant variables
As can be seen the model achieved a R
2
of 0.559,
with changes in independent variables explaining
55.9% of changes in dependent variable. The
variables that enabled these results were obtained
as answers to the following questions:
Question 19_3. Did the end users use the
data warehouse / BI solution provided?
Question 19_6. Were the business needs that
project tried to solve aligned to the
abovementioned strategic vision?
Question 19_10. Did project scope
definition concentrate on choosing the best
opportunities ("low-hanging fruit")?
Question 19_19. If technological issues
were present, were all of them resolved?
Question 19_20. Were non-technological
issues encountered during the project?
PLS is a linear regression method, but it is most
likely that a dependency of success from the factors
is not linear. The possibility of other dependencies
had to be explored. While other, non-linear
regressions could possibly be applied, a lack of any
knowledge of the nature of the dependency would
imply the need to try all of them, which was clearly
beyond the scope of this study. Instead, linear
regression could be applied to the transformed
factors. For example, if linear regression could be
done on squared factors, the outcome would be a
case of square regression for the original factors; if
various powers could be applied to these factors,
the outcome would be polynomial regression and
so on.
It is always possible to choose a function that
will fit the given finite, discrete data set. To avoid
such a pitfall, the transformations had to be limited
to basic mathematical functions – power (including
square root), logarithm, exponential functions,
reciprocal or simple composition of
aforementioned functions. Similar selection
criterion as for original data applied, but the
transformation had to improve the data. The
transformed data was considered better than
original if for a given function f and variable x, f(x)
had a stronger correlation with success than
variable x itself.
119
Before the abovementioned functions could
be applied, initial data transformations were
necessary. The data was centered on mean and
divided by a factor to produce more distinct values
instead of the same 5 values for all questions. None
of those operations changed the correlation of data
prior to applying the functions.
Based on the experience from model 1, the
variables were sorted by correlation. To facilitate
experimenting with the data model, a relative rank
of correlation value was included in variable name
while keeping the question number embedded. For
example, variable V05_N19_20 had the fifth
strongest correlation of all variables (05 in the
name) and was a result of applying transformation
(N in the name) to question Q19:20 answers
(19_20 in the name). For some variables, an
alternative version of the same variable was also
used. For example V18b_A19_15 was an
alternative variable to V18a_N19_15, containing
answers to question Q19:15 without applying the
function, with only the transformations described in
data preparation section above. The decision on
whether to include an alternative variable or not
was made based on how much the correlation has
improved as a result of applying a given function,
with 0.05 as a threshold. However, models
containing alternative versions of the variables
were not any different from those using
transformed variables and will not be presented
here. The summary of functions applied to
variables is presented below:
? V01a_N19_3: q19_3 transformed using
log(2, 1-X/4)
? V02_N19_6: q19_6 transformed using
power(X, 3)
? V03_N19_6: q19_19 transformed using
power(X, 3)
? V04_A19_3: q19_10 transformed using X
? V05_N19_20: q19_20 transformed using
1/power(2,X)
? V06a_N19_5: q19_5 transformed using
exp(X)
? V07_N19_18: q19_18 transformed using
1/power(2,X)
? V08_N19_22: q19_22 transformed using
exp(X)
? V09_N19_25: q19_25 transformed using
sqrt(abs(X))
? V10N19_17: q19_17 transformed using
1/power(X,3)
? V11_N19_8: q19_8 transformed using
sqrt(abs(X))
? V12a_N19_16: q19_16 transformed using
1/power(2,X)
? V13_N19_14: q19_14 transformed using
1/X
? V14_N19_2: q19_2 transformed using
sqrt(abs(X))
? V15a_N19_12: q19_12 transformed using
1/power(X,3)
? V16_N19_7: q19_7 transformed using
power(X, 2)
? V17_N19_11: q19_11 transformed using
1/power(X,3)
? V18a_N19_15: q19_15 transformed using
power(X, 3)
? V19_N19_24: q19_24 transformed using
1/X
? V20_N19_23: q19_23 transformed using
1/X
? V21_N19_9: q19_9 transformed using
sqrt(abs(X))
Figure 6: PLS regression model 2 - equivalent of model 1 with transformed variables
120
In case of untransformed variables, the regression
model with the best fit used only 5 independent
variables. Therefore, the analogical model was
tried using transformed data. This time, the
improvement in R
2
was smaller and the model was
able to explain 0.584 of the success. The resulting
regression model is presented in Figure 6 above.
Figure 7: PLS regression model with the best fit
After some experimentation with various different
models, R
2
=0.611 was achieved by one of them. It
is presented in Figure 7. This was the best model
found using either transformed or untransformed
data. The variables used in this model are
summarized as follows:
? V14_N19_2: Overall, was it possible to
define factors that were critical to how
successful the project was?
? V01a_N19_3: Did the end users use the data
warehouse / BI solution provided?
? V06a_N19_5: Was there a clear strategic
vision for Business Intelligence or data
warehouse?
? V02_N19_6: Were the business needs that
project tried to solve aligned to the
abovementioned strategic vision?
? V16_N19_7: Was project's underlying
business case preceded by a business needs
analysis? This does not mean your party
(vendor or customer) performed this
analysis as part of the project
? V11_N19_8: Did the final solution evolve
beyond initial scope? In case a program was
established, answer on a program level
? V04_A19_10: Did project scope definition
concentrate on choosing the best
opportunities ("low-hanging fruit")?
? V17_N19_11: When an expert in a
particular field was needed, was project
team always granted access to an expert or
enhanced by including one?
? V15a_N19_12: When an expert was
assigned to a team or made available to a
team, was it always the best expert
available?
? V13_N19_14: If there were data quality
issues at source systems, were they known
before the start of BI initiative?
? V12a_N19_16: Were there data quality
issues in BI system or data warehouse?
? V10N19_17: Were data quality issues in BI
system or data warehouse eventually
resolved?
? V07_N19_18: Were technological issues
encountered during the project?
? V03_N19_19: If technological issues were
present, were all of them resolved?
? V05_N19_20: Were non-technological
issues encountered during the project?
? V08_N19_22: Data architecture was
performed at the beginning of BI initiative
(note - data architecture is different from
121
systems architecture or solution
architecture)
? V09_N19_25: Was there a process in place
to measure business benefits?
4.2 Measures Analysis
From the three measures of data warehouse use
proposed in the survey, none were found bad or
even rather bad. Conversely, none of them were
found definitely good. The best from the three was
a weekly number of queries. The fact that it was
positively evaluated in most responses and that
mean value was close to being rather good makes
this measurement method adequate and fairly
universal. It is objective and can be automatically
computed.
Out of the two proposed measurement
methods of access to subject matter experts,
respondents found both “rather good”, with over
two thirds of respondents agreeing with this view.
Despite the fact that the mean response time turned
out to score slightly better, we propose to use both
as measurement for the “access to subject matter
experts”- variable. To compute any of them, a
formal process of request tracking should be made,
but with this process in place, both measures are
objective and easy to compute.
Only one of the three proposed measurement
methods for subject matter expert quality was
found applicable. It turned out to be the same
measurement method as for the access to subject
matter experts. Because of that, the same
arguments about measurement method objectivity
and computability apply.
From the three measures proposed in the
survey for the question of whether there was
enough business staff on the project, we found that
both percentages of business issues resolved within
a business week. The mean time to resolve a
business issue qualify as an adequate measure for
the variable was there enough business staff on a
project. We found little difference in those
measures as they are equally good and equally
objective. They are both calculated in a similar way
therefore either one of them could be used, or even
both at the same time.
Two measures of deliverable size in iterative
development were proposed and we found that only
a percentage of total project time for each iteration
is a good one. However, this measure is only
objective after the project ends. It can be calculated
during the planning phases and project schedule
can be used for that purpose. Unfortunately, this
makes the measure less objective and easy to
manipulate. This diminishes its applicability to
predict initiative success. Despite these drawbacks,
we postulate to use this measure in absence of
better alternatives. Perhaps other measures could be
examined in the future to replace this one, for
instance percentage of total budget allocated or a
similar measure.
Out of three measures proposed for data
quality issues at source systems, it was found that
both counting the total number of issues and the
number of unresolved issues are adequate and
almost identically rated. Therefore no distinction
can be made as to which one of them to use and we
recommend using both. They should be easily
computable if only the formal issue tracking
process is in place. However, both measurement
methods can only be used after the project end as
the number of issues is unpredictable (following
our earlier findings, a number of issues were
discovered during the BI project). For this reason it
would be hard to predict initiative success based
only on the upfront knowledge about source
systems data issues.
From the three measurement methods
proposed for data quality issues in BI system or
data warehouse, no one better than the others were
found. Therefore, it is conclude that this variable
should be measured using the number of
unresolved issues. The fact that this variable could
be objectively measured only after the project ends
makes it unsuitable for use in predicting project
success.
All three measurement methods proposed for
benefits quantification were found adequate for
quantification. Both ROI and IRR calculations
were objective yet difficult to perform at project
start, while relating capabilities to company
strategy was less objective but easier to apply early
in the project. It would be best if financial
indicators could be calculated at project start;
therefore those two methods would be preferred
over the one called strategy, however all three are
worth using and will probably work in practice.
Both measurement methods proposed for
business benefits definition were evaluated
positively by survey respondents with percentage
of business benefits objectively measureable being
significantly better than just a written definition of
the benefits. Both were objective and computable
in the early stages of the project. Although,
percentages were recommended as a better
measurement, we also note that both are likely to
be used in practice as it is unlikely that a company
will have the benefits that are measureable without
first defining them. Therefore, in practice, when the
better measurement method can be applied, then so
can the one that is worse off.
From the two measurement methods proposed
for business benefit measurements, only one was
approved by survey respondents - the presence or
absence of business benefits process owner. This
measurement process can and should be used in
predicting the project success.
122
Figure 8: Analysis of measurement methods of variables (Q22)
For each of the variables analyzed in question 22,
there was at least one measure that was deemed
good enough to be generally used for all projects.
However, almost no measure was found definitely
good. There are two possible explanations for this
phenomenon. Either there is no universally
acceptable measurement method for those variables
or such a variable exists but was not discovered in
this study. Neither of the above possibilities can be
verified using data gathered in this study.
5. Hypotheses Analysis
Hypothesis 1 (first part). It is possible to
identify critical success factors (CSFs) of BI
initiatives in large enterprises.
To determine if factors exist that, if present, are
more likely to lead to initiative success, an analysis
of the correlation between the success and different
factors surveyed was performed. To demonstrate
causality, the statistical evidence of correlation
between success and different key success criteria
are complemented with an argumentation on what
the casual relationship is. To show correlation
between different criteria and the success of the
initiatives, Spearman’s rank correlation
coefficient (Spearman’s ?) was chosen.
Motivation for choosing Spearman’s rank
correlation coefficient includes that it does not
require the relationship to be linear, but also the
fact that it is often used for Likert scales data sets
in e.g. medicine, biochemistry and other sciences
with success (Jamieson, 2004). The
questions/factors with correlation coefficients with
an absolute value of 0.25 or higher are (correlation
coefficient in parenthesis):
? Did project scope definition concentrate on
choosing the best opportunities ("low-
hanging fruit")? (0,43)
? Were non-technological issues encountered
during the project? (-0,42)
? Were the business needs that project tried to
solve aligned to the abovementioned
strategic vision? (0,41)
? Was the project meant to address specific
business needs of the sponsor? (0,38)
? Data architecture was performed at the
beginning of BI initiative (0,37)
? Was there a clear strategic vision for
Business Intelligence or data warehouse?
(0,35)
? Were there data quality issues at source
systems? (-0,30)
? If iterative development was chosen, did
each iteration provide business value by
itself or in connection with previous
deliverables? (0,29)
? Were technological issues encountered
during the project? (-0,28)
? Were data quality issues in BI system or
data warehouse eventually resolved? (0,27)
? Have business benefits been quantified?
(0,26)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Q
2
2
:
1
Q
2
2
:
2
Q
2
2
:
3
Q
2
2
:
4
Q
2
2
:
5
Q
2
2
:
6
Q
2
2
:
7
Q
2
2
:
8
Q
2
2
:
9
Q
2
2
:
1
0
Q
2
2
:
1
1
Q
2
2
:
1
2
Q
2
2
:
1
3
Q
2
2
:
1
4
Q
2
2
:
1
5
Q
2
2
:
1
6
Q
2
2
:
1
7
Q
2
2
:
1
8
Q
2
2
:
1
9
Q
2
2
:
2
0
Q
2
2
:
2
1
Q
2
2
:
2
2
Q
2
2
:
2
3
Q
2
2
:
2
4
Q
2
2
:
2
5
Q
2
2
:
2
6
Analysis of measurement methods of
variables
no answer 1 definitely yes 2 rather yes 3 neutral 4 rather not 5 definitely not
123
From the above it can be conclude that there are in
fact criteria (those factors with a large correlation
with success of the initiative) that, when present,
are associated with a higher likelihood of initiative
success.
Hypothesis 1 (second part). CSFs of BI
initiatives are not the same as for other IS in
large enterprises.
Looking at the top criteria for successful initiatives
with the highest ranking from the surveyed
participants shows that success factors related to
the project actually addressing a specific business
need is most important. Building a solution that is
tailored to user requirements and therefore enables
usage of the Business Intelligence solution is also
important. In the third place comes; documenting
success criteria for the initiative early on in the
initiative.
Resolving technological issues, having a clear
vision to guide the initiative and including a
business need analysis early on are other important
criteria. The survey clearly shows that non-
technological factors rank high amongst the
answers. To compare this to success factors for IS
in general, a framework proposed by Delone and
McLean (Delone & McLean, 1992; 2003) was
used. The original framework proposed by these
authors has been updated with an updated version
(Delone & McLean, 2003). As discussed
previously the factors leading to the success of IS
project in general include information quality,
system quality and service quality. The first thing
that strikes one in the framework proposed by
Delone and McLean is the heavy focus on technical
issues and factors, and little to none concentration
on the softer factors that pertain to management
issues.
One difference from the general IS framework
proposed by Delone and McLean and the findings
from this study is the addition of more non-
technical factors such as the presence of a specific
business need to be addressed by the initiative, a
clear vision to guide the initiative and a pre-
initiative definition of success factors.
Hypothesis 2. Non-technical and non-
technological CSFs play a dominant role in BI
initiatives’ success in large enterprises.
From what has been seen earlier, most of the CSFs
with high correlation with success are in fact non-
technical. Examples are:
1. Successful initiatives do focus, to a larger
extent, on choosing the best opportunities
i.e. “low-hanging fruit” or quick wins.
2. Successful initiatives are more likely to
have a clear strategic vision for Business
Intelligence or data warehouse.
3. Successful initiatives are more often meant
to address a specific business need of the
sponsor.
4. The business needs addressed by successful
initiatives are more often aligned with the
abovementioned strategic vision.
Looking at individual CSFs and the responses from
the survey, one can see that for non-technical
issues, there is a noteworthy difference between the
results for successful initiatives and the non-
successful ones.
One example of this is the area of iterative
development. For those initiatives that used an
iterative approach, each iteration was more likely
to provide business value by itself for the
successful ones. A total of 89% agreed to this,
compared to 50% for the non-successful initiatives.
Another example concerns the presence of a
strategic vision for Business Intelligence. A total of
81% of the successful initiatives had a clear
strategic vision compared to only 46% of the non-
successful ones.
In conjunction with the presence of a strategic
vision for Business Intelligence as a whole, the
successful initiatives are more likely to address a
specific business need that is aligned to this
strategic vision. Some 73% of the successful
initiatives were addressing a specific business need
aligned with the strategic vision. This is almost
twice as common as the 39% of the non-successful
ones.
When looking at the scope definition, the
successful initiatives are more often concentrating
on choosing the best opportunities as depicted by
the chart below. 57% of the successful initiatives
do this compared to the non-successful ones 23%.
All of the above examples are non-technical
factors that are (to a varying degree) more
commonly found in successful initiatives than
those initiatives that fail. This in turn, indicates that
non-technical and non-technological CSFs play a
dominant role in BI initiatives’ success in large
enterprises.
Hypothesis 3. Technology plays a secondary
role and is less critical for BI initiatives’
success in large enterprises
The results from the survey show clearly that non-
technical factors were the hardest to solve. Results
show that for all initiatives in the survey only
slightly more than 20% found the technology
related problems to be the hardest to solve. The fact
that the non-technology related problems were
harder to solve is even more evident when looking
at the non-successful initiatives only, where almost
124
as many as 70% of the initiatives found the non-
technology related problem to be the hardest.
Another view on this is the time spent on the
different types of problems. Again, as we can see a
larger part of the time is spent on resolving non-
technological problems. This is especially true
when looking at the non-successful initiatives, out
of which almost 40% indicate that non-
technological issues took the most time.
When comparing the successful initiatives
with the non-successful ones and the presence of
non-technological issues, it is clear that the
initiatives that fail more often encounter non-
technological problems. One interpretation of this
could be that the presence of non-technological
issues will have a detrimental effect on the
likelihood of initiative success.
Hypothesis 4. It is possible to develop
objective measures for each of CSFs of BI
initiatives in large enterprises.
The result of this study shows that all the CSFs
clearly have one or more measures that are ranked
higher than the rest. The following summarizes the
proposed measurements methods resulting from the
survey.
Area/Variable Proposed method(s)
Data warehouse
use
? Weekly number of queries submitted
Access to subject
matter experts
? Percentage of requests responded within one business day
? Mean response time for a request
Subject matter
expert quality
? Average number of questions needed to be asked per subject
matter issue
? Average number of available documents per subject matter issue
? Percentage of questions answered within one business day
Adequate level of
business staff on
the project
? Percentage of business issues resolved within a business week
? Mean time to resolve a business issue
Size of
deliverable in
iterative
development
? Percentage of total project time for each iteration
Data quality
issues at source
systems
? Total number of issues
? Number of unresolved issues
Quality issues in
BI system or data
warehouse
? Number of unresolved issues
Benefits
quantification
? Calculated ROI
? Proposition on how the business intelligence capabilities enables
the realization of business strategy
? Contribution to the internal rate of return
Business benefits
definition
? Percentage of business benefits that are objectively measureable
Business benefits
measurements
? Presence or absence of business benefits measurements process
owner
Table 1: Summary of variables analysis
6. Conclusions
The hypotheses of this study are focused on the
factors that make Business Intelligence initiatives
successful.
A quantitative approach was used to gather
data to support or reject the above hypotheses and a
primary research survey was used for gathering
data on the above hypotheses. The survey was
conducted online during Q2 of 2011. The findings
can be summarized as follows.
6.1 Non-technological problems dominate in
Business Intelligence projects
Business Intelligence projects wrestle with both
technological and non-technological problems, but
the non-technological problems are found to be
125
both harder to solve sand more time consuming
than their technological counterparts. For all
initiatives in the survey, only slightly more than
20% found the technology related problems to be
the hardest. When looking at the non-successful
initiatives only, almost as much as 70% found the
non-technology related problem to be the hardest.
The other view on this, the time spent on the
different types of problems, also shows that a larger
part of the time is spent on resolving non-
technological problems. This is especially true
when looking at the non-successful initiatives,
almost 40% of which indicate that non-
technological issues drove the most time.
6.2 Successful projects share common factors
Certain factors or attributes of Business
Intelligence projects are correlated with the success
of the initiative. The correlation has been
calculated using Spearman’s rank and shown in
the model that was developed using Partial Least
Squares. By using rational argumentation, it can be
shown that those factors do in fact have a causal
relationship with the likelihood of success. The
highest correlated factors are:
? A project scope definition that concentrate
on choosing the best opportunities ("low-
hanging fruit")
? The alignment of the business needs that
project tries to solve aligned with the
strategic vision of business intelligence
? A project that is meant to address specific
business needs of the sponsor
? Performing data architecture at the
beginning of BI initiative
? Having a clear strategic vision for Business
Intelligence or data warehouse
6.3 Successful and non-successful projects show
differences in certain factors
For a number of different factors, clear differences
can be observed for successful and non-successful
initiatives. This shows that successful initiatives are
doing specific things more frequently than the non-
successful. Factors with great differences are:
? The type of funding of the initiative
? Business value provided by each iteration
? An alignment of the business needs that
project tried to solve aligned to the strategic
vision of Business Intelligence
? A project that is meant to address specific
business needs of the sponsor
? A project scope definition that concentrate
on choosing the best opportunities ("low-
hanging fruit")
6.4 Business Intelligence project have success
factors that differ from IS projects in general
Comparing the critical factors for BI projects that
are correlated with success, or are otherwise more
commonly found in successful initiatives, with the
success factors for general IS projects found in
literature shows that they do in fact differ. One of
the major differences between the results found in
this study is the focus on non-technological factors.
One example of this are the frequently quoted
papers by Delone and McLean (1992; 2003), that
predominately discussed technological factors
while the results of this study point to the non-
technological factors as being more important.
6.5 Clear objective measures for CSFs can be
identified
Clear objective measures that are commonly agreed
upon according to the survey exist for each of the
CSFs analyzed in the survey. For some of the
CSFs, the results are clearly pointing towards one
of the suggested measures, while others can have
multiple measures that claim to be equally good
measures. However, due to practical reasons, to the
limit of the survey size, measurements for only 10
variables were proposed. For the remaining
variables, it is impossible to verify whether the
remaining variables do or do not have objective
measurement methods.
6.6 A statistical model can be constructed that, to a
certain extent, predicts the success of BI projects
A critical success factors framework for Business
Intelligence initiatives have been developed as part
of this study. The results from the survey show that
61% of variability of success can be explained by
changes in independent variables. Limiting the
model to 5 independent variables (out of 17 that
offer the highest explanatory power of the model)
makes the model only slightly less powerful,
allowing it to explain 58% of success variability. A
conclusion from this would be that managers need
only to monitor properly and manage 5 factors to
achieve better probability of success, than the 20%-
50% currently achievable in Business Intelligence.
For a Business Intelligence project to be successful:
1. Business Intelligence solution must be built
with end users in mind, as they need to use
it
2. The Business Intelligence system needs to
be closely tied to a company’s strategic
vision
3. Project needs to be properly scoped and
prioritized to concentrate on best
opportunities first
126
4. Although technological issues are
encountered, all of them need to be solved
5. Non-technological issues should be avoided
as they can hinder the success of the BI
initiative
Below is a summary of the support for the four
hypotheses of this study.
Hypothesis 1. It is possible to identify critical
success factors (CSFs) of BI initiatives and
they are different from the CSFs of
information systems in general. (supported).
Hypothesis 2. Non-technical and non-
technological CSFs play a dominant role in BI
initiatives’ success for initiatives (supported).
Hypothesis 3. Technology plays a secondary
role and is less critical for BI initiatives’
success for initiatives (supported).
Hypothesis 4. It is possible to develop
objective measures for each of CSFs of BI
initiatives primarily (partially supported - the
survey did not cover measurement analysis all
CSFs).
7. Future research
The survey data received describes project
primarily from Poland and primarily from a vendor
perspective. Conducting a similar research on a
more generalized sample spanning projects
worldwide with equal emphasis on vendors and
customers would be an obvious next step. It would
verify if the relationship between success and its
factors is a phenomenon local to Poland or rather,
as we suspect, general in nature.
Several other frameworks were reviewed in
this study. A carefully constructed study with
properly chosen variables would allow comparing
each framework's validity with regards to
contemporary Business Intelligence projects seeing
if one is significantly stronger than the rest of them.
This is not a trivial task, because of different
treatment of success and its factors, using proxy
variables for success or even modeling
dependencies between various parts of a success.
The word critical in its strict mathematical
meaning would suggest that if even one of critical
success factors is missing, success cannot happen.
Such a relationship is impossible to model using
linear regression. Even the application of data
transformation must be taken into account, similar
to the achieved linear combinations of some
functions of variables. Therefore, another study
could be performed to search for a model that
would better explain the success using the variables
used by us. It is possible that the variables used in
this study would be able to predict more of the
Business Intelligence success when organized in a
different fashion
Because almost 40% of success variability
remains unexplained in this model, it is possible
that it can be attributed to the weakness of the
linear relationship assumed in the model, to the
missing variable that remains unidentified or even
to a random factor that is influenced only by
chance and which cannot be controlled by
management. Before a model with higher
explanatory power is constructed, it is appropriate
to keep looking for a missing variable that might
potentially account for a significant portion of
unexplained success variability.
Although this work proposes several objective
measurement criteria for independent variables,
most of them cannot be considered universally
applicable in all (or even most) Business
Intelligence projects. Therefore, another study
could be conducted to verify whether such
universal and objective measures exist and attempt
to define them. In case such measures do not exist,
perhaps there is a set of measures for each variable
such that at least one of the quantification methods
would be universally applicable.
One of the delimitations in this work was lack
the treatment of NPV, ROI and other financial
measures of Business Intelligence projects. Since
success factors are different for BI than for IS in
general, perhaps Business Intelligence managers
could also employ different methods for calculating
financial measures than in the case of other IS. In
case such differences are identified, appropriate
methods should be proposed and aforementioned
differences should be uncovered.
Refrences
Adelman, S. 2003. Impossible Data Warehouse
Situations. Solutions from the Experts. Boston,
MA: Pearson Education.
Adèr, & Mellenbergh, G. 2011. Advising on
Research Methods: A consultant's companion.
3
rd
ed. Huizen, The Netherlands: Johannes van
Kessel Publishing.
Ariyachandra, T., & Watson, H. 2006. Which Data
Warehouse Architecture Is Most Successful?
Business Intelligence Journal , 11 (1).
Babbie, E. 2009. The Practice of Social Research.
Wadsworth Publishing.
Beal, B. 2005. Report: Half of data warehouse
projects to fail. Retrieved May 22, 2011, from
Search CRM:http://searchcrm.techtarget.com/news/1066086/
Report-Half-of-data-warehouse-projects-to-fail
Chen, L., Soliman, K., Mao, E., & Frolick, M.
2000. Measuring user satisfaction with data
warehouses: An exploratory study. Information
& Management , 37 (3), 103-110.
Daniel, D. 1961. Management Information Crisis.
Harvard Business Review.
127
Delone, W., & McLean, E. 1992. Information
Systems Success: The Quest for the Dependent
Variable. Journal of Information System
Research, 3 (1), 60-95.
Delone, W., & McLean, E. 2003. The DeLone and
McLean Model of Information Systems
Success: A Ten-Year Update. Journal of
Management Information Systems , 19 (4), 9-
30.
Duncan, W. 1996. A Guide to the Project
Management Body of Knowledge. Sylva, North
Carolina: PMI Publishing.
Ellison, R., & Moore, A. 2003. Trustworthy
Refinement Through Intrusion-Aware Design.
Carnegie Mellon Software Engineering
Institute.
Garthwaite, P. 1994. An Interpretation of Partial
Least Squares. Journal of the American
Statistical Association , 89 (425), 122-127.
Hwang, M., & Xu, H. 2008. A Structural Model of
Data Warehousing Success. Journal of
Computer Information Systems , 48-56.
Inmon, W. H. 2002. Building the Data Warehouse,
3rd ed. New York, NY: John Wiley & Sons.
Jamieson, S. 2004. Likert scales: how to (ab)use
them. Medical Education, 1217-1218.
Kimball, R., & Ross, M. 2002. The data warehouse
Toolkit, 2 ed. New York, NY: John Wiley and
Sons, Inc.
Kimball, R., Ross, M., Thornthwaite, W., Mondy,
J., & Becker, B. 2008. The Data Warehouse
Lifecycle Toolkit, 2nd ed. Indianapolis, IN:
Wiley Publishing.
Laskowski, N. 2001. Gartner BI Summit: Business
intelligence benefits lie in orchestration.
Retrieved May 22, 2011, fromhttp://searchbusinessanalytics.techtarget.com/ne
ws/2240035533/Gartner-BI-Summit-Business-
intelligence-benefits-lie-in-
orchestration?asrc=EM_NLN_13851353&track
=NL-544&ad=832107
Legodi, I., & Barry, M.-L. 2010. The current
challenges and status of risk management in
enterprise data warehouse projects in South
Africa. PICMET 2010 Technology Management
for Global Economic Growth.
Linstone, H. A., & Turoff, M. 2002. The Delphi
Method Techniques and Applications.
Markarian, J., Brobst, S., & Bedell, J. 2007.
Critical Success Factors Deploying Pervasive
BI.
Meehan, P. 2011. Patrick Meehan speech during
Gartner Business Intelligence Summit 2011.
London, UK. Retrieved fromhttp://link.brightcove.com/services/player/bcpid
1156010110?bctid=741282639001
Merriam-Webster Dictionary. 2011. Retrieved May
24, 2011, from Merriam-Webster.com
Numminen, E. (2010). On the Economic Return of
a Software Investment omManaging Cost, PhD
Dissertation.
Pirouz, D. 2006. An Overview of Partial Least
Squares. Retrieved May 27, 2011, from Social
Science Research Network:http://ssrn.com/abstract=1631359
Pisello, T. 2001. IT Value Chain Management rom
Social Science Research Network: htt Alinean,
LLC.
Popovic, A., & Jaklic, J. 2010. Benefits of Business
Intelligence System Implementation: An
Empirical Analysis of the Impact of Business
Intelligence System Maturity on Information
Quality.
Ringle, C., Wende, S., & Will, S. 2005. SmartPLS
2.0 (M3) Beta. Hamburg.
Shin, B. 2003. An Exploratory Investigation of
System Success Factors in Data Warehousing.
Communications of the Association for
Information Systems , 6 (4), 141-170.
Solberg Søilen, K., & Hasslinger, A. 2009. How
application integration, security issues and
pricing strategies in business intelligence shape
vendor differentiation. ECIS 2009: THIRD
EUROPEAN COMPETITIVE INTELLIGENCE
SYMPOSIUM, (pp. 252-260). Stockholm.
Walton, E. J., & Dawson, S. 2001. Managers J., &
Dawson, S. MPETITIVE INTELLIGENCE
SYMPOSIUMtrategiesJournal of Management
Studies, 173-179.
Watson, H., Annino, D., Wixom, B., Avery, L., &
Rutherford, M. 2001. Current Practices in Data
Warehousing. Information Systems
Management, 18 (1), 47-55.
Wixom, B., & Watson, H. 2001. An Empirical
Investigation of the Factors Affecting Data
Warehousing Success. MIS Quarterly , 25 (1),
17-41.
Yeoh, W., & Koronios, A. 2010. Critical Success
Factors for Business Intelligence Systems.
Journal of Computer Information Systems, 50
(3), 23-32.
Yeoh, W., Gao, J., & Koronios, A. 2007. Towards
a CSF Framework for Implementation of
Business Intelligence Systems: A Delphi Study
in Engineering Asset Management
Organisations. Research and Practical Issues of
Enterprise Information Systems II. Beijing,
PRC.
doc_281253399.pdf