Description
From Big Data To Better Decisions The Ultimate Guide To Business Intelligence Today
FROM BIG DATA TO BETTER DECISIONS
The ultimate guide to business intelligence today.
Introduction: What You’re Going to Learn.........................................................3
Ch. 1: A Flood of Data, and How BI Addresses It............................................5
Ch. 2: The Business Intelligence Market.............................................................7
Ch. 3: The BI Process | Step 1 - Ingestion..........................................................11
Ch. 4: The BI Process | Step 2 - Analysis.........................................................15
Ch. 5: The BI Process | Step 3 - Delivery........................................................20
Ch. 6: The Bene?ts of BI........................................................................................23
Ch. 7: The Challenges of BI.................................................................................27
Ch. 8: The Future of BI..........................................................................................30
THE BI GUIDE: WHAT YOU’RE GOING TO LEARN.
From sales opportunities to supply chain logistics, and from accounting software to social media stats,
your organization is bursting at the seams with data. Business intelligence (BI) is the combination of
tools, processes, and skills that help turn that vast amount of data into digestible information.
With information coming from every part of your organization, everyone needs better access to
data to do their job well. Chances are, you need it, too. It’s why you’re reading this guide.
Browse our list of chapters to go straight to information speci?c to your needs, or feel free to read
it cover to cover for a holistic view at how you can use BI to shape your work.
BI HAS OUTGROWN
SPREADSHEETS AND
DATA WAREHOUSES.
Before business intelligence was “business
intelligence,” it was nothing but numbers written
on spreadsheets (the actual paper variety). But as
technology grew, little changed with how business
leaders consumed information—they moved from
paper to digital, and when the volume got to be
big enough, they moved the data from desktop
spreadsheets to a massive table known as a
database. In the end, the results were the same: static
information presented in a document, maybe with a
few graphs thrown in for good measure.
But you don’t need new ways to replicate antiquated
business practices. If you can get real-time updates
on obscure college friends’ lives through social media,
then you should be able to access information from
your business anytime, anywhere.
Now it’s time to learn how to access the right data at
the right time. Keep reading—you’re in the right place.
You see BI as important
and want more info as
part of your professional
development.
You’d like to pursue a
career in BI.
You’ve got the gist of BI
and want to brush up on
some details.
You want to know what
the BI team in your
organization really deals
with day to day.
You’ve heard so many
buzzwords—“big data,”
“data science,” “business
analytics,” “predictive
analytics,” “BI,” etc.—and
want to know what all the
fuss is about.
Five reasons to
read this guide.
WHY YOU SHOULD READ THIS GUIDE.
Data is on everybody’s minds—from executives pushing their teams to take advantage of all the
data the business collects, to consumers worrying about sharing too much of their personal lives.
This guide cuts through the buzzwords and the technical jargon to give you an overview of business
intelligence—the tools, processes and skills that help us harness the data explosion to make better
and faster decisions. A state-of-the-art BI environment ensures the shortest and most reliable path
from data to decisions that make your business more successful.
CHAPTER ONE
A ?ood of data, and how BI addresses it.
A FLOOD OF DATA.
We are living in a data deluge. The amount of new data created annually will grow ten-fold between
2013 and 2020, according to IDC, from 4.4 trillion gigabytes to 44 trillion gigabytes.
If you can swim in this ?ood of data, you win. According to MIT researchers, companies that excel in
data-driven decision-making are 5% more productive and 6% more pro?table than their competitors,
on average. A study by IDC found that users of big data and analytics that use diverse data sources,
diverse analytical tools, and diverse metrics were ?ve times more likely to exceed expectations for
their projects than those who don’t.
DATA WITH NO ANALYSIS HAS NO VALUE.
Navigating the ?ood of data is much easier said than done. IDC predicts that
companies will continue to waste 80% of customer data they have collected. More
broadly, IDC estimates that in 2013 only 22% of all data in the world was useful (i.e.,
could be analyzed) and less than 5% of that was actually analyzed.
A University of Texas at Austin study put these general estimates in a business
context: it found that for the median Fortune 1000 company, a 10% increase in the
usability of its data translates to an increase of $2.01 billion in annual revenues and
a 10% increase in remote accessibility to data translates into an additional $65.67
million in net income per year.
$2 BILLION ON THE LINE, BUT NOTHING NEW FROM BI.
With $2 billion on the line, CIOs have reported in Gartner’s surveys that business intelligence has
been a top priority for the last nine years. CEOs are also getting on the bandwagon, demanding
more and more access to more and more data.
While the need for timely, accurate, and accessible business intelligence is greater than ever, the
use of business intelligence tools has plateaued at about 20%-25% of business users in a typical
organization over the past few years.
As Gartner recently observed, “despite the strong interest in BI and analytics, confusion around
big data is inhibiting spending on BI and analytics software.”
The frustration is widespread, according to surveys conducted by businessintelligence.com:
• Only 25% of CEOs say their reports contain the information they need and want.
• 44% of executives say that too many of their critical decisions were based on incomplete or
inaccurate data.
• 75% of vice presidents surveyed said that they were dissatis?ed with their access to the data they
need, and 69% were not happy with the speed of information delivery.
These data management challenges are compounded by bloated solutions, complex deployments,
and overly complicated user interfaces. The emergence of new tools and technologies for
harnessing the data deluge, aimed at solving these issues, may actually slow down the adoption
and widespread use of business intelligence.
Want to learn more? Read the executive brief, “The big BI disappointment: Troubling gaps
between BI expectations and reality.”
“
WHAT BUSINESS LEADERS NEED FROM BI.
Today’s leaders no longer make decisions based primarily on intuition. Instead, making decisions
today is a team sport, involving all the relevant people in the organization, and taking advantage of
new technologies to collect and analyze all the relevant data. In this all-hands-on-data environment,
decision makers expect:
• Data from all relevant sources in one place.
• Real-time data, not having to wait for an analyst to
deliver it or wait for IT to respond.
• Data that is accessible anytime and anywhere, on
any mobile device.
• Data that represents one version of the truth
• Self-service data and analysis, reducing the
reliance on experts.
• Data and analytics that help predict what’s coming.
Want to learn more? Read the analyst
report, “7 Steps to Making Big Data
Accessible to Executives.”
Today’s business intelligence is embedded in all
levels of the organization, allowing anyone that
needs to make a decision—operational, tactical,
or strategic decision—to make it based on the
best data available. Business Intelligence is the
combination of tools, processes and skills that
help us turn the data deluge into better and faster
decisions.
CHAPTER TWO
The business intelligence market.
HOW BIG IS THE BI MARKET?
Gartner estimates that the worldwide business intelligence and analytics market was $14.4 billion in
2013, growing at 8% annually. Assessing the larger market for business analytics, IDC estimates it
had reached $104.1 billion in 2013, at a growth rate of 10.8%. The big data segment of this market was
$12.6 billion in 2013, with a growth rate of 27%.
The business intelligence market is dominated by a few large players—SAP, Oracle, IBM, SAS,
Microsoft, Teradata—accounting for about 70% of worldwide revenues. The balance of the market is
accounted for by hundreds of small players, including numerous new startups, most of them focused
on one or two segments of the market. Established business intelligence-focused companies include
Actuate, Information Builders, Panorama, MicroStrategy, QlikTech, Tableau Software, and Tibco
Software. New startups include Alteryx, Birst, Domo, Good Data, and SiSense.
HOW IS THE BI MARKET CHANGING?
In older tools—and even in most current solutions—BI tells you what happened in a speci?c segment
of your business. With how quickly business is moving today, that kind of BI is as problematic as
driving down the freeway by looking only in your rear-view mirror.
With new technology and new expectations, BI is moving toward a more predictive model that shows
you what will happen. New BI systems are now beginning to show how all the various parts of your
organization work together to produce an outcome, and business leaders can ?nally see the big
picture and make faster, better-informed decisions.
This transformation started over a decade ago as more and more ?rms started to compete on the
basis of statistical analysis and data management prowess. It’s what drove today’s online giants like
Net?ix, Google, and Amazon—each with a reputation for mastering data, measurement, testing, and
analysis—to be what seem like unstoppable forces.
In response to these giants’ success—who barely existed 20 years ago—many established
companies now invest in statisticians and operations research personnel, build business analytics
departments, weave modeling, prediction, and forecasting into their processes, and acquire new
hardware and software tools to support these activities.
HOW ARE ORGANIZATIONS MEETING THE DEMAND?
More recently, another new layer of the business intelligence market has emerged and become
known by the somewhat misleading name of big data. Again the main culprits were online ?rms
such as Google, Yahoo, and LinkedIn but this time the new layer of the market was created around
the new technologies (e.g., Hadoop), and the new roles (e.g., data scientists) that were invented by
these companies to support data-driven decision-making and turn their data into revenue streams.
Now, every organization has to reconcile itself to the rapid growth of available data, the competitive
pressures to excel in data mining and analysis, and the increasing need to bring these capabilities to
all levels of the organization.
Global intelligence market size, by technologies, 2013–2018 ($ billion)
Sources: Gartner, Redwood Capital
Traditional BIC loud BI Mobile BI Social BI
2013 2014 2015 2016 2017 2018
25
20
15
10
5
0
IC
“Major changes are imminent to the world of BI and analytics including the dominance of
data discovery techniques, wider use of real-time streaming event data and the eventual
acceleration in BI and analytics spending when big data ?nally matures.”—Gartner
In developing their business intelligence capabilities, organizations have always had the option to
buy outside services to supplement their own in-house activities. They could buy specialized skills,
consulting, or even speci?c data from data aggregators. This segment of the market, now called
“data- as-a-service,” has recently grown rapidly with the emergence of new players providing data
services with embedded BI and analytic capabilities. To alleviate the analytics and data science talent
shortage, some vendors focus on providing the required skills on a project-by-project basis.
But no economy had enough trained talent—data scientists, analysts, systems managers, etc.—to
meet the sudden demand, prompting a burst of new technologies meant to ?ll the void. Thus,
investment in BI tools and technologies is primarily driven today by the trend towards wider adoption
of BI, giving end-users easy-to-use tools for accessing, viewing, analyzing and manipulating data.
This “democratization of business intelligence” or “self-service BI” is accompanied by growing
investments in embedding BI capabilities in various business processes and applications. These
new applications, leveraging new data types and new types of analysis, are increasingly installed on
mobile devices, drawing on data that resides in the cloud, supporting users anywhere, anytime.
WHAT ARE THE KEY COMPONENTS OF THE BI MARKET?
The BI market is typically segmented according to product functionality such as “query and
reporting,” “online analytical processing (OLAP),” and “dashboards.” It is easier, however, to
understand the BI market if we look at the process of business intelligence or the steps required
to get from data to decisions. In a nutshell, the process of business intelligence has three steps:
Ingestion, Analysis, and Delivery.
“
It starts with the ingestion of data—identifying the right data
sources and preparing the data for analysis; continuing through
the analysis stage, including processing the data and applying
analytical models to the data; and concluding with delivery—
presenting the results of the analysis in easy-to-consume
manner and at the most convenient point of consumption for
the user.
These three steps will be covered in detail in the following
three chapters.
“Experts often possess more data than judgment.” —Colin Powell
“
See the infographic: “The World
Needs Data Scientists.”
CHAPTER THREE: THE BI PROCESS
Step 1 - Ingestion
INGESTION:
HOW IS THE BI MARKET CHANGING?
Before an organization can take in data, business leaders need to understand where it’s coming
from, what format it’s in, and how to turn raw data into something useful.
Here are some of the basics:
The data for business intelligence comes from a variety of sources, internal and external to your
company. Internal sources include engineering and manufacturing processes, Enterprise Resource
Planning (ERP) systems, sales force automation and customer relationship management (CRM)
software, and ?nancial and accounting activities. External sources include supply-chain and logistics
systems, business and distribution partners, social networks, websites, location/GPS systems, mobile
and stationary sensors, and click streams. There are also many “open data” sources on the Web that
make data collected by government agencies, non-pro?ts and businesses available at no charge.
The process of business intelligence starts with identifying the data sources and the type of
data that can support speci?c decisions and business objectives. Once you have the data, you
need to make sure it is ready for processing and analysis.
DEALING WITH DATA STRUCTURE.
The data coming from these disparate sources is in many types and formats, including rows and
columns in traditional databases, images, text documents, video, PowerPoint and HTML ?les, email
Structured Data
(e.g., the numbers in a customer invoice) can be easily
ordered in the rows and columns of a traditional database
table (e.g., customer account number, invoiced amount) or
some other type of database with a de?ned structure.
Semi-Structured Data
(e.g., HTML or email ?les) conforms to a partial structure or
a standard format and contains speci?c markers that give it
some type of organization.
Unstructured Data
(e.g., an image) is not organized in any pre-de?ned manner.
THE STRUCTURING OF DATA: A HISTORY.
“Structured” and “unstructured” are somewhat misleading terms. All forms of human communications
have some structure (e.g., language), and machine-generated data typically has a structure because
it is designed to have one. What we have is a continuum that extends from a highly rigid structure,
which is de?ned before the processing and mining of the data to highly ?exible structure that is
de?ned after the processing and mining of the data.
The “highly rigid” end of the continuum gave rise in the 1970s to technologies such as relational
databases that exploited the structure imposed on the data. The focus on “structured” data, (i.e., data
with prede?ned structure), continued until the 2000s. At that point, online search and web analytics
companies started digging into “unstructured” data, (i.e., data without a prede?ned structure). New
techniques are now available that take in data that has loose structure (e.g., log ?les) or implicit
structure (e.g., natural language) and extract that structure rapidly and at scale, making it available for
analysis in a time frame where it is still useful.
messages, sensor data, web-based transactions, and IT systems logs. These data types are usually
classi?ed into three broad categories: Structured, semi-structured, and unstructured data:
Q3 Repo
From Big Data To Better Decisions The Ultimate Guide To Business Intelligence Today
FROM BIG DATA TO BETTER DECISIONS
The ultimate guide to business intelligence today.
Introduction: What You’re Going to Learn.........................................................3
Ch. 1: A Flood of Data, and How BI Addresses It............................................5
Ch. 2: The Business Intelligence Market.............................................................7
Ch. 3: The BI Process | Step 1 - Ingestion..........................................................11
Ch. 4: The BI Process | Step 2 - Analysis.........................................................15
Ch. 5: The BI Process | Step 3 - Delivery........................................................20
Ch. 6: The Bene?ts of BI........................................................................................23
Ch. 7: The Challenges of BI.................................................................................27
Ch. 8: The Future of BI..........................................................................................30
THE BI GUIDE: WHAT YOU’RE GOING TO LEARN.
From sales opportunities to supply chain logistics, and from accounting software to social media stats,
your organization is bursting at the seams with data. Business intelligence (BI) is the combination of
tools, processes, and skills that help turn that vast amount of data into digestible information.
With information coming from every part of your organization, everyone needs better access to
data to do their job well. Chances are, you need it, too. It’s why you’re reading this guide.
Browse our list of chapters to go straight to information speci?c to your needs, or feel free to read
it cover to cover for a holistic view at how you can use BI to shape your work.
BI HAS OUTGROWN
SPREADSHEETS AND
DATA WAREHOUSES.
Before business intelligence was “business
intelligence,” it was nothing but numbers written
on spreadsheets (the actual paper variety). But as
technology grew, little changed with how business
leaders consumed information—they moved from
paper to digital, and when the volume got to be
big enough, they moved the data from desktop
spreadsheets to a massive table known as a
database. In the end, the results were the same: static
information presented in a document, maybe with a
few graphs thrown in for good measure.
But you don’t need new ways to replicate antiquated
business practices. If you can get real-time updates
on obscure college friends’ lives through social media,
then you should be able to access information from
your business anytime, anywhere.
Now it’s time to learn how to access the right data at
the right time. Keep reading—you’re in the right place.
You see BI as important
and want more info as
part of your professional
development.
You’d like to pursue a
career in BI.
You’ve got the gist of BI
and want to brush up on
some details.
You want to know what
the BI team in your
organization really deals
with day to day.
You’ve heard so many
buzzwords—“big data,”
“data science,” “business
analytics,” “predictive
analytics,” “BI,” etc.—and
want to know what all the
fuss is about.
Five reasons to
read this guide.
WHY YOU SHOULD READ THIS GUIDE.
Data is on everybody’s minds—from executives pushing their teams to take advantage of all the
data the business collects, to consumers worrying about sharing too much of their personal lives.
This guide cuts through the buzzwords and the technical jargon to give you an overview of business
intelligence—the tools, processes and skills that help us harness the data explosion to make better
and faster decisions. A state-of-the-art BI environment ensures the shortest and most reliable path
from data to decisions that make your business more successful.
CHAPTER ONE
A ?ood of data, and how BI addresses it.
A FLOOD OF DATA.
We are living in a data deluge. The amount of new data created annually will grow ten-fold between
2013 and 2020, according to IDC, from 4.4 trillion gigabytes to 44 trillion gigabytes.
If you can swim in this ?ood of data, you win. According to MIT researchers, companies that excel in
data-driven decision-making are 5% more productive and 6% more pro?table than their competitors,
on average. A study by IDC found that users of big data and analytics that use diverse data sources,
diverse analytical tools, and diverse metrics were ?ve times more likely to exceed expectations for
their projects than those who don’t.
DATA WITH NO ANALYSIS HAS NO VALUE.
Navigating the ?ood of data is much easier said than done. IDC predicts that
companies will continue to waste 80% of customer data they have collected. More
broadly, IDC estimates that in 2013 only 22% of all data in the world was useful (i.e.,
could be analyzed) and less than 5% of that was actually analyzed.
A University of Texas at Austin study put these general estimates in a business
context: it found that for the median Fortune 1000 company, a 10% increase in the
usability of its data translates to an increase of $2.01 billion in annual revenues and
a 10% increase in remote accessibility to data translates into an additional $65.67
million in net income per year.
$2 BILLION ON THE LINE, BUT NOTHING NEW FROM BI.
With $2 billion on the line, CIOs have reported in Gartner’s surveys that business intelligence has
been a top priority for the last nine years. CEOs are also getting on the bandwagon, demanding
more and more access to more and more data.
While the need for timely, accurate, and accessible business intelligence is greater than ever, the
use of business intelligence tools has plateaued at about 20%-25% of business users in a typical
organization over the past few years.
As Gartner recently observed, “despite the strong interest in BI and analytics, confusion around
big data is inhibiting spending on BI and analytics software.”
The frustration is widespread, according to surveys conducted by businessintelligence.com:
• Only 25% of CEOs say their reports contain the information they need and want.
• 44% of executives say that too many of their critical decisions were based on incomplete or
inaccurate data.
• 75% of vice presidents surveyed said that they were dissatis?ed with their access to the data they
need, and 69% were not happy with the speed of information delivery.
These data management challenges are compounded by bloated solutions, complex deployments,
and overly complicated user interfaces. The emergence of new tools and technologies for
harnessing the data deluge, aimed at solving these issues, may actually slow down the adoption
and widespread use of business intelligence.
Want to learn more? Read the executive brief, “The big BI disappointment: Troubling gaps
between BI expectations and reality.”
“
WHAT BUSINESS LEADERS NEED FROM BI.
Today’s leaders no longer make decisions based primarily on intuition. Instead, making decisions
today is a team sport, involving all the relevant people in the organization, and taking advantage of
new technologies to collect and analyze all the relevant data. In this all-hands-on-data environment,
decision makers expect:
• Data from all relevant sources in one place.
• Real-time data, not having to wait for an analyst to
deliver it or wait for IT to respond.
• Data that is accessible anytime and anywhere, on
any mobile device.
• Data that represents one version of the truth
• Self-service data and analysis, reducing the
reliance on experts.
• Data and analytics that help predict what’s coming.
Want to learn more? Read the analyst
report, “7 Steps to Making Big Data
Accessible to Executives.”
Today’s business intelligence is embedded in all
levels of the organization, allowing anyone that
needs to make a decision—operational, tactical,
or strategic decision—to make it based on the
best data available. Business Intelligence is the
combination of tools, processes and skills that
help us turn the data deluge into better and faster
decisions.
CHAPTER TWO
The business intelligence market.
HOW BIG IS THE BI MARKET?
Gartner estimates that the worldwide business intelligence and analytics market was $14.4 billion in
2013, growing at 8% annually. Assessing the larger market for business analytics, IDC estimates it
had reached $104.1 billion in 2013, at a growth rate of 10.8%. The big data segment of this market was
$12.6 billion in 2013, with a growth rate of 27%.
The business intelligence market is dominated by a few large players—SAP, Oracle, IBM, SAS,
Microsoft, Teradata—accounting for about 70% of worldwide revenues. The balance of the market is
accounted for by hundreds of small players, including numerous new startups, most of them focused
on one or two segments of the market. Established business intelligence-focused companies include
Actuate, Information Builders, Panorama, MicroStrategy, QlikTech, Tableau Software, and Tibco
Software. New startups include Alteryx, Birst, Domo, Good Data, and SiSense.
HOW IS THE BI MARKET CHANGING?
In older tools—and even in most current solutions—BI tells you what happened in a speci?c segment
of your business. With how quickly business is moving today, that kind of BI is as problematic as
driving down the freeway by looking only in your rear-view mirror.
With new technology and new expectations, BI is moving toward a more predictive model that shows
you what will happen. New BI systems are now beginning to show how all the various parts of your
organization work together to produce an outcome, and business leaders can ?nally see the big
picture and make faster, better-informed decisions.
This transformation started over a decade ago as more and more ?rms started to compete on the
basis of statistical analysis and data management prowess. It’s what drove today’s online giants like
Net?ix, Google, and Amazon—each with a reputation for mastering data, measurement, testing, and
analysis—to be what seem like unstoppable forces.
In response to these giants’ success—who barely existed 20 years ago—many established
companies now invest in statisticians and operations research personnel, build business analytics
departments, weave modeling, prediction, and forecasting into their processes, and acquire new
hardware and software tools to support these activities.
HOW ARE ORGANIZATIONS MEETING THE DEMAND?
More recently, another new layer of the business intelligence market has emerged and become
known by the somewhat misleading name of big data. Again the main culprits were online ?rms
such as Google, Yahoo, and LinkedIn but this time the new layer of the market was created around
the new technologies (e.g., Hadoop), and the new roles (e.g., data scientists) that were invented by
these companies to support data-driven decision-making and turn their data into revenue streams.
Now, every organization has to reconcile itself to the rapid growth of available data, the competitive
pressures to excel in data mining and analysis, and the increasing need to bring these capabilities to
all levels of the organization.
Global intelligence market size, by technologies, 2013–2018 ($ billion)
Sources: Gartner, Redwood Capital
Traditional BIC loud BI Mobile BI Social BI
2013 2014 2015 2016 2017 2018
25
20
15
10
5
0
IC
“Major changes are imminent to the world of BI and analytics including the dominance of
data discovery techniques, wider use of real-time streaming event data and the eventual
acceleration in BI and analytics spending when big data ?nally matures.”—Gartner
In developing their business intelligence capabilities, organizations have always had the option to
buy outside services to supplement their own in-house activities. They could buy specialized skills,
consulting, or even speci?c data from data aggregators. This segment of the market, now called
“data- as-a-service,” has recently grown rapidly with the emergence of new players providing data
services with embedded BI and analytic capabilities. To alleviate the analytics and data science talent
shortage, some vendors focus on providing the required skills on a project-by-project basis.
But no economy had enough trained talent—data scientists, analysts, systems managers, etc.—to
meet the sudden demand, prompting a burst of new technologies meant to ?ll the void. Thus,
investment in BI tools and technologies is primarily driven today by the trend towards wider adoption
of BI, giving end-users easy-to-use tools for accessing, viewing, analyzing and manipulating data.
This “democratization of business intelligence” or “self-service BI” is accompanied by growing
investments in embedding BI capabilities in various business processes and applications. These
new applications, leveraging new data types and new types of analysis, are increasingly installed on
mobile devices, drawing on data that resides in the cloud, supporting users anywhere, anytime.
WHAT ARE THE KEY COMPONENTS OF THE BI MARKET?
The BI market is typically segmented according to product functionality such as “query and
reporting,” “online analytical processing (OLAP),” and “dashboards.” It is easier, however, to
understand the BI market if we look at the process of business intelligence or the steps required
to get from data to decisions. In a nutshell, the process of business intelligence has three steps:
Ingestion, Analysis, and Delivery.
“
It starts with the ingestion of data—identifying the right data
sources and preparing the data for analysis; continuing through
the analysis stage, including processing the data and applying
analytical models to the data; and concluding with delivery—
presenting the results of the analysis in easy-to-consume
manner and at the most convenient point of consumption for
the user.
These three steps will be covered in detail in the following
three chapters.
“Experts often possess more data than judgment.” —Colin Powell
“
See the infographic: “The World
Needs Data Scientists.”
CHAPTER THREE: THE BI PROCESS
Step 1 - Ingestion
INGESTION:
HOW IS THE BI MARKET CHANGING?
Before an organization can take in data, business leaders need to understand where it’s coming
from, what format it’s in, and how to turn raw data into something useful.
Here are some of the basics:
The data for business intelligence comes from a variety of sources, internal and external to your
company. Internal sources include engineering and manufacturing processes, Enterprise Resource
Planning (ERP) systems, sales force automation and customer relationship management (CRM)
software, and ?nancial and accounting activities. External sources include supply-chain and logistics
systems, business and distribution partners, social networks, websites, location/GPS systems, mobile
and stationary sensors, and click streams. There are also many “open data” sources on the Web that
make data collected by government agencies, non-pro?ts and businesses available at no charge.
The process of business intelligence starts with identifying the data sources and the type of
data that can support speci?c decisions and business objectives. Once you have the data, you
need to make sure it is ready for processing and analysis.
DEALING WITH DATA STRUCTURE.
The data coming from these disparate sources is in many types and formats, including rows and
columns in traditional databases, images, text documents, video, PowerPoint and HTML ?les, email
Structured Data
(e.g., the numbers in a customer invoice) can be easily
ordered in the rows and columns of a traditional database
table (e.g., customer account number, invoiced amount) or
some other type of database with a de?ned structure.
Semi-Structured Data
(e.g., HTML or email ?les) conforms to a partial structure or
a standard format and contains speci?c markers that give it
some type of organization.
Unstructured Data
(e.g., an image) is not organized in any pre-de?ned manner.
THE STRUCTURING OF DATA: A HISTORY.
“Structured” and “unstructured” are somewhat misleading terms. All forms of human communications
have some structure (e.g., language), and machine-generated data typically has a structure because
it is designed to have one. What we have is a continuum that extends from a highly rigid structure,
which is de?ned before the processing and mining of the data to highly ?exible structure that is
de?ned after the processing and mining of the data.
The “highly rigid” end of the continuum gave rise in the 1970s to technologies such as relational
databases that exploited the structure imposed on the data. The focus on “structured” data, (i.e., data
with prede?ned structure), continued until the 2000s. At that point, online search and web analytics
companies started digging into “unstructured” data, (i.e., data without a prede?ned structure). New
techniques are now available that take in data that has loose structure (e.g., log ?les) or implicit
structure (e.g., natural language) and extract that structure rapidly and at scale, making it available for
analysis in a time frame where it is still useful.
messages, sensor data, web-based transactions, and IT systems logs. These data types are usually
classi?ed into three broad categories: Structured, semi-structured, and unstructured data:
Q3 Repo