Description
With the Analysis Center, you get an extensive set of reports for web analytics and sales analysis.
1
© Copyright 2003, Blue Martini Software. San Mateo California, USA
Deriving Key Insights from Blue Martini
Business Intelligence
Ronny Kohavi
Vice President,
Business Intelligence
Blue Martini Software
Anne Ford
E-Commerce
Project Manager
MEC
Bill Maginn
Internet
Controller
Debenhams
2
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Agenda
? Debenhams and MEC
? Web site, DSSGen, and Bots
? Easy insights – out of the box reports
? Harder insights – what you could write
? Actions at Debenhams and MEC
? BIG ROI project
? Q&A
3
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
? Debenhams plc, “Britain's favourite department store”
– UK's fashion retail leader for more than 200 years
– Runs approximately 100 department stores in the UK and Ireland
– Offering brand-name women's, men's and children's apparel as well as
high-end housewares and cosmetics
– Debenhams also offers the award winning „Queen's Award for
Enterprise - April 2001? bridal registry service
? Debenhams online profile
– Over 500,000 customers
– About £18 million in total sales in the last year
– Wedding list purchases account for about half of sales
? Clickstreams
– About 2M page views / week
– About 6,000 new customers per week
Debenhams
4
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
MEC
? MEC – Mountain Equipment Co-op
? Canada?s leading supplier of quality outdoor gear and clothing
? MEC has
– 1.8 million members
– Sales over $160 million
– Seven physical retail stores
? As a co-op, MEC supports the community in several ways.
– One of them is gearswap, an area of the site for selling used gear,
where MEC makes no money
– Products link to information promoting environmental responsibility
5
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Survey
We asked early participants what they wanted to hear
in this webinar
0
2
4
6
8
10
12
R
e
p
o
r
t
i
n
g
V
i
z
B
o
t
M
i
g
r
a
t
o
r
A
s
s
o
c
C
r
o
s
s
-
s
e
l
l
C
a
m
p
a
i
g
n
U
s
a
b
i
l
i
t
y
S
e
a
r
c
h
A
d
e
f
f
G
e
o
R
e
f
e
r
r
e
r
Options
S
a
m
p
l
e
c
o
u
n
t
Observation:
• Everyone wants to learn
how to start fires
• Few are thinking about
fire safety
• What you don?t know
can hurt you - bots
6
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Bot Detection (Fire Safety)
? Bots are automated programs, sometimes
called crawlers/robots
Examples: search engines, shopping bots, performance monitors
? Significant traffic may be generated by bots
? Can you guess what percentage of sessions are
generated by bots at MEC and Debenhams?
23% at MEC
40% at Debenhams
? Without bot removal, your metrics will be
inaccurate
? Blue Martini has good heuristics, but look
at the bot report and make sure your
performance monitors are recognized
7
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Website Checklist
? Collect clickstreams at 100%
– You can?t improve what you don?t measure
? Turn ReverseDNS on to get host names, not IPs
(note, this is off by default)
? Setup home page redirect correctly so that you do not
lose referrers (avoid client-side redirects)
? When customizing site, don?t lose business events
(search, checkout, etc)
? Do not run reports against web site DB, unless they
are small. That?s why there is the DSS DB
? More details in “Business Intelligence - Getting
Started Guide” on http://developer.bluemartini.com
8
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Real Time Reporting Against Store
? Sometimes executing a small query against live DB makes sense.
? For example, on the www.bluemartini.com site, we provide our
salespeople with a live query JSP that shows visitors from any
domain to help sales activites. Example report below.
? For you, this could be useful for customer support, for example
IE6 user
Running Win 2000
Came from our
press release on
Yahoo
9
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Setting Session Timeout
Recommended timeout
duration is 60 mins
2.5% of sessions with an
item in cart will
experience timeout
? Set your session timeout to an hour
? Note the impact session timeout has on the percent of people
who will lose their shopping cart (or see annoying message)
Watch for an
upcoming article on
developer.bluemartini.com
around this
recommendation
10
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
DSSGen
? Reports should run against a data warehouse (DSS)
? DSSGen builds the data warehouse from the store/click/main DBs
? Normally, such an ETL (Extract/Transform/Load) process takes
months to develop, but with Blue Martini you get 95% of it out of
the box
? The other 5% are:
– Adding your custom tables
– Marking performance monitors/bots
– Changing things due to web site customizations
? Common mistakes
– Inappropriate hardware – Build and Reporting need strong hardware
– Bad database setup (Oracle must be tuned)
– Running DSSGen without –parallel flag
– Running all reports every DSSGen run instead of daily, weekly,
monthly options
11
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Reports
? Blue Martini provides two ways to build reports
– Crystal Reports
? Industry standard
? Easy to layout reports
? Limited capability to transform data (e.g., can?t sort by
percentages requiring multiple passes)
– Blue Martini Reports
? Use transformation chains
? Can integrate multiple investigations into unified report
? Generic reports easy, custom reports require JSP coding
? Visualizations are interactive
12
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Dataviz Webstart
? Every graph has a “dataviz” icon
? Example: the standard dashboard
Demo
? View->Scatterplot
? Right click Settings, X: Day of week Y: hour of day,
size: heatmap, color -> web visits
13
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Easy Insights – Out of the Box Reports
? With the Analysis Center, you get an
extensive set of reports for web analytics
and sales analysis
? Here are some examples from Debenhams
and MEC
14
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Search Effectiveness at MEC
? Customers that search are worth two times as much as
customers that do not search
? Failed searches hurt sales
Visit
Search
(64% successful)
No Search
Last Search Succeeded
Last Search Failed
10%
90%
Avg sale per visit: 2.2X
Avg sale per visit: $X
Avg sale per visit: 2.8X
Avg sale per visit: 0.9X
70%
30%
15
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Top Searches
? Top searched keywords (percent of searches)
– Empty search string (3.9%)
returns over 160 results
– GPS (1.2%)
– sunglasses (0.8%)
– watches (0.6%)
– nalgene (0.5%)
– ecological footprint (0.5%)
? Top failed keywords in the product category (percent of failed searches)
– gift certificate(s) (0.98%)
(already implemented since study)
– arc?teryx (0.44%)
– bear spray (0.44%)
– pedometer (0.37%)
– stroller(s) (0.36%)
– north face (0.33%)
– (gift) registry (0.21%)
Recommendation:
- Do not allow empty search
- Create custom pages for
often searched keywords
Recommendation:
- Parse search string to
remove special characters
- Build extensive thesaurus
- Consider carrying products
16
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Referrers at Debenhams
? Top Referrers
– Google
? 5.8% of all visits are referred by google
? Visit to purchase conversion 1.6%
? Average purchase per visit = 1.8X
– MSN (including search and shopping)
? 11% of all visits are referred by MSN
? Visit to purchase conversion 0.7%
? Average purchase per visit = X
– AOL search
? 0.62% of all visits are referred by AOL Search
? Visit to purchase conversion 2.6%
? Average purchase per visit = 4.8X
Recommendation
Define an ad strategy
based on ROI
Emphasize AOL
17
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
? Understand abandonment and
conversions
? Not just visitor to purchaser, but
also the micro-conversions
? Shopping Cart Abandonment
62% =55% + 45% * 17%
? Abandonment varies from about
25%-80% across sites
? Excellent opportunity
to identify problematic
steps in processes and
improve
? Also a good way to identify
abandoned products, send
targeted e-mails if those products
are on sale
2.0%
7.7%
25%
2.3%
45%
55%
83%
17%
6%
Micro-Conversion Rate at Debenhams
18
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Acxiom
? BMS supports ADN – Acxiom Data Network
? Seamless integration: get username/password
Note: Acxiom recently changed their interface, so you will
need a patch
? Comprehensive collection of US consumer and telephone
data available via the internet
– Multi-sourced database
– Demographic, socioeconomic, and lifestyle information.
– Information on most U.S. households
– Contributors? files refreshed a minimum of 3-12 times per year.
– Data sources include: County Real Estate Property Records, U.S.
Telephone Directories, Public Information, Motor Vehicle Registrations,
Census Directories, Credit Grantors, Public Records and Consumer
Data, Driver?s Licenses, Voter Registrations, Product Registration
Questionnaires, Catalogers, Magazines, Specialty Retailers, Packaged
Goods Manufacturers, Accounts Receivable Files, Warranty Cards
19
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Example - Income
? Graph showing
incomes for a
company that
targets high-end
customers based
on POS purchases
? Income of their
customers in blue
? The US population
in red
Note highest bracket
(30% vs. 5% for US)
Percent
20
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Product Affinities
? Which products sell well together
? Together these form a model which can be
used as a Product Recommender
Note: this does not ship by default as an AC report, but as an
example investigation
21
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Product Affinities at MEC
? Minimum support for the associations is 80 customers
? Confidence: 37% of people who purchased Orbit Sleeping Pad also purchased Orbit Stuff Sack
? Lift: People who purchased Orbit Sleeping Pad were 222 times more likely to purchase the Orbit Stuff Sack
compared to the general population
Product Association
Lift Confidence
Orbit
Sleeping Pad
Cygnet
Sleeping Bag Aladdin 2
Backpack
Primus Stove
Orbit
Stuff Sack
Website
Recommended Products
222 37%
Bambini
Tights Children?s
Bambini
Crewneck
Sweater
Children?s
195 52%
Yeti Crew Neck
Pullover Children?s
Beneficial T?s
Organic Long
Sleeve T-Shirt Kids?
Silk Crew
Women?s
Silk
Long Johns
Women?s
304 73%
Micro Check
Vee Sweater
Volant
Pants
Composite Jacket
Cascade
Entrant
Overmitts
Polartec
300 Double
Mitts
51 48%
Volant
Pants
Windstopper
Alpine Hat
Tremblant 575
Vest Women?s
22
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Product Affinities at Debenhams
? Minimum support for the associations is 50 customers
? Confidence: 41% of people who purchased Fully Reversible Mats also purchased Egyptian Cotton Towels
? Lift: People who purchased Fully Reversible Mats were 456 times more likely to purchase the Egyptian
Cotton Towels compared to the general population
Product Association
Lift Confidence
Website
Recommended
Products
J Jasper
Towels
Fully
Reversible
Mats
456 41%
Egyptian
Cotton
Towels
White Cotton
T-Shirt Bra
Plunge
T-Shirt Bra
246 25%
Black
embroidered
underwired bra
Confidence
1.4%
Confidence
1%
23
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Identifying Top Products - Debenhams
Elephant
Parade
Quilt Cover
Bombay
Quilt Cover
Bombay
Quilt Cover
Opulent
Check
Quilt Cover
Elephant
Parade
Throw
Border
Quilt Cover
Quilt Cover
Oxford edge
border
Polycotton
Mattress
Cover Set
Top Homeware – Bedroom products for the last year
Website recommended products for Homeware - Bedroom
Units: 10.1X
Revenue: 7.3Y
Units: 10.1X
Revenue: 7.3Y
Units: 5X
Revenue: 6.4Y
Units: 10X
Revenue: 8.2Y
Units: 9.7X
Revenue: 6.1Y
Units: X
Revenue: Y
Units: 1.7X
Revenue: 0.8Y
Units: 8.7X
Revenue: 2.6Y x x x
listed not listed not listed not listed
It?s hard to update top products manually
24
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Campaign Analysis - Debenhams
? Analyze the effectiveness of campaigns
0.01% 5.3%
(15.3p/email)
22%
(3.6p/email)
100%
(0.8p/email)
Campaign 3
0.01% 3%
(17.9p/email)
11%
(4.8p/email)
100%
(0.5p/email)
Campaign 2
0.07% 9.3%
(52p/email)
22%
(22.3p/email)
100%
(4.8p/email)
Campaign 1
Orders Click-
throughs
Opens Emails Sent Campaign
Recommendation:
Send targeted emails
Click-through rates are high - good
25
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Campaign and Ad “Tricks”
? To track clickthrough from external ads
– Use jump pages (e.g., www.foo.com/ad2.jsp) or
– Add unused extra parameter to URL
? http://www.bluemartini.com/bi&biwebinar=3
? The biwebinar=3 will be ignored, but you can then see
how many sessions have biwebinar=3 in the first
request
? Examples
– Use with Google adwords
– Use with rented lists, which you can?t mail using
Blue Martini?s campaign management / RM
26
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Page Effectiveness Report
Percentage of visits clicking on different links
14%
13%
9%
Top Menu 6%
8%
Any product link 7%
18% of visits exit at the welcome page
3%
3% 2%
2%
0.3%
2%
2%
2%
0.6%
27
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Top Links followed from the Welcome Page:
Revenue per session associated with visits
10.2X
1.4X 4.2X 1.4X
Top Menu 0.2X
2.3X
Product Links 2.1X
10X
2.3X X 1.3X
5X
3.3X
1.7X
1.2X
Note how effective physical
catalog item #s are
28
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
“Harder” Insights
? Blue Martini collects a lot of data for which
there are no out-of-the-box reports yet
? Some will be in future releases
? Some need to be written depending on your
attributes, hierarchy, etc
? Some answer specific questions you have
? BMAS has developed many such reports
29
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Top Searches On Search Engines /
Internet Portals
? Which keywords do people type on popular search
engines/internet portals to get to our site?
? The „Referrer? recorded as part of the Blue Martini
clickstream data contains these keywords
? For example
– Google Search:
http://www.google.com/search?hl=en&ie=ISO-8859-1&q=jasper+conran
&btnG=Google+Search
– AOL Search:
http://aolsearch.aol.co.uk/web.adp?query=department%20stores%20uk
? Extract the keywords, substitute HTML escape characters with their
ASCII equivalents (such as „space? for „%20?)
? Determine the top searched keywords
30
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Top Searched Keywords (Debenhams)
? Variations of Debenhams are by far the most frequent
– DEBENHAMS
– debenhams.com
– debenhams;direct
– debenhams department store
? Other interesting search keywords
MSN Search
Faith shoes
Nike
Wedding invitations
Luggage
Swimwear
Google
Burberry
Calvin Klein Crave
Cravela / Cravela Shoes
Jane Packer / Suede Boots
AOL Search
burberry
carvela
Mother of the bride
Skiwear
Jane Packer
BRIDESMAIDS
Recommendation
Monitor top searched
keywords to identify
interesting trends
31
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Site Usability
? Form errors logged on 5.0 and later
– Any form that is filled and has an error caught by the Blue
Martini architecture generates a Form Error event
– MINE_EVENT table in the DSS database has these
– BMAS has written initial reports to analyze typical form
errors and help website designers improve the form design
– For example, on MEC there were thousands of errors on
the member application page and address change page
? Many errors for fields that cannot be empty
? Many mismatches between postal code and region
32
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Usability – Form Errors
This was the Bluefly home
page
Looking at form errors, we saw
thousands of errors every day
on this page
Any guesses?
Approved by Bluefly
33
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Improved Home Page
This is the new Bluefly
home page
• Search box added
• E-mail box clearly
marked as email
Approved by Bluefly
34
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Building The Customer Signature
? Building a customer signature is a significant effort, but well worth
the effort
? A signature summarizes customer or visitor behavior across
hundreds of attributes, many which are specific to the site
? Once a signature is built, it can be used to answer many questions.
? The mining algorithms will pick the most important attributes for
each question
? Example attributes computed:
– Total Visits and Sales
– Revenue by Product Family
– Revenue by Month
– Customer State and Country
– Recency, Frequency, Monetary
– Latitude/Longitude from the Customer?s Postal Code
35
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Migration Study - MEC
Oct 2001 – Mar 2002
Apr 2002 – Sep 2002
Spent $1 to
$200
Spent over
$200
Spent over
$200
Spent
under $200
(5.5%)
(94.5%)
? Customers who migrated from low
spenders in one 6 month period to high
spenders in the following 6 month period
36
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Key Characteristics of Migrators at MEC
? During October 2001 – March 2002 (Initial 6
months)
– Purchased at least $70 of merchandise
– Purchased at least twice
– Largest single order was at least $40
– Used free shipping, not express shipping
– Live over 60 aerial kilometers from an MEC
retail store
– Bought from these product families, such as
socks, t-shirts, and accessories
– Customers who purchased shoulder bags and
child carries were LESS LIKELY to migrate
Recommendation:
Score light spending
customers based on their
likelihood of migrating
and market to high
scorers.
37
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
RFM Analysis
? RFM – Recency, Frequency, Monetary
(described in appendix). Insights from
Debenhams
– Anonymous purchasers have lower average order amount
– Customers who have opted out [of e-mail] tend to have
higher average order amount
– People in the age range 30-40 and 40-50 spend more on
average
38
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Customer Locations Relative to Retail Stores
Map of Canada with store locations.
Black dots show store locations.
Heavy purchasing areas away from retail
stores can suggest new retail store locations
No stores in several hot areas:
MEC is building a store in
Montreal right now.
39
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Distance From Nearest Store (MEC)
? People farther
away from
retail stores
– spend more
on average
– Account for
most of the
revenues
40
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
World Wide Revenue Detail
UK – 98.8%
US – 0.6%
Australia – 0.1%
NOTE: About 50% of the non-UK
orders are wedding list purchases
Low Medium High
Although Debenhams online site only ships in
the UK, we see some revenue from the rest of
the world.
41
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Other Results at MEC (See Appendix)
? Free shipping changed to flat-fee (C$6 flat charge)
– Orders - down 9.5%
– Total sales - up 6.5%
? Gear Swap (buy/sell used gear)
– Visit-to-Purchase very low: 0.34% vs. 2.1% for non gear-swap
– However, these visitors converted to purchasing customers (over
multiple visits) at a rate 62% higher than visitors who never visited
gear swap!
? Visits where an FYI page (For-Your-Information) page was
viewed had a Visit-to-Purchase conversion of 7.1%
42
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Other Results at Debenhams (See Appendix)
? People who got the timeout page for a high percentage of their
sessions are less likely to migrate (to heavy spenders)
? Revenue due to wedding list item purchases is clearly affected by
summer weather
– Weddings are more common in the summer in the UK
– In June/July, 65% of revenues were generated through the wedding
list
? A page-tagging-based service provider was used, but was about
30% inaccurate due to people hitting links before page download
was complete
43
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
MEC Actions Resulting From Analysis
? Done
? Implemented links to on-line shop from Gear Swap
(appendix has before and after pictures)
? Implemented gift certificates and increased their visibility
on the site (#1 failed search)
? Used the content page view information to inform our IA
redesign
? Planned
? Refine internal measures by removing bot, internal and
production host visits
? Design and implement controlled experiments to help
guide our content planning process in particular
? Product Recommender (as budget permits)
44
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Debenhams Actions
? Note: Debenhams analysis delivered recently, so
there was less time to take action
? Done
– Increased session timeout
? Planned
– Some failed searches were for products available at stores
but not online. Will import all brands and offer a store
locator for brands not available online
– Looking to enable „save basket? functionality
45
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
BIG ROI Project
? The Business Intelligence Group (BIG) Guarantees Return On
Investment in 6 months or You Don?t Pay
? Process
– Client provides us with a backup of databases
– BI group analyzes data and makes recommendations
– BI group provides the JSP changes using a test/control group
methodology
? Half the people will see the “old” site (control group)
? Half the people will see our recommendations (test groups)
– Client approves the changes, QAs, deploys
– Client provides us with a second backup for assessment
– Client pays 20% of the revenue delta between the test and control
groups, extrapolated to the next 6 months, assuming it would be
implemented for everyone
46
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
The ROI
? Assuming your profit margin is about 20%, this project has a 6-
month ROI
? With a test/control group methodology, the delta revenue is clear:
– Seasons, ads, campaigns will affect both the test group and the
control group in the same way
– Once the experiment is done, you can end the experiment and stop
the control group (old site)
? Example:
– Measurement time: 2 weeks
– Revenue for control group is $200,000
– Revenue for test group is $205,000
– Expected revenue if implemented for everyone: $410,000
– You pay: 26 weeks / 2 weeks * ($410,000-$400,000) * 20% = $26,000
47
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Qualifications
? Must run 4.1.4 or later, preferably 5.0 or 5.5
? Must have clickstream collection on at 100%
? Must commit to timeline
– Must be able to generate backup of DBs and
send to Blue Martini
– Commit to implement changes in 3 weeks from
the time code is provided by BMAS
? We will not recommend significant site changes
? We will provide the code to implement changes
48
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Examples and Commitments
? Potential recommendations:
– Improvements to usability
– Improvements to search
– Product recommender for cross sells, top products
– Identification and reduction of abandonment
? Commitments by Blue Martini
– We will fix code that the BI group provided for this
agreement, free of charge, to help deployment
– This SOW (Statement Of Work) will be governed by your
existing contract with Blue Martini.
(Easy contractual agreement.)
49
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Additional Resources
? Business Intelligence - Getting Started Guide on
http://developer.bluemartini.com
? Data Mining Tutorial on http://www.bluemartini.com/bi
? MEC case study at http://www.bluemartini.com/bi
? Debenhams case study at http://www.bluemartini.com/bi
? Appendix has more examples
For questions and a copy of these slides, send e-mail to
[email protected]
50
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Q&A
? Questions and Answers
? Type your questions into the Q&A
(upper-left)
51
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Appendix
? Here are additional slides with some
interesting insights
52
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
RFM Analysis (Debenhams)
? Recency, Frequency, and Monetary
calculations are used extensively in retail for
customer segmentation
? Implemented the Arthur-Hughes RFM Cube
– R, F, and M scores are binned into 5 equal sized
bins
– Each dimension is labeled 1 (best) – 5 (worst)
? Interactive visualization using Filter Charts
? Look at charts instead of cell-tables
53
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Complete RFM
Recommendation
Targeted marketing campaigns to
convert people to repeat
purchasers, assuming they did not
opt-out of e-mails
Majority of
customers have
purchased once
More frequent
customers have
higher average
order amount
Low Medium High Low Medium High
54
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Interacting with the RFM visualization
? Explore sub-segments with filter charts
? People in the age range 30-40 and 40-50 spend more on
average
? Anonymous purchasers have lower average order amount
Average Order
Amount mapped to
color
Low Medium High
55
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
RFM for Debenhams Card Owners
Debenhams card owners
Large group (> 1000)
High average order amount
Purchased once (F = 5)
Not purchased recently (R=5)
Recommendation
Send targeted email
campaign since these are
Debenham?s customers.
Try to “awaken” them!
Low Medium High Low Medium High
56
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Customers who have Opted Out
Recommendation
Send targeted emails to
prevent email fatigue
? Customers who have opted out tend to have
higher average order amount
Recommendation
Log changes to opt out
settings and track
unsubscribes to identify
email fatigue
Low Medium High
57
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Free Shipping Offer (MEC)
? Free shipping stopped on Aug 14, 2002
? A flat $6 Canadian Dollars shipping charge introduced
? Express shipping at higher charge continues
? Observations
– Total sales - up 6.5%
– Revenue (excluding shipping and tax) - up 2.8%
– Orders - down 9.5%
– Average Sales per Order – up 18%
58
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Free Shipping Offer (Cont.)
? The distribution shows fewer
orders from low spenders
(probably a good thing)
? No impact on rest of buyers
Fewer low spenders
(<= $50)
59
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Free Shipping Offer (Cont.)
? Breakdown of orders by shipping method
? More people used express shipping, probably because the delta to ship
express wasn?t as large (from C$6 instead of from C$0)
Free/Standard Shipping Express Shipping
60
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Gear Swap Pages (Cont.)
Recommendation:
Link back to MEC Shopping from Gear Swap
Shop MEC Cycling
61
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Gear Swap Pages (Cont.)
Done
62
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Definitions for Gear Swap Analysis
? A visitor is defined as someone who is registered
(MEC member) or is identified by a cookie
– Note that in the Blue Martini system a registered user will
have all of his/her cookies combined into a single visitor ID
? Comparing visitors who viewed gear swap with those
who did not
– Several non-bot sessions have 1 request that just visited the
MEC homepage (Main/home.jsp)
– To get to the Gear Swap section you have to click at least
twice
– To make a fair comparison we have excluded all 1 request
sessions that just visited the MEC homepage
(Main/home.jsp) from the following analysis
63
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Distribution of Gear Swap Visitors
? Visitors who viewed Gear Swap pages had a 62%
higher visitor to purchaser conversion ratio as
compared to those who did not view Gear Swap
Visitors: X
MEC members: Y
Purchasing
Customers: Z
Visitors: 14.3% of X
MEC members: 20.8% of Y
Purchasing
Customers: 21.1% of Z
Visitors: 85.7% of X
MEC members: 79.2% of Y
Purchasing
Customers: 78.9% of Z
Visitors who ever
viewed Gear Swap
Visitors who never
viewed Gear Swap
Overall
64
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Distribution of Orders (the real ROI)
Orders: X
Average Basket Value: $Y
Visitors who ever
viewed Gear Swap
Visitors who never
viewed Gear Swap
Overall
Orders: X
Average Basket Value: 1.05 * Y
Orders: 3,875 (78.3%)
Average Basket Value: 0.98*Y
65
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Distribution of Visits
? Although, Gear Swap visitors have lower visit-to-purchase
conversion than non Gear Swap visitors, they visit more often
and their overall visitor-to-purchase conversion is higher
Visits: X
Visitors who ever
viewed Gear Swap
Visitors who never
viewed Gear Swap
Overall
Visits: 24.8% of X
Visit to Purchase Conversion:
1.94%
Visits: 75.2% of X
Visit to Purchase Conversion:
2.3%
66
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Effectiveness of FYI Pages
? People viewing FYIs are more likely to purchase
Viewed FYI
Visits: 6.2% of all
Purchases: 23% of all
Visit-to-Purchase: 7.1%
Avg. Sales per Visit: 6.1X
Did Not View FYI
Visits: 93.8% of all
Purchases: 77% of all
Visit-to-Purchase: 1.2%
Avg. Sales per Visit: $X
Recommendation:
Controlled experiment to study the effect of FYIs
67
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
FYIs (Cont.)
? Setting up controlled experiments to study the cause-
effect relationship of FYI
– Select a handful of products (say 6) for introducing FYIs
– Randomly show the new FYIs to 50% of the visitors viewing
these products and don?t show the FYIs to the other 50% of
the visitors
– At the end of the trial period (say 2-3 weeks) measure the
visit-to-purchase conversion of the two groups
– Determine if there is a significant difference in the visit-to-
purchase conversion of the two groups
68
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Debenhams Migrators: Timeout
Some attributes are more useful
when combined with other
attributes
For each visitor we computed the
number of sessions which went to
the page timeout.jsp
This was binned as shown on the
X axis of the chart
The height shows the number of
visitors in each bin and color
shows the percentage of those
visitors who migrated
Just looking at this variable alone
it is difficult to tell what the pattern
is
69
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
By combining the number of timeout
sessions with the total number of
sessions for each visitor a pattern
emerges
In this heatmap the X axis shows the
total number of sessions, the Y axis
shows the number of timeout
sessions, and color shows the
percentage of migrators at each pair
of values
The green along the diagonal shows
that people who got the timeout page
for a high percentage of their
sessions are less likely to migrate
Migrators: Timeout
70
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Migrators: Timeout
The number of sessions a visitor has is a
good indicator of whether or not they will
migrate
However there are some inconsistencies
that are apparent. For example, why does
the percent of visitors who migrate drop at
19 sessions?
We can construct new attributes based on
the relationship we saw between the
number of timeouts and the number of
sessions
Two more attributes can be created:
• Number of sessions that did not time out
• Percentage of sessions that did not time out
71
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
68,000 visitors with no timeout sessions have been filtered out
*
Migrators: Timeout
Number of sessions without timeout
is a good predictor of migration
Percentage of sessions without timeout
is also a good indicator of migration
72
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Distribution of Wedding Purchases over Time
Revenue due to wedding list item purchases clearly affected by
summer weather, when weddings are more common in the UK
73
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Hidden Page Requests
? Page Tagging
– Debenhams used a third-party ASP that uses page tagging
to track users
– Tag generation implemented using a separate JSP
– This causes two requests to be executed
? Generates higher server load
? Hurts user experience
? Many users are clicking on links before the complete
page downloads
– Page Tag jsp is never executed
– Statistics are inaccurate: 34% of non-bot sessions did not
execute the page tagging jsp
doc_571626177.pdf
With the Analysis Center, you get an extensive set of reports for web analytics and sales analysis.
1
© Copyright 2003, Blue Martini Software. San Mateo California, USA
Deriving Key Insights from Blue Martini
Business Intelligence
Ronny Kohavi
Vice President,
Business Intelligence
Blue Martini Software
Anne Ford
E-Commerce
Project Manager
MEC
Bill Maginn
Internet
Controller
Debenhams
2
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Agenda
? Debenhams and MEC
? Web site, DSSGen, and Bots
? Easy insights – out of the box reports
? Harder insights – what you could write
? Actions at Debenhams and MEC
? BIG ROI project
? Q&A
3
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
? Debenhams plc, “Britain's favourite department store”
– UK's fashion retail leader for more than 200 years
– Runs approximately 100 department stores in the UK and Ireland
– Offering brand-name women's, men's and children's apparel as well as
high-end housewares and cosmetics
– Debenhams also offers the award winning „Queen's Award for
Enterprise - April 2001? bridal registry service
? Debenhams online profile
– Over 500,000 customers
– About £18 million in total sales in the last year
– Wedding list purchases account for about half of sales
? Clickstreams
– About 2M page views / week
– About 6,000 new customers per week
Debenhams
4
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
MEC
? MEC – Mountain Equipment Co-op
? Canada?s leading supplier of quality outdoor gear and clothing
? MEC has
– 1.8 million members
– Sales over $160 million
– Seven physical retail stores
? As a co-op, MEC supports the community in several ways.
– One of them is gearswap, an area of the site for selling used gear,
where MEC makes no money
– Products link to information promoting environmental responsibility
5
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Survey
We asked early participants what they wanted to hear
in this webinar
0
2
4
6
8
10
12
R
e
p
o
r
t
i
n
g
V
i
z
B
o
t
M
i
g
r
a
t
o
r
A
s
s
o
c
C
r
o
s
s
-
s
e
l
l
C
a
m
p
a
i
g
n
U
s
a
b
i
l
i
t
y
S
e
a
r
c
h
A
d
e
f
f
G
e
o
R
e
f
e
r
r
e
r
Options
S
a
m
p
l
e
c
o
u
n
t
Observation:
• Everyone wants to learn
how to start fires
• Few are thinking about
fire safety
• What you don?t know
can hurt you - bots
6
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Bot Detection (Fire Safety)
? Bots are automated programs, sometimes
called crawlers/robots
Examples: search engines, shopping bots, performance monitors
? Significant traffic may be generated by bots
? Can you guess what percentage of sessions are
generated by bots at MEC and Debenhams?
23% at MEC
40% at Debenhams
? Without bot removal, your metrics will be
inaccurate
? Blue Martini has good heuristics, but look
at the bot report and make sure your
performance monitors are recognized
7
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Website Checklist
? Collect clickstreams at 100%
– You can?t improve what you don?t measure
? Turn ReverseDNS on to get host names, not IPs
(note, this is off by default)
? Setup home page redirect correctly so that you do not
lose referrers (avoid client-side redirects)
? When customizing site, don?t lose business events
(search, checkout, etc)
? Do not run reports against web site DB, unless they
are small. That?s why there is the DSS DB
? More details in “Business Intelligence - Getting
Started Guide” on http://developer.bluemartini.com
8
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Real Time Reporting Against Store
? Sometimes executing a small query against live DB makes sense.
? For example, on the www.bluemartini.com site, we provide our
salespeople with a live query JSP that shows visitors from any
domain to help sales activites. Example report below.
? For you, this could be useful for customer support, for example
IE6 user
Running Win 2000
Came from our
press release on
Yahoo
9
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Setting Session Timeout
Recommended timeout
duration is 60 mins
2.5% of sessions with an
item in cart will
experience timeout
? Set your session timeout to an hour
? Note the impact session timeout has on the percent of people
who will lose their shopping cart (or see annoying message)
Watch for an
upcoming article on
developer.bluemartini.com
around this
recommendation
10
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
DSSGen
? Reports should run against a data warehouse (DSS)
? DSSGen builds the data warehouse from the store/click/main DBs
? Normally, such an ETL (Extract/Transform/Load) process takes
months to develop, but with Blue Martini you get 95% of it out of
the box
? The other 5% are:
– Adding your custom tables
– Marking performance monitors/bots
– Changing things due to web site customizations
? Common mistakes
– Inappropriate hardware – Build and Reporting need strong hardware
– Bad database setup (Oracle must be tuned)
– Running DSSGen without –parallel flag
– Running all reports every DSSGen run instead of daily, weekly,
monthly options
11
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Reports
? Blue Martini provides two ways to build reports
– Crystal Reports
? Industry standard
? Easy to layout reports
? Limited capability to transform data (e.g., can?t sort by
percentages requiring multiple passes)
– Blue Martini Reports
? Use transformation chains
? Can integrate multiple investigations into unified report
? Generic reports easy, custom reports require JSP coding
? Visualizations are interactive
12
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Dataviz Webstart
? Every graph has a “dataviz” icon
? Example: the standard dashboard
Demo
? View->Scatterplot
? Right click Settings, X: Day of week Y: hour of day,
size: heatmap, color -> web visits
13
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Easy Insights – Out of the Box Reports
? With the Analysis Center, you get an
extensive set of reports for web analytics
and sales analysis
? Here are some examples from Debenhams
and MEC
14
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Search Effectiveness at MEC
? Customers that search are worth two times as much as
customers that do not search
? Failed searches hurt sales
Visit
Search
(64% successful)
No Search
Last Search Succeeded
Last Search Failed
10%
90%
Avg sale per visit: 2.2X
Avg sale per visit: $X
Avg sale per visit: 2.8X
Avg sale per visit: 0.9X
70%
30%
15
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Top Searches
? Top searched keywords (percent of searches)
– Empty search string (3.9%)
returns over 160 results
– GPS (1.2%)
– sunglasses (0.8%)
– watches (0.6%)
– nalgene (0.5%)
– ecological footprint (0.5%)
? Top failed keywords in the product category (percent of failed searches)
– gift certificate(s) (0.98%)
(already implemented since study)
– arc?teryx (0.44%)
– bear spray (0.44%)
– pedometer (0.37%)
– stroller(s) (0.36%)
– north face (0.33%)
– (gift) registry (0.21%)
Recommendation:
- Do not allow empty search
- Create custom pages for
often searched keywords
Recommendation:
- Parse search string to
remove special characters
- Build extensive thesaurus
- Consider carrying products
16
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Referrers at Debenhams
? Top Referrers
? 5.8% of all visits are referred by google
? Visit to purchase conversion 1.6%
? Average purchase per visit = 1.8X
– MSN (including search and shopping)
? 11% of all visits are referred by MSN
? Visit to purchase conversion 0.7%
? Average purchase per visit = X
– AOL search
? 0.62% of all visits are referred by AOL Search
? Visit to purchase conversion 2.6%
? Average purchase per visit = 4.8X
Recommendation
Define an ad strategy
based on ROI
Emphasize AOL
17
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
? Understand abandonment and
conversions
? Not just visitor to purchaser, but
also the micro-conversions
? Shopping Cart Abandonment
62% =55% + 45% * 17%
? Abandonment varies from about
25%-80% across sites
? Excellent opportunity
to identify problematic
steps in processes and
improve
? Also a good way to identify
abandoned products, send
targeted e-mails if those products
are on sale
2.0%
7.7%
25%
2.3%
45%
55%
83%
17%
6%
Micro-Conversion Rate at Debenhams
18
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Acxiom
? BMS supports ADN – Acxiom Data Network
? Seamless integration: get username/password
Note: Acxiom recently changed their interface, so you will
need a patch
? Comprehensive collection of US consumer and telephone
data available via the internet
– Multi-sourced database
– Demographic, socioeconomic, and lifestyle information.
– Information on most U.S. households
– Contributors? files refreshed a minimum of 3-12 times per year.
– Data sources include: County Real Estate Property Records, U.S.
Telephone Directories, Public Information, Motor Vehicle Registrations,
Census Directories, Credit Grantors, Public Records and Consumer
Data, Driver?s Licenses, Voter Registrations, Product Registration
Questionnaires, Catalogers, Magazines, Specialty Retailers, Packaged
Goods Manufacturers, Accounts Receivable Files, Warranty Cards
19
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Example - Income
? Graph showing
incomes for a
company that
targets high-end
customers based
on POS purchases
? Income of their
customers in blue
? The US population
in red
Note highest bracket
(30% vs. 5% for US)
Percent
20
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Product Affinities
? Which products sell well together
? Together these form a model which can be
used as a Product Recommender
Note: this does not ship by default as an AC report, but as an
example investigation
21
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Product Affinities at MEC
? Minimum support for the associations is 80 customers
? Confidence: 37% of people who purchased Orbit Sleeping Pad also purchased Orbit Stuff Sack
? Lift: People who purchased Orbit Sleeping Pad were 222 times more likely to purchase the Orbit Stuff Sack
compared to the general population
Product Association
Lift Confidence
Orbit
Sleeping Pad
Cygnet
Sleeping Bag Aladdin 2
Backpack
Primus Stove
Orbit
Stuff Sack
Website
Recommended Products
222 37%
Bambini
Tights Children?s
Bambini
Crewneck
Sweater
Children?s
195 52%
Yeti Crew Neck
Pullover Children?s
Beneficial T?s
Organic Long
Sleeve T-Shirt Kids?
Silk Crew
Women?s
Silk
Long Johns
Women?s
304 73%
Micro Check
Vee Sweater
Volant
Pants
Composite Jacket
Cascade
Entrant
Overmitts
Polartec
300 Double
Mitts
51 48%
Volant
Pants
Windstopper
Alpine Hat
Tremblant 575
Vest Women?s
22
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Product Affinities at Debenhams
? Minimum support for the associations is 50 customers
? Confidence: 41% of people who purchased Fully Reversible Mats also purchased Egyptian Cotton Towels
? Lift: People who purchased Fully Reversible Mats were 456 times more likely to purchase the Egyptian
Cotton Towels compared to the general population
Product Association
Lift Confidence
Website
Recommended
Products
J Jasper
Towels
Fully
Reversible
Mats
456 41%
Egyptian
Cotton
Towels
White Cotton
T-Shirt Bra
Plunge
T-Shirt Bra
246 25%
Black
embroidered
underwired bra
Confidence
1.4%
Confidence
1%
23
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Identifying Top Products - Debenhams
Elephant
Parade
Quilt Cover
Bombay
Quilt Cover
Bombay
Quilt Cover
Opulent
Check
Quilt Cover
Elephant
Parade
Throw
Border
Quilt Cover
Quilt Cover
Oxford edge
border
Polycotton
Mattress
Cover Set
Top Homeware – Bedroom products for the last year
Website recommended products for Homeware - Bedroom
Units: 10.1X
Revenue: 7.3Y
Units: 10.1X
Revenue: 7.3Y
Units: 5X
Revenue: 6.4Y
Units: 10X
Revenue: 8.2Y
Units: 9.7X
Revenue: 6.1Y
Units: X
Revenue: Y
Units: 1.7X
Revenue: 0.8Y
Units: 8.7X
Revenue: 2.6Y x x x
listed not listed not listed not listed
It?s hard to update top products manually
24
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Campaign Analysis - Debenhams
? Analyze the effectiveness of campaigns
0.01% 5.3%
(15.3p/email)
22%
(3.6p/email)
100%
(0.8p/email)
Campaign 3
0.01% 3%
(17.9p/email)
11%
(4.8p/email)
100%
(0.5p/email)
Campaign 2
0.07% 9.3%
(52p/email)
22%
(22.3p/email)
100%
(4.8p/email)
Campaign 1
Orders Click-
throughs
Opens Emails Sent Campaign
Recommendation:
Send targeted emails
Click-through rates are high - good
25
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Campaign and Ad “Tricks”
? To track clickthrough from external ads
– Use jump pages (e.g., www.foo.com/ad2.jsp) or
– Add unused extra parameter to URL
? http://www.bluemartini.com/bi&biwebinar=3
? The biwebinar=3 will be ignored, but you can then see
how many sessions have biwebinar=3 in the first
request
? Examples
– Use with Google adwords
– Use with rented lists, which you can?t mail using
Blue Martini?s campaign management / RM
26
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Page Effectiveness Report
Percentage of visits clicking on different links
14%
13%
9%
Top Menu 6%
8%
Any product link 7%
18% of visits exit at the welcome page
3%
3% 2%
2%
0.3%
2%
2%
2%
0.6%
27
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Top Links followed from the Welcome Page:
Revenue per session associated with visits
10.2X
1.4X 4.2X 1.4X
Top Menu 0.2X
2.3X
Product Links 2.1X
10X
2.3X X 1.3X
5X
3.3X
1.7X
1.2X
Note how effective physical
catalog item #s are
28
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
“Harder” Insights
? Blue Martini collects a lot of data for which
there are no out-of-the-box reports yet
? Some will be in future releases
? Some need to be written depending on your
attributes, hierarchy, etc
? Some answer specific questions you have
? BMAS has developed many such reports
29
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Top Searches On Search Engines /
Internet Portals
? Which keywords do people type on popular search
engines/internet portals to get to our site?
? The „Referrer? recorded as part of the Blue Martini
clickstream data contains these keywords
? For example
– Google Search:
http://www.google.com/search?hl=en&ie=ISO-8859-1&q=jasper+conran
&btnG=Google+Search
– AOL Search:
http://aolsearch.aol.co.uk/web.adp?query=department%20stores%20uk
? Extract the keywords, substitute HTML escape characters with their
ASCII equivalents (such as „space? for „%20?)
? Determine the top searched keywords
30
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Top Searched Keywords (Debenhams)
? Variations of Debenhams are by far the most frequent
– DEBENHAMS
– debenhams.com
– debenhams;direct
– debenhams department store
? Other interesting search keywords
MSN Search
Faith shoes
Nike
Wedding invitations
Luggage
Swimwear
Burberry
Calvin Klein Crave
Cravela / Cravela Shoes
Jane Packer / Suede Boots
AOL Search
burberry
carvela
Mother of the bride
Skiwear
Jane Packer
BRIDESMAIDS
Recommendation
Monitor top searched
keywords to identify
interesting trends
31
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Site Usability
? Form errors logged on 5.0 and later
– Any form that is filled and has an error caught by the Blue
Martini architecture generates a Form Error event
– MINE_EVENT table in the DSS database has these
– BMAS has written initial reports to analyze typical form
errors and help website designers improve the form design
– For example, on MEC there were thousands of errors on
the member application page and address change page
? Many errors for fields that cannot be empty
? Many mismatches between postal code and region
32
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Usability – Form Errors
This was the Bluefly home
page
Looking at form errors, we saw
thousands of errors every day
on this page
Any guesses?
Approved by Bluefly
33
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Improved Home Page
This is the new Bluefly
home page
• Search box added
• E-mail box clearly
marked as email
Approved by Bluefly
34
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Building The Customer Signature
? Building a customer signature is a significant effort, but well worth
the effort
? A signature summarizes customer or visitor behavior across
hundreds of attributes, many which are specific to the site
? Once a signature is built, it can be used to answer many questions.
? The mining algorithms will pick the most important attributes for
each question
? Example attributes computed:
– Total Visits and Sales
– Revenue by Product Family
– Revenue by Month
– Customer State and Country
– Recency, Frequency, Monetary
– Latitude/Longitude from the Customer?s Postal Code
35
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Migration Study - MEC
Oct 2001 – Mar 2002
Apr 2002 – Sep 2002
Spent $1 to
$200
Spent over
$200
Spent over
$200
Spent
under $200
(5.5%)
(94.5%)
? Customers who migrated from low
spenders in one 6 month period to high
spenders in the following 6 month period
36
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Key Characteristics of Migrators at MEC
? During October 2001 – March 2002 (Initial 6
months)
– Purchased at least $70 of merchandise
– Purchased at least twice
– Largest single order was at least $40
– Used free shipping, not express shipping
– Live over 60 aerial kilometers from an MEC
retail store
– Bought from these product families, such as
socks, t-shirts, and accessories
– Customers who purchased shoulder bags and
child carries were LESS LIKELY to migrate
Recommendation:
Score light spending
customers based on their
likelihood of migrating
and market to high
scorers.
37
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
RFM Analysis
? RFM – Recency, Frequency, Monetary
(described in appendix). Insights from
Debenhams
– Anonymous purchasers have lower average order amount
– Customers who have opted out [of e-mail] tend to have
higher average order amount
– People in the age range 30-40 and 40-50 spend more on
average
38
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Customer Locations Relative to Retail Stores
Map of Canada with store locations.
Black dots show store locations.
Heavy purchasing areas away from retail
stores can suggest new retail store locations
No stores in several hot areas:
MEC is building a store in
Montreal right now.
39
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Distance From Nearest Store (MEC)
? People farther
away from
retail stores
– spend more
on average
– Account for
most of the
revenues
40
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
World Wide Revenue Detail
UK – 98.8%
US – 0.6%
Australia – 0.1%
NOTE: About 50% of the non-UK
orders are wedding list purchases
Low Medium High
Although Debenhams online site only ships in
the UK, we see some revenue from the rest of
the world.
41
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Other Results at MEC (See Appendix)
? Free shipping changed to flat-fee (C$6 flat charge)
– Orders - down 9.5%
– Total sales - up 6.5%
? Gear Swap (buy/sell used gear)
– Visit-to-Purchase very low: 0.34% vs. 2.1% for non gear-swap
– However, these visitors converted to purchasing customers (over
multiple visits) at a rate 62% higher than visitors who never visited
gear swap!
? Visits where an FYI page (For-Your-Information) page was
viewed had a Visit-to-Purchase conversion of 7.1%
42
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Other Results at Debenhams (See Appendix)
? People who got the timeout page for a high percentage of their
sessions are less likely to migrate (to heavy spenders)
? Revenue due to wedding list item purchases is clearly affected by
summer weather
– Weddings are more common in the summer in the UK
– In June/July, 65% of revenues were generated through the wedding
list
? A page-tagging-based service provider was used, but was about
30% inaccurate due to people hitting links before page download
was complete
43
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
MEC Actions Resulting From Analysis
? Done
? Implemented links to on-line shop from Gear Swap
(appendix has before and after pictures)
? Implemented gift certificates and increased their visibility
on the site (#1 failed search)
? Used the content page view information to inform our IA
redesign
? Planned
? Refine internal measures by removing bot, internal and
production host visits
? Design and implement controlled experiments to help
guide our content planning process in particular
? Product Recommender (as budget permits)
44
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Debenhams Actions
? Note: Debenhams analysis delivered recently, so
there was less time to take action
? Done
– Increased session timeout
? Planned
– Some failed searches were for products available at stores
but not online. Will import all brands and offer a store
locator for brands not available online
– Looking to enable „save basket? functionality
45
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
BIG ROI Project
? The Business Intelligence Group (BIG) Guarantees Return On
Investment in 6 months or You Don?t Pay
? Process
– Client provides us with a backup of databases
– BI group analyzes data and makes recommendations
– BI group provides the JSP changes using a test/control group
methodology
? Half the people will see the “old” site (control group)
? Half the people will see our recommendations (test groups)
– Client approves the changes, QAs, deploys
– Client provides us with a second backup for assessment
– Client pays 20% of the revenue delta between the test and control
groups, extrapolated to the next 6 months, assuming it would be
implemented for everyone
46
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
The ROI
? Assuming your profit margin is about 20%, this project has a 6-
month ROI
? With a test/control group methodology, the delta revenue is clear:
– Seasons, ads, campaigns will affect both the test group and the
control group in the same way
– Once the experiment is done, you can end the experiment and stop
the control group (old site)
? Example:
– Measurement time: 2 weeks
– Revenue for control group is $200,000
– Revenue for test group is $205,000
– Expected revenue if implemented for everyone: $410,000
– You pay: 26 weeks / 2 weeks * ($410,000-$400,000) * 20% = $26,000
47
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Qualifications
? Must run 4.1.4 or later, preferably 5.0 or 5.5
? Must have clickstream collection on at 100%
? Must commit to timeline
– Must be able to generate backup of DBs and
send to Blue Martini
– Commit to implement changes in 3 weeks from
the time code is provided by BMAS
? We will not recommend significant site changes
? We will provide the code to implement changes
48
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Examples and Commitments
? Potential recommendations:
– Improvements to usability
– Improvements to search
– Product recommender for cross sells, top products
– Identification and reduction of abandonment
? Commitments by Blue Martini
– We will fix code that the BI group provided for this
agreement, free of charge, to help deployment
– This SOW (Statement Of Work) will be governed by your
existing contract with Blue Martini.
(Easy contractual agreement.)
49
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Additional Resources
? Business Intelligence - Getting Started Guide on
http://developer.bluemartini.com
? Data Mining Tutorial on http://www.bluemartini.com/bi
? MEC case study at http://www.bluemartini.com/bi
? Debenhams case study at http://www.bluemartini.com/bi
? Appendix has more examples
For questions and a copy of these slides, send e-mail to
[email protected]
50
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Q&A
? Questions and Answers
? Type your questions into the Q&A
(upper-left)
51
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Appendix
? Here are additional slides with some
interesting insights
52
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
RFM Analysis (Debenhams)
? Recency, Frequency, and Monetary
calculations are used extensively in retail for
customer segmentation
? Implemented the Arthur-Hughes RFM Cube
– R, F, and M scores are binned into 5 equal sized
bins
– Each dimension is labeled 1 (best) – 5 (worst)
? Interactive visualization using Filter Charts
? Look at charts instead of cell-tables
53
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Complete RFM
Recommendation
Targeted marketing campaigns to
convert people to repeat
purchasers, assuming they did not
opt-out of e-mails
Majority of
customers have
purchased once
More frequent
customers have
higher average
order amount
Low Medium High Low Medium High
54
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Interacting with the RFM visualization
? Explore sub-segments with filter charts
? People in the age range 30-40 and 40-50 spend more on
average
? Anonymous purchasers have lower average order amount
Average Order
Amount mapped to
color
Low Medium High
55
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
RFM for Debenhams Card Owners
Debenhams card owners
Large group (> 1000)
High average order amount
Purchased once (F = 5)
Not purchased recently (R=5)
Recommendation
Send targeted email
campaign since these are
Debenham?s customers.
Try to “awaken” them!
Low Medium High Low Medium High
56
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Customers who have Opted Out
Recommendation
Send targeted emails to
prevent email fatigue
? Customers who have opted out tend to have
higher average order amount
Recommendation
Log changes to opt out
settings and track
unsubscribes to identify
email fatigue
Low Medium High
57
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Free Shipping Offer (MEC)
? Free shipping stopped on Aug 14, 2002
? A flat $6 Canadian Dollars shipping charge introduced
? Express shipping at higher charge continues
? Observations
– Total sales - up 6.5%
– Revenue (excluding shipping and tax) - up 2.8%
– Orders - down 9.5%
– Average Sales per Order – up 18%
58
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Free Shipping Offer (Cont.)
? The distribution shows fewer
orders from low spenders
(probably a good thing)
? No impact on rest of buyers
Fewer low spenders
(<= $50)
59
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Free Shipping Offer (Cont.)
? Breakdown of orders by shipping method
? More people used express shipping, probably because the delta to ship
express wasn?t as large (from C$6 instead of from C$0)
Free/Standard Shipping Express Shipping
60
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Gear Swap Pages (Cont.)
Recommendation:
Link back to MEC Shopping from Gear Swap
Shop MEC Cycling
61
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Gear Swap Pages (Cont.)
Done
62
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Definitions for Gear Swap Analysis
? A visitor is defined as someone who is registered
(MEC member) or is identified by a cookie
– Note that in the Blue Martini system a registered user will
have all of his/her cookies combined into a single visitor ID
? Comparing visitors who viewed gear swap with those
who did not
– Several non-bot sessions have 1 request that just visited the
MEC homepage (Main/home.jsp)
– To get to the Gear Swap section you have to click at least
twice
– To make a fair comparison we have excluded all 1 request
sessions that just visited the MEC homepage
(Main/home.jsp) from the following analysis
63
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Distribution of Gear Swap Visitors
? Visitors who viewed Gear Swap pages had a 62%
higher visitor to purchaser conversion ratio as
compared to those who did not view Gear Swap
Visitors: X
MEC members: Y
Purchasing
Customers: Z
Visitors: 14.3% of X
MEC members: 20.8% of Y
Purchasing
Customers: 21.1% of Z
Visitors: 85.7% of X
MEC members: 79.2% of Y
Purchasing
Customers: 78.9% of Z
Visitors who ever
viewed Gear Swap
Visitors who never
viewed Gear Swap
Overall
64
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Distribution of Orders (the real ROI)
Orders: X
Average Basket Value: $Y
Visitors who ever
viewed Gear Swap
Visitors who never
viewed Gear Swap
Overall
Orders: X
Average Basket Value: 1.05 * Y
Orders: 3,875 (78.3%)
Average Basket Value: 0.98*Y
65
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Distribution of Visits
? Although, Gear Swap visitors have lower visit-to-purchase
conversion than non Gear Swap visitors, they visit more often
and their overall visitor-to-purchase conversion is higher
Visits: X
Visitors who ever
viewed Gear Swap
Visitors who never
viewed Gear Swap
Overall
Visits: 24.8% of X
Visit to Purchase Conversion:
1.94%
Visits: 75.2% of X
Visit to Purchase Conversion:
2.3%
66
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Effectiveness of FYI Pages
? People viewing FYIs are more likely to purchase
Viewed FYI
Visits: 6.2% of all
Purchases: 23% of all
Visit-to-Purchase: 7.1%
Avg. Sales per Visit: 6.1X
Did Not View FYI
Visits: 93.8% of all
Purchases: 77% of all
Visit-to-Purchase: 1.2%
Avg. Sales per Visit: $X
Recommendation:
Controlled experiment to study the effect of FYIs
67
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
FYIs (Cont.)
? Setting up controlled experiments to study the cause-
effect relationship of FYI
– Select a handful of products (say 6) for introducing FYIs
– Randomly show the new FYIs to 50% of the visitors viewing
these products and don?t show the FYIs to the other 50% of
the visitors
– At the end of the trial period (say 2-3 weeks) measure the
visit-to-purchase conversion of the two groups
– Determine if there is a significant difference in the visit-to-
purchase conversion of the two groups
68
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Debenhams Migrators: Timeout
Some attributes are more useful
when combined with other
attributes
For each visitor we computed the
number of sessions which went to
the page timeout.jsp
This was binned as shown on the
X axis of the chart
The height shows the number of
visitors in each bin and color
shows the percentage of those
visitors who migrated
Just looking at this variable alone
it is difficult to tell what the pattern
is
69
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
By combining the number of timeout
sessions with the total number of
sessions for each visitor a pattern
emerges
In this heatmap the X axis shows the
total number of sessions, the Y axis
shows the number of timeout
sessions, and color shows the
percentage of migrators at each pair
of values
The green along the diagonal shows
that people who got the timeout page
for a high percentage of their
sessions are less likely to migrate
Migrators: Timeout
70
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Migrators: Timeout
The number of sessions a visitor has is a
good indicator of whether or not they will
migrate
However there are some inconsistencies
that are apparent. For example, why does
the percent of visitors who migrate drop at
19 sessions?
We can construct new attributes based on
the relationship we saw between the
number of timeouts and the number of
sessions
Two more attributes can be created:
• Number of sessions that did not time out
• Percentage of sessions that did not time out
71
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
68,000 visitors with no timeout sessions have been filtered out
*
Migrators: Timeout
Number of sessions without timeout
is a good predictor of migration
Percentage of sessions without timeout
is also a good indicator of migration
72
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Distribution of Wedding Purchases over Time
Revenue due to wedding list item purchases clearly affected by
summer weather, when weddings are more common in the UK
73
© Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Hidden Page Requests
? Page Tagging
– Debenhams used a third-party ASP that uses page tagging
to track users
– Tag generation implemented using a separate JSP
– This causes two requests to be executed
? Generates higher server load
? Hurts user experience
? Many users are clicking on links before the complete
page downloads
– Page Tag jsp is never executed
– Statistics are inaccurate: 34% of non-bot sessions did not
execute the page tagging jsp
doc_571626177.pdf