Description
The mobile phone has transitioned from its traditional role as a mere communication medium to that of a multipurpose personal gadget. Technological advancements and the proliferation of devices across operating systems (Apple iOS, Android and Windows Mobile) and platforms have made it more challenging for hardware manufacturers and application developers to develop and roll out new products.
Selecting the Right Mobile Test Automation
Strategy: Challenges and Principles
Understanding the various options and choosing the tools and
automation processes best suited to testing both mobile Web and
native mobile apps can reduce complexity and speed time to value.
Executive Summary
The mobile phone has transitioned from its tra-
ditional role as a mere communication medium
to that of a multipurpose personal gadget. Tech-
nological advancements and the proliferation
of devices across operating systems (Apple iOS,
Android and Windows Mobile) and platforms have
made it more challenging for hardware manufac-
turers and application developers to develop and
roll out new products.
For vendor peace of mind, mobile applica-
tions must be tested to ensure they run on key
platforms and across a multitude of networks.
Despite the pressures of short mobile develop-
ment cycles, quality testing of applications across
operating systems, device platforms and networks
is a necessary but daunting task to ensure
long-term success in what is a highly fragmented
and competitive global market. Moreover, non-
functional testing — including usability, security
and adaptability — is as critical as functional
testing. Effective testing can help device makers
and application developers collect appropriate
metrics that improve product quality.
This white paper explores industry best practices
on mobile testing and demonstrates effective
ways of managing mobile application quality
through test automation. As with conven-
tional applications, there is an increased need
to automate mobile applications testing to
improve scale and ef?ciency. A well-orchestrated
automation strategy enables hardware manufac-
turers and software developers to reduce effort
and accelerate time to market.
Mobile Technology’s Growth Promise
The growth of mobile devices is evidenced by the
following:
•
Globally, the user base of location-based
services has grown from 96 million in 2009
to more than 526 million in 2012, according to
Gartner Research.
1
•
Worldwide mobile advertising and content
revenue is forecast to reach $67 billion this
year.
2
•
IDC predicts that smartphone shipments will
reach 978 million in 2014.
3
•
By 2016, smartphones and tablets will
put power in the hands of a billion global
consumers, according to Forrester
Research, Inc.
4
•
Mobile device makers are providing
customized platforms to increase their share
of the customer’s wallet.
•
Cognizant 20-20 Insights
cognizant 20-20 insights | september 2012
cognizant 20-20 insights 2
Figure 1
Real-time Scenario for a Smartphone-based App: Sample Test Execution
Use Case 1
Mobile Applications
Use Case 2
Use Case n
Wi-Fi
iPhone
Blackberry
Android
Real Device Emulators
Devices/Browsers Sky Fire Dolphin
iphone3
iphone4
iphone5
Nexus One
Moto Droid
Samsung
Captivate
HTC Incredible
Nexus S
Moto Droid X
Curve 8330
Tour 9630
Bold 9650
Bold 9000 Touch
iOS Native
3,4&5
Android Native
(2.1,2.2,2.3, 3.0,4.0)
Blackberry Native
5,6 ,7,8 & 9
Opera Mini Firefox 4 Firefox 5 Firefox 6
Opera
Mobile 11.1
Device Anywhere M
o
b
i
l
e
T
e
s
t
L
a
b
Two Key Challenges in Mobile
Application Test Automation
Mobile Platforms and Technology Coverage
The primary factor that determines an automation
tool’s success is its ability to work across platforms
and technology stacks. The following challenges
in?uence automation success:
•
Device Diversity:
>
Multiple platforms and browsers.
>
Rendering differences.
>
Mobile devices with varied application run
times.
•
Network Challenges:
>
Multiple network types (e.g., GSM/GPRS/
Wi-Fi/Wi-Max).
>
Different speeds of connectivity across
geographies.
>
Multiple network operators with custom-
ized network features.
•
Hardware Challenges:
>
Limitations in processing speed.
>
Limitations of mobile memory size.
>
Differences in device communication
protocols (e.g., WAP/HTTP).
Test Execution Challenge
Test execution across different devices and brows-
ers is a huge, challenging effort. For example,
consider a test case that is created and needs to
be executed on three different platforms — iOS,
Android and Blackberry. In this example, the test
case is expected to be executed on 13 different
devices and in 10 different combinations, which
leads to 130 rounds of testing. Designing a test
execution matrix of this scope is both dif?cult and
time-consuming (see Figures 1 and 2).
Mobile Testing Industry Opportunities
In the growing mobile market, the need for spe-
cialized mobile testing is growing. The following
three opportunities contribute almost 80% of
total market demand.
Mobile Automation Testing
There are only a handful of tools available to suc-
cessfully automate testing. The ROI promised by
automation testing highlights the need for more
options.
Functional Testing
Mobile functional testing always involves a non-
heuristic component that cannot be automated.
Mobile Performance Testing
Compared with a desktop/notebook environment,
mobile device system resources (e.g., processing
power, memory, etc.) are limited. Hence, perfor-
mance testing of mobile applications is crucial.
Therefore, this domain is growing and has great
potential.
cognizant 20-20 insights 3
iOS
Current Version 6.0
Closed Source
Apple
Multitasking
Safari Web Browser
Android
Current Version 4.2
Open Source
Google
Multitasking
Windows Phone
Current Version 7.5
Key Version 6.5
Closed Source
Microsoft
Multitasking
Internet Explorer
Blackberry OS
Current Version 7.0
Closed Source
RIM
Multitasking
Webkit Browser
Symbian
Symbian4
Open Source
Symbian
Multitasking
Key Versions
(iPhone 4.0, iPad 2)
Key Version 2.3, 3.0, 4.0
Chrome/Webkit Browser
Key Version
6.x & 7.x series
Webkit Browser
Figure 2
Smartphone Platforms and Browser Combinations
Mobile Automation: What and Why?
Mobile applications are becoming increasingly
sophisticated, signi?cantly increasing the require-
ment for functional testing (see Figures 3 and
4). To tackle this, test organizations are increas-
ingly exploring alternatives to traditional manual
testing. Automated testing is a highly effective
approach to mobile app QA that can provide
signi?cant business returns, provided it is imple-
mented by using the right tools and architecture,
factoring in cross-platform challenges.
The following speci?c types of testing also need
to be automated:
•
Cross-platform compatibility testing: This
is necessitated by the growing number of
handsets and platforms.
•
User experience testing: Most testing orga-
nizations have limited experience with design
and execution of usability tests.
•
Field or network testing: Testing must be
performed in a geographically distributed envi-
ronment to account for a variety of network
types.
•
Structural challenges: Integrating mobile
testing toolsets into existing IT systems is a
crucial hurdle.
Test Automation Mechanisms
for Mobile Apps
A mobile Web site can be automated in one of the
following three manners:
•
User agent-based automation: This approach
leverages user agent add-ons that come with
popular browsers such as Mozilla Firefox
and Google Chrome. The user agents help
render the speci?c Web content that would be
displayed on the device onto a regular desktop
browser. The functionality can be automated
by using popular tools such as QTP, Selenium
or RFT — each of which supports all desktop
browsers.
•
Cloud-based automation: Cloud-based test
automation vendors support Web-based QA
across platforms. Cloud-based automated
solutions are highly effective as they are
offered as a uni?ed package.
•
Lab-based automation (devices/simulators):
This method involves setting up tools in a
test environment and leveraging simulators
or physical devices to automate testing using
popular open-source tools such as Selenium.
Web driver–supported device browsers and
simulators are best suited here.
Approaches for Native and Hybrid
Mobile Applications
Cloud-based Approach
Cloud-based test automation is one popular
means for achieving native and hybrid types of
test automation. The method is useful when
long-term automation is required across a set
of prede?ned devices. Commercial tool add-on
mechanisms such as HP QTP and IBM RFT are also
available for test engineers who are familiar with
cognizant 20-20 insights 4
Backend Middleware Device UI
SOA Testing
UI Testing
Security Testing
Interruption
Fully Automatable Partially Automatable Not Automatable
Communication
Framework/
Server
Mobile Web
Services
Enterprise
Data
Application
Data
Mobile Client
Admin Services
Integration Testing
Integration
Testing
Device Management
Testing
Authentication Testing
Installation
Testing
Functional
Testing
Compatibility
Testing
Device
Pro?ling
Mobile Website
A
p
p
l
i
c
a
t
i
o
n
L
a
y
e
r
s
T
y
p
e
s
o
f
T
e
s
t
i
n
g
Figure 3
Three-tier Mobile Application Testing Framework
industry-wide products. The ability to automate
from anywhere is a great advantage: All that
is needed is connectivity to the cloud lab. The
“automation-from-anywhere” feature is a distinct
advantage when using tools such as Perfecto
Mobile and Device Anywhere.
Lab-based Approach
There are multiple tool sets with different capa-
bilities available for organizations interested in
a lab-based approach. Tools such as EggPlant,
Jamo, See Test, ZAP, SilkTest and Test Quest
provide industry tool add-ons. They are useful
in enabling automation in a closed environment,
for continuous automation or to achieve test
regression functionality. They also cover nonfunc-
tional scenarios that handle mechanisms largely
relevant to mobile apps.
Open Source/Platform-speci?c Tools
Device platform-speci?c tools reliability is a high
priority. Platform vendors supply tools such as
Robotium for Android and iOS instruments for
iOS to provide this facility. However, to leverage
this effectively, it requires additional program-
ming skills.
Source: Cognizant; based on 350-plus projects performed across all vertical markets and our
analysis of industry trends.
Figure 4
0
10
20
30
40
50
60
70
80
90
100
Mobile
Automation
Testing
P
e
r
c
e
n
t
a
g
e
o
f
t
e
s
t
s
p
e
r
f
o
r
m
e
d
u
s
i
n
g
a
u
t
o
m
a
t
e
d
t
o
o
l
s
Functional
Testing
Mobile
Performance
Testing
Mobile Cross-
Browser
Testing
Security Testing SOA/
Middleware
Testing
Package
Application
Testing (e.g., ERP)
Enterprise
Application
Testing
Potential Areas of Automation in End-to-End Mobile Application Testing
cognizant 20-20 insights 5
Cognizant’s Tools
We have our own platform to easily automate
testing of mobile native and Web apps automation
across platforms.
Our platform ensures single type scripting that
works cross-platform and thus guarantees greater
reusability of scripts. The platform also offers
an easy interface for testers to auto-generate
scripts. For instance, scripts can be automated
without any programming or scripting language
knowledge. It also comes with a facility to
automate as applications are created to support
the Agile development methodology.
Mobile Application Testing Tools
Selection Criteria
There are many successful techniques that can
make testing applications on mobile devices
more effective and ef?cient. It is essential to
ensure functional and nonfunctional compliance
of mobile applications on devices and mobile
automation. It is also key to establish labs for
testing mobile applications.
Some basic criteria for mobile application testing
include the use of emulators, multiple browsers,
different operating systems and the unique types
of testing required for GUI, ?eld and interrup-
tion testing. There are roughly 10 key principles
that need to be considered while selecting the
right tool.
11 Key Principles in Selecting the Right Tools for
Mobile Testing
It is good practice to start with a proof of concept
(PoC) on critical platforms before narrowing
down to a tool or tool set. One tool may not
achieve cross-platform automation goals. When
test automation tools are evaluated, organiza-
tions must not overlook how they integrate with
test management environments. One starting
point is to identify reusable scenarios across
platforms and group them accordingly. This facili-
tates the creation of an appropriate, time-saving
framework.
For native apps, choose critical test cases that can
be run across releases as automation candidates
and ensure that there is a cross-platform reus-
ability and multi-browser compatibility. Always
keep in mind different form factors while deter-
mining test coverage, especially for the mobile
Web. Consider different connectivity aspects for
automation execution runs.
In addition, the following recommendations will
be helpful in informing your test automation
journey:
•
Perform tool feasibility tests, because mobile
technologies and platforms vary.
•
Select tools that support both platform
simulators and devices, because you can mix
and match devices and simulators to optimize
runs on different platforms.
•
Aim for automation in nonfunctional areas
— interruption and hardware scenarios that
include battery state changes.
•
Always optimize on platform support: In some
cases, there may be a need for one or more
tools to perform automation.
•
Identify multiple device support and versions
support.
•
Determine the use of utilities and reusable
functions that add value to automation.
•
Ensure ROI per each mobile platform and
duration of execution as part of the tool con-
sideration process.
•
Look for avenues for early automation, because
most mobile projects are Agile.
•
Integrated execution with the test manage-
ment tool and application lifecycle manage-
ment is an important success factor for a tool.
•
Look for data-driven automation support:
Iterations in the execution will increase
coverage and ROI.
•
Good candidates for automation are test
cases that have repeated business logic,
functional ?ows or data-driven tests: Smoke
test and build acceptance testing, regression
testing, synchronization testing, compatibility
testing, GUI testing and testing non-dynam-
ic UI components, as well as cross-platform
functional testing.
Moving Forward
The principles and success factors of test
automation for mobile applications are much
different than those traditionally applied to con-
ventional applications. The attributes of devices,
environment, tool sets and test coverage add
About Cognizant
Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process out-
sourcing services, dedicated to helping the world’s leading companies build stronger businesses. Headquartered in
Teaneck, New Jersey (U.S.), Cognizant combines a passion for client satisfaction, technology innovation, deep industry
and business process expertise, and a global, collaborative workforce that embodies the future of work. With over 50
delivery centers worldwide and approximately 145,200 employees as of June 30, 2012, Cognizant is a member of the
NASDAQ-100, the S&P 500, the Forbes Global 2000, and the Fortune 500 and is ranked among the top performing
and fastest growing companies in the world. Visit us online at www.cognizant.com or follow us on Twitter: Cognizant.
World Headquarters
500 Frank W. Burr Blvd.
Teaneck, NJ 07666 USA
Phone: +1 201 801 0233
Fax: +1 201 801 0243
Toll Free: +1 888 937 3277
Email: [email protected]
European Headquarters
1 Kingdom Street
Paddington Central
London W2 6BD
Phone: +44 (0) 20 7297 7600
Fax: +44 (0) 20 7121 0102
Email: [email protected]
India Operations Headquarters
#5/535, Old Mahabalipuram Road
Okkiyam Pettai, Thoraipakkam
Chennai, 600 096 India
Phone: +91 (0) 44 4209 6000
Fax: +91 (0) 44 4209 6060
Email: [email protected]
© Copyright 2012, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any
means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein is
subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.
multiple layers of complexity. It requires keen
insight and a customized approach for a well-
orchestrated automation strategy.
The best practices covered in this paper will
guide you toward setting test automation goals
and choosing appropriate coverage, the right
tool sets and a comprehensive approach. Making
automation effective and ef?cient ultimately
yields cost and time reduction in the app develop-
ment lifecycle.
About the Authors
Pradeep Kumar Govindasamy leads Cognizant’s Automation and Mobile Testing Center of Excellence,
a practice comprising more than 1,500 experts servicing global customers. He brings over 12 years of
experience in the information technology industry and is a valued speaker on topics such as automation,
test data management and tools at events worldwide including Starwest 2009 and 2011, Stareast 2011,
Swiss Testing Day, HP Universe and EuroStar. Pradeep can be reached at PradeepKumar.Govindasamy@
cognizant.com.
Ramakrishnan Venkatasubramanian heads Cognizant’s Automation R&D and Offshore Mobile Testing
teams. He has over 12 years of experience in the information technology industry. His experience spans
software development, product engineering, testing, test automation, mobile and technology consulting.
Ramakrishnan can be reached at [email protected].
Footnotes
1http://www.gartner.com/it/page.jsp?id=1230413
2http://www.mediapost.com/publications/search-marketing-daily/edition/2012/07/13/?print
3
IDC Worldwide Smartphone 2012-2016 Forecast Update, June 2012.
4
Ted Schadler, John C. McCarthy, “Mobile Is the New Face of Engagement,”
Forrester Research, February 2012.
About Cognizant’s Mobile Testing CoE
Cognizant‘s Mobile Testing Center of Excellence (CoE) provides state-of-the-art solutions for mobile testing require-
ments with best-of-the-breed testing practices and processes. It specializes in offerings such as unit testing, functional
testing, multi-language compatibility testing, security testing, automation testing, performance testing, device testing
and application testing. Our Mobile Testing CoE features a strong pool of certi?ed mobile testing experts, built through
long-term competency development programs. In addition, it houses smart products and frameworks powered by a
focused R&D team, which takes mobile application testing to a whole new dimension.
doc_975489243.pdf
The mobile phone has transitioned from its traditional role as a mere communication medium to that of a multipurpose personal gadget. Technological advancements and the proliferation of devices across operating systems (Apple iOS, Android and Windows Mobile) and platforms have made it more challenging for hardware manufacturers and application developers to develop and roll out new products.
Selecting the Right Mobile Test Automation
Strategy: Challenges and Principles
Understanding the various options and choosing the tools and
automation processes best suited to testing both mobile Web and
native mobile apps can reduce complexity and speed time to value.
Executive Summary
The mobile phone has transitioned from its tra-
ditional role as a mere communication medium
to that of a multipurpose personal gadget. Tech-
nological advancements and the proliferation
of devices across operating systems (Apple iOS,
Android and Windows Mobile) and platforms have
made it more challenging for hardware manufac-
turers and application developers to develop and
roll out new products.
For vendor peace of mind, mobile applica-
tions must be tested to ensure they run on key
platforms and across a multitude of networks.
Despite the pressures of short mobile develop-
ment cycles, quality testing of applications across
operating systems, device platforms and networks
is a necessary but daunting task to ensure
long-term success in what is a highly fragmented
and competitive global market. Moreover, non-
functional testing — including usability, security
and adaptability — is as critical as functional
testing. Effective testing can help device makers
and application developers collect appropriate
metrics that improve product quality.
This white paper explores industry best practices
on mobile testing and demonstrates effective
ways of managing mobile application quality
through test automation. As with conven-
tional applications, there is an increased need
to automate mobile applications testing to
improve scale and ef?ciency. A well-orchestrated
automation strategy enables hardware manufac-
turers and software developers to reduce effort
and accelerate time to market.
Mobile Technology’s Growth Promise
The growth of mobile devices is evidenced by the
following:
•
Globally, the user base of location-based
services has grown from 96 million in 2009
to more than 526 million in 2012, according to
Gartner Research.
1
•
Worldwide mobile advertising and content
revenue is forecast to reach $67 billion this
year.
2
•
IDC predicts that smartphone shipments will
reach 978 million in 2014.
3
•
By 2016, smartphones and tablets will
put power in the hands of a billion global
consumers, according to Forrester
Research, Inc.
4
•
Mobile device makers are providing
customized platforms to increase their share
of the customer’s wallet.
•
Cognizant 20-20 Insights
cognizant 20-20 insights | september 2012
cognizant 20-20 insights 2
Figure 1
Real-time Scenario for a Smartphone-based App: Sample Test Execution
Use Case 1
Mobile Applications
Use Case 2
Use Case n
Wi-Fi
iPhone
Blackberry
Android
Real Device Emulators
Devices/Browsers Sky Fire Dolphin
iphone3
iphone4
iphone5
Nexus One
Moto Droid
Samsung
Captivate
HTC Incredible
Nexus S
Moto Droid X
Curve 8330
Tour 9630
Bold 9650
Bold 9000 Touch
iOS Native
3,4&5
Android Native
(2.1,2.2,2.3, 3.0,4.0)
Blackberry Native
5,6 ,7,8 & 9
Opera Mini Firefox 4 Firefox 5 Firefox 6
Opera
Mobile 11.1
Device Anywhere M
o
b
i
l
e
T
e
s
t
L
a
b
Two Key Challenges in Mobile
Application Test Automation
Mobile Platforms and Technology Coverage
The primary factor that determines an automation
tool’s success is its ability to work across platforms
and technology stacks. The following challenges
in?uence automation success:
•
Device Diversity:
>
Multiple platforms and browsers.
>
Rendering differences.
>
Mobile devices with varied application run
times.
•
Network Challenges:
>
Multiple network types (e.g., GSM/GPRS/
Wi-Fi/Wi-Max).
>
Different speeds of connectivity across
geographies.
>
Multiple network operators with custom-
ized network features.
•
Hardware Challenges:
>
Limitations in processing speed.
>
Limitations of mobile memory size.
>
Differences in device communication
protocols (e.g., WAP/HTTP).
Test Execution Challenge
Test execution across different devices and brows-
ers is a huge, challenging effort. For example,
consider a test case that is created and needs to
be executed on three different platforms — iOS,
Android and Blackberry. In this example, the test
case is expected to be executed on 13 different
devices and in 10 different combinations, which
leads to 130 rounds of testing. Designing a test
execution matrix of this scope is both dif?cult and
time-consuming (see Figures 1 and 2).
Mobile Testing Industry Opportunities
In the growing mobile market, the need for spe-
cialized mobile testing is growing. The following
three opportunities contribute almost 80% of
total market demand.
Mobile Automation Testing
There are only a handful of tools available to suc-
cessfully automate testing. The ROI promised by
automation testing highlights the need for more
options.
Functional Testing
Mobile functional testing always involves a non-
heuristic component that cannot be automated.
Mobile Performance Testing
Compared with a desktop/notebook environment,
mobile device system resources (e.g., processing
power, memory, etc.) are limited. Hence, perfor-
mance testing of mobile applications is crucial.
Therefore, this domain is growing and has great
potential.
cognizant 20-20 insights 3
iOS
Current Version 6.0
Closed Source
Apple
Multitasking
Safari Web Browser
Android
Current Version 4.2
Open Source
Multitasking
Windows Phone
Current Version 7.5
Key Version 6.5
Closed Source
Microsoft
Multitasking
Internet Explorer
Blackberry OS
Current Version 7.0
Closed Source
RIM
Multitasking
Webkit Browser
Symbian
Symbian4
Open Source
Symbian
Multitasking
Key Versions
(iPhone 4.0, iPad 2)
Key Version 2.3, 3.0, 4.0
Chrome/Webkit Browser
Key Version
6.x & 7.x series
Webkit Browser
Figure 2
Smartphone Platforms and Browser Combinations
Mobile Automation: What and Why?
Mobile applications are becoming increasingly
sophisticated, signi?cantly increasing the require-
ment for functional testing (see Figures 3 and
4). To tackle this, test organizations are increas-
ingly exploring alternatives to traditional manual
testing. Automated testing is a highly effective
approach to mobile app QA that can provide
signi?cant business returns, provided it is imple-
mented by using the right tools and architecture,
factoring in cross-platform challenges.
The following speci?c types of testing also need
to be automated:
•
Cross-platform compatibility testing: This
is necessitated by the growing number of
handsets and platforms.
•
User experience testing: Most testing orga-
nizations have limited experience with design
and execution of usability tests.
•
Field or network testing: Testing must be
performed in a geographically distributed envi-
ronment to account for a variety of network
types.
•
Structural challenges: Integrating mobile
testing toolsets into existing IT systems is a
crucial hurdle.
Test Automation Mechanisms
for Mobile Apps
A mobile Web site can be automated in one of the
following three manners:
•
User agent-based automation: This approach
leverages user agent add-ons that come with
popular browsers such as Mozilla Firefox
and Google Chrome. The user agents help
render the speci?c Web content that would be
displayed on the device onto a regular desktop
browser. The functionality can be automated
by using popular tools such as QTP, Selenium
or RFT — each of which supports all desktop
browsers.
•
Cloud-based automation: Cloud-based test
automation vendors support Web-based QA
across platforms. Cloud-based automated
solutions are highly effective as they are
offered as a uni?ed package.
•
Lab-based automation (devices/simulators):
This method involves setting up tools in a
test environment and leveraging simulators
or physical devices to automate testing using
popular open-source tools such as Selenium.
Web driver–supported device browsers and
simulators are best suited here.
Approaches for Native and Hybrid
Mobile Applications
Cloud-based Approach
Cloud-based test automation is one popular
means for achieving native and hybrid types of
test automation. The method is useful when
long-term automation is required across a set
of prede?ned devices. Commercial tool add-on
mechanisms such as HP QTP and IBM RFT are also
available for test engineers who are familiar with
cognizant 20-20 insights 4
Backend Middleware Device UI
SOA Testing
UI Testing
Security Testing
Interruption
Fully Automatable Partially Automatable Not Automatable
Communication
Framework/
Server
Mobile Web
Services
Enterprise
Data
Application
Data
Mobile Client
Admin Services
Integration Testing
Integration
Testing
Device Management
Testing
Authentication Testing
Installation
Testing
Functional
Testing
Compatibility
Testing
Device
Pro?ling
Mobile Website
A
p
p
l
i
c
a
t
i
o
n
L
a
y
e
r
s
T
y
p
e
s
o
f
T
e
s
t
i
n
g
Figure 3
Three-tier Mobile Application Testing Framework
industry-wide products. The ability to automate
from anywhere is a great advantage: All that
is needed is connectivity to the cloud lab. The
“automation-from-anywhere” feature is a distinct
advantage when using tools such as Perfecto
Mobile and Device Anywhere.
Lab-based Approach
There are multiple tool sets with different capa-
bilities available for organizations interested in
a lab-based approach. Tools such as EggPlant,
Jamo, See Test, ZAP, SilkTest and Test Quest
provide industry tool add-ons. They are useful
in enabling automation in a closed environment,
for continuous automation or to achieve test
regression functionality. They also cover nonfunc-
tional scenarios that handle mechanisms largely
relevant to mobile apps.
Open Source/Platform-speci?c Tools
Device platform-speci?c tools reliability is a high
priority. Platform vendors supply tools such as
Robotium for Android and iOS instruments for
iOS to provide this facility. However, to leverage
this effectively, it requires additional program-
ming skills.
Source: Cognizant; based on 350-plus projects performed across all vertical markets and our
analysis of industry trends.
Figure 4
0
10
20
30
40
50
60
70
80
90
100
Mobile
Automation
Testing
P
e
r
c
e
n
t
a
g
e
o
f
t
e
s
t
s
p
e
r
f
o
r
m
e
d
u
s
i
n
g
a
u
t
o
m
a
t
e
d
t
o
o
l
s
Functional
Testing
Mobile
Performance
Testing
Mobile Cross-
Browser
Testing
Security Testing SOA/
Middleware
Testing
Package
Application
Testing (e.g., ERP)
Enterprise
Application
Testing
Potential Areas of Automation in End-to-End Mobile Application Testing
cognizant 20-20 insights 5
Cognizant’s Tools
We have our own platform to easily automate
testing of mobile native and Web apps automation
across platforms.
Our platform ensures single type scripting that
works cross-platform and thus guarantees greater
reusability of scripts. The platform also offers
an easy interface for testers to auto-generate
scripts. For instance, scripts can be automated
without any programming or scripting language
knowledge. It also comes with a facility to
automate as applications are created to support
the Agile development methodology.
Mobile Application Testing Tools
Selection Criteria
There are many successful techniques that can
make testing applications on mobile devices
more effective and ef?cient. It is essential to
ensure functional and nonfunctional compliance
of mobile applications on devices and mobile
automation. It is also key to establish labs for
testing mobile applications.
Some basic criteria for mobile application testing
include the use of emulators, multiple browsers,
different operating systems and the unique types
of testing required for GUI, ?eld and interrup-
tion testing. There are roughly 10 key principles
that need to be considered while selecting the
right tool.
11 Key Principles in Selecting the Right Tools for
Mobile Testing
It is good practice to start with a proof of concept
(PoC) on critical platforms before narrowing
down to a tool or tool set. One tool may not
achieve cross-platform automation goals. When
test automation tools are evaluated, organiza-
tions must not overlook how they integrate with
test management environments. One starting
point is to identify reusable scenarios across
platforms and group them accordingly. This facili-
tates the creation of an appropriate, time-saving
framework.
For native apps, choose critical test cases that can
be run across releases as automation candidates
and ensure that there is a cross-platform reus-
ability and multi-browser compatibility. Always
keep in mind different form factors while deter-
mining test coverage, especially for the mobile
Web. Consider different connectivity aspects for
automation execution runs.
In addition, the following recommendations will
be helpful in informing your test automation
journey:
•
Perform tool feasibility tests, because mobile
technologies and platforms vary.
•
Select tools that support both platform
simulators and devices, because you can mix
and match devices and simulators to optimize
runs on different platforms.
•
Aim for automation in nonfunctional areas
— interruption and hardware scenarios that
include battery state changes.
•
Always optimize on platform support: In some
cases, there may be a need for one or more
tools to perform automation.
•
Identify multiple device support and versions
support.
•
Determine the use of utilities and reusable
functions that add value to automation.
•
Ensure ROI per each mobile platform and
duration of execution as part of the tool con-
sideration process.
•
Look for avenues for early automation, because
most mobile projects are Agile.
•
Integrated execution with the test manage-
ment tool and application lifecycle manage-
ment is an important success factor for a tool.
•
Look for data-driven automation support:
Iterations in the execution will increase
coverage and ROI.
•
Good candidates for automation are test
cases that have repeated business logic,
functional ?ows or data-driven tests: Smoke
test and build acceptance testing, regression
testing, synchronization testing, compatibility
testing, GUI testing and testing non-dynam-
ic UI components, as well as cross-platform
functional testing.
Moving Forward
The principles and success factors of test
automation for mobile applications are much
different than those traditionally applied to con-
ventional applications. The attributes of devices,
environment, tool sets and test coverage add
About Cognizant
Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process out-
sourcing services, dedicated to helping the world’s leading companies build stronger businesses. Headquartered in
Teaneck, New Jersey (U.S.), Cognizant combines a passion for client satisfaction, technology innovation, deep industry
and business process expertise, and a global, collaborative workforce that embodies the future of work. With over 50
delivery centers worldwide and approximately 145,200 employees as of June 30, 2012, Cognizant is a member of the
NASDAQ-100, the S&P 500, the Forbes Global 2000, and the Fortune 500 and is ranked among the top performing
and fastest growing companies in the world. Visit us online at www.cognizant.com or follow us on Twitter: Cognizant.
World Headquarters
500 Frank W. Burr Blvd.
Teaneck, NJ 07666 USA
Phone: +1 201 801 0233
Fax: +1 201 801 0243
Toll Free: +1 888 937 3277
Email: [email protected]
European Headquarters
1 Kingdom Street
Paddington Central
London W2 6BD
Phone: +44 (0) 20 7297 7600
Fax: +44 (0) 20 7121 0102
Email: [email protected]
India Operations Headquarters
#5/535, Old Mahabalipuram Road
Okkiyam Pettai, Thoraipakkam
Chennai, 600 096 India
Phone: +91 (0) 44 4209 6000
Fax: +91 (0) 44 4209 6060
Email: [email protected]
© Copyright 2012, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any
means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein is
subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.
multiple layers of complexity. It requires keen
insight and a customized approach for a well-
orchestrated automation strategy.
The best practices covered in this paper will
guide you toward setting test automation goals
and choosing appropriate coverage, the right
tool sets and a comprehensive approach. Making
automation effective and ef?cient ultimately
yields cost and time reduction in the app develop-
ment lifecycle.
About the Authors
Pradeep Kumar Govindasamy leads Cognizant’s Automation and Mobile Testing Center of Excellence,
a practice comprising more than 1,500 experts servicing global customers. He brings over 12 years of
experience in the information technology industry and is a valued speaker on topics such as automation,
test data management and tools at events worldwide including Starwest 2009 and 2011, Stareast 2011,
Swiss Testing Day, HP Universe and EuroStar. Pradeep can be reached at PradeepKumar.Govindasamy@
cognizant.com.
Ramakrishnan Venkatasubramanian heads Cognizant’s Automation R&D and Offshore Mobile Testing
teams. He has over 12 years of experience in the information technology industry. His experience spans
software development, product engineering, testing, test automation, mobile and technology consulting.
Ramakrishnan can be reached at [email protected].
Footnotes
1http://www.gartner.com/it/page.jsp?id=1230413
2http://www.mediapost.com/publications/search-marketing-daily/edition/2012/07/13/?print
3
IDC Worldwide Smartphone 2012-2016 Forecast Update, June 2012.
4
Ted Schadler, John C. McCarthy, “Mobile Is the New Face of Engagement,”
Forrester Research, February 2012.
About Cognizant’s Mobile Testing CoE
Cognizant‘s Mobile Testing Center of Excellence (CoE) provides state-of-the-art solutions for mobile testing require-
ments with best-of-the-breed testing practices and processes. It specializes in offerings such as unit testing, functional
testing, multi-language compatibility testing, security testing, automation testing, performance testing, device testing
and application testing. Our Mobile Testing CoE features a strong pool of certi?ed mobile testing experts, built through
long-term competency development programs. In addition, it houses smart products and frameworks powered by a
focused R&D team, which takes mobile application testing to a whole new dimension.
doc_975489243.pdf