White paper
Mobile
Performance
Testing
Abstract
Why Mobile Performance Testing?
This whitepaper is to provide insight about the approach for Mobile
Performance Testing to collect Server Side, Network & Device Performance
Metrics. Tools used in this approach are HPE Load Runner, HPE Mobile
Center, tPacketCapture Android Application, SeeTest Automation.
“As long as you have a Cell Phone you’re never alone”
Stanley Victor Paskavich
A Nasscom and Akamai report, published on
17 August 2016, claims India will have over
730 million internet users and 702 million
smartphone users by 2020. This makes
smartphone internet usage 96% of overall
internet usage, leaving only 4% to Personal
Computers. In 2010 Steve Jobs termed this as
“Post PC Era”.
In this era, for every business it is important
to have their own High Performance Mobile
Application to survive in competitive market
conditions.According to a survey by Akamai,
47% of consumers expect web pages to
load in two seconds or less and 20% expect
mobile apps to load instantly. The indings of
a 2015 mobile app user survey conducted
by Dimensional Research showed that
80% of app users will only attempt to use
a problematic app three times or less and
36% said that an app with slow performance
issues made them have a lower opinion of the
company. With this increased usage of Mobile
Internet and high performance expectations
from consumers, it is inevitable that Mobile
Performance testing is of utmost importance.
2
Mobile Performance Testing
In Traditional Performance Testing of Browser based – web applications;
we solely concentrate on Server side performance, because Client side
performance is not impacted by how many users are using the website
concurrently. While this being true with Mobile devices as well, there are
few more new attributes added to performance impacting factors.
Client side/Mobile Device
resource ut
ilisation
It is important for a Native Mobile Application
to optimise the CPU usage, Memory usage,
Number of HTTP calls over internet, Battery
utilisation to achieve a better user experience,
resulting better user rating. So, measuring
Mobile Application Performance will not be
complete without Mobile Phone resource
utilisation stats.
Mobile Performance
Metrics to be collected
In Mobile Performance testing we need to capture…
Server side performance stats
• Server Response Time
• Server Side Resource utilisation (CPU,
Memory, Heap Utilisation, Session count,
Load balancer stats etc.)
• Network impact on performance
Mobile Device Performance Stats
• Response Time in respect of user experience
(including Page Rendering Time)
• Mobile Devise Resource utilisation
(CPU, Memory, Heap Utilisation, Battery
consumption stats etc.)
Network Dependency
Unlike PCs with stable broadband network
connectivity, Mobile phone relies on
Mobile Networks, which will vary based on
Geography, Varying Signal Strength, Network
Bandwidth (2G/3G/4G LTE), Packet Loss etc.
having deinite impact on performance. So,
for mobile performance testing it is important
to simulate similar network conditions and
measure the performance impact of the same.
Script recording challenges
For typical web based application scripting,
Load Runner set up a proxy after browser and
captures the HTTP Request – Response traic.
Now the same approach will not work directly
for mobile performance testing because...
1) LoadRunner being a PC based tool
(Windows/Linux), cannot be installed on
Mobile Devices running Android or iOS.
2) For Native Applications, setting up proxy
at Browser level will not work, as they do
not depend on browsers for network HTTP
Request/Response
3
What is new in
Mobile Performance Testing?
Mobile Performance Testing
Web HTTP/HTML
protocol:
Used for: Server side Performance Testing &
Metrics collection
How to record script?
To capture Mobile HTTP/HTML traic
using this protocol, we have to change the
recording approach a bit compared to our
approach for recording normal PC – Browser
based Web Applications.
To record Mobile HTTP traic, instead of
selecting Browser option (default selection)
in Start Recording window, we have to select
either of below 2 options…
1) Remote Application
via LoadRunner Proxy
Pre Requisite: LoadRunner – VuGen Machine
& Mobile device should be connected to
same WiFi enabled LAN network having
access to internet.
Approach:
1) In VuGen select “Remote Application via
LoadRunner Proxy” option in Start Recording
window instead of “Browser”
2) After selecting irst option, VuGen will ask
for “LoadRunner proxy listens on port:” – Port
number can be set to any 4 digit number or
also can be used the default one – 8080
3) Check “Display recording toolbar on client
machine”
4) Click on “Start Recording” – VuGen will start
listening to any traic traversing through it
proxy set on VuGen PC – deined port set on
point 2.
5) Now, as VuGen is listening to the requests,
we have to set the Mobile Device going
through VuGen machine.
6) In mobile device, we must connect to the
same WiFi enabled LAN network to which our
VuGen machine is also connected.
HPE LoadRunner protocol support
for Mobile performance testing
7) While connecting to the WiFi network in
mobile device, we must set Custom Proxy
with IP of the VuGen Machine and Port
Number as LoadRunner proxy Port number
8) Now, whatever traic lows to and fro this
Mobile device will be captured at VuGen
and script will be created once we stop the
recording.
9) While Recording we can use Recording
toolbar to put comment or transaction or
perform other recording time options
10) Once Script is generated, we can perform
Correlation, Parameterisation, Error Handling,
Transaction Naming etc. like any other Web
Script
11) We can Use LoadRunner Network
virtualisation to simulate Dierent types of
Mobile Networks, during scenario creation
using controller or Performance center.
Advantages:
1) Can be used without any help from any
other tool irrespective for Mobile OS – iOS/
Android/Windows
2) Advantages of Web HTTP/HTML protocol
can be used
3) Do not require the Mobile device to replay
the script or during execution
4) Network impact on performance can be
measured using HPE Network Virtualisation
Limitations:
1) Cannot be used if we require to create script
for a mobile device which is not connected to
the same Network
2) Mobile device performance metrics will not
be measured
2) Captured Traic File Analysis
Pre Requisite:
1) Packet Capture applications enabled
with saving captured traic in PCAP format
should be installed on Mobile Device (Ex. –
tPacketCapture application can be used for
Android mobile).
2) Permission/approvals should be taken from
concerned stakeholders to install WireShark in
LoadRunner – VuGen Machine.
3) WireShark should be installed in VuGen
machine.
Approach:
1) Using Packet Capture applications installed
on mobile device, capture the Network traic
related to the Application under test in PCAP
format & transfer the ile to VuGen machine
2) In VuGen Select “Captured Traic File
Analysis” option in Start Recording window
instead of “Browser”
3) Browse the recorded PCAP ile
4) Click on “Start Recording” – VuGen will
analyse the PCAP ile using WireShark and
generate the script.
5) Once Script is generated, we can perform
Correlation, Parameterisation, Error Handling,
Transaction Naming etc. like any other Web
Script
6) We can Use LoadRunner Network
virtualisation to simulate Dierent types of
Mobile Networks, during scenario creation
using controller or Performance center.
Advantages:
1) Mobile device need not to be on the same
network as VuGen machine
2) Advantages of Web HTTP/HTML protocol
can be used
3) Do not require the Mobile device to replay
the script or during execution
4) Network impact on performance can be
measured using HPE Network Virtualisation.
4
Mobile Performance Testing
Limitations:
1) Have dependency on other packet
capturing tool, having capability to same
captured traic in PCAP format. Selection of
tool is again dependent on mobile OS.
2) Have dependency on WireShark tool
to enable VuGen able to analyse PCAP ile
format
3) WireShark installation requires exceptional
approval from concerned stakeholders &
corporate network security team
4) Mobile device performance metrics will not
be measured.
TruClient – Mobile Web
protocol
Used for: End-to-End Performance Testing &
Metrics collection for Mobile Browser based
applications.
Approach:
1) Being a TruClient protocol, records /
simulates GUI level User actions / events, and
not the protocol level communications
2) To create a TruClient – Mobile Web script,
we will need to select a Mobile Device
simulator from the List of devices provided in
VuGen Mobile Setting window, which appears
after clicking on Develop Script option
3) Based on the Mobile Device selection,
Mobile Browser – OS – OS version – Screen
Resolution gets selected, which will be used
to create the script
4) Once Mobile Settings are selected, script
can be developed following same TruClient
script preparation steps and all TruClient
customisations can be applied to the script
5) We can Use LoadRunner Network
virtualisation to simulate Dierent types of
Mobile Networks, during scenario creation
using controller or Performance center.
Advantages:
1) No Mobile device is required for recording/
replay of the script
2) Advantages of HPE TruClient technology
can be used
3) Network impact on performance can be
measured using HPE Network Virtualisation.
Limitations:
1) Can be used for Mobile based application
testing only, Mobile Apps cannot be scripted.
2) Mobile device performance metrics will not
be measured, as Mobile device is not used to
execute the script
TruClient – Native
Mobile Protocol
Used for: End-to-End Performance Testing
& Metrics collection for Mobile native
Applications including Mobile Performance
metrics.
Approach:
1) Being a TruClient protocol, records /
simulates GUI level User actions / events
performed at Real Mobile device, and not the
protocol level communications
2) To create a TruClient – Native Mobile script,
we will need to provide below details to
VuGen
• Mobile Center Server address
• Mobile Center User Login credentials and
SSL details
• Mobile Device ID attached to Mobile Center
• Application to be tested
3) Mobile Center is integral part of TruClient
– Native Mobile protocol. Mobile Center is a
HPE product working in master slave concept.
Master part is called as Server and the Slave
is called as Connector. Mobile devices are
connected to connectors and Connectors are
connected to Mobile Center Server. This can
be integrated to LoadRunner using TruClient
– Native Mobile protocol script. Mobile Center
can…
• Connect multiple Mobile devices, supports
iOS, Android, Windows devices
• Mirror the Mobile Device screen
• Capture User Actions for TruClient script
• Replay TruClient Script in connected device
(any device, not necessarily the same
device which was used to record the script)
• Measure Mobile Resource utilisation during
Replay/Execution
4) Once Required details are provided and
Mobile Center mirrors the Mobile Device,
script can be developed following same
TruClient script preparation steps and all
TruClient customisations can be applied to
the script
5) During replay/execution actual mobile
Network of the connected device is used,
simulating actual network conditions.
Advantages:
1) Actual end user experience using actual
mobile device can be measured in this
protocol
2) Network impact on performance can be
measured using actual mobile device
3) Mobile Center connector component
enables simulating dierent geographies
4) Mobile Performance metrics & Server
Performance metrics both can be measured
5) Advantages of HPE TruClient technology
can be used.
Limitations:
1) To generate multiple virtual user equal
number of Mobile devices will be required,
making it more costly.
2) Mobile Center licence needs to be procured
on top of LoadRunner Licence, making it
more costly.
5
Mobile Performance Testing
Experitest SeeTest Automation
an alternate to HPE Mobile Center
SeeTest Automation can be considered as an
alternative to HPE Mobile Center.
Features of SeeTest Automation
1) Connect Mobile devices, supports iOS,
Android, Windows devices
2) Mirror the Mobile Device screen
3) Capture User Actions and creates script
4) Replay Script in connected device
5) Measure Mobile Resource utilisation during
Replay/Execution
6) Can create LoadRunner CVuser script, for
the seetest automation scripts
7) Provides LoadRunner add on using which
LoadRunner CVuser scripts can be executed
on real devices connected through SeeTest
Automation
8) Supports multiple device connection
enabling concurrent connection.
Advantages over Mobile Center
1) Licencing cost much less compared
to Mobile Center
2) Can be used for Mobile Automation testing
as well, without depending on any other tool
like QTP.
Limitations
1) LoadRunner licence for CVuser protocol will
be required to integrate with LoadRunner
2) Basic licence does not allow concurrent
connection to multiple Mobile devices, and
requires additional licence for each of the
mobile device to be connected.
6
Mobile Performance Testing
Suggested Scenario preparation approach
for Mobile Performance testing
1) Capture concurrent usage metrics of any
application for all access channels like PC –
Web Applications, Mobile Browser based web
applications, Mobile native Applications
2) Capture in scope business lows for each of
the access channels, and also capture Work
Load Model for each of the identiied Access
channel – business low combination
3) Capture Network simulation requirement
in terms of Geography, Packet loss, Signal
strength, network bandwidth, network
connection type etc.
4) Create Web HTTP/HTML script to simulate
“Synthetic” Load on server
5) Use HPE Network Virtualization to simulate
real life network condition
6) Use Mobile Center (TruClient – Native
Mobile protocol) or SeeTest Automation
(CVUser protocol) to simulate real Mobile
device Load and capture Mobile device
performance metrics – restrict the number
users to minimum simulating all required type
of devices only
7) Use TruClient - Web / TruClient – Mobile
Web protocol only when web Http/Html
protocol cannot be used
8) Run Test simulating maximum load on
server using Synthetic load & only required
load using Real Mobile device, running in
parallel
9) Correlate the Synthetic Load & Real Mobile
load results to identify probable performance
bottleneck.
7
Mobile Performance Testing
CT_J965_180507_RY_WP_MOBILEPERFOR
For more information: atos.net/contact-us
All trademarks are the property of their respective owners. Atos, the Atos logo, Atos Codex, Atos Consulting, Atos Worldgrid, Bull, Canopy,
equensWorldline, Unify, Worldline and Zero Email are registered trademarks of the Atos group. Atos reserves the right to modify this document
at any time without notice. Some oerings or parts of oerings described in this document may not be available locally. Please contact your
local Atos oice for information regarding the oerings available in your country. This document does not represent a contractual commitment.
May 2018. © 2018 Atos
About AtosAbout Atos
Atos is a global leader in digital transformation
with approximately 100,000 employees in
73 countries and annual revenue of around
€ 13 billion. European number one in Big
Data, Cybersecurity, High Performance
Computing and Digital Workplace, the Group
provides Cloud services, Infrastructure &
Data Management, Business & Platform
solutions, as well as transactional services
through Worldline, the European leader
in the payment industry. With its cutting-
edge technologies, digital expertise and
industry knowledge, Atos supports the
digital transformation of its clients across
various business sectors: Defense, Financial
Services, Health, Manufacturing, Media,
Energy & Utilities, Public sector, Retail,
Telecommunications and Transportation.
The Group is the Worldwide Information
Technology Partner for the Olympic &
Paralympic Games and operates under the
brands Atos, Atos Consulting, Atos Worldgrid,
Bull, Canopy, Unify and Worldline. Atos SE
(Societas Europaea) is listed on the CAC40
Paris stock index.
Find out more about us
atos.net
atos.net/blog
Let’s start a discussion together
