5 Things You Need To Know About Before Jumping into Mobile A/B Testing

As the Android vs. iOS saga continues, most major apps establish their presence on both platforms. Many opt for iOS first, since there are only a few device models (as opposed to the ever-expanding portfolio of Android devices and manufacturers).

Nevertheless, some have been swayed by the advantages of Android. Our friends at Stack Overflow cited iteration speed and first-party support for alpha and beta testing as the main reasons for choosing Android as the first platform for which they developed their native mobile presence.

Even if the platform differences are not immediately apparent to you, taking them into account is key to a successful cross-platform app. It may be easy to assume that what works for one platform will translate into success on the other, but this type of logic will get you in trouble down the road.

In case you’re new to A/B testing, here is the concept in a nutshell:

AB Testing Mobile Apps

Source: A/B Testing for Mobile Apps – Coca-Cola Mobile Innovation Workshop

Mobile A/B testing has a big part to play in identifying the differences needed to be successful on both iOS and Android. Here are some of the things we’ve seen that have the greatest impact when testing on the platforms:

1. Demographics

According to this comScore report, iOS users tend to be younger and wealthier: 19% of iPhone owners are between the ages of 18-24 years old (compared with just 16% of Android owners), and 41% of iOS users are in the $100,000+ income bracket (compared with just 24% of Android users).

Android has been shown to be popular with professional and business users. Hacker types also have been drawn to Android because of the possibilities that an open platform offers.

Takeaway: Think critically about the differences in general audiences across the two platforms, and how you can play to those differences through UI, UX, price points, and features.

For example, iPhone users may respond better to a promotion or feature that makes a cultural reference to something that young, affluent users will recognize. Android users, on the other hand, may be motivated more by a feature that lets them customize their experience.

2. Monetization

Despite Android’s impressive growth in market share of devices sold and active users, iOS has been shown to generate more money for developers year after year. iPhone users are more likely to purchase something from their mobile device, and 23% have purchased something on mobile previously (as opposed to 17% on Android).

Flurry took the opportunity at GDC this week to open their datasets on Android games, showing that not only is the Android population skewed toward young males, but some mix of in-app purchases and ad-based revenue is optimal for many mobile games.

iOS leads revenue charts

Source: Vision Mobile Developer Economics

Takeaway: A/B test different monetization strategies on iOS and Android in order to capture the most overall value from both platforms. iOS users generally are more likely to download paid apps and make in-app purchases, whereas Android users may be monetized more easily through advertising and lead generation. You also can test different price points, and you may find that one platform’s users have a greater tolerance for higher price points.

3. Usability

Engagement and native UX perhaps are where iOS and Android differ the most. For example, Android’s “intents” allows users to share content using any installed app from any other app, which doesn’t exist on iOS.

iOS has a tendency to attract “power users” who are more likely to engage in all major content categories (social media, news, e-commerce, and games) for longer average session times.

android intents

Source: Android SDK Quick Tip: Sending Pictures the Easy Way

Takeaway: Consider testing user flows and experiences that complement the behavior typical of users on the platforms. You may want to test more streamlined and straightforward activation flows for Android, whereas you may be able to rely on iOS users to engage unilaterally for longer periods of time before you offer them an in-app purchase or ask them to share some content with friends.

Additionally, there are distinct UX conventions in iOS and Android – such as navigation, or how actions are displayed – that may work for one platform and not the other.

4. Device Type and OS Versions

A key challenge in Android development is the relatively large range of different devices that use the platform. From Samsung to HTC and Google’s own phones, Android represents a mosaic of different price points, screen sizes and resolutions, and hardware that can make it an unpredictable platform at times.

iOS runs on significantly fewer device types, but, nevertheless, can be complicated by different hardware and OS versions that may not have the capability to run some apps correctly or at all.

Takeaway: When A/B testing new features on Android or iOS, segment by device type and OS to account for how changes affect users on each combination of hardware and software. You may be able to drive more desirable user behavior by pushing only new features on certain devices while keeping the old feature set for other devices. The same may apply to OS versions.

5. Speed of Iteration

At a recent Android meetup at Google’s NYC offices, we heard from Kinsa Health’s CEO Inder Singh about the benefits of developing for Android. One of his key perceived advantages of Android: the speed at which Kinsa can act upon user data, and iterate faster, through methods like A/B testing, without having to go through lengthy app approval processes.

Although new technology in A/B testing mobile apps has cut down the iteration time for both iOS and Android, there still is a strong case to be made that Android is a more flexible platform for which to implement iterative feedback loops and respond quickly to user data.

The time saved through quicker iteration should not be confused with quicker tests, though. Stopping tests early leads to the risk of a false positive (also known as a Type I error). In other words, a statistically underpowered test could indicate an effect (like a 20% boost in conversions) is present, when, in fact, the results are skewed because of a narrow sample population.

Takeaway: If you’re not already leveraging new technologies to quickly run A/B tests and make app feature or UI decisions for your mobile app based on data (and not opinions), there’s no reason not to get started.

And, when you’re ready to commit changes to your app’s binary that have been validated through rigorous testing and analysis, it will be faster to push those changes through the Google Play Store than the Apple App Store.

About the Author: Zac Aghion is the CEO and Co-Founder of Splitforce. Data is power, and it should be easy to leverage data to make better decisions. Splitforce is A/B testing mobile apps.

  1. Interesting considerations. I’ve never really thought about the demographics and statistics of IOS and Samsung’s users. Also good considerations where to choose to develop your app! After all, Samsung’s market isn’t the same as IOS’s, so they are likely to have different priorities or wants.

    • @Scott – it’s not only a difference in demographics! Distinct devices, UX conventions, and native OS functionalities on iOS vs. Android are all reasons to segment your users when testing and optimizing your apps.

  2. Thank you for your note, it brings a lot informations here. I even have no idea that each platform will bring us to a different target audiences :(. I guess, the first thing I must do now is to delete my old plan and creat a new one. Thank you a lot!

  3. Thanx for sharing this. I had absolutely no idea we had to take so many things into consideration.

  4. Thanks for the info. it’s really helpful. It gives basic information regarding viewers behavior and other aspects.

  5. iOS market is much better, since the probability of engagement in contents & UI in iOS is much higher than the that of Android, but amount is also matter a lot. A small amount can also generate a big conversion.

  6. it’s really helpful. It gives basic information regarding viewers behavior and other aspects.

  7. Great takeaways! If you haven’t considered the demographics of iOS and Android users as part of your A/B testing, Google Analytics provides some great statistics on your traffic coming from different devices.

14 comments

Please use your real name and a corresponding social media profile when commenting. Otherwise, your comment may be deleted.

← Previous ArticleNext Article →