Mobile Fragmentation and the Need for Automation: An Interview with Josh Galde

Josiah Renaudin: Today I'm joined by senior product marketing manager, Josh Galde. Josh, thank you very much for speaking with us.

Josh Galde: Thanks for having us. Thanks.

Josiah Renaudin: All right. Could you tell us just a bit about your experience in the industry before we dig into the mobile topics?

Josh Galde: Absolutely. My experience has been working with ... I've actually worked with DeviceAnywhere for the last six years. Part of the acquisition back in 2011 by Keynote. I've had extensive experience working with our customers DeviceAnywhere who primarily have to plan their app releases and need to do lots of testing for their variety of apps and websites that they're releasing to support their customer experiences. We work with hundreds of companies around the world who need to provide ... Who have an interest in requirement to do testing. They come to us to help them with that.

Josiah Renaudin: One of your main areas of expertise is mobile devices, and almost at this point, it feels like there's a new company entering the market every single year. We have Amazon recently with the Fire Phone. Are there just too many styles of mobile devices on the market to reasonably and thoroughly test an application on? Is it worth the time and effort to test more fringe devices that aren't as established as, let's say, iPhone, or HTC One?

Josh Galde: Yeah. That's a great question, Josiah. The market is getting more and more fragmented. While we've seen the demise of Blackberry and others, Nokia for one. There's still a need to test on as many devices as possible because whether you're in the US market or around the world, there will never be a one device everyone is using. Because of that, there will still be tons of devices on the market. Obviously, different OSs, different software.

You have different device configurations. It just is endless. Many of our customers have come to us for advice on which devices to target as a result. Nobody can ever test on every device on the market, and you wouldn't want that or need to. There's a couple things, three items, that I really feel like I would say companies need to take into consideration when looking at which devices to test on, how many devices to test on, etc.

First I would say you need to understand what are the operating systems that your customers are using. Sometimes it may be more than the latest version of the OS i.e., Android which has, I don't know, ten different versions, and you've seen the charts and the graphs that support the fact that customers as they upgrade the OS are not always all upgrading their OS to the latest version, unlike on iOS that's 98 percent upgrade rate. With Android it's still very fragmented. You have to be mindful of do I need to build multiple app versions for different OS versions because on Android, at least, consumers are not upgrading for one reason or another. Therefore, do I need to consider supporting back versions?

It also takes a value of looking at market data, looking at our own internal data to decide, "Okay, which OS version are we going to support? Which are the ones that are the most popular?" Really help narrow your focus a little bit too, so you're not trying to reach everyone, and be able to reach everyone in the industry. I don't think it is achievable, nor is it realistic.

You may have a small set of subscribers using a super old version of Android. Are you necessarily going to see any value in expending revenue, and expending costs to make sure that person is supported? Once you know that, you can determine which devices make sense to test on.

Is it the newest devices, or maybe it's a variety of those devices? Obviously, on Android that makes sense, because you have multiple devices built by multiple operators who support Android. On iOS it's probably a little more focused because obviously, it's the one operator, but you still have a need to support different form factors. As you know, there's the touch, obviously the 5, the 5s, and the 4s, which are all fully supported by Apple. Obviously, with new devices coming out by Apple hopefully this year, we expect to see that increase and expand.

Third you need to know if you're building an app for a phone, or a tablet, or just one of the other. I think we see a lot of that need where people think they can just build one app that's going to work on both devices. They realize there's more functionality on a tablet, so therefore want to end up building a second app. Something to consider when supporting multiple devices. I'll lastly end with this, and just say that I know in a recent survey of ours over 66 percent of respondents tested on less than ten device models before releasing their app.

All that really tells you is that, while that is the panacea to say, "Look. We're going to test on hundreds, and hundreds of devices," you can by looking at the market, by looking at your own internal needs and what you need to support based on features and functionality, can and should be able to, even with our help. We've helped a lot of customers say, "Look. These are the devices you should test on. You're going to get this market reach," and we're able to guide our customers. It is totally doable with less than ten devices. At least for automation. For functional testing, that is definitely makes sense to do it on more then ... Even upwards of a hundred if you need to.

Josiah Renaudin: When we're talking about mobile, we're not just talking about phones so what new challenges arisen with the advent of tablets? Is testing for an iPhone similar to testing for an iPad, or is there a more unique progression with that?

Josh Galde: Yeah. That's a great question. What we've seen, and a lot of our customers do a lot of tablet testing. What we see is that the processes are the same. The process of the apps, and the process of the testing of the app is the same, but the applications are different. As I mentioned before, tablets typically have more processing power, more memory, and larger screen size that allows for more features, more real estate, etc. Therefore the flow, or the test plan could vastly differ from a phone.

For example, if you're DirectTV, and you have an app that's on an iPhone let's say, that can control channels on your TV. That's going to be very limited compared to your tablet where you'll actually will be able to watch the TV, and watch what's happening on your TV on your tablet, and gets to interact with it through different features within the application.

The testing is still the same, but you actually are going to have a completely different test case when using tablets versus phones.

Josiah Renaudin: Would you say on average that we rely too much on automation, or too little in the mobile world?

Josh Galde: Yeah. Our view is that it's definitely too little. Specifically when it comes to mobile functional testing. While not all functional testing can be automated, much of it can be. What we've heard from existing customers is that they have been able to automate about 80 to 85 percent of their mobile functional testing.

What this does is, you're simply taking a simple manual test that someone was doing in your testing team and being able to replicate that and run that repeatedly, especially as you get more iterative, and you're running more agile cycles and deploying your updates to your app more frequently. That's going to be necessary if not required.

In a recent survey that we did, we found that only 14 percent of respondents have automated more than 50 percent of their testing. Essentially, more than 50 percent of their testing was automated, but we basically have a long way to go. This is a good thing. I would say, when it comes to automation there are many things to consider. Number one, would be the number and types of test cases. You also have the frequency of testing required. How often are you going to do your testing, which obviously increased in the last couple years?

The methodology needed to conduct proper testing, and the number of different devices you need to test on. All of these factors can have an influence into how much automation you need to do. Ironically, what we found from customers who have gone this route is while they have been able to speed up the amount of testing they're doing, decreasing the amount of time it takes to do that testing, they've actually been taking that cost savings and re-investing it into the application.

In a sense, it actually allows you to do more with the same amount of money. We're not seeing this major, one's going to replace the other, as much as now they get to do more testing that maybe they couldn't do before.

As a result, that has a direct impact into the functionality of the app, hopefully the success of the app, and the revenue on the business.

Josiah Renaudin: Do you think some of this lack of automation is due to just an ignorance of the benefits of automation? Do you think there's a reluctance to let automation do it and not someone in the field actually start testing something?

Josh Galde: That's a great question, too. My personal view is ... I think there's another question there too, and that is, "Where does automation live?" One of the things we're really finding is that, there's a real difference between automating if you're on the developer side. You're developing the application, you're developing different versions of the application. You need to run more iterative cycles, and you need to be able to perform automated testing. If you're in QA, it needs to be more of a robust testing they're able to do. Scripting has become extremely important for QA.

I think that could have an impact because there can be a learning curve to getting up to speed and getting your team up to speed. Automating on any platform is going to be unique, whether it's HP, or IBM, or others. You're still going to have a learning curve to understanding that automation. Building the scripts is also ... At least the first time, can be, like I said, a learning curve to be able to say, "Okay. How do I build this script?" Then I'm going to build all these scripts for, let's say, 300 test cases.

That's going to definitely take some time and I think that is the hump that people see. Once they get over it, they see the benefit of being able to re-run those tests. Suddenly they have more free time to allocate for different types of testing. It's the speed curve, I think, the beginning, but a lot of companies have realized that once they get past that, they're able to see the benefits of automating, particularly when they're a QA organization.

Suddenly, they have more available time. Automation to a QA specialist or QA manager, time is gold to them. Being able to free them up and give them more time to do things ... To do more testing, is definitely worth going down this route for them.

Josiah Renaudin: Every application is different, but how can you tell when enough testing has been done and it's actually time to publish a product? I think we've seen multiple times that just one flop can really crush a developer’s credibility. How careful should companies be when it comes time to actually picking a launch date?

Josh Galde: My view at least is that companies should be extremely careful with each release. I've seen a lot of releases come out, and either it's a big release, or even a minor patch can have a drastic impact on the flow of a customer's experience. I do think it's very important. Constantly be testing.

I think a lot of times they'll test it on the first version, even if it's final beta before it goes live, and then they don't perform testing or on-going testing once it's released. There's a lot of advantages to knowing what is it like in the field? How are people interacting with our application on these devices? There's a huge benefit to being able to perform on-going testing, even post release of the app.

As far as pre-release, you can only test it until you have an assurance that the app is working on the broadest range of devices. At least that's the goal. Many do fall short, and risk it by releasing the app anyway. People end up taking the risk. "I'm just going to go ahead and release my app, and see how it does, and then be able to make changes on the way." To be honest, there's also external factors. Carrier signal, or network issues where the carrier has a problem, or the app store, or whatever.

There's a lot of factors that the app developer can't control. A majority of it is controllable. We definitely here that from our customers. The testing services, and the ability to perform a lot of testing quickly is extremely valuable to them, and helps them. Gives them the assurance they need before they can push their app with marketing.

Josiah Renaudin: We've been talking about mobile during most of this interview, but there is a wide range of topics that you speak and write about. What do you consider your favorite subject matter that you dig into, and why? If it is mobile, that's totally fine. What's your favorite thing to discuss?

Josh Galde: Yeah. It definitely is mobile, but this idea of what I'm calling, "Quality in context." When we talk about quality in context we're saying, whether it's a desktop application, or a mobile application … Any customer experience that people interact with, whether it's a business, or even internal applications for the workforce apps, and stuff like that.

All of that is about you want to make sure as a company that you have a highest level of quality in whatever you deliver. A lot of companies out there can help with that, but it's this idea in context where I see the most value. We're able to provide different levels of contextual data, or analytics around a mobile application. I think there's huge advantages. I think that is where we're going.

I see mobile as being the future. I think because it's so fragmented. Because there's so many devices. You're not dealing with one desktop machine that's a standardized platform, that it's going to require extreme diverse level, and different approach, than a desktop application would.

At least our future is we're looking at to provide context around the quality of their experience, and how can we give companies insight into that information so they can better improve their apps. They can make more money off the application. It hopefully will move beyond it being bug free, or up or down.

Josiah Renaudin: Fantastic. We're going to move into our last question now. I really do appreciate your time and everything you've said so far, Josh. Like I said, we've talked about mobile almost the entire time, but is there another booming part of the industry that you think people need to pay more attention to? If so, what has you most excited about it?

Josh Galde: Sure. I think there's really three areas. I would say because I have strong belief that mobile is the future. Mobile is broad. We typically think in the history that mobile is typically a mobile phone. Now we've crossed over into tablets, and we have new devices out there that are kind of in between a tablet and mobile phone. I think for us, at least, where I think that it's really going to explode is in three areas. The first I would say in the wearables market.

Whether it's Google Glass, or the iWatch that's coming out, or even the Samsung watches that are out today, wearables will become another place to build and support applications. I do see, while it's not there yet, I think this is the future, and it's more of an extension of their mobile device, or of their tablet. I think this is going to be a new testing ground that companies are going to need, and be required to make sure that they assure that their applications are getting to market, and need to be tested properly. That would be number one.

Number two, I would say is iOS 8. Every time Apple does a major iOS version update, it causes headaches in the industry, particularly because of new functionality, new features. A lot of the, especially iOS 7, was that they changed for developers, and for QA testers that they had to deal with quickly. I see iOS 8 … it may not be at that level. Definitely has some major changes coming with it, and as you know or have heard, new devices and form factor. Certainly, iOS 8. Third I would say this idea of agile testing/continuous integration.

We're seeing a lot of movement around being able to test with an existing test frameworks like Selenium, or IntelliJ or others, that where they're to use existing testing framework and be able to test on real mobile devices, a lot of companies see that emulators is not enough.

Testing on real devices is incredibly valuable to give them a real user experience. Being able to do that from inside an existing framework is a huge benefit and a cost savings as well because they don't have to go re-learn another tool. We're working our hardest to integrate with as many platforms out there on the market as possible of course, so we can support as rapid an iteration of mobile app development.

Josiah Renaudin: I might have been lying about that being my last question, because you mentioned wearables. I did want your opinion on one thing. What do you think about Oculus Rift? We know it was bought by Facebook recently, and it's not exactly a casual wearable. You're not going to wear that around the mall, but what's your impression of virtual reality, and head tracking, and putting that device on your head.

Josh Galde: Yeah. No. That's a great question. Our view ... At least from a technology perspective, Keynote is agnostic. We pride ourselves on not necessarily picking one technology over the other. Our view is, if there is a place to put an app, you should be able to test on that, and be able to do that remotely. In the past, and I would say the last couple of years, we have integrated other devices. Not just mobile devices. We've integrated your set top boxes, we've integrated nav systems, we've integrated even, what do you call it? The scanners that support retail stores while they do their stocking and stuff like that.

Wherever apps can be supported, whether internal, or external, and wherever they go is where the testing will follow. We want to be ahead of that curve. Whether if that is on virtual reality device, or on your watch, or on your tablet, or on your computer, we want to make sure that our customers are going to be able to test on a variety of devices.

I do see a market. I think it's going to need to grow, and we'll see where it goes. I could see us adding other devices as we continue to look out in future what's coming and where do we see ... What's going to take off, and therefore where's the need going to be.

Josiah Renaudin: All right, great. Once again, I really do appreciate your time, Josh. I look forward to actually meeting you in person and hearing more of what you have to say at future conferences.

Josh Galde: Sounds good. Thanks.

Josh Galde, senior product marketing manager and mobile evangelist, brings more than thirteen years of experience in the wireless industry to Keynote, with a background working for companies ranging in size from startups to multinational corporations. He spent three-plus years managing marketing functions of wireless infrastructure for Alcatel Lucent as well as corporate marketing function for WebOS (formerly Palm) with 3Com Corporation. He has experience managing all aspects of marketing from brand awareness to web development to lead generation and social media. Josh holds a BA in Professional Studies and a minor in Business from Bethany University.