Optimizing Mobile Test Automation using Experitest SeeTest

In 2013 Blackstone evaluated a number of commercially available mobile test automation tools in our lab, and walked away from that exercise identifying Experitest’s (www.experitest.com) SeeTest product as one of the most robust tools on the market in terms of features, flexibility to accommodate a wide variety of mobile app architectures, predictability in terms of producing the most consistent results, and overall cost to value. We felt that SeeTest’s integration to multiple testing suites (HP vs open source, private cloud vs local, customizable reporting) further differentiated it from the competition. Thus we view SeeTest as a best-of-breed tool in the automated test development and execution realm. More below on what this tool can do to help your financial institution automate mobile app testing with a high degree of efficiency and reliability.

SeeTest Capabilities

Experitest’s SeeTest software product provides a robust Test Automation framework that we implemented for testing mobile apps and mobile websites. We’ve leveraged SeeTest in all stages of automated testing; authoring test cases, executing tests, recording test results, and generating reports. So far, we have successfully integrated SeeTest test execution with JUnit, HP QTP/UFT, Atlassian JIRA, HP ALM and Testopia. We took test automation one step further and automated defect reporting from the Experitest automated testing routines. When the automated test fails, we have built adapters to automatically create new issues with relevant supporting documentation (e.g., screenshot attachments, step where failed, etc.) in defect management systems such as HP ALM and JIRA. We find SeeTest to provide fairly robust capability across each step of the entire automated test development process:

Test Authoring: Author test scripts using the SeeTest Automation program

      • Manage your automated devices in a private cloud
      • Modify the execution steps and validations
      • Use either platform native UI, HTML, or OCR to identify elements in the app/website
      • Save the elements to a repository for easy reference

Test Execution: Execute tests using the SeeTest client libraries

      • These libraries are the “driver” between the computer and controlling mobile devices
      • SeeTest provides you with client libraries compatible with C#, VB, QTP, and JAVA

Test Reporting: Generate rich test summary and detailed reports

      • SeeTest’s out-of-box reporting automatically creates reports with all steps and associated screenshots
      • Test results can be customized to not report certain steps, change verbiage, only include screenshots on error, etc

Issue Logging: Automatically create issues with relevant details including screenshots in HP ALM, Atlassian JIRA, Testopia or other defect tracking systems

      • Because of SeeTest’s robust and consistent reporting, Blackstone created tools to automatically parse results
      • After parsing out the most pertinent information, our tools automatically create a defect with no user interaction
      • These tools are capable of integrating with ANY defect management system

Optimizing Mobile Test Automation Using Experitest SeeTest - Image 1

Test Authoring Tool Interface

Our Recommended Techniques to Get the Most out of SeeTest

Steps are added as you define the test case as a script, adding events to mimic real world use. We recommend using ids instead of text tags to identify objects. Often times, text will be duplicated on a screen and can cause test case ambiguity.

Our two recommended approaches are using SeeTest with Java JUnit tests or HP’s QTP/UFT test suite. The reasons to choose one over the other are licensing costs, team members existing skillsets, size of technology stack, or preference for scheduling test runs.

For HP’s QTP/UFT tests, test scripts are exported as VBscript. One advantage of QTP/UFT is the built in data tables for simple test case maintenance. Often times, you will want to change your test user’s credentials or test problematic inputs. QTP/UFT works best when integrated with HP ALM to schedule and report test results. However, there are licensing costs to consider related to this approach. Also, you need one computer or virtual machine for every test you want to run in parallel.

For JUnit tests, test scripts are exported to Java code. Java has many open source libraries that we use to enhance test cases and to allow for easier test maintenance. We use Maven to run multiple JUnit tests, and SureFire to generate the reports. This solution requires no licensing of additional software other than SeeTest. You can run multiple JUnit tests in parallel from one computer.

In terms of skillsets, a QA resource with some level of technical background/savvy is generally more suited for creating the foundational re-usable test objects, and to address the more robust Junit and QTP/UFT tests. This isn’t a limitation of SeeTest, rather we’ve found this to be generally true of any test automation tool we’ve worked with, whether focused on mobile, web, or other IT systems. Once the foundation is setup, a more functional QA analyst can generally adequately manage the ongoing test authoring, execution, and monitoring of the test automation solution.