Automated test reliability

Ask general questions here.
Crispy
Posts: 1
Joined: Wed Aug 18, 2010 12:34 pm

Automated test reliability

Post by Crispy » Wed Aug 18, 2010 12:52 pm

Hi there,

We have created a set of tests using Ranorex Studio, however we have found that they can be unreliable to run.

The playback of recordings fail on either
a) not being able to 'find' an element or form that we can see is on screen and available
or b) one or more keyboard or mouse click events not successfully completing, meaning that an action is not completed and thus that a future validation fails (as the application is not in a known state)

Are there any guidelines we should be following in order to ensure reliable tests?

We are testing a .NET 3.5 WPF application on Windows 7 using Ranorex 2.3.3.8879

Chris

User avatar
Ciege
Posts: 1336
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Automated test reliability

Post by Ciege » Wed Aug 18, 2010 4:05 pm

First off, how much automation experience (using any toolset) do you have? Sounds like you may need a little more experience with how automation works (and doesn't work).

When an item is not found after you search for it, have you tried extending your search timeout?
Does your AUT have a lot of items on it that can cause the search to take a while to complete?
Are you putting your automation code in try/catch blocks to catch any unexpected exceptions?

All that being said, my automation code with Ranorex (and other toolsets) works very reliably except for the occasional bug (which is usually in my automation code). Automated tests need to be written defensively since they cannot think and see the way a manual tester does. There needs to be checks and validations that what you asked for indeed did happen. I have tests that run for close to 8 hours and reliably run again and again on different builds of my AUT because I have coded them in a way to be defensive and grow with my AUT over time.

Good luck.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...