Improving reliability of tests run on the iOS Simulator

Mobile Testing, Android App Testing.
tvu
Posts: 195
Joined: Tue Apr 07, 2015 10:47 pm

Improving reliability of tests run on the iOS Simulator

Post by tvu » Thu Jun 18, 2015 12:01 am

Hello Support,

I am testing our iOS App on the iOS Simulator. I have a Jenkins matrix job that runs the same test sequentially multiple times base on two axis: Device and OS.

It does the following:
1) On Jenkins Mac Slave - Launch iOS Simulator with a specific Device / OS Version
2) On Jenkins PC Slave - Launch Ranorex test via EXE where my mobileDevice parameter is set to the iOS Simulator
3) After the Ranorex test is done, the iOS Simulator process is killed on the Jenkins Mac Slave

This job repeats with a different Device / OS configuration until all the possible configuration is covered.

80% of the time the test runs fine, but sometimes the iOS Simulator takes a little bit longer to load our app. The Ranorex test started running before the app completely loads. I thought this would be fine since the Timeout to find the app (about 2 minutes) is much longer than it takes to load the app (about 10 seconds when it fails). I found that in this situation Ranorex will always fail to find the app and will eventually timeout.

To get around this, I created a TestCase with a recording that waits for 10 seconds. I would run the RanorexTest.exe /tc:Wait /pa:waitDelay=10s. After that test run is complete, I immediately followed with a call to run the Test Suite. I tried to put the Wait inside the Test Suite, but it didn't improve the situation. It appears that once the EXE can't find the device, it would never find the device on that call.

This improved reliability to about 95%. But, every once in awhile Ranorex will still fail because it won't be able to find app/item.

I've even tried to load up Spy on another computer to see if it can find the iOS Simulator device and it does. Is there anyway I can improve this? I can try to add a longer delay than 10 seconds, but it would be better if I could find the root cause of this so I wouldn't even have to use the delay.

Thanks,
Tung

User avatar
Support Team
Site Admin
Site Admin
Posts: 12145
Joined: Fri Jul 07, 2006 4:30 pm
Location: Houston, Texas, USA
Contact:

Re: Improving reliability of tests run on the iOS Simulator

Post by Support Team » Mon Jun 22, 2015 1:33 pm

Hi Tung,

It will be hard to say why the test times out even if the emulator / application is already running. Therefore, I'd like you to ask if it is possible to get in touch with me by email ([email protected]) and to provide more information about your system and your Ranorex Solution. Please also let me know which version of Ranorex and Xcode you are running.

Thank you in advance.

Regards,

Markus (S)

tvu
Posts: 195
Joined: Tue Apr 07, 2015 10:47 pm

Re: Improving reliability of tests run on the iOS Simulator

Post by tvu » Mon Jun 29, 2015 10:59 pm

Just so everyone knows...

Markus suggested that I run the following line of code in my project:
Ranorex.Core.Remoting.RemoteServiceLocator.Service.GetByDisplayName("DEVICENAME").Reconnect();

I created a code module to call that one line of code and called it as part of my setup step. Seems to fix my issue.

Thanks Markus!