Scheduled test run in Jenkins doesn't work properly

Ranorex Studio, Spy, Recorder, and Driver.
mafi
Posts: 21
Joined: Thu Mar 21, 2019 2:43 pm

Scheduled test run in Jenkins doesn't work properly

Post by mafi » Thu May 09, 2019 10:24 am

Hi Everyone,
we set up a CI environment with Jenkins: Linux server where Jenkins runs and a slave Windows VM with Ranorex Studio 9.0 installed where test should run.
Everything works properly if each job is launched manually (Build Now option), but if the same TS is launched by the schedule of the Job, it starts properly, but TC of TS fail, everytime for different specific reasons, but always because some elements are not found at all.
That sounds strange to me, because the build is correct and all the TS works properly if launched manually.
I tried to have a look on the web, but the only similar problem I found was related to GUI on RDP session and I think that is not my case.
Any suggestions?

User avatar
odklizec
Ranorex Guru
Ranorex Guru
Posts: 7470
Joined: Mon Aug 13, 2012 9:54 am
Location: Zilina, Slovakia

Re: Scheduled test run in Jenkins doesn't work properly

Post by odklizec » Thu May 09, 2019 10:40 am

Hi,

At first, please make sure you are using latest 9.0.1. 9.0.0 was somewhat buggy ;)

At next, if the test fails on "not found" element, could you please post an example xpath (stored in repo) and also Ranorex Snapshot (NOT screenshot) of the problematic element?

Additionally, does it happen always at the same place or it fails randomly at multiple places in test? If test fails with specific element, could you please add a "save snapshot" action, right before the failing place and save entire page, containing the failing element? This could help with narrowing down the reason of failing xpath.
Pavel Kudrys
Ranorex explorer at Descartes Systems

Please add these details to your questions:
  • Ranorex Snapshot. Learn how to create one >here<
  • Ranorex xPath of problematic element(s)
  • Ranorex version
  • OS version
  • HW configuration

mafi
Posts: 21
Joined: Thu Mar 21, 2019 2:43 pm

Re: Scheduled test run in Jenkins doesn't work properly

Post by mafi » Thu May 09, 2019 10:55 am

Thank you for the reply.
Do you really think the version update can help? I will try.

As far as the problem I am having, tests don't fail always on the same item, but usually stop at the first element they are looking for.
I have several TS and all of them cannot find the first element. I tried to change the test beginning sequence, but nothing changes.
I remind you that if I run each TS from Jenkins manually (not scheduled) everything works properly (elements are found and tests completed successfully). I cannot understand why the scheduled sessions give me this problem!
Element of the repository are there and the VM can find them if Jenkins jobs are not schedule but run with "Build Now" option.

An example of Jenkins log is:
[2019/05/09 11:10:35.746][Error ][Module]: Failed to wait for item to exist.<br/>Failed to find item 'Proj_Repository.Proj.LoginSplash'. Please check that your whitelist allows Ranorex to access the process.<br/>No element found for path '/dom[@domain='xxx.xxxxx.xx:8080']/body/div/div/div' within 30s.
[2019/05/09 11:10:35.848][Failure][Test]: Test Module 'LoginSetup' completed with status 'Failed'
....
[2019/05/09 11:11:09.022][Debug ][Logger]: Console logger stopping.
Build step 'Run a Ranorex test suite' changed build result to FAILURE
[JK_FX_TS_XXXXXX] $ cmd /c call C:\Users\aut\AppData\Local\Temp\jenkins3445835144195212420.bat

C:\JenkinsDockerWorkspace\workspace\JK_FX_TS_XXXXXX>cd "C:\JenkinsDockerWorkspace\workspace\JK_FX_TS_XXXXXX\CompressedReports"

C:\JenkinsDockerWorkspace\workspace\JK_FX_TS_XXXXXX\CompressedReports>exit 0
Sending e-mails to: [email protected]
Finished: FAILURE

McTurtle
Posts: 297
Joined: Thu Feb 23, 2017 10:37 am
Location: Benedikt, Slovenia

Re: Scheduled test run in Jenkins doesn't work properly

Post by McTurtle » Wed May 15, 2019 10:30 am

Hi Mafi,

It super-likely that you are either not starting Jenkins as a process or your GUI user session is not open when running the test from Jenkins:
1. Disable service/Start as process
2. Keep user session alive after closing remote desktop connection

Which one was it? :D

Regards,
McTurtle

mafi
Posts: 21
Joined: Thu Mar 21, 2019 2:43 pm

Re: Scheduled test run in Jenkins doesn't work properly

Post by mafi » Thu May 23, 2019 8:18 am

Thank you.
Unfortunately I had already set both in the right way: I read them around in the net.
I think the problem is not Jenkins. I think the problem is Ranorex. Perhaps it is my knowledge of Rnx! :)
I noticed that main issues come from Edge: several objects, very often, are not recognized or clicked somewhere else. Sometimes, repository objects well recognized on physical machines are not well recognized in VM. And so on.
So, now, before deploy the test suites on CI system, I have to test it a lot in order to be sure everything is strong enough to be run in many different machines, but Edge is still a problem. :(

User avatar
odklizec
Ranorex Guru
Ranorex Guru
Posts: 7470
Joined: Mon Aug 13, 2012 9:54 am
Location: Zilina, Slovakia

Re: Scheduled test run in Jenkins doesn't work properly

Post by odklizec » Thu May 23, 2019 8:27 am

Hi,

At first, Edge engine is going to be replaced with new chrome-based engine, so it's most probably not the best idea to concentrate on Edge tests development right now? :D I believe there is already available a beta of new Edge?

At next, if you are experiencing problems mainly on VM (with tests developed on physical machine), then you should most probably develop/optimize them directly on VM, where the tests will be running? I'm always developing all my tests directly on VM and it helps a lot.

Unfortunately, without seeing an example of failing xpath, exact error you are getting and ideally, also Ranorex snapshot (NOT screenshot) of the problematic element (taken both on VM and physical machine), there is not much else anyone here can do or suggest.
Pavel Kudrys
Ranorex explorer at Descartes Systems

Please add these details to your questions:
  • Ranorex Snapshot. Learn how to create one >here<
  • Ranorex xPath of problematic element(s)
  • Ranorex version
  • OS version
  • HW configuration

mafi
Posts: 21
Joined: Thu Mar 21, 2019 2:43 pm

Re: Scheduled test run in Jenkins doesn't work properly

Post by mafi » Thu May 23, 2019 1:26 pm

Unfortunately it is not so easy to develop on VM directly due to logistics issue, but I will keep in mind your suggestion.
Thanks.