Running multiple test suites with run configurations when using Jenkins and command line

Ask general questions here.
JimLin
Posts: 70
Joined: Mon Jun 02, 2014 4:23 pm

Running multiple test suites with run configurations when using Jenkins and command line

Post by JimLin » Wed Apr 20, 2022 6:30 pm

Hi, I have found a number of questions/answers on this forum and elsewhere regarding having to use command line to run multiple test suites when using Jenkins, because the Ranorex Jenkins plugin doesn't support multiple test suites.

The problem I am having is as follows: My Ranorex/Jenkins/Remote VM setup is working fine and successfully runs each test suite individually, but I can't workout/find the structure of the command line to get the 3 test suites to run one after another with a run configuration for each. Although it doesn't look like it, the text shown in the screen grab is one long line, not 4 individual lines.
TestSuites.png
What is happening, is that the test run runs, but it will only run the last test suite in the line (in the screen grab that is the Execute test suite), rather than running the 3 sequentially.

Any ideas or suggestions on how I should do this will be greatly accepted.

On a slightly different note, does anyone from the Ranorex support team know when or whether the Jenkins Ranorex plugin is going to be updated to let it work with multiple spreadsheets? In this thread about Jenkins https://www.ranorex.info/need-help-rano ... tml#p50021, Marcus from the Ranorex Support team mentions that he is working to update it, but that comment is from August 2018 and the plugin still hasn't been updated. In fact, looking in GIT, the last update to the plugin is 2019.

Cheers JimLin
You do not have the required permissions to view the files attached to this post.

csaszi89
Posts: 41
Joined: Mon Jan 17, 2022 12:10 pm

Re: Running multiple test suites with run configurations when using Jenkins and command line

Post by csaszi89 » Wed Apr 20, 2022 8:58 pm

Hi JimLin,

as I see, you try to run your Tests.exe with multiple testsuite parameters.
Based on Ranorex documentation: https://www.ranorex.com/help/latest/ran ... execution/ this is currently not possible.
You can specify only one test suite to execute with this argument. If you specify more than one, only the last one will be executed.
What if you call your tests like this:
Tests.exe /testsuite...
Tests.exe /testsuite...
Tests.exe /testsuite...

This way the test suites are called sequentially one after another in different processes.

Greetings,
Gyuri

JimLin
Posts: 70
Joined: Mon Jun 02, 2014 4:23 pm

Re: Running multiple test suites with run configurations when using Jenkins and command line

Post by JimLin » Thu Apr 21, 2022 8:29 am

Hi csaszi89,

Thank you for your reply. It does seem to fit with with the behaviour I have witnessed and it seems I had misunderstood what was possible.

This does seem to be a bit of a limitation to me. @Ranorex support, is this expected behaviour and is the Jenkins Ranorex plugin going to be updated to make this easier to setup?

Regards JimLin

JimLin
Posts: 70
Joined: Mon Jun 02, 2014 4:23 pm

Re: Running multiple test suites with run configurations when using Jenkins and command line

Post by JimLin » Thu Apr 21, 2022 10:51 am

csaszi89 wrote:
Wed Apr 20, 2022 8:58 pm
What if you call your tests like this:
Tests.exe /testsuite...
Tests.exe /testsuite...
Tests.exe /testsuite...

This way the test suites are called sequentially one after another in different processes.

Greetings,
Gyuri
Gyuri, I have tried your suggestion and it does solve the problem and is a suitable way of doing it as far as I can work out. Thanks for your assistance.

Cheers JimLin

User avatar
doke
Posts: 112
Joined: Fri Mar 29, 2019 2:33 pm

Re: Running multiple test suites with run configurations when using Jenkins and command line

Post by doke » Mon Apr 25, 2022 10:22 am

besides the rx job (multibranchpipelinejob) in jenkins I have another job like this,
so i can run all predefined suites + runconfigs
please note I dont use the plugin, the rx job has to accept the parameters ( i my case this job builds rx & runs buildoutput with given parameters)

pipeline script:
def dobuild(String jobname,String testenv,String testsuitename,String runconfigname) { 
            try {
                ThisjobX=build( job: jobname,propagate: false,wait:true,
                parameters: [[$class: 'StringParameterValue', name: 'Testsuite', value: testsuitename],
                            [$class: 'StringParameterValue', name: 'Runconfig', value: runconfigname],  te rx branch
                            [$class: 'StringParameterValue', name: 'Testenv', value: testenv]]) ;   
                //println 'jobdone...';
    	        if (ThisjobX.result=='SUCCESS' ) {
    	        } 
    	        else {  
    	            unstable(ThisjobX.result);
    	        }
            }
            catch (Exception err) {
	                throw new Exception(err); 
            }
}

import java.net.URLEncoder

String downstreamjobname = "rx/" +  URLEncoder.encode(yourbranchname);

pipeline {
    agent none
    stages{
        stage('Parallel run of tests') {
            steps {           
                parallel ( 
                    T1      : { dobuild(downstreamjobname,testenv,'suite1.rxtst',runconfig1'); },																		
                    T2      : { dobuild(downstreamjobname,testenv,'suite1.rxtst',runconfig2'); },																		 
                    T3      : { dobuild(downstreamjobname,testenv,'suite2.rxtst',runconfig1'); },																		
                    T4      : { dobuild(downstreamjobname,testenv,'suite2.rxtst',runconfig2'); },																		 
                    T5      : { dobuild(downstreamjobname,testenv,'suite3.rxtst',runconfigx'); }