Way to parametrize a test/test iteration as an expected fail

Ask general questions here.
sorina
Posts: 4
Joined: Tue Jan 10, 2017 1:36 pm

Way to parametrize a test/test iteration as an expected fail

Post by sorina » Thu Jan 12, 2017 6:10 pm

Is there any way I can parametrize a test to signal it as an expected fail in the report?
When you use a tree structure with test cases on different levels, it is often very difficult to recognize the data sources and if they have been restricted to only some rows. Therefore it would be very useful to me to be able to mark a test as an expected fail in the report > as I'm building a regression suite, I really need to not forget issues by commenting them off in the data table, but keep track of things that may only be fixed in upcoming sprints.
How do you guys handle these cases?

jma
Posts: 107
Joined: Fri Jul 03, 2015 9:18 am

Re: Way to parametrize a test/test iteration as an expected fail

Post by jma » Fri Jan 13, 2017 1:27 pm

Hi sorina,

Would it be an option for you to change the error behavior of the corresponding test case to 'Continue with iteration'? In this case, the test will be continued for the remaining rows of the data source although one iteration failed.
continue_with_iteration.png
In my opinion it wouldn't be a good solution to mark those test cases as a success, because actually they failed. Information about changing the error behavior can be found on the Ranorex web page.

Furthermore, you could mark specific actions as 'Continue on fail' as described in the following user guide section.
You do not have the required permissions to view the files attached to this post.

sorina
Posts: 4
Joined: Tue Jan 10, 2017 1:36 pm

Re: Way to parametrize a test/test iteration as an expected fail

Post by sorina » Fri Jan 13, 2017 2:33 pm

Thank you for your reply. I was thinking of something in the line of: pytest mark-a-test-function-as-expected-to-fail (for some reason I'm not allowed to post links as a new user
It was a useful feature that I miss with Ranorex. I already use the continue with iteration error behavior and the failed validations are also marked so as not to halt the whole execution, but I wanted something similar to what is described in the pytest documentation - no traceback in the main report, executed and separately listed in a list of "expected fails".

krstcs
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Way to parametrize a test/test iteration as an expected fail

Post by krstcs » Fri Jan 13, 2017 3:02 pm

Normally our tests are testing against failure, so we want any failures to raise errors.

In your case it sounds like you want to test for failure, so you don't want a failure to raise an error.

I would suggest that you rethink how you have the test case coded so that it actually PASSES if the failure in the SUT happens, since that is the EXPECTED OUTCOME of the test case.
Shortcuts usually aren't...