TestRail: Reporting data-driven iterations

Best practices, code snippets for common functionality, examples, and guidelines.
Posts: 125
Joined: Wed Sep 19, 2018 10:05 am

TestRail: Reporting data-driven iterations

Post by HansSchl » Thu Jan 14, 2021 8:48 am

I have a question about reporting to TestRail the results of a test run that iterates over entries of a data source.
This is the structure of my test suite:

Code: Select all

+ test suite "ImportExport"
+- smart folder "Export"
  +-- smart folder "Export A"
  |  +-- test case 1
  +-- smart folder "Export B"
    +-- test case 2
(I created this structure because I am going to add an "Import" folder to the test suite, and more test cases to smart folders A and B.)

The smart folder "Export" contains a simple data source with two rows. Test case 1 contains a condition so that is executed for data row 1 only. When I run the suite manually, I am reported "3x Success, 1x Blocked" which comes close to what I would expect ("1x Ignored" would be perfect).
In TestRail however, I see "2 Passed"; the results of both runs of test case 2 are merged into one report, and there is no indication that one run of test case 1 was skipped. The latter information is important because it reminds me to fix a bug in the AUT which destroys the test data (and that is why I skip the run, rather than letting it fail).

Am I doing something wrong, or is that normal behaviour? Any ideas how to solve the issue?