We noticed that our CI process was executing individual Caraya tests that are defined in a Test Suite, but not running the Test Suite itself. In our particular case, it was noticed because we have a Suite that runs a bunch of tests in a parallel-for loop (mostly to cut down on total execution time, but there is also a slight confidence boost in seeing that there is no negative interaction between the VIs under test running in parallel). Upon investigation, we found that the "Discover Tests.vi" is explicitly excluding Test Suites from the results. There is a comment in "Find Test and Suite Index.vi" that gives a little hint as to the rationale:
However, I still don't quite understand why we would want to exclude Test Suites.
It's not necessarily a dealbreaker problem for the example of the call-tests-in-a-parallel-for-loop situation, since that's mostly just done to save time, but I could see a situation where someone might want to define tests to run in a particular order in a Test Suite, but then when automating it, they lose that explicit ordering.
Is there a way to get the test discovery to respect Test Suite grouping? I suppose we could roll our own child class of TestRunner to implement this behavior, but before we go too far down that road, I'd like to make sure we're not reinventing a wheel (or risking reintroducing a problem that this exclusion is specifically avoiding).
In my mind, I would imagine we could:
Find all defined tests
Find all define test suites
For each suite found, check its linker info to find the tests it contains
Remove those discrete tests from the list, as they are redundant with the test suite
Similarly, if a given test suite contains other suites as dependencies, remove those, as well (also redundant)
But, again, before I go too far down that path, is there already some other workaround for this problem? Or if not, does this sound like a viable solution?
Thanks!