turbophil Posted December 5, 2020 Report Share Posted December 5, 2020 We noticed that our CI process was executing individual Caraya tests that are defined in a Test Suite, but not running the Test Suite itself. In our particular case, it was noticed because we have a Suite that runs a bunch of tests in a parallel-for loop (mostly to cut down on total execution time, but there is also a slight confidence boost in seeing that there is no negative interaction between the VIs under test running in parallel). Upon investigation, we found that the "Discover Tests.vi" is explicitly excluding Test Suites from the results. There is a comment in "Find Test and Suite Index.vi" that gives a little hint as to the rationale: Quote Define Test Suite; we DON'T want to include anything with this in it because nested test suites aren't supported. However, I still don't quite understand why we would want to exclude Test Suites. It's not necessarily a dealbreaker problem for the example of the call-tests-in-a-parallel-for-loop situation, since that's mostly just done to save time, but I could see a situation where someone might want to define tests to run in a particular order in a Test Suite, but then when automating it, they lose that explicit ordering. Is there a way to get the test discovery to respect Test Suite grouping? I suppose we could roll our own child class of TestRunner to implement this behavior, but before we go too far down that road, I'd like to make sure we're not reinventing a wheel (or risking reintroducing a problem that this exclusion is specifically avoiding). In my mind, I would imagine we could: Find all defined tests Find all define test suites For each suite found, check its linker info to find the tests it contains Remove those discrete tests from the list, as they are redundant with the test suite Similarly, if a given test suite contains other suites as dependencies, remove those, as well (also redundant) But, again, before I go too far down that path, is there already some other workaround for this problem? Or if not, does this sound like a viable solution? Thanks! Quote Link to comment Share on other sites More sharing options...
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.