Code coverage

[Mathilde]

Hi,

I'd like to know if it's possible to calculate code coverage percentage with Caraya ? 

And if no, is it in the Caraya roadmap ? 

Or do you think it is possible for us to customize Caraya in order to add this functionality ?

 

Regards,

Mathilde

 

Edited March 1, 2021 by Mathilde

[Jim Kring]

Hi @Mathilde (CC: @Francois Normandin)

Thanks for your interest in code coverage with Caraya.

There's not currently a built-in feature for doing code coverage in CaraYA. Do you have thoughts on how you would want to calculate coverage?

I've implemented some simple coverage tools in projects by using naming conventions for my tests and enforcing that each public member of a library has a test associated with it.

[Mathilde]

Hi Jim, 

Thanks for your reply!

The calculation of code coverage is imposed by our customer. Currently, we calcul it "manually" by verifying each public method is covered by at least 1 test. So if you have a tool which automatically do that, it would already be great !

By discussing with the quality expert at our customer company, they would like us to calculate a real code coverage (see attached).

 

Do you think it is possible with Caraya ?

Best regards,

Mathilde

 

[Francois Normandin]

@Mathilde

(I just saw this thread today)

Caraya is not well suited for automated code coverage calculation as it executes at the VI level, not the application level. There is no inspection performed and the framework is all geared towards individual assertions.

An application could be developed, on top of Caraya, to inspect code and report on the minimum number of assertion tests to be performed to cover all conditions, but it would probably still be opened to interpretation since different assertions can be done on the same case to test all the limit conditions.

As an example of this statement, let's consider the number of assertions needed to fully test a single "enum" value in your screenshot. Just from the image you provide, I can think of those assertions: 

• "Numeric 2" is equal to 0
• "Numeric 1" is same sign as "Numeric 2" and not equal 0.
• "Numeric 1" is opposite sign from "Numeric 2" and not equal 0.
• "Numeric 1" is equal to NaN
• "Numeric 2" is equal to NaN

Those five assertions are possibly just a subset of the range of assertions to run, but they are in no mean related to achieving a 83.3% of coverage based on the following computation: "5 tests for 6 different paths". Indeed, the same limits should be tested for all 6 paths, which means that 30 assertions are the base for achieving 100% coverage. But this is really depending on the algorithm in the black box (case structures) because there is no way to determine the number of assertions needed for full coverage unless one knows the algorithm under test, unless the assertions happen directly in each case of the production code... 

Each generated test vector would need to be tested with multiple assertion results. If Caraya could aggregate more information that would be useful to such a wrapper application, that could certainly be entertained. At the moment, the call chain, assert name and test names are pretty much all that is available.