Sunday, August 30, 2009

Selecting A Coverage Tool

At work, we use Clover to track coverage.

For programming at home, a Clover license is way too expensive (not as bad as those software vendors that used to not list prices and instead said "Call for quote", but close).

Also, we've had some issues with Clover:
  1. It sometimes falsely claims that code is covered when it isn't.
  2. It doesn't support branch coverage.
  3. It can't instrument assignments in conditionals.
(The Clover rep confirmed the above problems a couple of years ago. These may have been fixed by now. However, because the licenses are so expensive, we haven't upgraded, so we're stuck with the problems even if they're fixed in the current version. Also, the Clover rep didn't think the third issue would ever be fixed due to the way they instrument the code.)

For a simple example of problem #2, start with a function like this:
public boolean eval(boolean x, boolean y, boolean z)
{
return x && (y || z);
}
Clover scores 100% coverage if the entire expression evaluates at least once to true and at least once to false, so this is sufficient:
assertTrue(eval(true, true, false));
assertFalse(eval(true, false, false));
But evaluating all of the meaningfully distinct ways for the expression to evaluate requires more tests:
assertTrue(eval(true, true, false));
assertTrue(eval(true, false, true));
assertFalse(eval(true, false, false));
assertFalse(eval(false, true, false));
Problem #3 means that this code can't be covered:
while (currentLine = stream.readLine() != null)
Covering that code with Clover requires rewriting it as a do/while.

For the above reasons, I went in search of a coverage tool other than Clover for programming at home.

Googling around, EMMA and EclEmma (an Eclipse plugin for EMMA) kept showing up, so I tried them.

Unfortunately, where Clover incorrectly says that uncovered code is covered, EMMA incorrectly says that covered code is not covered:



True, it only screws up occasionally, but if you're five lines short of 100% coverage, and the five lines are spurious tool errors, it's annoying.

This was unfortunate, because the EclEmma plugin is very fast and completely non-intrusive, and would be great to use. Plus, it's currently the only coverage tool that works with Android.

After reporting these problems, I looked around for alternatives. A bunch of coverage tools were listed here, and I tried each of them out. Most didn't install, or weren't compatible with the latest Eclipse, or hadn't been maintained for a number of years, etc.

But Cobertura works really well.

Unfortunately, there's no Eclipse plugin for Cobertura, and it's slower than the other two tools, but compared to getting the wrong answer, that's not too much to give up. The dream tool would combine Cobertura's accuracy with EclEmma's speed and usability, but no such tool exists.

This table compares the three tools:
















































CloverEMMA/EclEmmaCobertura
Statement coverageyesyesyes
Branch coveragenonoyes
No quirks or bugsnonoyes
Fastyesyesno
Reportsyesyesyes
Maven integrationyesyessort of
Works with mocking toolsyesyesyes
Eclipse pluginyesyessort of

Freenoyesyes
Active communityyesyessort of



Notes:
  • Maven integration for Cobertura is provided by a separate open-source project. This project has been around for a while. There are some problems using the integration.
  • Eclipse integration for Cobertura is provided by a separate open-source project. This project is new and not very mature yet.
  • Cobertura's community support is rated "sort of" because the only forum is via an email distribution list, and responses can take a couple of days. On the other hand, the code is mature and easy to use, so there isn't much need for help (most of the emails I sent were questions about setting up the ant scripts).
  • JMockit can be configured to generate coverage reports, but currently only statement covered is supported. The author plans to add additional features (including branch coverage), at which point it should be evaluated like the other three tools.
  • For a comparison of coverage tools that arrives at the opposite conclusion to mine (partly due to other requirements), see http://javapulse.net/2008/09/02/coverage-emma-cobertura-maven.

10 comments:

  1. You might take a look at our latest effort to create new code coverage backend:

    http://www.eclemma.org/jacoco/

    Currently only Ant tasks are available, but the mid term target is using JaCoCo as the backend for EclEmma.

    Please feel free to contribute your use-cases and requirements.

    Best regards
    -marc

    ReplyDelete
  2. Thanks for the link! I'll try it out. As I said above, EMMA is close, but it leaves a few lines uncovered here and there, and for a perfectionist it's really painful. I'm going to attach the above screenshots to the original bug reports to make it clearer what the gaps are.

    ReplyDelete
  3. I've just found your blog and I'm already going back to read the past entries, since we seem to have similar "working conditions": large (multi million LOC) Java legacy projects and trying to use open-source code-analysis tools on them. Thank you for writing about your experience!

    About code coverage: we went with Emma, mainly for speed. A little poking around seems to indicate that Emma simply uses an array of ints to store the coverage data and Coberture has a nice OO data model, which results in a lack of speed (on our test suite it was ~8x slower, which made it unusable). Also, both of them seem to have their quirks regarding the bytecode they instrument:

    - The current Coberture version (1.9.3) throws all kinds of exceptions for "synthetic" bytecodes (an issue with private inner classes). Its source code parser (taken from JavaNCSS) also fails in some circumstances which I wasn't able to narrow down.

    - Emma dislikes generate RMI stub classes: if they are present, you won't get a report generated at the end, even though the instrumentation process displays only warnings!

    Also, it would be nice if projects like Sonar (http://sonar.codehaus.org/) would support multi-million LOC out of the box and direct attribution to individual committers.

    ReplyDelete
  4. Forgot to say the things I started to write the comment for in the first place: thanks for sharing the information and I'm looking forward to more great posts!

    ReplyDelete
  5. EMMA is definitely faster.

    If you haven't done so already, please make sure the Cobertura team receives your bug reports--they have been very responsive to things I've reported.

    And thank you for the positive feedback--it's encouraging!

    ReplyDelete
  6. Interesting blog, i usally be aware all about all different kind of sofware. i am online all the time, and this action allow me to see a site costa rica homes for sale and i like it too much. beyond all doubt without my computer i never would have seen this site too.

    ReplyDelete
  7. THIS POST IS OBSOLETE. JMockit generates branch coverage, and is the most-accurate of the coverage tools I've used. If there was an Eclipse plugin with editor integration like eclEmma, JMockit would be perfect.

    ReplyDelete
  8. @Jim: how does JMockit gather the coverage data? Sourcecode instrumentation? Bytecode instrumentation? For the later - compile time or runtime? Also, how does the performance compare to JaCoCo? (which I found to be the fastest from the open-source ones available out there).

    ReplyDelete
  9. JMockit instruments bytecode. It's very fast. I haven't benchmarked it against JaCoCo, but would expect it to be about the same, because running JMockit on a very large Java project doesn't seem to slow it down. The reason I say JMockit is more accurate than JaCoCo: JMockit doesn't use "block" analysis, and JaCoCo (and its EMMA predecessor) does. So with JaCoCo you get dinged for not covering blocks that are actually compiler generated (for example, synthetic constructors and the various stock methods for enums like valueOf). And there were other blocks-related problems. The other thing is that we already use JMockit for mocking, so to get coverage in addition is basically free, and we have fewer javaagents competing on the classpath (http://sourceforge.net/p/eclemma/bugs/138/).

    ReplyDelete