Selecting A Coverage Tool

At work, we use Clover to track coverage.

For programming at home, a Clover license is way too expensive (not as bad as those software vendors that used to not list prices and instead said "Call for quote", but close).

Also, we've had some issues with Clover:

1. It sometimes falsely claims that code is covered when it isn't.
2. It doesn't support branch coverage.
3. It can't instrument assignments in conditionals.

(The Clover rep confirmed the above problems a couple of years ago. These may have been fixed by now. However, because the licenses are so expensive, we haven't upgraded, so we're stuck with the problems even if they're fixed in the current version. Also, the Clover rep didn't think the third issue would ever be fixed due to the way they instrument the code.)

For a simple example of problem #2, start with a function like this:

public boolean eval(boolean x, boolean y, boolean z)
{
return x && (y || z);
}

Clover scores 100% coverage if the entire expression evaluates at least once to true and at least once to false, so this is sufficient:

assertTrue(eval(true, true, false));
assertFalse(eval(true, false, false));

But evaluating all of the meaningfully distinct ways for the expression to evaluate requires more tests:

assertTrue(eval(true, true, false));
assertTrue(eval(true, false, true));
assertFalse(eval(true, false, false));
assertFalse(eval(false, true, false));

Problem #3 means that this code can't be covered:

while (currentLine = stream.readLine() != null)

Covering that code with Clover requires rewriting it as a do/while.

For the above reasons, I went in search of a coverage tool other than Clover for programming at home.

Googling around, EMMA and EclEmma (an Eclipse plugin for EMMA) kept showing up, so I tried them.

Unfortunately, where Clover incorrectly says that uncovered code is covered, EMMA incorrectly says that covered code is not covered:



True, it only screws up occasionally, but if you're five lines short of 100% coverage, and the five lines are spurious tool errors, it's annoying.

This was unfortunate, because the EclEmma plugin is very fast and completely non-intrusive, and would be great to use. Plus, it's currently the only coverage tool that works with Android.

After reporting these problems, I looked around for alternatives. A bunch of coverage tools were listed here, and I tried each of them out. Most didn't install, or weren't compatible with the latest Eclipse, or hadn't been maintained for a number of years, etc.

But Cobertura works really well.

Unfortunately, there's no Eclipse plugin for Cobertura, and it's slower than the other two tools, but compared to getting the wrong answer, that's not too much to give up. The dream tool would combine Cobertura's accuracy with EclEmma's speed and usability, but no such tool exists.

This table compares the three tools:

	Clover	EMMA/EclEmma	Cobertura
Statement coverage	yes	yes	yes
Branch coverage	no	no	yes
No quirks or bugs	no	no	yes
Fast	yes	yes	no
Reports	yes	yes	yes
Maven integration	yes	yes	sort of
Works with mocking tools	yes	yes	yes
Eclipse plugin	yes	yes	sort of
Free	no	yes	yes
Active community	yes	yes	sort of

Notes:

• Maven integration for Cobertura is provided by a separate open-source project. This project has been around for a while. There are some problems using the integration.
  
• Eclipse integration for Cobertura is provided by a separate open-source project. This project is new and not very mature yet.
• Cobertura's community support is rated "sort of" because the only forum is via an email distribution list, and responses can take a couple of days. On the other hand, the code is mature and easy to use, so there isn't much need for help (most of the emails I sent were questions about setting up the ant scripts).
• JMockit can be configured to generate coverage reports, but currently only statement covered is supported. The author plans to add additional features (including branch coverage), at which point it should be evaluated like the other three tools.
• For a comparison of coverage tools that arrives at the opposite conclusion to mine (partly due to other requirements), see http://javapulse.net/2008/09/02/coverage-emma-cobertura-maven.