Software, Technology, and Science

The Tyranny Of Extroverts

2012-01-28T11:06:00.000-08:00

Here's a book that provides insight into why some people love open cubicles and a highly collaborative/noisy environment, and others don't:

http://www.amazon.com/exec/obidos/ASIN/0307352145/

It's unfortunate that the extroverts got the upper hand, because it's easy to work as an extrovert in an environment designed for introverts, but the converse is not true.

The author lists some contributions made by introverts. For example, the theory of relativity. Who knows what other contributions we're forfeiting, or at least slowing down, by creating environments in which introverts cannot be productive?

Open Cubicles Must Die

2010-05-31T18:54:00.000-07:00

"The cynic knows the price of everything and the value of nothing." - Oscar Wilde

In 1987, Tom DeMarco and Timothy Lister published the first edition of "Peopleware: Productive Projects and Teams".

The book discusses many reasons software projects fail, and provides numerous recommendations for making projects succeed.

Part II of the book discusses the effect a bad office environment has on productivity, and recommends (among other suggestions) private offices as a significant productivity enhancer. The book provides quantitative evidence to substantiate the recommendation. And the book eviscerates the productivity claims of cubicleware vendors, which are shown to have no substantiation (the book calls those claims "proof by repeated assertion").

Fittingly, the book is out of print.

If anything, in the decades since the book was published, things have gotten worse (other than getting rid of P.A. systems). We are in the grip of an industry-wide pathology that produces expensive, flow-disrupting buildings.

How did we get here?

A lot of it has a lot to do with the counting of beans.

It is easy for a bean counter to count tangible things, like square feet.

It is not easy for a bean counter to count intangible things, like lost productivity due to a noisy work environment yanking programmers out of flow.

Suppose at some point a bean counter notices that their company spends more for facilities than "comparable" companies spend. This seems to the bean counter like fiduciary irresponsibility, so the bean counter launches an investigation.

Because the bean counter cannot count intangibles, the bean counter does a poor job of counting beans, and works up an Excel spreadsheet that shows how, if they get rid of walls and doors, and reduce the space each programmer gets, and eliminate bookshelves, and generally pack programmers into "pods" where they're practically sitting in each other's laps, the company will save a bundle.

On paper at least.

Intrigued by the putative savings to be had, the exec staff--sitting in their capacious, private, quiet offices--decides to engage the services of a cubicleware vendor to "work up" some designs for how the building could look. (Note that open cubicles are by this point a foregone conclusion, and that by involving a cubicleware vendor as the consultant, we have an instance of bid rigging so egregious that if this was a government contract somebody would go to jail.)

The designs are very slick. Spiffy in color, appealing in the abstract, the revamped building will "facilitate collaboration" (and who would question the need for that?), foster "inter-departmental serendipity" (which also sounds desirable, whatever that is), and so forth.

Nobody asks the programmers for their input. Or they do ask, but when the programmers say they want private offices so they can focus on work, this is dismissed as "impractical", even though drywall is cheaper than cubicleware.

The building is revamped, the programmers move into it, and are pissed off because they feel like they're regarded as interchangeable, revenue-producing, chunks of meat. Morale drops and productivity goes into a tailspin.

Adding insult to injury, the company newsletter makes a big deal about how the building, in a triumph of form over function, won an AIA award for "design excellence".

Those programmers who stick around find ways to circumvent the limitations of the building. For example, they take refuge in the server room (lots of white noise), or work at home.

But these strategies ultimately backfire, because the bean counter's minions notice that a lot of cubicles--the ones occupied by programmers hiding in the server room or working at home--are empty, and conclude that the building was in fact overprovisioned--that there could have been smaller cubicles, and fewer of them. Inevitably, the bean counter concludes that cubicles shouldn't even be allocated to individual programmers, but instead should come from a pool, first-come/first-served.

Because programmers are now sharing keyboards and mice, they share flu germs, and occupancy rates go down again. As does productivity.

To make up for the loss of productivity, the company needs to hire more programmers. But where to put them? Clearly cubicles take up too much room. Programmers should just work on long tables, like medieval craftsmen.

Meanwhile, the aggressive reduction in facilities costs is noticed by other companies' bean counters, who don't want to look like spendthrifts based on the latest "comparables".

And so continues the industry-wide death spiral.

Another reason we have open cubicles is due to differences between how programmers work and how the execs making the decisions work.

Execs are people people. They like to meet new people. They like to shake hands. They like conversation. They're extroverts.

Cubicleware vendors are also people people. They're in sales, so of course they're extroverts.

Programmers are, well, if you're a programmer you know what we are. It's not that we're necessarily ugly, or antisocial, but we're a lot more likely to send an IM to somebody 10 feet away than to get up and walk over there to talk. We like to get into flow and stay there. We like to solve problems.

When people people design a work environment, they design it how they like to work. Which involves a lot of face-to-face interaction with other people, long phone conversations, open airy spaces, etc. It's almost the exact opposite of the conditions that foster flow.

A few companies ago, Lisa was in a cube farm at a Fortune 50 company, concentrating, when some ninny of a people person walked into the area and, apparently discomfited by the unfamiliar sound of work getting done, said: "There isn't enough visitin' going on".

Visitin'.

Honestly?

Another reason we have open cubicles is that private offices are seen as a "perk". But they're not a perk, they're a software-development tool, and an effective one. Programmers don't care about status or prestige. They care about being effective. And they need good tools to do that.

Unfortunately, many execs do care about status and prestige, and they see private offices as a perk. It's a problem of semiotics.

Another reason we have open cubicles--and this is a self-inflicted wound--is that some of the newer methodologies emphasize collaboration, pair programming, etc.

But one person's collaboration is another person's pointless interruption.

When people are yakking around a programmer who's trying to get into flow or stay in flow, that constitutes involuntary collaboration. It's like being on an airplane before they banned smoking--there was no way to escape.

With private offices, programmers can always opt into collaboration by opening their door, sitting down in a communal work area, etc.

Without private offices, programmers can't opt out of collaboration. Which is a shame, because at some point, somebody, somehow, has to actually write some code.

But, you say, the programmer can always put on headphones. Yes, they can, except that page 78 of the second edition of Peopleware explains why that's a bad idea (a Cornell study showing that creativity is hampered by listening to music while programming).

The saddest part of all of this is that if companies would focus on maximizing programmer productivity instead of on minimizing facilities costs, they could accomplish a lot more work with far fewer programmers, which would save so much money in salaries it would dwarf the incremental cost of private offices.

Penny-wise, pound foolish.

Not all companies make this mistake. For example, when Adobe was building its headquarters in San Jose, they asked the programmers what they wanted, and then did something crazy: they listened. (Guess what the programmers wanted.)

Sometimes I get the argument that young programmers prefer open cubicles and more collaboration, but how would young programmers know? Did a lot of them previously work in large, quiet, private offices? Doubtful--cubiclitis has been infecting the industry since the 1960s, which is longer than young programmers have been working (indeed, longer than many of them have been alive).

Claiming that young programmers prefer it this way implies they ever had a choice. But they didn't have a choice, so there's no control group, and this is just more self-serving nonsense.

I'll close with a visit a few companies ago to one of the largest cubicleware vendors in Silicon Valley. We went there because we were opening up a development center, and needed furniture. The vendor took us on a tour of their facility, which doubled as a showroom and their headquarters. As the tour lady stood there telling us about the merits of this or that piece of furniture, I noticed some employees of the vendor in the background, in the accounting department.

They were glaring at us, because we were making too much noise.

Some interesting articles about or related to this topic:

Note: Although out of print, Peopleware can be found online as a PDF, or purchased used for around a hundred dollars.

Update: Look how far we have fallen--a friend sent: "In the '60's my father took me through his insurance company offices after hours. Most of the desks were in open bays. Then we came to the computer area and he pointed out the leased IBM 360/30. Nearby there were several special rooms along the wall with doors and glass walls. I asked why some of the desks were sealed off like that and he told me that the programmers needed quiet places to work without interruption because it was so important that their programs be correct. Your article brought back that memory."

Scrum Is A Flow-state Interrupter

2010-05-31T16:01:00.000-07:00

Overall, Scrum is useful.

But the Daily Interruption is an abomination.

First of all, because the Daily Interruption is supposed to be held every workday, it interferes with no-interruptions days.

Second, the Daily Interruption serves no purpose.

I realize this is heresy, but look at what is supposed to happen in the Daily Interruption. Every team participant is supposed to answer three questions:

What have you done since yesterday?
What are you planning to do today?
Is anything blocking you?

The first two questions can and should be replaced by emailed status reports. Why waste time telling me something I can read (if I care, which I don't)?

The third question is even more asinine. Why would a programmer wait an average of 12 hours to bring up something that's blocking them? Shouldn't they deal with the problem right away?

If someone comes to a Daily Interruption to kvetch about a blocker, my question to them is always why haven't you dealt with this already, dummy.

Because the Daily Interruption serves no purpose, requiring programmers to attend it is fiduciary irresponsibility.

The Daily Interruption was invented to solve communication problems on dysfunctional teams. If a team is not dysfunctional, the Daily Interruption is make-work. Wasteful, flow-killing make-work.

But just try convincing a ScrumMaster that the team can do without the Daily Interruption. They've been to ScrumSchool, you see. They're ScrumCertified.

Except they didn't read the whole book. If they had, they'd know that Agile methodologies (of which Scrum is an instance) support Method Tailoring. And Method Tailoring allows a team to modify a methodology to fit the needs of the team and the project the team is working on.

In other words, a team can, with the blessing of Agile, tailor Scrum to eliminate the Daily Interruption.

As a compromise, if a team simply must have the Daily Interruption, at least only have it on the three days of the week that are already contaminated by other meetings. Leave the no-interruptions days alone.

Because our team has enlightened management, the Daily Interruption is not scheduled on no-interruptions days, and attendance is optional on the other three days.

Mandatory Meetings

2010-05-31T15:50:00.001-07:00

Unless required by law, there should be no such thing as a mandatory meeting.

The best way to get people to attend a meeting is to make the meeting worth attending.

Mandatory meetings are particularly annoying when they're supposed to be "fun" (for example, a team-building offsite). Mandatory fun is an oxymoron.

Feedback On Non-recurring Meetings

2010-05-31T15:42:00.001-07:00

Because meetings are flow-state interrupters, companies should have a process for improving the quality of all meetings, including non-recurring meetings.

After every non-recurring meeting, a survey should be sent to everyone who attended the meeting, asking some questions:

Did the meeting start on time?
Did the meeting end on time?
Do you understand the purpose of this meeting?
Do you feel that this meeting met its stated purpose?
Based on the average fully burdened salary of the attendees, the cost of this meeting (not counting opportunity cost) was $<put cost here>. Do you think that cost was justified?
Is there any information communicated during the meeting that should be communicated by some other means (for example, in email)?
Which of the following actions would have improved the meeting (check as many as apply)?:

Add ______ to the agenda.
Remove ______ from the agenda.
Change the allocated time to ______.
Reduce the number of attendees.

Meeting ratings should be considered in employee's annual reviews.

Winnowing Recurring Meetings

2010-05-31T14:58:00.000-07:00

Whenever someone takes over an IT department, the first thing they're supposed to do is cancel all reports.

The thinking is that many reports are leftovers nobody cares about anymore, and shutting them off is the best way to find out which ones matter to someone.

Have you ever found yourself in a recurring meeting where you wonder what the point of the meeting is?

That meeting is like a zombie report. Nobody really needs it anymore, but there's no process in place for getting rid of the meeting. It just lurches along on inertia, wasting everyone's time.

Because meetings are flow-state interrupters, companies should be proactive about getting rid of recurring meetings nobody needs anymore.

Every recurring meeting should have to interview for its job on an ongoing basis.

After every recurring meeting--or at least quarterly--a survey should be sent to everyone who attended the meeting, asking some questions:

Did the meeting start on time?
Did the meeting end on time?
Do you understand the purpose of this meeting?
Do you feel that this meeting met its stated purpose?
Based on the average fully burdened salary of the attendees, the cost of this meeting (not counting opportunity cost) was $<put cost here>. Do you think that cost was justified?
If you do not feel this meeting is justified, do you feel that it should simply be cancelled, or improved?
If you feel the meeting should be cancelled, is there any information communicated during the meeting that should be communicated by some other means (for example, in email)?
If you do not feel that this meeting should be cancelled but instead should be improved, which of the following actions do you think should be taken to improve it (check as many as apply)?:

Add ______ to the agenda.
Remove ______ from the agenda.
Change the allocated time to ______.
Hold it less often.
Make it non-recurring (only hold it when needed).
Reduce the number of attendees.
Combine it with the ______ meeting.
Reschedule it to ______ to leave a larger contiguous block of uninterrupted time available.

If a meeting fails the interview, it should be cancelled, no matter how important the meeting owner thinks the meeting is.

Constant review and winnowing of meetings can, over time, convert this:to this:If a meeting passes the interview but needs improvement, it should be improved per the survey feedback.

Meeting ratings should be considered in employee's annual reviews.

A Huge Gain In Productivity For A Tiny Amount Of Effort

2010-05-31T14:08:00.000-07:00

Does this look familiar to you?:If so, there's no way you can get into flow during work hours. You either don't get into flow (bad), or work evenings and weekends to get anything done (also bad).

Fortunately, there is a simple, free, and ridiculously effective solution to this problem.

Working with management, institute two no-interruptions days per week.

Stagger the two days to avoid being out of contact with others for two days in a row. (I recommend Tuesday and Thursday, but Tuesday and Friday also work well.)

Make the two days sacrosanct.

Have everyone on your team reserve the two days on their calendars:Make your calendars public so everyone can clearly see the two days blocked out.

Politely decline all meeting invites on the two days, with the explanation that the two days are for focused work, and suggest scheduling the meeting on one of the three non-focus days that can be wasted playing office.

Be firm.

And also be firm with yourself. The two days are not for running personal errands, goofing off, etc. The two days are for focused working. If you screw this up, management will consider the experiment a failure, you'll lose credibility, and it will be hard to convince them to remove other, more difficult obstacles.

On the two days, shut off IM, Twitter, RSS, your cellphone, etc. It will be hard to do that at first. You'll feel cut off, out of the loop, going through withdrawal.

Good. That means it's working.

Ideally, assuming you have a good home-office work environment, work at home on the two days. That way you can roll out of bed, grab something quick to eat, and start coding.

Be sure to put in a full eight hours of focused time on each of the two days. (You will find this easy to do, because you'll be in flow.)

Congratulations, you just worked your 16 hours/week! You are 33% more efficient than the industry average. If you manage to work a few more hours on the other three days of the week, that's all upside.

Some people can get really offended that you declined their super-important meeting. Boo hoo. This is when it's vital to have management backing, so you can send the invitee to management for goal alignment. You need management to make it clear that interrupting a programmer on a no-interruptions day is stealing from the company.

Because you will actually be able to clock 16+ hours a week with this system, management will give you the backing you need.

You also need management backing if you work across teams, so that the no-interruptions days can be aligned. (It does no good to declare Tuesday as a no-interruptions day if a team you have to work with picks a different day.)

If you can't get management backing for this and/or if upper management is for it but your own manager is against it, change jobs. Seriously.

How dogmatic should you be about no interruptions on those days? Well, there's sort of an art to getting it correct. Obviously if you are blocked because you have to ask somebody else a question, it would be nice to be able to get an answer right away. But it's a slippery slope. You have to be careful to only interrupt when it's really important. And you should never, ever, accept an invite for a recurring meeting on a no-interruptions day.

The irony of this system is that it costs the company nothing to implement. Mostly it just involves changing the days for some meetings. A simple, mechanical process.

Flow Versus "New Cognition"

2010-05-31T13:05:00.000-07:00

There has been some speculation online that kids these days aren't actually interrupted all the time and incapable of concentrating, but are instead just better at multitasking, and are in effect changing the way people think--that we're witnessing the emergence of a new kind of cognition. Evidence for this includes measurable changes to brain structures from playing lots of video games, and so forth. Some posts have gone so far as to say that because of this new cognitive style, kids shouldn't have to learn to spell, because good spelling isn't required for online communication. And of course anyone who disagrees is an old fogey who just can't keep up. ADHD isn't a disease, it's a New Way Of Thinking.

Uh huh.

Andrew Wiles spent seven years secluded in an attic. When he emerged, he had proven Fermat's Last Theorem as a side-effect of proving a much more general and profound result.

Now suppose he had been one of these New Cognition types. How far do you suppose he'd have gotten on that proof?

I think he'd still be in the attic, banging away on his Wii.

As an old fogey who can spend hours at a time in flow, I'm worried that kids will grow up never entering a state of flow at work, and the cost to humanity in lost productivity will be gigantic. Our best and brightest will constantly thrash, never achieving their potential.

They'll be totally pwnd.

Fighting For Flow

2010-05-31T12:33:00.000-07:00

"And George, while his intelligence was way above normal, had a little mental handicap radio in his ear. He was required by law to wear it at all times. It was tuned to a government transmitter. Every twenty seconds or so, the transmitter would send out some sharp noise to keep people like George from taking unfair advantage of their brains." - Harrison Bergeron, by Kurt Vonnegut

Flow is the highest-productivity mental state of which humans are capable.

A person operating in flow can be vastly more productive than a person who is not in flow.

A person in flow often starts working in the morning when it is light, and snaps out of flow late at night when it is dark, with no sense of the passage of time.

A person in flow sometimes forgets to eat.

A programmer in flow can do a week's worth of work in an afternoon, and a month's worth of work in a couple of days.

A company that makes it easy for programmers to work in a state of flow has a strategic advantage over companies that have not figured this out.

A programmer who learns to maintain a state of flow has a career advantage over programmers who have not learned how to do this.

Unfortunately, it is becoming increasingly difficult to maintain a state of flow.

Some of this is the fault of programmers. It is easy to be seduced by email, IM, Twitter, etc.

Fortunately, this can be rectified by self-discipline. Shut down email, log out of IM and Twitter, turn off the cellphone, and focus.

But companies also contribute to the problem. Perversely, they seem to go out of their way to squander programmer productivity. Instead of uninterrupted blocks of time, days are broken up with meetings. Instead of quiet offices, they stick programmers in noisy "pods". Instead of encouraging working at home, they disproportionally terminate remote workers during layoffs (then wonder why so few employees become remote workers).

It can become pathological. Several companies ago, a colleague took two weeks of vacation so he could come to work every day and focus on something he'd been trying to get done for over a year. If anyone stopped by to ask him something, he'd tell them he was on vacation, and tell them to go away.

I didn't make that up.

Programmers need to fight back. It's a matter of self-preservation.

Just complaining won't change anything. You'll just be seen as a "troublemaker".

But there is one stick big enough to beat management over the head with to get them to listen.

Money.

Programmers cost money. They cost more money than anything else in the budget. A Silicon Valley programmer, fully burdened, can cost $250k/year.

Wasting programmer productivity is fiduciary irresponsibility.

Mechanisms that preserve programmer productivity deliver tangible benefits to the bottom line.

The industry average for under-the-headphones actual coding--and this is criminal--is only about 12 hours/week. Not per day. Per. Week.

That's crazy.

There are 168 hours in a week.

Where do the rest of the hours in the week go? WoW? Air hockey? American Idol?

Now suppose instead you can eke out a whopping 16 hours/week. That doesn't seem like much, but it's a 33% improvement. It's like being able to hire 33% more programmers without spending another penny. Think what the company could do with that much additional engineering resource! They could deliver more features. They could fix more bugs. They could crush their competition.

But it's not even just a 33% improvement, because you're shooting for achieving 16 hours a week of hours in flow, so the productivity gains are much higher than a mere tally of hours indicates. Both the number of hours and the quality of hours increase. It's a force multiplier.

Now you're talking a language management understands.

So go to management, and make the business case. Tell them you want to help the company be more successful by removing obstacles to programmer productivity. List the obstacles, and suggest how each could be eliminated.

Start with something small, so their heads don't explode with concerns about "risk". I recommend starting with this.

Then when you show that the small change resulted in a measurable improvement, move to the next obstacle.

Your team will be vastly more productive than teams that still suffer from the obstacles. Other managers will start to panic and want to know what the "secret" is, and start removing obstacles as well. Management will stop muttering about reducing costs by "offshoring" (because you found a way to reduce costs while keeping the jobs local).

The Joy Of Legacy Code

2010-05-30T18:20:00.000-07:00

The project I'm currently working on consists of more than a million lines of legacy code.

Every other project I worked on was a "greenfield" project--a blank sheet of paper.

It took a while to adjust.

I'd always thought of legacy code as something that dull people maintained in the basements of banks.

I was an idiot.

In fact, legacy code is important code. If it wasn't important, a company would simply decommission it, but instead a company continues to fund its development (or at least its maintenance), despite all the problems the programmers who work on the code complain about.

Legacy code is virtually bug-free code, at least on the happy path. If it wasn't virtually bug-free, end users would complain and/or the code would crash. But it doesn't do that. Instead, it hums merrily away, the bugs having been dispatched years ago.

Working on legacy code, you don't have to worry about the project getting cancelled, the start-up running out of money, etc. You already have customers, and you already have revenue.

Legacy code pays well. A company can find millions of programmers who can write hello world, but not many programmers can do things like:

Migrate a legacy database to a non-backward-compatible schema without impacting end users
Revamp a legacy user interface to use Flex
Convert a legacy build system to Maven
Replace homegrown legacy infrastructure components with open-source libraries
Refactor unstructured legacy code into layers

Those are hard problems, like cutting a diamond without shattering it. In comparison, writing code from scratch is trivial.

Besides, new code is just legacy code in waiting. Assuming some new code proves useful and survives, it eventually turns into legacy code. Yesterday's spiffy greenfield project is today's legacy codebase.

In most contexts, the word "legacy" has a positive connotation: freedom is the Founding Fathers' legacy; the National Park System is Teddy Roosevelt's legacy; prudent saving and investing leaves a legacy for one's children to inherit. This should be true for legacy code as well.

Consider yourself lucky if you get an opportunity to work on legacy code!

Automatic File Renaming While Downloading

2010-05-09T13:35:00.000-07:00

I got tired of having to rename files when downloading them, so I wrote a Windows Service to automate it.

A Tool That Does Not Exist

2010-03-20T18:40:00.000-07:00

Moved to http://free-inventions.blogspot.com/2010/04/super-tool-for-refactoring.html.

Structuring An Android Application For Reuse

2009-12-19T14:04:00.000-08:00

The Android application I'm working on consists mostly of code that's not Android-specific:

- Database schema creation.
- SQL queries.
- Data access (objects mapped to and from database).
- Business logic.

There is only a small amount (less than 10%) of Android-specific GUI and notification code.

Other than the code that creates the database schema (which is database-specific), all of the non-Android code is device- and OS-agnostic. It just needs a Java runtime.

I wanted to be able to easily reuse the portable code to run the application on a website, and on other Java-based handhelds.

The first step involved abstracting the database so that any database can be used, instead of just Android's binding to SQLite:

http://code.google.com/p/blog-code-hosting/source/browse/#svn/trunk/android/reuse/CommonPortable/src/main/java/com/jimandlisa/common/database

and also abstracting the logger:

http://code.google.com/p/blog-code-hosting/source/browse/#svn/trunk/android/reuse/CommonPortable/src/main/java/com/jimandlisa/common/logging

Once that was done, I could test the database and logger independently from Android:

http://code.google.com/p/blog-code-hosting/source/browse/#svn/trunk/android/reuse/CommonPortableTestUtils/src/main/java/com/jimandlisa/common/testutils

http://code.google.com/p/blog-code-hosting/source/browse/#svn/trunk/android/reuse/CommonDatabaseTestUtils/src/main/java/com/jimandlisa/common/testutils/database

http://code.google.com/p/blog-code-hosting/source/browse/#svn/trunk/android/reuse/CommonPortableTests/src/test/java/com/jimandlisa/common/tests

The portable business logic is in another set of projects:

MyAppPortable
com.jimandlisa.myapp.common
com.jimandlisa.myapp.controller
com.jimandlisa.myapp.database
com.jimandlisa.myapp.model

MyAppPortableTestUtils
com.jimandlisa.myapp.testutils

MyAppPortableTests
com.jimandlisa.myapp.tests

Finally, the Android-specific code is in two more projects:

MyAppAndroid
com.jimandlisa.myapp

MyAppAndroidTests
com.jimandlisa.myapp.android.tests

The inter-project dependencies look like this:

CommonPortable:
n/a

CommonPortableTestUtils:
CommonPortable

CommonPortableDatabaseTestUtils:
CommonPortable
CommonPortableTestUtils

CommonPortableTests:
CommonPortable
CommonPortableTestUtils
CommonPortableDatabaseTestUtils

MyAppPortable:
CommonPortable

MyAppPortableTestUtils:
CommonPortable
MyAppPortable

MyAppPortableTests:
CommonPortable
CommonPortableTestUtils
CommonDatabaseTestUtils
MyAppPortable
MyAppPortableTestUtils

MyAppAndroid:
CommonPortable
MyAppPortable

MyAppAndroidTests:
CommonPortable/target/classes
MyAppPortable/target/classes
MyAppAndroid/bin
CommonPortableTestUtils
MyAppPortableTestUtils

The CommonDatabaseTestUtils are broken out from CommonPortableTestUtils because they depend on JARs that aren't loaded on the device, so they can't be included in the dependencies for MyAppAndroidTests.

MyAppAndroidTests depends on target/classes and bin for reasons explained in http://jimshowalter.blogspot.com/2009/10/developing-android-with-multiple.html.

So was it worth it?

Well, yes, and no.

There are definitely advantages:

I can write any other database-intensive application by cloning this setup and reusing the code in com.intuit.jimandlisa.common.*.
The database-abstraction layer doesn't have the problems reported in Google Android issues #3302, 3296, and 3304.
Having much of the code be regular, portable Java makes it easier to write coverage tests because EasyMock, JMockit, and Cobertura are available.

But there are also disadvantages:

This approach creates a lot of separate projects, and that's kind of a pain to set up.
I have to maintain the database-abstraction layer.
More work is required to mature the database-abstraction layer. (It currently doesn't support binding args or compound keys, all keys must be longs, etc.)
Because the database-abstraction layer isn't a native binding, it's slower. For a cellphone app with a few database operations per hour, this doesn't matter, but it will matter on a website.

On balance, the approach makes sense for the particular application I'm working on, because it is almost entirely not tied to Android. But the approach wouldn't be appropriate for a GUI-intensive application that makes a lot of calls to Android APIs.

It might be better to model the application, and generate the code. With so many devices, OSs, and vendors, code generation might be the only pragmatic way to develop a cross-platform application.

Update: The sample code has been updated to add createSchema and updateSchema to AbstractDatabase in order to make execSql private.

Developing Android With Multiple Eclipse Projects

2009-10-04T13:20:00.001-07:00

The Android application I'm developing at home has non-Android-specific schema, database access, and business logic, so it makes sense to partition the application into multiple Eclipse projects, some Android-specific and others not.

One benefit of using separate projects is that I can write a second version of the application that runs on a website instead of on a handheld without having to rewrite the bulk of the code.

But there's a catch: when the Android-specific unit tests are moved into a separate Eclipse project, the loader reports "Class resolved by unexpected DEX error" and the unit tests fail with "java.lang.IllegalAccessError: cross-loader access from pre-verified class". (In 2.1 the error remains, but the message has changed to "Class ref in pre-verified class resolved to unexpected implementation".)

I first ran into this problem in 1.5 Android, and upgraded to 1.6 hoping it would go away, because in 1.6 using separate Eclipse projects for unit tests is recommended:

http://mobilebytes.wordpress.com/2009/09/19/new-eclipse-android-test-features

http://www.danielswisher.com/2009/06/as-new-android-developer-i-have-been.html

There is even a toolbar icon and wizard to create an Android test project that points to a separate Android project to be tested.

Googling around, I found other programmers having the same or similar problems:

http://groups.google.com/group/android-platform/browse_thread/thread/20ff41b925e04dd4

But googling also found programmers with multiple projects but not the problem:

http://www.anddev.org/unit_testing_private_methods-t7847.html

What makes some programmers have no problem using multiple Eclipse projects to separate Android application code from Android unit-test code, while other programmers get the loader errors?

The problem shows up when the loader sees the same class loaded more than once:

http://groups.google.com/group/android-developers/browse_thread/thread/3440dd8e11a1b481

which can happen if the main Android code and the Android unit tests share code from another Eclipse project:

http://groups.google.com/group/android-developers/browse_thread/thread/5537ae10e4143240

which is the setup I have.

To reproduce the problem:

Create an ordinary Java project called LoaderProblemCommon, and create this class in it:

package com.loaderproblem;

public class LoaderLogger
{
public static void log(String message)
{
System.out.println(message);
}
}

Create an Android project called LoaderProblem, add the LoaderProblemCommon project to its build path, and create this class in it:

package com.loaderproblem;

import android.app.Activity;
import android.os.Bundle;

public class Hello extends Activity
{
public void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
setContentView(R.layout.main);
logHello();
}

public static void logHello()
{
LoaderLogger.log("Hello");
}
}

Add LoaderProblem to the Android manifest.
Create an ordinary Java project called LoaderProblemTestUtils, and create this class in it:

package com.loaderproblem.testutils;

public class TestUtil
{
public static final void doNothing()
{
}
}

Create an Android test project called LoaderProblemTests, add the other three projects to its build path, and create this class in it:

package com.loaderproblem.tests;

import android.test.AndroidTestCase;

import com.loaderproblem.Hello;
import com.loaderproblem.LoaderLogger;
import com.loaderproblem.testutils.TestUtil;

public class LogHelloTest extends AndroidTestCase
{
public void testLogHello()
{
LoaderLogger.log("Hello");
Hello.logHello();
TestUtil.doNothing();
}
}

The result should look like this:

Compile all four projects, create a brand-new AVD, and run LoaderProblem as an Android application. It should deploy and display:

Now try to run LoaderProblemTests as an Android JUnit test. It will fail with:

Or, for 2.1, it will fail with:

I first tried to fix this problem by editing the build path for LoaderProblemTests so it only depends on the LoaderProblemTestUtils Eclipse project, and then adding back the necessary classes by adding dependencies on the other projects' class folders:

This "worked", but it introduced several unwanted side effects:

Now that the dependent project's source is seen as a class folder, one can no longer smoothly refactor across projects or navigate into classes (via F3, for instance), because these files are now treated as a compiled objects. Attaching the source to the classfolder library sort of works, but then it brings up an uneditable version of the source. [reported by a reader]
It screws up EclEmma. For details see https://sourceforge.net/tracker/?func=detail&atid=883351&aid=2934081&group_id=177969.

The proper way to fix this problem was reported by another reader:

In the build path for LoaderProblem, click on "Order and Export".
Check the box for LoaderCommon, to let LoaderProblem export the LoaderCommon classes as well as its own.
In the build path for LoaderProblemTests, remove the dependency on the LoaderCommon project. The dependency is already now covered by the LoaderProblem dependency.
Clean, build, rerun LoaderProblem, then rerun LoaderProblemTests.

This solution also fixes the EclEmma problem.

Using Eclipse With Large Code Bases, Part V

2009-09-26T10:53:00.000-07:00

Previous posts described how to use file-system links and various Eclipse features to make a large, single-rooted source tree look like a set of smaller, multi-rooted Eclipse projects.

But there's a catch: the Perforce Eclipse plugin doesn't work, because the Perforce client spec specifies the root of the actual source tree on disk, but the Eclipse Perforce plugin thinks the source tree is rooted at the directory that contains the Eclipse projects.

For example, if we start with this client spec:

Client: originalclient
Root: C:\
View:
//trunk/src/... //originalclient/src/...

and use links to create this Eclipse structure:

C:\projects\
  Child1\
      .classpath
      .project
      src\
          com\
              parent\
                  child1 → linked to
                  C:\src\com\parent\child1
  Child2\
      .classpath
      .project
      src\
          com\
              parent\
                  child2 → linked to
                  C:\src\com\parent\child2

then the Perforce root is C:, but the Eclipse Perforce plugin thinks the root is C:\projects.

To fix this, we updated our link-generation script to generate a second client spec, where the client spec's root is the directory that contains the Eclipse projects, and the client spec maps individual sub-roots to their counterparts in the Eclipse projects (a one-for-one mapping between Perforce and the links):

Client: myclient
Root: C:\projects
View:
//trunk/src/com/parent/child1/... //myclient/Child1/src/com/parent/child1/...
//trunk/src/com/parent/child2/... //myclient/Child2/src/com/parent/child1/...

Note that although we have three different "views" of the source tree (two in Perforce and one in Eclipse), there is only one actual occurrence of each source file on disk, which means changes are kept in sync among all three views.

So if Perforce can do this mapping, why do we need the links at all? In many cases, you might not. In our case, though, the file layout in Perforce is really complicated, and it would be painful to create a client spec that maps all of the files to the file system. For example, there are sixty or so files at the very root of the Perforce tree, and these would have to be mapped to a directory on disk while using client-view exclusions to prevent mapping everything else in the root to that same directory. For us, it's easier to use the second client spec to map only the files we need in Eclipse--and to map them to directories that are actually links into the full tree.

A Powerful And Flexible Java Dependency Analyzer

2009-09-26T10:27:00.001-07:00

An earlier post evaluated two Java dependency-analysis plugins for Eclipse. Both plugins are very good at providing an interactive, dynamic view of the code, which allows a programmer to explore dependencies and evaluate the impact of changes.

Recently I had a different requirement: generate reports that summarize dependencies, and provide those reports to management in a format that doesn't require having the code or Eclipse available.

A bit of googling brought up Dependency Finder, which turned out to be an excellent tool--scalable, fast, flexible, well-documented, and easy to install. It comes with a set of reports, with numerous options, and it can be customized to produce specialized reports. It also has a GUI, although that wasn't needed for what I'm using it for.

Support is excellent--a question to the author was answered within a few minutes.

Test Utils are now open-source

2009-09-12T15:06:00.000-07:00

At work, we've been encouraged to find ways to open-source technologies that are useful but aren't core to our business.

While writing unit tests over the past several years, I developed some utilities that make it easier to perform some common test tasks, particularly when used with JMockit and EasyMock.

The utilities are now available as open source.

Update: The genius who develops JMockit has added so many features it's not necessary to use reflection to populate fields, so there's no need for TestUtils to be used with JMockit. TestUtils are still useful with other mocking tools, and when you just want to use reflection for something in a test where you aren't using mocking.

Selecting A Coverage Tool

2009-08-30T15:52:00.000-07:00

At work, we use Clover to track coverage.

For programming at home, a Clover license is way too expensive (not as bad as those software vendors that used to not list prices and instead said "Call for quote", but close).

Also, we've had some issues with Clover:

It sometimes falsely claims that code is covered when it isn't.
It doesn't support branch coverage.
It can't instrument assignments in conditionals.

(The Clover rep confirmed the above problems a couple of years ago. These may have been fixed by now. However, because the licenses are so expensive, we haven't upgraded, so we're stuck with the problems even if they're fixed in the current version. Also, the Clover rep didn't think the third issue would ever be fixed due to the way they instrument the code.)

For a simple example of problem #2, start with a function like this:

public boolean eval(boolean x, boolean y, boolean z)
{
return x && (y || z);
}

Clover scores 100% coverage if the entire expression evaluates at least once to true and at least once to false, so this is sufficient:

assertTrue(eval(true, true, false));
assertFalse(eval(true, false, false));

But evaluating all of the meaningfully distinct ways for the expression to evaluate requires more tests:

assertTrue(eval(true, true, false));
assertTrue(eval(true, false, true));
assertFalse(eval(true, false, false));
assertFalse(eval(false, true, false));

Problem #3 means that this code can't be covered:

while (currentLine = stream.readLine() != null)

Covering that code with Clover requires rewriting it as a do/while.

For the above reasons, I went in search of a coverage tool other than Clover for programming at home.

Googling around, EMMA and EclEmma (an Eclipse plugin for EMMA) kept showing up, so I tried them.

Unfortunately, where Clover incorrectly says that uncovered code is covered, EMMA incorrectly says that covered code is not covered:

True, it only screws up occasionally, but if you're five lines short of 100% coverage, and the five lines are spurious tool errors, it's annoying.

This was unfortunate, because the EclEmma plugin is very fast and completely non-intrusive, and would be great to use. Plus, it's currently the only coverage tool that works with Android.

After reporting these problems, I looked around for alternatives. A bunch of coverage tools were listed here, and I tried each of them out. Most didn't install, or weren't compatible with the latest Eclipse, or hadn't been maintained for a number of years, etc.

But Cobertura works really well.

Unfortunately, there's no Eclipse plugin for Cobertura, and it's slower than the other two tools, but compared to getting the wrong answer, that's not too much to give up. The dream tool would combine Cobertura's accuracy with EclEmma's speed and usability, but no such tool exists.

This table compares the three tools:

	Clover	EMMA/EclEmma	Cobertura
Statement coverage	yes	yes	yes
Branch coverage	no	no	yes
No quirks or bugs	no	no	yes
Fast	yes	yes	no
Reports	yes	yes	yes
Maven integration	yes	yes	sort of
Works with mocking tools	yes	yes	yes
Eclipse plugin	yes	yes	sort of
Free	no	yes	yes
Active community	yes	yes	sort of

Notes:

Maven integration for Cobertura is provided by a separate open-source project. This project has been around for a while. There are some problems using the integration.
Eclipse integration for Cobertura is provided by a separate open-source project. This project is new and not very mature yet.
Cobertura's community support is rated "sort of" because the only forum is via an email distribution list, and responses can take a couple of days. On the other hand, the code is mature and easy to use, so there isn't much need for help (most of the emails I sent were questions about setting up the ant scripts).
JMockit can be configured to generate coverage reports, but currently only statement covered is supported. The author plans to add additional features (including branch coverage), at which point it should be evaluated like the other three tools.
For a comparison of coverage tools that arrives at the opposite conclusion to mine (partly due to other requirements), see http://javapulse.net/2008/09/02/coverage-emma-cobertura-maven.

Eclipse Plugins For Java Dependency Analysis

2009-08-25T18:48:00.000-07:00

Once I got our project's code imported into Eclipse in a usable way, the real work started.

The plan is to upgrade the code to use newer APIs and services. To do that, we need a way to analyze the dependencies, to identify cycles, determine the easiest places to refactor, etc.

A web search found a useful evaluation of various dependency-analyzers for Eclipse.

I tried all of the tools listed in the evaluation, and liked STAN the best.

STAN installs easily, is intuitive to use, and doesn't choke on our million-plus lines of code (after configuring it to analyze class-to-class dependencies instead of at the method level). Plus, questions emailed to STAN are answered quickly--the support is good. We bought a license.

But STAN is a commercial product, and I also needed something to use at home. Something inexpensive.

That's when I noticed a comment at the bottom of the evaluation. (It had been there all the time, but who reads comments, right?)

The comment recommends CAP, and it's a good recommendation. CAP is similar to STAN in terms of ease of installation and use, it doesn't choke on our million-line project, and it's faster than STAN and uses less memory. Plus it's free, which is hard to beat.

However, CAP hasn't been upgraded for a while, and the author is intermittently difficult to contact. Also, STAN has a better display of rolled-up package dependencies (for example, if you have com.abc and com.abc.def, you can see dependencies on com.abc, or com.abc.def, or com.abc*, but in CAP you can only see com.abc or com.abc.def individually).

STAN doesn't offer floating licenses, which makes it pretty expensive if more than a couple engineers will be using it. Dependency analysis is something an engineer might do while learning or refactoring a code base, and then not do again for months, so floating licenses would make sense.

Both tools are good. STAN's package roll-ups are really useful when you have a lot of subpackages. CAP's price is hard to beat.

If you do use CAP, please support open source and send the author a donation. I was the first person to do that, which is kind of a shame.

Update: See this post for another good Java dependency analyzer.

Using Eclipse With Large Code Bases, Part IV

2009-08-23T12:25:00.000-07:00

Previous posts described how we created a set of Eclipse projects that break up our million-line, single-rooted source tree into manageable chunks.

Today, I'll describe how we solved a similar problem with our runtime classes:

5. JAXB is used to generate some .class files into a runtime directory "rt", but that same directory contains all of the .class files for the system

Some details on the situation:

Our project currently uses gmake to compile the million-plus lines of code.
Builds output to a runtime directory on the developer's machine called "rt". All classes required to launch the application are either in rt, or in JARs in a "lib" directory.
Many of the classes in the rt directory are duplicates of classes Eclipse compiles in the projects, but some of the classes in the rt directory are generated by JAXB (which is executed by gmake as part of the builds), and are not in the source tree.
Eclipse needs to see the generated classes in order to compile.

For example:

package com.parent.child1;

import com.parent.Parent;

import com.parent.child1.generated.Gen1;

public class Child1 extends Parent
{
   private Gen1 gen1;
}

Eclipse supports linked resources, so at first it seemed like we just needed to define a linked resource for external classes that pointed to rt. Unfortunately, there are so many classes in rt that Eclipse again ground to a halt.

Fortunately, by now we were experienced with using links to subset a source directory hierarchy, so I just used the same approach to subset the runtime directory hierarchy.

First, run commands to add a link into the rt directory at the desired location:

mkdir C:\projects\Child1\rt\com\parent\child1\generated
junction C:\projects\Child1\rt\com\parent\child1\generated C:\rt\com\parent\child1\generated

Then add the rt directory to the .classpath:

<classpathentry kind="lib" path="rt"/>

Refresh in Eclipse, and the code builds:

Notes:

As was the case with source files, multiple links to runtime-class directories can be created for a single project.
If multiple projects need the same .class files from an rt directory, the rt link should be set only in the project that generates the .class files. Other projects should point to that rt link via Build Path → Add Class Folder...

In Part V, I'll describe how we fixed a problem with the Perforce Eclipse plugin caused by using links.

Using Eclipse With Large Code Bases, Part III

2009-08-22T13:25:00.000-07:00

In Part II, we saw how to use links to break a large single-rooted source tree into separate Eclipse projects.

In those examples, the packages didn't have mutual dependencies--they were a directed, acyclic graph.

But in my project's legacy code base, there is another complication:

4. Packages and layers have mutual dependencies (for example, business logic in the UI layer), but Eclipse treats cycles among projects as compile errors

Of course, it would be better not to have cycles, but in a legacy code base it's not always easy, or even tractable, to remove them. Remember that my project has to work with this constraint:

6. None of this can be changed, at least not any time soon

So, a more realistic example has child1 depend on child2, and vice-versa:

package com.parent.child1;

import com.parent.Parent;
import com.parent.child2.Child2;

public class Child1 extends Parent
{
  private Child2 child2;
}

package com.parent.child2;

import com.parent.Parent;
import com.parent.child1.Child1;

public class Child2 extends Parent
{
  private Child1 child1;
}

This is allowed by Java, but not allowed (by default) by Eclipse for packages in separate projects, even after adding the project dependencies:

Fortunately, Eclipse allows cycles to be a warning instead of an error:

Unfortunately, that just converts the errors into warnings, which clutter the window (we already know we have cycles):

Fortunately, Eclipse supports filtering out specific warnings. In the Problems window, click the down-arrow icon on the right (View Menu), select Configure Contents..., and add a filter:

And now the Problems window is empty.

Part IV describes how we fixed this remaining issue:

5. JAXB is used to generate some .class files into a runtime directory "rt", but that same directory contains all of the .class files for the system

Using Eclipse With Large Code Bases, Part II

2009-08-21T18:51:00.000-07:00

In Part I, linked source with excludes failed to solve the problem of how to break up a large, single-rooted source tree into multiple projects in Eclipse.

After a night off, I thought of using file-system links to create the illusion of multiple directory roots. But we needed directory-to-directory links, not file-to-file links, and Windows doesn't directly support those.

A coworker found the Windows-specific "junction" command, which can be downloaded from Microsoft. It's an add-on to Windows, not part of the standard set of shell commands. With the junction command, I was able to create a multi-rooted source tree that points to the source it needs from the single-rooted source tree.

For example, starting with:

C:\src\
  com\
      parent\
          Parent.java
          child1\
              Child1.java
          child2\
              Child.java

Create a parallel structure:

C:\projects\
  Child1\
      .classpath
      .project
      src\
  Child2\
      .classpath
      .project
      src\

The .project files don't have linked-source directives in them, and the .classpath files just have the standard <classpathentry including="**/*.java" kind="src" path="src"/> entries.

From a command prompt, execute commands to create links from the project src directories into the real source tree:

mkdir C:\projects\Child1\src\com\parent\child1
junction C:\projects\Child1\src\com\parent\child1 C:\src\com\parent\child1

mkdir C:\projects\Child2\src\com\parent\child2
junction C:\projects\Child2\src\com\parent\child2 C:\src\com\parent\child2

(The commands for creating directory and file links in linux/Mac are of course different, but the concepts are the same.)

The resulting directory structure looks like this:

C:\projects\
  Child1\
      .classpath
      .project
      src\
          com\
              parent\
                  child1 → linked to
                  C:\src\com\parent\child1
  Child2\
      .classpath
      .project
      src\
          com\
              parent\
                  child2 → linked to
                  C:\src\com\parent\child2

After refreshing the projects in Eclipse, the unwanted packages and source files are gone:

Unfortunately, the code doesn't compile:

Remember the third item in the list of problems?:

3. Some source files used throughout the code are located in the top of the source tree

It has come back to haunt us. We have to have visibility to Parent.java in both projects, but we can't link to the root of the source tree, because that's the problem we're trying to solve with links.

To fix this, create another project, Parent, but use a file link instead of a directory link:

mkdir C:\projects\Parent\src\com\parent
fsutil hardlink create C:\projects\Parent\src\com\parent\Parent.java C:\src\com\parent\Parent.java

Then add the Parent project to the dependencies of Child1 and Child2, and now it does compile:

This approach works very well--the entire million-plus lines of code is broken up into 40+ projects in Eclipse, and the code compiles quickly after the initial import.

You can envision an Eclipse plugin that would semi-automate this process. At a minimum it would be nice to generate the projects and link scripts from some kind of description, instead of editing the files by hand. Unfortunately, by the time I had worked out the pattern, the projects and links were mostly already finished.

Note: Although it's not shown in the examples above, this approach can also be used to link to multiple child nodes in a source tree to produce a combined tree for a project. For example:

mkdir C:\projects\Child1\src\com\parent\child1
junction C:\projects\Child1\src\com\parent\child1 C:\src\com\parent\child1

mkdir C:\projects\Child1\src\com\parent\otherChild
junction C:\projects\Child1\src\com\parent\otherChild C:\src\com\parent\otherChild

In Part III, I'll describe how we dealt with this issue:

4. Packages and layers have mutual dependencies (for example, business logic in the UI layer), but Eclipse treats cycles among projects as compile errors

Using Eclipse With Large Code Bases, Part I

2009-08-20T19:25:00.000-07:00

The project I'm currently working on has more than a million lines of source code. Some of the code was written as long ago as 1998, so as odd as it sounds to call anything involving Java "legacy", this is a legacy Java codebase.

I wanted to bring the code into Eclipse, but not as one giant million-line project. Instead, I wanted to break it up into smaller projects.

But there were complications:

The source tree has a single root directory
Eclipse can't nest projects
Some source files used throughout the code are located in the top of the source tree
Packages and layers have mutual dependencies (for example, business logic in the UI layer), but Eclipse treats cycles among projects as compile errors
JAXB is used to generate some .class files into a runtime directory "rt", but that same directory contains all of the .class files for the system
None of this can be changed, at least not any time soon

Items #1 and #2 mean that the Eclipse projects have to be located outside the source tree, and point to source in the source tree.

Fortunately, Eclipse supports linked source, so I created the Eclipse projects in a different location, and set their build paths to have linked-source entries that pointed to the source tree.

The first step is to create a global linked-resource variable that points to the root of the source tree:

Then right-click on each project and select Build Path → Configure Build Path... → Link Source... → Variables..., and select the global linked-resource variable.

Saving the changes results in a .classpath entry like this:

<classpathentry kind="src" path="src"/>

and a .project entry like this:

<linkedResources>
   <link>
       <name>src</name>
       <type>2</type>
       <locationURI>src</locationURI>
   </link>
</linkedResources>

for each project.

(Because I had so many projects to manage, I edited the .project files directly, instead of interactively.)

Unfortunately, item #1 complicated linking to the source, because every Eclipse project linked to the same root directory, which meant every Eclipse project saw the same source instead of just seeing the source for that project.

For example, if the source tree looks like:

C:\src\
   com\
       parent\
           Parent.java
               child1\
                   Child1.java
               child2\
                   Child.java

and we want two Eclipse projects, "Child1" and "Child2", they both have to start their source trees at C:\src. So Child1 sees Child2's code in the child2\ directory, and Child2 sees Child1's code in the child1\ directory:

Fortunately, Eclipse supports source exclusion. To exclude source, right-click on a project and select Build Path → Configure Build Path... → Source, Excluded → Edit... → Exclusion Patterns: → Add..., and add every package and/or file you don't want included.

This modifies the .classpath files to have entries like:

<classpathentry kind="src" path="src"
excluding="com/parent/child1/"/>

Using this approach, I was able to exclude unwanted packages from each project. Because there are a lot of packages, this was tedious and took hours, but it worked:

Unfortunately, once everything was configured and I launched Eclipse, it took 35 minutes to load.

35. Minutes.

It turned out that Eclipse bogs down if it has to import a lot of source code and then filter it out. Ideally it would filter it out first and only load the remainder, but it doesn't seem to do that.

After reporting this problem, I switched to plan B, which is described in Part II.