Monday, May 31, 2010

Open Cubicles Must Die


"The cynic knows the price of everything and the value of nothing." - Oscar Wilde
In 1987, Tom DeMarco and Timothy Lister published the first edition of "Peopleware: Productive Projects and Teams".

The book discusses many reasons software projects fail, and provides numerous recommendations for making projects succeed.

Part II of the book discusses the effect a bad office environment has on productivity, and recommends (among other suggestions) private offices as a significant productivity enhancer. The book provides quantitative evidence to substantiate the recommendation. And the book eviscerates the productivity claims of cubicleware vendors, which are shown to have no substantiation (the book calls those claims "proof by repeated assertion").

Fittingly, the book is out of print. (Corrected: The same material is now available in "Peopleware: Productive Projects and Teams, 3rd Edition".)

If anything, in the decades since the book was published, things have gotten worse (other than getting rid of P.A. systems). We are in the grip of an industry-wide pathology that produces expensive, flow-disrupting buildings.

How did we get here?

A lot of it has a lot to do with the counting of beans.

It is easy for a bean counter to count tangible things, like square feet.

It is not easy for a bean counter to count intangible things, like lost productivity due to a noisy work environment yanking programmers out of flow.

Suppose at some point a bean counter notices that their company spends more for facilities than "comparable" companies spend. This seems to the bean counter like fiduciary irresponsibility, so the bean counter launches an investigation.

Because the bean counter cannot count intangibles, the bean counter does a poor job of counting beans, and works up an Excel spreadsheet that shows how, if they get rid of walls and doors, and reduce the space each programmer gets, and eliminate bookshelves, and generally pack programmers into "pods" where they're practically sitting in each other's laps, the company will save a bundle.

On paper at least.

Intrigued by the putative savings to be had, the exec staff--sitting in their capacious, private, quiet offices--decides to engage the services of a cubicleware vendor to "work up" some designs for how the building could look. (Note that open cubicles are by this point a foregone conclusion, and that by involving a cubicleware vendor as the consultant, we have an instance of bid rigging so egregious that if this was a government contract somebody would go to jail.)

The designs are very slick. Spiffy in color, appealing in the abstract, the revamped building will "facilitate collaboration" (and who would question the need for that?), foster "inter-departmental serendipity" (which also sounds desirable, whatever that is), and so forth.

Nobody asks the programmers for their input. Or they do ask, but when the programmers say they want private offices so they can focus on work, this is dismissed as "impractical", even though drywall is cheaper than cubicleware.

The building is revamped, the programmers move into it, and are pissed off because they feel like they're regarded as interchangeable, revenue-producing, chunks of meat. Morale drops and productivity goes into a tailspin.

Adding insult to injury, the company newsletter makes a big deal about how the building, in a triumph of form over function, won an AIA award for "design excellence".

Those programmers who stick around find ways to circumvent the limitations of the building. For example, they take refuge in the server room (lots of white noise), or work at home.

But these strategies ultimately backfire, because the bean counter's minions notice that a lot of cubicles--the ones occupied by programmers hiding in the server room or working at home--are empty, and conclude that the building was in fact overprovisioned--that there could have been smaller cubicles, and fewer of them. Inevitably, the bean counter concludes that cubicles shouldn't even be allocated to individual programmers, but instead should come from a pool, first-come/first-served.

Because programmers are now sharing keyboards and mice, they share flu germs, and occupancy rates go down again. As does productivity.

To make up for the loss of productivity, the company needs to hire more programmers. But where to put them? Clearly cubicles take up too much room. Programmers should just work on long tables, like medieval craftsmen.

Meanwhile, the aggressive reduction in facilities costs is noticed by other companies' bean counters, who don't want to look like spendthrifts based on the latest "comparables".

And so continues the industry-wide death spiral.

Another reason we have open cubicles is due to differences between how programmers work and how the execs making the decisions work.

Execs are people people. They like to meet new people. They like to shake hands. They like conversation. They're extroverts.

Cubicleware vendors are also people people. They're in sales, so of course they're extroverts.

Programmers are, well, if you're a programmer you know what we are. It's not that we're necessarily ugly, or antisocial, but we're a lot more likely to send an IM to somebody 10 feet away than to get up and walk over there to talk. We like to get into flow and stay there. We like to solve problems.

When people people design a work environment, they design it how they like to work. Which involves a lot of face-to-face interaction with other people, long phone conversations, open airy spaces, etc. It's almost the exact opposite of the conditions that foster flow.

A few companies ago, Lisa was in a cube farm at a Fortune 50 company, concentrating, when some ninny of a people person walked into the area and, apparently discomfited by the unfamiliar sound of work getting done, said: "There isn't enough visitin' going on".

Visitin'.

Honestly?

Another reason we have open cubicles is that private offices are seen as a "perk". But they're not a perk, they're a software-development tool, and an effective one. Programmers don't care about status or prestige. They care about being effective. And they need good tools to do that.

Unfortunately, many execs do care about status and prestige, and they see private offices as a perk. It's a problem of semiotics.

Another reason we have open cubicles--and this is a self-inflicted wound--is that some of the newer methodologies emphasize collaboration, pair programming, etc.

But one person's collaboration is another person's pointless interruption.

When people are yakking around a programmer who's trying to get into flow or stay in flow, that constitutes involuntary collaboration. It's like being on an airplane before they banned smoking--there was no way to escape.

With private offices, programmers can always opt into collaboration by opening their door, sitting down in a communal work area, etc.

Without private offices, programmers can't opt out of collaboration. Which is a shame, because at some point, somebody, somehow, has to actually write some code.

But, you say, the programmer can always put on headphones. Yes, they can, except that page 78 of the second edition of Peopleware explains why that's a bad idea (a Cornell study showing that creativity is hampered by listening to music while programming).

The saddest part of all of this is that if companies would focus on maximizing programmer productivity instead of on minimizing facilities costs, they could accomplish a lot more work with far fewer programmers, which would save so much money in salaries it would dwarf the incremental cost of private offices.

Penny-wise, pound foolish.

Not all companies make this mistake. For example, when Adobe was building its headquarters in San Jose, they asked the programmers what they wanted, and then did something crazy: they listened. (Guess what the programmers wanted.)

Sometimes I get the argument that young programmers prefer open cubicles and more collaboration, but how would young programmers know? Did a lot of them previously work in large, quiet, private offices? Doubtful--cubiclitis has been infecting the industry since the 1960s, which is longer than young programmers have been working (indeed, longer than many of them have been alive).

Claiming that young programmers prefer it this way implies they ever had a choice. But they didn't have a choice, so there's no control group, and this is just more self-serving nonsense.

I'll close with a visit a few companies ago to one of the largest cubicleware vendors in Silicon Valley. We went there because we were opening up a development center, and needed furniture. The vendor took us on a tour of their facility, which doubled as a showroom and their headquarters. As the tour lady stood there telling us about the merits of this or that piece of furniture, I noticed some employees of the vendor in the background, in the accounting department.

They were glaring at us, because we were making too much noise.

Some interesting articles about or related to this topic:
Update: Look how far we have fallen--a friend sent: "In the '60's my father took me through his insurance company offices after hours. Most of the desks were in open bays. Then we came to the computer area and he pointed out the leased IBM 360/30. Nearby there were several special rooms along the wall with doors and glass walls. I asked why some of the desks were sealed off like that and he told me that the programmers needed quiet places to work without interruption because it was so important that their programs be correct. Your article brought back that memory."

Update: http://www.nytimes.com/2012/05/20/science/when-buzz-at-your-cubicle-is-too-loud-for-work.html

Update:


Update: http://online.wsj.com/article/SB10001424127887324123004579057212505053076.html

Update:

Update:
The evidence continues to pile up, not that I expect it to change anyone's mind just because of the facts:
http://www.newyorker.com/online/blogs/currency/2014/01/the-open-office-trap.html

Update:


Update:

https://hbr.org/2014/01/to-raise-productivity-let-more-employees-work-from-home

Update:

http://www.ted.com/talks/jason_fried_why_work_doesn_t_happen_at_work

Update:

http://www.businessinsider.com/noise-hurts-productivity-of-millennials-2016-6

Regression:

http://money.cnn.com/2017/05/19/technology/ibm-work-at-home/index.html

White elephant:

https://medium.com/make-better-software/apple-is-about-to-do-something-their-programmers-definitely-dont-want-fc19f5f4487

Scrum Is A Flow-state Interrupter

Overall, Scrum is useful.

But the Daily Interruption is an abomination.

First of all, because the Daily Interruption is supposed to be held every workday, it interferes with no-interruptions days.

Second, the Daily Interruption serves no purpose.

I realize this is heresy, but look at what is supposed to happen in the Daily Interruption. Every team participant is supposed to answer three questions:
  1. What have you done since yesterday?
  2. What are you planning to do today?
  3. Is anything blocking you?
The first two questions can and should be replaced by emailed status reports. Why waste time telling me something I can read (if I care, which I don't)?

The third question is even more asinine. Why would a programmer wait an average of 12 hours to bring up something that's blocking them? Shouldn't they deal with the problem right away?

If someone comes to a Daily Interruption to kvetch about a blocker, my question to them is always why haven't you dealt with this already, dummy.

Because the Daily Interruption serves no purpose, requiring programmers to attend it is fiduciary irresponsibility.

The Daily Interruption was invented to solve communication problems on dysfunctional teams. If a team is not dysfunctional, the Daily Interruption is make-work. Wasteful, flow-killing make-work.

But just try convincing a ScrumMaster that the team can do without the Daily Interruption. They've been to ScrumSchool, you see. They're ScrumCertified.

Except they didn't read the whole book. If they had, they'd know that Agile methodologies (of which Scrum is an instance) support Method Tailoring. And Method Tailoring allows a team to modify a methodology to fit the needs of the team and the project the team is working on.

In other words, a team can, with the blessing of Agile, tailor Scrum to eliminate the Daily Interruption.

As a compromise, if a team simply must have the Daily Interruption, at least only have it on the three days of the week that are already contaminated by other meetings. Leave the no-interruptions days alone.

Because our team has enlightened management, the Daily Interruption is not scheduled on no-interruptions days, and attendance is optional on the other three days.

Mandatory Meetings

Unless required by law, there should be no such thing as a mandatory meeting.

The best way to get people to attend a meeting is to make the meeting worth attending.

Mandatory meetings are particularly annoying when they're supposed to be "fun" (for example, a team-building offsite). Mandatory fun is an oxymoron.

Feedback On Non-recurring Meetings

Because meetings are flow-state interrupters, companies should have a process for improving the quality of all meetings, including non-recurring meetings.

After every non-recurring meeting, a survey should be sent to everyone who attended the meeting, asking some questions:
  • Did the meeting start on time?
  • Did the meeting end on time?
  • Do you understand the purpose of this meeting?
  • Do you feel that this meeting met its stated purpose?
  • Based on the average fully burdened salary of the attendees, the cost of this meeting (not counting opportunity cost) was $<put cost here>. Do you think that cost was justified?
  • Is there any information communicated during the meeting that should be communicated by some other means (for example, in email)?
  • Which of the following actions would have improved the meeting (check as many as apply)?:
    • Add ______ to the agenda.
    • Remove ______ from the agenda.
    • Change the allocated time to ______.
    • Reduce the number of attendees.
Meeting ratings should be considered in employee's annual reviews.

Winnowing Recurring Meetings

Whenever someone takes over an IT department, the first thing they're supposed to do is cancel all reports.

The thinking is that many reports are leftovers nobody cares about anymore, and shutting them off is the best way to find out which ones matter to someone.

Have you ever found yourself in a recurring meeting where you wonder what the point of the meeting is?

That meeting is like a zombie report. Nobody really needs it anymore, but there's no process in place for getting rid of the meeting. It just lurches along on inertia, wasting everyone's time.

Because meetings are flow-state interrupters, companies should be proactive about getting rid of recurring meetings nobody needs anymore.

Every recurring meeting should have to interview for its job on an ongoing basis.

After every recurring meeting--or at least quarterly--a survey should be sent to everyone who attended the meeting, asking some questions:
  • Did the meeting start on time?
  • Did the meeting end on time?
  • Do you understand the purpose of this meeting?
  • Do you feel that this meeting met its stated purpose?
  • Based on the average fully burdened salary of the attendees, the cost of this meeting (not counting opportunity cost) was $<put cost here>. Do you think that cost was justified?
  • If you do not feel this meeting is justified, do you feel that it should simply be cancelled, or improved?
  • If you feel the meeting should be cancelled, is there any information communicated during the meeting that should be communicated by some other means (for example, in email)?
  • If you do not feel that this meeting should be cancelled but instead should be improved, which of the following actions do you think should be taken to improve it (check as many as apply)?:
    • Add ______ to the agenda.
    • Remove ______ from the agenda.
    • Change the allocated time to ______.
    • Hold it less often.
    • Make it non-recurring (only hold it when needed).
    • Reduce the number of attendees.
    • Combine it with the ______ meeting.
    • Reschedule it to ______ to leave a larger contiguous block of uninterrupted time available.
If a meeting fails the interview, it should be cancelled, no matter how important the meeting owner thinks the meeting is.

Constant review and winnowing of meetings can, over time, convert this:to this:If a meeting passes the interview but needs improvement, it should be improved per the survey feedback.

Meeting ratings should be considered in employee's annual reviews.

A Huge Gain In Productivity For A Tiny Amount Of Effort

Does this look familiar to you?:If so, there's no way you can get into flow during work hours. You either don't get into flow (bad), or work evenings and weekends to get anything done (also bad).

Fortunately, there is a simple, free, and ridiculously effective solution to this problem.

Working with management, institute two no-interruptions days per week.

Stagger the two days to avoid being out of contact with others for two days in a row. (I recommend Tuesday and Thursday, but Tuesday and Friday also work well.)

Make the two days sacrosanct.

Have everyone on your team reserve the two days on their calendars:Make your calendars public so everyone can clearly see the two days blocked out.

Politely decline all meeting invites on the two days, with the explanation that the two days are for focused work, and suggest scheduling the meeting on one of the three non-focus days that can be wasted playing office.

Be firm.

And also be firm with yourself. The two days are not for running personal errands, goofing off, etc. The two days are for focused working. If you screw this up, management will consider the experiment a failure, you'll lose credibility, and it will be hard to convince them to remove other, more difficult obstacles.

On the two days, shut off IM, Twitter, RSS, your cellphone, etc. It will be hard to do that at first. You'll feel cut off, out of the loop, going through withdrawal.

Good. That means it's working.

Ideally, assuming you have a good home-office work environment, work at home on the two days. That way you can roll out of bed, grab something quick to eat, and start coding.

Be sure to put in a full eight hours of focused time on each of the two days. (You will find this easy to do, because you'll be in flow.)

Congratulations, you just worked your 16 hours/week! You are 33% more efficient than the industry average. If you manage to work a few more hours on the other three days of the week, that's all upside.

Some people can get really offended that you declined their super-important meeting. Boo hoo. This is when it's vital to have management backing, so you can send the invitee to management for goal alignment. You need management to make it clear that interrupting a programmer on a no-interruptions day is stealing from the company.

Because you will actually be able to clock 16+ hours a week with this system, management will give you the backing you need.

You also need management backing if you work across teams, so that the no-interruptions days can be aligned. (It does no good to declare Tuesday as a no-interruptions day if a team you have to work with picks a different day.)

If you can't get management backing for this and/or if upper management is for it but your own manager is against it, change jobs. Seriously.

How dogmatic should you be about no interruptions on those days? Well, there's sort of an art to getting it correct. Obviously if you are blocked because you have to ask somebody else a question, it would be nice to be able to get an answer right away. But it's a slippery slope. You have to be careful to only interrupt when it's really important. And you should never, ever, accept an invite for a recurring meeting on a no-interruptions day.

The irony of this system is that it costs the company nothing to implement. Mostly it just involves changing the days for some meetings. A simple, mechanical process.

Flow Versus "New Cognition"

There has been some speculation online that kids these days aren't actually interrupted all the time and incapable of concentrating, but are instead just better at multitasking, and are in effect changing the way people think--that we're witnessing the emergence of a new kind of cognition. Evidence for this includes measurable changes to brain structures from playing lots of video games, and so forth. Some posts have gone so far as to say that because of this new cognitive style, kids shouldn't have to learn to spell, because good spelling isn't required for online communication. And of course anyone who disagrees is an old fogey who just can't keep up. ADHD isn't a disease, it's a New Way Of Thinking.

Uh huh.

Andrew Wiles spent seven years secluded in an attic. When he emerged, he had proven Fermat's Last Theorem as a side-effect of proving a much more general and profound result.

Now suppose he had been one of these New Cognition types. How far do you suppose he'd have gotten on that proof?

I think he'd still be in the attic, banging away on his Wii.

As an old fogey who can spend hours at a time in flow, I'm worried that kids will grow up never entering a state of flow at work, and the cost to humanity in lost productivity will be gigantic. Our best and brightest will constantly thrash, never achieving their potential.

They'll be totally pwnd.

Fighting For Flow

"And George, while his intelligence was way above normal, had a little mental handicap radio in his ear. He was required by law to wear it at all times. It was tuned to a government transmitter. Every twenty seconds or so, the transmitter would send out some sharp noise to keep people like George from taking unfair advantage of their brains." - Harrison Bergeron, by Kurt Vonnegut

Flow is the highest-productivity mental state of which humans are capable.

A person operating in flow can be vastly more productive than a person who is not in flow.

A person in flow often starts working in the morning when it is light, and snaps out of flow late at night when it is dark, with no sense of the passage of time.

A person in flow sometimes forgets to eat.

A programmer in flow can do a week's worth of work in an afternoon, and a month's worth of work in a couple of days.

A company that makes it easy for programmers to work in a state of flow has a strategic advantage over companies that have not figured this out.

A programmer who learns to maintain a state of flow has a career advantage over programmers who have not learned how to do this.

Unfortunately, it is becoming increasingly difficult to maintain a state of flow.

Some of this is the fault of programmers. It is easy to be seduced by email, IM, Twitter, etc.

Fortunately, this can be rectified by self-discipline. Shut down email, log out of IM and Twitter, turn off the cellphone, and focus.

But companies also contribute to the problem. Perversely, they seem to go out of their way to squander programmer productivity. Instead of uninterrupted blocks of time, days are broken up with meetings. Instead of quiet offices, they stick programmers in noisy "pods". Instead of encouraging working at home, they disproportionally terminate remote workers during layoffs (then wonder why so few employees become remote workers).

It can become pathological. Several companies ago, a colleague took two weeks of vacation so he could come to work every day and focus on something he'd been trying to get done for over a year. If anyone stopped by to ask him something, he'd tell them he was on vacation, and tell them to go away.

I didn't make that up.

Programmers need to fight back. It's a matter of self-preservation.

Just complaining won't change anything. You'll just be seen as a "troublemaker".

But there is one stick big enough to beat management over the head with to get them to listen.

Money.

Programmers cost money. They cost more money than anything else in the budget. A Silicon Valley programmer, fully burdened, can cost $250k/year.

Wasting programmer productivity is fiduciary irresponsibility.

Mechanisms that preserve programmer productivity deliver tangible benefits to the bottom line.

The industry average for under-the-headphones actual coding--and this is criminal--is only about 12 hours/week. Not per day. Per. Week.

That's crazy.

There are 168 hours in a week.

Where do the rest of the hours in the week go? WoW? Air hockey? American Idol?

Now suppose instead you can eke out a whopping 16 hours/week. That doesn't seem like much, but it's a 33% improvement. It's like being able to hire 33% more programmers without spending another penny. Think what the company could do with that much additional engineering resource! They could deliver more features. They could fix more bugs. They could crush their competition.

But it's not even just a 33% improvement, because you're shooting for achieving 16 hours a week of hours in flow, so the productivity gains are much higher than a mere tally of hours indicates. Both the number of hours and the quality of hours increase. It's a force multiplier.

Now you're talking a language management understands.

So go to management, and make the business case. Tell them you want to help the company be more successful by removing obstacles to programmer productivity. List the obstacles, and suggest how each could be eliminated.

Start with something small, so their heads don't explode with concerns about "risk". I recommend starting with this.

Then when you show that the small change resulted in a measurable improvement, move to the next obstacle.

Your team will be vastly more productive than teams that still suffer from the obstacles. Other managers will start to panic and want to know what the "secret" is, and start removing obstacles as well. Management will stop muttering about reducing costs by "offshoring" (because you found a way to reduce costs while keeping the jobs local).

Related: http://www.paulgraham.com/makersschedule.html

Sunday, May 30, 2010

The Joy Of Legacy Code

The project I'm currently working on consists of more than a million lines of legacy code.

Every other project I worked on was a "greenfield" project--a blank sheet of paper.

It took a while to adjust.

I'd always thought of legacy code as something that dull people maintained in the basements of banks.

I was an idiot.

In fact, legacy code is important code. If it wasn't important, a company would simply decommission it, but instead a company continues to fund its development (or at least its maintenance), despite all the problems the programmers who work on the code complain about.

Legacy code is virtually bug-free code, at least on the happy path. If it wasn't virtually bug-free, end users would complain and/or the code would crash. But it doesn't do that. Instead, it hums merrily away, the bugs having been dispatched years ago.

Working on legacy code, you don't have to worry about the project getting cancelled, the start-up running out of money, etc. You already have customers, and you already have revenue.

Legacy code pays well. A company can find millions of programmers who can write hello world, but not many programmers can do things like:
  • Migrate a legacy database to a non-backward-compatible schema without impacting end users
  • Revamp a legacy user interface to use Flex
  • Convert a legacy build system to Maven
  • Replace homegrown legacy infrastructure components with open-source libraries
  • Refactor unstructured legacy code into layers
Those are hard problems, like cutting a diamond without shattering it. In comparison, writing code from scratch is trivial.

Besides, new code is just legacy code in waiting. Assuming some new code proves useful and survives, it eventually turns into legacy code. Yesterday's spiffy greenfield project is today's legacy codebase.

In most contexts, the word "legacy" has a positive connotation: freedom is the Founding Fathers' legacy; the National Park System is Teddy Roosevelt's legacy; prudent saving and investing leaves a legacy for one's children to inherit. This should be true for legacy code as well.

Consider yourself lucky if you get an opportunity to work on legacy code!