To TDD, or not to TDD?

Summary: Some thoughts on Test-Driven Development (TDD).

A few weeks ago, Stack Overflow co-founders Joel Spolsky and Jeff Atwood discussed criticized certain aspects of Test-Driven Development (TDD) and SOLID Principles of Object-Oriented Design (OOD) [the latter advocated by Robert C. Martin, AKA Uncle Bob]. Check out the excerpts from the original podcast and the follow-up talk (you may also want to listen to the Scott Hanselman's interview with Uncle Bob, which prompted Joel's rant):

• The Stack Overflow: Podcast #38, Excerpt (14:32) [see also podcast page and transcript]
• The Stack Overflow: Podcast #39, Excerpt (18:35) [see also podcast page and transcript]

As expected, the discussion criticism triggered rather emotional rebuttals from TDD supporters, including Uncle Bob himself; see (and read comments):

• "Quality Doesn't Matter That Much" -- Jeff and Joel
• An Open Letter to Joel Spolsky and Jeff Atwood

And then, a counter-rebuttals by Jeff Atwood:

• The Ferengi Programmer
• Real Ultimate Programming Power

Eventually, Joel, Jeff, and Uncle Bob met for a makeup session, which you can find -- along with the summary -- at:

• The Stack Overflow: Podcast #41

Although all participants of the podcast #41 seemed to agree on many issues, I have a feeling that the opposing sides stuck to their points of disagreement: uncle Bob remained convinced that TDD could do only good (the more TDD, the more good), while Jeff and Joel continued emphasizing that other aspects of software development may be more important than TDD.

I happen to lean more towards Jeff and Joel for the following reasons (in addition to the reasons mentioned in The Stack Overflow podcasts).

First, Joel mentioned the issues related to GUI testing, but it's not just the GUI: the database layer poses another challenge. When unit testing database-driven apps, a common approach is to rely on mock objects (in which case, you totally skip database testing) or use test-specific data in a simplified version of the database (here you will have to figure out how to maintain this database and integrate it into your build process). Even with the 100% code coverage of the business layer, if your unit test skip the GUI and database, about half of the application (more or less) will remain untested, and you'll have to address this gap using functional testing, which will overlap with unit tests. So why spend time writing unit test for the code that will be tested functionally anyway?

Second, claims that TDD improves code quality are rarely, if ever, substantiated by data. You may hear statements like:

"Since I (we) started using TDD, my (our) code has become much better."

I'm not claiming that these statements are false, but without data they are just personal opinions. Even assuming that code did in fact improve, could it be due to other factors, such as better programming experience, tools, team composition, processes?

Analyzing effects of TDD on code quality is extremely difficult. One of the better studies on the subject was conducted by Microsoft and IBM (see the 8-minute interview with one of the researchers and the original paper). The study found that TDD improved code quality, but it also increased development time.

There is a trade-off: either use TDD and spend more time writing, maintaining, and running unit tests now or do not use TDD and spend time fixing bugs later. The question is: how much time? If TDD activities take 2 weeks out of an 8-week development cycle (a typical 25% TDD penalty) to prevent defects that could've been found and fixed in 3 days during functional testing, would TDD give you the best return on investment? By the way, it is worth noting that the team with the smallest test coverage (62%) in the study achieved the best results (90% drop of defects), while the team with the highest test coverage (95%) achieved the "worst" (as compared to other teams) results (40% drop of defects; still not bad when compared to the control group). I would also emphasize that the use of TDD in the study was close to ideal: the projects were appropriate for TDD and the teams were not pushed to achieve the maximum code coverage.

The Microsoft-IBM study had a few problems, though. First, the number of participants was rather small. Second, and more important, it tried to compare similar teams and projects, but there were many difference between them, such as team/project sizes, and project durations, which is expected in a study focusing on real life projects. Studies performed in academic settings -- with undergraduate students of approximately the same grade working on identical projects -- attempted to minimize some of the differences and make experiments more controlled. According to an abstract from one of such studies:

"[T]est-first students on average wrote more tests and, in turn, students who wrote more tests tended to be more productive. We also observed that the minimum quality increased linearly with the number of programmer tests, independent of the development strategy employed."

How this sounds depends on how you interpret the study's findings. If you're a TDD supporter, you may join Phil Haack and announce that Research Supports The Effectiveness of TDD. Or, you can look more closely at the results of the study and come to a different conclusion. For example, Jacob Proffitt suggested (see also post comments) that:

"[W]ithout question, testing first leads to having more tests per functional unit. The question is if this is valuable. This study would seem to indicate that this is probably not the case, at least if quality is your intended gain. But then, I'm not that surprised that number of tests doesn't correspond to quality just as I'm not surprised that the number of lines of code doesn't correspond to productivity."

I would also point that the code in the experiments was not written for database-driven apps with web-based GUI, so again it was an ideal usage of TDD. So I would speculate that if either of these studies focused on database-driven apps with web front ends, the results would probably be even less encouraging.

I'm not an opponent of TDD and agree that it can be valuable when applied to certain types of projects. My gripe against TDD is that it is being often sold as yet another silver bullet and forced into adoption indiscriminately, in the projects where TDD offers minimal, if any, benefits.

I also resent the assumption that TDD always leads to good code. Your My project can have 100% code coverage, but it can still suck. It will pass all unit test, alright, but at the same time it can be unreadable, unnecessarily complex, uncommented, resulting in an app with bad GUI and unintuitive behavior, an app which both the users and support teams hate. But don't you dare to criticize my code: since it's 100% unit tested, it's good by definition. Usability testing? Never heard of it. And as far as everything else goes (like comments, better design, etc), do I have time for this when I spend my better hours working on unit test? Something's gotta give.

Finally, what about all those great apps written by developers, who did not use TDD or any other silver-bullet-like methodology? Take Google developers, who apparently do not follow any software development methodology or approach, but somehow manage to write apps that people love. If they can write great apps (which is one of the goals of good code), why would one want to impose TDD on them?

Now, Google developers are good, but what do you do if your developers are bad? [I'm not talking about developers who make occasional, sometimes serious, errors (this happens even to the best of us); I'm talking about developers who consistently write bad code.] Wouldn't TDD help them write better code? I wouldn't expect it, but if you can share a success story, please leave a comment.

If your organization suffers from bad code, consider adopting TDD (when applied correcly, it may help), but more importantly, consider other factors, such as:

• The Joel Test: 12 Steps to Better Code by Joel Spolsky
• What Makes Good Code Good? by Paul DiLascia

Once you the adopt common-sense practices, you may not need TDD after all.

See also:
Hanselminutes Podcast 146 - Test Driven Development is Design - The Last Word on TDD
Hanselminutes Podcast 31 - Test Driven Development
TDD Tests are not Unit Tests by Stephen Walther
Test-After Development is not Test-Driven Development by Stephen Walther

Real men don't need PowerPoint

Summary: Steve Yegge explains the importance of branding at Open Source Convention 2007. And he ain't got no PowerPoint!

Imagine yourself invited to speak at a technical conference. You do your homework: pick a topic, build the slides, bring a laptop. You are ready to go, but once you get on stage, you realize that the projector does not work and tech support guys can't fix it. What do you do?

As Steve Yegge showed at OSCON 2007, you do not need PowerPoint to give an interesting speech. Sure, a couple of slides would be helpful, but they were not essential. If you want to hear Steve's perspective on the importance of branding (branding is not only relevant to major corporate brands, such as Coca-Cola, but equally applies to the work of your development team), or if you are interested in learning how other corporations handle justifiably bad brand perceptions (so that you know what to expect on the next screw-up), watch this 25-minute video (if the link does not work, try this one). As an introduction, you can also read Steve's How To Make a Funny Talk Title Without Using The Word "Weasel" post.

Additional references:
How I made my presentations a little better

On iPhone, Paris Hilton, Six Stupid, and more

Summary: This week's reading recommendations focused on processes and other topics.

Today is the iPhone day. Am I the only one who noticed that iPhone had turned into Paris Hilton of the technology world? Haven't we had enough of both? Really,

"Hitler got less coverage when he invaded Poland."

Thank you John Dvorak. And thank you Mika Brzezinski. Now, about things that matter.

Bob Lewis, the publisher of IS Survivor and president of IT Catalist, just discovered a new methodology called Six Stupid:

"Everyone knows that a group of people is dumber than its least intelligent member. Six Stupid is based on this insight. Unlike the better-known Six Sigma, Six Stupid requires the collaboration of at least six idiots, to design process flows that defy reason and preclude exceptions."

It's unfortunate that many Six Stupid processes (methodologies, rules, etc) start as great ideas, but at some point get overtaken by zealots and eventually turn into entities that exist mostly for the goals of their own. Instead of helping, they become obstacles. Instead of reducing time and effort, they require more time and effort. To make sure your process does not turn into Six Stupid, remember:

"Turning a new process on does not justify employees turning their brains off."

Now, here is a process from Google described in Life at Google - The Microsoftie Perspective, which is definitely not Six Stupid:

"Google has the concept of "Tech Stops." Each floor of each building has one. They handle all of the IT stuff for employees in the building including troubleshooting networks, machines, etc. If you’re having a problem you just walk into a Tech Stop and someone will fix it. [...] If your laptop breaks you bring it to a Tech Stop and they fix it or give you another one (they move your data for you). If one of your test machines is old and crusty you bring it to the Tech Stop and they give you a new one. [...] If you need more equipment than your job description allows, your manager just needs to approve the action."

As far as I know this process is not patented. Someone at Intel IT management chain, please give John Johnson a call.

And here is another process: hiring -- or should I say "non-hiring" -- of technology workers in the US. In this video, Lawrence Lebowitz, vice president and director of marketing for the Cohen & Grigsby law firm, explains how employers can hire cheap foreign workers while following the legal process imposed by the U.S. government that is inteneded to guarantee that no qualified U.S. worker is left behind. According to Lebowitz,

"[T]he goal [of this process] is clearly not to find a qualified and interested U.S. worker."

It's not stupid, but it's so wrong, so deviant... And what were these guys thinking when they were shooting the video. OK, maybe it's One Stupid. By the way, now I understand the purpose of the job advertisements placed in professional magazines.

Well, some processes directed against us have antidotes. In case you're shopping for a Dell laptop, read 22 Confessions Of A Former Dell Sales Manager. Smart, uh! Dell's reaction on the other hand was not so smart.

Finally, Dare Obasanjo (AKA Carnage4Life), a son of the president of Nigeria and a current Microsoft employee, described a meeting with Bill Gates, who was planning a trip to Nigeria, in a blog post, which I recommend to readers. You can learn a few things about Bill Gates and Obasanjo (the president): how he became a president after a military coup in 1976, served one term, held free elections, was jailed, was elected again, became a born-again Christian... What a life! The last part of the post made me laugh. As Obasanjo was leaving the building, he overheard the following exchange between the receptionist of the building and a visitor:

Visitor: Where is Bill Gates's office?
Receptionist: I'm not at liberty to divulge that information.
Visitor: I need to see him, I just downloaded Windows Vista and I have a number of complaints.

Enjoy the weekend, and please don't mention iPhone (or Paris Hilton) to me.