Jurgen Appelo on Agile and more

Summary: Some interesting stuff from Jurgen Appelo.

Jurgen Apello just published a new quarterly list of the Top 100 Blogs for Developers. I'm glad that Jurgen's own blog made the list because it's gooood. Here are some quotes from a few recent articles:

From AINO: Agile In Name Only:

"I want to suggest the acronym AINO (Agile In Name Only) for those cases where there is a significant gap between preaching and practising the agile approach."

From The Complex Manifesto for Software Development:

"I believe it is time for a new manifesto. A manifesto of complexity. With this manifesto we must recognize that it is human to prefer simple solutions, but also that the world is more complex than we think."

From The Decline and Fall of Agilists:

"Agility is about staying successful in ever-changing environments. That's it. There's nothing more to it. This includes defying any expert who claims that practice X is essential for your organization. Because, quite likely, practice X happens to be something this expert is very good at, and is willing to assist you with, for a considerable consultancy fee."

While we're on the topic of Agile development, here is a very good and brief interview with Jurgen Appelo, where he discuss his experiences of introducing and using Scrum and the curse of dogmatic use of "recipes for success" in software development:


On a lighter note, here is some humor:

From Thank You, Stupid Americans (WARNING: If you easily get offended, do not read this piece):

"Americans are stupid, and I love them for it. Because stupidity leads to simplicity, and simplicity is exactly what we need in software development."

See also:
100 Interview Questions for Software Developers

To TDD, or not to TDD?

Summary: Some thoughts on Test-Driven Development (TDD).

A few weeks ago, Stack Overflow co-founders Joel Spolsky and Jeff Atwood discussed criticized certain aspects of Test-Driven Development (TDD) and SOLID Principles of Object-Oriented Design (OOD) [the latter advocated by Robert C. Martin, AKA Uncle Bob]. Check out the excerpts from the original podcast and the follow-up talk (you may also want to listen to the Scott Hanselman's interview with Uncle Bob, which prompted Joel's rant):

• The Stack Overflow: Podcast #38, Excerpt (14:32) [see also podcast page and transcript]
• The Stack Overflow: Podcast #39, Excerpt (18:35) [see also podcast page and transcript]

As expected, the discussion criticism triggered rather emotional rebuttals from TDD supporters, including Uncle Bob himself; see (and read comments):

• "Quality Doesn't Matter That Much" -- Jeff and Joel
• An Open Letter to Joel Spolsky and Jeff Atwood

And then, a counter-rebuttals by Jeff Atwood:

• The Ferengi Programmer
• Real Ultimate Programming Power

Eventually, Joel, Jeff, and Uncle Bob met for a makeup session, which you can find -- along with the summary -- at:

• The Stack Overflow: Podcast #41

Although all participants of the podcast #41 seemed to agree on many issues, I have a feeling that the opposing sides stuck to their points of disagreement: uncle Bob remained convinced that TDD could do only good (the more TDD, the more good), while Jeff and Joel continued emphasizing that other aspects of software development may be more important than TDD.

I happen to lean more towards Jeff and Joel for the following reasons (in addition to the reasons mentioned in The Stack Overflow podcasts).

First, Joel mentioned the issues related to GUI testing, but it's not just the GUI: the database layer poses another challenge. When unit testing database-driven apps, a common approach is to rely on mock objects (in which case, you totally skip database testing) or use test-specific data in a simplified version of the database (here you will have to figure out how to maintain this database and integrate it into your build process). Even with the 100% code coverage of the business layer, if your unit test skip the GUI and database, about half of the application (more or less) will remain untested, and you'll have to address this gap using functional testing, which will overlap with unit tests. So why spend time writing unit test for the code that will be tested functionally anyway?

Second, claims that TDD improves code quality are rarely, if ever, substantiated by data. You may hear statements like:

"Since I (we) started using TDD, my (our) code has become much better."

I'm not claiming that these statements are false, but without data they are just personal opinions. Even assuming that code did in fact improve, could it be due to other factors, such as better programming experience, tools, team composition, processes?

Analyzing effects of TDD on code quality is extremely difficult. One of the better studies on the subject was conducted by Microsoft and IBM (see the 8-minute interview with one of the researchers and the original paper). The study found that TDD improved code quality, but it also increased development time.

There is a trade-off: either use TDD and spend more time writing, maintaining, and running unit tests now or do not use TDD and spend time fixing bugs later. The question is: how much time? If TDD activities take 2 weeks out of an 8-week development cycle (a typical 25% TDD penalty) to prevent defects that could've been found and fixed in 3 days during functional testing, would TDD give you the best return on investment? By the way, it is worth noting that the team with the smallest test coverage (62%) in the study achieved the best results (90% drop of defects), while the team with the highest test coverage (95%) achieved the "worst" (as compared to other teams) results (40% drop of defects; still not bad when compared to the control group). I would also emphasize that the use of TDD in the study was close to ideal: the projects were appropriate for TDD and the teams were not pushed to achieve the maximum code coverage.

The Microsoft-IBM study had a few problems, though. First, the number of participants was rather small. Second, and more important, it tried to compare similar teams and projects, but there were many difference between them, such as team/project sizes, and project durations, which is expected in a study focusing on real life projects. Studies performed in academic settings -- with undergraduate students of approximately the same grade working on identical projects -- attempted to minimize some of the differences and make experiments more controlled. According to an abstract from one of such studies:

"[T]est-first students on average wrote more tests and, in turn, students who wrote more tests tended to be more productive. We also observed that the minimum quality increased linearly with the number of programmer tests, independent of the development strategy employed."

How this sounds depends on how you interpret the study's findings. If you're a TDD supporter, you may join Phil Haack and announce that Research Supports The Effectiveness of TDD. Or, you can look more closely at the results of the study and come to a different conclusion. For example, Jacob Proffitt suggested (see also post comments) that:

"[W]ithout question, testing first leads to having more tests per functional unit. The question is if this is valuable. This study would seem to indicate that this is probably not the case, at least if quality is your intended gain. But then, I'm not that surprised that number of tests doesn't correspond to quality just as I'm not surprised that the number of lines of code doesn't correspond to productivity."

I would also point that the code in the experiments was not written for database-driven apps with web-based GUI, so again it was an ideal usage of TDD. So I would speculate that if either of these studies focused on database-driven apps with web front ends, the results would probably be even less encouraging.

I'm not an opponent of TDD and agree that it can be valuable when applied to certain types of projects. My gripe against TDD is that it is being often sold as yet another silver bullet and forced into adoption indiscriminately, in the projects where TDD offers minimal, if any, benefits.

I also resent the assumption that TDD always leads to good code. Your My project can have 100% code coverage, but it can still suck. It will pass all unit test, alright, but at the same time it can be unreadable, unnecessarily complex, uncommented, resulting in an app with bad GUI and unintuitive behavior, an app which both the users and support teams hate. But don't you dare to criticize my code: since it's 100% unit tested, it's good by definition. Usability testing? Never heard of it. And as far as everything else goes (like comments, better design, etc), do I have time for this when I spend my better hours working on unit test? Something's gotta give.

Finally, what about all those great apps written by developers, who did not use TDD or any other silver-bullet-like methodology? Take Google developers, who apparently do not follow any software development methodology or approach, but somehow manage to write apps that people love. If they can write great apps (which is one of the goals of good code), why would one want to impose TDD on them?

Now, Google developers are good, but what do you do if your developers are bad? [I'm not talking about developers who make occasional, sometimes serious, errors (this happens even to the best of us); I'm talking about developers who consistently write bad code.] Wouldn't TDD help them write better code? I wouldn't expect it, but if you can share a success story, please leave a comment.

If your organization suffers from bad code, consider adopting TDD (when applied correcly, it may help), but more importantly, consider other factors, such as:

• The Joel Test: 12 Steps to Better Code by Joel Spolsky
• What Makes Good Code Good? by Paul DiLascia

Once you the adopt common-sense practices, you may not need TDD after all.

See also:
Hanselminutes Podcast 146 - Test Driven Development is Design - The Last Word on TDD
Hanselminutes Podcast 31 - Test Driven Development
TDD Tests are not Unit Tests by Stephen Walther
Test-After Development is not Test-Driven Development by Stephen Walther

Problem solvers vs. process keepers

Summary: "Turning a new process on does not justify employees turning their brains off." Bob Lewis

Here is a fascinating recording of a conversation between a Dell support representative and a customer (WARNING: the clip contains strong language; if you get offended by profanity, cursing, or swearing, do not play it):

In case you skipped the clip, here is a brief summary. A technical support representative (a help desk guy) receives a call from a customer who cannot turn his Dell laptop off. After learning about the problem, the help desk guy asks the customer to answer a seemingly endless number of questions. He asks for the service code (which the customer has apparently provided before being connected to the support desk), name on the account, phone number of the purchaser, name of the caller, his last name, phone number, email address... With every new question, the customer gets more frustrated and angry: he swears, curses, and keeps begging to tell him how to turn the laptop off. The help desk guy remains calm; he tolerates the verbal abuse, but insists on getting the information. The customer eventually answers all but the last question. Finally, the help desk guy tells the customer to hold down the power button for 10 seconds. The trick works (the laptop shuts down), but the customer bursts with outrage:

"Oh, Jesus Christ, you could've told me that. You know, you're [...] not very freaking helpful. You know that? You're a [beep]ing asshole. And you can put this in your [beep]ing records, alright! You know, if I had you right here, I'd [beep]ing kick your [beep]ing ass."

There is no good-bye.

I assume that the recording is genuine. It has to be. Even if not, it can serve as a great illustration of a conflict between humans and processes.

Before I move on, let me state the obvious: the customer is disturbed. And he is wrong. There is absolutely no... okay, there is not enough... reason for being so obnoxious. If I were the help desk guy, I would've hung up on him.

On the other hand, why is the help desk guy so stringent? Would it kill him to tell the customer to "hold down the power button for 10 second" right away? And why does he need all this information (email address, phone number, and so on) in the first place? How would the email address help him verify the system owner?

I'm not sure if the help desk guy knew why he needed the caller's email address. He was probably just following the process, and the process had asked for an email address.

It's ironic that the process, no matter how well intended, didn't help anyone in this case. If the help desk guy hadn't followed the process, he would've solved the customer's problem in less than a minute. He could've told the customer to hold down the power button first and then asked him to provide the information. Or he could've said: "Listen, this seems like a simple issue, but if my first suggestion does not work, I may need to keep you on the phone a bit longer, and I will have to ask you a few questions for the support ticket." But instead, the help desk guy turned into a process keeper, blindly following the letter of the process, even when it did no good.

A few posts ago, I mentioned Bob Lewis' article in which he made the following point:

"Make sure every process has a process bypass to handle situations that just don't fit."

An emotionally unstable customer asking a simple question is an example of a situation requiring a process bypass.

Talking to -- or hearing someone else talking to -- a process keeper insisting on following the process no matter how inappropriate the process is (in general, or in a particular context) reminds me of this classic conversation from One Flew Over the Cuckoo's Nest:

McMurphy: Nurse Ratched, Nurse Ratched! The Chief put his hand up. He voted! Would you please turn the television set on.
Nurse Ratched [opens the glass window]: Mr. McMurphy, the meeting was adjourned and the vote was closed.
McMurphy: But the vote was 10 to 8. The Chief, he's got his hand up! Look!
Nurse Ratched: No, Mr. McMurphy. When the meeting was adjourned, the vote was 9 to 9.
McMurphy [exasperated]: Aw come on, you're not gonna say that now! You're not gonna say that now! You're gonna pull that hen house shit? Now when the vote... the Chief just voted - it was 10 to 9. Now I want that television set turned on right now!
[Nurse Ratched closes the glass window]

Most effective organizations follow processes, which help them maintain efficiency and achieve other (noble) goals. However, once a process stands on the way of problem solving and starts defying common sense, it defeats the very same goals it intends to achieve.

If your team enforces the process no matter what, whether it does or does not make sense, consider focusing more on problem solving than process keeping; otherwise, you are at risk of turning into an organization run by the spirit of Nurse Ratched, which is not good even for a mental institution.

Additional references:
Six Stupid process controls
Are You a Doer or a Talker?

Love thy critics

Summary: Don't hate your critics; you can learn something from them.

If you follow technical blogs, you must have heard the news about stackoverflow.com: a recent joint venture between Jeff Atwood and Joel Spolsky. If not, here it is in a nutshell: about a month ago, two of the most popular technical bloggers created "a programming Q&A site" which is "by programmers, for programmers, with the ultimate intent of collectively increasing the sum total of good programming knowledge in the world." The new site is still at an early stage, but hopefully it will continue the traditions of Coding Horror/Joel on Software and turn into the source of knowledge and entertainment for programmers and other tech lovers.

I'm delighted that my favorite bloggers now produce podcasts and other technical content together, but I'm even more fascinated about the personal dynamics between the duo. In case you are not familiar with the story, Jeff has not been always kind to Joel. Not long ago, he called Joel blind, illogical, and just stopped short of calling him insane. In fact, he did call Joel insane... right before posting this picture:


If you look at this story closer, you will realize that I did tweak the truth a bit. You see, Jeff did not actually call Joel blind, illogical, and insane in general. No, sir. He merely implied that certain technical decisions made by Joel could have been characterized as such. Do you see the difference? Joel did, but many software professional don't.

After having worked in IT for over a decade, I witnessed numerous times how technical disagreements produced animosity between otherwise rational professionals. So it's refreshing (and encouraging) to see Jeff and Joel embracing their technical differences. This is how grown-ups behave.

Note to self: next time you hear people criticizing your technical position (your design, code, chosen methodology, etc), or call you (technically) blind, illogical, or insane, think about this:

• It is not personal.
  They are not trying to make a fool out of you or ridicule you; they just don't agree with your technical position. So do not fall into a trap of making it a personal issue between you and them. Although they may be fighting for their opinions for personal benefits, it's most likely that they want the best outcome as much as you do, so don't treat them as personal enemies.
• They may be right.
  Don't assume that you know everything. Try to find out if you are missing something that they take for granted. Biases are sticky. They may be genuinely misguided, but so may be you.
• Be gracious.
  Take a deep breath and stay cool. Be mature. Be patient. Do not call names. Don't make personal attacks even when being attacked. Don't whine. Whether you are right or wrong, you may lose. If you lose, don't be bitter. Don't seek revenge.

If you are surrounded by people who always agree with you, you are limiting your personal and professional growth. Instead, foster relationships with bright people (colleagues) who are not shy to criticize you; they will make you better. And in the meantime, check out stackoverflow.com.

Additional references:
Introducing Stackoverflow.com
Stackoverflow.com

Can a man change?

Summary: How can you improve efficiency of an organization? Hire great employees. Fire bad ones. But, first check out the references mentioned in this article.

	Madame de Merteuil:	"You are an awful man."
	Monsieur de Valmont:	"Do you think a man can change?"
	Madame de Merteuil:	"Yes. For the worse."
From "Valmont"

In the article The causes of greatness, Bob Lewis explains what makes an organization great focusing on the value of people. The article offers several astute observations, but the main idea can be summarized in the following statement:

"[I]f you want an organization that works, you'll get more leverage from hiring great employees than from any other single effort you can undertake."

Among high-tech companies, Google understands the importance of hiring quality workers better than most. In case you haven't heard about Google's hiring philosophy, it's based on two simple rules:

1. Only hire candidates who are above the mean of your current employees;
   and
2. There is no hiring manager.

Peter Norvig, Director of Google Research, explains how this approach helps Google address the shortcomings of traditional hiring practices:

"Whenever you give project managers responsibility for hiring for their own projects they'll take the best candidate in the pool, even if that candidate is sub-standard for the company, because every manager wants some help for their project rather than no help. That's why we do all hiring at the company level, not the project level."

Google's hiring philosophy seems to pay off (at least, if you believe Google employees who like to brag how smart their co-workers are); unfortunately, it is not that common. Most companies make their hiring decisions under the following premises:

1. If you do not hire someone today, your open req* may be gone tomorrow (due to cost saving initiatives, reorganization, outsourcing, etc).
2. Considering the previous statement, don't waste time searching for the best candidate; just pick the first applicant, who seems to fit the job description.

Organizations operating under such premises tend to underestimate the implications of bad hiring decisions because they succumb to a couple of fallacies.

Fallacy #1: Bad workers will do a good job if they follow an approved process/methodology/procedure (CMMi, XP, Agile, SCRUM, etc).

This belief is based on the assumption that processes are more important than people. Those who hold this belief think that any organizational problem can be solved with the help of a good process.

While processes have their place in the organization, managers tend to overestimate their value because relying on processes gives them a (false) sense of control. If your organization is not effective, it is comforting to believe that once you adopt a new process, things will change to the better. If you admit that bad employees are at the root of your problems, you will have to confront people and either make them change or control their behavior, something good managers do not enjoy doing.

The good news is that certain processes adopted by software development teams can reduce the number of problems caused by bad developers. Juval Löwy once wrote:

"Rapid development tools will make bad developers worse, because they allow them to produce bad code faster."

Likewise, by introducing certain bureaucratic overhead, processes can make bad developers less bad, because they will force them to produce bad code slower. However, if you're stuck with mediocre developers, no process will make your organization sufficiently effective. If you agree with Steve Yagge, who suggested that

"Bad developers, who constitute the majority of all developers worldwide, can write bad code in any language you throw at them,"

you should agree that bad developers can write bad code under any process you throw at them. This brings us to the fallacy of personal improvement.

Fallacy #2: Certain measures -- such as effective supervision, mentoring, a development plan -- can turn bad workers into good workers.

I'm not questioning a possibility of human self-improvement, but I cannot seem to recall a single case of a bad worker turning good. I have witnessed good workers becoming better, average workers staying about the same, but bad workers tend to remain bad.

Some bad workers do not improve because they don't even understand that they are bad (or how bad they are). Cornell University researches conducting the Unskilled and Unaware of It study concluded:

"When one fails to recognize that he or she has performed poorly, the individual is left assuming that they have performed well. As a result, the incompetent will tend to grossly overestimate their skills and abilities." (read the review)

The study suggests that people can be trained to "recognize the limitations of their abilities," but what about actual working skills, such as ability to write software? According to the findings of A cognitive study of early learning of programming, most people can't be taught programming. Nevertheless, many of those who can't learn to program somehow manage to get Computer Science degrees and eventually get programming jobs.

So what's a manager to do? To paraphrase Bob Lewis,

"If you want an organization that works, you'll get more leverage from firing bad employees than from any other single effort you can undertake."

This may sound harsh, but for organizations which do not invest in a rigorous -- a'la Google -- hiring procedure, getting rid of bad workers (not necessarily via layoffs) may be the most effective method of achieving efficiency. I'm not advocating such draconian measures as Jack Welch's annual firing of the bottom 10% of his managers (although, if implemented correctly it could be quite effective), but getting rid of people, who cannot do their job, is essential to the health of the organization. If you are interested in better ideas, read the references below.

Additional references:
How Do You Find the Best Employees for Your Company?: Things that do not help in hiring great developers.
Done, and Get Things Smart: How do you find great employees?
A Field Guide to Developers: What it’s going to take to be a top choice for top developers.
A holiday card to the industry - 2007: People, process, and technology: it's a simple formula that describes what makes any business operate.
Converting Capital Into Software That Works: A software company has to think of recruiting the right people as its number one problem.
Finding Great Developers: You can receive thousands of job applications and never see a great software developer.
Hazards of Hiring: Eric Sink offers guidelines for handling tough hiring decisions in a small ISV.
Google: Ten Golden Rules: How Google gets the most out of knowledge workers.
Hitting the High Notes: The lagniappe that you get from the talented software developers is your only hope for remarkableness.
How Google woos the best and brightest: Going from 150,000 resumes per month to 9 hires per day.
Interviewing with Google: Benji Smith shares first-hand experience with the Google interviewing process.
Iz.I.T.4.V.S.P: Suggestions how Intel should handle redeployment (layoffs).
News: Analysis of the "we hire the top 1%" claims.
No Matter What They Tell You, It's a People Problem: Usually the things that make or break a project are process and people issues.
Separating Programming Sheep from Non-Programming Goats: There are two populations: those who can program, and those who cannot.
Sorting Resumes: Looking for negative clues in resumes.
The Guerrilla Guide to Interviewing: Don’t hire people that you aren’t sure about.
Whaddaya Mean, You Can't Find Programmers?: How to find people and convince them to work for you.
Why Can't Programmers... Program?: 199 out of 200 applicants for every programming job can’t write code at all.

Bonus:
Employees Fight Abusive Supervisors with Lower Productivity: Researchers have quantified the ways employees quietly fight back despotic supervisors.

---

*	Open req is a corporate slang for an open requisition (position).

From real Steve to fake Bill

Summary: A few bits of entertainment and knowledge related to Apple and Microsoft, Steve Jobs and Bill Gates.

If you're following the buzz surrounding Steve Jobs and/or Apple (iPhone price cut, rebates, etc), you may enjoy the following articles:

The Puppet Master: Love Steve Jobs or hate him, just don't ignore him by Robert X. Cringely (interesting)
Dear early iPhone adopters: Yeah, we f****d you by "Fake Steve Jobs" (funny)
Larry's bold idea by "Fake Steve Jobs" (funny)

Not to be forgotten, here is your weekly dose of (well-deserved, yet good-natured) Microsoft mockery:

Channeling Microsoft Execs by John C. Dvorak (funny)

And a twist of iPod vs. Zune humor:

California by Joel Spolsky

Dvorak prediction 4,652

Summary: Lame rant on the importance of buzzwords and fads in high tech.

John C. Dvorak, an award-winning technical columnist, is not known for avoiding (technical) controversies. Some of his opinions and predictions (such as suggestion that Apple would abandon MacOS and switch to Windows) were proved wrong (at least thus far); others came true (for example, he predicted that Apple would release iPod despite Steve Jobs' denial). In a recent column Dvorak responds to Sam Palmisano (head of IBM) who has just pronounced client-server computing dead. Says Palmisano:

"The PC client-server model has run its course."

According to Palmisano, businesses must "escape the economic waste that has plagued traditional client-server architectures" and move to "a new architecture for data centers" coupled with "software as a service (SaaS) environments and service-oriented architectures (SOAs)."

Dvorak calls Palmisano's "new" approach nothing more than:

"what they do now, but with a fancy name"

and concludes:

"In the next few years we will see more and more new monikers that rename what we already do but change absolutely nothing."

You may discard Dvorak's opinion as irrelevant, but if you decide to analyze Palmisano's statement seriously, you will have to answer similar questions. Do client-server architectures really plaque businesses with economic waste? If they do, what (or who) can guarantee that the "new" architecture will do any better? By emphasizing the end of PC client-server model, does Palmisano mean to somehow distinguish it from non-PC client-server models? Does he suggest that non-PC client-server models (whatever they are) are alive and well? What is so particular about the PC (vs. non-PC) client-server model other than the fact that with the recent sale of PC unit to Lenovo IBM has exited the PC business? And which "traditional" client-server architectures is Palmisano talking about: 2-tier, 3-tier, n-tier, ...? Isn't Web-based architecture also client-server, as Dvorak rightly points out? Aren't all distributed application architectures based on the client-server architecture (even peer-to-peer can be viewed as client-server at certain angle, as well as SOA and others)?

I guess, Palmisano, who majored in history at school and specialized in sales at work, did not intend to be technically precise, or even correct. His likely goal -- and it's hard to argue with Dvorak about this -- must be promotion of the IBM's current lines of businesses (servers, consulting services), and buzzwords seem to be the best promotional drivers in the today's corporate world. In fact, if he substituted a new architecture for data centers with IBM-built hardware and software as a service (SaaS) environments and service-oriented architectures (SOAs) with IBM-built software, Palmisano's talking points would make more sense (at least, from the IBM's perspective), but they would not generate as much hype.