If Agile isn't working, it's your fault

 Everybody is doing Agile development these days, or so they claim. Especially Scrum seems to be ubiquitous. Despite that, software development in general is not much more successful than it was 20 years ago. Surely that proves that Agile, and particularly Scrum, isn't working? Not so fast.

Let's take a look at what the Agile manifesto says about the values Agile is based on:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
 

Let that sink in while we take a look at what research says about teamwork in general. In particular, research has tended to focus on the difference between real teams, pseudo-teams and groups.

Real teams are distinguished by doing all of the following things:

1. The members claim to belong to a team
2. The members of the team have a common goal.
3. The members are dependent on each other and have to communicate and coordinate their work.
4. The team regularly reflects over the way they've worked and change things to improve their way of working.

Pseudo-teams claim to be a team, but do not do all the other three things, while groups do not claim to be a team (even if they may do the other things). The interesting thing to note is that real teams perform much better than pseudo-teams. Another interesting thing to note is that groups perform almost as well as real teams. I suspect that is because groups don't assume things will magically work out, they take the steps necessary to coordinate the work and achieve success.

The single most effective tool for organizational learning is the After Action Review. It's no coincidence that it fulfils point 4 above. There are a few similar things that also come in under point 4, like Post-mortems, or the Scrum "Inspect & Adapt" principle, a.k.a. "the retro". Taking it all the way back to the Agile manifesto, it favours "Individuals and interactions over processes and tools".

Clearly there is no point in reflecting about the work unless action is taken to make things work better going forward. If you spend your retros complaining, celebrating and blame-throwing, you've missed the point.

If you can't change the process, you're not doing Agile. If you don't change the process, you're the only one to blame. You're the software development professional, and as Uncle Bob puts it, you're supposed to profess how software can and should be developed. You don't tell the manager how to run the business or the product owner how to decide which features are most important, neither should they tell you what you can develop, how fast, or how you should do it.

What process should you have?

The only agile answer is that you should have the process that helps you do the work. And that process should change as circumstances change. Does the daily stand-up help you have a more pleasant and productive day? If it does, keep it, if it doesn't, change the way you do it or abolish it.

The first priority is "Working software over comprehensive documentation". One thing I like to do is to write down or draw an outline of a technical design, in just enough detail to understand it and discuss it. I feel that helps me develop working software. I will not maintain the technical design document as the software changes. Unless the team decides the document does help to continue developing working software, in which case we all commit to keeping it up to date.

A more controversial topic is perhaps the sprint plan. Does the sprint plan help you create working software? If not, it's just useless paperwork. Does estimating your stories help you deliver working software? (I'd say that's a no, but YMMV)

Hang on, you say, the manager needs the plan to do their job. This is where we pull out "Customer collaboration over contract negotiation". The manager is in a sense a customer receiving the plan, so what can the team provide instead? It turns out that the manager really needs a forecast, not a plan, and it has been shown that just counting tickets in Jira (or lines on the spreadsheet or whatever you're using) is at least as accurate for forecasting as any estimating and planning.

If you add more tickets as you learn more, that's still fine, good managers also know the value of "Responding to change over following a plan". What they hate is being told that day before they thought you would be done that it's going to be a few more weeks.

When it comes to collaborating with the manager and product owner on what can be delivered and when, there are four factors in play: resources, quality, time and scope.

Increasing resources, like adding people or using better tools, usually helps in the long term, but it will be detrimental in the short term. You may have heard of teams going through the stages of "forming, storming, norming and performing" and as soon as theteam changes, they go back to square one. It takes time to get back to efficiency. Likewise new tools can have a long rampup before they can be used better than the old ones. Often it is probably not even worth the trouble.

Decreasing resources can't usually be helped, but the negative effect on team productivity needs to be recognized.

Skimping on quality is a dangerous route, but can be done to some extent. Deliberately not handling edge cases or error cases in the short term to make a deadline is often doable. It may also be acceptable to do something the quick way instead of the right way. The trick is managing the technical debt later. The "wall of technical debt" may be a way to do it. Make sure that it is a deliberate decision to take on the debt.

That leaves time and scope that can be played with more freely. In the spirit of collaboration, the tech team needs to be honest about what can be done and when, but should also be able to come up with alternative solutions. Often the customers' biggest needs can be met in simpler ways, cutting down scope. At other times the manager just needs to accept that it will take longer.

Risks also need honest communication and evaluation. When tasks are no longer estimated for time, there are still two types of tasks: those the team understands how to do and those the team doesn't understand how to do in enough detail. The tasks that are not understood are a risk and risks need to be defused, so the team will need to spend time to dig into them more and plan them out better.

There are lots more potential topics to dig into, like the technical development practices, or the myriad of ways management can sabotage teamwork by performance reviews or issuing individual rewards, but I think this is a good place to stop, having outlined a decent basis for an agile development process.

Don't just listen to me, though, also take a look at what Ron Jeffries, one of the original signatories of the Agile Manifesto, has to say: Scrum is not Agile, but it's not a bad place to start.