If agile isn’t dead, why is it still not working?

Agile software development has outlived wave after wave of technology innovation, but in 2023, is the methodology on life support?

Agile needs a retrospective.

It’s been 22 years since the signing of the venerated Manifesto for Agile Software Development. Yet many organizations still struggle to embrace agility. Despite billions spent on consultancies, certifications, frameworks, and whole departments, there are a lot more caterpillars and chrysalides than butterflies to show for the agile revolution. Which brings us to ask:

• Why don’t more organizations have the finish lines of their agile transformations in sight? 
• Is it because something created by 17 white men over a weekend more than 20 years ago can’t hold up anymore?
• Is it because things like Agile HR and Agile Marketing and DevOps are all the rage and it was only ever meant for the development of software?
• Have companies really embraced agile or are they just going about agile-in-name-only?
• Are Waterfall or Prince2 methodologies having a resurgence, or do plenty of enterprises still think agility is cutting edge?
• Do we really need daily stand-ups? Is anybody still standing anyway?

The end of agile has been talked about since at least 2014, but it has always been a complex debate, colored by the tech industry’s constant hunger for the next way of doing things and a habit of creating more and more complexity to contend with.

Why do we still want to be agile?

“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.”

That’s it. Just 68 words. That’s the whole Agile Manifesto that sparked a movement to help the software industry respond to an increasingly complex world.

It’s a simple set of principles that still have weight. The 2022 State of Agile report found that 70% of agile practitioners are satisfied with these practices and that they most appreciate the increase in cross-company collaboration and continuous improvement they drive. The report also found that organizations are embracing agility mostly to improve time to market, with more predictability, and lower risk.

Enterprise agility consultant Almudena Rodríguez Pardo says most organizations embrace agile to combat slow time to market, software quality issues, more nimble competition, and unsatisfied employees.

She is clearly an evangelist, calling the Agile Manifesto “a piece of poetry of just four lines that have held strong for 22 years.” But, while she says it is “pure common sense put to words,” it’s also “so simple it can be manipulated.”

Still, 22 years in and we continue to talk about agile. Why? “Because nobody understands it,” says Erin Davies, an agile consultant at the 1ovmany consultancy. Whether it’s agile, DevOps, or OKRs, these things all “look conceptually very easy on paper and it makes sense, but for reasons nobody can understand, we manage to make it look very difficult in the workplace.”

Part of the confusion is in the name – it says it’s about software development, but it’s really anchored on that first value: people over tools.

In fact, 89% of respondents to the State of Agile report believe that high-performing agile teams have:

• People-centric values
• Clear culture
• Tools 
• Leadership engagement

“If you think it’s a technical problem, it’s always a people one,” Rodríguez Pardo says, paraphrasing Weinberg’s Secrets of Consulting. “It’s about complex, adaptive systems.”

Where are organizations still struggling?

By reframing agile as a people problem, organizations can set themselves up for success. Last year’s State of Agile report found that a lack of leadership participation or support, and resistance to change were the leading blockers to agile adoption.

It is notable that across all culture markers, there was an improvement from 2021 to 2022, but it remains to be seen, especially as budgets tighten, if 2023 will continue to follow this downward trend.

“It’s not about agility. It’s about change management,” Rodríguez Pardo says. For fear of introducing another framework, John Kotter’s eight steps for leading change would be valuable in these instances:

1. Create a sense of urgency
2. Build a guiding coalition
3. Form a strategic vision
4. Enlist a volunteer army
5. Enable action by removing barriers
6. Generate short-term wins
7. Sustain acceleration 
8. Institute change

The goal is “to get to the agile vision we define at the beginning to the transformation – or to get close to it,” Rodríguez Pardo says, independent of the framework. You have to know why you are introducing such a change into an organization and only then measure progress.

“Agility is a change process,” she says. “Success or failure depends on the vision you’ve had and if you’re able to apply the change process without falling back.”

What hasn’t changed year after year is that communication and leadership are the two most important factors holding organizations back from achieving agility.

Is it still all the consultants’ fault?

Agile can’t be dead, Davies says, because the industry continues to develop better ways to develop software – it’s just that “we’ve forgotten why we do what we do.”

“Consultants, we’ve ruined everything,” she says. “Somebody has gone ‘I can turn [agile] into a sellable product and sell expertise that they are experts in’ and everyone forgets the whole thing about agility. […] If you are basing your coaching and consultancy around a specific framework and you are inflexible yourself, you are part of the problem.” 

That’s why Davies says she signed up to Agnostic Agile, which embraces a more iterative, improvisational form of agility. You start cooking, have a taste, and then try something else if that doesn’t work.

For Davies, agile isn’t dead, it’s just fundamentally misunderstood. “For 17 white dudes, they came up with something that’s remarkably egalitarian,” she says. The main problem she sees was in making it solely about software delivery, instead of simply a manifesto for “agile working together.” 

Take the 15-minute daily standup as a way to keep everyone engaged in their work or scrum events, and retrospectives as a way to communicate and pivot, and Kanban to foster continuous feedback loops.

“If a team understands why we do the things that we do and has a better way of doing it, I will scrap every single one of those meetings,” she says. “Once teams are really cohesive then they can leave the framework.” All you need is a small team with a product and a good sense of goals to be successfully agile.

It’s just that most organizations, she’s found, are performing these rituals, but are doing what she calls mini-Waterfall: “I make a plan. I know where I want to be. I execute on the plan. And I never think how this is aligning to the end user value that I need.” They are practicing sprints and demos, but it’s just smaller, she says, and “you don’t actually feed back into the loop.”

“We’ve overcomplicated it”

Along the way, in the land of certification factories, “We’ve forgotten the purpose and value,” Davies says, “we’ve overcomplicated it,” pointing to the current 2020 version of the Scrum Guide, which is 13 pages long.

“More people are becoming cynical about wholesale implementations of frameworks,” Davies says, pointing to the Scaled Agile Framework, commonly called SAFe, as an example. Since 2011, SAFe has expanded the Agile Manifesto to seven core competencies of business agility, which can be learned through a variety of paid courses, all geared towards an enterprise audience.

Like it or not, according to the State of Agile Report, more than half of enterprises responded that they are currently following SAFe.

Rodríguez Pardo has been a SAFe consultant for seven years now, at multinationals across a wide variety of sectors, saying that “SAFe is not perfect, but it’s the best of all the possibilities to help organizations that have 100,000 employees or more.”

“It’s a framework which can support you to introduce agility in big, established organizations with a legacy which needs an evolution and not a revolution towards agility.” She calls it an evolution because business agility cannot be adopted overnight, particularly in the highly regulated environments she works in. In fact, it often takes two to six years. And even then, she says, “Maybe you will never make it across 150,000 employees and 180 countries.”

Still, with or without SAFe, she says failure happens. 

Be more adaptive

The barriers to agility are in part agile practitioners themselves, according to Sophia Ashley, senior product analyst and scrum master at AND Digital consultancy. “As a consultant, you may have the perception that you’re trying to sell something. I’d ask why do you want to do [that]? Why do you need it?” 

She compares agile to a tool. “People buy something and think it fixes all their issues. Any tool is designed with a specific purpose in mind, and then they use it for something else. You can use a hammer to put a nail in a wall, or bash someone’s head in.”

An agile consultant has to use whatever tool or combination of tools works for their team, she says. “I’m not very religious when it comes to scrum. Doing scrum by the book is unwise,” even though the Scrum Framework calls itself immutable. 

Ashley advocates for the use of “whatever tool is available to maximize your success and the output.” She continued that if all you’re doing is “stressing about deadlines, outages, customers going to competition, then you have shitty agile.”

In the end, she asks: What do you care about more, following a process or methodology, or releasing a really great product? Learn to identify what works for your team and where a toolset may hinder you.

Can only startups (still) be agile?

A personal trainer would argue, in order to get into shape, you should start small, and yet, in agile, Davies says, “we get big bang implementations, because “people don’t like change. It’s still too top-down.” This is why she suspects that the only truly agile company is small, slightly chaotic, and relationship-based, while also continuously inspecting and adapting. As these organizations formalize, they lose some of those connections, and the ability to change direction and learn.

“We never use the word ‘agile.’ It just happens,” Jas Schembri-Stothart, cofounder of UK-based health and wellbeing app Luna says. “We don’t stand around a Kanban board. We have a Notion [mini Kanban] page with tickets and buckets. We have a weekly team meeting reviewing tickets [that] start with feature updates and what’s being rolled out in a 25-minute all hands,” which is seated or online for the seven-person hybrid team. This weekly meeting is focused at the highest level, not individual work.

They also have a weekly one-hour product meeting that not everyone attends, but uses tickets to go granular on what developers are working on. Finally, they have a half-hour meeting at the end of the week to discuss highlights and lowlights.

“For us, agile means being super connected to what everyone’s doing but in a way that requires lots of meetings and communications,” she says. But, she admits, this could change as they scale. The company is not even two years old yet, and its minimum viable product was launched last November.

For her, every pivot they make has to be in service to their core audience. In-app feedback is connected to a channel that everyone is in, most often highlighting new features and what’s not working. That creates transparency so “we don’t have to put a lot of pressure top down like the next feature will be this,” Schembri-Stothart says, which in turn drives change management. 

Final thoughts

Agility really comes down to building psychological safety to allow people to fail, learn, and pivot. In the end, does it matter if agile software development is alive or dead, as long as the industry is continuously looking to improve? Or, as Rodríguez Pardo asks, “If we don’t do agility, what do we do?”