Adapting to a complex world: why we must always experiment and bounce back from failure

Bestselling author Tim Harford shows how a spirit of experimentation with variation, selection and survivability are key for innovators and entrepreneurs.

Disruptive innovation and political complexity are creating a fast-changing and often confusing world. From startups to governments, a mindset of experimentation and coping with failure is key for long-term success, according to the author of the book Adapt: Why Success Always Starts With Failure.

Tim Harford is the author of the bestsellers Fifty Things that Made the Modern Economy and Messy: How to be Creative and Resilient in a Tidy-Minded World (see my book reviews and author interview). His other books are The Undercover Economist and The Logic of Life. Tim is a columnist for the Financial Times, presenter at BBC, and visiting fellow of Nuffield College at Oxford University.

The eight chapters are thoroughly referenced and spread across 310 pages; the material is sprinkled with stories and humour as well. The content spans business, finance, politics, biology, arts, neuroscience, and even military history, and is an entertaining and informative read.

Here are my eight takeaways from the book; see also my reviews of the related books The Up Side of Down, The Wisdom of Failure, Fail Better, Fail Fast, and Failing to Succeed.

1. Approaches towards failure

Psychologist James Reason identifies three types of errors: slips (due to clumsiness or lack of attention), violations (deliberately doing the wrong thing), and mistakes (done on purpose but with unintended consequences due to the wrong mental model). Latent errors are those that lurk unnoticed till the worst possible moment, causing larger problems.

It is important to be willing to fail and accept failure, and also create an environment where failures are instructive and recoverable, Tim advises. At the individual level, some failures should be private, with oneself as the critic. Other kinds of failure can be in front of a limited or test audience, so one can learn from the mistakes.

“First, try new things; second, try them in a context where failure is survivable. But the third and final essential step is how to react to failure,” advises Tim, drawing on the work of Russian economist Peter Palchinsky.

Unfortunately, typical responses to failure are denial, cognitive dissonance in accepting failure in an otherwise clean track record, futile pursuit of the failed path hoping it will eventually work out, trying to convince oneself that the mistake was not significant, or even trying to spin the failure as a success by some other metric.

But it is better to make peace with one’s losses than chase losses. A healthy mix of self-doubt along with confidence is needed during the experimentation process, and we must nurture our inner critic while also cultivating a “validation squad,” in the words of award-winning choreographer-dancer Twyla Tharp.

These validators are family, friends, colleagues and associates who are willing to give you honest criticism. It also helps to withdraw and unplug for some time so that the true picture can emerge when the scenario is revisited and reinterpreted.

2. The adaptive mindset

A sense of ‘disciplined pluralism’ is important, according to John Kay, author of The Truth About Markets. Pluralism is a mix of exploration and experiments via small steps and large leaps (and even wild gambles), and discipline brings in process and focus. There should be the inner confidence that the cost and process of failure is something that one has to bear in the longer journey to success.

Tim cites examples of some artistes who have made many dramatic conceptual leaps (eg. Joyce’s Ulysses, Picasso’s Guernica, Beatles’ Sgt. Pepper’s Lonely Hearts Club Band, Bob Dylan), and others who are tentative experimentalists (eg. Robert Frost).

3. Adaptivity in business

Businesses should conduct experiments at the right scale – big enough to make an impact, but small enough to avoid catastrophic failure, along with assessment measures to differentiate between the two. “Trial and error is a tremendously powerful process for solving problems in a complex world, while expert leadership is not,” says Tim. There is a time and place for expert inputs, but hard data from experiments can reveal better insights.

More frontline inputs and decision-making autonomy are called for, as shown with successful companies as varied as Whole Foods and Google. While it is tempting to move toward centralised decision-making, this tendency should be balanced with an empowered workforce, with its own discretionary budget as well. Centralised control should co-exist with decentralised experimentation; monitoring and reinforcement should exist at the peer level also, Tim advises.

This can extend to external audiences via crowdsourcing. Serendipity can be accelerated by creating a greenhouse environment to promote “massively parallel” experiments for the emergence, collision, adaptation and assessment of ideas.

The research on disruptive innovation shows how established players in sectors like mini-computer disk manufacturing were sidelined by makers of smaller drives for desktops; digital cameras challenged photo film makers, and webmail changed the game for desktop Outlook.

Even if incumbents had or acquired the new technology, they were unwilling to cannibalise themselves, as evinced in Kodak. Finding new niche customers with products that seem inferior compared to incumbents gives disruptors a new edge and momentum.

To conduct safe experiments and isolate failures, large companies have tried approaches like ‘skunk works’ (eg. Lockheed, Charles Schwab) or separate brands and companies (eg. Richard Branson’s Virgin Group). However, companies need to go beyond capacity to transform – they should also have the will to transform, otherwise good ideas get trapped within or emerge but are unable to scale, cautions Tim.

4. Encouraging variation: patents, grants and prizes

Allowing for variations in project bids by industry or government opens up new possibilities. For example, the Spitfire fighter aircraft helped change the course of World War II during airfights in Europe, at a time when the focus was largely on bombers.

The global internet is enabling so much experimentation that it is almost like “failure for free,” according to Clay Shirky. Silicon Valley and the startup movement are good examples of bottom-up exploration of the power of emerging technologies, irrespective of a grand coordinating plan.

Patents and grants are regarded as a cornerstone of innovation, but Tim questions their impact and efficiency. While some important domains are overlooked, other patents are given to frivolous ideas. The size of teams to create patents is growing as well. Large companies may prefer the trade secret option to patents. Many grants promote incremental innovations, but few fund radical leaps or revolutionary breakthroughs.

“The problem of encouraging expensive, world-changing innovations remains daunting,” cautions Tim. An interesting approach, and an alternative to grants, is innovation prizes and competitions.

This dates way back to 1714, at least, when the Royal Observatory announced a prize to solve the problem of determining longitude accurately. Similar approaches based on awarding a prize have been used by Netflix (to outperform its CineMatch algorithm), X Prize, Innocentive and Clay Mathematics Institute.

“The wonderful thing about prizes is that they don’t cost a penny until success is achieved,” explains Tim. “This allows the ultimate combination: a completely open field, where failures are tolerated, and the boldest, riskiest ideas could succeed, alongside large sums of money that are spent only when the problem is solved,” he adds. The challenge is to spot opportunity areas that we don’t know yet (a prize to invent the internet would have not made sense when we didn’t know what it was).

The combination of profits, prizes and philanthropic initiatives is an interesting trend to watch, says Tim, as a number of academic institutes, foundations, corporations and philanthropists are joining hands to tackle large problems.

5. Failure in governments and military

Governments in general may tend to be wary of experimenting with taxpayer money and acknowledge failures; they prefer large flagship projects. One of the reasons for the Soviet Union’s poor performance and eventual demise was an inability to experiment, trust dissenting voices and adapt to new courses, Tim explains.

“Adaptive organisations need to decentralise and become comfortable with the chaos of different local approaches and the awkwardness of dissent from junior staff,” cautions Tim. This is difficult for traditional top-down hierarchies, where power is in the boardroom and not on the ground.

“Distinguishing success from failure, oddly, can be the hardest task of all: arrogant leaders can ignore the distinction,” says Tim, drawing on the botched US wars in Vietnam and Iraq as examples. Group-think, conformity, and self-deluding propaganda are no match for frontline insights and ability to adapt. Creative thinkers in such organisations unfortunately run the risk of “borderline insubordination,” says Tim.

Good leaders should not just tolerate dissent – they should encourage it and seek it out. Alternative options and risk assessments should be explored. “For an organisation that needs to quickly correct its own mistakes, the org chart can be the worst possible roadmap,” cautions Tim.

Ironically, leaders tend to draw only those lessons from history that match their own agenda, rather than providing alternative interpretations. New technological solutions and quantitative tools also have seductive power, but Tim warns against the illusion of information without broader and even contradictory insights.

6. The development sector

Much aid has gone into the development sector in emerging economies, but sometimes the projects that seem appealing to donors are not as effective or sustainable on the ground. For example, the PlayPump, powered by a children’s roundabout, was found to be better suited to community settings and not other contexts. It is more effective to have a “worm’s eye” view on the ground, as advocated by Muhammad Yunus, Founder of Grameen Bank.

Adapting to failure and selecting the right solution in this context requires humility, willingness to admit mistakes, share the lessons, and try new approaches; having a “God complex” is not helpful. “Rigorous evidence often overturns years of received practice,” observes Tim.

Randomised trials dig beneath the assumptions of well-meaning and plausible-sounding initiatives. Development sector challenges are complex, and hard to unpack to identify key causality relations. For example, better nutrition programmes may be more effective than books and flipcharts in ensuring that students in poor communities come to school and study properly.

Bottom-up insights and community support are key for development projects to succeed. There are also well-planned top-down initiatives such as Taiwan mastering orchid exports and Chile succeeding in salmon farming. Singapore, Hong Kong and Dubai are other examples of well-planned urban economies. Unfortunately, there is a long history of failed “white elephant” projects also around the world.

7. Climate change

One of the challenges of dealing with climate change is how to account for the carbon footprint of each component and process that went into making a product, Tim explains. A number of measures such as carbon tax have been proposed at the national level for various product ranges.

Some rules have been passed for buildings to generate their own power (eg. the Merton Rule), but challenges lie in making this generative capacity actually work, and maintaining this infrastructure. “There is a difference between the letter and the spirit of the law,” cautions Tim. Many of these requirements can be gamified and thus may fail, thus nullifying the intended effect.

8. Failure in the economy

As systems become more complex and interconnected, especially at a global level, it is important to be able to “decouple” the components and prevent cascading domino effects, as seen during nuclear reactor disasters, oil rig failures, and economic crashes.

Challenges arise when safety mechanisms trip, and create unforeseen scenarios through snowballing effects. Solutions include having crucial indicators readily accessible in easily-understandable formats, well-designed heat maps of the situation, and a focus on the crucial problem areas. Proposed measures in banking include bridge banks and narrow banks, for example, so that banking failures are survivable.

Spotting and recovering from failures in such complex situations calls for frontline alertness, whistleblower rewards, contingency plans, and far-sighted regulators.

In sum, in the context of an increasingly complex, fast-moving and inter-connected world, we are unlikely to get things right the first time. As shown by nature, we have many lessons to learn from biological evolution: variation, selection and survivability.

While it can be crushing to experience failure, correcting the mistakes can be even more liberating, Tim observes. “Whatever its source, we need that willingness to risk failure. Without it, we will never succeed,” he wraps up.

The book has a number of witty and inspiring quotes, and it would be good end this review with the sample below.

The real measure of success is the number of experiments that can be crowded into 24 hours. – Thomas Edison

Evolution is cleverer than you are. – Leslie Orgel

Your first try will be wrong. Budget and design for it. – Aza Raskin

It is very difficult to make good mistakes. – Goro Shimura

The curious task of economics is to demonstrate to men how little they really know about what they imagine they can design. – Friedrich von Hayek

The end of surprise would be the end of science. To this extent, the scientist must constantly seek and hope for surprises. – Robert Friedel

An empiricist, I was willing to learn by my mistakes and those of others. - Muhammad Yunus

The barrier to change is not too little caring; it is too much complexity. - Bill Gates