Why the best thing you can do is fail

Editor's note: Eddie Obeng is an author and founder of Pentacle, a virtual business school. He spoke at the TED Global conference in Edinburgh in June. TED is a nonprofit dedicated to "ideas worth spreading", which it makes available through talks posted on its website.

(CNN) -- After the 2008 financial crisis had taken hold, the Queen of England asked, in essence: Why did no one tell us that the disaster was coming?

The answer from a group of eminent economists, some of the smartest people on the planet, was to blame it on "the failure of the collective imagination of a number of bright minds."

Do you wonder why a search on Amazon for the word "creativity" returns a list of about 90,000 books? Why a search on Google for "innovation + creativity" delivers 45 million results, and adding the word "consultants" boosts the number to 60 million? And yet, in spite of all that professional advice, it's the rare idea that is making money or delivering benefits two years after its implementation.

Wonder why businesses insist that their expensive executives spend a significant proportion of their time (over three months every year) carefully preparing budgets, only to find that they are often out of date and need changing before they can be published?

Have you been troubled by so much energy dedicated to summits and visions on how we intend to rebalance the environment or on how we intend to grow wealth for all, and yet with little in the way of accomplishments?

Isn't it clear that the key is not the vision but the implementation?

TED.com: Why you will fail to have a great career

Despite businesses spending decades professionalizing implementation in project management and change management, the results are underwhelming.

2012's business hall of shame

Facing economic challenges head on

Singapore's labor pains

Drug giant embraces innovation

To use an analogy to illustrate how woeful the results are, imagine a family of five planning to go on holiday from London to Hong Kong, with a budget of £3,000. If the vacation turned out like the typical result of a business plan, the family would end up in Makassar, Indonesia, at a cost of £4,000, while leaving two of the children behind.

What's happened in business is that the rules of the real 21st century aren't clear to us, so instead we spend our time responding rationally to a world which we understand and recognize, but which no longer exists.

In the past, the world in which we created our institutions, enterprises and organizations was built on a number of truths. In that world, the immediate past and near future were usually similar enough to allow us to predict the future with some confidence.

TED.com: Keep your goals to yourself

As individuals or organizations, our experiences were mostly influenced by local events, and so by understanding our local business or national environment we could learn about the world around us much faster than it was changing.

This meant that our knowledge and experience had a long "sell by" date. To be successful, we would ensure that the person who knew best had the power to decide and lead effectively from the top. Enterprises sought growth often by choosing to do "more of the same" of what had produced their success.

But over the past few decades we have moved our world to a point where institutions, enterprises, organizations and individuals most often need to seek change. At the same time we have connected the world through cyberspace, allowing distant events, ideas and social phenomena to influence us instantly. And we've enabled people to connect to each other, fueling the density of human interaction.

TED.com: What we learned from teetering on the fiscal cliff

The only things holding us back are the ever weakening inertia of habit, and tradition. And it is this shift in balance which has resulted in a silent transition, flipping from an environment which was far more predictable, Newtonian and laminar, to a turbulent and more complex new world.

I call the transition point "Midnight" (the punch line of a weak joke about the fact no one noticed the flip because it happened at night while they were asleep) and I have been exploring how to think and what to do in this World After Midnight.

I proposed an experiment to many clients: "Keep a diary on a spreadsheet, for a week. Wait for four months and then score the time spent which either improved the bottom line or made your life better." A typical result is about 15%. But these are not idle people. They are smart, well-meaning people, in top, well-resourced enterprises, who are working full weeks, often from dawn to beyond dusk. The result horrified me -- almost all of the time spent acting rationally in response to a world which doesn't exist turned out to be wasted.

All the CEOs around me, my clients, want innovation, so they seek innovation. They say to people, "Take risks and be creative!" But unfortunately the words get transformed as they travel through the air. By the time the words enter their ears, what people hear is, "Do crazy things and then I'll fire you." Why?

Because in the old world, getting stuff wrong was unacceptable. If you got something wrong, you'd failed. How should you be treated? Well, harshly, because you could have asked somebody who had experience. The answer is, don't do things which are different. And then suddenly we tell people to do things differently, and it doesn't work.

TED.com: How ideas trump crises

In reality, there are two ways you can fail in our new world. One, you're doing something for which you should follow a set procedure, and it's a very difficult thing, you're sloppy, and you get it wrong. How should you be treated? You should probably be fired.

On the other hand, you're doing something new, no one's ever done it before, and you get it completely wrong. How should you be treated? Well, free pizzas! You should be treated better than the people who succeed. It's called smart failure.

My goal is to get executives around the world to realize that we're in a new world and that they have to rethink everything they do as a result.

Follow @CNNOpinion on Twitter.

Join us at Facebook/CNNOpinion.

The opinions expressed in this commentary are solely those of Eddie Obeng.