“Surely,” said Dorothea, “it is better to spend money in finding out how men can make the most of the land which supports them all, than in keeping dogs and horses only to gallop over it. It is not a sin to make yourself poor in performing experiments for the good of all.”
— Mary Ann Evans (aka George Elliot), Middlemarch (1871)
I’m an engineer by training. And as engineers, the strive for flawless execution is so deeply ingrained in our mindset that we often overlook the obvious but daunting larger question: Are we building something that’s actually valuable?
After all, what does a typical engineering-driven project, like the many I myself have kicked off over the years, look like? Obviously, we have a clear vision of what we want to build from the outset. So we concentrate all our time, money, and energy on implementing the best solution we possibly can. We, and our teams, labor day and night, pouring our hearts and souls into the endeavor. And, if we try hard enough, we may truly achieve greatness. It shines. It sparkles. It comes with all the bells and whistles we could think of. It’s configurable, scalable, customizable and and so-much-more-able that we don’t even know where to begin when talking about Its merits.
Then, finally, has the time come to unleash our beloved creation into the wildness of the market. But we don’t expect any problems here, do we? We did our best, worked as hard as we could, and thought of everything for sure?
Turns out, more often than not, that’s precisely the point where things start falling apart. The premise “if you build it, they will come,” probably never held true for anyone other than Noah himself. So, maybe consumers didn’t care quite as much as we assumed about the highly sophisticated, extensible, generic multi-cloud architecture we devised? Or about most of the features and functions that made It seem so perfect in our heads? Whatever the individual reasons may be, if we’re forced to admit defeat at that stage, the amount of time, energy, and capital wasted will make returning to square one now horribly painful. Hence we blindly march on, heading straight for the abyss of the sunk cost fallacy, full recovery from which is nearly impossible.
But where did we go wrong? Isn’t an innovative idea, combined with meticulous planning and splendid execution a proven recipe for success?
Well, Alberto Savoia says no. In his 2019 book The Right It, the serial entrepreneur, Stanford professor, and former engineering director and failure analyst at Google assumes a more systematic view of why so many novel ventures don’t survive in the market, and what can be done about that. His premise is simple, but convincing: First and foremost, we need to focus on building the right thing, and only later should we worry about building it right. But that realization comes with a humbling lesson: Regardless of how convinced we ourselves may be, we must not mistake intuitions about what we think customers might want for hard facts.
The right it
Savoia’s argument doesn’t discount the relevance of competent execution though. He acknowledges that the right thing built wrong is still unlikely to succeed. But when it comes to prioritizing one’s efforts, especially during the early stages of a new endeavor, focusing too much on details such as picking the right technology stack or architecture can be disadvantageous. Of course those things matter, but maybe, they don’t matter so much at this point in time. Hence Savoia defines The Right It in quite a pragmatic way:
“The Right It is an idea for a new product that, if competently executed, will succeed in the market.”
Data
To gauge whether something is or isn’t the right It, the author heavily emphasizes the importance of data. But not all data are created equal, as he reminds us: There’s external, 3rd-party data such as market research gathered by analysts, competitor’s revenue numbers, or economic growth projections. Savoia sums up all of those those under the umbrella term OPD—Other People’s Data—and warns against granting them too much influence over our decisions. After all, particularly in a pre-chasm environment, you don’t even know what exact problem you’re solving yourself, so how should a research agency be able to tell you what percentage of consumers are bothered by that kind of problem? And how should your potential customers seriously assess if they have the need for something that doesn’t exist yet?
Therefore, Savoia sends us on a field trip to collect data ourselves first-handedly, which he, tongue in cheek, refers to as YODA—Your Own DAta. To do that, we’ll have to put our potential customers in a situation where they’re faced with a product or service like the one we’re aiming to build, and observe and measure their interactions and engagement. By doing this early, we avoid overinvesting in a potentially flawed idea. By doing it efficiently, we allow ourselves to try out many alternatives. And by doing it often, we can iteratively hone in on the core of our right It.
Hypotheses and hypozooming
The better part of Savoia’s book is designated to the design and execution of such experiments. The author suggests starting with large, overall hypotheses ("X of Y would do Z"), then making those concrete and falsifiable ("At least x% of such-and-such people would … if…"), and finally zooming in on a local environment where we can actually run real-world tests. Such a microcosm should allow us to cheaply and easily gather high-quality, first-party data, interpret them, and make informed decisions.
That approach, which the author refers to as hypozooming, is in my view both a huge asset, as well as the potential Achilles heel of his entire methodology. We need to be mindful of the risk of over-interpreting positive signals from the tiny slice of the market which we chose to run our experiments in, and thus make predictions which may not scale up to the wider world. Nevertheless, if the alternative is not to run any experiments at all, executing them in small, lab-like surroundings is still way better.
Or, as Savoia puts it:
“Think globally, test locally.”
Pretotyping
But how exactly should we run experiments on products that don’t yet exist? We could of course build prototypes, but in many cases that process itself is time-consuming and expensive. How do you prototype a new railroad for example? An apartment building? An airline? Instead, Savoia introduces the idea of pretotyping (pre as in “before” and typing as in “prototyping”) as an early-stage alternative. The sole purpose of those simple, often throw-away, faux-products is to validate some of our assumptions, hence it is key that we build them as fast and cheap as possible.
In the book, eight distinct pretotyping strategies are covered, not all of which are suitable for each situation. But the author emphasizes that his list is by no means complete or exhaustive, and that different types of products will require different strategies, or a combination of some of them. Further ideas and examples can be found on the pretotyping website, as well as the author’s.
Here are a couple of approaches that I found most interesting, especially when what we’re trying to build is not a mass-produced physical product targeted at consumers. This space comes with its very own merits and trapdoors which the book also talks about, but my personal interest naturally lies more in the intricacies of software and services.
(1) Mechanical Turk
TL;DR: Manually perform the tasks your product will one day do automatically, but let the users think the magic has already happened.
For example, let’s say that a couple of years ago, I came up with the ingenious idea to build a mobile app that would help its users distinguish flowers by analyzing pictures taken of them. Of course, building the underlying image recognition algorithms would have been immensely time-consuming before the advent of the cloud-based AI services available today. Therefore it’s highly uncertain whether I would have been able to string together a financially viable business model around such a small, gadget-like app.
But instead of investing years into the actual R&D work of developing the required tech, I could have built an app that pretended to identify flowers automatically, but instead forwarded the photos to a human who would quickly look them up and send back the results. That way, I’d be able to gauge general interest in such an offering, probably play around with different marketing and pricing strategies, and find out whether or not automatic flower identification is actually a problem worth solving.
By leveraging services such as the aptly named Amazon Mechanical Turk, we could even run such a business at scale for a long time, whilst getting to work on the underlying algorithms required to optimize our bottom-line. And, many companies today are doing exactly that. Mary L. Gray and Siddharth Suri of Microsoft Research explored that seemingly opaque corner of the digital economy in their 2020 book Ghost Work and found that quite a few big players, even those focused on AI and automation themselves, covertly rely on human labor to provide some of their services.
(2) Fake Door
TL;DR: Pretend you have a storefront when you don’t even have a store.
By installing a fake door, we create a way for our potential customers to engage with our yet-to-be offering which unmasks itself as non-existent as soon as the interaction actually takes place. What we’re interested in mostly is the ratio between those who notice our fake door but pass on in ignorance, and those who actually come knocking.
This approach is particularly en vogue with software startups today. A great example is the way the social media automation tool Buffer got started: All they put up at first was a simplistic landing page which talked about (potential) features of the product. Once the user clicked the “Plans and Pricing” link, they were redirected to a sign-up form that unveiled that the offering wasn’t quite available yet, but allowed them to sign up for email updates.
Later, they extended the experiment to explore which pricing options would work best (i.e., resulted in the most sign-ups), and iteratively moved towards a deeper understanding of how a viable offering needed to be designed. As an added benefit, once they actually launched, they already had a list of highly interested leads that they could get in touch with to build an initial user base.
(3) Façade
TL;DR: Act as if your offering already existed in the world when it doesn’t.
The façade prototype takes the idea of the fake door one step further. It also makes customers believe that a non-existent product is already available, but it doesn’t stop at merely measuring engagement. Instead of leaving the prospect possibly irritated or confused, something actually happens and the advertised service is performed to one degree or another. For instance, a mechanical turk could kick in and execute the request.
Savoia uses the example of CarsDirect.com, an early online auto retailer. The would-be entrepreneurs around Bill Gross launched a simple website with ads for a few used cars, waited until somebody actually ordered one of them, and only then purchased the desired vehicle from another dealer and delivered it to the customer. Of course, that mode of operation wouldn’t be scalable, but it allowed them to gain valuable insights about their idea. First and foremost they managed to validate their assumption that consumers were in fact ready and willing to purchase cars online in 1999, which I imagine must have been hugely uncertain at the time.
(4) YouTube
TL;DR: Show your customers what your product will one day do and ask for their help in bringing it to life.
Remember the original Google Glass teaser? That was a great example for how a relatively simple video can be used to validate how much general interest there is in a product that’s not yet available. But be aware of the risks: If you just put together an awesome clip of your offering and count only the number of views or thumbs-ups, the siren song of those vanity metrics might lure you into building something that nobody will pay for once you’re done.
Instead, Google made a genius move: They asked potential customers to sign up for the Google Glass Explorer Program, the act of which alone already showed an additional level of commitment to the product. Then they randomly selected some of those who had demonstrated interest and invited them to New York or Mountain View for further discussions—but they had to go on their own expenses. Finally, those daring individuals had to put more than $ 1000 down to get access to a working prototype.
Even without a corporate behemoth such as Google at our side, we can do similar things nowadays by working with kickstarter or Indiegogo. If our idea, and our demonstration of its value, is promising enough, chances are that lots of actual consumers will help us to fund its development by backing us on one of those platforms. If they don’t, well, it’s still better to learn that sooner rather than later.
But Google Glass, as Savoia rightly acknowledges, offers a further cautionary tale: Even if we have identified a problem worth solving and many consumers demonstrate a willingness to purchase our solution, things can still go wrong. With Glass, the issue was that despite positive feedback from the market, many others strongly disagreed with the premise of the product. But then, who could have guessed that enabling people to covertly take pictures and videos of their surroundings all the time would lead to a societal backlash?
Skin in the game
The aforementioned idea that those who participate in our experiments need to show some level of interest (“skin in the game”) is a further cornerstone of Savoia’s methodology.
Consider what Tesla did when they announced Model 3: Potential customers had to make a down payment of $ 1000 so that they would be first in line to get a car once production ramped up—years later. That way, Tesla could be sure that hundreds of thousands of people actually would be willing to buy such a car, and had not only great proof for that assumption, but also heaps of cash to work with.
For all the pretotyping approaches discussed above, the same ground rule holds true: If we don’t get people to offer some kind of skin in the game, all experimentation in the world won’t make us much wiser. On the contrary, we might end up believing that the market truly desires what we plan to build, when in fact it doesn’t. Depending on the situation, the form of commitment we should ask for can take on very different shapes though. For a quick landing-page experiment for example, asking people for a (validated) email address might be good enough. For others, a phone number, the willingness to join a video-call, or visit us at a physical store could be more appropriate. Especially in the B2B world, it’s even not uncommon to charge a (symbolic) fee for participation in a closed beta program
Summary & Opinion
Overall, Savoia’s book offers a comprehensive framework for how to bring innovative ideas to market whilst reducing the risk of outright failure after large, upfront investments. For me, his thinking ties in nicely with agile principles in software development, as opposed to the waterfall or V-Model-based approaches of the days of yore. In either field, the traditional methodologies force us into tremendous initial assumptions that can only be validated at the very end of the entire process, once potentially millions of dollars and years of our lives have been wasted building something nobody wanted.
What I found a bit disconcerting is that the author makes us feel as though his ideas exist in a vacuum, unrelated to anything else that’s been going on in industry or academia in the last fifteen years. For example, the term MVP doesn’t make a single appearance throughout the entire text, neither does validated learning, growth hypothesis, actionable metrics, or anything else that originated from the Lean Startup cosmos.
Other interesting works on innovation culture, such as the Rework methodology of Basecamp’s Jason Fried, Simon Sinek’s Start With Why, or Clayton M. Christensen’s Competing Against Luck also don’t get any attention by Savoia. But, luckily, his ideas are a lot tamer than Fried’s or Sinek’s which makes it easier to apply them in real-world projects.
Finally, let me say one thing I rarely say about a book but that’s definitely true in this case: Savoia’s work has both the exact right scope and length. It’s densely packed with useful information and insightful real-world examples, but boiled down into a crisp 200 pages. Furthermore, I greatly enjoyed Savoia’s creative and beautiful use of language, which isn’t common in non-fiction books these days. Oh, and, his puns really aren’t as bad as he would make us believe ;-)