Testing the Future: Navigating Uncertainty in Emerging Products
How to unlock the creativity required to test the value of products that are hard to build and don't yet exist.
When building a product that solves a problem that has never been solved before, it’s easy to conclude that it cannot be tested or that it must be built first to gauge viability. There is no doubt that emerging technology products are the hardest to test, and in many cases the tests themselves are imperfect, but doing no testing for these expensive and deeply difficult challenges is simply a failure of imagination. I’d like to share three ways for reducing market uncertainty even against these headwinds.
Find a Way To Fake It
My favorite method is to come up with a way to deliver nearly the same value to the customer without having to actually build the product. In the early days at mePrism, we were building tools to help consumers get their private data (search history, YouTube watch history, friend lists, advertising interactions, etc) back from the tech titans, and then we helped them monetize it with brands they wanted to sell it to. Perhaps unsurprisingly, the tech titans did not make it easy to reclaim that data on the user’s behalf. If we wanted to test, for example, the value of a person’s Spotify music play history with a buyer, the assumption was that we first had to build a first class integration with Spotify. The same goes for if we wanted to test the consumer’s willingness to sell Spotify information. A simple way to fake it was to avoid trying to perform a full integration. We could achieve the same result and test the same value propositions by offering people cash on Craigslist to collect their Spotify histories themselves through the manual process all companies subjected to GDPR or CCPA have to follow. By telling the consumer who the data would go to if they transacted, they were in essence performing the same exact end-to-end workflow that we would be offering in the product, and we didn’t have to build a thing. This sort of validation strategy taught us an immeasurable amount about what markets would sustain, what data was most interesting to businesses, and what data consumers were willing to collect and sell.
But this example pertains exclusively to software, and I’d argue that the hardware-software hybrid startups have an even harder validation job. Last year, one of my clients was building an advanced, first of its kind satellite constellation that does remote sensing of other satellites in space. Imagine the owner of a $250M satellite wondering if physical damage to their space vehicle is repairable. The customer needs some images of the satellite to really know how bad it is, but those images cannot be captured from Earth. It would be easy to give up here and assume that you have to build a space-worthy vehicle with world class optics before you can test this. My advice was to first fake it on the ground. A buyer of this sort of service would need to know what an image of a satellite taken from space would look like and what sorts of details it might reveal. For this challenge, there’s an easy option, and a more aggressive option. The easy option is to create a design rendering of example scenarios and use these with customer prospects. The higher octane option is to take the same sensors that would go into space, set them up on Earth, in the dark (in the desert), and take images of a real satellite, also on Earth, at a similar distance. Of course, companies need to be honest with their prospects about whether they are seeing the real thing or a synthetic, but it’s crucial for the buyer to really get to appreciate the exact thing they are purchasing to get true feedback. In this case, is the camera resolution high enough? Can the image show the type of damage desired? Is a different sensor payload required to capture a different type of observation? All of this can be de-risked ahead of buying components, building and assembling, reserving rocket capacity, and launching into outer space.
Test the Assumptions Instead
The second method is to test the assumptions underlying the product instead of testing the actual product. Again, this requires some creativity on the part of the product team. A lot of new companies are generating text from AI such as through ChatGPT for different domains, and the work of labeling, training, and tuning a new model is non-trivial engineering. Consider a startup working to assist marketers in writing marketing copy for websites. One crucial assumption is that users would trust a bot to write these important materials in the first place. This assumption can easily be tested by creating text for specific products that mimics what the AI would produce, placing it alongside prose written by humans, and setting up experiments to learn how much value having it faster and cheaper has over the perfect human response. No need to build a thing.
Sometimes the market itself gives us evidence of the answer to the test. For example, in the consumer and commercial drone markets, both types of users want longer battery lives so they can fly for longer. The longer the drone is in the air, the more value it can provide in either context. A company working to build a next generation drone with 2X the battery life can analyze this trend in the market and have reasonably high confidence that a drone with all the same characteristics and yet twice the flight duration, will be accepted. Reading reviews of existing drones and noting an overwhelming desire among those reviewers to have longer battery life probably is a sufficient start. I would always caution to understand the upper limits of these assumptions too, however. It’s not a problem today since most drones have such short battery lives, but at some point, building a 1,000-hour battery life drone, for example, won’t be a meaningful improvement if the past generation already goes for 500 hours, and actual users never fly for more than a few hours at a time. At that stage, even a seemingly obvious improvement may lose the power to compel a sale of the product.
Your Deck is Your Product
When all else fails, perhaps my favorite value testing hack is the sales deck. When building complex emerging technology products that cannot be instantiated in the world for years, your sales deck is your product. What I mean by this is that the specific value propositions, bullet points, and context shared in the sales deck is what you are selling until the product becomes real. If you can’t earn pre-orders from the sales deck, this is functionally similar to when the product exists and users won’t buy it. Change the deck (in essence, change the solution/features), and re-test and see what happens. Once you can sell the value through the words on the page, you have strong validation that what you’re building is real, at least with business buyers. It’s really fun too because these are absolutely the fastest, cheapest features you’ve ever built.
A Note on Ethics
I should caveat that for this entire set of suggestions, ethics are paramount. If you can’t eventually build what’s in the deck or deliver what’s in the faked-out test, it doesn’t belong in there. If you pretend that everything in your deck exists now, and it doesn’t, then you’re no better than Theranos, and jail time is in your future. Same goes for pretending the fake versions are actually the real things. I find that customers with real problems to solve are more than willing to entertain future-looking statements, and even assign future cash to those solutions, so long as the people doing the selling and validating are honest about what’s possible now and what belongs to the future. An easy way to handle this is to make the contracts contingent on successful delivery of the stated capabilities.
The Exception
Finally, when a team has unlimited budget and the best engineers on the planet, and when they can build things quickly, by all means, those prototypes make even better tests than anything above. For the very few where this is true, keep on hacking and enjoy, but for the rest, or even for those incredible people, when the problems are more intractable, consider these options.
Hi! If you enjoyed these insights, please subscribe, and if you are interested in tailored support for your venture, please visit our website at First Principles, where we focus on product to help the world’s most ambitious Founders make a difference.