Embarking on product R&D is akin to a scientific expedition; define your hypothesis and success criteria as the North Star. Envision them not just as checkpoints but as constellations guiding your journey. Picture your hypothesis as a bold hypothesis, a daring exploration into the unknown, and success criteria as the compass steering your ship. It's not merely an experiment; it's a visionary quest. Think of your R&D endeavors as uncharted territories waiting to be conquered, with each hypothesis and success criterion charting the course. As the modern-day explorer, your map to feasibility and scalability unfolds, transforming experimentation into an epic adventure where innovation meets strategic success.

When doing exploration and research about products, you are creating a couple models. The first is that there is an unaddressed NEED out there, which you can build some THINGS to address; and if addressed, will result in an OBSERVABLE CHANGE. The second model is that if you can cause or enable an OBSERVABLE CHANGE it will lead to an OUTCOME which is good for your business. Most experimentation is of the first model, and you develop a solution hypothesis which describes your belief. The hypothesis needs to be disprovable - if you run a test and don't see particular results, you need to consider the approach to have not succeeded.

Hypothesis: Formulate a clear prediction. E.g., "Adding a 'Recommended Products' section will increase average order value by 10%." Key Metrics: Identify what you'll measure, like average order value. Success Criteria: Define clear, quantifiable success, such as a 10% increase in the key metric with 95% statistical significance. Assumptions: Note any underlying assumptions for context. Analysis Plan: Plan your data analysis method, considering sample size and test duration. Review: Post-experiment, compare results against criteria, and use insights for product iteration.

Choosing the right experimental method is driven more by your constraints than anything else. One set of constraints comes from your cadence of experimentation and time pressures. Do you have a process where you need to learn something every week? Are you running a one-off experiment to de-risk a specific element of product strategy? The second set of constraints emerges from your solution hypothesis. Are you trying to observe direct measurable improvements in a process (fix something broken and see X improvement), something outside of your control (improve a customer experience and see Y change in behavior), or something you must infer (if X were improved, we would expect to also see Y)?

Test small, gather user feedback, analyze data, and ensure alignment with business goals. Gradually invest more as experiments prove successful and scalable.

Your experimental process (testing paper prototypes with 7 people, launching a concierge mockup, etc.) is driven primarily by the constraints of what you must observe and the level of fidelity in results you need to achieve to meet the conditions of generating an insight from your results. Always define the way you will interpret results before running the experiment. Most experiments are straightforward to run, although the ones based on inference are hardest to conceptualize and leave the greatest room for skepticism even after evaluating results.

The choice to iterate your experiment, just as the decision to originally run it, is one based on removing uncertainty from your original beliefs. Your models of how a product will work is fraught with implicit jeopardy. Non-trivial products embody more than one belief - multiple customer groups, multiple contexts of usage, multiple issues to address to varying degrees. A good heuristic for jeopardy is the combination of the likelihood of being wrong with the consequence of being wrong. If, after your experiment, this hypothesis is still the greatest source of jeopardy, design a new experiment for it. If not, move on to the next source of concern.

Scaling an experiment to a larger audience is about exploring the extension of findings into a broader context. Can you extrapolate from what you observed with a single group of people in a single situation to other groups and other circumstances? This is really just one aspect of iterating an experiment - considering the applicability of findings to a broader audience. You could just as easily iterate on other dimensions (if we did less, would it still work? if we addressed things in a different way, would it still work?)

Maintaining an impact map which combines your models of behavior and hypotheses is the most effective way to create a shared understanding of how you believe things work. This model is also useful for creating a shared understanding of where the jeopardy in the plan lies - where beliefs, when wrong, hurt the most. As you learn through experimentation you reduce the uncertainty which creates implicit jeopardy in the plan. This allows you to make an informed choice about when to launch your product or the update to your product.

There is a beautiful article called "A more rational approach to new-product development" by Eric Bonabeau and colleagues where he explicitly talks about this dilemma. Dr. Bonabeau suggests separating the team in certain cases, committing the first team to the Truth and the second team to the Success of the new product.

Uncertainty can never be fully eliminated. You have to balance the risk of being wrong (with your models) and the risk of being late (with your product). Those decisions are a function of the risk-tolerance of your organization. Key factors to consider - how well do you understand your customers, and how agile are your competitors? Further, you want to understand how rapidly the playing field changes, through evolution of technology, through changing customer expectations, and through changing competitive responses to change.

Start with a small pilot, like testing a new feature with a select user group. If successful, gradually expand, ensuring resources and processes can handle growth. Constant feedback and data analysis are key.