May 7, 2017
After discovering his books as well as technical reports and research, I’ve had my head buried in anything written by Michael Mauboussin. Mauboussin’s writings are among the most lucid examinations of business and investing. After reading his latest book, The Success Equation an idea for a new post came to me. In this book, Mauboussin analyzes the relative importance of luck and skill — in sports, business, and life. Mauboussin offers valuable lessons on how to deal with luck, which I think is highly relevant to those of us working in the biopharmaceutical industry.
First, a few of Mauboussin’s words:
Luck
“Luck is a chance occurrence that affects a person or group (e.g., a sports team or company). Luck can be good or bad. Furthermore, if it is reasonable to assume that another outcome was possible, then a certain amount of luck is involved.”
Skill
“The dictionary defines skill as ‘ability to use one’s knowledge effectively and readily in execution or performance.’ It’s hard to discuss skill in a particular activity without recognizing the role of luck. Some activities allow little luck, such as running races and playing the violin or chess. In these cases, you acquire skill through deliberate practice of physical or cognitive tasks. Other activities incorporate a large dose of luck. Examples include poker and investing.”
With these definitions in hand one can place activities along a luck-skill continuum. As shown below in Figure 1 (adapted from Mauboussin’s book), activities that are influenced more by luck, like gambling and investing in the stock market, are placed closer to the right side of the continuum. While skill-dominated activities, such as chess, exist closer to the left side of the continuum.
For those that have studied the history of drug discovery and development, as well as the biopharmaceutical industry at large, the role of luck in our industry is obvious and outsized. Notable, but not exhaustive examples of serendipity in the discovery and development process include:
- The unintended discovery of penicillin in 1928 by Alexander Fleming. Fleming was studying influenza and accidentally contaminated his staphylococcus culture plates with a mold, which prevented the growth of the staphylococcus bacteria. The mold was isolated and found to produce penicillin, a substance that is highly toxic to bacteria.
- Viagra (sildenafil citrate) was originally acquired by Pfizer for the treatment of hypertension and angina. It’s safe to say Pfizer’s 2012 $2B+ worldwide Viagra sales weren’t a result of this medicine’s success in treating angina.
- Anxiolytic drugs were discovered as a byproduct of antibiotics research. Research that led to the development of meprobamate begin in 1945 in the laboratories of the British Drug Houses Ltd in London as an effort to discover non-toxic antibacterial agents. Work on meprobamate inspired the synthesis of benzodiazepines in the 1950’s at the Hoffmann-La Roche research facility in Nutley, New Jersey.
Not only does luck show up on the positive side of drug development, it also frequently rears its head when the drug discovery or development process fails. Drugs fail in development due to unintended toxicities that didn’t show up in preclinical testing, or because a lack of clinical benefit. Predicting the success or failure of a large sample of drug candidates a priori with a 100% hit rate is impossible due to the large amount of luck and uncertainty involved in the process.
The folly of rewarding and relying on successful outcomes in luck-dominated activities
The tendency of the human mind to draw a straight line between cause and effect frequently leads to the post hoc fallacy. For example, we often assume that if event A proceeded event B, then A caused B. In reality, a causal connection between A and B is often spurious. For example, some parents may believe vaccines cause autism because of a diagnosis they received after vaccinating their child. In this case, there is plenty of strong evidence demonstrating A does not cause B, but the natural tendency to form a narrative around cause and effect can easily override logic and reason.
Consulting firms that seek to decode the success of managers and organizations that have experienced success in the biopharmaceutical business often suffer from the same logical flaw. The oft-used formula typically identifies commonalities between winning firms or corporate strategies, in order to make sweeping generalities on how best to conduct business. As Mauboussin points out in his book, the flaw in this kind of analysis is due to the blend of skill and luck that makes up performance. Therefore, in heavily luck-dominated industries such as biopharma, we should expect true winning strategies to fail at a significant rate when examining the industry as a whole. Using winning biopharmaceutical firms to derive a formula for success results in an under-sampling of failure. As Mauboussin describes:
“Since we draw our sample from the outcome, not the strategy, we observe the successful company and assume the strategy was good. In other words, we assume that the favorable outcome was the result of a skillful strategy and overlook the influence of luck. We connect cause and effect where there is no connection. We don’t observe the unsuccessful company because it no longer exists. If we had observed it we would see the same strategy failing rather than succeeding and realized that copying the strategy blindly may not work.”
Let’s look at an example to illustrate the point. Imagine a world in which there exists a total of 100 biotech companies, each pursuing one of two strategies. Half of the 100 biotech firms pursue a risky development strategy that has a 10% probability of success with high payoffs; we’ll call this strategy ‘Risky Strategy A.’ Alternatively, the other half choose a lower-risk strategy that has a 50% probability of success with average payoffs; let’s call this strategy ‘Average Strategy B.’ Both strategies with their respective probabilities and payoffs are illustrated below.
Figure 2 depicts the error of omission that frequently occurs when attempting to draw lessons from the successes of other biotech companies. A consultant performing a typical outcomes-based analysis would ignore any firms that pursued Average Strategy B and only examine the 10% of firms that succeeded in pursuing Risky Strategy A. This analysis would effectively omit the 90% of companies that failed in applying Risky Strategy A. The recommendations generated from such an analysis would expound the virtues of Risky Strategy A without any appreciation for probability of success, and therefore, the role of luck in achieving high payoffs.
“When luck plays a part in determining the consequences of your actions, you don’t want to study success to learn what strategy was used but rather study strategy to see whether it consistently led to success.”
How to deal with luck in biopharma: checklists and adopting a process
Mauboussin on dealing with luck-dominated activities:
“If an activity involves luck, then how well you do in the short run doesn’t tell you much about skill, because you can do everything right and still fail, or you can do everything wrong and succeed. For activities near the luck side of the continuum, a good process is the surest path to success in the long run.”
So having the right processes in the biopharmaceutical industry is critical. What I find fascinating is that due to the outsized role luck plays poor processes can often lead to success. Because of this, companies must be careful to guard against the post hoc fallacy by examining success in isolation. Instead, executives should focus on analyzing the merits of a potential strategy or process.
An example of a thoughtful process in biopharma is the 5R framework, a five-dimensional framework developed by AstraZeneca (AZ) after conducting an exhaustive portfolio review in 2011. AZ examined success rates for 142 pipeline projects from 2005 to 2010 at the preclinical, phase 1, and phase 2 stages comparing against the industry median. The analysis uncovered a significant difference in phase 2 success rates for AZ versus the industry median (15% vs. 29%). AZ executives and scientists uncovered the five most important technical determinants of success and decided to implement them in a company-wide process, which spans drug discovery through commercialization.
The beauty of the 5R framework is that it is essentially a checklist. Checklists can be highly effective tools in dealing with probabilistic tasks that follow a set of rules or steps; AZ employs the 5Rs at every key decision point as drug projects progress from discovery to commercialization (for more on checklists I recommend Atwul Gawande’s The Checklist Manifesto). Through their longitudinal analysis, AZ was able to clearly pinpoint root causes of project failure. The aim of each bullet in the 5Rs is to therefore prevent the organization from repeating past mistakes. Developing drugs by using the 5R framework is akin to using basic strategy when playing blackjack at the casino — over the long run it’s likely to give AZ an incremental statistical advantage against the competition.
In summary, the relevant lessons from Mauboussin’s The Success Equation for the biopharmaceutical industry:
- Biopharma lies near the luck side on the luck-skill continuum.
- Don’t study success when deriving a new process or strategy; instead, study strategy to include both winners and losers and resulting probabilistic implications.
- Adopting a thoughtfully conceived process is the best way to deal with luck in biopharma; checklists can be a critical process component.
Questions or comments? Email Garrett at garrett@oncologydiscovery.com
Disclaimer: All opinions expressed on Oncology Discovery are my own and do not necessarily represent the position of my employer. The information presented within this article is not a solicitation for investment.
Copyright © 2017 Oncology Discovery. All Rights Reserved. Unauthorized use and/or duplication of this material without permission is strictly prohibited.
Comments are closed.