Questions First, Data Second
Data-driven insights address previously unanswered questions, enabling more accurate economic decisions, but this is only possible if you ask the right questions.
Welcome to the Data Score newsletter, composed by DataChorus LLC. The newsletter is your go-to source for insights into the world of data-driven decision-making. Whether you're an insight seeker, a unique data company, a software-as-a-service provider, or an investor, this newsletter is for you. I'm Jason DeRise, a seasoned expert in the field of data-driven insights. As one of the first 10 members of UBS Evidence Lab, I was at the forefront of pioneering new ways to generate actionable insights from alternative data. Before that, I successfully built a sell-side equity research franchise based on proprietary data and non-consensus insights. After moving on from UBS Evidence Lab, I’ve remained active in the intersection of data, technology, and financial insights. Through my extensive experience as a purchaser and creator of data, I have gained a unique perspective, which I am sharing through the newsletter.
Decision-making often falls short due to a lack of critical questioning. Jumping to solutions without fully understanding the problem can lead to missed opportunities and ineffective strategies. The article emphasizes the importance of asking the right questions to gain data-driven insights for effective decision-making in economics. It covers techniques like the "5 Whys", "Question Bursts", and "Jobs to Be Done1" interviews, highlighting the need for curiosity and thorough inquiry in business and financial contexts.
The critical role of questions
Extracting information to understand the problem and the outcomes needed at a deeper level is the most often skipped step in solution design. Often, we hastily jump to solutions upon hearing just a fragment of the problem, particularly if it seems familiar. Humans naturally think back to analogous situations. This can be dangerous if situations are similar on the surface but actually structurally different. It’s important to slow down and ask the deeper questions to truly understand if the analogy is appropriate or if something brand new must be designed to answer the critical questions.
Asking the right questions starts with legitimately being curious about how things work. If there weren’t natural tensions between the goals we want to achieve, tradeoffs between choices, and constraining factors, there wouldn’t even be a challenge to solve. Decisions need to be made at the point where there is tension in a choice, supported by data. However, those key breaking points in a decision process as a consumer or a business are where the economic opportunities are and when data is at its most important.
Understanding the outcomes, constraints, and trade-offs—and how each of those is changing in response to changes in the economy, competition, regulations, prices, and disruptive factors—reveals a deeper understanding of how corporate decisions and consumer behaviors are likely to play out as new information is provided.
How do we know if we are answering the right question?
“The most serious mistakes are not being made as a result of wrong answers. The true dangerous thing is asking the wrong question.” - Peter Drucker
The Clairvoyance Test
The question needs to pass the clairvoyance test. If we had an accurate and perfect answer to the question, would we have the right information to make an accurate decision? The phrasing of the question is important. It needs to be measurable, and it needs to have a time frame. We also need to start with a base-case answer. We can then test the usefulness of the question by considering different specific answers as scenarios. What decisions would we make if the answer to the question was different from our base-case expectations?
In the financial markets, it's not only crucial to have a clearly defined question that passes the clairvoyance test but also to ensure it's the question other market participants are asking.
In the corporate world, we need to know that the answer to the question and answer will be mirrored by customer behavior in the context of their economic situation, alternative products, regulation changes, and the outcomes they need.
Unpack the big question into smaller, answerable questions
The right question is often a big question that is often unanswerable today. Therefore, we need to break the question down into smaller questions that can be answered. We aim to build a mosaic of questions and answers that narrow down the potential answers.
What I mean by unanswerable is that the question requires information that isn’t available at present to be answered with perfect confidence. I believe in a Bayesian approach2 where each piece of information allows for an updated view of the probability of an outcome being more likely than other outcomes. The Bayesian approach combines multiple questions and data points together.
Answer “What’s going on here?” before “What will happen?”
In the book Radical Uncertainty, John Kay and Mervyn King take a different approach to problem solving that de-emphasizes the role of specific data points in decision-making and cares more about robust narratives, which may need to change as new information is uncovered by deeper questions. As a data-driven decision-maker, you might think I’d dislike the book, but I thought it was great. It challenges the notion that all data points or models hold value amidst uncertainty. I think successful investors who use data in their decision-making would agree that not all data points are equally valuable and that we should initially be skeptical of new data points. The narrative that the data points support also needs to make logical sense, especially in the context of the questions being asked. I think the authors make a great point that the most important question to ask before trying to predict anything is “What’s going on here?” because understanding what’s actually happening right now is more important than “what happened before” or “what will happen.”
https://www.amazon.com/Radical-Uncertainty-Decision-Making-Beyond-Numbers/dp/1324004770
Practical Methods: Approaches to asking the questions and getting to a deeper understanding
5 whys
The '5 Whys' technique is an effective method for quickly and efficiently identifying the core of a problem. It does so by asking the question "Why?" five times in succession, with each question building upon the answer of the previous one. This method helps to delve deeper into the crux of the issue at hand, avoiding the trap of focusing purely on surface-level symptoms and instead seeking to uncover the fundamental root cause.
The implementation of the "5 Whys" technique is simple, yet it requires a thoughtful and methodical approach. It begins with clearly stating the problem. This initial step is crucial, as it sets the direction for the subsequent investigation. Once the problem has been defined, you pose your first "Why?" question. The answer to this first question then forms the basis for the next "Why?" question.
This iterative process continues until you've asked "Why?" five times, or until it's no longer beneficial or feasible to proceed further. It's important to note that while five is a guideline, the exact number may vary based on the complexity of the problem. The goal is not simply to ask five questions, but to understand and address the fundamental cause.
Pro Tip: Get to the 5 Why’s without sounding like a four-year-old!
While I appreciate this method’s intent, I can’t help but think about having a conversation with a four-year-old who’s trying this approach. The tone and thoughtfulness of how the “whys” are asked are really important for this to be effective.
“Why” is considered a dangerous question by Chris Voss, author of “Never Split the Difference.” His book discusses the use of calibrated questions to uncover information. “Why” questions often trigger a defensive response. This may lead the person answering to begin defending the existing state rather than problem-solving a solution. Instead, open-ended questions that start with “how” or "what” can invite a more collaborative response that still leads to a deeper understanding. You can be much more effective by mirroring the information being provided or simply politely asking them to “tell me more.”
https://www.amazon.com/Never-Split-Difference-Negotiating-Depended/dp/0062407805
Question bursts
Often times, many people are shy about asking questions because they don’t want to appear unknowledgeable on a subject. In reality, it is the simple questions that uncover deeper understanding. A technique called “question bursts” can allow for lots of questions to be generated while allowing anyone uncomfortable asking questions in a public forum to participate.
Here's my approach. It can be done live or as a pre-scoping meeting step. Each stakeholder in the project is asked to come up with as many questions as possible about the project, both independently and anonymously. If live, set a time limit of 5 minutes. The lists are combined, and the most upvoted questions are addressed (Slido is a great resource for this).
The questions could end up being about highly informed, detailed nuances of the project, or they could be as basic as possible. Importantly, the question lists are created independently. It’s also important that the goal is to generate as many questions as possible under time pressure. This approach allows real uncertainties to emerge. It also alleviates any apprehension within the team about asking 'simple questions,' as many might have similar queries but hesitate to voice them.
For more info on the benefits of question bursts, check out: https://mitsloan.mit.edu/ideas-made-to-matter/heres-how-question-bursts-make-better-brainstorms
“Jobs To Be Done” outcome-based interviews
The outcome-based research3 approach, a comprehensive methodology, centers on understanding the desired results or 'outcomes' for users or customers. The process involves conducting in-depth interviews with users to gain a thorough understanding of their needs, goals, and challenges in the context of the service or product in question. This method shifts focus from the product or service itself to the user's ultimate goals and their desired outcomes.
In these outcome-based research interviews, questions are carefully crafted to elicit detailed information about users' experiences, their processes, and the challenges they face. These interviews provide researchers with a more nuanced understanding of the user's perspective, allowing them to identify both explicit and implicit needs.
By focusing on outcomes rather than on specific features or services, researchers and product developers can identify what success looks like from the user's point of view. This understanding is crucial for designing or modifying solutions that help users achieve these outcomes more efficiently and effectively. This could involve refining existing features, removing unnecessary elements, or introducing new solutions that address previously unmet needs.
The outcome-based research approach is instrumental in creating products or services that are closely aligned with user needs and expectations. By ensuring that the solutions provided are geared towards helping users achieve their desired outcomes, this approach can significantly increase user satisfaction and product usage. Furthermore, it can also aid in identifying potential market gaps and opportunities for innovation, giving businesses a competitive edge.
Overall, the Outcome-Based Research approach is a user-centric methodology that prioritizes understanding users' desired outcomes, enabling the creation of more effective and satisfying products or services.
To uncover the "jobs to be done" during an interview with a potential user of a product, you need to ask questions about the “pushes, pulls, anxieties and habits” as described by Bob Moesta who is the co-creator of the Jobs To Be Done framework.
The pushes are understanding what is driving the consumer or user to change.
The pulls are drawing the consumer or user to the new outcomes, for example a “have to have” new feature.
Anxiety is about changing to something new.
Habits are about their current process or the inertia of what they are currently doing.
As Bob Moesta points out, someone will not move to the new if the compelling nature of the pushes and pulls is outweighed by the anxieties and habits.
Pro Tip: Say it back wrong on purpose to get more information
Lenny (you know, of Lenny’s Newsletter and Podcast) did a great interview of Bob Moesta. The interview shared lots of practical guidance of how to practically apply the approach beyond the theory, which his partner Clay Christensen is most focused on. He also shared a great technique to uncover a deeper understanding of the outcomes needed and the context behind it. When mirroring the answers and replaying back the summary, say it wrong on purpose. It’s going to get a reaction where the interviewee is going to share much more information and get closer to the true outcomes needed by a product. Bob specifically references Chris Voss in his approach to getting a person to say “no,” first instead of “yes” because its going to open up the conversation more and reveal more information. The conversation ends at “Yes” (especially if its a “false yes”), but when trying to understand “what’s going on here” at a deeper level, it requires more dialogue and willingness to share more.
In addition to clarifying some of the nuances of the methodology with real examples, some of Bob’s personal stories help confirm neurodiversity4 is a super power.
Once you have the right questions, move on to the data
Hopefully people haven’t gotten the wrong impression from the references to Chris Voss’ “Never Split the Difference”: I’m not saying user and consumer interviews are like a life or death negotiation. The approach to questions and conversation are very helpful in a very low stakes environment.
Starting with questions makes sure the work to be done is aligned with the outcomes needed.
Armed with the right questions and a deeper understanding of “What’s going on here” will allow you and the team to move on to the data driven solution, knowing that the work means the right questions will be answered.
Join the conversation
"Found this article helpful? Share it with friends or colleagues who might benefit too.
For more insightful content like this, make sure to subscribe.
And I'd appreciate hearing from you. There’s many more methods around questions - drop a comment to join the conversation and share your favorite question techniques.”
- Jason DeRise, CFA
Jobs To Be Done: A theory and methodology for understanding customer motivations and needs in business and product development, based on the idea that customers "hire" products or services to fulfill specific jobs.
Bayesian approach: The Bayesian approach is a statistical method that uses prior knowledge or beliefs to update and revise probabilities based on new evidence or data. It allows for the incorporation of prior information and the updating of probabilities as new information becomes available.
Outcome-based research: A research approach focused on understanding the desired outcomes or goals of users or customers, rather than just the features of a product or service.
Neurodiversity: A concept and movement acknowledging and respecting neurological differences among people, such as autism, ADHD, dyslexia, etc., as natural variations within the human population.