The Single Metric That Reveals Whether Your Data Product Will Succeed
A practical framework for measuring repeat usage and linking it to value, adoption, and budget decisions.
In the world of data product teams and insight-driven companies, a familiar scene unfolds. A new dataset is launched, early adopters arrive, and leadership waits for proof that the product is driving value. By the time that proof shows up in ROI reports, revenue attribution, or alpha1 calculations, the moment to act has often passed. Markets have shifted, decisions have been made, and the product’s trajectory is already set.
Early in my career as an equity analyst, I saw this pattern clearly with scanner data2. Every major firm had the monthly numbers. Everyone paid for access. Yet a question hung in the air. If the data were widely available, why did analysts and investors still race to interpret it the moment it was released? The insights rarely shifted consensus3 in dramatic ways. Most simply confirmed what the market already believed. And still the dataset became indispensable.
I only understood why after years spent building data products. The products that succeeded were not the ones that consistently produced alpha. They were the ones that became embedded in the user workflow. These were the data tools people returned to at critical moments, and the ones they would immediately miss if they lost access. Repeat usage created trust, embedment, and eventually pricing power. Usage, not outcome, was the earliest and most reliable signal of long-term value.
This article will
Explain why lagging indicators fail to guide investment decisions
Explain the case for repeat usage as the leading indicator
How to measure repeat usage (operational playbook)
How to use repeat usage in an ROI and budget framework
Welcome to the Data Score newsletter, composed by DataChorus LLC. This newsletter is your source for insights into data-driven decision-making. Whether you’re an insight seeker, a unique data company, a software-as-a-service provider, or an investor, this newsletter is for you. I’m Jason DeRise, a seasoned expert in the field of data-driven insights. I was at the forefront of pioneering new ways to generate actionable insights from alternative data. Before that, I successfully built a sell-side equity research franchise based on proprietary data and non-consensus insights. I remain active in the intersection of data, technology, and financial insights. Through my extensive experience as a purchaser and creator of data, I have a unique perspective, which I am sharing through the newsletter.
Understanding why repeat usage is so important requires first understanding the limits of the metrics many teams rely on today. These lagging indicators can validate success long after the fact, but they cannot guide product or budget decisions in real time. To see the problem clearly, we need to examine how these indicators form and why they fail to provide the timely signals that data product leaders depend on.
1. Why lagging indicators fail to guide investment decisions
The decision chain problem
Data product companies and centralized data teams within companies have difficulty in assessing the ROI of their data products because the data product is ultimately enabling users to make better decisions.
Data products generate trusted insights that directly enable the economic decisions and outcomes needed by the user.
The decision falls to the user, which then sets off a cascade of actions and market or competitive responses that may lead to a positive or negative outcome.
Time-lag examples
Consider the chain of events that connects the data product-driven decision to the end outcome.
For business decision makers competing in an industry, they may make marketing, pricing, selling, or other decisions, which will generate a response from competitors and consumers. The actual ROI of the data product cannot be directly linked to the confirmation of results because it will take time to execute and monitor the response, over which time many factors beyond the decision set could affect the results (economic, regulatory, etc.).
In financial markets we talk about the alpha generation of the investment decision. However, while the decision may ultimately prove correct, the alpha won’t be observable until a later date. The market is inefficient in the near term and efficient in the long term. The catalyst where the market learns the truth and proves the investment thesis right or wrong may be the next quarter’s results or many quarters into the future for longer-term investors.
And furthermore, this assumes that the data product has shown a surprising insight that leads to an active decision instead of sticking with the current plan.
In financial markets, the data products are used to find an insight that is different than the consensus. But, we shouldn’t be surprised if consensus and data products are mostly in agreement since the market is digesting info rapidly and reflecting it in security valuations and consensus estimates.
The same analogy plays out at a slower pace in industries leveraging data for decision-making.
Attribution problem
Now let’s consider a future state where the economic outcome is known and measurable. There’s still a challenge to assign the value of the outcome back to the data product’s insight that triggered the decision to act.
In investing, it’s a challenge to then convert that alpha of an investment decision into a return relative to the cost of the data product, especially if it’s a fundamental discretionary investor4. While a systematic5 investor could isolate the alpha contribution of a data product signal by comparing the model with and without the data point, a discretionary investor applied human judgment in an informal Bayesian process. How much was the data point versus the investor’s good judgment? What about the value of other data points—traditional, alternative, and anecdotal? And even if one could split those factors with many assumptions in a model, the internal data product team and the data product company would want their users to be the star of the story.
Similarly for corporate decision-makers, assigning value from a known outcome back to the original data-driven insight is a challenge. How much of the decision was based on the specific data point compared to other data points, the decision makers’ good judgment, or collaboration across multiple teams to execute the idea effectively in a competitive market?
I’ve always felt that for my data products, I want the decision maker to be the hero of the story. Taking claim for some or all of the outcome after the fact puts an unneeded tension into the dynamic between data product creators and users.
Consequences for product and budget decisions
While the investors are waiting for proof that they were right (aided by the data), data teams and companies won’t know the value of their data either. That’s a problem because in a constrained resource world, tough decisions about which data products to invest in and which to cull are needed on a more frequent basis.