8 point approach to evaluating data partners
The first pitch and demo look great. But then we look under the hood…
Welcome to the Data Score newsletter, your go-to source for insights into the world of data-driven decision-making. Whether you're an insight seeker, a unique data company, a software-as-a-service provider, or an investor, this newsletter is for you. I'm Jason DeRise, a seasoned expert in the field of alternative data insights. As one of the first 10 members of UBS Evidence Lab, I was at the forefront of pioneering new ways to generate actionable insights from data. Before that, I successfully built a sell-side equity research franchise based on proprietary data and non-consensus insights. Through my extensive experience as a purchaser and creator of data, I have gained a unique perspective that allows me to collaborate with end-users to generate meaningful insights.
In the early years of Evidence Lab, it was rare for a week to go by without multiple potential data vendor presenting to us. Each of the original team members would find relevant data vendors to address pressing investment debates that we were working on with our analyst colleagues.
Prior to 2019 the Alternative Data Council at FISD1 did not exist. The Due Diligence Questionnaire (DDQ) is now commonplace (https://fisd.net/alternative-data-council/). However, UBS and Evidence Lab maintained the highest standards for risk management, which provided the guidelines for the approach we adopted. It seems we weren’t the only one who felt this way about managing the risks associated with data procurement. Eventually, it has become industry standard.
I’d like to share some of the guidelines that are important to me when assessing data partners.
Use a completed Due Diligence Questionnaire (DDQ) to understand the compliance and risk associated with a dataset.
Assess the Return on Investment (ROI) by considering how many decisions can be influenced and the potential limitations of the data.
Conduct common sense, first principles tests to ensure the data behaves as expected and reflects known events and expected seasonality. It’s surprising how often these types of tests are failed.
Perform back testing against benchmarks to measure the dataset's correlation with a known KPI, while avoiding common statistical mistakes that lead to incorrect conclusions.
Assess the transparency of the methodology used for harvesting, cleansing, and enriching the data, while respecting proprietary trade secrets.
Evaluate how the data vendor handles feedback and whether they have the capacity for custom work, understanding the potential implications on competitive advantage.
Understand the vendor's competitive set by asking about their closest competitors and their target customer base.
Examine the Service Level Agreement (SLA) for post-delivery service, including response times for errors, communication of code-breaking changes, and availability of sales engineering support.
The article provides details on each of these steps, which should better help data providers understand the rigorous process of a data buyer. It should also help data buyers compare/contrast their process to a benchmark for what good looks like.