Data Deep Dive: Web Mining Shows Inflections in Apparel Sector Results
Pricing data reveals strategy and performance inflection points for apparel companies, providing an early warning system for investors for upside and downside risk versus consensus estimates.
Welcome to the Data Score newsletter, your go-to source for insights into the world of data-driven decision-making. Whether you're an insight seeker, a unique data company, a software-as-a-service provider, or an investor, this newsletter is for you. I'm Jason DeRise, a seasoned expert in the field of alternative data insights. As one of the first 10 members of UBS Evidence Lab, I was at the forefront of pioneering new ways to generate actionable insights from data. Before that, I successfully built a sell-side equity research franchise based on proprietary data and non-consensus insights. Through my extensive experience as a purchaser and creator of data, I have gained a unique perspective that allows me to collaborate with end-users to generate meaningful insights.
Web-mined pricing and inventory data
Monitoring price trends is an incredibly important analysis for investment professionals and corporations. The price of a good or service is the only way any business monetizes its competitive value. Prices are not set randomly. Companies agonize over the correct price of their product or service to be able to maximize market share, maximize margins, and effectively manage inventory. Observed price actions provide insights into a brand's equity and corporate strategy.
This article goes into more details as a dataset deep dive explaining the use cases, methodology, and limitations associated with the analytics. The goal is to help insight seekers better understand the potential of various types of datasets and to help data providers think through how they can align their products with the outcomes needed by their clients.
Recent interview on the topic
I was recently interviewed by Ascential’s Alternative Data team, where we discussed the value of web-mined pricing data, including some examples. Check out the videos here to get:
Why collect prices with a web mining approach?
By collecting prices systematically with web mining1 techniques and applying the fundamentally driven analytics used by each industry to the data, one can reverse engineer the company’s pricing strategy and assess their ability to execute against that strategy.
The corporate behaviors around pricing are hiding in plain view, as seen in the price changes and product availability across the web.
By leveraging technology to collect and process the data, the insights can be revealed as early indicators for investment decisions.
This allows for improved forecasting of company operations, including the ability to catch inflection points in company performance.
To assess price strategy and execution, pricing data is gathered from multiple public sources across the web. With the application of appropriate analytical tools, millions of data points collected each week are cleansed, enhanced, and analyzed to provide comprehensive insights into the pricing strategies of corporations.
When analyzing web-mined price data, it's important to use the frameworks used by the industries themselves in order to see the data the way business decision-makers see and react to it. For example
Fashion industry pricing analytics should be built around merchandise planning frameworks,
Pricing in the consumer staples industry should be analyzed around category replenishment as a framework.
Pricing that is set based on demand relative to fixed supply, such as airlines or hotels, means that web-mined pricing data analytics should follow yield management as the framework.
Some products and services are priced together, which requires a framework to make bundled offerings comparable in price.
Financial products like mortgages and savings products require pricing data to be analyzed differently than other sectors, for example, by considering the offered rates relative to the underlying interest rate costs associated with the business.
In this deep dive into web-mined pricing data for fashion sector use cases, we are going to
Start with the common questions addressed with the data
Describe the underlying data sources
Explain the thought process behind cleansing the data
Discuss how the data should be enriched, including relevant metrics
How to use the metrics to find insights
Explore the limitations of the data to ensure appropriate use of the insights
Common questions addressed with the data
Fashion Pricing: A Deep Dive
This deep dive into pricing will focus on the fashion pricing use case. In future entries in the Data Score Newsletter, we will go into detail on how to generate valuable insights from pricing data in other sectors.
Is Nike able to manage its elevated inventory levels back to historic levels while maintaining its merchandising margins?
By collecting price and product availability on Nike, Adidas, and other competitors direct-to-consumer websites as well as from sportswear retailer websites, one can reverse engineer Nike and its competitors execution by monitoring like-for-like price trends through the selling season compared to prior years at the same time as monitoring the y/y trend in inventory age. If inventory age is rising and prices are falling faster through the selling season, this is a negative signal, meaning the lower prices are still not clearing the inventory. If its prices are holding firm while inventory age is falling, that's a positive signal. Furthermore, the retail channel data harvested can provide best seller share trends as a proxy for demand market share shifts.
Considerations: different sportswear companies have different product and geographic contributions to their total revenue. It’s important to make sure multiple geographies are collected as part of the data harvest. Classification is also important for footwear, apparel, and accessories, as each company has a different mix. It's also important to compare the direct and retail channels, as they can give different signals.
How are fast fashion companies handling the increased competition from SHEIN's lower price points?
There are multiple alternative datasets that should be used to monitor the rising popularity of low-cost fast fashion brand SHEIN (transaction data, clickstream data, Google Trends, Social Media interactions, etc.). However, an important part of the equation for the public companies is there response. Web-mined pricing and inventory data focused on the percentage of new SKU2s introduced by each competitor as well as monitoring the distribution of retail prices to see if the companies are introducing more low-priced products in response will help assess if there is an actual response from the companies. This, in addition to the approach outlined above, will monitor the execution of merchandise planning via price reductions and inventory age to reveal if the strategy is working.
Considerations: When new entrants have a large disruptive impact on industries, it’s important to assess the situation from multiple angles. A new entrant's adoption curve typically follows a sigmoid (S-curve) pattern. However, incumbent competitors can influence when adoption plateaus through their strategic actions. The defense of their market position could be seen in other datasets in terms of overall awareness, brand health, and revenue trends. But the underlying cost of the defense versus the new entrant could be seen in the pricing dataset.
Can Luxury companies introduce new price increases in China?
As a long-term trend, China has been a source of profit growth for luxury companies. Historically, web-mined pricing data shows that prices for identical items are more expensive in China compared to other markets. However, when price gaps widen too far, luxury shoppers will travel abroad to make there purchases, limiting the ability of luxury companies to increase prices in China. As the economy in China reopens, demand for Luxury products could be high, but a return to traveling abroad and allowing luxury shoppers to purchase luxury products cheaper in other countries could limit the ability to raise prices in China. Monitoring both the like-for-like price trend on evergreen items3 and the price positioning of identical products in China compared to the US and Europe could help assess if price increases are happening and likely to stick.
Considerations: In addition to the harvested prices from the web-mined approach, it’s important to adjust the prices to be in the same currency and also adjust for VAT and other taxes, which may be included in the price but refunded to foreign travelers. Pairing this data with datasets on duty-free sales and travel trends will help assess the overall picture with more clarity. Luxury companies like LVMH and Kering typically raise prices at specific moments and then hold prices firm, so if price increases are spotted, it’s important to build in alerting into the product pipeline because the news will make it into the market quickly through other sources (like in-person channel checks).
How will apparel inflation affect US "back-to-school" shoppers in 2023?
Using data from retailers that sell multiple fashion brands, monitor the best-seller share of brands compared to the like-for-like price trend. If inflation is beginning to have an impact on the consumer, we should see higher price sensitivity where products that reduce prices sharply move up the best-seller rankings faster. However, if brands and product appeal are more important, the price trends would not be as correlated with the best-seller rankings.
Considerations: When aggregating the sequential, like-for-like price trends, it’s important to seasonally adjust the trends to match the core CPI methodology. Choosing the right sample of retailers that sell multiple brands is important for the analysis. The retailers need to be representative of the broader market. See the limitation section of this article for reasons why 3 to 5 retailers should be used for the sample.
These questions can be addressed even more accurately by combining multiple datasets together to fill in the limitations of each dataset. The approach can either be done by applying a Bayesian approach to probabilistic outcomes or by combining datasets in a nowcasting model.
As a summary, the table below outlines sample investment outcomes needed, the related metrics, and the logic connecting the two.
This isn’t a complete list of use cases and investment debates to be addressed.
Data providers operating in this space should feel free to chime in with other use cases in the comments. Let’s get the conversation going.
Collecting the Underlying Data
Web-mined pricing data refers to the gathering of price and product data by systematically harvesting product listings across retailer sites, brand sites, and other sources. Key fields collected include unique identifiers, pricing, best-seller rankings, and metadata product information like brand and categorization. Compliant scraping is essential to minimizing site disruption.
The underlying data is harvested from various web-based sources, including retailer and manufacturer websites, as well as third-party data sources when retail prices are not available.
Web robots need to be designed to systematically gather the entire, public facing assortment of products and prices, including metadata about the product. Important fields to gather include:
Unique identifier for each SKU. Note that websites may use their own identifier system or leverage a system that works across multiple sites.
current price (after promotion)
Retail price (also called original price, list price, or price before promotions)
The department, category and subcategories where the item is located in the navigation
The brand name
The product name and short description of the product
Package size, multi-unit size,
Geography
Price Currency
Collection date
Best Seller Rank
Website classification as a branded website (direct channel) or a retailer of multiple brands (retail channel)
The most important challenge in web-mined price and inventory analytics is the product discovery process. This is deep web mining. The easier part is collecting the relevant information from the website once the URLs of the products are known.
Plan ahead
When the web robot first turns on, that will be the start of your history. It will take time to build up enough history to automate anomaly detection, and even more time to build up enough history to assess the takeaways from each new season’s performance. Third-party data companies specializing in web-mined price and inventory collection will have longer histories available for purchase.
Have a consistent schedule
The web robots need to be on a consistent schedule. We need to consider the data harvested as a sample of the real-world available products and their prices. All price and inventory changes cannot be completely captured while maintaining a low traffic footprint on the website. So we know the data is going to have noise and bias compared to the real world. Maintaining a tight schedule helps make the noise and bias consistent by capturing the price and assortment at the same time each day or week. A constant bias and level of noise can then be removed by appropriate trend analytics, allowing for clear signals to be generated.
Have compliance and risk management procedures in place before turning on your first web robot
It is important to have policies and procedures in place to ensure compliance with industry standards. The web mining work needs to follow compliance standards.
When collecting the data, it is extremely important to not disrupt the operations of the website, which means minimizing the number of page views and maintaining a slow pace of collection that blends in with other traffic. Any information behind a login is off-limits. Any information gathered by putting items in a basket is off limits because it could prevent a paying customer from buying the item. It’s also important to be careful about collecting and storing trademarked content. Have procedures in place to deal with communication from website owners to stop scraping the site. A deeper conversation about data web mining rules and guidelines will be covered in a future entry in the Data Score Newsletter.
Cleaning the Data
After collection, the data will need to undergo extensive cleansing and preparation, including deduplication, parsing, anomaly detection, classification, and metric generation. Web-mined data in its raw form is extremely messy and, frankly, unusable. The ability to catch incomplete web scrapes and identify outliers to ensure high quality datasets is critical. The ability to parse relevant information into clean information makes enrichment and analytics possible.
Unit testing
The challenge with unit testing4 in this case is that the website is the golden source of truth, but if we already had that golden record source in a database, we wouldn’t need to web scrape it! So the unit tests are looking for signs that the run didn’t complete as expected.
Effectively, we’re testing to identify “incorrect nulls and zeros”.
If a website usually has 10,000 products on it, in the current web robot run, only 8,000 are collected. Are the 2,000 missing from the collection accurately reflecting that they have been removed from the website? Or did the web robot make an error interpreting the website, and the 2,000 items are still there?
The unit tests need to be completed at the end of each run and compared to historical collections. Harvesting errors will need to be addressed quickly because the prices and assortment on the websites will change eventually, and it’s not possible to go back in time to collect the prices. If the robot cannot be adapted quickly to the website’s reality, the collection will need to be completely removed from the database. This approach is better than accidentally leaving an incomplete run that distorts the analytics.
Each fashion company will have a unique seasonal cadence of item availability, so the total number of records collected needs to be compared not only to the prior run but also to runs in prior years at the same part of the selling season.
The number of records collected as well as distinct SKUs need to be checked. The number of departments, categories, sub-categories, and brands needs to be tested as well.
We’re looking for signs that the robot did not complete the collection, so thresholds need to be set to trigger manual interventions where a human reviews the website to confirm the robot collected the site accurately.
As more time passes, the calibration of the thresholds for exception checking improves.
Parsing and cleansing rules at the row level
The information collected will be embedded in text strings and needs to be separated into a well-defined schema with appropriate data types.
Price conventions vary across websites and geographies. For example, European websites may use a comma instead of a decimal to separate the cents from the euros. Brand names may be embedded in the product description. Package sizes may be embedded in the product name.
Machine learning with natural language has continued to improve the efficiency and effectiveness of parsing price and metadata information.
Deduplication
Websites are designed with the customer journey in mind, which may mean that products are listed in multiple locations on the website. The shoes could be in their own category, paired with bags as a cross-selling effort, in a promotion category, or in a most popular item category.
From an analytical point of view, we need to deduplicate the records such that each website, geography, data harvest date, and SKU identifier exist only once in the pricing database. There’s additional logic needed for products where price ranges are provided across different SKU variants. This is also a point where robot logic should be checked as a test.
We should not see different prices for the same SKU variant on the same run in the same geography on the same website. This event should trigger an investigation of the website and the robot.
For deduplicating the SKUs and deciding on which best seller rank to main
Anomaly detection
As part of the data pipelines, individual SKUs and aggregate metric trends need to be tested against history to find abnormal activity.
For example, sometimes web robots and databases are confused by commas in European prices, which leads to the price of a “€20,00” on the website showing up as €2000 in the database instead of €20. A check of more than a 900% increase or 90% decrease will catch these breaks in parsing.
It’s important to remember in price time-series testing some basic math that if an item is 50% off one week and full price the next week, the price increase is 100% sequentially. So don’t set your price tests with equal upside and downside movement thresholds. Also, a 50% price cut is not uncommon; even a 70% drop in price for fashion brands is possible. The anomaly detection at the SKU level needs to be calibrated with an appropriately wide range to avoid triggering too many false alarms.
Another check is to look at the distribution of prices to find outliers in the range. If 99% of the prices are between $5 and $150, the anomaly detection should be set up to catch times when the price parsing resulted in cents or thousands of dollars inaccurately.
The end metrics needed to be tested in trend versus history, looking for changes in trend by more than a standard deviation or two, depending on the level of accuracy needed and capacity for manually checking exceptions. It’s important to test not only against the entire history but also factor in seasonality into the checks. Key aggregated metrics to test include
Average and median prices
Promotion depth and breadth
Number of like-for-like items
Number of new SKUs
Number of removed SKUs
It’s important to remember that the design of the analysis benefits from the law of large numbers. As metrics are aggregated, the noise of individual SKU anomalies is reduced. We don’t need perfection in the data harvest to get actionable insights. Having a process to catch when the robot is incomplete or the price parsing has led to extreme outliers that can move the aggregate measures incorrectly is the primary focus.
Web mining community - there’s definitely many more checks that should be executed that I’ve left off the summary above. Do you believe any of them should be listed here as the top priority?
Enriching the data
Additional techniques like classification and bespoke metric development allow for tailored analysis frameworks suited to different industries. For example, fashion pricing analytics focuses on metrics aligned with retail merchandise planning.
Classification
Even with the narrow focus of this article on web-mined pricing for fashion companies, it’s important to classify the items to improve comparability. Web mining and metric creation result in observation-weighted metrics, which can differ from demand-weighted metrics. Demand data is not freely and publicly available on most websites. In the absence of demand data, it’s important to categorize the items collected to improve comparability between one site and another. One retailer may do more business in the footwear department compared to a similar competitor who has more exposure to handbags and another that is more focused on apparel.
Classification has come a long way from the early days of web mining. In the beginning, much of the classification was handled by regex5 looking for key words. For example:
Dress → Apparel,
Shoe → Footwear.
But this breaks down when the lexicon-based approach tries to classify “Dress Shoe.” Natural Language Processing and Machine Learning are much better at classification, and Large Language Models are even better at classification into primary categories.
Similarly, brand classification is very important when harvesting data from websites that sell many brands. Some websites do not provide the information in its own field, which requires parsing to identify the brand. This is improved by machine learning approaches. However, it’s also important to consider what brands are important to break out and leave the rest to “other” in the final aggregations. This allows for more time and effort to be focused on brands that matter, while retaining the information of other brands in case they should be broken out in the future.
The brands that are mapped in the data should be further mapped to tickers for ease of use by the investment community.
Metric creation
When analyzing web-mined price data, it's important to use the analytic frameworks used by the industries themselves in order to see the data the way business decision-makers see and react to it.
Merchandise Planning is used when the goal of the inventory management system is to end with zero inventory.
Think about seasonal categories like winter apparel that must be cleared to make room for the spring season.
In this structure, the initial price is the maximum price the company will aim to achieve, but from there, there is only one direction for the price: down.
The company needs to navigate competitive pressures and changes in the health of consumers to successfully manage its inventory.
The company will continue to drop prices while monitoring the rate of sale and remaining inventory levels to maximize the price achieved while reaching zero inventory.
Note: This is different than category replenishment where the items are constantly replaced on shelves For example, when the milk is sold out, the milk inventory is replaced on the shelves. In this system, prices and product mix are managed to maximize market share and margin. Also note that merchandise planning can apply to other seasonal categories, not just fashion.
Critical metrics
The metrics listed below are the ones that I believe are critical to being able to reverse engineer pricing strategies and monitor if companies are tactically executing well.
Sequential like-for-like price change based on the start of the selling season and compared to prior year’s selling season
Year-over-year change in median inventory age
Year-over-year change in sell-through rate (% of SKUs no longer available)
Year-over-year change in discount factor (discount depth x discount breadth)
Year-over-year change in full price factor (1 minus discount factor)
Year-over-year change in percentage of new SKUs
Year-over-year change in percentage of removed SKUs (sold out)
Year-over-year change in total SKUs
Year-over-year change in price distribution
Year-over-year Best-seller share percentage (for retail channel)
Analytics
So how would we use these structures to assess how a company is performing in Merchandise Planning? What metrics matter most?
Let’s take a fast-fashion apparel company like Inditex. The web-mined data on Inditex shows its historically best-in-class merchandise planning performance. The items are introduced at full price, and as the measured inventory age rises through the selling season, the like-for-like prices fall. When prices are cut faster, inventory is cleared faster, especially in the final month of the season, which is typically the month where most items are at their lowest inventory clearance prices. And at the start of the next selling season, the measured inventory age drops, and the process starts over.
Here’s data from Ascential where we use Inditex as a best-in-class example. What the charts show is that at the start of each season, promotions are low and inventory clearance is low; however, as the season progresses, promotions increase, particularly in the clearance month when prices are dropped to turn over the remaining inventory.
Sometimes, when they get the fashion wrong and demand isn’t available to support the inventory level purchased, you will see in the row-level data an item priced at €20 in its first week of availability. Then the next week it's €10 because it's obvious to Inditex that the rate of sale is off plan and it needs to clear the inventory faster. And by the 3rd week, it's €5 and then gone from availability in week 4. They take the margin hit to make sure they don’t have a longer-term cash flow problem from poor inventory turnover.
In the data, one should expect to see that when prices are moved lower at faster rate than the prior year, the inventory clears at a faster rate.
Merchandise planning becomes an even harder task to manage for fashion companies that have longer inventory management cycles compared to Inditex’s version of just-in-time inventory management.
For many fashion brands and retailers, Inventory is purchased quarters ahead of time, factoring in estimates for the health of the consumer, input cost inflation, and likely competitive pressures.
That’s a lot to get right six to nine months ahead of time. So monitoring the data reveals the strategy and if the execution happened as planned.
So what do we look for in the data to see if it's going well or poorly?
There are times when companies get the fashion wrong compared to competitors, or they miss estimated demand or overbuy inventory; either way, there isn’t enough demand to support the amount of inventory purchased, and dropping prices doesn’t lead to inventory being cleared fast enough. Remember, they have to get to zero inventory to make room for the next season or get stuck with excess inventory that is not relevant for the next season.
If we see prices falling faster this year’s selling season compared to last year’s selling season, combined with inventory age or inventory clearance performing worse than last year, the data is providing two flashing red lights that something has gone very wrong.
And other times, the fashion companies get the product very right, such that they clear inventory faster than the year before and with less downward pressure on prices than the year before: Double green lights!
Using a case study from H&M’s poor share price performance between 2016 and 2018 relative to Inditex, we can see from the data many times that H&M would drop prices more aggressively but fail to clear inventory at a faster rate. In its financial statements, this showed up as lower margins and worse cash flow due to lower inventory turnover.
By setting up a two-by-two grid comparing the pace of inventory clearance versus price reduction activity, an investor can spot the double red and double green lights. Below, the made-up data shows
Company B has most of its categories and geographies in the double green signal quadrant
The made-up data for Company A is in the double red signal quadrant.
The other quadrants require a more nuanced approach to analysis, leveraging other datasets and conversations with the company's management to better assess what is happening.
Deeper metrics to consider
Further metrics can be created and leveraged to go deeper into price trends. For example, by exploring the breadth and depth of like-for-like price changes, one can find that the overall like-for-like price trend was influenced by large changes in just a few SKUs. Likewise, more metrics can be generated to understand the price positioning of new items compared to prior years new items. Prices of evergreen items should be compared on a like-for-like basis y/y instead of sequentially. Prices of items available in multiple countries can be compared on a cross-country like-for-like basis to understand price parity (important for luxury items).
Pricing analytics community - what else would you look at in the fashion context?
Limitations to consider
As with all alternative datasets, there are limitations to consider when interpreting the data. By properly understanding the constraints on what the data can do, it allows for more confidence in using the data for appropriate use cases.
Differences Between Observation-Weighted and Demand-Weighted Aggregations
Web-mined data collection does not factor in the demand weight of each SKU. The actual demand across time and categories can differ significantly from the observation-weighted approach, which means absolute values on their own are biased. Collecting pricing data systematically from websites on a schedule helps provide a standardized view of the products and price, which allows for the trend to reveal insights (e.g. the results are consistently biased), but the trend of the key metrics removes the bias and allows for insights to flow through)
Website-wide promotions
Website-wide promotions, which can sometimes change hourly, are not accounted for in the web-mined approach taken. It is possible to collect the site-wide discounts and apply them to a standardized basket, but it opens up more complexity. While “take 10% of the basket with this coupon code” is easy to implement across the prices collected at that time, “free shipping” or “20% off baskets of $100 or more” and other conditional promotions require more data collection and modeling assumptions. In my experience, it's been an area of R&D to factor these promotions in. However, my current view is that the trends in the data, without factoring in these promotions, provide enough signal to get an edge on investment decisions without boiling the ocean.
I'm happy to be disproved on this view and take on the approach of collecting and implementing website-side promotions into the analysis. Let me know your thoughts
Data gaps
The limitation of retrospectively handling data gaps in web-mined pricing datasets is a significant constraint. Once a robot fails to collect the public data, it's not possible to go back and get the historic data. The robot can rerun as a close approximation to the scheduled run time, but if the robot cannot rerun soon enough, the data is going to be missing for the date. While many will prefer to create some sort of interpolation to generate the missing data point, I would prefer to leave it as a data gap.
Retail channel versus direct channel6 price interpretations.
For investors wanting to monitor the prices of brands sold in the retail channel (the wholesale channel, as the brands report it), the sample of websites needs to be kept constant in each time period of the analysis. If data is unavailable for one of the websites in a week, the entire week's data for the retail channel view should be removed to prevent a sample shift in time series analytics from giving a false signal.
Why using just one retailer is a poor choice: It’s important to consider the source of the pricing information for the analytics. When scraping a brand’s direct-to-consumer websites, we know that the price decisions are 100% controlled by the brand owner. However, when we look at a retailer that sells multiple brands, it’s unclear where the price decision is coming from. For example, promoting the most popular brands could be a decision from the retailer to win market share versus its competitors, which may also be contrary to what the brand owner wants to see. Promotions from the brand owner’s point of view may be rotated across retailers, spreading the benefit. Using one retailer as the sample introduces decisions by that retailer as a distorting signal in the data.
How big should the sample be? Three to five websites should be enough to capture the relevant sample to remove the noise of individual retailer activity while also being able to manage data gaps to a minimum in the retail channel view of brand performance. It’s therefore important to make sure the retailers in the sample cover the most popular websites for purchasing the specific brands in question to act as a representative sample of the entire market. However, there’s a trade off in the number of retailers included in the sample because of the issue with data gaps. If one retailer has a data gap, it will cause a shift in the sample and cause the metrics to move in a way that doesn’t reflect reality. As an analogy, think about running a survey where the answers in one run didn’t include 21–34-year-olds. It wouldn’t be possible to compare trends for that period to the other periods, including the 21–34-year-olds. It’s natural for robots to break, so it’s possible that with too many websites in the retail channel sample, there are constant data gaps.
Inventory age
Also, it is important to distinguish that inventory age, as defined by web mining, differs from the traditional accounting and financial analysis term.
The web-mining based metric measures the number of weeks each SKU is available from its first sighting to the present day.
The metric is not the accounting-based metric of the days of inventory as calculated based on the inventory balance sheet item and cost of goods sold on the income statement (converted to days of inventory).
Cross-website like-for-like comparison limitations
Challenges can arise when matching like-for-like items across different geographies and websites, especially when different SKU systems are in use. Website differences can also impact brand comparability. A single SKU on one website may include several variants, whereas another website might treat each variant as a unique SKU. This could affect SKU count measures and any measures reflecting the percentage distribution of SKUs. Furthermore, a navy 2-button polo shirt by Ralph Lauren and a similar navy 2-button polo shirt by Lacoste are not the same SKU, even though they may be nearly perfect substitutes (will I get some challenging feedback from fashion experts telling me these are not anywhere close to perfect substitutes?).
Classification restatements
As classification methods improve over time, historical data may need to be restated when reclassifying products. While beneficial for analysis going forward, restatements can impact category-level historical data. These need to be disclosed in the data and also need to be captured with point-in-time7 data so users can see what the data looked like when it was first available.
Best seller share
Keep in mind that best seller share' metrics might be influenced by paid product placements, which are common both in physical and online retail. However, the trend in best-seller share is more revealing of brand popularity than focusing on any single week's data point. Websites need to lead with their best products to keep customers engaged because the next website is just a click away. Retailers can’t consistently risk putting less popular products in the best-seller rankings.
Some common pricing metrics that should be given less analytical weight
There’s a number of “go to” pricing metrics that I think are not useful or misleading. Many investors like to look at these, but I would suggest its best to reduce the weighting of their importance in your analytic interpretation.
Absolute Average or Median Price. These are going to observation weighted and therefore are biased vs the actual demand weighted price trend that investors would care about. The observation weighted price aggregations can’t be put into a financial model x a volume estimate to get to revenue. The y/y trend in aggregated absolute prices is helpful in revealing if a brand has repositioned its portfolio, but that’s where the distribution is more important to understand the price strategy.
Discount depth and discount breadth are independent of other metrics. Sometimes retailers raise list prices and then promote more, but it leads to a higher current price. Other times, retailers introduce products during a promotion but show the original price crossed out. By using like-for-like pricing trends, it removes the importance of the promotion activity and focuses on what the company is actually charging for the item. Again, I’m not saying to avoid these altogether, but consider the limitations and consider the other metrics with higher analytic impact.
The possible differences between online and offline prices
This limiting factor has become less and less of an issue over time, but it is worth mentioning. Omni-channel pricing8 and assortment have become the industry standard because consumers have demanded that the in-store and online assortments and prices be the same.
However, it is possible that prices during clearance months vary between the website and the stores because the inventory at the store level may be different than the overall level and require a different level of promotion intensity.
Importantly, the overall trends in aggregate cannot materially differ between in-store and online because the consumer would notice and take advantage of the channel, or if not consistently priced between the channels, may distrust the brand and shop elsewhere altogether.
Action items to begin using the data
To leverage the web-mined pricing data in your data driven insights repertoire, follow these high level steps:
Have proper risk and compliance procedures in place before executing any web mining-based data harvesting. There are third parties who offer web mining as a service and third parties that expertly collect, cleanse, and enrich the data into a data product that should be heavily considered in the sourcing process.
Design and implement compliant web robots to harvest price and inventory data.
Build unit tests and anomaly detection algorithms to alert you when robots break.
Create a taxonomy of product classifications and brands
Calculate aggregated metrics such as like-for-like price trends, inventory age, and best seller share.
Consider the assumptions and limitations inherent in this model in the interpretation of the data, such as the observation-weighted metrics compared to demand-weighted metrics. Plan ahead for how data gaps will be handled analytically.
Look for signals where the clearance process is going materially better or worse than prior seasons and competitors.
- Jason DeRise, CFA
Web mining: The process of extracting valuable information from web pages using algorithms, typically used to understand customer behaviors, product popularity, and sales trends.
SKU (Stock Keeping Unit) is a unique code consisting of letters and numbers that identifies each distinct product in a store's inventory.
Evergreen: Products that are consistently available for sale over the long-term.
Unit testing: In software development, unit tests are methods that verify the correctness of a specific section of code. This also applies to data pipeline coding.
Regex: short for regular expressions, is a programming tool for pattern matching within text. It uses specific sequences of characters to find, replace, or manipulate strings in data. It's essential for search functions and data manipulation.
Retail Channel: The various routes that retailers use to sell products to consumers, including physical stores and online platforms.
Direct Channel: The means by which a company sells its products directly to consumers, bypassing any third-party retailers, wholesalers, or any other intermediaries.
Point in time-stamped history: This phrase refers to a dataset that provides the time data to show how data has been revised. So it includes not only the time period the data was related to but also the date when the entire data set was originally released or revised. This allows investors to use the data in back-testing models as if it were seen in real time before revisions.
Omni Channel: A pricing strategy that offers customers a consistent, seamless shopping experience across all channels (in-store, online, mobile, etc.)