Use Data Mining to Predict If Your Product Will Crash and Burn

Career AdviceWorking Life

The benefits of predictive analytics for clients, including the ability to better identify which customers are more likely to succeed.

It appears to go against Marketing 101, but is it possible that in some instances, strong sales and positive consumer feedback for a new product can be… bad?

According to marketing expert Eric Anderson and his colleagues, consumers who have a history of buying items that fail are "harbingers of failure." This means that if these consumers buy a new product, it is likely that the product will also fail.

The market for Diet Crystal Pepsi and Frito Lay Lemonade appeared to be supportive, as people who bought the former also kept buying the latter while it was still available.

According to the researchers, a one-time purchase of Diet Crystal Pepsi is only partially informative about a consumer’s preferences. They added that a consumer who repeatedly purchases Diet Crystal Pepsi is much more likely to have unusual preferences and is more likely to choose other new products that will fail in the future.

The research went against traditional marketing models, like the Bass diffusion model, that suggest a strong beginning sales indicates a greater chance at long-term success. What the research found was that a head start does not always make one a successful marathon runner.

The study also found commonalities with retail's early days of big data utilization. The researchers analyzed two large data sets from a national drugstore: one of individual customer transactions, amounting to over 10 million transactions made with customer loyalty cards over two years; and a sample of aggregate store-level transaction data, involving 111 store locations in 14 states over more than six years.

The academic analysis may have been more post hoc than what we often see in contemporary business architectures, but it was nonetheless a triumph of using large volumes of data to unearth significant patterns and anomalies.

TOP BENEFITS OF DATA MINING IN MARKETING

• Basket Analysis: How can data mining be used to uncover which products or services are often purchased together?
• Product Recommendation: What is the role of data mining in tailoring product suggestions to individual users?
• Customer Segmentation: Grouping customers or clients into subsets based on common characteristics and habits. How does cluster analysis use data mining?
• Customer Lifetime Value: Decision trees and boosting to quantifying how much money a customer is likely to generate for a company.
• Churn Prediction: How can data mining be used to quantify the likelihood that a customer will stop doing business with a company?

Harbingers of Failure has been a significant and long-term contribution to marketing, and has even spawned a 2019 sequel. The paper's findings show that companies should avoid such harbingers of big data as much as possible.

"I keep running into people who know about this ... but firms are really struggling to figure out what to do about it," Anderson. "If they don't broaden their view, they won't be able to figure it out."

"A cosmetics company, for example, is likely to buy beauty-related data from a consumer packaged goods market research firm like Nielsen or IRI," said Anderson. "But that probably doesn't tell them who's drinking the 2021 version of Frito Lay Lemonade."

Anderson recalled how, while at Ocean Spray, his team would purchase data that only covered red drinks; however, this only provided eyes on the cranberry juice world and didn't give any insight into other beverages.

Anderson believes that retailers and e-commerce sites like Amazon and Walmart, who have access to a wealth of sales data across many categories, are best equipped to put his research into practice.

The harbinger effect is a bizarro-world version of basket analysis, which is a technique that marketers use to understand how consumers who like product X also tend to like product Y.

This could also have implications for customer lifetime value, which tells companies which clients are the most valuable and should be given more attention. They can be counted on to keep shopping, but they might not buy the right items. Designating customers as red flags for failure is a new twist on customer segmentation, which is dividing consumers into subsets based on their characteristics or habits.

Data mining has been shown to be effective in marketing in the following ways.

Basket Analysis

The story of beer and diapers has been around for a long time. A retailer who was good with numbers supposedly discovered that shoppers often bought these two unrelated products together at the same time of day. The story goes that young fathers, when out on late-night diaper runs, would often times award themselves with a six-pack. Though the story may not be accurate, it does show how purchase patterns can influence how companies choose to cross-promote and target their marketing efforts.

By integrating disparate data sources and applying association rule learning, analytics firm Quantzig was able to maintain a dashboard of real-time product-bundling recommendations for a European food retailer in 2019.

Given the large amount of data that Instacart has, it is not surprising that the company uses basket analysis to find hidden connections. It has even shared some of these findings with the public. A recent Instacart blog post suggests that vegetable buyers are typically "heavy meal preppers" who plan their weekly meals in advance, often using tortillas, cucumbers and watermelons, while fruit buyers are more likely to snack and grab yogurt and hummus.

The 2017 Instacart Grocery Data Set is a great way for marketers to learn about consumer behavior prediction through data science.

Product recommendation data mining marketing

These examples come from the world of packaged goods, but the same idea-finding treasure-is of course central to contemporary e-commerce, where it is called product recommendation.

According to the Wall Street Journal, Etsy chief technology officer Mike Fisher said that because the website has more than 80 million items for sale, it uses a sophisticated recommendation system to help prevent shoppers from feeling overwhelmed by too many choices (a phenomenon known as "paradox of choice" paralysis). The system has evolved over time into a natural language processing framework that takes into account past searches and purchases - "billions of historical data points," as the Journal said. The recent publication of research by Etsy data scientists has proposed that by mining recent user activity data, “within-session” personalization of preferences for attributes such as color, size, and material options can be driven.

Data mining techniques like collaborative filtering and content-based recommendation can be used to drive success for streaming platforms like Netflix and Spotify.

Customer Lifetime Value

As customer lifetime value (CLV) expert and marketing pioneer Peter Fader mentioned, companies should focus their promotional efforts on those customers who will generate the most profit, rather than trying to turn every customer into a loyal, high-value individual.

As of today, this means implementing complex machine learning algorithms rooted in data mining — for example, the gradient-boosting decision trees of Cars.com or the neural networks that help power CLV software provider Retina. According to Fader, one of the most sophisticated examples is game publisher Electronic Arts, which updates CLV estimates daily based on gamer behavior data. After updating its CLV model, EA decreased its marketing expenses from 22 percent to 12 percent of revenue, as stated by Zach Anderson, former chief analytics officer, in the Customer Equity Accelerator podcast in 2018.

Customer segmentation data mining marketing

Dividing a company's customer base into distinct subgroups has many benefits, the most obvious of which is that marketers can tailor messaging and promotions to fit how that specific group interacts with the brand.

K-means clustering is a technique used in cluster analysis to determine which users are similar and dissimilar. From this analysis, we can see that there are six different customer types, ranging from “medium income, low annual spend” to “very high income, high annual spend.”

In addition to non-numerical data, latent class cluster analysis (LCCA) is another cluster-based data mining technique that allows modelers to build segments. The Dallas-area analytics firm Decision Analyst used LCCA for a customer segmentation job when introducing a new appliance to market, in order to build clusters of like-minded panel responses and determine the best way to position the new wares.

Churn prediction data mining marketing

According to marketing experts, it is cheaper to keep a customer than to acquire a new one. Churn prediction is the term used for assessing the likelihood that a client will cancel or not renew a service. If marketers know this in advance, they can try to stop the turnover from happening.

Churn prediction traditionally relies on data mining techniques like humble regression analysis and classification.

As Sadrach Pierre stated earlier this month, companies that use Python libraries such as Streamlit can create classification/churn models with easy-to-use interfaces. Furthermore, there is an expanding ecosystem of machine learning-based churn prediction tools that can provide more accurate churn scores, thanks to the increased number of data points they process.

According to Kristen Hayer, founder of consulting firm The Success League, "You can figure out how you're going to tackle churn with your product shifts, and how you're going to make it easier for the customer to work with you. It changes the conversation because it gives you enough time to actually plan."