Glitches are inevitable, and consumers know it: how anomaly detection can safely corral a glitch stampede

When there are legions of online retailers, each with thousands of products, and competing, overlapping promotions and discounts blinking into and out of existence – all powered by logical code written by unreliably logical humans, glitches are inevitable and are a constant reality of selling on the web.

Today’s consumers know this, and are always sharing with each other not only the deals they find, but also the best current strategies for finding them, putting a modern ecommerce spin on the ancient proverb:

Tell a man about a glitch, and he will get 19 free bottles of Ajax dish detergent,

Teach a man how to find glitches, and he will get free merchandise for a lifetime.

Whether they’re mining their social media feeds or being tipped off by sites devoted to sharing them, a stampede of shoppers from around the world can exploit your pricing error faster than you can even find it…unless you have a detection system that can identify the anomalies in your real-time pricing, volume and revenue data.

A glitch in EBT causes unexpected benefits

You may not consider food stamp programs to have much to do with ecommerce, but there’s a lot of overlap between the two:

  • Since the benefit is accessed via a card (called an electronic benefit transfer, or EBT card), almost all the program’s financial transactions are electronic, transmitted over the web.
  • Remote servers have to be queried in order to confirm that the cardholder’s balance is sufficient to cover the food purchase at the grocery store, just like all our online purchases made with a credit or debit card.
  • Since these programs are electronic and dependent on the web, they are vulnerable to the very same types of infrastructure and connection problems large ecommerce sites are.
  • Word quickly spreads about glitch discounts, resulting in a stampede of opportunistic shoppers.

One glitch incident which demonstrates all four of these points is the one Walmart suffered at a few stores in Louisiana back in October of 2013, when a test at EBT vendor Xerox caused a backup generator to fail, knocking the whole EBT system offline for a few hours. After the system came back online, the EBT cardholders in a few areas discovered that even though they could use the cards to buy food, their spending limits were gone. And thus began the stampede.

In emergency situations like this, retailers like Walmart are allowed to set an emergency $50 spending limit on each EBT cardholder, and if they do, they could avoid the losses incurred by all purchases made possible only by the glitch. Walmart store management did notice something was wrong during the glitch, but decided to honor those purchases, and thereby ate the losses. Other grocery stores in the area made the opposite decision: refusing to accept EBT cards until the limits were restored.

Walmart faced both a PR and ethical quandary: decline purchases made by the EBT cards (and thereby deny low-income families on financial assistance the ability to buy food for their families), institute the $50 limit (and allow them to buy only a small amount of food, but effectively punish them for a problem they didn’t cause) or just allow all the purchases go through, which is Walmart ultimately did. Every one of those options carry negative consequences for Walmart’s bottom line, for their non-EBT customers, for their company image and reputation, and for the EBT recipients themselves.

How anomaly detection and correlation could have helped

Had they known from the beginning that this was more of a systemic problem across an entire region and not an isolated problem, they may have been more inclined to enforce the $50 limit, and thereby drastically decrease the losses Walmart had to eat.

An anomaly detection system, monitoring company-wide as well as store-level metrics in real time would have detected the many signals which would have appeared the moment this glitch occurred: a sharp increase in the average dollar amount of EBT purchases, the rapidly shrinking shelf inventory of the affected stores, and so on. Additionally, a smart system would have then correlated all those separate events into a singular, concise alert which showed the scale of the problem. With a clearer picture of the scale of the glitch at the moment it began, would Walmart’s management have made the choice to use the emergency $50 limit?

With limited information in fast-moving crises (the glitch only lasted about two hours), Walmart had a tough call to make, one which Walmart would not have had to make had anomaly detection been in use at Xerox.

Had Xerox implemented a robust IoT-based solution to proactively monitor operational data from the backup generator and then feed that data to anomaly detection system, the limit-erasing failure could have been prevented to begin with, and along with it the losses Walmart incurred for Xerox’s mistake. This is because anomalies could have been spotted in that data, leading to effective preventative maintenance long before the generator itself failed. Anomaly detection could have prevented the whole cascade of failures from happening to begin with. This is because they find anomalies in time series data, and don’t care where that data comes from: an ecommerce company, an adtech broker or an industrial-grade backup generator.

Glitches will happen to either your company, or a critical vendor, that much is guaranteed. Don’t let a glitch stampede overrun you with losses and bad PR. Use an anomaly detection system to monitor everything from sensor data to sales volume, and fix the glitch before the stampede even arrives.