Wrapping Your Arms Around Big Retail Data

In 2012, the federal government announced $200 million in research for Big Data computing.  Why?  Each and every day an estimated 2.5 quintillion (that is 17 zeroes after 2.5) bytes of data are created, and nearly 90 percent of the world’s data has been created in the past two years, according to IBM.  In the four years since the term Big Data was originally adopted within the IT community, the number of data sources has exploded, now including smart phones, social media posts, web interactions and much more.  As a result, experts predict that by 2015, the global market for Big Data technology and services will swell to $16.9-billion, from just $3.2-billion in 2010.  So what does that mean for retail?  A lot!    

Understanding Retail Big Data

 Within the next two years, 30 percent of retailers will run out of storage capacity, according to The Edgell Knowledge Networks, and while nearly 80 percent are aware of the Retail Big Data concept, less than half understand the implications it has on their business.  One immediate implication is clear, if retailers continue to ignore this issue they will be overrun by it.

 Beyond the amount of data, the types and sources of the data are key to understanding and managing this growing stream of information.  There are two types of data, structured and unstructured.  Structured data is typically numbers and letters laid out in a consistent format that is readable and searchable by a computer or person.  In retail, this data can come from the POS, alarms and access control, RFID and EAS, returns management, customer loyalty programs and much more.  Unstructured data can be much larger, and can spawn an entirely new category of structured data.  The primary form of unstructured retail data is video.  But don’t be fooled, that single source is a huge data contributor.  To provide some context, storing 30 days of footage from a high definition (HD) camera, running at 15 frames per second, requires nearly 1TB of space.  And that’s just one camera!

 With that being said, the Retail Big Data concept becomes a bit clearer, but it still lacks a formalized definition.  Not anymore.

 Retail Big Data (n.) – The collection of large amounts of (structured and unstructured) data from multiple, disparate, and often unrelated systems into a single repository to enable retailers to more efficiently and effectively understand their business and their customers in a timely manner.

 Managing Retail Big Data

Armed with a definition and a thorough understanding of the Retail Big Data epidemic, it’s time to take action.  Here are four steps to help retailers turn their Big Data headaches into Big Data profits.

1. Find the gaps – The first step to manage Retail Big Data is to understand the unique data needs.  Whether it’s a chain of two stores or 2,000, there needs to be a system in place to collect and process every data point.  According to experts, each store visitor can generate up to 10,000 data points, for national chains that can mean millions of data points every day. 

One way retailers can combat Big Data growth is through cloud storage solutions. While some retailers may be hesitant to transition their entire system to cloud, due to the loss of data visibility, there is a middle ground that many retailers are taking advantage of – summary cloud storage. Many retailers are utilizing the cloud to store summary data from individual stores and enterprise headquarters for an enterprise-wide view of the retail chain. This approach allows retailers to retain their store-hosted data, to meet the individual store needs, yet also gain the larger bird’s eye view of integrated data in real-time, through cloud computing.

1. Tear Down the Walls – As the number of data sources continue to grow, information is being collected, stored and processed in numerous locations.  Choosing to keep this data siloed immediately limits a retailers’ ability to succeed.  Integrated data provides a full view of the retail space.  It allows for side-by-side comparisons and enables retailers to uncover potential savings and opportunities. 

 In one instance, a grocery chain’s inventory and TLog were not adding up following a recent bottled water promotion.  With siloed data, the retailer would have needed to identify each of the item transactions, and then watch hours of surveillance footage to locate each bottled water scan.  However, with a fully integrated system, a retailer can identify each of the bottled water transactions, and with the click of a button see the individual item being scanned.  This technology allowed the grocer to uncover a major scanning issue.  Rather than scanning the case of water that was on sale, the scanner read an individual bottle’s barcode, thus accounting for the inventory discrepancy and thousands of dollars in lost revenue.  

1. Take Action – Now that retailers have a full view of their store environment, they can determine how to analyze and act based on the information available.  Beyond overall information access, advanced data integration systems allow retailers to create customized reports to review anything from Human on Transaction (HOT) refunds and sweethearting, to gself check-out (SCO) and facial recognition. 
   
   
2. Turn up the Speed – Finally, retailers can take these insights a step further by increasing the processing speed.  Big Data’s true potential is realized when the data lag is removed and retailers are able to interact with their information in real-time.  With real-time reporting retailers can create customized alerts for each store, or the enterprise as a whole, so managers are aware of issues as they are happening.

 For example, one client set an alert for drops in average department sales.  Since sales in the produce department had dropped below the specified figure, the manager got an immediate alert sent to his phone and email.  Rather than finding out a week later and searching for an answer, he was able to call the store manager and determine that the department manager hadn’t come in, so the department had never been fully stocked for the day.  The managers were able to rectify the situation in real-time and prevent further revenue loss.  

No matter how you look at retail data, one thing is clear – it’s on the rise.  In fact, the world's total data is more than doubling every two years, according to a 2011 IDC study.  Retailers cannot afford to wait.  Now is the time to get a handle on Big Data and prepare your company for the data surge.  Is your store ready?

 About the Author:

Derek Rodner is the vice president of product strategy at Agilence.  Derek’s prior experience includes strategy for open source database technologies, enterprise-class servers, and mid-range ERP software.  Derek is currently spending his time at Agilence working with retailers making the transition to big data environments and helping to shape the next generation of solutions. To learn more about Agilence’s Retail 20/20 solution for Big Data, contact Derek at This email address is being protected from spambots. You need JavaScript enabled to view it. or learn more at www.agilenceinc.com.

Related Articles:

Consumers are spending more time engaged (some might say, enslaved) digitally, and retailers have taken note. Never before have retailers paid closer attention to consumer technology. The benefits of mobility ...

When it comes to selecting retail software, there are more and more offerings from which to choose every year. You may be surprised, but there is actually quite a lot of variation in the different software ...

... bring mobile commerce to life.  Isis allows customers to pay brick and mortar retail stores with just a tap of the phone.  Banks, mobile carriers, payment networks and merchants will have access to the ...