This year, the big giant of retail, Walmart, made headlines in the technology media space when it announced its intentions to acquire Kosmix, a startup focused on social media. When this announcement happened, many analysts were surprised and were analyzing the key reasons for a giant retailer to acquire a startup company.

Post acquisition, the Kosmix team operates as part of the newly formed company called WalmartLabs. Kosmix understands context extremely well. For example, if a user tweets ‘I enjoyed Salt,’ it knows that the tweet is related to a movie starring Angelina Jolie, and not the salt we use for cooking. By applying semantic analysis to social media, Kosmix understands the connections between people, topics, location and products.

Kosmix stands out for its ability to search and analyze connections in real-time data streams to deliver highly personalized insights to users. The platform powers TweetBeat, a real-time social media filter for live events. By using this intelligence, Kosmix is building a giant knowledge base called the ‘Social Genome.’ This giant knowledge base captures information and relationships about entities such as people, events, topics, products, locations and organizations.

Using tweets to make product recommendations

By analyzing their social media activity, Social Genome can make recommendations about products, events or any other activity that the user is interested in. For example, by using publically available social media data, the Walmart product store can suggest product recommendations, based on recent tweets or Facebook wall posts.

“There is an unprecedented amount of data about people, places, products, companies, brands, and pretty much anything we can think of. Companies that can mine this treasure trove of data and glean insights to gain a huge competitive advantage,” opines Anand Rajaraman, SVP, Walmart Global eCommerce and Head of WalmartLabs. Walmart can even filter this data to understand location-based preferences and hold inventory that is preferred in certain locations. Monitoring social media can even help Walmart create or stock products that are in demand.

Dealing with Big Data

While the idea sounds great, doing this in reality is a huge problem — especially since there are thousands of data pieces flowing in a torrent from live data sources such as tweets, Facebook posts and blogs. The data flow was so fast that Kosmix could not rely on the traditional Map-Reduce or Hadoop framework that is typically used to solve Big Data problems.

“Social Media data is the fastest growing source of Big Data today. In addition to being Big Data, social media data such as Twitter also has a real-time nature — it’s not just Big Data, but also Fast Data. With mobile devices, location data is now a new source of both Big and Fast data,” explains Rajaraman, on the technical challenges faced by his firm while building the platform.

To address this Big Data and Fast Data problem, Kosmix developed its own in-house solution called Muppet, which processes streaming fast data in a lightening fashion, over large clusters of machines. Today, Muppet can manage and track data streams with billions of updates a day.

"At WalmartLabs, we deal with both Big Data and Fast Data. We use Hadoop and other open source tools to analyze social media data from Twitter, Facebook, Foursquare and other sources. We also have developed our own proprietary technology code named Muppet to analyze Fast Data such as Twitter firehose (full data feed of all public tweets on Twitter). Just as Hadoop has the primitives “Map” and “Reduce,” Muppet uses “Map” and “Update.” For example, Muppet can analyze in real-time which stores have the biggest crowds on Black Friday (the biggest shopping day of the year that happens a day after Thanksgiving) based on Foursquare check-ins,” states Rajaraman. Foursquare is a location-based mobile platform that users can use to share their location with friends.

Our technology platform, Social Genome, allows us to make product recommendations to customers shopping online or in the store

Anand Rajaraman SVP, Walmart Global eCommerce and Head, WalmartLabs

Using this technology, Walmart can track social media mentions on locations, people or products and use this intelligence to better tailor its product offerings. “The deeper the understanding we have about our customers and our products, the better we can connect with them. Our technology platform, which we call the Social Genome, tracks connections between people, products, brands and other relevant entities. We can use the Social Genome to make product recommendations to customers, whether they are in the store or shopping online,” explains Rajaraman.

As a firm that some analysts say is the future of the giant retailer, WalmartLabs is in an interesting position as more and more people are increasingly being influenced by what their friends say on Facebook or what a group of strangers tweet on Twitter. If Walmart can tap into this torrent of information, and can use this to understand customer behavior better, it can redefine shopping experience.