One of the things marketers and brands alike are excited about at the moment is the potential of Big Data. This excitement is understandable – the ability to tap into previously unheard of sources of information about our customers is a very big thing indeed.

Whether the excitement being generated is fully warranted is another thing, though, especially given the fact that Big Data more than lives up to its name when it comes to the reality of using it effectively.

At a conference earlier this year, one of the speakers – from a data analysis company – spoke of the craziness of trying to make sense of the amount of data we have access to. By her reckoning, it would take 1,000 data analysts working 24 hours a day, 7 days a week, more than 300 years to sift through everything currently available to us.

And that’s just with today’s data. As more users come online and begin to share their own information and preferences, the numbers continue to escape the folks trying to make sense of it.

Even with that, though, Big Data is, and continues to be, a valuable resource when used in the right context. However, there’s another opportunity just waiting for us – that of finding context in the meaningless data we discard.

Big Data – Beyond the Obvious

For most companies mining data, the goal is to find the nugget of gold that can help them with a variety of business goals – lead generation, customer acquisition, customer retention, crisis prevention, brand reputation, HR head-hunting and more.

All good stuff; all the kinds of the things businesses should be looking for, and all the kinds of questions that Big Data can answer. Yet while this kind of approach has been proven to yield results, the opportunities when we go beyond the obvious is where it gets really exciting.

For instance, a typical data mine might look like this:

  • Identify keywords, topics, and user groups/personas;
  • Start indexing search matches;
  • Use natural language processing (NLP) to identify sentiment, context, etc.;
  • Weight keywords against each other based on importance, relevance and frequency;
  • Create user groups of results for the relevant business team to take over;
  • Rinse and repeat.

Given, that’s a pretty basic overview of what a typical social search/data mine comprises of – but it does show you how the data can be found, filtered and used.

Ontology discussion

However, this is going after specific pre-defined targets – keywords and groups based on the business goal. So, it’s fair to say that the results achieved are only meeting the immediate targets set.

But what would happen if we stepped outside the immediate target area and started thinking beyond the obvious?

Out of Context Data, In Context Opportunities

One of the biggest challenges facing monitoring platforms, even with today’s technology, is they’re still (mostly) relying on scripted conversations to glean data from.

Sure, NLP and text analytics can help filter out certain emotions and sentiment around a conversation to give us the kind of data we need to make decisions – but the human mind is a far more complex beast than the flow of conversation traditionally used for monitoring reports.

It’s this complexity, and the way it adapts on the fly while continuing the same conversation – or even taking an action based on a non-targeted conversation – that offers the greatest opportunity for analytics, monitoring and data companies to build for.

Example A – The Social Graph Data

Let’s say Mary is the target audience of a business that sells shoes. They might set up certain searches around how she decides what shoes to buy, and when – historical purchases, brands she follows, age group and similar consumer follows, seasonal choices (back to school, new job, etc.)

Based on these searches, any time Mary takes an action that involves the specified keyword(s) – a Like on Facebook, sharing a video on YouTube, an extended conversation on Google+, participating in a fashion chat on Twitter, etc. – will pop up as an opportunity for that brand to engage with her, either directly (a tweet, a blog comment) or indirectly (banner ad, Sponsored Story).

However, let’s take it a little bit further. Undetected by the search algorithm, Mary occasionally uses a hashtag on some of her updates. The hashtags don’t seem related – they’re innocuous, random, and spread across multiple networks.

While the automated search is ignoring them, though, a behavioural analyst – or just someone that has a curious nature on the other end of the search – decides that there is a pattern to the hashtags, no matter how infrequent and haphazard they seem.

Haphazard sanity

Using human intuition and personal legwork, the analyst discovers the hashtag refers to Mary’s crowning moment at high school, when she beat the high jump record. The hashtag – say, #WIWYAF – stands for “When I was young and fit”, and is a reminder of Mary’s youth that she’d love to get back, hence her love for certain sports shoes.

Sending a new search spider out connects her social graph together and uncovers multiple conversations and images around her reminiscing.

This little nugget allows the brand to reach out and say, “Hi, Mary, wouldn’t it be great to revisit the summer of school sports ’85? Well guess what – our new Running Shoe X is built from the memories that made that year so great.”

Instant connection. Instant relevance. High on context and memories and direct to Mary.

Example B – The Alternative Thinking Data

Another way to look at is is by thinking of alternatives to what we believe we’re being told by public updates.

John is in Vancouver, and posts an update to his networks that he hates the cold. Being Canadian, this could mean that John hates the winters in Vancouver, and wishes he was elsewhere.

A vacation company monitoring opportunities could see this update, and perhaps reach out with a special offer valid for the next 48 hours. The time-sensitive offer, and the likelihood that John is in that company’s target audience, could see a sale and a new customer.

Then there’s the thinking beyond that.

  • Does John hate the cold because he can’t afford his heating bills?
  • Does John hate the cold because he has a hole in his window?
  • Does John hate the cold because he has seasonal allergies?
  • Does John hate the cold because his roof isn’t insulated properly and letting heat escape?
  • Does John hate the cold because it usually means Christmas and crappy family dinners he hates attending?

One simple statement has now opened up a myriad of possibilities that, if we dig deep enough, could offer several opportunities to meet John’s need.

  • His bank could reach out discreetly to see if they can help;
  • A glazier could offer a low-cost emergency repair to his window;
  • A consumer advice group could offer tips on better roof insulation and heat preservation.

Each opportunities; each resolving a need. All that’s needed is the hidden context of an unremarkable update.

The Permission Factor of Data

Now, given, this assumes a lot of permission marketing and public acceptance of how data is used. Then again, who says data needs to be the sole domain of the marketer?

Think of identifying and activating new donors or activists for a non-profit or cause. Think about helping people in danger – depression, loneliness, abuse – by proactively digging beyond what may be a limited call for help but goes much deeper. Think about law enforcement spotting dangerous new drug avenues before they hit the streets.

The data we monitor today can often be hiding the real data we can use tomorrow. It’s going to take experimentation and respecting, as well as garnering the respect of, the people we’re monitoring to start the process.

However, as a starting point in truly meeting the needs of the people we say we want to help, it’s not a bad goal to be thinking of now. Is it?

image: kris krug
image: Francis Rowland
image: Argonne National Laboratory

Get my latest posts as soon as they're published

Join over 12,000 smart subscribers and receive my latest posts as soon as they're published - simply enter your email below (I respect your privacy and will never spam you).

Alternatively, click here to subscribe to the RSS feed instead.

Ivan Widjaya
Ivan Widjaya

This is a rather painstaking task. Imagine trying to look for organization amidst the chaos. Although this can be quite valuable in extracting data, it is still very important to have some sort of software that will do everything automatically. This takes too much time which is better off allotted to more productive work.

Jonny Ross
Jonny Ross

Great blog, Danny - really insightful. It's such a fascinating topic area. For me as a web agency and someone who helps people plan digital campaigns, big data is always in the back of my mind in terms of the touchpoints where data can be captured. It's about foreseeing which information you want to collect about your customers and potential customers, then planning in how and where to collect it. Big data has big implications for website development. We need to be ready to think like CRM experts to really make the most of this for our clients! Jonny

Chilly Willy from Philly
Chilly Willy from Philly

This is a great post. Twitter is great for this. You can do it in 2 ways. 1] Shoot an ad around that key word(s) with an offer like you mentioned. Very impersonal. Non-threatening. 2] You use humans. I have some key words for clients twitter accounts. For my mobile gourmet dessert client I can do a search for anyone in a certain town within the last hour and reply to come by and send a photo to make them drool. And customize it based on what they tweeted (or their bio). Much more powerful the second. But if you come across the wrong way people get put off. For some reason that kind of tracking has people feel like you are stalking. But I have had great results over all. Would you suggest your example about the running shoes have a human doing that? My biggest issue is cost. Humans cost more. Can I sell more shoes in a different way that is more profitable? While I can easily show building a connection has long tail value on the short end a business owner might go for the higher up front profits.

Danny Brown
Danny Brown

Automation will only get you so far. You still need human analysis and understanding of the human psyche to get the real information you need. Painstaking it may be, but no long-term success was built overnight. Besides, wouldn't you say knowing what makes your customers tick the most productive work of all?

Danny Brown
Danny Brown

That's the biggest advancement for sure, mate - that ability to see what message resonates most at a particular time for a customer is gold. Of course, then you have to make sure you have the right people putting the right message out, which is another discussion altogether... ;-)

Danny Brown
Danny Brown

See, I knew there was a reason I should have left Livefyre switched on, to combat all your crazy aliases, Mr. Goldfarb. ;-) That's definitely the biggest problem, the question of scale. How do businesses - especially smaller ones - react to multiple queries and opportunities? There are companies like Kaypok and Maven Social that are trying to answer that question, but for me the human angle is still key. Automation can filter out the qualified opps, and the human eye quantify them. Still legwork; but the payoff is more often than not worth it. As shown by your examples.

Jonny Ross
Jonny Ross

Would have to agree with Danny, some of this stuff is simply too insightful to miss it! Jonny

Jonny Ross
Jonny Ross

It is indeed, Danny! The right message to the right person at the right time IS the "holy grail" - but then it is with the people you know personally too! ;)