One of the things marketers and brands alike are excited about at the moment is the potential of Big Data. This excitement is understandable – the ability to tap into previously unheard of sources of information about our customers is a very big thing indeed.

Whether the excitement being generated is fully warranted is another thing, though, especially given the fact that Big Data more than lives up to its name when it comes to the reality of using it effectively.

At a conference earlier this year, one of the speakers – from a data analysis company – spoke of the craziness of trying to make sense of the amount of data we have access to. By her reckoning, it would take 1,000 data analysts working 24 hours a day, 7 days a week, more than 300 years to sift through everything currently available to us.

And that’s just with today’s data. As more users come online and begin to share their own information and preferences, the numbers continue to escape the folks trying to make sense of it.

Even with that, though, Big Data is, and continues to be, a valuable resource when used in the right context. However, there’s another opportunity just waiting for us – that of finding context in the meaningless data we discard.

Big Data – Beyond the Obvious

For most companies mining data, the goal is to find the nugget of gold that can help them with a variety of business goals – lead generation, customer acquisition, customer retention, crisis prevention, brand reputation, HR head-hunting and more.

All good stuff; all the kinds of the things businesses should be looking for, and all the kinds of questions that Big Data can answer. Yet while this kind of approach has been proven to yield results, the opportunities when we go beyond the obvious is where it gets really exciting.

For instance, a typical data mine might look like this:

  • Identify keywords, topics, and user groups/personas;
  • Start indexing search matches;
  • Use natural language processing (NLP) to identify sentiment, context, etc.;
  • Weight keywords against each other based on importance, relevance and frequency;
  • Create user groups of results for the relevant business team to take over;
  • Rinse and repeat.

Given, that’s a pretty basic overview of what a typical social search/data mine comprises of – but it does show you how the data can be found, filtered and used.

Ontology discussion

However, this is going after specific pre-defined targets – keywords and groups based on the business goal. So, it’s fair to say that the results achieved are only meeting the immediate targets set.

But what would happen if we stepped outside the immediate target area and started thinking beyond the obvious?

Out of Context Data, In Context Opportunities

One of the biggest challenges facing monitoring platforms, even with today’s technology, is they’re still (mostly) relying on scripted conversations to glean data from.

Sure, NLP and text analytics can help filter out certain emotions and sentiment around a conversation to give us the kind of data we need to make decisions – but the human mind is a far more complex beast than the flow of conversation traditionally used for monitoring reports.

It’s this complexity, and the way it adapts on the fly while continuing the same conversation – or even taking an action based on a non-targeted conversation – that offers the greatest opportunity for analytics, monitoring and data companies to build for.

Example A – The Social Graph Data

Let’s say Mary is the target audience of a business that sells shoes. They might set up certain searches around how she decides what shoes to buy, and when – historical purchases, brands she follows, age group and similar consumer follows, seasonal choices (back to school, new job, etc.)

Based on these searches, any time Mary takes an action that involves the specified keyword(s) – a Like on Facebook, sharing a video on YouTube, an extended conversation on Google+, participating in a fashion chat on Twitter, etc. – will pop up as an opportunity for that brand to engage with her, either directly (a tweet, a blog comment) or indirectly (banner ad, Sponsored Story).

However, let’s take it a little bit further. Undetected by the search algorithm, Mary occasionally uses a hashtag on some of her updates. The hashtags don’t seem related – they’re innocuous, random, and spread across multiple networks.

While the automated search is ignoring them, though, a behavioural analyst – or just someone that has a curious nature on the other end of the search – decides that there is a pattern to the hashtags, no matter how infrequent and haphazard they seem.

Data visualizer

Using human intuition and personal legwork, the analyst discovers the hashtag refers to Mary’s crowning moment at high school, when she beat the high jump record. The hashtag – say, #WIWYAF – stands for “When I was young and fit”, and is a reminder of Mary’s youth that she’d love to get back, hence her love for certain sports shoes.

Sending a new search spider out connects her social graph together and uncovers multiple conversations and images around her reminiscing.

This little nugget allows the brand to reach out and say, “Hi, Mary, wouldn’t it be great to revisit the summer of school sports ’85? Well guess what – our new Running Shoe X is built from the memories that made that year so great.”

Instant connection. Instant relevance. High on context and memories and direct to Mary.

Example B – The Alternative Thinking Data

Another way to look at is is by thinking of alternatives to what we believe we’re being told by public updates.

John is in Vancouver, and posts an update to his networks that he hates the cold. Being Canadian, this could mean that John hates the winters in Vancouver, and wishes he was elsewhere.

A vacation company monitoring opportunities could see this update, and perhaps reach out with a special offer valid for the next 48 hours. The time-sensitive offer, and the likelihood that John is in that company’s target audience, could see a sale and a new customer.

Then there’s the thinking beyond that.

  • Does John hate the cold because he can’t afford his heating bills?
  • Does John hate the cold because he has a hole in his window?
  • Does John hate the cold because he has seasonal allergies?
  • Does John hate the cold because his roof isn’t insulated properly and letting heat escape?
  • Does John hate the cold because it usually means Christmas and crappy family dinners he hates attending?

One simple statement has now opened up a myriad of possibilities that, if we dig deep enough, could offer several opportunities to meet John’s need.

  • His bank could reach out discreetly to see if they can help;
  • A glazier could offer a low-cost emergency repair to his window;
  • A consumer advice group could offer tips on better roof insulation and heat preservation.

Each opportunities; each resolving a need. All that’s needed is the hidden context of an unremarkable update.

The Permission Factor of Data

Now, given, this assumes a lot of permission marketing and public acceptance of how data is used. Then again, who says data needs to be the sole domain of the marketer?

Think of identifying and activating new donors or activists for a non-profit or cause. Think about helping people in danger – depression, loneliness, abuse – by proactively digging beyond what may be a limited call for help but goes much deeper. Think about law enforcement spotting dangerous new drug avenues before they hit the streets.

The data we monitor today can often be hiding the real data we can use tomorrow. It’s going to take experimentation and respecting, as well as garnering the respect of, the people we’re monitoring to start the process.

However, as a starting point in truly meeting the needs of the people we say we want to help, it’s not a bad goal to be thinking of now.

Is it?

image: kris krug
image: Francis Rowland

Sign up for free weekly content

Enter your first name and email below to get my free weekly newsletter with the latest posts, recommended reading, content tips and more.

(I respect your privacy and will never spam you)

Blog consulting with Danny Brown

Enjoy this post? Share your thoughts below:


  1. says

    Another excellent article, Danny. You do get back the answers to the questions you pose and if you haven’t framed the question well or considered the possibilities you may not get the insight you were hoping for. That’s not unique to big data – ask any survey designer.
    I can see the possibilities of  better data mining too. For years I’ve been saying that CRM should do this. When someone is identified as a potential customer, it should look for not just current role but who influences them, where they wer previously and even where they went to Uni, to find matches or useful data for the salesperson. For example if this person masterminded a project to put in SAP at their previous company and is now enquiring from SAP about this one, you can be fairly sure where this is headed.
    Practice is very different from theory. I wanted a guest house in Wales for one night three years ago. Because all the places I wanted were full or too dear, I ended up in a little village called Criccieth. Ever since, however, has decided that Criccieth is my number one destination and begins all its emails to me with Places to stay in Criccieth and X.
    Lastly there is a paternalistic “we know best” view involved. And a correlation/causation debate. When I was young the car I aspired to was the Lotus Elan. But I wouldn’t want a current one. And if you sent me an email like that I would say “I can make my own decisions, thanks” and buy something else. I would already have researched the shoe market and while 1985 may have been a factor, it wouldn’t be straight “See it-must have it”. I wonder how much better data improves success. I suspect people are making a more informed decision than marketers credit them with.

  2. says

    PeterJ42  Hi Peter,
    Have you checked out They do an excellent job of connecting the dots from social profiles – all you need is one touchpoint. So, you drop an email address into Nimble, or a Twitter profile, or LinkedIn profile, etc., and Nimble then spiders the web for that person’s other digital footprints. It also shows their most frequent connections, milestones, etc. is another example of the same idea, except their solution also shows you who influences that person the most based on topic as well as demography and geography.
    Like you say, practice is very different from theory which is why I firmly believe businesses need more people analysts as much as business or data analysts. There’s only so much automation and algorithms can offer, because people will always be smarter than them.
    Cheers, sir!

  3. DannyBrown says

    Kabolobari Ironically, I think I may have kept the font the same colour, just switched around the background with the new theme. :)

  4. DannyBrown says

    sabreangel Cheers, Sabre – definitely one of the topics that still interest me in this “space”

  5. says

    You hit a key here Danny Brown because this dovetails with scalability and costs. The data exists to do some amazing things with analysis and targeting. The question is the resources needed to do this currently cost more than the benefit. It really takes humans and it also takes narrow vs broad queries. Meaning brands and Facebook want to do algorithms to sift the masses for opportunities and the results suck. And listening tools are still primititive. Obviously Facebook doesn’t use our actual posts and written comments/content for targeting or it would be so much better. They only use our graph and the pages and interests we entered in. I might Like the Chobani page but it would take a human to learn on that page I complain I hate strawberry and why don’t they have maple? Which then a local creamery could then say ‘opportunity’ because they have maple. Right now that takes a human and to win $10, $50, $100 of business from them is too costly. So if I buy Facebook ads I can’t look for people complaining in Chobani’s page why they don’t have Maple. But if the Creamery had a smart person they could easily set up their own proprietary listening and response for people wishing the majors had maple. BUT they still can’t do this on Facebook…really just Twitter. Unless the user hashtagged #maple on facebook and other networks.

    I think Big Data works best with your own customer data. Like Target did for the expectant mothers. Or the Supermarkets with club cards could though they waste this data by not using it right.
    I think one day this will change but the computing skills needed to create such intricate AI doesn’t exist….yet.

  6. says

    Howie Goldfarb Great points, Howie, and completely agree. I had an interesting meeting with a couple of Facebook ad execs the other week (and a more deep dive meeting is planned), and I was surprised to hear their take that a large and engaged fan base will trump an ad bid in their marketplace from a brand with a higher budget but less fans.
    Given Facebook has essentially nixed the reason to have a large fan base (poor organic reach, dwindling participation, etc.), why on earth would they take that route? It seems to me you may as well give up an a Facebook page and simply use it as an acquisition channel via ads and newsfeed sponsored stories.
    And given that many Facebook users have their updates locked down to private or friends only, it’s impossible for monitoring solutions to pick up on the type of conversations that are truly valid – that still boils down to human watching.
    And that, as you rightly say, is where the prohibitions start. There’s a huge opportunity for a tech company somewhere to create something that counters this – question is, will they, or will they go for the big bucks through big data instead?

  7. says

    Danny Brown Howie Goldfarb  It seems to me that marketers have weed in their own nests. Instead of being so interesting that people want to hear from them they have just done the old interruption marketing thing. Now they have a double whammy – Facebook is trying to charge them and prospects are shutting them out. Will they never learn?

  8. says

    PeterJ42 But therein lies the problem, Peter. Brands are trying to be interesting (many actually are, but that’s another discussion!), and that interesting content isn’t being seen due to Facebook’s supposed QA process (which they themselves have shown to be faulty). When the fun disappears from social, perhaps it’ll be less the brands at fault and more teh networks that force them to be anything but approachable. Howie Goldfarb

  9. says

    Thanks for referencing my blog,
    Danny. Really interesting discussion here about big data potential and possible

    My experience is that many
    people are excited about the potential, but few have a clear idea of the
    practicalities of utilising it. Which brands do you think are using big data
    Also interesting to see public response to the
    UK NHS Care.Data project – still an issue of public fear/negative perceptions
    to overcome.

  10. says

    Jonny Ross  Hi mate,
    Just Googled the Care.Data story, fascinating. Like you say, the real staller of progress is perception – brands and organizations need to do a much better job of clarifying what’s being done, and why, and – more importantly – how it truly benefits the end user as opposed to just the company running a program.
    Ironically, I find many of the best uses come from organizations that don’t see it as a marketing platform/solution, but a people and information one. Non-profits seeing how certain regions and stories affect people; law enforcement organizations closing loops on unsolved cases; emergency services being more effective due to increased knowledge of prior locations and situations of missing people, etc.
    Perhaps the businesses looking purely from an acquisition mindset need to take a bigger leaf out of those other organizations’ books.

  11. says

    I always get excited when reading posts around contextual social monitoring case studies – even if hypothetical. We have the monitoring solution, but it is still going to take that ANALYSIS beyond the tool to dig deep (beyond the first question answered) and get the full and varied context. I hope to see more of these hypothetical scenarios become live case studies…and I hope we get to be a part of those case studies.