The Future of Insight is From Language Data

Converseon explores the value of language data and they share ten ways you can easily leverage unstructured data with advanced machine learning.

The Future of Insight is From Language Data

Converseon explores the value of language data and they share ten ways you can easily leverage unstructured data with advanced machine learning.

Rob Key
Founder and CEO
Converseon.aiOrganizations are awash in untapped, insight-rich unstructured data. Customer feedback and unprompted opinion through social media, product reviews, long-form survey verbatims, call center transcripts and more are veritable goldmines of insight for those customer-obsessed companies that can effectively harness, filter, process, and understand this massive, messy data set. Computer World magazine forecasts that unstructured information might account for more than 70%–80% of all data in organizations.Yet organizations today face a conundrum: even as this data set grows exponentially, most brands are processing and using only a small portion of it— Forrester Research says most organizations are processing less than 21% of this unstructured data. And with some good reason: this unprompted “language data” is complex. Implicit meaning, sarcasm, slang context and much more make it challenging to separate the signals from the noise and make the data actionable in a time span needed for competitive advantage.Today, however, a growing number of organizations are leveraging advanced natural language processing and text analytics solutions powered by artificial intelligence that are proving to be game-changers and allowing these firms to begin to fully leverage the long untapped value of this data set.But doing so requires a thoughtful and clear methodology and approach that builds on the latest data science, machine learning validation, and processes. While this article focuses largely on social media, the approaches and lessons can also be applied, with some modification, to other unstructured data sources.What is social media intelligence? You’ve probably heard the term social intelligence, but there is a big difference between a hype-laden phrase and real business value. With the social media explosion now a decade old, it’s time for marketers to finally extract the value out of social media conversations—that’s social intelligence.With social intelligence technology, you can identify the conversations that matter to you and classify them into standard or custom categories that allow you to see trends and make better business decisions. “Intelligence” refers not only to the accuracy of the model but also how well the model meets business requirements. And by combining social intelligence with advanced analytics, researchers and analysts are discovering vast consumer predictive intelligence. Properly utilized and implemented, social intelligence is proving to have strong quantitative value and predictors of business determinants, such as sales, survey-based tracker results, new trends and more.What can social intelligence models tell us?Language is highly complex. Most social-based conversations, for example, do not use specific keywords but are instead conveying implicit opinions. A person likely will not say, “I trust x brand,” but instead likely will “use it with my baby at night because my baby sleeps better.” In the latter, there is no word specific to connoting trust, but most humans will understand it, and now today, with more advanced natural language understanding technology powered by machine learning, algorithms can process and categorize this level of nuance too - “like humans do.” Social intelligence requires applying these more advanced annotations —data that is added to the original social conversations to help us categorize and aggregate them -- -- powered by more sophisticated and accurate algorithms. But since we’re dealing with the complexities of human language, these more advanced and accurate NLP algorithms (or “models”) need to be steeped in a framework -- organizing principles that help define specific annotations and how those annotations get applied to specific sectors. What does “trust” mean for example? What are the inputs and drivers of trust? And can you define it in a way where humans agree on it? Social intelligence requires “intelligent” algorithms - highly precise, and accurate, detailed, able to classify even nuanced concepts and language and grounded in meaningful frameworks that are consistent and effective for use. These new intelligent algorithms are able to classify:Some standard annotations that social intelligence models produce are:

Sentiment: Is the conversation positive towards your brand, negative, or neutral? Sentiment models are designed to answer that question correctly.

Emotion: Certainly, anger and sadness both convey a negative sentiment, but they might warrant different responses to the customers experiencing the Plutchik’s Wheel of Emotions, which is one way of categorizing emotions, while other social scientists have designed alternative models.

Intensity: The strength of passion behind an opinion can itself be measured. “It had a slightly butter off-note” is not as strong as “That is the worst-tasting $^#% I have ever tasted”.

Voice of Customer: Only about 10% of social discussion is from actual consumers (the rest being noise from job listings, advertisements and more). VoC models can filter out all that noise to get to actual consumer opinion and a critical first step for building more complex models for product innovation, customer experience, and brand measurement.

Trust: Brand trust reflects a customer’s expectation that a product or service (and sometimes corporate behavior) reflect the promises of the brand. Trust is a key quality of any relationship where customers make a purchase, yet brand trust sometimes fluctuates significantly over time. Identifying comments that exhibit brand trust can be challenging, as in the example, “I would be reluctant to use a different brand of shampoo on my infant’s hair.”

Values: Consumers today expect brands to not only build superior products but also help make the world a better place and take action on important “value.” CSR (“corporate social responsibility”_ models can measure how consumers perceive company efforts including in the areas of brand purpose, environment, and related social issues.

Innovation: In high tech, consumer electronics, and many other industries, a brand’s reputation for innovation is a critical part of the buyer’s decision-making. Finding the right conversations that reflect innovation is also not easy, because people often comment on things that are new, but not all of them are innovative.

Often, however, what really matters are specific insights about your unique business questions, which requires a custom model that captures specific concepts unique to your company. Custom models classify language “like humans do”, and you’ll finally be able to realize the full value of social data:

Buyer Journey Stages: Each company (and sometimes each product within a company) overlays a buyer journey onto its customer interactions. One company might have a four-step journey and another might have a six-step journey and each one uses different names for its steps.

Brand Health/Attributes: Do conversations about your brand align with attributes or not, and are those attributes even relevant or in demand in social conversations?

Customer Intent: Is the customer trying to buy something or looking for customer service? Or something else? Different businesses attract people for different purposes at different times.

Emerging Trends: Social data is rich in insight into emerging trends and discoveries before they hit mainstream recognition. Machine learning models can be critical to accelerating and improving innovation at brands by finding these new trends first.

How can you use social intelligence models?Social intelligence can serve a number of use case within your enterprise, including:

Advocacy: Who is engaging in conversation that “sell” your brand? As with customer experience, it is sometimes valuable to understand the overall numbers by also sometimes important to identify individual influencers.

Brand Tracking: Trust, innovation, and safety are all common attributes that many companies need to track on an ongoing basis.

Crisis Management: Communication professionals need to monitor breaking stories that negatively affect brand image.

Customer Experience (CX): Customer experience is the product of an interaction between an organization and a customer over the duration of their relationship. It can be measured in aggregate across many customers’ testimonials on social media or it can be segmented down to the individual level to support focused retention efforts.

Customer Service: Support team identify complaints in social media and reach out to resolve them.

Market Research: Marketers use social media conversation to understand the wants and needs of their market.

Product Development: Product managers mine social media to determine popular product features and identify needed features.

Recruiting: Human resources personnel can identify potential employees through social media conversations.

Reputation Management: Marketers and communications professionals assess the brand image across the social media population.

Sales Leads: Salespeople use social media to identify potential purchasers for their offerings.

Why measurement mattersIt’s often said that you can’t manage what you can’t measure. For many years, the quality of the data processed through social listening platforms has been opaque at best and disappointing-to-unusable for insights at worst.It has been difficult, if not impossible, for analysts to clearly measure the accuracy of data in their social listening platforms. As a result, the adoption of social intelligence has too often been stunted by a lack of trust in this massive, messy, unstructured dataset. Market research professionals often look at social data with skepticism because of concerns about accuracy. Senior executives naturally hesitate to accept insights and findings without a clear understanding of the true, quantitative nature of the data.And with good reason. “Accuracy”—how well systems match the consensus of humans—has often been only slightly better than a coin flip. Additionally, many technologies miss many customer opinions, leading to well-deserved hesitation by market research professionals who cannot effectively integrate this data into advanced analytics models, or use the data to report on key trends to senior executives.There is good news, however. By directly learning from humans, machine learning algorithms are beginning to unlock the full value of this massive, real-time insight resource.The convergence of AI with Natural Language Processing (NLP) and social Voice-of-Customer (VoC) data is clearly a critically important development for customer-centric organizations. For many leading organizations, it is representing an entirely new generation of insights, including predictive insights, that are helping to transform brand guidance, reputation management, customer care, customer experience, trend discovery, and marketing research more generally.