Data Sources and Search Volumes Results

There's a lot of variance in data sources and volumes.

There's big differences in data sources and search volumes.In an ideal world, the query would have pulled similar, if not identical, data sources and search volumes across the five vendors.

However, in the real world, this did not happen. The search produced varying results across all five vendors. In most cases, the discrepancies were not extreme, but still large enough to raise some concerns.This project has given us new questions about the sourcebooks and crawling techniques of each vendor. Hopefully, The Social Intelligence Lab will be able to provide more details on these techniques. The discrepancy in the volumes was most noticeable on Twitter. We believe that this could have been caused by several factors:

Language: The ability to recognize and assign particular languages to a piece of content

Automatic Wildcard Operators: NetBase runs wildcard operators as a default option so “craft beer” automatically pulls mentions of “craft beerS” too

Twitter API Access and Decahose Restrictions: These are subject to the vendor and Twitter’s contractual agreements.

Crawling Script Criteria: Some platforms look at the tweet and if the Tweet contains a link, the algorithm checks for the words featured in the URL. In a similar fashion, some platforms search the forum thread title and the post for keywords, not just the forum post.

Proprietary Crawlers: The use of proprietary in house developed crawlers, buying content from third-party providers (Twitter, LexisNexis/MoreOver, SocialGist.com, etc.) or a combination of both.

Examples of Crawling Script Criteria.The first example across highlights that there is no mention of "beer" in the original twee, but there was a mention of "beer" in the link and article. In the second example, you see the mention of "beer" in the thread rather than in the title of the forum thread.