With social media use gaining strong ground in the midmarket, it's never too soon to check that metrics are in place and working well. That's easier said than done, of course. There's no shortage of vendors who offer social media metric solutions, even claiming to mine useable real-time data from the flood of feedback flowing along a range of social media channels.
One sharp challenge for social media metrics is sentiment analysis. Evidently, the channels through which customers react and respond to marketing initiatives throb with sentiment. Whether primarily concerned with B2B or B2C strategies, the midmarket enterprise needs to be sensitive to the ripples of sentiment about their offerings detectable in the social stream.
The problem is that sentiment analysis is notoriously unreliable, especially when automated -- and when channels are super-saturated with data, automated analytics are really the only game in town.
In a 2010 study, the social media agency FreshNetworks compared results of automated sentiment analysis of the Starbucks brand by several social media tools with the findings of a human analyst. It was all bad news for the automated tools. Although accuracy levels looked good, the results were skewed by the typically high level of neutral comments. The sentiment metrics tools could identify these correctly, giving an impression of high accuracy: The problem lay in correctly distinguishing positives from negatives in the smaller pool of non-neutral comments.
Compared with human results, automated sentiment analysis was right about positive or negative comments about 30 percent of the time, a result appropriately described as "disastrous." Have things gotten any better since 2010?
A recent discussion at Social Media Explorer doesn't provide much comfort, but does indicate some ways in which sentiment analysis tools might be improved, and in particular, what enterprises should look for when assessing the competing claims of analysis vendors.
Be aware, for example, of the challenges facing automation. Identifying the general sentiment of a message board post is one thing, identifying sentiment about a specific topic of interest within the post may be more difficult.
Fantastic day, very happy and pleased with everything (not so much the coffee).
It's easy to imagine sentiment metrics rating such a message as highly positive -- which is unhelpful if you're the coffee vendor.
Look out for proffers which allow you "train" the tool. There is no need to settle for "out of the box" accuracy. Generalized algorithms and taxonomies may not be well adapted to the needs of specific enterprises. Look for the option of customizing the analysis by importing your own word or phrase lists, and make sure the tool can be tuned to the channels of most interest.
Finally, be aware that accuracy is not necessarily everything. Capturing small numbers of highly positive leads using a sentimental analysis tool which misses large numbers of averagely positive leads might have sufficient business impact to be worthwhile. Also, depending on the business in question, it might be much more important to identify powerfully negative responses than powerfully positive (or vice versa).
When it comes to sentiment analysis, as with so many other smart business solutions, one-size-fits-all is not the way to go.
— Kim Davis , Community Editor, Internet Evolution