Blog
Social media sentiment analysis: what it measures and when to trust it
Social media sentiment analysis classifies what people write about your brand (comments, messages, mentions) as positive, negative or neutral, and sums that classification up in a score or a percentage. To know whether the number a tool gives you can be trusted, compare its labels with the ones you assign by hand to a sample of your own comments. That tells you how accurate it is with your audience, your language and your industry.
Take this comment: “The coffee is great. But shipping took ten days and nobody told me.” It praises the product and complains about delivery. A tool that returns one score per comment may land close to zero, and a dashboard will show it as neutral. For your team, it’s a complaint that needs a reply.
Bing Liu, a researcher at the University of Illinois Chicago, defines sentiment analysis and opinion mining as the field of study that analyzes people’s opinions, sentiments, evaluations, attitudes and emotions from written language. In his 2020 book he adds that those opinions point at entities and their attributes: a product, and also its price, its shipping or its customer service. The coffee comment holds a different opinion about each.
Sentiment is one of the possible readings in social listening, which starts from a question and tries to understand conversations. This article stays with the number: where it comes from and how far you can rely on it.
How a tool scores a comment
Every tool uses its own scale. Google Cloud Natural Language makes a useful example because its documentation explains how to read it. According to its basics page, it returns two numbers per text:
- Score: from -1.0 (negative) to 1.0 (positive). It tells you which way the emotion points.
- Magnitude: from 0 to infinity. It tells you how much emotion there is, adding up positive and negative. It isn’t normalized for length: a longer text tends to have a higher magnitude.
The documentation gives these sample values:
| Sentiment | Score | Magnitude |
|---|---|---|
| Clearly positive | 0.8 | 3.0 |
| Clearly negative | -0.6 | 4.0 |
| Neutral | 0.1 | 0.0 |
| Mixed | 0.0 | 4.0 |
A score near zero can mean two opposite things: little sentiment, or a lot of sentiment pulling in opposite directions. The documentation tells them apart with magnitude: a truly neutral text has low magnitude, and a mixed one has high magnitude. If your tool only shows a “neutral” label, find out whether it separates those two cases. The coffee comment is the second kind.
Google also scores each sentence separately, with a magnitude that does run from 0 to 1 at that level. For the coffee comment, that would let you see the praise and the complaint each with its own sign.
On cutoffs, the documentation says what counts as clearly positive or clearly negative varies by use case and customer, and recommends defining a threshold that works for you. Its example: treat any score over 0.25 as clearly positive. That threshold is your decision, and the way to make it is with a sample, like the one below.
Where it goes wrong
Sarcasm
A preprint posted on arXiv in April 2025 (Bhargava et al.) measured the effect of sarcasm in one specific case: 5,929 English tweets about nuclear power, labeled by two people, with the tweets they disagreed on removed. About 43% were sarcastic.
Three language models of around 7 billion parameters (Falcon, Llama-2 and Mistral), fine-tuned by the authors on tweets about nuclear power, got the sentiment right on 50.2%, 52.5% and 51.9% of those tweets. When GPT-3.5 rewrote the same tweets without the sarcasm, accuracy rose to 67.5%, 67.5% and 69.5%. Evaluated on the sarcastic tweets alone, they scored between 31.3% and 36.9%. The paper’s abstract mentions an improvement “of up to 21%”; in its tables, the largest gain is 17.6 points, Mistral’s.
The same models fine-tuned on general-topic tweets did better on the sarcastic ones (between 60.2% and 73.6%), though worse on the full set (between 41.3% and 48.0%).
It’s one study, one topic, one language and models trained by the authors themselves. It doesn’t tell you how often the tool you use gets things wrong. It does show two things worth keeping: sarcasm can drag accuracy down a lot, and the texts a model was trained on change how much it’s affected.
Emojis, less than you’d expect
In the same study, replacing emojis with their text description barely moved the results of the three large models: Falcon gained 0.8 points, Llama-2 gained 2.9 and Mistral lost 0.9. The authors conclude that, in certain contexts, emojis may only reinforce the text’s sentiment rather than reveal it. The tricky case in your comments is a different one: the emoji that contradicts the text, like a 🙃 after a complaint, which in practice is sarcasm.
Your language and your slang
Tools publish which languages they analyze sentiment in. Google’s includes Spanish, English and Portuguese in its list of supported languages. Your language being on the list says nothing about how the tool handles your slang, your abbreviations or comments that mix two languages. None of the sources in this article measures that for your case: your sample does.
Overall accuracy
SemEval-2017 Task 4 was an academic competition on sentiment analysis in Twitter. In the task of classifying English tweets as positive, negative or neutral, 38 teams took part. Almost half the test tweets were neutral: 5,937 of 12,284, or 48.3%. A system that answered “neutral” every time would therefore be right 48.3% of the time without catching a single complaint.
That’s why the organizers didn’t use overall accuracy as the main measure. They used the average, across the three classes, of the share of each class the system identifies correctly. That measure holds up better when one class is far more common than the others. On it, the “all neutral” system scores 0.333, and the two best teams reached 0.681.
If most of your comments are questions or greetings, “80% accurate” can hide the fact that the tool catches few negatives. Ask for accuracy by class, or work it out yourself.
How to check whether a tool gets your comments right
- Gather a real sample
Recent comments from your accounts, across several posts, without picking the most striking ones. Note the dates, accounts and language so you can repeat the test the same way later.
- Write the rule before you label
Use three labels (positive, negative, neutral) and decide what to do with mixed comments. A useful rule for a brand: if a comment praises and complains at once, it counts as negative, because it needs a reply.
- Label by hand without looking at the tool
If two people can do it separately, even better. Where they disagree, revisit the rule before going on. Both studies cited here used more than one person per text.
- Run the same comments through the tool
Record the score, not just the label. With the score you can try different thresholds without running anything again.
- Compare by class and choose the threshold
For negatives, look at two things: of the ones you labeled, how many the tool finds; and of the ones it flags, how many really are negative.
Example: 20 comments for a coffee brand
Grano Lento is a made-up brand. The comments, labels and scores are fictional: the scores use the -1 to 1 scale described above, but they aren’t output from any real tool.
The Grano Lento team labels 20 comments from one month using the rule in step 2: 8 positive, 6 negative and 6 neutral.
| No. | Comment | Team’s label | Tool’s score |
|---|---|---|---|
| 1 | The Huila is the best coffee I’ve ever made at home. | Positive | 0.9 |
| 2 | Arrived in two days and well packed, thanks. | Positive | 0.8 |
| 3 | Tasty, smooth, just what I was looking for. | Positive | 0.8 |
| 4 | Loved the tasting note that comes in the bag. | Positive | 0.7 |
| 5 | The French press you recommended works perfectly. | Positive | 0.6 |
| 6 | Good coffee, good price. | Positive | 0.5 |
| 7 | Thanks for replying so fast by message. | Positive | 0.4 |
| 8 | On my third bag of the house blend now. | Positive | 0.2 |
| 9 | The package arrived broken and nobody answers. | Negative | -0.8 |
| 10 | I ordered a French press grind and it came as whole beans. Second time this happened. | Negative | -0.6 |
| 11 | Three weeks waiting for the refund. | Negative | -0.4 |
| 12 | Great, express shipping took ten days again 🙃 | Negative | 0.3 |
| 13 | Excellent service: four days waiting for someone to reply. | Negative | 0.2 |
| 14 | The coffee is great. But shipping took ten days and nobody told me. | Negative | -0.1 |
| 15 | Do you ship outside the city? | Neutral | 0.0 |
| 16 | Is the Ethiopian more acidic than the Colombian? | Neutral | 0.0 |
| 17 | Price per kilo? | Neutral | 0.0 |
| 18 | How long does ground coffee last once opened? | Neutral | 0.1 |
| 19 | Do you have decaf? | Neutral | -0.1 |
| 20 | What happens if I’m not home when the order arrives? | Neutral | -0.3 |
What the comparison shows
With cutoffs at 0.25 and -0.25 (positive from 0.25 up, negative from -0.25 down, neutral in between), the tool agrees with the team on 15 of the 20 comments: 75%. By class, the picture changes: it gets 7 of the 8 positives, 5 of the 6 neutrals and only 3 of the 6 negatives.
The totals mislead too. The tool counts 8 positives out of 20 (40%), the same as the team, but they aren’t the same comments: it includes number 12, a sarcastic complaint, and leaves out number 8, from someone already on their third bag. For negatives it counts 4 of 20 (20%) against the team’s 6 of 20 (30%). A matching final percentage doesn’t prove the tool classifies well.
The team uses the tool for one job only: deciding which comments to read first. It tries three thresholds for flagging a comment as negative:
| Flag as negative if the score is | Comments flagged | Truly negative among those flagged | Negatives found, out of 6 |
|---|---|---|---|
| -0.25 or lower | 4 | 3 of 4 (75.0%) | 3 (50.0%) |
| -0.1 or lower | 6 | 4 of 6 (66.7%) | 4 (66.7%) |
| 0 or lower | 9 | 4 of 9 (44.4%) | 4 (66.7%) |
It picks -0.1. That finds one more negative than -0.25 (number 14, the mixed comment) at the cost of reading one more question (number 19). The two questions left in the queue, 19 and 20, need an answer anyway. Going down to 0 finds no new negatives and adds three more comments to the queue. If the tool returned magnitude, number 14 would be the case Google’s documentation describes as mixed: a score near zero with high magnitude.
No threshold recovers 12 or 13: both are sarcastic and have positive scores. For those, the team adds a rule that doesn’t depend on the score: any comment that mentions shipping, a refund or customer service gets read, whatever it scores.
Twenty comments is a small sample: each one moves the percentage by 5 points. It’s enough to see how the tool goes wrong with your audience. To set the threshold with more confidence, repeat the test with a bigger sample from another month.
Which decisions a sentiment metric can support
| Decision | Does it support it? | Condition |
|---|---|---|
| Deciding which comments to read first | Yes | With a threshold tested on your sample and a person reading what gets flagged |
| Spotting a rise in negatives after a post | Yes, as an alert | Read those comments before concluding what caused it |
| Seeing whether the tone changed from one month to the next | With care | Same tool, same threshold, same accounts and same language. With small samples, a change of a few points may be noise |
| Saying what your customers or the market think | No | Only people who choose to comment have a say. That question belongs to social listening, with its sources and limits |
| Comparing your sentiment with another brand’s measured in a different tool | No | Different scales and thresholds can’t be compared |
| Hiding, deleting or replying to a comment based on its score alone | No | Sarcasm and mixed comments get misclassified. That decision calls for reading the comment |
If you reply from an inbox that brings comments and messages together, like HeyMark’s, where Mark replies with your approval and moderates comments, your team already reads those comments every day. That inbox also shows the sentiment of the comments on each post, and it’s an indicator like any other: labeling twenty comments once a month takes little time and tells you how much to trust that indicator, or the one from whatever tool you use.
Plan your next post in HeyMark.
Keep the idea, review the draft with your team, and see how it performed in the accounts you connected.