Nine in Ten Reports Never Reach a Human: A Moderation Q&A
Archi4 min read
Contents
- The Queue That Closes Itself
- Your headline number is nine in ten. Is that unusual?
- What does the machine actually decide?
- How often is the classifier right?
- What happens to the cases that do reach a person?
- Where the time goes
- Do appeals change anything?
- If you wanted one number to change, which one?
- What should an ordinary user do differently?
The Queue That Closes Itself#
Nine out of ten reports filed on a typical mid-sized social platform are closed by software, not a person. The classifier reads the flag, scores it against a policy list, and resolves the ticket in roughly two seconds. The tenth that reaches a human gets about 45 seconds of attention before the reviewer moves to the next case.
We spoke with a moderation operations lead who has run queues at two platforms and now advises a third. They asked not to be named because they still do contract work for one of them. The conversation below has been edited for length and clarity.
Your headline number is nine in ten. Is that unusual?#
It is the shape of the pipeline more than a fixed ratio, and it varies by week. Every platform I have worked on front-loads automation because the volume is unmanageable otherwise. A mid-sized app can take a few hundred thousand reports in a bad week. Human review runs at a small fraction of that pace even when quality is high. So the default is: machine closes it, and a person sees it only if the score lands in a narrow band.
What does the machine actually decide?#
Three things, in order. Is this report about a real policy line, or is it a disagreement with someone's opinion? Is the content still up? And does the classifier score cross the threshold for removal? Most reports die on the first question. People flag things they find rude, or wrong, or aimed at them personally, and none of that is against the rules.
How often is the classifier right?#
In the audits I have sat through, roughly seven in ten automated closures hold up when a human later looks at them. That sounds decent until you see where the other three sit. The errors are not spread evenly. They cluster. , Sarcasm and deadpan jokes, especially in text-only posts with no image., Reclaimed language used inside a community that has claimed it., Languages with thin training data, where a slur list was translated and never checked by a native speaker., Coded terms that rotate every few weeks, so the model is always a step behind., Quoted abuse inside a post that is condemning the abuse.
What happens to the cases that do reach a person?#
Forty-five seconds is honest. The reviewer sees the post, the report reason, sometimes a short thread, and a checkbox. No history, no sense of whether these two accounts have been fighting for a month. The reviewer is measured on decisions per hour and on agreement with the quality sample, and those two pressures pull in opposite directions. People learn to move fast and pick the answer that is easiest to defend later.
Where the time goes#
A reviewer's shift is mostly context reconstruction, not judgment. Finding the post, checking whether it was edited, checking whether the reporter was also in the thread. The actual decision is a few seconds. That is why context in the report matters more than people expect.
Do appeals change anything?#
Only a small share of takedowns get appealed, often single digits as a percentage. Of those that do, a minority are reversed, and the reversal rate is noticeably higher when the appeal includes the exact line, a timestamp, and a policy citation. Appeals that say "this is unfair" with nothing else almost never move.
If you wanted one number to change, which one?#
Time to first human look, not the overall automation rate. A post that sits for two days in an automated hold has already done whatever damage it was going to do, and a wrongful removal that takes a week to reverse has already cost the account its reach. Speed of the second look is the number I would put on a dashboard.
What should an ordinary user do differently?#
Report the specific line, not the account. Quote it, add a timestamp, name the behavior. Do not organize a mass report against someone you are arguing with, because coordinated flagging trips fraud filters and buries the real complaint. If you are on the receiving end, appeal once, with the text of the rule you think was misapplied, and skip the essay about your intentions.
The two-second closure is not going away. The platforms cannot afford the alternative, and users would not like the version where every report waits for a person. What can change is the handoff: which cases get pulled out of the automated stream, and how quickly a human sees the ones that matter. That is a staffing and design decision, and it is made by people who read dashboards, not by the classifier.
More articles