AI Content Moderation for Comments, Reviews and Chats

How it works in short

AI moderation automatically checks comments, reviews and messages before or right after publication. A working setup is almost always layered:

1. Fast rules: banned words, links, phone numbers, repeated messages.
2. Moderation API: ready-made models that score text by category (insults, threats, sexual content and so on) and return probabilities.
3. LLM classifier: a language model with your policy in the prompt, for context, sarcasm and platform-specific rules.
4. A human: for borderline cases and appeals.

Each layer filters part of the stream, so only what truly needs attention reaches the expensive stages.

Start with a policy, not a model

Without a written policy, moderation is unpredictable for both users and the model. The policy should answer:

• which categories are forbidden (insults, spam, competitor ads, personal data, threats);
• what is allowed: harsh but reasoned criticism in a review is not an insult;
• what happens in each case: hide, send for review, warn the author, block;
• "allowed / forbidden" examples for every category.

The same examples later go into the LLM classifier prompt and the test set.

Uzbek, Russian and slang

Off-the-shelf moderation APIs work best in English. For Uzbek, especially in Latin, Cyrillic and mixed spelling, and for Russian-Uzbek code-switching, accuracy is usually lower. What to do:

• Build your own test set from real comments on your platform: a few hundred labeled examples per category and language.
• Test every tool on that set before choosing. Do not rely on claimed language support.
• Normalize text before checking: lowercase it, collapse repeated letters, handle character substitutions (digits for letters, Latin for Cyrillic).
• Add a slang glossary to the LLM classifier prompt, explaining what each word means and when it is offensive.
• Remember both Uzbek alphabets: the same word may arrive in Cyrillic or Latin.

An LLM classifier example

Ask the model to reply in a strict format so code can process the result:

{
  "category": "insult",
  "confidence": 0.82,
  "language": "uz-latn",
  "reason": "Direct insult aimed at another user"
}

The prompt contains the policy, examples, the list of categories and a requirement to return JSON only. The confidence value from the model is a rough signal, not a calibrated probability, so tune thresholds on your own test set.

Thresholds and decisions

Do not split everything into "allow" and "delete". Three zones work better:

Thresholds differ by category: threats and personal data should go to review even at modest confidence, while mild rudeness can pass more often. Revisit thresholds using real data: how many false blocks and how much slipped through.

Appeals and human review

• The author of a blocked message should see the reason and an Appeal button.
• Appeals and the medium zone land in one moderator queue with the text, category and the model's explanation.
• Save moderator decisions: they become new labeled examples for the prompt and tests.
• Keep a decision log to investigate complaints and check whether the model became stricter or softer after changes.

Common mistakes

• Moderating without a policy, so every decision looks random.
• Testing only on English examples.
• Automatic blocking with no explanation and no appeal.
• Sending full personal data to an LLM when it is not needed.