As soon as a SaaS product lets users publish or share content — comments, profiles, images, reviews, public pages — the question arises how abusive content is handled. Large platforms have dedicated teams. Small SaaS providers do not, and AI models promise to close that gap. They can do a lot, but only as part of a process that starts with clear rules and ends with people on difficult cases.
Policy first
Moderation without written rules is arbitrary, with or without AI. Before a model evaluates anything, you need a policy that defines what is not allowed in the product and why: illegal content, harassment, spam, fraud, content that does not fit the product's purpose. The policy is part of the terms of service, written so users understand it, and the basis for every decision — including the instructions given to the model.
Tiered review
A proven process has several stages. First come simple, fast checks: known spam patterns, blocked links, rate limits. Then a model evaluates the content against the policy and assigns a category with a confidence estimate. Clearly harmless content is published, clearly problematic content is held back, and everything in between goes into a queue for human review.
Specialised moderation models from several providers return categories such as harassment or violence at low cost. For product-specific rules — such as competitor advertising in a review section — a general language model with your own policy in the prompt works better.
People decide edge cases
Models err in both directions: they miss abuse that is cleverly phrased and they block legitimate content such as irony, quotations or professional discussions of sensitive topics. For decisions that noticeably affect users — removal, account suspension — a person should have the final say, at least in uncertain cases. For a small team this is manageable when automated pre-sorting handles most clear cases.
Reasons and appeals
Anyone who removes or restricts content should tell the affected user which rule was broken and how to appeal. That is not only fair, it also reduces support effort and frustration. Appeals are reviewed by a person, and wrong decisions flow back as test cases into the model's evaluation.
Obligations under the Digital Services Act
Providers that store their users' content and offer services to people in the EU — including from Switzerland — fall under the Digital Services Act. Hosting services must provide a mechanism for reporting illegal content and give reasons for restrictions to the users concerned. Online platforms that disseminate content publicly have further obligations, some of which do not apply to micro and small enterprises. Which obligations apply depends on the type of service and is worth checking properly once.
Data protection
Moderation processes content and often personal data. If an external model provider is used, it belongs on the sub-processor list, and the privacy policy names the processing. Moderation decisions and their reasons are kept as long as needed for appeals and evidence, and no longer.
Measure and adjust
Two metrics matter most: how much problematic content is missed, and how much legitimate content is wrongly blocked. Both can be estimated through samples and appeals. Thresholds for automated decisions are set accordingly — more cautious for publicly visible content, more generous for private areas.
A lean setup for small teams
- A written policy as the basis for users, team and model.
- Fast rules first, model evaluation second, a queue for edge cases.
- Human decisions on removal and suspension in uncertain cases.
- Reasons for affected users and a simple appeal path.
- A reporting mechanism for illegal content under the DSA.
- Regular samples to measure errors in both directions.
Innopulse builds moderation workflows that a small team can sustain and connects them with the data protection and transparency requirements that apply to DACH providers.