Content moderation
Screened as it is sent, and kept either way
Messages are checked against a word list before they leave, a fuller scan follows behind, and anything a person needs to weigh goes to a queue. Blocking does not mean deleting — the message somebody tried to send is exactly the thing a safeguarding review needs later.
- 5
- milliseconds, at most, before a message is let go
- 0
- blocked messages that are deleted
- 0
- buttons that approve more than one thing at a time
- 0
- reviewers ever shown suspected illegal material
- Inline check in under five milliseconds — nobody waits to send
- A blocked message is stored, not destroyed
- Anything that needs judgement goes to a named person, not a score
As it is sent
Two checks at two speeds, because they answer different questions
One has to be instant because a person is waiting. The other can afford to be thorough because nobody is.
A local word list, inline
Runs against a list held on our own servers, in under five milliseconds. No network call, nothing sent anywhere, and no perceptible pause between pressing send and the message going.
Then a fuller scan, behind
The slower checks run after the message is on its way, so a scanner having a bad minute never becomes a young person unable to reach an adult. Anything it finds raises a flag rather than reaching back to undo a delivery.
And a blocked message is still stored
This is the part people assume works the other way. Blocking means the recipient never sees it — not that it stops existing. The message somebody tried to send is often exactly what a safeguarding review needs to see, and destroying it at the moment of creation is the wrong instinct on a product like this.
The word list is a blunt instrument and is treated as one. It catches what it catches; the concern-alert scanning is what looks for behaviour, and a leader reading a thread is what catches the rest.
The part a person does
Automatic screening decides what to show a human, not what to conclude
Every automated check here ends at a person. None of them closes anything.
The leaders who already oversee that young person
Not a moderation team somewhere else, and not a shared inbox. The same people who can already read the conversation, because they are the ones with the context to judge it.
The message, in its thread
With what came before it and what came after. A flagged sentence on its own is how a reviewer reaches a confident wrong conclusion, and the queue links straight into the conversation rather than quoting a fragment at them.
And nothing can be actioned in bulk
There is no button anywhere in this product that approves more than one thing. Approving means a person looked at it, and a control that approves forty means somebody did not.
Every decision is recorded with the person who made it, in the same transaction as the decision itself.
Built, and deliberately dark
Image moderation is finished and not switched on
Said plainly because the alternative is a reader assuming it is running, and finding out otherwise at the worst moment.
What exists
Classifiers, the mapping from a verdict to an action, the hold-approve-reject workflow, the reviewer's queue, the record of who revealed what, and the decision trail. All of it built, tested, and reachable.
Why it is off
Pictures themselves are not enabled, and no jurisdiction configured in this product currently permits them. Turning images on is a decision about children's data that belongs to an organization and its regulator, not to us — so the machinery waits rather than shipping ahead of the decision.
And one rule that will not change when it is on
A reviewer is never shown suspected illegal material. It is reported, not reviewed. That is a duty of care to the adults doing the reviewing, and it is why the reporting route exists separately from the review route rather than as a button beside it.
When images are switched on, the limits an organization may set are bounded by its jurisdiction in both directions — it can be stricter than the floor and never looser, exactly as every other safety setting here behaves.
What you decide
An organization tightens its own screening. It cannot loosen it
The same rule that runs through every safety setting in this product.
Yours to set
Additional terms your organization wants caught, how strict the scanning is, and who reviews the queue. Anything that makes screening tighter is yours.
Not yours to set
Whether screening runs at all, whether images are permitted, and the categories a jurisdiction requires. An organization under pressure to reduce false alarms is exactly who would switch those off, which is why it cannot.
And every change is attributable
Who changed a screening setting, when, and the reason they gave. A safety setting that changed with nobody's name against it is the thing an audit exists to prevent.
The refusal comes from the database, so it holds whether the request arrived from a settings screen, an API route, or a script somebody wrote in a hurry.
In short
What to tell your board
Three sentences, all of them checkable.
- Messages are checked against a local list before they send and by a fuller scan behind, and a blocked message is stored rather than destroyed.
- Everything automated ends at a leader who already oversees that young person, and nothing in this product can be approved in bulk.
- Image moderation is built and switched off, because enabling pictures is a decision for an organization and its regulator rather than for us.