Skip to content

Moderation provider docs #2619

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 15, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions src/data/nav/chat.ts
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,14 @@ export default {
name: 'Hive (Dashboard)',
link: '/docs/chat/moderation/direct/hive-dashboard',
},
{
name: 'Tisane',
link: '/docs/chat/moderation/direct/tisane',
},
{
name: 'Bodyguard',
link: '/docs/chat/moderation/direct/bodyguard',
},
],
},
{
Expand Down
10 changes: 1 addition & 9 deletions src/pages/docs/chat/moderation/custom/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,7 @@ Before publish moderation is where your moderation logic is invoked before the m

### Integration configuration <a id="configuration"/>

To fine-tune how Ably handles messages according to your use-case, you can configure before publish rule behavior using the following fields:

| Field | Description |
| ----- | ----------- |
| Retry timeout | Maximum duration (in milliseconds) that an attempt to invoke the rule may take (including any retries). |
| Max retries | Maximum number of retries after the first attempt at invoking the rule. |
| Failed action | The action to take in the event that a rule fails to invoke. Options are reject the request or publish anyway. |
| Too many requests action | The action to take if your endpoint returns `429 Too Many Requests`, which may happen if your endpoint is overloaded. The options are to fail moderation, or retry. |
| Room filter (optional) | A regular expression to match to specific chat rooms. |
To fine-tune how Ably handles messages according to your use-case, see the [common configuration fields](/docs/chat/moderation#common-config) shared across all before-publish moderation rules.

### The API <a id="api"/>

Expand Down
29 changes: 29 additions & 0 deletions src/pages/docs/chat/moderation/direct/bodyguard.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
title: Bodyguard
meta_description: "Detect and remove unwanted content in a Chat Room using Bodyguard AI."
---

[Bodyguard](https://bodyguard.ai/) is a powerful contextual analysis platform that can be used to moderate content in chat rooms.

The Bodyguard integration can be applied to chat rooms so that you can use Bodyguard's content moderation capabilities to detect and handle inappropriate content before it's published to other users.

## Integration setup <a id="setup"/>

Configure the integration in your [Ably dashboard](https://ably.com/accounts/any/apps/any/integrations) or using the [Control API](/docs/account/control-api).

The following fields are specific to Bodyguard configuration:

| Field | Description |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've now got 4 rules that will have a lot of field crossover (the before publish rule config). Worth extracting the common fields to an overview page (like we have for "custom") so that we don't have to repeat ourselves on every page (and just have a callout to that page on the individual provider pages)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point! have extracted them all out now, see f0dbdf0

| ----- | ----------- |
| Bodyguard API key | The API key for your Bodyguard account. |
| Channel ID | The ID of your Bodyguard channel where moderation rules are configured. |
| Default Language (optional) | The default language to use for content analysis. This will be used as a fallback in case automatic language detection fails. |
| Model URL (optional) | A custom URL if using a custom moderation model. |

For additional configuration options shared across all before-publish moderation rules, see the [common configuration fields](/docs/chat/moderation#common-config).

Messages will be rejected if Bodyguard's analysis returns a `REMOVE` recommended action based on the moderation rules configured in your Bodyguard channel.

## Handling rejections <a id="rejections"/>

Messages are rejected when they fail Bodyguard's analysis. Bodyguard returns a REMOVE action in these instances and the messages will not be published to your channel. The publish request will be rejected. Moderation rejections will use the error code `42213`.
7 changes: 2 additions & 5 deletions src/pages/docs/chat/moderation/direct/hive-model-only.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,8 @@ The following are the fields specific to Hive (model only) configuration:
| Hive API key | The API key for your Hive account. |
| Thresholds | A map of text [text moderation classes](https://docs.thehive.ai/reference/text-moderation) to [severity](https://docs.thehive.ai/docs/detailed-class-descriptions-text-moderation). When moderating text, any message deemed to be at or above a specified threshold will be rejected and not published to the chat room. |
| Model URL (optional) | A custom URL if using a custom moderation model. |
| Retry timeout | Maximum duration (in milliseconds) that an attempt to invoke the rule may take (including any retries). The possible range is 0 - 5000ms. |
| Max retries | Maximum number of retries after the first attempt at invoking the rule. |
| Failed action | The action to take in the event that a rule fails to invoke. Options are reject the request or publish anyway. |
| Too many requests action | The action to take in the event that Hive returns a 429 (Too Many Requests Response). Options are to fail rule invocation, or retry. |
| Room filter (optional) | A regular expression to match to specific chat rooms. |

For additional configuration options shared across all before-publish moderation rules, see the [common configuration fields](/docs/chat/moderation#common-config).

## Text length <a id="text-length"/>

Expand Down
29 changes: 29 additions & 0 deletions src/pages/docs/chat/moderation/direct/tisane.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
title: Tisane
meta_description: "Detect and remove unwanted content in a Chat Room using Tisane AI."
---

[Tisane](https://tisane.ai/) is a powerful Natural Language Understanding (NLU) platform that can be used to moderate content in chat rooms.

The Tisane integration can be applied to chat rooms so that you can use Tisane's text moderation capabilities to detect and handle inappropriate content before it's published to other users.

## Integration setup <a id="setup"/>

Configure the integration in your [Ably dashboard](https://ably.com/accounts/any/apps/any/integrations) or using the [Control API](/docs/account/control-api).

The following are the fields specific to Tisane configuration:

| Field | Description |
| ----- | ----------- |
| Tisane API key | The API key for your Tisane account. |
| Thresholds | A map of [text moderation categories](https://docs.tisane.ai/apis/tisane-api-response-guide#supported-types) to severity levels (`low`, `medium`, `high`, `extreme`). When moderating text, any message deemed to be at or above a specified threshold will be rejected and not published to the chat room. |
| Default Language | The language to use for content analysis. |
| Model URL (optional) | A custom URL if using a custom moderation model. |

For additional configuration options shared across all before-publish moderation rules, see the [common configuration fields](/docs/chat/moderation#common-config).

## Handling rejections <a id="rejections"/>

If a message fails moderation the message will not be published and the publish request will be rejected.

Moderation rejections will use error code `42213`.
28 changes: 27 additions & 1 deletion src/pages/docs/chat/moderation/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,18 @@ When using before publish moderation, a message is reviewed by an automated mode

This approach provides additional safety guarantees, but may come at the cost of a small amount of latency, as messages must be vetted prior to being published.

#### Common configuration fields <a id="common-config"/>

All before-publish moderation rules share the following configuration fields:

| Field | Description |
| ----- | ----------- |
| Retry timeout | Maximum duration (in milliseconds) that an attempt to invoke the rule may take (including any retries). The possible range is 0 - 5000ms. |
| Max retries | Maximum number of retries after the first attempt at invoking the rule. |
| Failed action | The action to take in the event that a rule fails to invoke. Options are reject the request or publish anyway. |
| Too many requests action | The action to take in the event that the moderation provider returns a 429 (Too Many Requests Response). Options are to fail rule invocation, or retry. |
| Room filter (optional) | A regular expression to match to specific chat rooms. |

### After publish <a id="after-publish"/>

When using after publish moderation, a message is published as normal, but is forwarded to a moderation engine after the fact. This enables you to avoid the latency penalty of vetting content prior to publish, at the expense of bad content being visible in the chat room (at least briefly). Many automated moderation solutions are able to process and delete offending messages within a few seconds of publication.
Expand All @@ -37,12 +49,26 @@ Alternatively, you might have a custom solution you wish to integrate with, or a

[Hive](https://hivemoderation.com) provide automated content moderation solutions. The first of these is the [model only](/docs/chat/moderation/direct/hive-model-only) solution, which provides access to a powerful ML model that takes content and categorises it against various criteria, for example, violence or hate speech. For each classification, it also provides an indication of the severity of the infraction. Using this information, you can determine what level of classification is appropriate for your chat room and filter / reject content accordingly. Hive offer free credits to allow you to experiment with this solution.

The second solution is the [dashboard](/docs/chat/moderation/direct/hive-dashboard). This is an all-in-one moderation tool, that allows you to combine automated workflows using ML models as well as human review and decisions to control the content in your chat room.
The second solution is the [dashboard](/docs/chat/moderation/direct/hive-dashboard). This is an all-in-one moderation tool, that allows you to combine automated workflows using ML models as well as human review and decisions to control the content in your chat room.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not this PR but a quick fix - there's a few links in the direct vs custom section that are broken

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will fix in a separate PR!


Ably is able to integrate your chat rooms directly directly with both of these solutions, to allow you to get up and running with Hive moderation with minimal code required.

### Tisane <a id="tisane"/>

Tisane provides automated content moderation through their [API solution](https://tisane.ai/). The platform uses Natural Language Understanding (NLU) to analyze user-generated content and categorize it against various criteria for content moderation. Each analysis includes information about the type and location of potentially problematic content, enabling you to determine appropriate moderation actions for your chat room.

Tisane also offers an [on-premises solution](https://tisane.ai/) for organizations that need to deploy the moderation engine within their own infrastructure. Both options can be integrated with Ably to provide automated content moderation for your chat rooms.

### Bodyguard <a id="bodyguard"/>

Bodyguard provides content moderation through their [API solution](https://bamboo.bodyguard.ai/api). The service performs contextual analysis of messages to detect and categorize inappropriate content based on configurable criteria. Each analysis provides detailed information about the content, enabling you to implement appropriate moderation actions in your chat room.

The platform is accessed through a REST API and is integrated with Ably to provide automated content moderation for your chat rooms.

## What's next? <a id="whats-next"/>

* [Add moderation with Hive (model only)](/docs/chat/moderation/direct/hive-model-only)
* [Add moderation with Hive Moderation Dashboard](/docs/chat/moderation/direct/hive-dashboard)
* [Add moderation with Tisane](/docs/chat/moderation/direct/tisane)
* [Add moderation with Bodyguard](/docs/chat/moderation/direct/bodyguard)
* [Build custom moderation with the Moderation API](/docs/chat/moderation/custom)