Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/styles/Snowplow/Acronyms.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ exceptions:
- ZIP

# Added for Snowplow
- BDP
- CDI
- DNS
- SQS
- UUID
Expand Down
2 changes: 1 addition & 1 deletion .github/styles/config/vocabularies/snowplow/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Data Product Studio
Data Model Packs
Data Model Pack
AWS Console
BDP Console
Snowplow Console
Iglu Central
Iglu Server
Igluctl
Expand Down
7 changes: 1 addition & 6 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@
* Mix prose and lists appropriately: use prose to explain concepts, lists for configuration options or step-by-step items

### Snowplow terminology
* **Capitalized**: Data Product Studio, Snowtype, Snowplow BDP, Signals
* **Capitalized**: Data Product Studio, Snowtype, Snowplow CDI, Signals
* **Context-capitalized**: Collector, Enrich, specific Loaders
* **Not capitalized**: entities, events, schemas, data structures
* Use "entity" not "context", "self-describing event" not "unstructured event"
Expand Down Expand Up @@ -120,11 +120,6 @@ Important cautions about data loss, security, or breaking changes
* **Tip**: performance improvements, recommended workflows, pro tips
* **Warning**: data loss risks, security considerations

### BDP vs self-hosted
* Mark BDP-only features in frontmatter: `sidebar_custom_props: offerings: - bdp`
* Note when BDP provides built-in functionality, mention self-hosted alternatives
* **Do not mention "Community Edition"** - use "self-hosted" instead but otherwise prefer not to mention

### Images
* Formats: `.webp` (preferred), `.png`, `.jpg`
* Descriptive filenames and alt text
Expand Down
21 changes: 0 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ This is the source for https://docs.snowplow.io/docs.
- [Organizing content](#organizing-content)
- [Sidebar](#sidebar)
- [Updating sidebar attributes for multiple sections at once](#updating-sidebar-attributes-for-multiple-sections-at-once)
- [Offerings](#offerings)
- [Links](#links)
- [Concepts](#concepts)
- [Reusable fragments](#reusable-fragments)
Expand Down Expand Up @@ -143,26 +142,6 @@ It'll update the `index.md` files as appropriate.

You can now delete the `update_attributes_here.txt` file.

### Offerings

Some documentation is only relevant to a particular offering. You can indicate it like this:
```
---
title: ...
...
sidebar_custom_props:
offerings:
- bdp
...
---
```

This will result in an icon appearing in the sidebar, as well as an automatic banner on the page, specifying that the docs only apply to a given offering.

The available values are: `bdp` and `community`. Do not specify both values at once — if a piece of documentation is relevant to all offerings, there should be no `offerings` property as that’s the default.

Whenever the same functionality can be achieved in multiple offerings but in a different way (e.g. managing schemas), create a parent folder (“Managing schemas”) that’s offering-neutral, and then add offering-specific pages inside it. This way, other pages can link to the generic page without having to specify different methods for different offerings.

### Links

For links within this documentation, please end the link with `/index.md`. This way all links will be checked, and you’ll get an error if a link is broken at any point.
Expand Down
12 changes: 5 additions & 7 deletions docs/account-management/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,21 @@ date: "2020-02-15"
sidebar_position: 9
sidebar_custom_props:
header: " "
offerings:
- bdp
sidebar_label: "Account management"
---

Manage your account configuration and users using the Snowplow BDP Console. You can also use the underlying API directly. This page describes how to acquire an API key.
Manage your account configuration and users using the Snowplow Console. You can also use the underlying API directly. This page describes how to acquire an API key.

## Credentials API

The API that drives BDP Console's functionality is [publicly documented](https://console.snowplowanalytics.com/api/msc/v1/docs/index.html?url=/api/msc/v1/docs/docs.yaml) and available for our customers to invoke via code. All calls to it need to be properly authenticated using JSON Web Tokens (JWT) that can be acquired via the Credentials API.
The API that drives Console's functionality is [publicly documented](https://console.snowplowanalytics.com/api/msc/v1/docs/index.html?url=/api/msc/v1/docs/docs.yaml) and available for our customers to invoke via code. All calls to it need to be properly authenticated using JSON Web Tokens (JWT) that can be acquired via the Credentials API.

The process for creating a key has been improved over time. We recommend using the v3 process.


### Version 3

The following view is available to all customers under [BDP Console settings](https://console.snowplowanalytics.com/credentials):
The following view is available to all customers under [Console settings](https://console.snowplowanalytics.com/credentials):

![](images/accessing-generated-api-keys.png)

Expand Down Expand Up @@ -95,10 +93,10 @@ Authenticating with v2 only required the API key secret. While this method and t

### Version 1

Previously, BDP Console was using the Password authentication flow to support machine-to-machine (m2m) applications. Under that scenario a BDP customer had to create a bot user in their account, retrieve a client ID and a client secret, and use all three to acquire a JWT. Customers who have enabled these credentials in the past will see the following panel in their Console account settings:
Previously, Console was using the Password authentication flow to support machine-to-machine (m2m) applications. Under that scenario a customer had to create a bot user in their account, retrieve a client ID and a client secret, and use all three to acquire a JWT. Customers who have enabled these credentials in the past will see the following panel in their Console account settings:

![](images/image-2.png)

Legacy Snowplow BDP credentials management
Legacy Snowplow credentials management

This method and the respective credentials still work for those who have been using them, however we strongly advise that customers upgrade to the current iteration where the only secret to be used by m2m applications is an API key which can be exchanged for a JWT.
2 changes: 1 addition & 1 deletion docs/account-management/managing-permissions/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ To set a users permissions, navigate to `Manage users` and then to the user whos

## What permissions can be set?

Snowplow BDP Console sets permissions for each area of Console as summarized below:
Snowplow Console sets permissions for each area of Console as summarized below:

| **Console feature** | **Description** | **Possible permissions** |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ To accept the terms of license and run Enrich, set the `ACCEPT_LIMITED_USE_LICEN
| `monitoring.metrics.statsd.prefix` | Optional. Default: `snowplow.enrich`. Pefix of StatsD metric names. |
| `monitoring.healthProbe.port` (since *6.0.0*) | Optional. Default: `8000`. Open a HTTP server that returns OK only if the app is healthy. |
| `monitoring.healthProbe.unhealthyLatency` (since *6.0.0*) | Optional. Default: `2 minutes`. Health probe becomes unhealthy if any received event is still not fully processed before this cutoff time. |
| `telemetry.disable` | Optional. Set to `true` to disable [telemetry](/docs/get-started/snowplow-community-edition/telemetry/index.md). |
| `telemetry.userProvidedId` | Optional. See [here](/docs/get-started/snowplow-community-edition/telemetry/index.md#how-can-i-help) for more information. |
| `telemetry.disable` | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/index.md). |
| `telemetry.userProvidedId` | Optional. See [here](/docs/get-started/self-hosted/telemetry/index.md#how-can-i-help) for more information. |
| `validation.acceptInvalid` (since *6.0.0*) | Optional. Default: `false`. Enrich *3.0.0* introduces the validation of the enriched events against atomic schema before emitting. If set to `false`, a failed event will be emitted instead of the enriched event if validation fails. If set to `true`, invalid enriched events will be emitted, as before. |
| `validation.atomicFieldsLimits` (since *4.0.0*) | Optional. For the defaults, see [here](https://github.com/snowplow/enrich/blob/master/modules/common/src/main/resources/reference.conf). Configuration for custom maximum atomic fields (strings) length. It's a map-like structure with keys being atomic field names and values being their max allowed length. |
| `validation.maxJsonDepth` (since *6.0.0*) | Optional. Default: `40`. Maximum allowed depth for the JSON entities in the events. Event will be sent to bad row stream if it contains JSON entity with a depth that exceeds this value. |
Expand Down
2 changes: 1 addition & 1 deletion docs/api-reference/failed-events/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ In order for an event to be processed successfully:

If your pipeline is generating schema violations, it might mean there is a problem with your tracking, or a problem with your [Iglu resolver](/docs/api-reference/iglu/iglu-resolver/index.md) which lists where schemas should be found. The error details in the schema violation JSON object should give you a hint about what the problem might be.

Snowplow BDP customers should check in the Snowplow BDP Console that all data structures are correct and have been [promoted to production](/docs/data-product-studio/data-structures/manage/index.md). Snowplow Community Edition users should check that the Enrichment app is configured with an [Iglu resolver file](/docs/api-reference/iglu/iglu-resolver/index.md) that points to a repository containing the schemas.
Snowplow customers should check in the Snowplow Console that all data structures are correct and have been [promoted to production](/docs/data-product-studio/data-structures/manage/index.md). Snowplow Self-Hosted users should check that the Enrichment app is configured with an [Iglu resolver file](/docs/api-reference/iglu/iglu-resolver/index.md) that points to a repository containing the schemas.

Next, check the tracking code in your custom application, and make sure the entities you are sending conform to the schema definition.

Expand Down
2 changes: 1 addition & 1 deletion docs/api-reference/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ sidebar_label: "Reference"

This section contains detailed technical information about Snowplow components.

Some of the information is relevant only for [Community Edition](/docs/get-started/snowplow-community-edition/index.md) users, as [Snowplow BDP](/docs/get-started/snowplow-bdp/index.md) customers won't need to configure all their own components.
Some of the information is relevant only for [Snowplow Self-Hosted](/docs/get-started/index.md#self-hosted) users, as [Snowplow CDI](/docs/get-started/index.md#customer-data-infrastructure) customers won't need to configure all their own components.
Original file line number Diff line number Diff line change
Expand Up @@ -94,11 +94,11 @@ import Link from '@docusaurus/Link';
</tr>
<tr>
<td><code>telemetry.disable</code></td>
<td>Optional. Set to <code>true</code> to disable <Link to="/docs/get-started/snowplow-community-edition/telemetry/">telemetry</Link>.</td>
<td>Optional. Set to <code>true</code> to disable <Link to="/docs/get-started/self-hosted/telemetry/">telemetry</Link>.</td>
</tr>
<tr>
<td><code>telemetry.userProvidedId</code></td>
<td>Optional. See <Link to="/docs/get-started/snowplow-community-edition/telemetry/#how-can-i-help">here</Link> for more information.</td>
<td>Optional. See <Link to="/docs/get-started/self-hosted/telemetry/#how-can-i-help">here</Link> for more information.</td>
</tr>
<tr>
<td><code>http.client.maxConnectionsPerServer</code></td>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -112,11 +112,11 @@ import Link from '@docusaurus/Link';
</tr>
<tr>
<td><code>telemetry.disable</code></td>
<td>Optional. Set to <code>true</code> to disable <Link to="/docs/get-started/snowplow-community-edition/telemetry/">telemetry</Link>.</td>
<td>Optional. Set to <code>true</code> to disable <Link to="/docs/get-started/self-hosted/telemetry/">telemetry</Link>.</td>
</tr>
<tr>
<td><code>telemetry.userProvidedId</code></td>
<td>Optional. See <Link to="/docs/get-started/snowplow-community-edition/telemetry/#how-can-i-help">here</Link> for more information.</td>
<td>Optional. See <Link to="/docs/get-started/self-hosted/telemetry/#how-can-i-help">here</Link> for more information.</td>
</tr>
<tr>
<td><code>inMemBatchBytes</code></td>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -92,11 +92,11 @@ import Link from '@docusaurus/Link';
</tr>
<tr>
<td><code>telemetry.disable</code></td>
<td>Optional. Set to <code>true</code> to disable <Link to="/docs/get-started/snowplow-community-edition/telemetry/">telemetry</Link>.</td>
<td>Optional. Set to <code>true</code> to disable <Link to="/docs/get-started/self-hosted/telemetry/">telemetry</Link>.</td>
</tr>
<tr>
<td><code>telemetry.userProvidedId</code></td>
<td>Optional. See <Link to="/docs/get-started/snowplow-community-edition/telemetry/#how-can-i-help">here</Link> for more information.</td>
<td>Optional. See <Link to="/docs/get-started/self-hosted/telemetry/#how-can-i-help">here</Link> for more information.</td>
</tr>
<tr>
<td><code>output.good.jdbcLoginTimeout</code></td>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -170,8 +170,8 @@ Only Snowflake Loader can be run on Azure at the moment.
| `initRetries.strategy` | Backoff strategy used during retry. The possible values are `JITTER`, `CONSTANT`, `EXPONENTIAL`, `FIBONACCI`. |
| `initRetries.attempts` | Optional. How many attempts to make before sending the message into retry queue. If missing, `cumulativeBound` will be used. |
| `initRetries.cumulativeBound` | Optional. When backoff reaches this delay, eg '1 hour', the loader will stop retrying. If both this and `attempts` are not set, the loader will retry indefinitely. |
| `telemetry.disable` | Optional. Set to `true` to disable [telemetry](/docs/get-started/snowplow-community-edition/telemetry/index.md). |
| `telemetry.userProvidedId` | Optional. See [here](/docs/get-started/snowplow-community-edition/telemetry/index.md#how-can-i-help) for more information. |
| `telemetry.disable` | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/index.md). |
| `telemetry.userProvidedId` | Optional. See [here](/docs/get-started/self-hosted/telemetry/index.md#how-can-i-help) for more information. |

## Common monitoring settings

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ It is possible to run Snowflake Loader on AWS, GCP and Azure.

### Setting up Snowflake

You can use the steps outlined in our [quick start guide](/docs/get-started/snowplow-community-edition/quick-start/index.md?warehouse=snowflake#prepare-the-destination) to create most of the necessary Snowflake resources.
You can use the steps outlined in our [quick start guide](/docs/get-started/self-hosted/quick-start/index.md?warehouse=snowflake#prepare-the-destination) to create most of the necessary Snowflake resources.

There are two different authentication methods with Snowflake Loader:
* With the `TempCreds` method, there are no additional Snowflake resources needed.
Expand Down Expand Up @@ -43,7 +43,7 @@ Finally, use the `transformedStage` [configuration setting](/docs/api-reference/

### Running the loader

There are dedicated terraform modules for deploying Snowflake Loader on [AWS](https://registry.terraform.io/modules/snowplow-devops/snowflake-loader-ec2/aws/latest) and [Azure](https://github.com/snowplow-devops/terraform-azurerm-snowflake-loader-vmss). You can see how they are used in our full pipeline deployment examples [here](/docs/get-started/snowplow-community-edition/quick-start/index.md).
There are dedicated terraform modules for deploying Snowflake Loader on [AWS](https://registry.terraform.io/modules/snowplow-devops/snowflake-loader-ec2/aws/latest) and [Azure](https://github.com/snowplow-devops/terraform-azurerm-snowflake-loader-vmss). You can see how they are used in our full pipeline deployment examples [here](/docs/get-started/self-hosted/quick-start/index.md).

We don't have a terraform module for deploying Snowflake Loader on GCP yet. Therefore, it needs to be deployed manually at the moment.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,11 @@ import Link from '@docusaurus/Link';
</tr>
<tr>
<td><code>telemetry.disable</code></td>
<td>Optional. Set to <code>true</code> to disable <Link to="/docs/get-started/snowplow-community-edition/telemetry/">telemetry</Link>.</td>
<td>Optional. Set to <code>true</code> to disable <Link to="/docs/get-started/self-hosted/telemetry/">telemetry</Link>.</td>
</tr>
<tr>
<td><code>telemetry.userProvidedId</code></td>
<td>Optional. See <Link to="/docs/get-started/snowplow-community-edition/telemetry/#how-can-i-help">here</Link> for more information.</td>
<td>Optional. See <Link to="/docs/get-started/self-hosted/telemetry/#how-can-i-help">here</Link> for more information.</td>
</tr>
<tr>
<td><code>monitoring.sentry.dsn</code></td>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ sidebar_position: 500

# Telemetry Configuration

You can read about our telemetry principles [here](/docs/get-started/snowplow-community-edition/telemetry/index.md).
You can read about our telemetry principles [here](/docs/get-started/self-hosted/telemetry/index.md).

## Configuration options

Expand All @@ -16,7 +16,7 @@ Enabling telemetry:
# Optional. Set to true to disable telemetry.
disable_telemetry = false

# Optional. See here for more information: https://docs.snowplow.io/docs/get-started/snowplow-community-edition/telemetry/#how-can-i-help
# Optional. See here for more information: https://docs.snowplow.io/docs/get-started/self-hosted/telemetry/#how-can-i-help
user_provided_id = "[email protected]"
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ where `service_name` can be one of the following: `collector`, `enrich`, `esLoad

#### Configuring telemetry

See our [telemetry principles](/docs/get-started/snowplow-community-edition/telemetry/index.md) for more information on telemetry.
See our [telemetry principles](/docs/get-started/self-hosted/telemetry/index.md) for more information on telemetry.

HTTP GET to get current configuration

Expand Down
6 changes: 3 additions & 3 deletions docs/api-reference/snowplow-mini/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ sidebar_position: 120

Snowplow Mini is similar to [Snowplow Micro](/docs/data-product-studio/data-quality/snowplow-micro/index.md), with the following differences:
* Micro is more portable and can easily run on your machine or in automated tests.
* Mini has more features, mainly an OpenSearch Dashboards UI, and is better integrated with Snowplow BDP.
* Mini has more features, mainly an OpenSearch Dashboards UI, and is better integrated with Snowplow.

:::

Expand All @@ -21,9 +21,9 @@ You might use Snowplow Mini when:

## Getting started

Snowplow BDP users can request a Snowplow Mini instance through the console (go to `“Environments” → “Sandboxes” → “Setup a sandbox”`).
Snowplow users can request a Snowplow Mini instance through Console (go to `“Environments” → “Sandboxes” → “Setup a sandbox”`).

For Community Edition, see the setup guides for [AWS](/docs/api-reference/snowplow-mini/setup-guide-for-aws/index.md) and [GCP](/docs/api-reference/snowplow-mini/setup-guide-for-gcp/index.md).
For Snowplow Self-Hosted, see the setup guides for [AWS](/docs/api-reference/snowplow-mini/setup-guide-for-aws/index.md) and [GCP](/docs/api-reference/snowplow-mini/setup-guide-for-gcp/index.md).

## Conceptual diagram

Expand Down
Loading