Skip to content

Updates to collections description & zero persistence update #667

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 23 additions & 5 deletions adaptors/collections.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,29 @@ the collection (generated from the app at /profile/tokens).
Learn more about Collections and common use cases in the
[Collections Docs](/documentation/build/collections).

:::caution

Collections must be created in the
[Platform Admin page](https://docs.openfn.org/documentation/build/collections#managing-collections)
before they can be used.
:::caution Collections is _not_ a long-term data storage solution!

The OpenFn Collections feature is designed to be a _temporary_ data store or
integration “buffer” during data transfer and processing between digital
systems.

- This feature is optimized for throughput and performance (rather then
long-term retention and making the data accessible to end users for
querying/analysis)
- Technically, collections is a key value store, so performance for complex
aggregations or queries is either unsupported or inefficient
- Collections doesn't support advance archiving, backup, and recovery features
that you’d want in a long-term storage solution
- Most buffer implementations have retention mechanisms that automatically
expire/purge data after a certain period of time to clear up space and memory
to ensure resources aren’t exhausted for active processing

Therefore, if you're interested in storing data collected longer term–we
recommend you set up a dedicated database and configure OpenFn to route data to
there as a long-term storage and/or backup option. Ask on
[Community](https://community.openfn.org) or get in touch with
[our support team](mailto://[email protected]) to learn more about related
OpenFn services.

:::

Expand Down
2 changes: 1 addition & 1 deletion docs/deploy/options.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ We're committed to a **no vendor lock-in** experience.

:::

| Pathway | Free | Standard | Dedicated | Do-it-yourself (DIY) |
| Pathway | Free Cloud | OpenFn Cloud | Dedicated | Do-it-yourself (DIY) |
| :-------------------: | :------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------: |
| Description | Go live today on OpenFn.org for small scale projects | Scale up and down, pay only for what you need | A dedicated, unrestricted OpenFn installation anywhere in the world on our servers or yours | Deploy and manage your own solutions with OpenFn |
| License | Free forever with usage limitations | **SaaS** [plans](https://www.openfn.org/pricing); contact [email protected] for custom/invoice agreements | **SDaaS** includes deployment, maintenance, security patches, upgrades, and troubleshooting as a service; contact [email protected] | LGPLv3 means use freely as part of any closed or open-source solution, but make all _derivative_ works open source |
Expand Down
9 changes: 4 additions & 5 deletions docs/get-started/security-compliance.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,21 +24,20 @@ See our main website to learn more about OpenFn

## OpenFn and data storage

In your digital ecosystem, **OpenFn serves as a data processing and transfer
In your digital ecosystem, typically **OpenFn serves as a data processing and transfer
solution—not as a data storage service.**

As an open source Digital Public Good, OpenFn can be deployed anywhere and
Workflows can be configured to adhere to your organization's specific data
workflows can be configured to adhere to your organization's specific data
sharing agreements and security policies.

Consult the `Manage Projects` docs pages for more on project and
[data storage settings](../manage-projects/io-data-storage.md).

See the below diagram for an example architecture where even the cloud-hosted
OpenFn platform can be configured as a **“zero-persistence” data pipeline** to
See the below diagram for an example architecture where even the OpenFn Cloud can be configured as a **“zero-persistence” data pipeline** to
ensure compliance with data security and residency requirements. This enables
partners to rapidly configure and pilot projects using the turnkey OpenFn
cloud-hosted platform, before migrating to a local deployment when they’re ready
to scale.

![Sample Architecture](/img/zero_persistence_architecture.png)
![Sample Architecture](/img/zero-persistence.png)
Binary file added static/img/zero-persistence.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.