Important
On June 26 2024, Linux Foundation announced the merger of its financial services umbrella, the Fintech Open Source Foundation (FINOS), with OS-Climate, an open source community dedicated to building data technologies, modeling, and analytic tools that will drive global capital flows into climate change mitigation and resilience; OS-Climate projects are in the process of transitioning to the FINOS governance framework; read more on finos.org/press/finos-join-forces-os-open-source-climate-sustainability-esg
Welcome to OS Climate's Ecosystem platform!
We are part of something called Project-X, which is an OS-Climate project under the "Data Commons" work package. Our objective is to make climate data easier to find, consume share, and trust.
This repo is the starting point for understanding Project-X components, which are included under what we call our Ecosystem Platform. It contains several functional units:
- Climate Data Bazaar: A catalog for climate data which makes climate data easy to find, consume, share, and trust
- Geospatial Index services: Services that provide a uniform grid index for geospatial data (ie. for data that has a location, or latitude and longitude)
- Several sample "data products" that make climate data accessible in the Bazaar
- Several geospatial sample "data products" that illustrate use of location-based data
The Ecosystem Platform was originally developed by Broda Group Software with key contributions by:
If you just want to try out the Ecosystem Platform then the best place to start is in our Getting Started tutorial.
If you are a developer and wish to modify the code and create your own Docker images (which we encourage, after all this is open source software), then follow the instructions in the Building the Platform tutorial.
If you want to see where we are going with the platform, then check out our Roadmap (note this is still under construction and will be available soon).
If you want to participate in setting the direction for the platform, or just want to talk to the developers, then we have the place for you. We get together regularly and all of the meeting details are on the OS-Climate event calendar (look for "Project-X" on Thursdays at 9:30 EST). We would love to talk with you!
If you have any feedback at all, then we have a few venues that let you interact with us:
- OS-Climate Slack
- GitHub Issues for the project or component (we will try to respond quickly to any issues you find)
- LinkedIn: You can find me (Eric Broda, Leader for this project) on LinkedIn at: https://www.linkedin.com/in/ericbroda/ and I would be happy to respond if you DM me.
Otherwise, there is a bunch of stuff below that tells you about our platform, the problems it solves, and the architecture and components that make it all happen.
Enjoy!
This repo is contains or references the consolidated documentation for our Ecosystem Platform components. There are several components, each with their source code, that are available:
- osc-dm-mesh-srv: Execution environment for Ecosystem Platform
- osc-dm-mesh-cli: Command language interface to interact with Ecosystem Platform components using a terminal window command line
- osc-dm-product-srv: Ecosystem Platform Data product agent server
- osc-dm-search-srv: Ecosystem Platform search server
- osc-dm-registrar-srv: Ecosystem Platform registrar
- osc-dm-marketplace-ux: Ecosystem Platform marketplace user interface
- osc-dm-proxy-srv: Ecosystem Platform proxy server
- osc-dm-monitor-srv: Ecosystem Platform monitor services
The Geospatial components are defined in:
- bgs-geo-h3index-srv: Geospatial index server based upon H3 index schemes
Introductions for each component follows below as well as links to further documentation for each component.
osc-dm-mesh-srv is the runtime environment for the Ecosystem Platform, implemented using Docker Compose, starts up several components, including the Registry UX, Registry Service, and various Data Products necessary for data management within the mesh.
A quick note on our Docker Compose execution environment. Every runnable component has a Docker image which means that we can support just about any on-premises or cloud-native environment. But with a goal of geting our platform running in a few minutes on a local machines, we have offered a simple run-time environment based upon docker which is prevelent in most environments (and is easily downloaded) that can run easily and quickly on most laptops.
osc-dm-meshdp-srv is the runtime environment for the Ecosystem Platform data products. It is implemented using Docker Compose (see above for reason for Docker Compose support) and starts up a Data Products.
osc-dm-mesh-cli is the CLI for the Ecosystem Platform Registry provides a suite of capabilities that facilitate the management and interaction with a Ecosystem Platform environment. Here are the key functions and operations it supports:
-
User Management:
- Register different types of users (administrators, publishers, subscribers, guests), each with distinct privileges
- View user details and manage user authentication and sessions
-
Data Product Management:
- Register and configure data products
- View and manage data product configurations
- Generate unique identifiers (UUIDs) for new data products and their artifacts
-
Artifact Management:
- View and retrieve details about artifacts, and manage artifact-specific settings and metadata, directly from data product
-
Carts and Orders:
- Manage shopping carts for subscribers, allowing them to add and remove data products and artifacts.
- Process orders, converting carts into finalized purchases.
-
Search:
- Perform natural language searches within the Ecosystem Platform to discover products and artifacts based on specified criteria
osc-dm-product-srv is a FastAPI server designed for managing data products within a data mesh environment offers a comprehensive suite of functionalities to streamline data operations across an interconnected network.
This server enables users to register new data products and access metadata about data products. Soon it will offer observability, usage, and management capabilities, enabling administrators to modify system information or data products. In some respects, this server acts as a hub for managing the lifecycle of a data product.
osc-dm-search-srv is a FastAPI-based search service for our Ecosystem Platform. This service leverages a Vector database to facilitate natural language similarity searches, allowing users to find content that closely matches their queries in semantic meaning rather than just keyword alignment. It is designed to keep its content store updated with the most current and recent data product information.
osc-dm-registrar-srv is a FastAPI service for a Ecosystem Platform that is designed to handle the registration of, access to, and management of users, products, carts, and orders, consistently managing crucial operational data in one platform.
Utilizing ETCD as its underlying data store, the service benefits from ETCD's strong consistency, high availability, and key-value storage capabilities, ensuring a scalable and reliable infrastructure.
osc-dm-marketplace-ux is a React/MUI-based user interface for the Ecosystem Platform environment presents a "marketplace" style platform that revolutionizes the way users access and interact with products, carts, and orders.
By adopting a familiar shopping metaphor, users can "buy" data products, adding them to carts and processing orders. Note nothing is actually bought, rather we use a "shopping cart" metaphor for data requests - this way the orders can be used to represent past, or presumably frequently used, data which can be easily accessed and consumed.
One quick comment on our UX styling. Yes, it is minimal on purpose. And, as a typical React/MUI application, it is quite easy to customize the look and feel. So, we do expect to customize it a bit (logos, colours, icons etc) but actually want folks that download and use the kit to do much of the look-and-feel customization based upon their needs (but we do hope the share it with us at some point!).
osc-dm-proxy-srv is a FastAPI service that acts as a proxy/router for Ecosystem Platform service requests. It is the primary interface point for external UX functionality and provides dynamic routing to a ecosytem of virtually unlimited number of data products.
osc-dm-monitor-srv is a FastAPI service that monitors components in the Ecosystem Platform. It occasionally pings core services as well as active data products to gather health information and operational metrics. This data will soon be available on both our CLI and UX.
osc-dm-samples-dat is a repository of sample data that populates the registrar and various data products within a Ecosystem Platform environment, integrating with both the marketplace UX and the Ecosystem Platform CLI. These samples are offered to allow users to see data product capabilities and understand data product concepts. And, it provides the necessary data to support a simple and outstanding "Getting Started" experience.
This repository includes a small subset of pre-configured datasets that demonstrate core data product capabilities, including:
- RMI: utilities data
- PUDL: details emissions data for utilities
- And, soon, many others.
NOT YET IMPLEMENTED (link not active)
osc-dm-utilities-lib is a library of common utilities is designed to streamline and enhance development processes by providing a comprehensive toolkit for HTTP requests, data access, and general utility functions.
Tailored for developers working on complex applications, the library includes robust methods for performing secure and efficient HTTP communications, simplified interfaces for database interactions, and a wide range of general-purpose tools for tasks such as data manipulation, format conversions, and error handling.
Use of this source code is governed by an MIT-style license that can be found in the LICENSE file or at https://opensource.org/licenses/MIT.