[FSTORE-1789] Data source docs

docs/concepts/fs/feature_group/external_fg.md

@@ -1,6 +1,6 @@
-External feature groups are offline feature groups where their data is stored in an external table. An external table requires a storage connector, defined with the Connector API (or more typically in the user interface), to enable HSFS to retrieve data from the external table. An external feature group doesn't allow for offline data ingestion or modification; instead, it includes a user-defined SQL string for retrieving data. You can also perform SQL operations, including projections, aggregations, and so on. The SQL query is executed on-demand when HSFS retrieves data from the external Feature Group, for example, when creating training data using features in the external table.
+External feature groups are offline feature groups where their data is stored in an external table. An external table requires a data source, defined with the Connector API (or more typically in the user interface), to enable HSFS to retrieve data from the external table. An external feature group doesn't allow for offline data ingestion or modification; instead, it includes a user-defined SQL string for retrieving data. You can also perform SQL operations, including projections, aggregations, and so on. The SQL query is executed on-demand when HSFS retrieves data from the external Feature Group, for example, when creating training data using features in the external table.
 
-In the image below, we can see that HSFS currently supports a large number of data sources, including any JDBC-enabled source, Snowflake, Data Lake, Redshift, BigQuery, S3, ADLS, GCS, and Kafka
+In the image below, we can see that HSFS currently supports a large number of data sources, including any JDBC-enabled source, Snowflake, Data Lake, Redshift, BigQuery, S3, ADLS, GCS, RDS, and Kafka
 
 <img src="../../../../assets/images/concepts/fs/fg-connector-api.svg">
 

docs/concepts/fs/feature_group/fg_overview.md

@@ -19,4 +19,4 @@ The online store stores only the latest values of features for a feature group.
 
 The offline store stores the historical values of features for a feature group so that it may store much more data than the online store. Offline feature groups are used, typically, to create training data for models, but also to retrieve data for batch scoring of models.
 
-In most cases, offline data is stored in Hopsworks, but through the implementation of storage connectors, it can reside in an external file system. The externally stored data can be managed by Hopsworks by defining ordinary feature groups or it can be used for reading only by defining [External Feature Group](external_fg.md).
+In most cases, offline data is stored in Hopsworks, but through the implementation of data sources, it can reside in an external file system. The externally stored data can be managed by Hopsworks by defining ordinary feature groups or it can be used for reading only by defining [External Feature Group](external_fg.md).

docs/index.md

@@ -40,42 +40,49 @@ hide:
               </div>
             </div>
             <div class="side-content">
-              <div class="name_item ingrey">Storage connectors</div>
+              <div class="name_item ingrey">Data Sources</div>
               <div class="w-layout-grid">
                 <div class="db_frame">
                   <div class="icondb">
                     <div class="db_frame-top"></div>
                     <div class="db_frame-mid"></div>
                   </div>
-                  <div class="name_item db"><a href="./user_guides/fs/storage_connector/creation/jdbc/">JDBC</a></div>
+                  <div class="name_item db"><a href="./user_guides/fs/data_source/creation/jdbc/">JDBC</a></div>
                 </div>
                 <div class="db_frame">
                   <div class="icondb">
                     <div class="db_frame-top"></div>
                     <div class="db_frame-mid"></div>
                   </div>
-                  <div class="name_item db"><a href="./user_guides/fs/storage_connector/creation/bigquery/">BigQuery</a></div>
+                  <div class="name_item db"><a href="./user_guides/fs/data_source/creation/bigquery/">BigQuery</a></div>
                 </div>
                 <div class="db_frame">
                   <div class="icondb">
                     <div class="db_frame-top"></div>
                     <div class="db_frame-mid"></div>
                   </div>
-                  <div class="name_item db"><a href="./user_guides/fs/storage_connector/creation/s3/">Object Store</a></div>
+                  <div class="name_item db"><a href="./user_guides/fs/data_source/creation/s3/">Object Store</a></div>
                 </div>
                 <div class="db_frame">
                   <div class="icondb">
                     <div class="db_frame-top"></div>
                     <div class="db_frame-mid"></div>
                   </div>
-                  <div class="name_item db"><a href="./user_guides/fs/storage_connector/creation/snowflake/">Snowflake</a></div>
+                  <div class="name_item db"><a href="./user_guides/fs/data_source/creation/snowflake/">Snowflake</a></div>
                 </div>
                 <div class="db_frame">
                   <div class="icondb">
                     <div class="db_frame-top"></div>
                     <div class="db_frame-mid"></div>
                   </div>
-                  <div class="name_item db"><a href="./user_guides/fs/storage_connector/creation/redshift/">RedShift</a></div>
+                  <div class="name_item db"><a href="./user_guides/fs/data_source/creation/redshift/">RedShift</a></div>
+                </div>
+                <div class="db_frame">
+                  <div class="icondb">
+                    <div class="db_frame-top"></div>
+                    <div class="db_frame-mid"></div>
+                  </div>
+                  <div class="name_item db"><a href="./user_guides/fs/data_source/creation/rds/">RDS</a></div>
                 </div>
               </div>
             </div>

docs/setup_installation/admin/roleChaining.md

@@ -109,6 +109,6 @@ Add mappings by clicking on *New role chaining*. Enter the project name. Select
   <figcaption>Create Role Chaining</figcaption>
 </figure>
 
-Project member can now create connectors using *temporary credentials* to assume the role you configured. More detail about using temporary credentials can be found [here](../../user_guides/fs/storage_connector/creation/s3.md#temporary-credentials).
+Project member can now create connectors using *temporary credentials* to assume the role you configured. More detail about using temporary credentials can be found [here](../../user_guides/fs/data_source/creation/s3.md#temporary-credentials).
 
 Project member can see the list of role they can assume by going the _Project Settings_ -> [Assuming IAM Roles](../../../user_guides/projects/iam_role/iam_role_chaining) page.

docs/setup_installation/on_prem/external_kafka_cluster.md

@@ -60,8 +60,8 @@ As mentioned above, when configuring Hopsworks to use an external Kafka cluster,
   </figure>
 </p>
 
-#### Storage connector configuration 
+#### Data Source configuration 
 
-Users should create a [Kafka storage connector](../../user_guides/fs/storage_connector/creation/kafka.md) named `kafka_connector` which is going to be used by the feature store clients to configure the necessary Kafka producers to send data.
+Users should create a [Kafka Data Source](../../user_guides/fs/data_source/creation/kafka.md) named `kafka_connector` which is going to be used by the feature store clients to configure the necessary Kafka producers to send data.
 The configuration is done for each project to ensure its members have the necessary authentication/authorization.
-If the storage connector is not found in the project, default values referring to Hopsworks managed Kafka will be used.
+If the data source is not found in the project, default values referring to Hopsworks managed Kafka will be used.

docs/user_guides/fs/storage_connector/creation/adls.md → docs/user_guides/fs/data_source/creation/adls.md

@@ -1,14 +1,14 @@
-# How-To set up a ADLS Storage Connector
+# How-To set up a ADLS Data Source
 
 ## Introduction
 
-Azure Data Lake Storage (ADLS) Gen2 is a HDFS-compatible filesystem on Azure for data analytics. The ADLS Gen2 filesystem stores its data in Azure Blob storage, ensuring low-cost storage, high availability, and disaster recovery. In Hopsworks, you can access ADLS Gen2 by defining a Storage Connector and creating and granting permissions to a service principal.
+Azure Data Lake Storage (ADLS) Gen2 is a HDFS-compatible filesystem on Azure for data analytics. The ADLS Gen2 filesystem stores its data in Azure Blob storage, ensuring low-cost storage, high availability, and disaster recovery. In Hopsworks, you can access ADLS Gen2 by defining a Data Source and creating and granting permissions to a service principal.
 
-In this guide, you will configure a Storage Connector in Hopsworks to save all the authentication information needed in order to set up a connection to your Azure ADLS filesystem.
+In this guide, you will configure a Data Source in Hopsworks to save all the authentication information needed in order to set up a connection to your Azure ADLS filesystem.
 When you're finished, you'll be able to read files using Spark through HSFS APIs. You can also use the connector to write out training data from the Feature Store, in order to make it accessible by third parties.
 
 !!! note
-    Currently, it is only possible to create storage connectors in the Hopsworks UI. You cannot create a storage connector programmatically.
+    Currently, it is only possible to create data sources in the Hopsworks UI. You cannot create a data source programmatically.
 
 ## Prerequisites
 
@@ -19,27 +19,35 @@ Before you begin this guide you'll need to retrieve the following information fr
 - **Service Principal Registration:** Register the service principal, granting it a role assignment such as Storage Blob Data Contributor, on the Azure Data Lake Storage Gen2 account.
 
 !!! info
-    When you specify the 'container name' in the ADLS storage connector, you need to have previously created that container - the Hopsworks Feature Store will not create that storage container for you.
+    When you specify the 'container name' in the ADLS data source, you need to have previously created that container - the Hopsworks Feature Store will not create that storage container for you.
 
 ## Creation in the UI
-### Step 1: Set up new storage connector
+### Step 1: Set up new Data Source
 
-Head to the Storage Connector View on Hopsworks (1) and set up a new storage connector (2).
+Head to the Data Source View on Hopsworks (1) and set up a new data source (2).
 
 <figure markdown>
-  ![Storage Connector Creation](../../../../assets/images/guides/fs/storage_connector/storage_connector_create.png)
-  <figcaption>The Storage Connector View in the User Interface</figcaption>
+  ![Data Source Creation](../../../../assets/images/guides/fs/data_source/data_source_overview.png)
+  <figcaption>The Data Source View in the User Interface</figcaption>
 </figure>
 
 ### Step 2: Enter ADLS Information
 
 Enter the details for your ADLS connector. Start by giving it a **name** and an optional **description**.
 
 <figure markdown>
-  ![ADLS Connector Creation](../../../../assets/images/guides/fs/storage_connector/adls_creation.png)
+  ![ADLS Connector Creation](../../../../assets/images/guides/fs/data_source/adls_creation.png)
   <figcaption>ADLS Connector Creation Form</figcaption>
 </figure>
 
+1. Select "Azure Data Lake" as the storage.
+2. Set directory ID.
+3. Enter the Application ID.
+4. Paste the Service Credentials.
+5. Specify account name.
+6. Provide the container name.
+7. Click on "Save Credentials".
+
 ### Step 3: Azure Create an ADLS Resource
 
 When programmatically signing in, you need to pass the tenant ID with your authentication request and the application ID. You also need a certificate or an authentication key (described in the following section). To get those values, use the following steps:
@@ -48,20 +56,20 @@ When programmatically signing in, you need to pass the tenant ID with your authe
 2. From App registrations in Azure AD, select your application.
 3. Copy the Directory (tenant) ID and store it in your application code.
     <figure markdown>
-    ![ADLS select tenant-id](../../../../assets/images/guides/fs/storage_connector/adls-copy-tenant-id.png)
-    <figcaption>You need to copy the Directory (tenant) id and paste it to the Hopsworks ADLS storage connector  "Directory id" text field.</figcaption>
+    ![ADLS select tenant-id](../../../../assets/images/guides/fs/data_source/adls-copy-tenant-id.png)
+    <figcaption>You need to copy the Directory (tenant) id and paste it to the Hopsworks ADLS Data Source "Directory id" text field.</figcaption>
     </figure>
 
 4. Copy the Application ID and store it in your application code.
     <figure markdown>
-    ![ADLS select app-id](../../../../assets/images/guides/fs/storage_connector/adls-copy-app-id.png)
-    <figcaption>>You need to copy the Application id and paste it to the Hopsworks ADLS storage connector "Application id" text field.</figcaption>
+    ![ADLS select app-id](../../../../assets/images/guides/fs/data_source/adls-copy-app-id.png)
+    <figcaption>>You need to copy the Application id and paste it to the Hopsworks ADLS Data Source "Application id" text field.</figcaption>
     </figure>
 
 5. Create an Application Secret and copy it into the Service Credential field.
     <figure markdown>
-    ![ADLS enter application secret](../../../../assets/images/guides/fs/storage_connector/adls-copy-secret.png)
-    <figcaption>You need to copy the Application Secret and paste it to the Hopsworks ADLS storage connector "Service Credential" text field.</figcaption>
+    ![ADLS enter application secret](../../../../assets/images/guides/fs/data_source/adls-copy-secret.png)
+    <figcaption>You need to copy the Application Secret and paste it to the Hopsworks ADLS Data Source "Service Credential" text field.</figcaption>
     </figure>
 
 #### Common Problems
@@ -76,4 +84,4 @@ If you get an error "StatusCode=404 StatusDescription=The specified filesystem d
 
 ## Next Steps
 
-Move on to the [usage guide for storage connectors](../usage.md) to see how you can use your newly created ADLS connector.
+Move on to the [usage guide for data sources](../usage.md) to see how you can use your newly created ADLS connector.

docs/user_guides/fs/data_source/creation/bigquery.md

@@ -0,0 +1,80 @@
+# How-To set up a BigQuery Data Source
+
+## Introduction
+
+A BigQuery data source provides integration to Google Cloud BigQuery.
+BigQuery is Google Cloud's managed data warehouse supporting that lets you run analytics and 
+execute SQL queries over large scale data. Such data warehouses are often the source of raw data for feature 
+engineering pipelines.
+
+In this guide, you will configure a Data Source in Hopsworks to connect to your BigQuery project by saving the
+necessary information.
+When you're finished, you'll be able to execute queries and read results of BigQuery using Spark through 
+HSFS APIs.
+
+The data source uses the Google `spark-bigquery-connector` behind the scenes.
+To read more about the spark connector, like the spark options or usage, check [Apache Spark SQL connector for Google BigQuery.](https://github.com/GoogleCloudDataproc/spark-bigquery-connector#usage
+'github.com/GoogleCloudDataproc/spark-bigquery-connector')
+
+!!! note
+        Currently, it is only possible to create data sources in the Hopsworks UI. You cannot create a data source programmatically.
+
+## Prerequisites
+
+Before you begin this guide you'll need to retrieve the following information about your GCP account:
+
+- **BigQuery Project:** You need a BigQuery project, dataset and table created and have read access to it. Or, if 
+  you wish to query a public dataset you need its corresponding details.
+- **Authentication Method:** Authentication to GCP account is handled by uploading the `JSON keyfile for service
+  account` to the Hopsworks Project. You will need to create this JSON keyfile from GCP. For more information on
+  service accounts
+  and creating keyfile in GCP, read [Google Cloud documentation.](https://cloud.google.com/docs/authentication/production#create_service_account
+  'creating service account keyfile')
+
+!!! note
+        To read data, the BigQuery service account user needs permission to `create read sesssion` which is available in **BigQuery Admin role**.
+
+## Creation in the UI
+### Step 1: Set up new Data Source
+
+Head to the Data Source View on Hopsworks (1) and set up a new data source (2).
+
+<figure markdown>
+  ![Data Source Creation](../../../../assets/images/guides/fs/data_source/data_source_overview.png)
+  <figcaption>The Data Source View in the User Interface</figcaption>
+</figure>
+
+
+### Step 2: Enter source details
+Enter the details for your BigQuery storage. Start by giving
+it a unique **name** and an optional
+**description**.
+
+<figure markdown>
+  ![BigQuery Creation](../../../../assets/images/guides/fs/data_source/bigquery_creation.png)
+  <figcaption>BigQuery Creation Form</figcaption>
+</figure>
+
+1. Select "Google BigQuery" as the storage.
+2. Next, set the name of the parent BigQuery project. This is used for billing by GCP.
+3. Authentication: Here you should upload your `JSON keyfile for service 
+   account` used for authentication. You can choose to either
+   upload from your local using `Upload new file` or choose an existing file within project using `From Project`.
+4. Read Options:
+  In the UI set the below fields, 
+    1. *BigQuery Project*: The BigQuery project to read
+    2. *BigQuery Dataset*: The dataset of the table (Optional)
+    3. *BigQuery Table*: The table to read (Optional)
+
+
+!!! note
+    *Materialization Dataset*: Temporary dataset used by BigQuery for writing. It must be set to a dataset where the GCP user has table creation permission. The queried table must be in the same location as the `materializationDataset` (e.g 'EU' or 'US'). Also, if a table in the `SQL statement` is from project other than the `parentProject` then use the fully qualified table name i.e. `[project].[dataset].[table]` 
+         (Read more details from Google documentation on usage of query for BigQuery spark connector [here](https://github.com/GoogleCloudDataproc/spark-bigquery-connector#reading-data-from-a-bigquery-query)).
+
+5. Spark Options: Optionally, you can set additional spark options using the `Key - Value` pairs.
+6. Click on "Save Credentials". 
+
+## Next Steps
+
+Move on to the [usage guide for data sources](../usage.md) to see how you can use your newly created BigQuery 
+connector.