Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get snapshots information from json file #341

Merged
merged 3 commits into from
Nov 1, 2023
Merged

Conversation

bianchi2
Copy link
Collaborator

@bianchi2 bianchi2 commented Oct 23, 2023

This PR keeps an existing way to read snapshot IDs from tfvars file but adds an alternative method - parse json file and get EBS and RDS snapshots IDs from it. Depending on the product, version and dataset size snapshots IDs are passed to each product module.

Snapshots json takes precedence over existing snap variables if they are defined.

Checklist

  • I have successful end to end tests run (with & without domain)
  • I have added unit tests (if applicable)
  • I have user documentation (if applicable)

@bianchi2 bianchi2 force-pushed the CLIP-1829-snapshots-json branch from f26effc to c32a204 Compare October 23, 2023 02:26
@bianchi2 bianchi2 added the e2e label Oct 23, 2023
working-directory: test/unittest
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes unit tests. We need to flush /tmp after each test otherwise it occupies >20GB of disc space and GitHub actions fail with no space left on device error

config.tfvars Outdated
@@ -85,6 +85,9 @@ max_cluster_capacity = 5
# List of addtional namespaces to be created in the cluster
#additional_namespaces = ["extra_namespace"]

# Path to a JSON file with EBS and RDS snapshot IDs
#snapshots_json_file_path = "test/dcapt-snapshots.json"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/nit we should be consistent with the space after the # comment symbol. I think leaving a space in is better for readability and removing the space is more convenient as I can just uncomment the line and don't delete the leading space.

I am leaning towards leaving a space

config.tfvars Outdated
@@ -215,6 +218,9 @@ jira_db_iops = 1000
# Set `null` if the snapshot does not have a default db name.
jira_db_name = "jira"

# Dataset size. Used only when snapshots_json_file_path is defined. Defaults to large
# jira_dataset_size = "small"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# jira_dataset_size = "small"
# jira_dataset_size = "large"

When we default to something, we should keep the value here.

config.tfvars Outdated
@@ -310,6 +316,9 @@ confluence_db_iops = 1000
# Set `null` if the snapshot does not have a default db name.
confluence_db_name = "confluence"

# Dataset size. Used only when snapshots_json_file_path is defined. Defaults to large
# confluence_dataset_size = "small"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# confluence_dataset_size = "small"
# confluence_dataset_size = "large"

config.tfvars Outdated
@@ -420,6 +429,9 @@ bitbucket_db_name = "bitbucket"
#bitbucket_elasticsearch_storage = "<REQUESTS_STORAGE>"
#bitbucket_elasticsearch_replicas = "<NUMBER_OF_NODES>"

# Dataset size. Used only when snapshots_json_file_path is defined. Defaults to large
# bitbucket_dataset_size = "small"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# bitbucket_dataset_size = "small"
# bitbucket_dataset_size = "large"

config.tfvars Outdated
@@ -590,6 +602,9 @@ crowd_db_name = "crowd"
# if you encounter such an issue. This will apply to Crowd pods.
#crowd_termination_grace_period = 0

# Dataset size. Used only when snapshots_json_file_path is defined. Defaults to large
# crowd_dataset_size = "small"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# crowd_dataset_size = "small"
# crowd_dataset_size = "large"

value = local.confluence_ebs_snapshot_id
}

output "confluence_db_snapshot_build_number" {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only Confluence and Crowd require build number?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

install.sh Outdated
@@ -147,6 +151,9 @@ pre_flight_checks() {
fi
fi
RDS_SNAPSHOT_ID=$(get_variable ${RDS_SNAPSHOT_VAR} "${CONFIG_ABS_PATH}")
if [ "${SNAPSHOTS_JSON_FILE_PATH}" ]; then
RDS_SNAPSHOT_ID=$(cat ${SNAPSHOTS_JSON_FILE_PATH} | jq ".${PRODUCT}.versions[] | select(.version == \"${PRODUCT_VERSION}\") | .data[] | select(.size == \"large\" and .type == \"rds\") | .snapshots[] | .[\"${REGION}\"]" | sed 's/"//g')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't this always read the large dataset from the JSON file and uses that?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that's the debug version. Let me fix it.

variables.tf Outdated
type = string
default = "large"
validation {
condition = var.jira_dataset_size == null || can(regex("large|small", var.jira_dataset_size))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
condition = var.jira_dataset_size == null || can(regex("large|small", var.jira_dataset_size))
condition = var.jira_dataset_size == null || contains(["large", "small"], var.jira_dataset_size)

I find this easier to read.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the snapshot file that is going to be used with DCAPT or we are distributing the DCAPT snapshot json via DCAPT framework?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is just for tests. DCAPT will distribute a proper one.

@@ -17,4 +17,100 @@ locals {
local_bamboo_chart_path = var.local_helm_charts_path != "" && var.bamboo_install_local_chart ? "${var.local_helm_charts_path}/bamboo" : ""
local_agent_chart_path = var.local_helm_charts_path != "" && var.bamboo_install_local_chart ? "${var.local_helm_charts_path}/bamboo-agent" : ""
local_crowd_chart_path = var.local_helm_charts_path != "" && var.crowd_install_local_chart ? "${var.local_helm_charts_path}/crowd" : ""

# snapshots_json = var.snapshots_json_file_path != "" ? jsondecode(file(var.snapshots_json_file_path)) : jsondecode("{\"jira\": {\"versions\":[]},\"confluence\": {\"versions\":[]},\"crowd\": {\"versions\":[]},\"bitbucket\": {\"versions\":[]}}")
snapshots_json = var.snapshots_json_file_path != "" ? jsondecode(file(var.snapshots_json_file_path)) : null
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we put a sentence or two describing the filtering process and how the snapshot values are determined?

I had to read it a couple of times to see how this is used, and I am sure I will forget it before I will see this code again :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If snapshots_json_file_path is set then we parse the json and get snap IDs from there. If legacy var $product_db_snapshot_id is defined we will use it if either snapshots_json_file_path is undefined or we failed to find snapshot id for the product version in the json. This way we don't change an existing way of supplying snap IDs, just introduce a new one.

@bianchi2 bianchi2 added e2e and removed e2e labels Nov 1, 2023
@bianchi2 bianchi2 merged commit 4293761 into main Nov 1, 2023
2 checks passed
@bianchi2 bianchi2 deleted the CLIP-1829-snapshots-json branch November 1, 2023 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants