-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Get snapshots information from json file #341
Conversation
f26effc
to
c32a204
Compare
working-directory: test/unittest |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This fixes unit tests. We need to flush /tmp after each test otherwise it occupies >20GB of disc space and GitHub actions fail with no space left on device error
config.tfvars
Outdated
@@ -85,6 +85,9 @@ max_cluster_capacity = 5 | |||
# List of addtional namespaces to be created in the cluster | |||
#additional_namespaces = ["extra_namespace"] | |||
|
|||
# Path to a JSON file with EBS and RDS snapshot IDs | |||
#snapshots_json_file_path = "test/dcapt-snapshots.json" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/nit we should be consistent with the space after the #
comment symbol. I think leaving a space in is better for readability and removing the space is more convenient as I can just uncomment the line and don't delete the leading space.
I am leaning towards leaving a space
config.tfvars
Outdated
@@ -215,6 +218,9 @@ jira_db_iops = 1000 | |||
# Set `null` if the snapshot does not have a default db name. | |||
jira_db_name = "jira" | |||
|
|||
# Dataset size. Used only when snapshots_json_file_path is defined. Defaults to large | |||
# jira_dataset_size = "small" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# jira_dataset_size = "small" | |
# jira_dataset_size = "large" |
When we default to something, we should keep the value here.
config.tfvars
Outdated
@@ -310,6 +316,9 @@ confluence_db_iops = 1000 | |||
# Set `null` if the snapshot does not have a default db name. | |||
confluence_db_name = "confluence" | |||
|
|||
# Dataset size. Used only when snapshots_json_file_path is defined. Defaults to large | |||
# confluence_dataset_size = "small" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# confluence_dataset_size = "small" | |
# confluence_dataset_size = "large" |
config.tfvars
Outdated
@@ -420,6 +429,9 @@ bitbucket_db_name = "bitbucket" | |||
#bitbucket_elasticsearch_storage = "<REQUESTS_STORAGE>" | |||
#bitbucket_elasticsearch_replicas = "<NUMBER_OF_NODES>" | |||
|
|||
# Dataset size. Used only when snapshots_json_file_path is defined. Defaults to large | |||
# bitbucket_dataset_size = "small" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# bitbucket_dataset_size = "small" | |
# bitbucket_dataset_size = "large" |
config.tfvars
Outdated
@@ -590,6 +602,9 @@ crowd_db_name = "crowd" | |||
# if you encounter such an issue. This will apply to Crowd pods. | |||
#crowd_termination_grace_period = 0 | |||
|
|||
# Dataset size. Used only when snapshots_json_file_path is defined. Defaults to large | |||
# crowd_dataset_size = "small" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# crowd_dataset_size = "small" | |
# crowd_dataset_size = "large" |
value = local.confluence_ebs_snapshot_id | ||
} | ||
|
||
output "confluence_db_snapshot_build_number" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only Confluence and Crowd require build number?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
install.sh
Outdated
@@ -147,6 +151,9 @@ pre_flight_checks() { | |||
fi | |||
fi | |||
RDS_SNAPSHOT_ID=$(get_variable ${RDS_SNAPSHOT_VAR} "${CONFIG_ABS_PATH}") | |||
if [ "${SNAPSHOTS_JSON_FILE_PATH}" ]; then | |||
RDS_SNAPSHOT_ID=$(cat ${SNAPSHOTS_JSON_FILE_PATH} | jq ".${PRODUCT}.versions[] | select(.version == \"${PRODUCT_VERSION}\") | .data[] | select(.size == \"large\" and .type == \"rds\") | .snapshots[] | .[\"${REGION}\"]" | sed 's/"//g') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't this always read the large
dataset from the JSON file and uses that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, that's the debug version. Let me fix it.
variables.tf
Outdated
type = string | ||
default = "large" | ||
validation { | ||
condition = var.jira_dataset_size == null || can(regex("large|small", var.jira_dataset_size)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
condition = var.jira_dataset_size == null || can(regex("large|small", var.jira_dataset_size)) | |
condition = var.jira_dataset_size == null || contains(["large", "small"], var.jira_dataset_size) |
I find this easier to read.
test/dcapt-snapshots.json
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this the snapshot file that is going to be used with DCAPT or we are distributing the DCAPT snapshot json via DCAPT framework?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This one is just for tests. DCAPT will distribute a proper one.
@@ -17,4 +17,100 @@ locals { | |||
local_bamboo_chart_path = var.local_helm_charts_path != "" && var.bamboo_install_local_chart ? "${var.local_helm_charts_path}/bamboo" : "" | |||
local_agent_chart_path = var.local_helm_charts_path != "" && var.bamboo_install_local_chart ? "${var.local_helm_charts_path}/bamboo-agent" : "" | |||
local_crowd_chart_path = var.local_helm_charts_path != "" && var.crowd_install_local_chart ? "${var.local_helm_charts_path}/crowd" : "" | |||
|
|||
# snapshots_json = var.snapshots_json_file_path != "" ? jsondecode(file(var.snapshots_json_file_path)) : jsondecode("{\"jira\": {\"versions\":[]},\"confluence\": {\"versions\":[]},\"crowd\": {\"versions\":[]},\"bitbucket\": {\"versions\":[]}}") | |||
snapshots_json = var.snapshots_json_file_path != "" ? jsondecode(file(var.snapshots_json_file_path)) : null |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we put a sentence or two describing the filtering process and how the snapshot values are determined?
I had to read it a couple of times to see how this is used, and I am sure I will forget it before I will see this code again :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If snapshots_json_file_path
is set then we parse the json and get snap IDs from there. If legacy var $product_db_snapshot_id
is defined we will use it if either snapshots_json_file_path
is undefined or we failed to find snapshot id for the product version in the json. This way we don't change an existing way of supplying snap IDs, just introduce a new one.
This PR keeps an existing way to read snapshot IDs from tfvars file but adds an alternative method - parse json file and get EBS and RDS snapshots IDs from it. Depending on the product, version and dataset size snapshots IDs are passed to each product module.
Snapshots json takes precedence over existing snap variables if they are defined.
Checklist