Skip to content
This repository has been archived by the owner on Aug 19, 2024. It is now read-only.

Conversation

rm3l
Copy link
Member

@rm3l rm3l commented Jun 10, 2024

Description

This PR fixes a reconciliation error occurring against existing CRs after upgrading the operator from the released 1.1.x branch to either main or 1.2.x.

This issue is currently due to a PATCH operation that was denied by Kubernetes against the existing database Service object.
#369 (fix for https://issues.redhat.com/browse/RHIDP-2432) introduced a mechanism to "replace" StatefulSet objects when the PATCH operation fails, by deleting it (without deleting its dependents), so that the next reconciliation will recreate it.
To prevent similar issues (patching fields that are not allowed by K8s) from occurring again in the future, this PR generalizes this logic based on the patch error status and cause (either Field Update Forbidden or Field Value Invalid), i.e., it tries to delete the object anyway if patching it failed, so that it can be recreated upon the next reconciliation.

Which issue(s) does this PR fix or relate to

PR acceptance criteria

  • Tests
  • Documentation
  • If the bundle manifests have been updated, make sure to review the rhdh-operator.csv.yaml file accordingly

How to test changes / Special notes to the reviewer

Repro steps provided in #382.

  • To reproduce the issue using the E2E tests just run make test-e2e-upgrade. It will leverage the default operator image (not patched) and will not pass.
  • Run make test-e2e-upgrade IMG=quay.io/janus-idp/operator:0.3.0-pr-384 to see the E2E tests passing using the operator image from this PR.

rm3l added 4 commits June 10, 2024 17:10
This is because this test requires uninstalling the installed operator, in order to install a previous version, then try out the upgrade
…on fails

This way, it will be recreated at the next reconciliation.
This should prevent similar issues from happening again in the future.

The DB Service could not be patched to set its clusterIP field to None (to make it headless),
when we were upgrading from 1.1 (where it was not headless) to 1.2.

In fact, regarding K8s Services, users can upgrade (add another clusterIP or ipFamily),
downgrade (remove secondary clusterIP or ipFamily),
but *CAN NOT* change primary/secondary clusterIP || ipFamily
*UNLESS* they are changing from/to/ON ExternalName
@rm3l
Copy link
Member Author

rm3l commented Jun 10, 2024

/cherry-pick 1.2.x

@openshift-cherrypick-robot

@rm3l: once the present PR merges, I will cherry-pick it on top of 1.2.x in a new PR and assign it to you.

In response to this:

/cherry-pick 1.2.x

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@rm3l rm3l requested a review from gazarenkov June 10, 2024 15:35
@rm3l rm3l changed the title fix: Fix reconciliation error preventing existing Backstage operands from being upgraded when the operator is upgraded from 1.1.x Fix reconciliation error preventing existing Backstage operands from being upgraded when the operator is upgraded from 1.1.x Jun 10, 2024
@rm3l rm3l changed the title Fix reconciliation error preventing existing Backstage operands from being upgraded when the operator is upgraded from 1.1.x fix: Fix reconciliation error preventing existing Backstage operands from being upgraded when the operator is upgraded from 1.1.x [RHIDP-2597] Jun 10, 2024
controllers/backstage_controller.go Outdated Show resolved Hide resolved
controllers/backstage_controller.go Outdated Show resolved Hide resolved
tests/e2e/e2e_test.go Outdated Show resolved Hide resolved
tests/e2e/e2e_test.go Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
@rm3l rm3l requested a review from gazarenkov June 11, 2024 11:58
Copy link
Member

@gazarenkov gazarenkov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And please add some note that annotations and labels will be lost to the log

@nickboldt
Copy link
Member

/cherry-pick 1.2.x

@openshift-cherrypick-robot

@nickboldt: once the present PR merges, I will cherry-pick it on top of 1.2.x in a new PR and assign it to you.

In response to this:

/cherry-pick 1.2.x

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

…ost when the object is deleted

Co-authored-by: Gennady Azarenkov <[email protected]>
Copy link

sonarcloud bot commented Jun 11, 2024

Quality Gate Passed Quality Gate passed

Issues
0 New issues
2 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

@rm3l rm3l requested a review from gazarenkov June 11, 2024 13:46
@openshift-ci openshift-ci bot added the lgtm PR is ready to be merged. Required by Prow. label Jun 11, 2024
Copy link

openshift-ci bot commented Jun 11, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gazarenkov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved PR has been approved by an approver from all required OWNERS files. Required by Prow. label Jun 11, 2024
@openshift-merge-bot openshift-merge-bot bot merged commit 7086951 into janus-idp:main Jun 11, 2024
8 checks passed
@rm3l rm3l deleted the 382-existing-backstage-operand-not-upgraded-stuck-on-mounting-a-configmap-after-upgrading-operator-from-1.1.x-to-1.2.x branch June 11, 2024 14:05
@openshift-cherrypick-robot

@rm3l: new pull request created: #385

In response to this:

/cherry-pick 1.2.x

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
approved PR has been approved by an approver from all required OWNERS files. Required by Prow. lgtm PR is ready to be merged. Required by Prow.
Projects
None yet
4 participants