CLOUD-861: Automate operator release #320

pooknull · 2024-10-22T14:50:51Z

https://perconadev.atlassian.net/browse/CLOUD-861

This PR introduces a new binary, operator-tool, which automates the generation of source files for operators.

See README.md for more details.

Still to be done (could be addressed in future PRs):

option to update of tests in api-tests directory
option to generate *.dep.json files
add unit-tests

JNKPercona · 2024-10-22T14:58:00Z

Test name	Status
api-tests	failed

jvpasinatto · 2024-10-24T16:57:35Z

A few points:

The tool is returning the full list of pmm images(or the diff when providing a file), but we only need the latest. If we gonna put more than one, we need to align with the team.
Currently, we set the status to "recommended" for the latest of each one of major version(ex: for "postgresql" we have 5 recommended db versions), we need to align what should be the proper way, as the tool is setting only the latest major version as "recommended".
I think we should have an error when an object is empty. For ex, if I specify --operator "pg-operator" --version "2.5.1" the operator object is empty because this image does not exist yet, but we have no feedback.
The keys for the PXC images are missing the versions after dash. Ex: 5.7.44 but should be 5.7.44-31.65
It would be nice to have an option to have a "patch release", so the tool does not try to get the latest images of the softwares, but only update the operator images, and needed software tags.

egegunes · 2024-10-25T08:13:05Z

tools/operator-tool/README.md

+$ ./bin/operator-tool --operator "psmdb-operator" --version "1.17.0" # outputs source file for psmdb-operator
+...
+$ ./bin/operator-tool --operator "pg-operator" --version "2.5.0"     # outputs source file for pg-operator
+...
+$ ./bin/operator-tool --operator "ps-operator" --version "0.8.0"     # outputs source file for ps-operator
+...
+$ ./bin/operator-tool --operator "pxc-operator" --version "1.15.1"   # outputs source file for pxc-operator


since the flag is already --operator maybe we can just use --operator psmdb or --operator pxc, wdyt?

JNKPercona · 2024-10-28T12:38:05Z

Test name	Status
api-tests	failed

JNKPercona · 2024-10-29T12:38:51Z

Test name	Status
api-tests	failed

JNKPercona · 2024-10-31T12:36:56Z

Test name	Status
api-tests	failed

JNKPercona · 2024-11-04T12:04:59Z

Test name	Status
api-tests	failed

JNKPercona · 2024-11-13T15:25:21Z

Test name	Status
api-tests	failed

JNKPercona · 2024-11-13T17:42:52Z

Test name	Status
api-tests	failed

JNKPercona · 2024-11-27T16:07:14Z

Test name	Status
api-tests	failed

JNKPercona · 2024-11-27T16:08:45Z

Test name	Status
api-tests	failed

jvpasinatto · 2025-01-14T18:27:17Z

tools/operator-tool/internal/matrix/matrix.go

+}
+
+func PSMDB(f *filler.VersionFiller, version string) (*vsAPI.VersionMatrix, error) {
+	mongoVersions, err := productsapi.GetProductVersions("percona-server-mongodb-", "percona-server-mongodb-7.0", "percona-server-mongodb-6.0", "percona-server-mongodb-5.0")


please include mongo 8.0

and remove mongo 5.0

jvpasinatto · 2025-01-14T18:27:31Z

tools/operator-tool/internal/matrix/matrix.go

+		return nil, err
+	}
+
+	xtrabackupVersions, err := productsapi.GetProductVersions("Percona-XtraBackup-", "Percona-XtraBackup-8.0", "Percona-XtraBackup-2.4")


please include pxc 8.4

jvpasinatto · 2025-01-14T18:28:06Z

tools/operator-tool/internal/matrix/matrix.go

+)
+
+func PG(f *filler.VersionFiller, version string) (*vsAPI.VersionMatrix, error) {
+	pgVersions, err := productsapi.GetProductVersions("", "postgresql-distribution-16", "postgresql-distribution-15", "postgresql-distribution-14", "postgresql-distribution-13", "postgresql-distribution-12")


please include pg 17

and remove pg 12

tools/operator-tool/README.md

jvpasinatto · 2025-01-14T18:34:24Z

tools/operator-tool/internal/util/arch.go

+
+import "strings"
+
+func GetArchSuffixes() []string {


we actually only need -multi for now

This function shows all known arch suffixes for the image. It's used for truncating suffixes from images, not for including them.

This commit makes operator-tool to include only multi and amd64 images: 6ebf2b3

jvpasinatto · 2025-01-14T18:43:15Z

tools/operator-tool/internal/filler/filler.go

+// Logic:
+//   - If an image supports both amd64 and arm64 architectures and has a "-multi" suffix in its tag,
+//     the function includes a version of the image tag without the "-multi" suffix in the map.
+//   - If no image with both amd64 and arm64 builds is found, separate images for amd64 and arm64


we don't need separated images for arm64 and amd64, as we don't need arm64 images individually.
When a image does not have arm64, just provide the it's tag with only amd64 hash.

jvpasinatto · 2025-01-14T19:08:57Z

tools/operator-tool/README.md

+        Specify an older source file. The operator-tool will exclude any versions that are older than those listed in this file
+  -include-arch-images
+        Include images with "-multi", "-arm64", "-aarch64" suffixes in the output file
+  -only-latest


I'm using ./bin/operator-tool -operator psmdb --version 1.19.0 --only-latest --include-arch-images --file sources/operator.1.18.0.psmdb-operator.json

This makes backup, operator and pmm have 2 images, but ideally we want just the latest.

When --file is used with --only-latest, operator-tool appends latest images to the provided file.

This commit makes operator-tool to output only the latest images if the --file is not specified.: fdc56d5

JNKPercona · 2025-02-27T12:00:32Z

Test name	Status
api-tests	passed

operator-tool init commit

c591e1e

hors requested review from jvpasinatto, nmarukovich, egegunes and ptankov October 22, 2024 19:03

egegunes reviewed Oct 25, 2024

View reviewed changes

pooknull added 4 commits October 28, 2024 09:07

use latest pmm image

06c3504

update --operator option values

221623c

return error if image was not found

cd15072

add more versions from docker hub

53a8acd

add --patch option

e7c3f6a

pooknull added 4 commits October 31, 2024 09:20

fix adding versions from registry

b345ec2

move packages to pkg

bf19005

fix bugs

09339b3

refactor util.go

42d83e0

pooknull added 2 commits November 4, 2024 11:49

add -include-arch-images option

267c2a8

prefer prerelease versions

e86480f

pooknull added 2 commits November 13, 2024 13:38

get latest backups

e9e8c9b

implement --cap

e5bd255

refactor

9a01dc6

add --only-latest

0fc80cc

use includeArchSuffixes for latest images

dff5d64

jvpasinatto requested changes Jan 14, 2025

View reviewed changes

pooknull changed the title ~~WIP: Automate operator release~~ CLOUD-861: Automate operator release Feb 27, 2025

pooknull added 4 commits February 27, 2025 10:05

Merge remote-tracking branch 'origin/main' into dev/operator-tool

16c4f0e

update matrix

49e01e5

use --only-latest without --file

fdc56d5

use only '-multi' and '-amd64' images

6ebf2b3

CLOUD-861: Automate operator release #320

Are you sure you want to change the base?

CLOUD-861: Automate operator release #320

Conversation

pooknull commented Oct 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JNKPercona commented Oct 22, 2024

Uh oh!

jvpasinatto commented Oct 24, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JNKPercona commented Oct 28, 2024

Uh oh!

JNKPercona commented Oct 29, 2024

Uh oh!

JNKPercona commented Oct 31, 2024

Uh oh!

JNKPercona commented Nov 4, 2024

Uh oh!

JNKPercona commented Nov 13, 2024

Uh oh!

JNKPercona commented Nov 13, 2024

Uh oh!

JNKPercona commented Nov 27, 2024

Uh oh!

JNKPercona commented Nov 27, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pooknull Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JNKPercona commented Feb 27, 2025

Uh oh!

Uh oh!

pooknull commented Oct 22, 2024 •

edited

Loading

pooknull Feb 27, 2025 •

edited

Loading