cmd-fetch: Derive konflux lockfiles from rpm-ostree #4298

jbtrystram · 2025-09-04T14:12:32Z

Let's use our lockfiles to get hermeto download a set of matching RPMs. This is done through injecting includepkg filters to the repofiles then feeding that to rpm-lockfile-prototype.

The overview is:
we parse the manifest for a package list and enabled repos. For each package, we take the expected EVRA in the rpm-ostree lockfile and inject it in the includepkg filter in the repos files. Finally we feed that package list and the custom repo definition to rpm-ostree-prototype.
The result is a hermeto/cachi2 lockfile we will commit to the repo.

This --konflux flag would then be added to the Jenkins bump-lockfiles job to update both lockfiles in the same commit.

This will allow hermetic builds in Konflux.

I also added a sanity check at the end that iterate over both lockfiles and compare all versions string to make sure we have exact matches all around.

See also coreos/fedora-coreos-config#3644 for the inital implementation.

gemini-code-assist

Code Review

This pull request introduces a valuable feature for generating Konflux-compatible lockfiles to enable hermetic builds. The approach of deriving the new lockfile from the existing rpm-ostree one and including a sanity check is solid. I've identified a couple of high-severity issues that could cause script failures, including a syntax error and a potential TypeError. I've also included several medium-severity suggestions to improve code clarity, fix a typo in a user-facing message, and enhance maintainability by removing hardcoded values.

src/cmd-fetch

src/konflux-rpm-lockfile

src/cosalib/cmdlib.py

src/konflux-rpm-lockfile

Using coreos/coreos-assembler#4298

src/konflux-rpm-lockfile

jbtrystram · 2025-09-22T19:10:59Z

I pushed a second commit here that use dnf repoquery using the rpm-ostree lockfiles values and generate the hermeto lockfile directly. without needing rpm-lockfile-prototype, as suggested by @jlebon

@joelcapitao PTAL

Using coreos/coreos-assembler#4298

jbtrystram · 2025-09-26T12:52:23Z

This ends up being a whole more complex than I tought so turning it back to draft

Using coreos/coreos-assembler#4298

jlebon

Nice, looking better! I'm much more confident about this personally now that it's not doing a separate depsolve.

src/konflux-rpm-lockfile

src/cosalib/cmdlib.py

jlebon · 2025-09-26T15:20:41Z

src/cosalib/cmdlib.py

+    return json.loads(data)
+
+
+def get_locked_nevras(srcdir):


Hmm, I don't think we need this. Why not use the $outfile from the rpm-ostree step?

Good point ! I grafted my stuff into cmd-fetch without looking too much at the integration. Thanks for the suggestion

follow-up suggestion that I only came up with, with some changes I have not push yet : using dnf repoquery we can get all the packages location from the x86 builder in one process, using --forcearch so we can load all locks from the x86 main machine.
So I could move the --konflux flag to be a separate thing that is called after the tests at the end of the pipeline.

Fine either way. Seems mostly equivalent I think, no? cosa fetch is also called on each architecture individually.

I see two minor advantages to just doing it part of the same cosa fetch build:

keeps the pipeline code simpler

if repoquery flakes on infra, we fail early and not after all the expensive tests have run

Related to this discussion, I think we probably don't actually need a --konflux flag and could just unconditionally do this if we detect there are Konflux locks. The flag is nice if we want to gain confidence first. Though now that this is much more mechanical, I'm personally OK just going straight to the unconditional logic.

* keeps the pipeline code simpler

Does it ? Because if each builder write it's own lockfile we have to add code to the pipeline that merges all of them back to the same file at the end anyway.

* if repoquery flakes on infra, we fail early and not after all the expensive tests have run

Yeah agreed. Looking at the pipeline code this can still be done on the x86 builder before the tests, as all the lockfiles are synced back (for change detection and other things). I'll come up with something and we can discuss it further

Ok I updated the script with your suggestion. See last commit for details.

* keeps the pipeline code simpler

Does it ? Because if each builder write it's own lockfile we have to add code to the pipeline that merges all of them back to the same file at the end anyway.

Sorry, I missed that a key difference here is that Konflux lockfiles keep the different arches in the same lockfile. Is there an option to support separating them out? Otherwise, yeah I guess just doing the merge step separately SGTM.

jbtrystram · 2025-09-29T14:58:43Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a new --konflux flag to generate a hermetic lockfile for Konflux builds. The implementation adds a new script konflux-rpm-lockfile and a helper function in cosalib. The overall approach is sound, but I've identified a critical bug in the new script where command-line arguments for dnf are incorrectly accumulated within a loop, which will lead to incorrect package resolution. I've also found a logic issue in how override lockfiles are merged and suggested a few minor code quality improvements. Please address these points to ensure the new functionality is robust.

src/konflux-rpm-lockfile

src/cosalib/cmdlib.py

src/konflux-rpm-lockfile

src/cmd-fetch

We are about to commit lockfiles to enable hermetic builds, we'll bump definitions with cosa within bump-lockfile Jenkins job [1]. So we don't want Renovate/MintMaker do the maintenance of this file. [1] coreos/coreos-assembler#4298

Let's use our lockfiles to get hermeto download a set of matching RPMs. This is done by simply getting the package URL with `dnf repoquery` using the full NEVRA from the rpm-ostree lockfile. This will allow hermetic builds in Konflux. I also added an override yaml file to allow merging packages to the lock this will allow injecting arbitrary package. This will help enabling workarounds such as coreos/fedora-coreos-config@fb167ed See also coreos/fedora-coreos-config#3644 for the inital implementation and discussion.

Using coreos/coreos-assembler#4298

Refactor `konflux-rpm-lockfile` to support parallel, per-arch builds. The script is now split into `generate` and `merge` subcommands. `cmd-fetch` now calls `generate` to create arch-specific lockfile (e.g., `x86_64.rpms.lock.yaml`). It also look for an existing lockfile and call generate in that case. The `merge` subcommand combines these files in one, while injecting overrides. The override mechanism allow developpers to add extra RPMs needed in the build, e.g. in `buildroot-prep` [1] The pipeline will call merge after running `generate` on each builder. [1] coreos/fedora-coreos-config@fb167ed See: coreos#4298 (comment)

Using coreos/coreos-assembler#4298

src/cmd-fetch

Using coreos/coreos-assembler#4298

Refactor `konflux-rpm-lockfile` to support parallel, per-arch builds. The script is now split into `generate` and `merge` subcommands. `cmd-fetch` now calls `generate` to create arch-specific lockfile (e.g., `x86_64.rpms.lock.yaml`). It also look for an existing lockfile and call generate in that case. The `merge` subcommand combines these files in one, while injecting overrides. The override mechanism allow developpers to add extra RPMs needed in the build, e.g. in `buildroot-prep` [1] The pipeline will call merge after running `generate` on each builder. [1] coreos/fedora-coreos-config@fb167ed See: coreos#4298 (comment)

Using coreos/coreos-assembler#4298

Refactor `konflux-rpm-lockfile` to support parallel, per-arch builds. The script is now split into `generate` and `merge` subcommands. `cmd-fetch` now calls `generate` to create arch-specific lockfile (e.g., `x86_64.rpms.lock.yaml`). It also look for an existing lockfile and call generate in that case. The `merge` subcommand combines these files in one, while injecting overrides. The override mechanism allow developpers to add extra RPMs needed in the build, e.g. in `buildroot-prep` [1] The pipeline will call merge after running `generate` on each builder. [1] coreos/fedora-coreos-config@fb167ed See: coreos#4298 (comment)

joelcapitao · 2025-09-30T15:54:40Z

src/cmd-fetch

  --with-cosa-overrides    Don't ignore cosa overrides in `overrides/rpm`
  --autolock=VERSION       If no base lockfile used, create one from any arch build of `VERSION`
-
+  --konflux                Generate hermeto lockfile for Konflux derived from the rpm-ostree lockfiles.


We could inform the user that if cosa fetch finds rpms.lock.yaml in config directory, then this flag is enabled.

joelcapitao

Tested it locally, LGTM 👍

jlebon · 2025-09-30T16:36:22Z

src/konflux-rpm-lockfile

+    # We want to ensure that hermeto creates repo definitions for every repository.
+    # A round-robin assignment ensures each repo is mentioned at least once.
+    # This is needed because rpm-ostree uses the full list of repos to
+    # resolve packages and errors out if a repository is missing.


Interesting. And I guess hermeto won't inject a repo if it's not mentioned in the lockfile?

And I guess it doesn't matter than in actuality the package URL isn't coming from that given repo?

yes for both. The repoid in the lockfile is for the local repo creation : the rpm have been downloaded, and it will be available under that repo id, so the url is no longer relevant.

jlebon · 2025-09-30T16:36:59Z

src/konflux-rpm-lockfile

+    i = 0
+    for pkg in pkgs:
+        packages.append({"url": pkg, "repoid": local_repos[i % repo_numbers]})
+        i += 1


Suggested change

i = 0

for pkg in pkgs:

packages.append({"url": pkg, "repoid": local_repos[i % repo_numbers]})

i += 1

for i, pkg in enumerate(pkgs):

packages.append({"url": pkg, "repoid": local_repos[i % repo_numbers]})

jlebon · 2025-09-30T16:41:53Z

src/cosalib/cmdlib.py

    return int(subprocess.check_output(['kola', 'ncpu']))
+
+
+def get_locked_nevras(srcdir, arch=None):


Hmm, not sure I understand why we still have this. There's no need for merge semantics here I don't think. And at that point, the caller can just load the base lockfile directly.

we still want overrides for pins don't we ?
If i understand your comment it means the lockfile already include those after cosa fetch --update-lockfile runs ? In which case yeah, we don't we need this

If i understand your comment it means the lockfile already include those after cosa fetch --update-lockfile runs ? In which case yeah, we don't we need this

Correct, yes. --update-lockfile initially did not do that in fact and ignored overrides, but then we had to change it because sometimes things are so broken that you can't even do a depsolve without the overrides.

I mean, ideally yes we do handle override lockfiles as well in a similar way. E.g. if someone today opens PR to fast-track something, they only update the overrides lockfile, not the base one. And things still work.

But now with this Konflux lockfile, it wouldn't pick up that change until the next bump-lockfile run.

We probably want to call this script from https://github.com/coreos/fedora-coreos-config/blob/testing-devel/ci/overrides.py here as well, which is the script that powers the fast-tracking GHA workflow.

Longer-term of course, the right fix is to have a single lockfile language that's understood by both Konflux and the actual build tools (i.e. rpm-software-management/dnf5#833 and the work going on in https://github.com/rpm-software-management/libpkgmanifest).

Correct, yes. --update-lockfile initially did not do that in fact and ignored overrides, but then we had to change it because sometimes things are so broken that you can't even do a depsolve without the overrides.

Oh OK, yeah that make sense.

But now with this Konflux lockfile, it wouldn't pick up that change until the next bump-lockfile run.

Unless you query the url manually, add it to the override file and run merge. But then it's a more complex process to add an override.

We probably want to call this script from coreos/fedora-coreos-config@testing-devel/ci/overrides.py here as well, which is the script that powers the fast-tracking GHA workflow.

I'll look into that

Longer-term of course, the right fix is to have a single lockfile language that's understood by both Konflux and the actual build tools

Agreed. I read a bunch of discussion about this but it looks like it's going to take a little while

jlebon · 2025-09-30T16:42:52Z

src/konflux-rpm-lockfile

+        print("Some packages from the lockfile could not be resolved. The rpm-ostree lockfile is probably out of date.")
+        sys.exit(1)
+
+    print(f"Done. Solved {len(pkg_urls)} packages.")


Suggested change

print(f"Done. Solved {len(pkg_urls)} packages.")

print(f"Done. Resolved location for {len(pkg_urls)} packages.")

jbtrystram mentioned this pull request Sep 4, 2025

Konflux: generate rpm-lockfile-prototype input file coreos/fedora-coreos-config#3644

Closed

jbtrystram requested a review from joelcapitao September 4, 2025 14:13

gemini-code-assist bot reviewed Sep 4, 2025

View reviewed changes

src/cmd-fetch Outdated Show resolved Hide resolved

src/konflux-rpm-lockfile Outdated Show resolved Hide resolved

src/cosalib/cmdlib.py Show resolved Hide resolved

src/konflux-rpm-lockfile Outdated Show resolved Hide resolved

src/konflux-rpm-lockfile Outdated Show resolved Hide resolved

jbtrystram added a commit to jbtrystram/fedora-coreos-config that referenced this pull request Sep 4, 2025

konflux: generate hermeto lockfiles

2b87dfd

Using coreos/coreos-assembler#4298

jbtrystram mentioned this pull request Sep 4, 2025

konflux: enable hermetic builds coreos/fedora-coreos-config#3723

Open

jbtrystram changed the title ~~cmd-fetch: Derive kofnlus lockfiles from rpm-ostree~~ cmd-fetch: Derive konflux lockfiles from rpm-ostree Sep 4, 2025

jlebon reviewed Sep 5, 2025

View reviewed changes

src/konflux-rpm-lockfile Outdated Show resolved Hide resolved

jbtrystram mentioned this pull request Sep 19, 2025

write RPM url in lockfiles coreos/rpm-ostree#5495

Closed

jbtrystram requested a review from jlebon September 22, 2025 19:11

jbtrystram force-pushed the konflux-lockfiles branch from 4818082 to 2ca3c27 Compare September 22, 2025 19:11

jbtrystram added a commit to jbtrystram/fedora-coreos-config that referenced this pull request Sep 23, 2025

konflux: generate hermeto lockfiles

f277ad4

Using coreos/coreos-assembler#4298

jbtrystram force-pushed the konflux-lockfiles branch from 2ca3c27 to 12937d1 Compare September 23, 2025 09:25

jbtrystram added a commit to jbtrystram/fedora-coreos-config that referenced this pull request Sep 25, 2025

konflux: generate hermeto lockfiles

57d2e17

Using coreos/coreos-assembler#4298

joelcapitao mentioned this pull request Sep 26, 2025

Migration to Konflux for a Secure FCOS Release Supply Chain - Phase 1 coreos/fedora-coreos-tracker#2031

Open

15 tasks

jbtrystram marked this pull request as draft September 26, 2025 12:51

openshift-ci bot added the do-not-merge/work-in-progress label Sep 26, 2025

jbtrystram removed request for jlebon and joelcapitao September 26, 2025 12:51

jbtrystram added a commit to jbtrystram/fedora-coreos-config that referenced this pull request Sep 26, 2025

konflux: generate hermeto lockfiles

8bca9f4

Using coreos/coreos-assembler#4298

jlebon reviewed Sep 26, 2025

View reviewed changes

jbtrystram force-pushed the konflux-lockfiles branch from 9bd68a3 to 442cd35 Compare September 29, 2025 14:58

gemini-code-assist bot reviewed Sep 29, 2025

View reviewed changes

joelcapitao reviewed Sep 29, 2025

View reviewed changes

src/cmd-fetch Outdated Show resolved Hide resolved

jbtrystram force-pushed the konflux-lockfiles branch from 442cd35 to d014342 Compare September 30, 2025 08:46

jbtrystram added a commit to jbtrystram/fedora-coreos-config that referenced this pull request Sep 30, 2025

konflux: generate hermeto lockfiles

6cee677

Using coreos/coreos-assembler#4298

jbtrystram force-pushed the konflux-lockfiles branch from 23c2587 to 156db4a Compare September 30, 2025 12:38

jbtrystram added a commit to jbtrystram/fedora-coreos-config that referenced this pull request Sep 30, 2025

konflux: generate hermeto lockfiles

396b7e1

Using coreos/coreos-assembler#4298

joelcapitao reviewed Sep 30, 2025

View reviewed changes

src/cmd-fetch Outdated Show resolved Hide resolved

jbtrystram added a commit to jbtrystram/fedora-coreos-config that referenced this pull request Sep 30, 2025

konflux: generate hermeto lockfiles

af8d40c

Using coreos/coreos-assembler#4298

jbtrystram force-pushed the konflux-lockfiles branch from 156db4a to b961d04 Compare September 30, 2025 15:00

jbtrystram added a commit to jbtrystram/fedora-coreos-config that referenced this pull request Sep 30, 2025

konflux: generate hermeto lockfiles

96cc9cf

Using coreos/coreos-assembler#4298

jbtrystram force-pushed the konflux-lockfiles branch from b961d04 to eb5bd01 Compare September 30, 2025 15:05

joelcapitao reviewed Sep 30, 2025

View reviewed changes

joelcapitao approved these changes Sep 30, 2025

View reviewed changes

jlebon reviewed Sep 30, 2025

View reviewed changes

		return int(subprocess.check_output(['kola', 'ncpu']))


		def get_locked_nevras(srcdir, arch=None):

	print(f"Done. Solved {len(pkg_urls)} packages.")
	print(f"Done. Resolved location for {len(pkg_urls)} packages.")

cmd-fetch: Derive konflux lockfiles from rpm-ostree #4298

Are you sure you want to change the base?

cmd-fetch: Derive konflux lockfiles from rpm-ostree #4298

Uh oh!

Conversation

jbtrystram commented Sep 4, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbtrystram commented Sep 22, 2025

Uh oh!

jbtrystram commented Sep 26, 2025

Uh oh!

jlebon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbtrystram Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbtrystram commented Sep 29, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joelcapitao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

jbtrystram Sep 29, 2025 •

edited

Loading