scripts: Start of new requirements installer

[aarnq]

Warning

This is a public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request, nor
• business confidential information, such as customer names.

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• [kind/adr](set-me)

What does this PR do / why do we need this PR?

This aim to develop a lightweight requirements installer that can be used in a container as well as on workstations, using the current PURL format.

The goal is to be able to serve the tests/Dockerfile with the following types:

• deb packages
• helm plugins
• npm packages
• direct bin/tar/zip downloads with checksum verification

As much as possible is to be done in bash alone, debs are always installed first as that may install other package managers, but if needed they are expected to be installed before running the script.

The last type is provided by mapping, which isn't the greatest method, but the one that we kind of already use in the current ansible based approach.

Note that the purl parser does not attempt to be compliant, but tries to be enough for our purposes.

Information to reviewers

Currently I've tested to just run the script against the base requirement file when building a container.

We probably need to have a discussion around the purl types as we are using strange ways to refer to golang packages.
Logically it does not make sense for us to refer to golang packages, as we are not fetching golang packages. It should be github packages for a lot of these as we fetch them from their github releases.

I'll wait with implementing more mappings until we have discussed this.

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
  • The change updates CRDs
  • The change updates the config and the schema
• Documentation checks:
  • The public documentation required no updates
  • The public documentation required an update - [link to change](set-me)
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts required no updates)
  • The metrics names did change (Grafana dashboards and Prometheus alerts required an update)
• Logs checks:
  • The logs do not show any errors after the change
• PodSecurityPolicy checks:
  • Any changed Pod is covered by Kubernetes Pod Security Standards
  • Any changed Pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any Pods to be blocked by Pod Security Standards or Policies
• NetworkPolicy checks:
  • Any changed Pod is covered by Network Policies
  • The change does not cause any dropped packets in the NetworkPolicy Dashboard
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Bug checks:
  • The bug fix is covered by regression tests

[simonklb]

The reason why I picked the golang PURL type was because vulnerabilities are usually reported for the go modules.

For example:

$ curl -s -d '{"package": {"purl": "pkg:golang/helm.sh/helm"}}' "https://api.osv.dev/v1/query" | jq '.vulns | length'
10
$ curl -s -d '{"package": {"purl": "pkg:github/helm/helm"}}' "https://api.osv.dev/v1/query" | jq '.vulns | length'
0

[aarnq]

The reason why I picked the golang PURL type was because vulnerabilities are usually reported for the go modules.

Sure but if that is going to be a consideration then we should have a at least a plan to take advantage of that, so please create an issue for it else I don't see why we should have roundabout ways to manage it.
And if that is going to be a requirement then many purls are mismatched against OSV.
Would there be any other sources we want to query purls against?

As an alternative, what we have talked about is for the requirement script to be able to download newer versions and at release create a pinned requirements listing. Would it make sense in a such requirements list to write golang purls as an additional alias for that type of use?
Then we could allow the "floating" requirements to have more potential non standard shenanigans to make the fetching easier for us, and the pinned one to be as compliant as we can make it.

[Zash]

Looks promising!

[simonklb]

The reason why I picked the golang PURL type was because vulnerabilities are usually reported for the go modules.

Sure but if that is going to be a consideration then we should have a at least a plan to take advantage of that, so please create an issue for it else I don't see why we should have roundabout ways to manage it.

My understanding was that this issue would cover that.

And if that is going to be a requirement then many purls are mismatched against OSV. Would there be any other sources we want to query purls against?

All the PURLs ideally. But matching CVEs against PURLs isn't a straight forward job, talking from experience from previous engagements... 😄

As an alternative, what we have talked about is for the requirement script to be able to download newer versions and at release create a pinned requirements listing. Would it make sense in a such requirements list to write golang purls as an additional alias for that type of use? Then we could allow the "floating" requirements to have more potential non standard shenanigans to make the fetching easier for us, and the pinned one to be as compliant as we can make it.

Yea I'm not against having a list of "installable" PURLs and then a sublist of aliases/additional resources which can be used to, for example, discover source code for deb packages and vulnerabilities.

[aarnq]

The reason why I picked the golang PURL type was because vulnerabilities are usually reported for the go modules.

Sure but if that is going to be a consideration then we should have a at least a plan to take advantage of that, so please create an issue for it else I don't see why we should have roundabout ways to manage it.

My understanding was that this issue would cover that.

Well, there was nothing there even suggesting that it should be implemented using queries on purls rather than our standard method of subscribing to security notices.
So I want added it.

And if that is going to be a requirement then many purls are mismatched against OSV. Would there be any other sources we want to query purls against?

All the PURLs ideally. But matching CVEs against PURLs isn't a straight forward job, talking from experience from previous engagements... 😄

Note, I'm asking for sources to query against so I can evaluate which purls to use before I create specific mappings that are going to replaced, not which purls to query as I assume we want that for everything. And as you have previous experience, I thought you might have some suggestions. 😉

As an alternative, what we have talked about is for the requirement script to be able to download newer versions and at release create a pinned requirements listing. Would it make sense in a such requirements list to write golang purls as an additional alias for that type of use? Then we could allow the "floating" requirements to have more potential non standard shenanigans to make the fetching easier for us, and the pinned one to be as compliant as we can make it.

Yea I'm not against having a list of "installable" PURLs and then a sublist of aliases/additional resources which can be used to, for example, discover source code for deb packages and vulnerabilities.

Aight, then I'll move towards this because it make more logical sense to me for the requirements installer, and then we can create a framework for increasing our query coverage with mappings later on. 👍

[simonklb]

And if that is going to be a requirement then many purls are mismatched against OSV. Would there be any other sources we want to query purls against?

All the PURLs ideally. But matching CVEs against PURLs isn't a straight forward job, talking from experience from previous engagements... 😄

Note, I'm asking for sources to query against so I can evaluate which purls to use before I create specific mappings that are going to replaced, not which purls to query as I assume we want that for everything. And as you have previous experience, I thought you might have some suggestions. 😉

Sorry I misread it. I am not aware of any other sources that can query PURLs directly so we'll likely have to parse and translate.

Aight, then I'll move towards this because it make more logical sense to me for the requirements installer, and then we can create a framework for increasing our query coverage with mappings later on. 👍

Yea that sounds good. Let's focus on getting installations properly implemented and figure out security scanning later. Having extras/metadata part of the requirements structure doesn't hurt though.

[simonklb]

What is the reason for the DIY PURL parser over using one of the official ones? Other than that it's a fun exercise of course! 😄

[aarnq]

What is the reason for the DIY PURL parser over using one of the official ones? Other than that it's a fun exercise of course! 😄

Zero dependencies.
I want to be able to use this in a container, so I don't want to have to add any runtime or external tool on top of it, and there are no official script version.

[simonklb]

What is the reason for the DIY PURL parser over using one of the official ones? Other than that it's a fun exercise of course! 😄

Zero dependencies. I want to be able to use this in a container, so I don't want to have to add any runtime or external tool on top of it, and there are no official script version.

If you truly want to achieve zero dependencies it would make sense to me to build this in a compiled language. Otherwise you will have to bundle this with bash. The go library is really straight forward if you want to give it a go (hehe)!

[aarnq]

What is the reason for the DIY PURL parser over using one of the official ones? Other than that it's a fun exercise of course! 😄

Zero dependencies. I want to be able to use this in a container, so I don't want to have to add any runtime or external tool on top of it, and there are no official script version.

If you truly want to achieve zero dependencies it would make sense to me to build this in a compiled language. Otherwise you will have to bundle this with bash. The go library is really straight forward if you want to give it a go (hehe)!

Definitely in the future, but given how we have it today this is the simplest solution, and it doesn't require anything but a file to be included.

Edit: Also as a note, the base image target we have is ubuntu so bash is preloaded.
Had we a decision to go with scratch or distroless then it would become higher prio.
Do not want to port this to plain sh. 🫠

[aarnq]

I have been working on this on and off for a while and now it is only missing a handler for helm-plugins, then it can install everything the current requirements installer can.
And it can find the latest versions and checksums of them automagically.

[simonklb]

This looks great! The only thing that is unfortunate is that it will be all done in the same layer in the Dockerfile. Would it be possible to perform them as separate layers so that you don't have to re-download everything if you just change one dependency?

[aarnq]

This looks great! The only thing that is unfortunate is that it will be all done in the same layer in the Dockerfile. Would it be possible to perform them as separate layers so that you don't have to re-download everything if you just change one dependency?

Not in a Docker/Containerfile directly, if we were to use something like Buildah and script the build process it would be easy, dunno if Docker has something similar 🤔

Perhaps if we had it generate/update a Docker/Containerfile with the steps to install it instead 🤔

My thinking though is that it will be a separate PR to make this ready for containers, and focus on the requirements first.

[simonklb]

This looks great! The only thing that is unfortunate is that it will be all done in the same layer in the Dockerfile. Would it be possible to perform them as separate layers so that you don't have to re-download everything if you just change one dependency?

Not in a Docker/Containerfile directly, if we were to use something like Buildah and script the build process it would be easy, dunno if Docker has something similar 🤔

Perhaps if we had it generate/update a Docker/Containerfile with the steps to install it instead 🤔

My thinking though is that it will be a separate PR to make this ready for containers, and focus on the requirements first.

Yep, that makes sense!

[aarnq]

helm-plugin support!

next is to refactor, simplify, and harmonise how output is done.