Handling admission controller request timeouts

[etrexel]

When Ratify receives a request from an admission controller to validate a deployment there are strict timeouts for those requests. A number of factors can impact the amount of time that it takes Ratify to validate a request:

• Number of registries involved in the deployment
• Round trip time to each registry
• Number of artifacts associated with each container image
• Number of verifiers configured
• Others

Due to these factors it is challenging to determine if a request can be fully validated within the allotted time. We need to come up with a plan to handle these scenarios.

[etrexel]

Current plan is to approach this over time in a couple ways:

Short Term

Add a step to the Ratify API that will check if the requested container's artifacts are in cache. If they are not, return failure and cache all of the container's artifacts. This will ensure that we are blocking images that we will not be able to check within the timeout. The side effect of this approach is that valid container images will be blocked the first time a deployment request is made for them. Each subsequent request for the same image should be able to be validated.

Medium Term

Add functionality to Ratify to crawl and pre-cache container image artifacts from configured registries. This will reduce the number of round trips necessary during the admission request and result in more images being validated on the first request. We will need to configure caching correctly and ensure that we refresh artifacts on a reasonable interval.

Long Term

TBD

[akashsinghal]

Helm Exploration

Is it possible for us to do artifact fetching or even full validation prior to deployment?

Motivating Questions:

1. Can we have a hard dependency on helm for Ratify? Currently we use kubectl for pod creation in cluster.
2. What helm features can be leveraged for pre-validation?
3. Chart Hooks: If the hook resource is a Pod, will Gatekeeper also block it? If the hook resource is a Job, could Ratify be configured for Job admission control like Pod admission control is being used currently?
4. Chart Hooks: How can the Chart Hook get access to the image name trying to be deployed for the Pod?
5. Chart Hooks: Can the hook send a request to the Ratify service running in the cluster?

[susanshi]

We are not currently exploring this but just to understand if this is an option.
In short, there are no concerns from helm invoking ratify as a pre install step.

Some other takeaways:

• The default timeout for the preinstall step is 15min
• This step should probably be a "Job" instead of a "pod" so we can wait on the job completion (e.g. can wait on ratify completing verification and caching the result).
• There maybe issues on getting the image name for the deployment, suggestion is to use label instead.

The disadvantage of this options are:

• customer would have to configure the preinstall step in their helm charts
• not all deployment are through helm

[akashsinghal]

We have been compiling experiments/analysis on improving ORAS latency here: https://hackmd.io/@akashsinghal/rkEZqxxW5

[akashsinghal]

We have asked for Gatekeeper to increase the default timeout. The original reason for decreasing the default timeout from 5 to 3 seconds was due to a leader election issue. That issue no longer applies post k8 1.20. open-policy-agent/gatekeeper#1956

In the meantime, Ratify has updated helm charts and deployment instructions to set validating webhook timeout to 7 seconds. #160