Experimenting with various CSI drives for Docker/yarn caching on GKE

[Diontsoumas]

We are experimenting around docker caching on our GCP cluster and the suggested solution with the pool of PVs is working great, but has some obvious downsides:

• you need to be cautious on how much you autoscale, since the GCE disks can be very expensive in the long run, especially if teams are not cleaning after their images.
• you need to find a way to clear any unused images after X amount of time, otherwise you should just delete the disks and start from the beginning
• if you add yarn/go caching on top of it, things get even more expensive

Other than the GCE CSI provider, we have also experimented with:

• GCS CSI provider (with dynamic provisioning so all cache will be on a pool of cheap GCS buckets)
• Filestore GCS provider (which supports ReadWriteMany, so hopefully a single instance will be enough)

Unfortunately, none of the above solutions is working as intended, the performance is actually worse:

• With the GCS CSI provider, when sharing /var/lib/docker, docker couldn't start (Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?)
• Sharing another folder (ex /home/runner/.cache/ as the doc example suggests) it has no difference by default. I can see that docker create runs with -v "/runner/_work":"/__w", so I assume that having a volume mount like the following

        volumeMounts:
        - name: var-lib-docker
          mountPath: /runner/_work/.cache/

will be mounted into /__w/.cache on the pod (we use dockerdWithinRunnerContainer: true). This is working (tested on yarn cache), but performance is terrible, running yarn never finishes (ot at least takes >30'), while the original job takes ~2'. GCS bucket is properly populated with files, so the solution is working, albeit rather slowly.

• Filestore experiments are similar.

Long story short, none of our tests offered any good alternatives for cheaper storage solution when it comes to caching. In fact, most of them barely worked. Is anyone else using a different approach? Maybe we miss something?