Support loading into K3s with k3s.local

[kameshsampath]

• use k3s.local domain
• use LIMA_INSTANCE as way to detect if its Rancher Desktop or vanilla lima-vm

[imjasonh]

This looks pretty good so far! I'm glad to see how (relatively) little work it is to get these images loaded into k3s!

I'd love to see some kind of e2e test, sort of like we have in kind-e2e.yaml -- to ensure this works, and that future changes don't break it. Is that something we could add relatively easily?

[imjasonh]

I see this is largely copied from code in publish/kind.go -- I think there's an opportunity to share this code instead of duplicating it, especially since I think there are improvements we could make to it later. Could you refactor this into some shared method?

[kameshsampath]

yeah I thought so to refactor, no worries I could that now

[kameshsampath]

@imjasonh - fixed this one with latest commit

[kameshsampath]

I'd love to see some kind of e2e test, sort of like we have in kind-e2e.yaml -- to ensure this works, and that future changes don't break it. Is that something we could add relatively easily?

I am exploring the ways to do it as installing Rancher Desktop or Lima requires the hypervisors on the runner and I am finding way to install/enable it. IMHO from my previous experiences I was not able to do it. 🤔

[kameshsampath]

This looks pretty good so far! I'm glad to see how (relatively) little work it is to get these images loaded into k3s!

I'd love to see some kind of e2e test, sort of like we have in kind-e2e.yaml -- to ensure this works, and that future changes don't break it. Is that something we could add relatively easily?

@imjasonh tried doing a similar one https://github.com/kameshsampath/ko-demo-app/blob/main/.github/workflows/k3s-e2e.yaml but some reasons I am getting this error running the ko build

Run # Test with k3s load
2022/03/22 [12](https://github.com/kameshsampath/ko-demo-app/runs/5643866079?check_suite_focus=true#step:10:12):31:37 Using base gcr.io/distroless/static:nonroot@sha256:80c956fb0836a17a565c43a4026c9c80b20[13](https://github.com/kameshsampath/ko-demo-app/runs/5643866079?check_suite_focus=true#step:10:13)c83bea09f74fa4da[19](https://github.com/kameshsampath/ko-demo-app/runs/5643866079?check_suite_focus=true#step:10:19)5a59b7a99 for github.com/kameshsampath/ko-demo-app/cmd
[20](https://github.com/kameshsampath/ko-demo-app/runs/5643866079?check_suite_focus=true#step:10:20)[22](https://github.com/kameshsampath/ko-demo-app/runs/5643866079?check_suite_focus=true#step:10:22)/03/22 12:31:37 Using build config ko-demo-app for github.com/kameshsampath/ko-demo-app/cmd
2022/03/22 12:31:37 Building github.com/kameshsampath/ko-demo-app/cmd for linux/arm64
2022/03/22 12:31:37 Using build config ko-demo-app for github.com/kameshsampath/ko-demo-app/cmd
2022/03/22 12:31:37 Building github.com/kameshsampath/ko-demo-app/cmd for linux/arm
2022/03/22 12:31:37 Using build config ko-demo-app for github.com/kameshsampath/ko-demo-app/cmd
2022/03/22 12:31:37 Building github.com/kameshsampath/ko-demo-app/cmd for linux/s390x
Unable to connect to the server: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2022/03/22 12:32:09 Unexpected error running "go build": signal: killed
2022/03/22 12:32:09 Unexpected error running "go build": signal: killed
2022/03/22 12:32:09 Unexpected error running "go build": signal: killed
Error: error executing 'kubectl apply': exit status 1
2022/03/22 12:32:09 error during command execution:error executing 'kubectl apply': exit status 1
Error: Process completed with exit code 1.

any clue ?

[imjasonh]

any clue ?

Not really, honestly. 😕 Does this work when you try it on your own machine? Maybe there's some problem with k3s as it's set up by the action?

FWIW, I can't imagine that --platform=all is needed, since the code just selects the correct platform anyway. I don't think that's the cause of the issue, but it couldn't hurt, and should make the test faster at least.

[imjasonh]

I didn't see anything obvious that might cause loading into k3s to fail. Just a few small comments in the meantime though.

[kameshsampath]

any clue ?

Not really, honestly. 😕 Does this work when you try it on your own machine? Maybe there's some problem with k3s as it's set up by the action?

FWIW, I can't imagine that --platform=all is needed, since the code just selects the correct platform anyway. I don't think that's the cause of the issue, but it couldn't hurt, and should make the test faster at least.

Kind of got through the apiserver issue, but I see GHA macos runners are very flaky getting the error,

2022/03/25 08:14:24 Using build config ko-demo-app for github.com/kameshsampath/ko-demo-app/cmd/pingpong
2022/03/25 08:14:24 Building github.com/kameshsampath/ko-demo-app/cmd/pingpong for linux/amd64
Unable to connect to the server: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2022/03/25 08:14:55 Unexpected error running "go build": signal: killed
Error: error executing 'kubectl apply': exit status 1
2022/03/25 08:14:55 error during command execution:error executing 'kubectl apply': exit status 1

IMHO - the runner is lacking in memory or CPU 🤔 . The same works very well in my local mac.

[imjasonh]

IMHO - the runner is lacking in memory or CPU 🤔 . The same works very well in my local mac.

It sounds like they should have plenty of resources to do a ko build and apply -- 3 CPU, 14 GB RAM, according to https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

Just in case, it might make sense to lower the requirements specified here to match: https://github.com/kameshsampath/ko-demo-app/blob/main/ko-k3s-demos.yaml#L39-L49

The only one that exceeds the available runner resources is disk, maybe that's the problem? I'm sort of running out of ideas (if that's not obvious 😅), is there some Rancher Desktop expert we could tag in to help? Having a working Rancher Desktop CI setup seems useful in lots of other scenarios too, maybe we can borrow from some existing setup?

[kameshsampath]

IMHO - the runner is lacking in memory or CPU 🤔 . The same works very well in my local mac.

It sounds like they should have plenty of resources to do a ko build and apply -- 3 CPU, 14 GB RAM, according to https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

Just in case, it might make sense to lower the requirements specified here to match: https://github.com/kameshsampath/ko-demo-app/blob/main/ko-k3s-demos.yaml#L39-L49

The only one that exceeds the available runner resources is disk, maybe that's the problem? I'm sort of running out of ideas (if that's not obvious 😅), is there some Rancher Desktop expert we could tag in to help? Having a working Rancher Desktop CI setup seems useful in lots of other scenarios too, maybe we can borrow from some existing setup?

yeah tried reducing the CPU to smallest possible values and trying . I am also checking with Rancher Desktop folks on their slack to see if they any thoughts 🤞🏾

[kameshsampath]

@imjasonh Rancher is adding rdctl to do all rancher desktop stuff using CLI https://docs.rancherdesktop.io/references/rdctl-command-reference, its not yet complete. I have to wait for that to have the e2e completed as the other techniques are not reliable

[github-actions]

This Pull Request is stale because it has been open for 90 days with
no activity. It will automatically close after 30 more days of
inactivity. Keep fresh with the 'lifecycle/frozen' label.