K0s cluster

Private Kubernetes cluster setup on a home lab using k0sctl and Helm charts.

Installation

First install the prerequisites:

# install kubectl, helm
brew install kubectl helm
# install k0sctl for cluster setup
brew install k0sproject/tap/k0sctl

Usage

Setup K8s Cluster

See detailed blog post Setup a private Kubernetes cluster with k0sctl

Inspect/configure ./cluster/k0sctl.yaml and run:

k0sctl apply --config cluster/k0sctl.yaml
# save kubeconfig
k0sctl kubeconfig --config cluster/k0sctl.yaml > ~/.kube/config && chmod go-r ~/.kube/config
k get nodes
# make openebs-hostpath the default storage class
kubectl patch storageclass openebs-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

To reset:

k0sctl reset --config cluster/k0sctl.yaml

It's common to encounter host key mismatch: knownhosts: key mismatch error. In that case, remove the host entries in ~/.ssh/known_hosts and retry.

Install Cluster Addons

See detailed blog post Setting up Kubernetes Addons.

Install the cluster components with Helm:

• cert-manager: certificate management
• cluster-autoscaler: to dynamically autoscale cluster by adding or reducing nodes
• metrics-server: for monitoring and and HPA (HorizontalPodAutoscaler) to work
• kubernetes-dashboard: basic cluster monitoring (if Lens is not available)
• Loki (scalable) to aggregate and index all logs in the cluster, with retention policy; the logs are searchable in Grafana. Additionally:
  • promtail to aggregate logs
  • Note: Elasticsearch charts (hence ELK) have been deprecated in favor of their licensed ECK; plus Loki is much easier to run and maintain
• kube-prometheus-stack for cluster monitoring with many useful preconfigured cluster Prometheus metrics in Grafana dashboards. Additionally:
  • prometheus-adapter for custom metrics API, e.g. for HPA to scale using custom-defined metrics.
  • prometheus-pushgateway to push application metrics
  • prometheus-blackbox-exporter to probe endpoints for uptime monitoring

bash ./cluster/cluster-addons.sh

Additionally, install Lens for GUI monitoring and access to the cluster. Get a free license to use.

Accessing Dashboards

See more on blog post Setting up Kubernetes Addons.

• Lens
  • just open the app, it will use ~/.kube/config to connect
• Kubernetes Dashboard
  • get token: kubectl -n kubernetes-dashboard create token admin-user
  • run kubectl proxy and visit http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
• Grafana for cluster and logging monitoring
  • data sources include kube-state-metrics, node-exporter, prometheus, and custom-added loki for logs
  • run kubectl port-forward -n monitoring svc/prometheus-grafana 6060:80 and visit http://localhost:6060 to find the preconfigured dashboards
  • (one-time) import this Loki Kubernetes Logs and this Blackbox exporter dashboards
• Prometheus for cluster monitoring
  • run kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 and visit http://localhost:9090

Troubleshoot

• delete pod stuck in terminating state:

  kubectl delete pod --grace-period=0 --force <PODNAME>

• decode secret:

  kubectl get secret <SECRETNAME> -n <NAMESPACE> -o jsonpath="{.data.admin-password}" | base64 --decode ; echo