fix(uninstall):remove kubearmor annotations from kubernetes resources

[Prateeknandle]

fixes: #434

1. If annotations are added at the owner level then we remove annotations from deployments, replicasets, statefulsets, daemonsets, job, cronjob.
2. If annotations are added at pod level then all the pods with annotations will be restarted(deleted).
3. Lists all the pods with karmor uninstall that will be restarted when --force flag will be used

• karmor uninstall output:

prateek@bot:~ $ ~/go/bin/karmor uninstall 
ℹ️   Resources not managed by helm/Global Resources are not cleaned up. Please use karmor uninstall --force if you want complete cleanup.
ℹ️   Following pods will get restarted with karmor uninstall --force: 
+-----+--------------------------+-----------+
| NO  |         POD NAME         | NAMESPACE |
+-----+--------------------------+-----------+
|   1 | example-replicaset-6k4z4 | default   |
|   2 | example-replicaset-6m7nq | default   |
|   3 | example-replicaset-727ds | default   |
|   4 | example-replicaset-7bskf | default   |
|   5 | example-replicaset-7ckcn | default   |
|   6 | example-replicaset-9npvb | default   |
|   7 | example-replicaset-bshxd | default   |
|   8 | example-replicaset-cjxm9 | default   |
|   9 | example-replicaset-fmc58 | default   |
|  10 | example-replicaset-ltwl7 | default   |
|  11 | example-replicaset-mpxvc | default   |
|  12 | example-replicaset-n4xsw | default   |
|  13 | example-replicaset-pn4mb | default   |
|  14 | example-replicaset-s774m | default   |
|  15 | example-replicaset-sqzm4 | default   |
|  16 | example-statefulset-0    | default   |
|  17 | example-statefulset-1    | default   |
|  18 | example-statefulset-10   | default   |
|  19 | example-statefulset-11   | default   |
|  20 | example-statefulset-12   | default   |
|  21 | example-statefulset-13   | default   |
|  22 | example-statefulset-14   | default   |
|  23 | example-statefulset-15   | default   |
|  24 | example-statefulset-16   | default   |
|  25 | example-statefulset-17   | default   |
|  26 | example-statefulset-18   | default   |
|  27 | example-statefulset-19   | default   |
|  28 | example-statefulset-2    | default   |
|  29 | example-statefulset-3    | default   |
|  30 | example-statefulset-4    | default   |
|  31 | example-statefulset-5    | default   |
|  32 | example-statefulset-6    | default   |
|  33 | example-statefulset-7    | default   |
|  34 | example-statefulset-8    | default   |
|  35 | example-statefulset-9    | default   |
+-----+--------------------------+-----------+
❌  KubeArmor resources removed
🔄  Checking if KubeArmor pods are stopped...
🔴  Done Checking; all services are stopped!             
⌚️  Termination Time: 5.294239144s 

[DelusionalOptimist]

@Prateeknandle let's add a warning as well that policies and annotations will be removed when running karmor uninstall
cc @daemon1024

[DelusionalOptimist]

LGTM

[DelusionalOptimist]

KubeArmor v1.4.0 stable, BPF LSM node - pods belonging to deployments are also getting restarted with --force and being presented with this warning.

IMO pods which don't have the apparmor annotation should not be restarted?

$ ./karmor uninstall
ℹ️   Resources not managed by helm/Global Resources are not cleaned up. Please use karmor uninstall --force if you want complete cleanup.
ℹ️   Following pods will get restarted with karmor uninstall --force:

+-----+-----------------------------------------+-------------+
| NO  |                POD NAME                 |  NAMESPACE  |
+-----+-----------------------------------------+-------------+
|   1 | nginx-bf5d5cf98-99lcw                   | default     |
|   2 | coredns-576bfc4dc7-55xmp                | kube-system |
|   3 | local-path-provisioner-6795b5f9d8-7vt46 | kube-system |
|   4 | metrics-server-557ff575fb-r67vv         | kube-system |
+-----+-----------------------------------------+-------------+
❌  KubeArmor resources removed
🔄  Checking if KubeArmor pods are stopped...
🔴  Done Checking; all services are stopped!
⌚️  Termination Time: 4.329732048s

kubectl describe pod after uninstall

Name:             nginx-bf5d5cf98-99lcw
Namespace:        default
Priority:         0
Service Account:  default
Node:             kubearmor-dev-next/10.0.2.15
Start Time:       Tue, 06 Aug 2024 11:44:33 +0000
Labels:           app=nginx
                  pod-template-hash=bf5d5cf98
Annotations:      kubearmor-policy: enabled
                  kubearmor-visibility: process,file,network,capabilities
Status:           Running
IP:               10.42.0.57
IPs:
  IP:           10.42.0.57
Controlled By:  ReplicaSet/nginx-bf5d5cf98
Containers:
  nginx:
    Container ID:   docker://4ac64f9c6e035a814d1ff752745c36a479f7cedf4a187df81018e56bbb7ad439
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:6af79ae5de407283dcea8b00d5c37ace95441fd58a8b1d2aa1ed93f5511bb18c
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 06 Aug 2024 11:44:38 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2ph48 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  kube-api-access-2ph48:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

[DelusionalOptimist]

Even with --force some resource are left, just like the current behavior.
cc @daemon1024

$ kubectl api-resources | grep kubearmor
kubearmorconfigs                                 operator.kubearmor.com/v1         true         KubeArmorConfig
kubearmorclusterpolicies            csp          security.kubearmor.com/v1         false        KubeArmorClusterPolicy