Kubernetes Manifest Policies

tl;dr: Test Kubernetes manifests for known production issues before deploying to production.

This repo contains Open Policy Agent policies for testing Kubernetes manifests with conftest. The policies close the gap between kubectl apply --dry-run --validate or kubeval, and the Kubernetes API. These policies identify semantic validation issues such as Deployment selector not matching template labels and higher level best practices.

Getting Started

$ git clone [email protected]:ahawkins/opa-kubernetes.git
$ conftest test -p opa-kubernetes/policy your_manifests/*.yml
$ conftest test -p opa-kubernetes/policy -n combined your_manifests/*.yml

Rules

Goals:

1. Identify manifest that may be rejected by the Kubernetes API
2. Identify functional issues in manifests not identified by existing tools
3. Enforce best practices

Metadata

• MTA-01: no namespace set
• MTA-02: mandatory labels
• MTA-03: name well formed
• MTA-04: Non-empty annotations and labels
• MTA-05: String vlaues for annotations and labels

Workloads

Applies to: Deployment, Job, CronJob

• WRK-01: containers set resource requests and limits
• WRK-02: volumeMount matches volume
• WRK-03: volumes are mounted
• WRK-04: container names do not contain invalid characters
• WRK-05: env values are explicit strings

Deployments

• DPL-01: containers set liveness and readiness probes
• DPL-02: selector matches template labels
• DPL-03: liveness and readiness probes match container port
• DPL-04: reusing liveness and readiness probe

Job

• JOB-01: explicit backoffLimit set

ConfigMap

• CFG-01: values are String type

Secrets

• SEC-01: base64 encoded secrets contain valid Base64 encoded keys

HorizontalPodAutoScaler

• HPA-01: Less minimum than maximum replicas

Combined

• CMB-01: container envFrom matches a ConfigMap or Secret in the manifests
• CMB-02: volume from matches ConfigMap or Secret
• CMB-03: Service selector matches a Deployment
• CMB-04: HorizontalPodAutoscaler scaling target matches a Deployment
• CMB-05: Service port matches container port
• CMB-06: HPA managed deployment does not set replicas
• CMB-07: Deployment connected to Service sets a readiness probe

DataDog

Applies to: Deployment, Job, CronJob

• DOG-01: Unified Service Tagging
• DOG-02: APM setup
• DOG-03: Autodiscovery Setup

Sandbox

Applies to sandbox environments

• SBX-01: VPN only access
• SBX-02: Max 2 replicas

Staging

Applies to the staging environment

• STG-01: VPN only access

Developing

Add a new acceptance test in test/ for the rule. Tests take valid data then modify them with yq to break the rule. Tests assert that the relevant conftest test command exists non-zero and outputs the rule number.

$ task test:acceptance