aro-hcp-keepalived

A Go service that monitors an Azure AKS cluster and automatically redeploys it if the cluster disappears. It polls the AKS cluster's provisioning state via the Azure SDK, and when the cluster is confirmed missing (after a configurable threshold of consecutive NotFound checks), it clones the ARO-HCP repo and runs make personal-dev-env to recreate it. Failed deployments trigger automatic cleanup via make cleanup-entrypoint/Region. Slack notifications are sent for state transitions.

Build & Run

make build        # builds ./aro-hcp-keepalived binary
make run          # builds and runs
make test         # go test ./...
make docker-build # builds container image
make tidy         # go mod tidy
make clean        # removes binary

Environment Variables

Variable	Required	Default	Description
AZURE_SUBSCRIPTION_ID	Yes	-	Azure subscription ID
AZURE_RESOURCE_GROUP	Yes	-	Azure resource group containing the AKS cluster
AKS_CLUSTER_NAME	Yes	-	Name of the AKS cluster to monitor
SLACK_WEBHOOK_URL	No	-	Slack incoming webhook URL
GIT_REPO_URL	No	git@github.com:Azure/ARO-HCP.git	ARO-HCP git repo to clone for deploy
DEPLOY_USER	No	oasis	User for personal-dev-env deploy
GO_BINARY_PATH	No	/opt/homebrew/bin/go	Path to Go binary used during deploy
HEALTHY_INTERVAL	No	1h	Poll interval when cluster is healthy
UNHEALTHY_INTERVAL	No	5m	Poll interval when cluster is unhealthy
PROVISIONING_INTERVAL	No	5m	Poll interval when cluster is provisioning
NOT_FOUND_THRESHOLD	No	5	Consecutive NotFound checks before triggering redeploy
RUN_ONCE	No	false	Exit after one check cycle instead of running continuously

Azure authentication uses azidentity.NewDefaultAzureCredential (supports env vars, managed identity, Azure CLI, etc.).

Architecture

.
├── main.go              # Event loop: polls cluster state, manages transitions, triggers deploy/cleanup
├── pkg/
│   ├── config/          # Environment-based configuration loading
│   ├── monitor/         # AKS cluster state checking via Azure SDK
│   ├── deployer/        # Clones ARO-HCP repo, runs make targets for deploy and cleanup
│   └── notifier/        # Slack webhook notifications for state changes and deploy events
├── Dockerfile
└── Makefile

Key Behaviors

• Cluster states: Succeeded, NotFound, Creating, Deleting, Upgrading, Failed
• NotFound threshold: Requires crossing a configurable threshold before triggering redeploy (avoids false positives)
• Single deploy: Only one deploy runs at a time (mutex-guarded)
• Auto cleanup: Failed deploys automatically run cleanup before allowing retry
• Adaptive polling: Poll interval adapts based on cluster state (healthy=slow, unhealthy/provisioning=fast)

macOS LaunchAgent

A sample plist file (com.hchenxa.aro-hcp-keepalived.plist) is included for running as a macOS LaunchAgent. Copy it to ~/Library/LaunchAgents/, fill in the environment variables, then load it:

cp com.hchenxa.aro-hcp-keepalived.plist ~/Library/LaunchAgents/
launchctl load ~/Library/LaunchAgents/com.hchenxa.aro-hcp-keepalived.plist