Skip to content

Commit

Permalink
prombench: Added parca for continous profiling support.
Browse files Browse the repository at this point in the history
Fixes #552

Signed-off-by: bwplotka <[email protected]>
  • Loading branch information
bwplotka committed Aug 30, 2023
1 parent b839cb7 commit 1178793
Show file tree
Hide file tree
Showing 6 changed files with 185 additions and 2 deletions.
1 change: 1 addition & 0 deletions prombench/docs/eks.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ make cluster_resource_apply
- Grafana :: `http://<DOMAIN_NAME>/grafana`
- Prometheus :: `http://<DOMAIN_NAME>/prometheus-meta`
- Logs :: `http://<DOMAIN_NAME>/grafana/explore`
- Profiles :: `http://<DOMAIN_NAME>/profiles`

## Usage

Expand Down
1 change: 1 addition & 0 deletions prombench/docs/gke.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ make cluster_resource_apply
- Grafana :: `http://<DOMAIN_NAME>/grafana`
- Prometheus :: `http://<DOMAIN_NAME>/prometheus-meta`
- Logs :: `http://<DOMAIN_NAME>/grafana/explore`
- Profiles :: `http://<DOMAIN_NAME>/profiles`

## Usage

Expand Down
1 change: 1 addition & 0 deletions prombench/docs/kind.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ export NODE_PORT=$(kubectl get -o jsonpath="{.spec.ports[0].nodePort}" services
echo "Grafana: http://$INTERNAL_IP:$NODE_PORT/grafana"
echo "Prometheus: http://$INTERNAL_IP:$NODE_PORT/prometheus-meta"
echo "Logs: http://$INTERNAL_IP:$NODE_PORT/grafana/explore"
echo "Profiles: http://$INTERNAL_IP:$NODE_PORT/profiles"
```
## Usage

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,12 @@ data:
**Compared versions:** [**`PR-{{ index . "PR_NUMBER" }}`**](http://{{ index . "DOMAIN_NAME" }}/{{ index . "PR_NUMBER" }}/prometheus-pr) and [**`{{ index . "RELEASE" }}`**](http://{{ index . "DOMAIN_NAME" }}/{{ index . "PR_NUMBER" }}/prometheus-release)
After successful deployment, the benchmarking metrics can be viewed at:
After successful deployment, the benchmarking results can be viewed at:
- [Prometheus Meta](http://{{ index . "DOMAIN_NAME" }}/prometheus-meta/graph?g0.expr={namespace%3D"prombench-{{ index . "PR_NUMBER" }}"}&g0.tab=1)
- [Prombench Dashboard](http://{{ index . "DOMAIN_NAME" }}/grafana/d/7gmLoNDmz/prombench?orgId=1&var-pr-number={{ index . "PR_NUMBER" }})
- [Grafana Explorer, Loki logs](http://{{ index . "DOMAIN_NAME" }}/grafana/explore?orgId=1&left=["now-6h","now","loki-meta",{},{"mode":"Logs"},{"ui":[true,true,true,"none"]}])
- [Parca profiles (e.g. in-use memory)](http://{{ index . "DOMAIN_NAME" }}/profiles?expression_a=memory%3Ainuse_space%3Abytes%3Aspace%3Abytes%7Bpr_number%3D%22{{ index . "PR_NUMBER" }}%22%7D&time_selection_a=relative:minute|15
**Other Commands:**
To stop benchmark: `/prombench cancel`
Expand All @@ -50,11 +51,12 @@ data:
**Compared versions:** [**`PR-{{ index . "PR_NUMBER" }}`**](http://{{ index . "DOMAIN_NAME" }}/{{ index . "PR_NUMBER" }}/prometheus-pr) and [**`{{ index . "RELEASE" }}`**](http://{{ index . "DOMAIN_NAME" }}/{{ index . "PR_NUMBER" }}/prometheus-release)
After successful deployment, the benchmarking metrics can be viewed at:
After successful deployment, the benchmarking results can be viewed at:
- [Prometheus Meta](http://{{ index . "DOMAIN_NAME" }}/prometheus-meta/graph?g0.expr={namespace%3D"prombench-{{ index . "PR_NUMBER" }}"}&g0.tab=1)
- [Prombench Dashboard](http://{{ index . "DOMAIN_NAME" }}/grafana/d/7gmLoNDmz/prombench?orgId=1&var-pr-number={{ index . "PR_NUMBER" }})
- [Grafana Exlorer, Loki logs](http://{{ index . "DOMAIN_NAME" }}/grafana/explore?orgId=1&left=["now-6h","now","loki-meta",{},{"mode":"Logs"},{"ui":[true,true,true,"none"]}])
- [Parca profiles (e.g. in-use memory)](http://{{ index . "DOMAIN_NAME" }}/profiles?expression_a=memory%3Ainuse_space%3Abytes%3Aspace%3Abytes%7Bpr_number%3D%22{{ index . "PR_NUMBER" }}%22%7D&time_selection_a=relative:minute|15
**Other Commands:**
To stop benchmark: `/prombench cancel`
Expand Down
31 changes: 31 additions & 0 deletions prombench/manifests/cluster-infra/8a_parca_rbac.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: parca
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: parca
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/proxy
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: parca
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: parca
subjects:
- kind: ServiceAccount
name: parca
namespace: default
147 changes: 147 additions & 0 deletions prombench/manifests/cluster-infra/8b_parca.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: parca
data:
# TODO(bwplotka): Add config reloader, otherwise pod has to manually deleted.
parca.yaml: |-
"object_storage":
"bucket":
"config":
"directory": "/var/lib/parca"
"type": "FILESYSTEM"
scrape_configs:
- job_name: prometheus
scrape_interval: 30s
scrape_timeout: 40s
scheme: http
kubernetes_sd_configs:
- role: endpoints
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- action: keep
source_labels: [__meta_kubernetes_service_label_app]
regex: prometheus|prometheus-meta
- source_labels: [__meta_kubernetes_service_label_prometheus]
target_label: prometheus
# Each prometheus is configured with external path option, so it serves on certain hardcoded path. Adjust it.
- source_labels: [__meta_kubernetes_namespace]
regex: prombench-(.*)
target_label: pr_number
- source_labels: [__profile_path__]
target_label: __init_profile_path
- source_labels: [__meta_kubernetes_service_label_app, __init_profile_path]
regex: prometheus-meta;(.*)
replacement: /prometheus-meta$1
target_label: __profile_path__
- source_labels: [prometheus, pr_number, __init_profile_path]
regex: test-.*;(.*);(.*)
replacement: /$1/prometheus-release$2
target_label: __profile_path__
- source_labels: [prometheus, pr_number, __init_profile_path]
regex: test-pr-.*;(.*);(.*)
replacement: /$1/prometheus-pr$2
target_label: __profile_path__
- regex: __init_profile_path
action: labeldrop
- source_labels: [__address__]
target_label: instance
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: parca
labels:
app: parca
spec:
replicas: 1
selector:
matchLabels:
app: parca
template:
metadata:
labels:
app: parca
spec:
serviceAccountName: parca
securityContext:
fsGroup: 65534
runAsUser: 65534
containers:
- image: ghcr.io/parca-dev/parca:v0.18.0
args:
- /parca
- "--http-address=:7070"
- "--config-path=/etc/parca/parca.yaml"
- "--path-prefix=/profiles"
- "--log-level=info"
- "--cors-allowed-origins=*"
- "--debuginfod-upstream-servers=https://debuginfod.systemtap.org"
- "--debuginfod-http-request-timeout=5m"
livenessProbe:
exec:
command:
- /grpc_health_probe
- -v
- -addr=:7070
initialDelaySeconds: 5
name: parca
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- mountPath: /etc/parca
name: config
- mountPath: /var/lib/parca
name: storage
ports:
- name: parca-web
containerPort: 7070
volumes:
- name: config
configMap:
name: parca
# TODO(bwplotka): Make it persistent at some point.
- name: storage
emptyDir: {}
terminationGracePeriodSeconds: 120
nodeSelector:
node-name: main-node
---
apiVersion: v1
kind: Service
metadata:
name: parca
labels:
app: parca
spec:
type: NodePort
ports:
- name: parca-web
port: 80
targetPort: parca-web
selector:
app: parca
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-parca
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- backend:
service:
name: parca
port:
name: parca-web
path: /profiles
pathType: Prefix

0 comments on commit 1178793

Please sign in to comment.