Skip to content

Commit 04f845e

Browse files
subrata71claudewyattwalter
authored
fix(helm): make health probes tolerate in-app server restarts (#42571)
## Description Saving some admin settings restarts the server inside the Appsmith container for about 45 seconds. The chart's probes ran every 60 s with Kubernetes' default 1 s timeout and a failure threshold of 3. When three restarts happened back to back, each one could land on a single liveness check. Kubernetes counted three failures in a row and killed the container mid-boot. This happened on nightly-saml on 2026-11-06 during the SSO Sanity run. `/api/v1/health` checks Mongo (1 s timeout) and Redis (3 s timeout) in parallel. A slow but working server could therefore fail a 1 s probe. Fixes https://linear.app/appsmith/issue/APP-16424 ### Changes - `templates/deployment.yaml`: each probe now sets `timeoutSeconds`, which can be overridden. New defaults: | Probe | periodSeconds | timeoutSeconds | failureThreshold | Effect | |---|---|---|---|---| | startup | 60 → 10 | 1 → 5 | 3 → 60 | 600 s to boot | | liveness | 60 → 10 | 1 → 5 | 3 → 18 | restart after 180 s of continuous failure | | readiness | 60 → 10 | 1 → 5 | 3 | unready after 30 s, ready again within 10 s | One successful check resets the liveness count. A 45 s restart causes at most about 5 failures, so even several restarts in a row cannot reach 18. - `values.yaml`: documents the `probes` value, which the template already read through `dig` but no one had documented. Keys: `port`, `api` (the HTTP path), `periodSeconds`, `timeoutSeconds`, `failureThreshold`. The `dig` fallbacks in the template match the new defaults. - `values.schema.json` and `README.md`: regenerated. - Startup gets 600 s because a first boot runs every migration before the server listens. On a remote Atlas cluster this took more than 3.5 minutes (APP-16275). The startup probe stops after its first success, so the longer window costs nothing on a running server. (From #42574 by @wyattwalter, merged into this branch.) - `values.schema.json`: probe `periodSeconds`, `timeoutSeconds` and `failureThreshold` accept an integer of at least 1, or a string of digits with no leading zero. Chart 3.9.1 rendered quoted numbers such as `"60"` as integers, so an upgrade that keeps such an override still passes. - `tests/probes_test.yaml`: new helm-unittest suite. It covers the defaults, a partial override, and schema rejection of bad values. It also checks that the template fallbacks equal the `values.yaml` defaults when `probes` is unset. That matters because `helm upgrade --reuse-values` from 3.9.1 renders through the fallbacks. The tests fail against the old template and schema, and pass with this change. - `scripts/deploy_preview.sh`: deploy previews set the startup probe to 10 s × 180 (30 minutes). The period is set explicitly, so the window holds on published charts that still default to 60 s. Published charts do not render `timeoutSeconds` yet, so previews keep the 1 s timeout until a chart with this change is published. (From #42574.) - `Chart.yaml`: 3.9.1 → 3.9.2. The snapshot is updated to match. Call sites checked: the probes are defined only in `templates/deployment.yaml`, which renders both the StatefulSet and the Deployment. No other template defines a probe. ### Impact on existing instances | State | What happens | |---|---| | Fresh install | Gets the new defaults. Liveness and readiness each call `/api/v1/health` every 10 s (was every 60 s). That is 12 calls a minute instead of 2. Each call is a Mongo ping and a Redis ping. | | Upgrade from default | The pod spec changes, so `helm upgrade` rolls the pod once. After that, behaviour matches a fresh install. | | Upgrade from customized `probes` | Keys the user set still win, because Helm merges user values over chart defaults key by key. Keys the user did not set now take the new defaults. For example, a user who set only `livenessProbe.failureThreshold: 3` now gets a 10 s period and a 5 s timeout, so the restart window shrinks from about 180 s to 30 s. Users who tuned `failureThreshold` for a 60 s period should also set `periodSeconds: 60`. The schema now validates `probes`. Quoted numbers such as `failureThreshold: "5"` still work. Values that are not positive integers, such as `0` or `"ten"`, now fail `helm install`/`upgrade` with a schema error. Before, they reached Kubernetes and failed there. | | Rollback to 3.9.1 | Restores the old probe values (60 s period, 1 s timeout, threshold 3). An override that sets `probes.*.timeoutSeconds` has no effect on 3.9.1, because that template ignores the key. 3.9.1 has no schema for `probes`, so the rollback does not fail. | ### CE → EE sync The EE chart is at 3.9.3, so the hourly sync will conflict on the `Chart.yaml` version and the snapshot file. Resolution: set the EE version to 3.9.4 and regenerate the snapshot with `helm unittest -u`. I made a local copy of that end state on the EE side and checked it: 99/99 unit tests pass, the schema and README regenerate cleanly, and `helm lint` passes. ### Follow-up (not in this PR) The probe port defaults to `80` even when `HTTPContainerPort` is set. A user who sets `HTTPContainerPort` must also set every `probes.*.port`. ## Automation Not run. Cypress runs against the Docker image, not a Helm install, so it cannot exercise this change. The chart's own CI covers it: helm-unittest, the values schema check and helm-docs. ## Testing Run locally from `deploy/helm`: - `helm lint .` passes. - `helm unittest .`: 21 suites, 103 tests pass. - `helm schema …` and `helm-docs --sort-values-order file --dry-run` both match the committed files. - `helm lint . --set probes.livenessProbe.timeoutSeconds=0` is rejected by the schema. - Chart 3.9.2 is not yet published at helm.appsmith.com. - [ ] Client unit tests - [ ] Server unit tests - [ ] Cypress - [ ] Playwright - [ ] Deploy preview - [x] Not applicable ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [x] No 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: Cypress test results --> > [!WARNING] > Tests have not run on the HEAD 788e08a yet > <hr>Fri, 06 Nov 2026 18:56:14 UTC <!-- end of auto-generated comment: Cypress test results --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Wyatt Walter <wyattwalter@gmail.com>
1 parent 3eb3f36 commit 04f845e

8 files changed

Lines changed: 316 additions & 21 deletions

File tree

‎deploy/helm/Chart.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ sources:
1111
- https://github.com/appsmithorg/appsmith
1212
home: https://www.appsmith.com/
1313
icon: https://assets.appsmith.com/appsmith-icon.png
14-
version: 3.9.1
14+
version: 3.9.2
1515
dependencies:
1616
- condition: redis.enabled
1717
name: redis

‎deploy/helm/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,9 @@ for prerequisites, step-by-step setup, and platform-specific instructions (EKS,
4747
| customCAcert | string | `nil` | Custom CA certificates to trust at runtime (map of filename to PEM content) |
4848
| resources.limits | object | `{}` | Resource limits for the Appsmith container |
4949
| resources.requests | object | `{"cpu":"500m","memory":"3000Mi"}` | Resource requests for the Appsmith container |
50+
| probes.startupProbe | object | `{"api":"/api/v1/health","failureThreshold":60,"periodSeconds":10,"port":"80","timeoutSeconds":5}` | Startup probe. Liveness and readiness checks start only after it succeeds. Defaults allow 600 seconds to boot. |
51+
| probes.livenessProbe | object | `{"api":"/api/v1/health","failureThreshold":18,"periodSeconds":10,"port":"80","timeoutSeconds":5}` | Liveness probe. Kubernetes restarts the container when it fails failureThreshold times in a row. Defaults tolerate 180 seconds of continuous failure. |
52+
| probes.readinessProbe | object | `{"api":"/api/v1/health","failureThreshold":3,"periodSeconds":10,"port":"80","timeoutSeconds":5}` | Readiness probe. Kubernetes stops sending traffic to the pod while it fails. Defaults mark the pod unready after 30 seconds of failure. |
5053
| workload | object | `{"kind":"StatefulSet"}` | Select workload resource type: Deployment or StatefulSet |
5154
| replicas | int | `1` | Number of replicas when autoscaling is disabled |
5255
| autoscaling.enabled | bool | `false` | Enable the HorizontalPodAutoscaler |

‎deploy/helm/templates/deployment.yaml‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -140,20 +140,23 @@ spec:
140140
httpGet:
141141
port: {{ dig "startupProbe" "port" "80" $probes }}
142142
path: {{ dig "startupProbe" "api" "/api/v1/health" $probes }}
143-
failureThreshold: {{ dig "startupProbe" "failureThreshold" 3 $probes }}
144-
periodSeconds: {{ dig "startupProbe" "periodSeconds" 60 $probes }}
143+
failureThreshold: {{ dig "startupProbe" "failureThreshold" 60 $probes }}
144+
periodSeconds: {{ dig "startupProbe" "periodSeconds" 10 $probes }}
145+
timeoutSeconds: {{ dig "startupProbe" "timeoutSeconds" 5 $probes }}
145146
livenessProbe:
146147
httpGet:
147148
port: {{ dig "livenessProbe" "port" "80" $probes }}
148149
path: {{ dig "livenessProbe" "api" "/api/v1/health" $probes }}
149-
failureThreshold: {{ dig "livenessProbe" "failureThreshold" 3 $probes }}
150-
periodSeconds: {{ dig "livenessProbe" "periodSeconds" 60 $probes }}
150+
failureThreshold: {{ dig "livenessProbe" "failureThreshold" 18 $probes }}
151+
periodSeconds: {{ dig "livenessProbe" "periodSeconds" 10 $probes }}
152+
timeoutSeconds: {{ dig "livenessProbe" "timeoutSeconds" 5 $probes }}
151153
readinessProbe:
152154
httpGet:
153155
port: {{ dig "readinessProbe" "port" "80" $probes }}
154156
path: {{ dig "readinessProbe" "api" "/api/v1/health" $probes }}
155157
failureThreshold: {{ dig "readinessProbe" "failureThreshold" 3 $probes }}
156-
periodSeconds: {{ dig "readinessProbe" "periodSeconds" 60 $probes }}
158+
periodSeconds: {{ dig "readinessProbe" "periodSeconds" 10 $probes }}
159+
timeoutSeconds: {{ dig "readinessProbe" "timeoutSeconds" 5 $probes }}
157160
resources:
158161
{{- toYaml .Values.resources | nindent 12 }}
159162
volumeMounts:

‎deploy/helm/tests/__snapshot__/defaults_snapshot_test.yaml.snap‎

Lines changed: 18 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
app.kubernetes.io/instance: RELEASE-NAME
2525
app.kubernetes.io/managed-by: Helm
2626
app.kubernetes.io/name: appsmith
27-
appsmith.sh/chart: appsmith-3.9.1
27+
appsmith.sh/chart: appsmith-3.9.2
2828
name: RELEASE-NAME-appsmith
2929
namespace: NAMESPACE
3030
3: |
@@ -35,7 +35,7 @@
3535
app.kubernetes.io/instance: RELEASE-NAME
3636
app.kubernetes.io/managed-by: Helm
3737
app.kubernetes.io/name: appsmith
38-
appsmith.sh/chart: appsmith-3.9.1
38+
appsmith.sh/chart: appsmith-3.9.2
3939
name: RELEASE-NAME-appsmith
4040
namespace: NAMESPACE
4141
spec:
@@ -72,11 +72,12 @@
7272
image: index.docker.io/appsmith/appsmith-ee:latest
7373
imagePullPolicy: IfNotPresent
7474
livenessProbe:
75-
failureThreshold: 3
75+
failureThreshold: 18
7676
httpGet:
7777
path: /api/v1/health
7878
port: 80
79-
periodSeconds: 60
79+
periodSeconds: 10
80+
timeoutSeconds: 5
8081
name: appsmith
8182
ports:
8283
- containerPort: 80
@@ -93,19 +94,21 @@
9394
httpGet:
9495
path: /api/v1/health
9596
port: 80
96-
periodSeconds: 60
97+
periodSeconds: 10
98+
timeoutSeconds: 5
9799
resources:
98100
limits: {}
99101
requests:
100102
cpu: 500m
101103
memory: 3000Mi
102104
securityContext: {}
103105
startupProbe:
104-
failureThreshold: 3
106+
failureThreshold: 60
105107
httpGet:
106108
path: /api/v1/health
107109
port: 80
108-
periodSeconds: 60
110+
periodSeconds: 10
111+
timeoutSeconds: 5
109112
volumeMounts:
110113
- mountPath: /appsmith-stacks
111114
name: data
@@ -154,7 +157,7 @@
154157
app.kubernetes.io/instance: RELEASE-NAME
155158
app.kubernetes.io/managed-by: Helm
156159
app.kubernetes.io/name: appsmith
157-
appsmith.sh/chart: appsmith-3.9.1
160+
appsmith.sh/chart: appsmith-3.9.2
158161
name: RELEASE-NAME-appsmith-headless
159162
namespace: NAMESPACE
160163
spec:
@@ -185,7 +188,7 @@
185188
app.kubernetes.io/instance: RELEASE-NAME
186189
app.kubernetes.io/managed-by: Helm
187190
app.kubernetes.io/name: appsmith
188-
appsmith.sh/chart: appsmith-3.9.1
191+
appsmith.sh/chart: appsmith-3.9.2
189192
name: RELEASE-NAME-appsmith-redis-password-init
190193
namespace: NAMESPACE
191194
6: |
@@ -200,7 +203,7 @@
200203
app.kubernetes.io/instance: RELEASE-NAME
201204
app.kubernetes.io/managed-by: Helm
202205
app.kubernetes.io/name: appsmith
203-
appsmith.sh/chart: appsmith-3.9.1
206+
appsmith.sh/chart: appsmith-3.9.2
204207
name: RELEASE-NAME-appsmith-redis-password-init
205208
namespace: NAMESPACE
206209
rules:
@@ -230,7 +233,7 @@
230233
app.kubernetes.io/instance: RELEASE-NAME
231234
app.kubernetes.io/managed-by: Helm
232235
app.kubernetes.io/name: appsmith
233-
appsmith.sh/chart: appsmith-3.9.1
236+
appsmith.sh/chart: appsmith-3.9.2
234237
name: RELEASE-NAME-appsmith-redis-password-init
235238
namespace: NAMESPACE
236239
roleRef:
@@ -253,7 +256,7 @@
253256
app.kubernetes.io/instance: RELEASE-NAME
254257
app.kubernetes.io/managed-by: Helm
255258
app.kubernetes.io/name: appsmith
256-
appsmith.sh/chart: appsmith-3.9.1
259+
appsmith.sh/chart: appsmith-3.9.2
257260
name: RELEASE-NAME-appsmith-redis-password-init
258261
namespace: NAMESPACE
259262
spec:
@@ -264,7 +267,7 @@
264267
app.kubernetes.io/instance: RELEASE-NAME
265268
app.kubernetes.io/managed-by: Helm
266269
app.kubernetes.io/name: appsmith
267-
appsmith.sh/chart: appsmith-3.9.1
270+
appsmith.sh/chart: appsmith-3.9.2
268271
spec:
269272
containers:
270273
- command:
@@ -317,7 +320,7 @@
317320
app.kubernetes.io/instance: RELEASE-NAME
318321
app.kubernetes.io/managed-by: Helm
319322
app.kubernetes.io/name: appsmith
320-
appsmith.sh/chart: appsmith-3.9.1
323+
appsmith.sh/chart: appsmith-3.9.2
321324
name: RELEASE-NAME-appsmith
322325
namespace: NAMESPACE
323326
spec:
@@ -338,7 +341,7 @@
338341
app.kubernetes.io/instance: RELEASE-NAME
339342
app.kubernetes.io/managed-by: Helm
340343
app.kubernetes.io/name: appsmith
341-
appsmith.sh/chart: appsmith-3.9.1
344+
appsmith.sh/chart: appsmith-3.9.2
342345
name: RELEASE-NAME-appsmith
343346
namespace: NAMESPACE
344347
secrets:

‎deploy/helm/tests/probes_test.yaml‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Tests for the startup, liveness and readiness probes on the Appsmith container.
2+
suite: health probes
3+
templates:
4+
- deployment.yaml
5+
tests:
6+
- name: default probes tolerate in-app server restarts
7+
asserts:
8+
- equal:
9+
path: spec.template.spec.containers[0].startupProbe
10+
value: &defaultStartupProbe
11+
httpGet:
12+
port: 80
13+
path: /api/v1/health
14+
periodSeconds: 10
15+
timeoutSeconds: 5
16+
failureThreshold: 60
17+
- equal:
18+
path: spec.template.spec.containers[0].livenessProbe
19+
value: &defaultLivenessProbe
20+
httpGet:
21+
port: 80
22+
path: /api/v1/health
23+
periodSeconds: 10
24+
timeoutSeconds: 5
25+
failureThreshold: 18
26+
- equal:
27+
path: spec.template.spec.containers[0].readinessProbe
28+
value: &defaultReadinessProbe
29+
httpGet:
30+
port: 80
31+
path: /api/v1/health
32+
periodSeconds: 10
33+
timeoutSeconds: 5
34+
failureThreshold: 3
35+
36+
# `helm upgrade --reuse-values` from a chart without `probes` in values.yaml renders
37+
# without these values, so the template fallbacks must match the values.yaml defaults.
38+
- name: template fallbacks match the values.yaml defaults when probes is unset
39+
set:
40+
probes: null
41+
asserts:
42+
- equal:
43+
path: spec.template.spec.containers[0].startupProbe
44+
value: *defaultStartupProbe
45+
- equal:
46+
path: spec.template.spec.containers[0].livenessProbe
47+
value: *defaultLivenessProbe
48+
- equal:
49+
path: spec.template.spec.containers[0].readinessProbe
50+
value: *defaultReadinessProbe
51+
52+
- name: a partial override changes only the keys it sets
53+
set:
54+
probes:
55+
livenessProbe:
56+
timeoutSeconds: 15
57+
port: 8080
58+
asserts:
59+
- equal:
60+
path: spec.template.spec.containers[0].livenessProbe
61+
value:
62+
httpGet:
63+
port: 8080
64+
path: /api/v1/health
65+
periodSeconds: 10
66+
timeoutSeconds: 15
67+
failureThreshold: 18
68+
- equal:
69+
path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds
70+
value: 5
71+
72+
- name: rejects a probe timeoutSeconds below 1
73+
set:
74+
probes:
75+
readinessProbe:
76+
timeoutSeconds: 0
77+
asserts:
78+
- failedTemplate: {}
79+
80+
# Chart 3.9.1 rendered quoted numbers from probes overrides as integers.
81+
# Upgrades that keep those values must still pass the schema.
82+
- name: accepts a quoted positive number from an older override
83+
set:
84+
probes:
85+
livenessProbe:
86+
periodSeconds: "60"
87+
failureThreshold: "3"
88+
asserts:
89+
- equal:
90+
path: spec.template.spec.containers[0].livenessProbe.periodSeconds
91+
value: 60
92+
- equal:
93+
path: spec.template.spec.containers[0].livenessProbe.failureThreshold
94+
value: 3
95+
96+
- name: rejects a quoted probe number that is not a positive integer
97+
set:
98+
probes:
99+
startupProbe:
100+
failureThreshold: "0"
101+
asserts:
102+
- failedTemplate: {}
103+
104+
- name: rejects a probe number that is not numeric
105+
set:
106+
probes:
107+
startupProbe:
108+
periodSeconds: "ten"
109+
asserts:
110+
- failedTemplate: {}

0 commit comments

Comments
 (0)