test: enforce restricted PSS for CI user namespace - #3487
Conversation
|
Welcome to the Kubeflow Manifests Repository Thanks for opening your first PR. Your contribution means a lot to the Kubeflow community. Before making more PRs: Community Resources:
Thanks again for helping to improve Kubeflow. |
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR updates CI to validate/enforce Kubernetes Pod Security Standards (PSS) settings for Kubeflow profile namespaces, and wires the related test scripts into multiple GitHub Actions path filters so relevant workflows run when these scripts change.
Changes:
- Update
kubeflow_profile_install.shto label the Kubeflow user namespace withpod-security.kubernetes.io/enforce=restrictedandenforce-version=latest, and assert the labels are applied. - Add
tests/kubeflow_profile_install.shandtests/PSS_enable.shto workflowon.pull_request.pathsfilters across several test workflows. - Broaden some workflow path filters (e.g.,
tests/pipeline*,tests/katib*) to cover more related changes.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/kubeflow_profile_install.sh | Labels the profile namespace for PSS restricted + adds assertions for applied labels. |
| .github/workflows/workspaces_pipeline_run_test.yaml | Triggers workflow on profile/PSS test script changes. |
| .github/workflows/volumes_web_application_test.yaml | Triggers workflow on profile install script changes. |
| .github/workflows/training_operator_test.yaml | Triggers workflow on profile/PSS test script changes. |
| .github/workflows/trainer_test.yaml | Triggers workflow on profile/PSS test script changes. |
| .github/workflows/pipeline_test.yaml | Broadens pipeline path trigger + triggers on profile/PSS test script changes. |
| .github/workflows/pipeline_run_from_notebook.yaml | Adds pipeline wildcard + profile/PSS script triggers. |
| .github/workflows/kserve_test.yaml | Triggers workflow on profile/PSS test script changes. |
| .github/workflows/kserve_models_web_application_test.yaml | Triggers workflow on profile install script changes. |
| .github/workflows/katib_test.yaml | Broadens katib path trigger + triggers on profile/PSS test script changes. |
| .github/workflows/istio_validation.yaml | Triggers workflow on profile/PSS test script changes. |
| .github/workflows/dex_oauth2-proxy_test.yaml | Triggers workflow on profile/PSS test script changes. |
| kubectl get pods -n kubeflow-system -l app.kubernetes.io/name=trainer | ||
| kubectl get clustertrainingruntimes torch-distributed | ||
|
|
||
| kubectl patch clustertrainingruntime torch-distributed --type=json -p='[ |
There was a problem hiding this comment.
this must also be a proper overlay and maybe upstreamed.
There was a problem hiding this comment.
There might already bea recent Pr for this.
There was a problem hiding this comment.
Moved the torch-distributed runtime changes into applications/trainer/overlays/runtimes-restricted; upstream follow-up is noted separately.
There was a problem hiding this comment.
Checked kubeflow/trainer#3066 ; it helps with override support, but this PR still needs the local runtime default overlay.
There was a problem hiding this comment.
Split to kubeflow/trainer#3702; #3487 keeps the CI-only runtime overlay until that lands.
There was a problem hiding this comment.
please a proper overlay. this has to survive the next release and then we can already remove it. Please raise a follow up PR to remove it such that i can merge it afte rthe 26.03.1 release.
There was a problem hiding this comment.
Moved this into applications/training-operator/overlays/kubeflow-restricted-pss and noted the follow-up removal after 26.03.1.
There was a problem hiding this comment.
Kept as a proper overlay and logged the removal follow-up after 26.03.1.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (2)
tests/workspaces_pipeline_run_test.sh:1
- This JSONPatch adds nested fields under
/spec/podTemplate/securityContext/...and/spec/podTemplate/containerSecurityContext/.... JSONPatchaddwill fail if the parent object (e.g.,/spec/podTemplate/securityContext) does not already exist. To make this robust across different base WorkspaceKind manifests, patch the parent objects in one operation (e.g., add/merge the fullsecurityContext/containerSecurityContextobjects) or use a strategic/merge patch (--type=merge) targeting the parent keys.
tests/kserve_test.sh:1 - The polling logic treats any response other than 403 as success (twice). This can still false-positive on 404/503 while routes/backends are not ready—exactly the failure mode mentioned in the comment above the loop. Prefer checking for the expected success code (typically 200) or at least explicitly excluding 404/503 (and possibly 000) from counting toward
STABLE_POLL_COUNT, to prevent passing the policy wait while the service is still unreachable.
#!/bin/bash
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
`Remove legacy Training Operator (kubeflow#3540)` deleted `applications/training-operator/upstream`, so `applications/training-operator/overlays/kubeflow-restricted-pss` no longer builds: accumulating resources from '../../upstream/overlays/kubeflow': evalsymlink failure ... no such file or directory The matching `tests/training_operator_*.sh`, `tests/training_operator_job.yaml` and `.github/workflows/training_operator_test.yaml` were removed by the same upstream change while rebasing, so nothing references this overlay any more. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bm4s3z25BrfMky6b4R2ej6 Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
`applications/pipeline/overlays` is referenced from `example/kustomization.yaml:69`, so setting `defaultSecurityContextRunAsUser`, `defaultSecurityContextRunAsGroup` and `defaultSecurityContextRunAsNonRoot` there changed every regular installation, not only GitHub Actions. Restore the upstream empty values in `applications/pipeline/overlays` and move the restricted Pod Security Standards defaults into a new continuous-integration only overlay `applications/pipeline/restricted-pss`, selected by `tests/pipelines_install.sh` when `GITHUB_ACTIONS` is set. This matches how `tests/katib_install.sh` and `tests/trainer_install.sh` already gate their restricted overlays. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bm4s3z25BrfMky6b4R2ej6 Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
`black` 26.5.1 wraps the `FileNotFoundError` warning in `tests/pipeline_run_and_wait_kubeflow.py`, which failed the `format_python_files` check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bm4s3z25BrfMky6b4R2ej6 Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
1badb96 to
8f216ba
Compare
|
Rebased onto
Verified locally on On the Katib overlay question ( |
8f216ba to
f25ebfb
Compare
The KServe overlay on master already sets the restricted security context on the default ClusterStorageContainer since kubeflow#3515, without runAsUser. Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
f25ebfb to
8c4d0d4
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Three Pipelines workflows omit the new restricted overlay from path filters, and several introduced names violate mandatory repository guidance.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (4)
tests/pipeline_run_and_wait_kubeflow.py:119
- The newly introduced exception variable
eviolates the repository requirement for explicitly long, expressive names inAGENTS.md:7. Rename it toexception.
except Exception as e:
tests/pipeline_run_and_wait_kubeflow.py:127
- The newly introduced exception variable
eviolates the repository requirement for explicitly long, expressive names inAGENTS.md:7. Rename it toexception.
except Exception as e:
tests/pipeline_run_and_wait_kubeflow.py:174
- The newly introduced exception variable
eviolates the repository requirement for explicitly long, expressive names inAGENTS.md:7. Rename it toexception.
except Exception as e:
tests/pipeline_run_and_wait_kubeflow.py:183
- The newly introduced exception variable
eviolates the repository requirement for explicitly long, expressive names inAGENTS.md:7. Rename it toexception.
except Exception as e:
- Files reviewed: 28/28 changed files
- Comments generated: 5
- Review effort level: Balanced
| - common/cert-manager/** | ||
| - common/oauth2-proxy/** | ||
| - common/istio*/** | ||
| - tests/pipeline* |
| - tests/pipeline_v1_test.py | ||
| - tests/pipeline_v2_test.py | ||
| - experimental/security/PSS/* | ||
| - tests/pipeline* |
| - tests/dashboard_install.sh | ||
| - tests/multi_tenancy_install.sh | ||
| - tests/pipelines_install.sh | ||
| - tests/kubeflow_profile_install.sh |
| PSS_ENFORCE=baseline | ||
| if [[ "${GITHUB_ACTIONS:-false}" == "true" ]]; then | ||
| PSS_ENFORCE=restricted | ||
| fi | ||
|
|
||
| kubectl label namespace "$KF_PROFILE" \ | ||
| "pod-security.kubernetes.io/enforce=${PSS_ENFORCE}" \ | ||
| --overwrite | ||
|
|
||
| if [[ "$PSS_ENFORCE" == "restricted" ]]; then | ||
| kubectl label namespace "$KF_PROFILE" \ | ||
| pod-security.kubernetes.io/enforce-version=latest \ | ||
| --overwrite | ||
| fi | ||
|
|
||
| kubectl get namespace "$KF_PROFILE" \ | ||
| -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}' | grep -qx "$PSS_ENFORCE" | ||
|
|
||
| if [[ "$PSS_ENFORCE" == "restricted" ]]; then | ||
| kubectl get namespace "$KF_PROFILE" \ | ||
| -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce-version}' | grep -qx latest |
| except Exception as e: | ||
| logger.warning( | ||
| "Unable to run diagnostic command. Exception: %s: %s", | ||
| e.__class__.__name__, | ||
| str(e), | ||
| ) |
Summary of Changes
This pull request makes continuous integration validate Kubeflow user workloads under Kubernetes Pod Security Standards
restricted.kubeflow-user-example-comtopod-security.kubernetes.io/enforce=restrictedafter the Profile Controller creates it, only whenGITHUB_ACTIONSis set.kubectl patchor generated temporary-overlay logic.Scope: no change to the regular installation
Every restricted Pod Security Standards default in this pull request is selected only when
GITHUB_ACTIONSis set:tests/katib_install.shtests/trainer_install.shtests/pipelines_install.shtests/profile_controller_install.shAn earlier revision of this pull request set
defaultSecurityContextRunAsUser,defaultSecurityContextRunAsGroupanddefaultSecurityContextRunAsNonRootdirectly inapplications/pipeline/overlays/pipeline-install-config-securitycontext.yaml. That path is referenced fromexample/kustomization.yaml, so it changed every regular installation and not only GitHub Actions.That file is now byte-identical to
masteragain, and the restricted defaults live in a new continuous-integration only overlayapplications/pipeline/restricted-pss, selected bytests/pipelines_install.sh. This matches howtests/katib_install.shandtests/trainer_install.shalready gate their overlays.Verified by rendering:
Rebase onto
masterThis branch is rebased onto
masteratd8504b91(2026-08-28).mastermoved under the pull request and made three of its temporary continuous-integration patches redundant, so they are gone:ClusterStorageContainerwith the restricted security context inapplications/kserve/kserve/kustomization.yaml#L24-L43, withoutrunAsUser, because the image declares OCI user1000. The GitHub-Actions-onlykubectl patch clusterstoragecontainerintests/kserve_test.shis removed. The downstream pull request kserve: set storage initializer security context #3527 is superseded by the same change; fix(storage): set restricted initializer security context kserve/kserve#5855 remains the upstream fix.v2.0.0-beta.0) addedtests/workspacekind.test.yamlwithseccompProfile,runAsNonRoot, dropped capabilities andfsGroup, and no fixedrunAsUser.tests/workspaces_pipeline_run_test.shonmasterapplies that file, so thepatch_workspacekind_for_restricted_pssfunction and itsrunAsUser: 1000are dropped. This also resolves the review comment that a fixed user identifier should not be hard-coded there.masteralready waits180sper Notebooks deployment intests/notebooks_install.shwith a state dump on failure, so this pull request no longer touches that file.Earlier rebase notes still apply: the legacy Training Operator overlay was dropped after #3540, the failure-log collection steps live in #3560, and
.github/workflows/kserve_models_web_application_test.yamlwas deleted upstream.tests/trainer_test.shkeepsmaster'skubeflow-trainer-api==2.3.0pin plus thetorch-distributedruntime dump.The change set is now 28 files,
+635 / -104.Notes
This is intentionally continuous-integration and test focused. It does not change the default customer Profile namespace policy.
The new application changes are local Kubeflow manifests overlays. They do not edit synchronized
applications/*/upstreamfolders orcommon/*manifests.A follow-up note is still tracked for keeping the copied Katib continuous-integration configuration in sync during future Katib synchronizations.
Related
Follow-up to #3444.
Validation
Current head is
8c4d0d43.Local validation on
8c4d0d43:kustomize build examplerenders empty Pipelines identity defaults, as shown above.kustomize buildsucceeds forapplications/pipeline/overlays,applications/pipeline/restricted-pss,applications/katib/overlays/securityandapplications/trainer/overlays/runtimes-restricted.git diff master -- applications/pipeline/overlays tests/notebooks_install.sh tests/workspaces_pipeline_run_test.shis empty.black --check tests/*.pypasses withblack26.5.1, the version continuous integration uses.applications/*/upstream/**file is modified.Contributor Checklist
Related Issues
Tracking issue:
#3562