[WIP] Add MLflow to the Kubeflow Community Distribution by kramaranya · Pull Request #3582 · kubeflow/community-distribution

kramaranya · 2026-08-17T13:52:55Z

✏️ Summary of Changes

Adds MLflow to the Kubeflow Community Distribution in the kubeflow namespace.

Adds the MLflow Helm chart and Kubeflow overlay configuration.
Enables Kubernetes workspace support with profile-based workspace discovery.
Configures local SQLite backend storage and file-based artifact storage on a PVC.
Adds Kubernetes RBAC for workspace discovery and SubjectAccessReview authorization.
Adds Istio Gateway routing under /mlflow with ingress authorization and destination TLS policy.
Adds network policy, generated upstream resources, installation integration, and manifest assertions.
Aligns MLflow host validation with the configured Istio hosts.
Configures the Kubeflow identity headers used by the SubjectAccessReview authorization mode.
Regenerates the synchronized MLflow resources after the configuration updates.

📦 Dependencies

MLflow Kubernetes integration source at commit d276153b84844c076d92c74519a3c405936220de.
Requires the MLflow integration container image referenced by applications/mlflow/values-kubeflow.yaml.
Uses the existing Kubeflow Istio gateway, OAuth2/Kubeflow identity flow, and profile RBAC aggregation.

🐛 Related Issues

Not applicable.

✅ Contributor Checklist

I have tested these changes with kustomize. See Installation Prerequisites.
All commits are signed-off to satisfy the DCO check.
I have considered adding my company to the adopters page to support Kubeflow and help the community, since I expect help from the community for my issue (see 1. and 2.).

You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-community-distribution.

Signed-off-by: kramaranya <kramaranya15@gmail.com>

github-actions · 2026-08-17T13:53:23Z

Welcome to the Kubeflow Community Distribution Repository

Thanks for opening your first PR. Your contribution means a lot to the Kubeflow community.

Before making more PRs:
Please ensure your PR follows our Contributing Guide.
Please also be aware that many components are synchronized from upstream via the scripts in /scripts.
So in some cases you have to fix the problem in the upstream repositories first, but you can use a PR against kubeflow/community-distribution to test the platform integration.

Community Resources:

CNCF Slack channels: https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels
Community meetings: https://www.kubeflow.org/docs/about/community/

Thanks again for helping to improve Kubeflow.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

google-oss-prow · 2026-08-17T14:11:22Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kimwnasptd for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

applications/mlflow/README.md:59

This command names a file that does not exist; the added test is tests/test_mlflow_manifests.py, so following these synchronization instructions fails immediately.

./tests/mlflow_manifests_test.py

tests/mlflow_install.sh:14

This check only waits for Deployment availability; it never exercises the newly introduced multi-user authentication and Profile authorization path. The comprehensive integration workflow can therefore pass while identity headers or role mappings make MLflow unusable. Add a gateway request with a Profile-bound identity and a negative cross-Profile authorization assertion.

kubectl wait --for=condition=Available deployment/mlflow \
  --namespace kubeflow \
  --timeout=300s

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Suppressed comments (3)

applications/mlflow/values-kubeflow.yaml:45

The platform kubeflow-gateway accepts every host, and the integration environment accesses it through localhost:8080, but this VirtualService exposes /mlflow only for the two example domains. The Dashboard link therefore receives an ingress 404 in the standard port-forward installation before MLflow can apply its own strict host validation. Keep mlflow.allowedHosts restricted, but allow this route on the platform gateway like the other Kubeflow VirtualServices.

    hosts:
      - kubeflow.example.com
      - "*.kubeflow.example.com"

.github/workflows/full_kubeflow_integration_test.yaml:73

This integration step verifies only that the Deployment becomes available; it never requests /mlflow or verifies workspace isolation. Profile discovery, normalized identity headers, SubjectAccessReview authorization, and cross-Profile denial can therefore all be broken while the integration job passes. Add an MLflow verification test using Profile tokens for allowed same-Profile access and denied cross-Profile access, then run it after the gateway port-forward starts.

    - name: Install MLflow
      run: ./tests/mlflow_install.sh

applications/mlflow/README.md:59

This command references a file that does not exist; the added manifest test is tests/test_mlflow_manifests.py. Following the synchronization instructions currently ends with “No such file or directory.”

./tests/mlflow_manifests_test.py

kramaranya · 2026-08-20T12:14:30Z

CI is failing because mlflow-integration 1.6.0 hasn't been released yet

juliusvonkohout · 2026-08-21T07:26:02Z

+and submit Kubernetes SubjectAccessReviews. Callers receive MLflow permissions
+through the standard Kubeflow Profile roles.
+
+## Install


Install -> Installation

juliusvonkohout · 2026-08-21T07:27:50Z

@danish9039 please take a look

juliusvonkohout · 2026-08-21T07:29:35Z

Where is mlflow tested?

juliusvonkohout · 2026-08-21T07:30:40Z

@christian-heusel

We might need to get this upstream into the dashboard repository.

juliusvonkohout · 2026-08-21T07:35:36Z

+- `overlays/kubeflow` adds Profile-aggregated view, edit, and administrator roles.
+
+The deployment uses the `kubeflow` namespace, the `/mlflow` URL prefix, the
+`kubeflow-gateway` Istio gateway, and the identity headers supplied by the


Can I also directly use the service account default-editor token without the headers from the istio request authentication?

All of this must be tested

juliusvonkohout · 2026-08-21T07:42:18Z

@danish9039

juliusvonkohout · 2026-08-21T07:45:23Z

+          port: 15090
+      from:
+        - podSelector: {}
+        - namespaceSelector: {}


why do you add namespaceSelector: {}
Why not just part of kubeflow profile namespaces similar to the other nwtwork policies ?

juliusvonkohout · 2026-08-21T07:50:10Z

+    - kubeflow.example.com
+    - "*.kubeflow.example.com"


This might have to be changed once you add proper tests. Please check how pipelines and other components do it.

juliusvonkohout · 2026-08-21T07:51:12Z

@danish9039 please compare with the other synchronization scripts.

juliusvonkohout · 2026-08-21T07:51:36Z

we also need tests/mlflow_test.sh

juliusvonkohout · 2026-08-21T07:52:12Z

 | KServe | applications/kserve/kserve | [v0.20.0](https://github.com/kserve/kserve/tree/v0.20.0) | 600m | 1200Mi | 0GB |
 | Kubeflow Pipelines | applications/pipeline/upstream | [2.17.0](https://github.com/kubeflow/pipelines/tree/2.17.0/manifests/kustomize) | 970m | 3552Mi | 35GB |
 | Kubeflow Hub | applications/hub/upstream | [v0.3.14](https://github.com/kubeflow/hub/tree/v0.3.14/manifests/kustomize) | 510m | 2112Mi | 20GB |
+| MLflow | applications/mlflow/upstream | [d276153](https://github.com/kubeflow/mlflow-integration/tree/d276153b84844c076d92c74519a3c405936220de/charts/mlflow) | Not measured | Not measured | 2GB |


please add measurements

see https://github.com/kubeflow/community-distribution/blob/master/tests/metrics-server_resource_table.py

juliusvonkohout · 2026-08-21T07:54:57Z

the filename prefix is wrong. it must start with mlflow_

juliusvonkohout · 2026-08-21T07:56:56Z

+MLflow maps workspaces to Kubeflow Profile namespaces and authorizes requests
+through Kubernetes role-based access control. Install the multi-user overlay:
+
+```sh
+./tests/mlflow_install.sh
+```
+
+The example uses SQLite metadata, file-based artifacts, and a 2 GiB persistent
+volume for single-replica evaluation. Configure shared database and artifact
+stores before production use. See
+[`applications/mlflow/README.md`](applications/mlflow/README.md) for the storage,
+authorization, and synchronization details.


Suggested change

MLflow maps workspaces to Kubeflow Profile namespaces and authorizes requests

through Kubernetes role-based access control. Install the multi-user overlay:

```sh

./tests/mlflow_install.sh

```

The example uses SQLite metadata, file-based artifacts, and a 2 GiB persistent

volume for single-replica evaluation. Configure shared database and artifact

stores before production use. See

[`applications/mlflow/README.md`](applications/mlflow/README.md) for the storage,

authorization, and synchronization details.

```sh

./tests/mlflow_install.sh

juliusvonkohout · 2026-08-21T11:27:15Z

And first of all thank you for the PR.

thesuperzapper · 2026-08-25T15:40:27Z

Lets make sure we are following the new KDC projects policy for incuding new projects: https://github.com/kubeflow/community/blob/master/committee-distribution/PROJECTS.md

Signed-off-by: kramaranya <kramaranya15@gmail.com>

juliusvonkohout · 2026-09-02T10:59:08Z

Please do not merge from master it destroys the commit history. Fix it with a proper rebase to master and fix the conflicts.

Add MLflow to the Kubeflow Community Distribution

ed4a056

Signed-off-by: kramaranya <kramaranya15@gmail.com>

Copilot AI balanced review requested due to automatic review settings August 17, 2026 13:52

Copilot started reviewing on behalf of kramaranya August 17, 2026 13:54 View session

Copilot AI reviewed Aug 17, 2026

google-oss-prow Bot added the size/XL label Aug 17, 2026

google-oss-prow Bot requested review from juliusvonkohout and tarekabouzeid August 17, 2026 14:11

juliusvonkohout requested a balanced review from Copilot August 17, 2026 14:19

Copilot started reviewing on behalf of juliusvonkohout August 17, 2026 14:19 View session

Copilot AI reviewed Aug 17, 2026

kramaranya marked this pull request as ready for review August 19, 2026 13:57

kramaranya requested a balanced review from Copilot August 19, 2026 13:58

Copilot started reviewing on behalf of kramaranya August 19, 2026 13:59 View session

Copilot AI reviewed Aug 19, 2026

View reviewed changes

Comment thread applications/mlflow/values-kubeflow.yaml

Comment thread applications/mlflow/values-kubeflow.yaml Outdated

kramaranya added 2 commits August 20, 2026 12:18

fix: align MLflow Kubeflow ingress and identity headers

75b741c

fix: regenerate MLflow Kubeflow resources

4031b2f

kramaranya requested a balanced review from Copilot August 20, 2026 11:31

Copilot started reviewing on behalf of kramaranya August 20, 2026 11:31 View session

Copilot AI reviewed Aug 20, 2026

View reviewed changes

juliusvonkohout reviewed Aug 21, 2026

View reviewed changes

Comment thread .github/workflows/full_kubeflow_integration_test.yaml

juliusvonkohout Aug 21, 2026

Copy link
Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is mlflow tested?

juliusvonkohout reviewed Aug 21, 2026

View reviewed changes

Comment thread .github/workflows/helm-kustomize-comparison.yml

juliusvonkohout Aug 21, 2026

Copy link
Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@danish9039

juliusvonkohout reviewed Aug 21, 2026

View reviewed changes

Comment thread tests/mlflow_install.sh

juliusvonkohout Aug 21, 2026

Copy link
Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we also need tests/mlflow_test.sh

juliusvonkohout reviewed Aug 21, 2026

View reviewed changes

docs: rename MLflow README installation section

f4c979c

test: add MLflow platform e2e through the Kubeflow gateway

35b2235

Signed-off-by: kramaranya <kramaranya15@gmail.com>

google-oss-prow Bot added size/XXL and removed size/XL labels Aug 31, 2026

kramaranya changed the title ~~Add MLflow to the Kubeflow Community Distribution~~ [WIP] Add MLflow to the Kubeflow Community Distribution Aug 31, 2026

google-oss-prow Bot added the do-not-merge/work-in-progress label Aug 31, 2026

Merge origin/master into mlflow-manifests

ea2193b

google-oss-prow Bot added size/XL and removed size/XXL labels Aug 31, 2026

Uh oh!

Conversation

kramaranya commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✏️ Summary of Changes

📦 Dependencies

🐛 Related Issues

✅ Contributor Checklist

Uh oh!

github-actions Bot commented Aug 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

google-oss-prow Bot commented Aug 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

kramaranya commented Aug 20, 2026

Uh oh!

juliusvonkohout Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout commented Aug 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thesuperzapper commented Aug 25, 2026

Uh oh!

juliusvonkohout commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

kramaranya commented Aug 17, 2026 •

edited

Loading

juliusvonkohout Aug 21, 2026 •

edited

Loading

juliusvonkohout Aug 21, 2026 •

edited

Loading

juliusvonkohout commented Aug 21, 2026 •

edited

Loading

juliusvonkohout commented Sep 2, 2026 •

edited

Loading