Support namespaced TrainingRuntime in the SDK

[kramaranya]

What you would like to be added?

We want the SDK to work with both cluster-scoped and namespaced runtimes, not just ClusterTrainingRuntime. Users should be able to list and select a TrainingRuntime from their namespace.

Why is this needed?

Right now, users can only reference ClusterTrainingRuntime for their TrainJob via the SDK. Users who need TrainingRuntime per team or per project in their own namespaces can’t use the SDK directly, they have to use YAMLs instead.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[kramaranya]

/remove-lifecycle stale
/remove-label lifecycle/needs-triage
/good-first-issue

cc @kubeflow/kubeflow-sdk-team @abhijeet-dhumal

[google-oss-prow]

@kramaranya:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/remove-lifecycle stale
/remove-label lifecycle/needs-triage
/good-first-issue

cc @kubeflow/kubeflow-sdk-team @abhijeet-dhumal

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[abhijeet-dhumal]

Thanks @kramaranya for raising issue on my behalf 🙌

Just to add in description : The namespace abstraction based TrainingRuntime allows us to specify all resources (volumes, configmaps and training images) for that particular namespace..

For example :
In kubernetes the teams such as Data-Pre-processing/Model-training/Serving/monitoring/inferencing can use separate namespaces for their work, and the idea behind using namespace is just to get the isolation for all namespace scoped resources which other teams mentioned in ClusterTrainingRuntime might not have access to.

Using ClusterTrainingRuntime for each TrainJob (current situation) in such scenarios will be impossible.

The fix for this issue will allow data scientists to access not just clustertrainingruntime but also the TrainingRuntimes once authenticated in the namespace via TrainingClient :

trainer_client = TrainerClient(backend_config= KubernetesBackendConfig(namespace="<test-namespace>", client_configuration=api_client.configuration))

[shaikmoeed]

Hi, I’d like to pick this up as a first contribution. Could you please clarify what steps are needed to reproduce the issue and what the expected outcome / definition of done would be?

[abhijeet-dhumal]

Hii @shaikmoeed, Thanks for picking this up !

To reproduce this issue try initialising TrainerClient like this and then try listing TrainingRuntimes available :

from kubeflow.trainer import TrainerClient
from kubeflow.trainer.backends.kubernetes.types import KubernetesBackendConfig
from kubernetes import client

api_server = "<api-server-url>"
token = "<auth-token>"

configuration = client.Configuration()
configuration.host = api_server
configuration.api_key = {"authorization": f"Bearer {token}"}

# Un-comment if your cluster API server uses a self-signed certificate or an un-trusted CA
configuration.verify_ssl = False

api_client = client.ApiClient(configuration)
trainer_client = TrainerClient(backend_config= KubernetesBackendConfig(namespace="<test-namespace>", client_configuration=api_client.configuration))

for r in trainer_client.list_runtimes():
    print(f"Runtime:{r}")

Usually In Kubeflow Trainer V2 we have 2 types of runtimes :

1. ClusterTrainingRuntime (Cluster scoped)
2. TrainingRuntimes (Namespace scoped)

By default the trainer_client.list/get_runtimes() methods will fetch the available cluster scoped ClusterTrainingRuntimes .. but won't be able to fetch the namespace scoped Training Runtimes..
We need to make SDK TrainerClient methods to handle both these type of runtimes..

I hope this much info is enough to start with..

cc: @szaher @kramaranya

[shaikmoeed]

Thanks for detailing @abhijeet-dhumal! Looks like we need to list_namespaced_custom_object along side list_cluster_custom_object to get the TrainingRuntime of the given/current namespace(assuming SDK code is running inside k8s echo system) and append result accordingly?

[abhijeet-dhumal]

@shaikmoeed Yeah I would suggest to validate API support first for Trainer V2 and then need to think about while Creating a trainjob how someone can utilise namespace scoped TrainingRuntime to be utilised for runtime configuration

[NarayanaSabari]

Hi @kramaranya @abhijeet-dhumal 👋

I'd like to work on this issue and contribute to adding support for namespaced TrainingRuntime in the SDK.

Based on my analysis of the codebase and the discussion above, I understand that we need to:

• Add constants for namespace-scoped TrainingRuntime (similar to existing ClusterTrainingRuntime constants)
• Update list_runtimes() to fetch both cluster-scoped and namespace-scoped runtimes and merge the results
• Modify get_runtime() to check both cluster and namespace scopes (with fallback logic)
• Update the Runtime dataclass to track scope information (cluster vs namespace)
• Ensure train() method works seamlessly with namespace-scoped runtimes
• Add comprehensive tests for both runtime types

My proposed approach is to:

• First try fetching from cluster scope, then fall back to namespace scope in get_runtime()
• Return merged results from both scopes in list_runtimes())
• Add a scope field to the Runtime type to track whether it's cluster or namespace scoped

Questions:

• Should namespace-scoped runtimes take precedence over cluster-scoped ones if there's a name collision, or should we return both with scope differentiation?
• Are there any specific edge cases or backward compatibility concerns I should be aware of?
• Would you prefer the scope to be explicitly visible to users (e.g., in the Runtime object), or should it be handled transparently?

Please let me know if my understanding is correct or if there are any specific implementation details you'd like me to follow. Happy to discuss further!

[shaikmoeed]

@kramaranya 👋, I have created a PR. Could you please guide me on the review process? I noticed that the /ok-to-test label has been added, but the CI tests don’t seem to be running. Is there anything I need to do to trigger them?