istio-installation: mesh-wide egress prune Sidecar blocks Model Registry UI -> per-profile server (HTTP 403)

[NeutryFD]

Summary

The mesh-wide Sidecar named default in common/istio/istio-install/base/sidecar-prune-egress.yaml (no workloadSelector -> applies to every sidecar in the cluster) restricts egress to:

hosts:
- "./*"
- "kubeflow/*"
- "kubeflow-system/*"
- "knative-serving/*"
- "knative-eventing/*"
- "istio-system/*"

This appears to break the documented multi-user Model Registry installation pattern, where the Model Registry UI lives once in kubeflow but the server + DB are deployed per profile namespace. The UI->server call crosses namespaces, gets no egress cluster, falls back to plaintext passthrough, loses Istio identity (principal: -), and the server's AuthorizationPolicy (source.namespaces: ["kubeflow"]) can never match -> HTTP 403.

The documented pattern

applications/hub/upstream/README.md:

"Kubeflow Central Dashboard uses profiles to handle user namespaces and permissions. You will need to deploy Model Registry into a profile namespace."

PROFILE_NAME=<your-profile>
kubectl apply -k overlays/db -n $PROFILE_NAME      # DB, per profile
kubectl apply -k options/istio -n $PROFILE_NAME    # server, per profile
kubectl apply -k options/ui/overlays/istio         # UI, once, namespace kubeflow

options/ui/overlays/istio/kustomization.yaml sets namespace: kubeflow, and the UI VirtualService routes to model-registry-ui-service.kubeflow.svc.cluster.local. So the UI (in kubeflow) must reach model-registry-service.<PROFILE_NAMESPACE> - a cross-namespace hop that the mesh-wide prune excludes (no <profile-namespace>/* in the allowed list).

Symptom (observed in production)

• The UI pod's Envoy has no cluster for model-registry-service.<profile>.svc.cluster.local; the request is handled via passthrough plaintext. The server's Envoy access logs show principal: - (no source.principal) and rbac_access_denied_matched_policy[none] -> HTTP 403.
• The server pod's sidecar sees ~2367 outbound clusters; the UI proxy sees only the allowed namespaces (kubeflow, kubeflow-system, knative-serving, knative-eventing, istio-system) and zero profile namespaces (e.g. hpc-user-*).
• The UI->server call works end-to-end when both are colocated in the same namespace (./* keeps mTLS), confirming egress scope is the only blocker.

Context

sidecar-prune-egress.yaml already carries targeted exceptions for other kubeflow-namespace workloads that need cross-namespace egress by design:

# notebook-controller-exception (kubeflow): egress "*/*"  (kernels in all user namespaces)
# ml-pipeline-ui-exception (kubeflow): egress "*/*"       (artifact-proxy in pipeline-enabled namespaces)

The Model Registry UI appears to be a third such workload (its per-profile server is its designed target), but it has no exception.

Question for maintainers

Is a per-workload Sidecar exception for model-registry-ui (scoped via workloadSelector, mirroring the notebook-controller-exception / ml-pipeline-ui-exception precedent) the intended way to handle this? Or is the per-profile Model Registry layout meant to be excluded from the mesh-wide egress hardening - or is something else expected (e.g. including profile namespaces in the default Sidecar's egress)?

Environment

• Version: kubeflow/community-distribution (branch master, common/istio/istio-install)
• Kubernetes 1.32.x
• Model Registry component from applications/hub/upstream

[juliusvonkohout]

@Raakshass