Skip to content

Latest commit

 

History

History
205 lines (160 loc) · 9.53 KB

configurable-kubecontext.md

File metadata and controls

205 lines (160 loc) · 9.53 KB

Configurable kubecontext

  • Author(s): Cornelius Weig (@corneliusweig)
  • Design Shepherd: Balint Pato (@balopat)
  • Date: 29 June 2019
  • Status: On hold

Note ❗ As the global config option adds considerable complexity, that part of this design proposal has been put on hold to await more evidence. Ideally, the CLI and skaffold.yaml options are already enough for the great majority of Skaffold users.

Background

So far, Skaffold always uses the currently active kubecontext when interacting with a Kubernetes cluster. This is problematic when users want to deploy multiple projects with different kubecontexts, because the user needs to manually switch the context before starting Skaffold. In particular when working on multiple such projects in parallel, the current behavior is limiting.

Open issues concerning this problem are

  • Allow option to specify the kubectl context (#511)
  • Support kube.config and kube.context for specifying alternative Kubernetes config file or context (#2325)
  • Feature: Support regex in profile activation via kubeContext (#1677)
  • Skaffold.yaml files are not portable (#480)
  • Support forcing a context and a namespace for a profile/command (#2426)

There also was an attempt to add a configuration option to skaffold.yaml (Support for overriding kubectl context during deployment #1540).

The goal of this document is to create an agreement on what options should be supported and identify edge cases.

Recommended use-cases

As Skaffold user, I want to define the kubecontext for a single skaffold run.

Use CLI flag or environment variable.

As enterprise user, I want to define a default kubecontext for a project to be used across different machines.

Use the kubecontext configuration in skaffold.yaml. Think twice before using this approach in open source projects, as the setting will not be portable.

As individual user, I want to define a default kubecontext for a project.

Use kubecontext setting in the global Skaffold config (via skaffold config set ...).

As Skaffold user with multiple profiles, I want to use different kubecontexts for different profiles.

Use the kubecontext configuration in skaffold.yaml.

Design

There are four places where kubecontext activation can be added:

Precedence Kind Details
1. (highest) CLI option The Kubernetes standard to set the kubecontext is --context. However, in Skaffold this term is so overloaded that it should more precisely be named --kube-context. This flag is necessary for IDE integration.
2. Env variable SKAFFOLD_KUBE_CONTEXT, similar to other Skaffold flags.
3. skaffold.yaml Json-path deploy.kubeContext. This option is shareable, and requires some error handling for profile activation by kubecontext (see below).
4. (lowest) Global Skaffold config This should give users the possibility to define a default context globally or per project. This variant is not shareable.

Beside the kubecontext, also the namespace needs to be specified. Ideally, the namespace should also offer the same override variants as kubecontext. This is out of scope for this design proposal. As long as this is not implemented, there is always the workaround, to duplicate a kubecontext and set the default namespace for this kubecontext to the desired value. Then this kubecontext/namespace pair can be activated with the kubecontext activation machinery detailed in this design proposal.

Detailed discussion

Option in skaffold.yaml

A configuration option in skaffold.yaml has the advantage of being most discoverable: it is in the place where users configure all aspects of their pipeline. In addition, it allows to define the kubecontext per Skaffold profile.

A natural place for the config in skaffold.yaml is in latest.DeployConfig, resulting in a json path deploy.kubeContext.

Profiles have a double role, because they may override the kubecontext to a different value as before, but they may also be activated by a kubecontext. To catch unexpected behavior, the profile activation will perform the following logic:

  1. All profiles are checked if they become active with the original kubecontext.
  2. If any profile was activated by the current kubecontext, the effective kubecontext may not change. In other words, the effective kubecontext after profile activation must match the kubecontext before profile activation.
    • If the context has changed, return an error.
    • If the context has not changed, continue.

It is therefore possible that subsequent profiles repeatedly switch the kubecontext, in which case the last one wins.

For example, the following sequence will result in an error:

  • The current context is minikube and activates some profile.
  • Some profile also overrides the kubecontext and deploys to kubecontext gke_abc_123.
  • Thus the minikube-specific profile would be deployed to gke_abc_123, and this will be forbidden.

Note: skaffold.yaml is meant to be shared, but kubecontext names vary across users. Sharing therefore makes only sense in a corporate setting where context names are the same across different machines.

Option in global Skaffold config

Specifying a default kubecontext globally is straightforward. For example, via new config option

global:
  default-context: docker-for-desktop

On the other hand, building a relation between projects and kubecontext needs to solve two questions:

  1. How to identify projects
  2. How to save the project/kubecontext relation
How to identify projects

There are at least three possibilities:

  • Identify projects by their absolute host path. This is guaranteed to be unique, but may break if a user moves his project to another location.
  • Identify projects by a new metadata.name entry in skaffold.yaml (see also #2200). This has the drawback of being potentially not unique, so that users accidentally pin the kubecontext for more projects than intended. On the other hand, this is the standard approach taken by Kubernetes resources.
  • Identify project by their initial commit. This variant is stable against relocations. It is also unique unless a user forks a project and wants to define different kubecontexts for each fork. Compared to the other alternatives, it is rather slow.

<What option has the best tradeoffs?>

Resolution: We will go ahead with the metadata.name approach. As the name may not be unique, this requires special documentation.

How to save project/kubecontext relations

Currently, the Skaffold config uses the kubecontext as identifier.

There are two possibilities to add the relation:

  1. Reverse the mapping project/kubecontext and save as list under kube-context entries: 
    kubecontexts:
- kube-context: my-context
  skaffoldConfigs:
  - config-name
    The drawback here is that the data structure does not forbid conflicting entries, such as: 
    kubecontexts:
- kube-context: context1
  skaffoldConfigs:
  - my-project
- kube-context: context2
  skaffoldConfigs:
  - my-project
  2. Add a new top-level entry in Skaffold config: 
    global: {}
kubecontexts: []
skaffoldConfigs:
  my-project: my-context
  my-other-project: my-other-context
  '*': default-context
    This option will be more complex to implement wrt skaffold config.

Open Issues/Questions

<What Skaffold config structure has the best tradeoffs?>

Resolution: The top-level entry (option 2) overall has the better trade-offs.

Implementation plan

  1. Implement the CLI flag and env var variant first. This should also be the most important for the IDE integration. #2447
  2. Implement skaffold.yaml variant. #2510
  3. Implement the global Skaffold config variant to override a kubecontext for a skaffold config (skaffoldConfigs). #2558
  4. Implement the global Skaffold config variant to set a default kubecontext (skaffoldConfigs['*']). #2558
  5. Implement the namespace functionality. (out of scope)

Integration test plan

A single test covers the overall kubecontext override functionality sufficiently (part of #2447). Other test-cases such as precedence of the different variants and error cases should be covered by unit tests.