NO_PROXY support for per-object proxy configuration

[darkowlzz]

Per-object proxy background

In the recent releases, we have added per-object proxy support for various Flux
object kinds. I believe that initially, it was added in GitRepository API. The
proxy configuration is written to a Secret object and ready by the GitRepository
reconciler during reconciliation. See
https://fluxcd.io/flux/components/source/gitrepositories/#proxy-secret-reference
for detailed docs.

I believe that the proxy configuration was inspired from how it's used in
go-git, refer
https://github.com/go-git/go-git/blob/v5.12.0/plumbing/transport/common.go#L123-L127.
At that time, this worked perfectly as during a GitRepository reconciliation,
the outgoing connection is made only to the configured Git server.
When per-object proxy is configured, a custom transport is created with proxy
based on the provided proxy configuration. Environment variable proxy
configurations are ignored.

Need for NO_PROXY

Recently, the GitRepository API started supporting contextual login for Azure
Devops. For obtaining Git auth token, GitRepository now connects to various
Azure services to obtain auth token that can be used with the Git server.
GitRepository reconciliation makes multiple outgoing connections. When using
proxy, this creates the need to configure certain services for which proxy isn't
used. In the above example, one may want to not use proxy for connections to
Azure services but have the Git operations take place over the proxy.

Similarly, the other API objects that have proxy support may also have multiple
outgoing connections to different services and some of them may have a need to
not go through the proxy, depending on the architectural requirements.

Proposed solution

In case of environment variable proxy configurations, this is achieved by
setting NO_PROXY. When proxy is configured in Flux controller pods using
environment variables, connections to services that are defined in NO_PROXY
will not go through the proxy. For per-object proxy configuration, we need the
same behavior.

Because the proxy configuration in Flux is inspired by go-git, the configuration
includes an address, a username and a password. In case of GitRepository,
this address is used for configuring the Git client, http or ssh. Except for
Git, most of the other clients in flux are http clients. We can introduce a new
field no_proxy in the proxy Secret that can be used to configure no proxy in
the custom transport for per-object proxy configuration.

apiVersion: v1
kind: Secret
metadata:
  name: proxy
type: Opaque
stringData:
  address: http://proxy.com
  username: user
  password: pass
  no_proxy: example.com,fluxcd.io

Suggested implementation

In terms of implementation, at present, the custom transport for http clients is
configured with proxy using Go's http.ProxyURL() helper.
golang.org/x/net/http/httpproxy provides a Config type which takes into
account any NoProxy configuration. This can be used as

pc := httpproxy.Config{
    HTTPProxy:  "<ADDRESS>",
    HTTPSProxy: "<ADDRESS>",
    NoProxy:    "<NO_PROXY>",
}
proxyFunc := func(req *http.Request) (*url.URL, error) {
    return pc.ProxyFunc()(req.URL)
}
c := http.Client{
    Transport: &http.Transport{
        Proxy: proxyFunc,
        ...
    },
}

to create a new transport and a http client. Since the input address is generic
and has no notion of http/https, both HTTPProxy and HTTPSProxy are
configured to use the same address to make sure it's a fixed proxy address
regardless of the protocol scheme.

In case of ssh, the ssh client only talks to a single remote, the Git server. It
doesn't have the use case for configuring no_proxy. But if there's a need to
configure no_proxy for non-http client, we can use PerHost proxy dialer from
https://github.com/golang/net/blob/v0.30.0/proxy/per_host.go#L15.

I believe this should be applicable to source-controller and
image-reflector-controller as they connect to external services and have
per-object proxy support in the APIs.

[stefanprodan]

I'm in favour of allowing no_proxy to be set in the secret