Add proxy and custom cert options

[mdavis-xyz]

summary

I'm running in a corporate environment where all traffic must go through a proxy.

Proxies and custom certs are common in many organisations, so I think this library should be able to handle them.
I think this can be added as easily as adding a few more options to _ensure_session, and then exposing them as optional arguments in OEClient.__ini__().

my environment

I am unsure of the details of my organisation's proxy. I think that if you don't set an explicit proxy, then it goes through a MitM inspector, which results in TLS errors unless you install the corporate certificate.

Currently, I get errors:

aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host api.openelectricity.org.au:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1081)')]

minimal working example

To debug this, I wrote a MWE with just aiohttp:

from aiohttp import ClientResponse, ClientSession
import asyncio
import ssl
import os

proxy_url = "http://proxy.mycompany.org:8080"
os.environ["HTTPS_PROXY"] = proxy_url
os.environ["HTTP_PROXY"] = proxy_url

api_key_path = "c:\\Users\\MYUSER\\Documents\\Keys\\key.txt"

with open(api_key_path, 'r') as f:
    api_key = f.read().strip()

headers = {
    "Authorization": f"Bearer {api_key}"
}

test_url = f"https://api.openelectricity.org.au/v4/facility/"


async def main():
    #ssl_ctx = ssl.create_default_context(cafile=corp_ca_path)

    async with ClientSession(headers=headers) as session:
        async with session.get(test_url) as resp:
            print("Status:", resp.status)
            text = await resp.text()
            print("Body preview:", text[:200]) 

            assert resp.status == 200, f"Bad status: {resp.status}"


asyncio.run(main())

This gives the same error:

aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host api.openelectricity.org.au:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1081)')]

solution

Documentation for proxies with aiohttp is here.
There are two ways to get aiohttp to use a proxy.

We already have environment variables as HTTPS_PROXY, but it won't use it unless we set trust_env.

async with ClientSession(headers=headers, trust_env=False) as session:

or pass the proxy explicitly:

async with ClientSession(headers=headers, proxy=proxy_url) as session:

If we modify openelectricity to allow users to pass proxy config, that should also include proxy_auth.
As well as certificates:

ssl_context = ssl.create_default_context(cafile=certifi.where())
async with ClientSession(connector=TCPConnector(ssl=ssl_context)) as sess:

Alternatively this library could just always set trust_env=True. It would maximize the number of people for whom this library works out of the box. I don't know if there are security implications for that though.

[mdavis-xyz]

Actually, I can see that a ClientSession() is created in potentially many different places. I am not familiar with async Python. Could we maybe refactor this so that there is one function which creates a session, and that is called from each of the places where ClientSession() is currently called? That way you only have to worry about headers, proxies, retries etc once.