Accessing HTTP connection response body for debugging

[DrPyser]

Hi!

I'm trying to debug failing connection attempts, probably an authentication issue with our API gateway.
The gateway server responds with a 400, refusing the connection. The websockets debug logs show the response status and headers, but not the response body, which in this case should contain an error message.

Is there a way to have the library output the response body as well?

I would have tried using a proxy like https://docs.mitmproxy.org to monitor the http exchange, but the library doesn't support connecting through such a proxy.

Thanks!

[aaugustin]

Probably, the fastest solution will be to patch the library around here:

https://github.com/aaugustin/websockets/blob/017a072705408d3df945e333e5edd93e0aa8c706/src/websockets/client.py#L101

Add something along the lines of await self.reader.read(...) and see if you get something.

Definitely something worth improving, at least for simple cases (e.g. when there's a Content-Length header).

[DrPyser]

Thanks!

[aaugustin]

#676 (WIP) parses HTTP response bodies in the easy cases (no Transfer-Encoding) which is a good step towards getting this done.

[nosalan]

If anyone is interested I'm patching it in the way below. Just call patch_websockets_lib_in_order_to_see_unsuccessful_websocket_connection_response_body

from typing import Tuple

import websockets
import websockets.client as websockets_client_for_patching
from websockets.http import Headers


class InvalidResponse(Exception):
    def __init__(self, status_code: int, body: str) -> None:
        self.status_code = status_code
        self.body = body

    def __str__(self) -> str:
        return f"Code {self.status_code}, Body {self.body}"


async def read_http_response_patched(self) -> Tuple[int, Headers]:
    status_code: int
    body: str
    try:
        status_code, reason, headers = await websockets.http.read_response(self.reader)
    except Exception as exc:
        raise websockets.InvalidMessage("did not receive a valid HTTP response") from exc

    if status_code != 101:
        body = await self.reader.read()
        body = body.decode()
        body = body.replace('\n', "").replace('\r', "")
        raise InvalidResponse(status_code, body)

    websockets.client.logger.debug("%s < HTTP/1.1 %d %s", self.side, status_code, reason)
    websockets.client.logger.debug("%s < %r", self.side, headers)

    self.response_headers = headers

    return status_code, self.response_headers


def patch_websockets_lib_in_order_to_see_unsuccessful_websocket_connection_response_body():
    websockets_client_for_patching.WebSocketClientProtocol.read_http_response = read_http_response_patched

[newvicx]

#676 (WIP) parses HTTP response bodies in the easy cases (no Transfer-Encoding) which is a good step towards getting this done.

Out of curiosity, what is the reasoning for not supporting chunked Transfer-Encoding? I notice you reference section 3.3.3 of the HTTP RFC in the code, but I didn't see anything in the RFC that would prevent you from processing the body of the response (as long as the final encoding is chunked). Maybe something like this...

if 100 <= status_code < 200 or status_code == 204 or status_code == 304:
          body = None
else:
    if "Transfer-Encoding" in headers:
        if headers["Transfer-Encoding"] == "chunked":
            body = b""
            while True:
                # The first line contains the chunk size
                chunksize = yield from parse_line(read_line)
                n = int(chunksize, 16)
                if n > 0:
                    body += yield from read_exact(n + 2)
                else:
                    # Chunksize is 0, consume the last line
                    data = yield from parse_line(read_line)
                    assert not data
                    break
        else:
            raise NotImplementedError(
                f"'{headers["Transfer-Encoding"]}' transfer codings aren't supported"
            )
    else:
        # Do content length stuff...

[aaugustin]

The reason is that I'm maintaining a WebSocket library, not a HTTP library.

Deciding not to implement a HTTP library is purely a decision of where I want to spend my free time :-)

[aaugustin]

It will always be possible to move the line "just a bit".

At some point I have to draw the line. I believe that the place where it's currently drawn covers reasonable use cases of websockets as a WebSocket client.

If you come across a use case where support for Transfer-Encoding would be needed, I will assess if I deem that use case reasonable to support :-)

[aaugustin]

This is now available in Sans-I/O implementation, the threading implementation, and the new asyncio implementation that was just merged to the main branch (#1332), which solves this issue. I'm not going to add it to the legacy asyncio implementation.