[TlsInterception] How to parse JSON (chunked, brotli compressed) and modify it ?

[ghostlexly]

Hello, i'm trying a man-in-the-middle attack, and i can't find a way to change the JSON response.

here is my code

    def handle_upstream_chunk(self, chunk: memoryview) -> memoryview:
        self.response.parse(chunk.tobytes())

        if self.response.body_expected:
            self.response.body = b'\nTEST'
            self.client.queue(memoryview(self.response.build_response()))
            return okResponse(content=b'Hello from man in the middle')

        return chunk

I can't read the body as JSON and can't find a way to replace or search for a text inside..

[abhinavsingh]

As handle_upstream_chunk suggests, incoming data is a chunk. You must assemble all the chunks before you can make sense of them together. Incoming chunks can be assembled and parsed using ChunkParser class. See https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/http/parser/chunk.py#L27-L35

Once you have all chunks assembled, you should be able to parse that data as json (if it is a valid json). You can ensure it's json by checking Content-Type for responses

[abhinavsingh]

I'm trying to change the netflix.com's homepage text for MITM attack test purposes

Do you have any code example of https mitm ? i have found one in the plugins folder but it's doesn't read the page's content

Sorry, I use TlsInterception for analysis, but never for MITM. I added support for TlsInterception mainly for analysis purposes. That's another thing that it also allows MITM. Support was added because proxy.py users requested for it.

[ghostlexly]

Thanks for the br, i was looking for gzip lol

here is my final code :

    import brotli

    def handle_upstream_chunk(self, chunk: memoryview) -> memoryview:
        self.response.parse(chunk.tobytes())

        if type(self.response.body) is bytes:
            if self.response.chunk_parser.state == chunkParserStates.COMPLETE:
                print(self.response.headers)
                print(brotli.decompress(self.response.body))

        return chunk

But now i have "Decompression error: b'CL_SPACE'" and "Decompression error: b'HUFFMAN_SPACE'" errors on all brotli requests. I think it's because the chunks are incomplete. I don't understand why.

[abhinavsingh]

Sorry I am unsure what's happening here either. But if chunk parser has reached COMPLETE state, we can be rest assured that we have received expected data. How does len(self.response.body) compares to Content-Length header value?

[ghostlexly]

Where can i find the content-length value ?
i don't have it in the headers:

    def handle_upstream_chunk(self, chunk: memoryview) -> memoryview:
        self.response.parse(chunk.tobytes())

        if type(self.response.body) is bytes:
            if self.response.chunk_parser.state == chunkParserStates.COMPLETE:
                print(self.response.headers)
                print(len(self.response.body))

        return chunk

{b'content-type': (b'Content-Type', b'application/javascript; charset=utf-8'), b'transfer-encoding': (b'Transfer-Encoding', b'chunked'), b'connection': (b'Connection', b'keep-alive'), b'date': (b'Date', b'Mon, 10 Jan 2022 19:07:31 GMT'), b'server': (b'Server', b'nginx'), b'etag': (b'ETag', b'W/"1ac59-zzvgiaPS/0qnPUAtRCdiNA"'), b'cache-control': (b'Cache-Control', b'public, max-age=120'), b'strict-transport-security': (b'Strict-Transport-Security', b'max-age=31536000; includeSubDomains'), b'content-encoding': (b'Content-Encoding', b'gzip'), b'vary': (b'Vary', b'Accept-Encoding'), b'x-cache': (b'X-Cache', b'Hit from cloudfront'), b'via': (b'Via', b'1.1 9d2c93ece5a5ccb2b5952a40f7502a04.cloudfront.net (CloudFront)'), b'x-amz-cf-pop': (b'X-Amz-Cf-Pop', b'CDG3-C1'), b'x-amz-cf-id': (b'X-Amz-Cf-Id', b'lEZOFQUz2aTn2bRuqnSw4PLdWWphmKazRB_1poHma7LQsWuWe2tlDw=='), b'age': (b'Age', b'46')}

body length is : 42849

I attach my full plugin code here if you wanna do some tests on your side.
Big thank you, it's an important project for my company

filter_by_url_regex.txt

[abhinavsingh]

You had a content-length header in previous example. Your current example don't.

Irrespective this is an issue unrelated to proxy.py. You will run into variety of variations in the wild. proxy.py only provides you an ability to get hold of the packets. Further you must use your favorite library to parse/split/modify them.

I tried following, so as such I see no issue with receiving brotli compressed data (via proxy.py)

curl -x localhost:50660 http://httpbin.org/brotli | brotli -d -                                            ─╯
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   173  100   173    0     0    208      0 --:--:-- --:--:-- --:--:--   210
{
  "brotli": true, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.77.0", 
    "X-Amzn-Trace-Id": "Root=1-61dc8df1-14c6329d103c4239605114a8"
  }, 
  "method": "GET", 
}