moqrelayserver crash - ending session by closing bidi control results in UAF

[gmarzot]

Core was generated by `./build/bin/moqrelayserver -port 4433 -cert ./certs/certificate.pem -key ./cer'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 folly::detail::CancellationState::requestCancellation (this=0x7fe098083440)
--Type for more, q to quit, c to continue without paging--
at /home/gmarzot/Projects/moq/moxygen/build/deps/folly/folly/CancellationToken.cpp:135
135 head = callback->next;
[Current thread is 1 (Thread 0x7fe09ea25640 (LWP 1023106))]
(gdb) bt
#0 folly::detail::CancellationState::requestCancellation (this=0x7fe098083440)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/folly/folly/CancellationToken.cpp:135
#1 0x00005578341f20fa in folly::detail::CancellationState::requestCancellation (this=)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/folly/folly/CancellationToken.cpp:180
#2 0x00005578343be055 in folly::CancellationSource::requestCancellation (this=)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/include/folly/CancellationToken-inl.h:238
#3 proxygen::WebTransportImpl::StreamReadHandle::deliverReadError (this=0x7fe098085598, ex=...)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/proxygen/proxygen/lib/http/webtransport/WebTransport Impl.cpp:315
#4 0x00005578343c009f in proxygen::WebTransportImpl::StreamReadHandle::readError (this=0x7fe098085598,
id=, error=...)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/proxygen/proxygen/lib/http/webtransport/WebTransport Impl.cpp:309
#5 0x00005578344cf83d in quic::QuicTransportBaseLite::cancelAllAppCallbacks (this=0x7fe098006b58,
err=...)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/mvfst/quic/api/QuicTransportBaseLite.cpp:2257
#6 0x00005578344ca47a in quic::QuicTransportBaseLite::closeImpl (this=, errorCode=...,
drainConnection=, sendCloseImmediately=)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/mvfst/quic/api/QuicTransportBaseLite.cpp:1443
#7 0x00005578344d3989 in quic::QuicTransportBaseLite::onNetworkData (this=0x7fe098006b58, peer=...,
networkData=...) at /usr/include/c++/11/bits/char_traits.h:357
#8 0x0000557834479db0 in operator() (__closure=0x7fe09ea1c2e0, transport=0x7fe098006a60)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/mvfst/quic/server/QuicServerWorker.cpp:848
--Type for more, q to quit, c to continue without paging--
#9 0x0000557834481a02 in quic::QuicServerWorker::dispatchPacketData (this=, client=..., routingData=..., networkData=...,
quicVersion=..., isForwardedData=) at /usr/include/c++/11/bits/shared_ptr_base.h:1295
#10 0x0000557834458cd0 in quic::QuicServer::routeDataToWorker (this=0x55783dd2ac20, client=..., routingData=..., networkData=...,
quicVersion=..., isForwardedData=false) at /home/gmarzot/Projects/moq/moxygen/_build/deps/mvfst/quic/server/QuicServer.cpp:465
#11 0x000055783447d991 in quic::QuicServerWorker::forwardNetworkData (this=0x55783dd2c150, client=..., routingData=..., networkData=...,
quicVersion=..., isForwardedData=)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/mvfst/quic/server/QuicServerWorker.cpp:649
#12 0x000055783447f916 in quic::QuicServerWorker::handleNetworkData (this=0x55783dd2c150, client=..., udpPacket=...,
isForwardedData=) at /usr/include/c++/11/bits/stl_iterator.h:1027
#13 0x00005578344803a9 in quic::QuicServerWorker::onDataAvailable (this=0x55783dd2c150, client=..., len=44, truncated=,
params=...) at /home/gmarzot/Projects/moq/moxygen/_build/deps/mvfst/quic/server/QuicServerWorker.cpp:375
#14 0x0000557834653a56 in folly::AsyncUDPSocket::handleRead (this=0x7fe0980014e0)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/folly/folly/io/async/AsyncUDPSocket.cpp:1331
#15 0x000055783420aefc in folly::EventHandler::libeventCallback (fd=, events=, arg=0x7fe0980014e0)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/folly/folly/io/async/EventHandler.cpp:159
#16 0x00007fe09f6cff58 in ?? () from /lib/x86_64-linux-gnu/libevent-2.1.so.7
#17 0x00007fe09f6d18a7 in event_base_loop () from /lib/x86_64-linux-gnu/libevent-2.1.so.7
#18 0x00005578341fe84e in (anonymous namespace)::EventBaseBackend::eb_event_base_loop (flags=1, this=)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/folly/folly/io/async/EventBase.cpp:99
#19 folly::EventBase::loopMain (this=0x55783dd3d900, flags=, options=...)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/folly/folly/io/async/EventBase.cpp:606
#20 0x00005578341fec35 in folly::EventBase::loopBody (this=0x55783dd3d900, flags=0, options=...)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/folly/folly/io/async/EventBase.cpp:516
#21 0x00005578341fecd0 in folly::EventBase::loop (this=0x55783dd3d900)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/folly/folly/io/async/EventBase.cpp:477
#22 0x0000557834200e5d in folly::EventBase::loopForever (this=this@entry=0x55783dd3d900)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/folly/folly/io/async/EventBase.cpp:784
#23 0x000055783465adf3 in folly::run (ebm=0x55783dd90060, eb=0x55783dd3d900, stop=0x55783dd3db88, name=...)
at /home/gmarzot/Projects/moq/moxygen/_build/deps/folly/folly/io/async/ScopedEventBaseThread.cpp:40
#24 0x00007fe09ee47253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#25 0x00007fe09eacdac3 in start_thread (arg=) at ./nptl/pthread_create.c:442
#26 0x00007fe09eb5f850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

[gmarzot]

packet trace before crash

Frame 20: 74 bytes on wire (592 bits), 74 bytes captured (592 bits) on interface lo, id 0
Interface id: 0 (lo)
Interface name: lo
Encapsulation type: Ethernet (1)
Arrival Time: Feb 2, 2025 22:44:17.266086899 EST
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1738554257.266086899 seconds
[Time delta from previous captured frame: 3.002524982 seconds]
[Time delta from previous displayed frame: 3.002524982 seconds]
[Time since reference or first frame: 3.010845797 seconds]
Frame Number: 20
Frame Length: 74 bytes (592 bits)
Capture Length: 74 bytes (592 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:udp:quic]
Ethernet II, Src: 00:00:00:00:00:00, Dst: 00:00:00:00:00:00
Destination: 00:00:00:00:00:00
Address: 00:00:00:00:00:00
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Source: 00:00:00:00:00:00
Address: 00:00:00:00:00:00
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.14.25.253, Dst: 10.14.25.253
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
0000 00.. = Differentiated Services Codepoint: Default (0)
.... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
Total Length: 60
Identification: 0x51f3 (20979)
Flags: 0x40, Don't fragment
0... .... = Reserved bit: Not set
.1.. .... = Don't fragment: Set
..0. .... = More fragments: Not set
...0 0000 0000 0000 = Fragment Offset: 0
Time to Live: 64
Protocol: UDP (17)
Header Checksum: 0xa0a8 [validation disabled]
[Header checksum status: Unverified]
Source Address: 10.14.25.253
Destination Address: 10.14.25.253
User Datagram Protocol, Src Port: 54420, Dst Port: 4433
Source Port: 54420
Destination Port: 4433
Length: 40
Checksum: 0x484f [unverified]
[Checksum Status: Unverified]
[Stream index: 0]
[Timestamps]
[Time since first frame: 3.010845797 seconds]
[Time since previous frame: 3.002524982 seconds]
UDP payload (32 bytes)
QUIC IETF
QUIC Connection information
[Connection Number: 0]
[Packet Length: 32]
QUIC Short Header DCID=4000000b5c076264 PKN=11
0... .... = Header Form: Short Header (0)
.1.. .... = Fixed Bit: True
..1. .... = Spin Bit: True
...0 0... = Reserved: 0
.... .0.. = Key Phase Bit: False
.... ..01 = Packet Number Length: 2 bytes (1)
Destination Connection ID: 4000000b5c076264
Packet Number: 11
Protected Payload: e09c2051df8259d341d70363d3c5a432b872ec7ae6
STREAM id=4 fin=1 off=35 len=0 dir=Bidirectional origin=Client-initiated
Frame Type: STREAM (0x000000000000000f)
.... ...1 = Fin: True
.... ..1. = Len(gth): True
.... .1.. = Off(set): True
Stream ID: 4
.... .... .... .... .... .... .... .... .... .... .... .... .... .... .... ...0 = Stream initiator: Client-initiated (0)
.... .... .... .... .... .... .... .... .... .... .... .... .... .... .... ..0. = Stream direction: Bidirectional (0)
Offset: 35
Length: 0
Stream Data:

[gmarzot]

pip install aiomoqt

minimal script to recreate

#!/usr/bin/env python3

import argparse
import logging
import ssl

import asyncio
from aioquic.h3.connection import H3_ALPN
from aioquic.quic.configuration import QuicConfiguration

from aiomoqt.client import MOQTClient
from aiomoqt.utils.logger import get_logger, set_log_level, QuicDebugLogger


def parse_args():
    parser = argparse.ArgumentParser(description='MOQT WebTransport Client')
    parser.add_argument('--host', type=str, default='localhost',
                        help='Host to connect to')
    parser.add_argument('--port', type=int, default=4433,
                        help='Port to connect to')
    parser.add_argument('--endpoint', type=str, default='moq',
                        help='MOQT WT endpoint')
    parser.add_argument('--debug', action='store_true',
                        help='Enable debug output')
    return parser.parse_args()


async def main(host: str, port: int, endpoint: str, debug: bool):
    log_level = logging.DEBUG if debug else logging.INFO
    set_log_level(log_level)
    logger = get_logger(__name__)
    try:
        secrets_log = open("/tmp/keylog.client.txt", "a") if debug else None
        quic_logger = QuicDebugLogger() if debug else None
        configuration = QuicConfiguration(
            alpn_protocols=H3_ALPN,
            is_client=True,
            verify_mode=ssl.CERT_NONE,
            quic_logger=quic_logger,
            secrets_log_file=secrets_log
        )
        client = MOQTClient(
            host, port, endpoint=endpoint,
            configuration=configuration, debug=debug
        )
        logger.info(f"Open MoQT client session: {client}")
        async with client.connect() as session:
            await asyncio.wait_for(session.initialize(), timeout=30)
            logger.info(
                f"Closing control stream: {session._control_stream_id}"
            )
            session._quic.send_stream_data(
                session._control_stream_id, b"", end_stream=True
            )
        logger.info(f"MoQT client session closed: {client}")
    except asyncio.TimeoutError:
        logger.error("MoQT client session connection timed out")
        raise

if __name__ == "__main__":
    args = parse_args()
    asyncio.run(main(
        host=args.host,
        port=args.port,
        endpoint=args.endpoint,
        debug=args.debug
    ))

[afrind]

I think this is because we WebTransportImpl.cpp needs to clear the stream read callback before it erases the stream state after FIN receipt. Requires a new API in WebTransportImpl::Provider.

[afrind]

I believe this is fixed by facebook/proxygen#550