Best way to load test live HLS

[danroot]

We would like to load test several always-on live HLS streams at once. As best I can find, the documents all describe load testing a "on demand" style HLS stream. I'm new to artillery, so not 100% sure I'm reading the output correctly. When I attempt load testing a "live" always-on stream, the .m3u8 request completes, but it's not clear to me where to set how long each user views the stream.

Is there a way to test, for example "600 users watching these two streams for 10 minutes each"?

config:
  target: 'https://video.company.com'
  phases:
      - duration: 10
        arrivalRate: 1
  plugins:
    hls: {}
      
scenarios:
  - flow:
    - get:
        url: "/some.stream/playlist.m3u8"
        hls:
          concurrency: 2
          streamSelector:
            resolution:
              width: 720
              height: 480
    - get:
        url: "/some_other.stream/playlist.m3u8"
        hls:
          concurrency: 2
          streamSelector:
            resolution:
              width: 720
              height: 480

Output:

All virtual users finished
Summary report @ 22:58:31(-0600) 2021-03-01
  Scenarios launched:  10
  Scenarios completed: 10
  Requests completed:  20
  Mean response/sec: 0.44
  Response time (msec):
    min: 492.5
    max: 10677.3
    median: 2370.4
    p95: 9166.6
    p99: 10677.3
  Scenario counts:
    0: 10 (100%)
  Codes:
    200: 20
  HLS: segment download time:
    min: 2846
    max: 24564
    median: 12914
    p95: 24479
    p99: 24564
  HLS: stream download time:
    min: 0
    max: 31618
    median: 7358.5
    p95: 30839.5
    p99: 31618
  HLS: stream download started: 20
  HLS: segment download started: 30
  HLS: segment download completed: 30
  HLS: stream download completed: 20

[hassy]

Awesome use-case @danroot. Been a while since I looked at HLS myself. Could you remind me, what does a client do with a live HLS stream? Are they expected to re-request the playlist URL to find out where to get new segments from?

Artillery's HLS plugin does not do that at the moment iirc, but it should be fairly straightforward to make it re-load the playlist (if that's the case).

[hassy]

Re plugin's output:

• HLS: stream download started: 20 (and completed) -- 20 streams have been streamed (each of the 10 virtual users created in your config.phase definition made requests to two streams)
• 30 segments (across both streams) have been downloaded
• HLS segment download time p99 is 24564, i.e. most (99%) of the segments were downloaded in 24.5s or less

[danroot]

Thanks - your reply helps me understand the output a bit better. In our case (RTSP traffic cameras restreamed from a Wowza server as HLS), it looks to me like a normal HLS client (HLS.js in this case) will just continually re-request the chunklist m3u8 (not initial playlist). I'm not versed enough in "on demand" HLS to know what would signal it to stop HLS stream. I can give you a public URL if you need, though we are doing all this to debug some issues under load so if you do use it, please don't hit it too hard. The traffic looks like this:

GET playlist.m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=904503,CODECS="avc1.4d401e",RESOLUTION=810x480
chunklist_w625761017.m3u8

GET chunklist_w625761017.m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:11
#EXT-X-MEDIA-SEQUENCE:2269
#EXT-X-DISCONTINUITY-SEQUENCE:0
#EXTINF:10.01,
media_w625761017_2269.ts
#EXTINF:10.01,
media_w625761017_2270.ts
#EXTINF:10.01,
media_w625761017_2271.ts

GET media_w625761017_2269.ts
(.ts bytes)

GET media_w625761017_2270.ts
(.ts bytes)

GET blob:https://video.somesite.com/8874efe8-8db3-47c9-ae83-ef5bcdd9752d
(javascript - I think this is more HLS.js related than HLS spec, but including for completeness)

GET media_w625761017_2271.ts
(.ts bytes)

GET chunklist_w625761017.m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:11
#EXT-X-MEDIA-SEQUENCE:2270
#EXT-X-DISCONTINUITY-SEQUENCE:0
#EXTINF:10.01,
media_w625761017_2270.ts
#EXTINF:10.01,
media_w625761017_2271.ts
#EXTINF:10.01,
media_w625761017_2272.ts

chunklist_w625761017.m3u8 is called indefinitely, and fresh .ts served up until the client is stopped.

Reading your description, I'm not clear why it stopped at 30 segments - I would have expected 20 streams, each with 3 .ts segments, to result in 60 segments downloaded. Or if it only did the first segment, 20. Maybe because at 30 it was past duration:10?

Regardless, it sounds like what we're wanting is a duration or numberOfSegments on the hls plugin?

Thanks!

[hassy]

Yes it sounds like the plugin needs to allow an option to keep reloading the playlist until a certain number of segments have been downloaded / some amount of time has passed.

An URL for testing would be great (feel free to email me on [email protected]), though it might be a good few days before I have a chance to look at it.

[danroot]

Artillery has already helped us look at some static files under load- great product. One clarification - I believe it's the chunklist , not the playlist, that would need to be reloaded. Honestly, we're deeper in the HLS spec than I usually dig, but in the HTTP above, that's all one stream. Player requests playlist.m3u8, which contains a chunklist_nnnn.m3u8. Player requests chunklist_nnnn.m3u8 which contains the .ts segments. Player requests the .ts segments, renders them, then repeats request of chunklist_nnnn.m3u8 until the player is stopped. I'd need to diff with a "non-live" m3u8 to figure out what signals it to continue requesting. I suspect it's one of the EXT-X-* codes. Thanks for your help on this. Will send a URL offline.