[SoundCloud] Fix SoundCloud HLS expiry and add extractor logging #1325

absurdlylongusername · 2025-07-08T16:59:07Z

I carefully read the contribution guidelines and agree to them.
I have tested the API against NewPipe.
I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API.

Fix SoundCloud HLS stream urls expiring

This PR implements extractor side changes to facilitate refreshing expires HLS stream URLs for SoundCloud streams.

Add Extractor Logging

Added extractor logging, which helped a lot to fix this issue.

In a future PR I will add a proper logging framework that will only print logs if running in debug mode, but for now since the logs are not in high-traffic code then it should be fine.

Please Note

Includes changes from:

So those PRs must be merged before this.

See TeamNewPipe/NewPipe#12418 for the full writeup for the fix

Add StreamingServiceId enum to map service id to name for logging

…rate building Hls and Progressive streams Add HlsAudioStream to facilitate refreshing expired Hls playlists

TobiGr · 2025-10-02T13:14:04Z

extractor/src/main/java/org/schabi/newpipe/extractor/utils/ExtractorLogger.java

+        logger.error(tag, msg, t);
+    }
+
+    // default logger that prints to stdout


Suggested change

// default logger that prints to stdout

/**

* Default logger that prints to stdout.

*/

Stypox

Thank you! My review gives some thoughts that we should decide on before continuing. There definitely needs to be a way to refresh streams in the extractor, which will come in handy not only for SoundCloud but also for YouTube.

I think it is a good idea to store refresh information in the streams themselves, and to make them expose methods that allow refreshing the stream. I guess this would also play well with #858 (comment). What do you think @AudricV?

Regarding logging, if I understand correctly the commit structure, it should be possible to extract it in a separate PR, that we can merge much faster. Could you put the logging in a separate PR? Also the

Stypox · 2025-10-02T12:56:15Z

extractor/src/main/java/org/schabi/newpipe/extractor/utils/ExtractorLogger.java

+    public static void d(final String tag, final String msg) {
+        logger.debug(tag, msg);
+    }
+
+    public static void w(final String tag, final String msg) {
+        logger.warn(tag, msg);
+    }


These should also optionally take a Throwable (that's how they work in Android, and it's often useful)

Stypox · 2025-10-02T13:01:39Z

extractor/src/main/java/org/schabi/newpipe/extractor/stream/RefreshableStream.java

+@SuppressWarnings("checkstyle:LeftCurly")
+public interface RefreshableStream  {


Suggested change

@SuppressWarnings("checkstyle:LeftCurly")

public interface RefreshableStream {

public interface RefreshableStream {

Stypox · 2025-10-02T13:19:01Z

extractor/src/main/java/org/schabi/newpipe/extractor/StreamingServiceId.java

Is this file used anywhere? Also, the reason why services are not implemented as an enum is also because it should be possible to dynamically add service implementations (think e.g. of plugins). Obviously this has never happened yet, but iirc that's why the API is like that.

It's used in the NewPipe PR for logging the names of services.

It's possible to have enums and still allow for dynamic implementations. Since plugins are a long way away, I think it makes a lot more sense to have enums for each service to improve readability, traceability, logging and type safety.

Howbeit, I'm not going to change the entire API to use enums because that's long: I just want the nice id -> name static method.

Can't you use this utility method instead? https://github.com/TeamNewPipe/NewPipe/blob/2a9c6f05387866bef2c42ab67226c14936917d27/app/src/main/java/org/schabi/newpipe/util/ServiceHelper.java#L138

But yeah I agree this is not the best. We should not have the concept of service ID at all except for saving stuff to database, so every StreamInfo should not have any getServiceId() method but rather getService() directly. So we don't have to keep switching back and forth between int and Service

Stypox · 2025-10-02T13:22:11Z

extractor/src/main/java/org/schabi/newpipe/extractor/utils/ExtractorLogger.java

+    }
+
+    // default logger that prints to stdout
+    private static final class ConsoleLogger implements Logger {


This should be only set by users of the library and should default to a logger that logs nothing. In each extractor test then you would set this ConsoleLogger as the logger (i.e. you just add this in the common tests initialization methods).

Stypox · 2025-10-02T13:28:32Z

extractor/src/main/java/org/schabi/newpipe/extractor/stream/HlsAudioStream.java

+
+import javax.annotation.Nonnull;
+
+public class HlsAudioStream extends AudioStream implements RefreshableStream {


I don't think it's a property of all HLS streams that they are refreshable. I think we have two alternative ways we can handle this instead:

RefreshableStream could just be an interface that some Streams can implement, then each Service would override VideoStream/AudioStream and also implement RefreshableStream if needed, and the player could check for refreshable streams using instanceof. This would require a lot of boilerplate though.

Add a (Audio/Video)Stream refresh() {} method to the base Stream class which builds a new refreshed stream, and make it return null if refreshing is not available.

Stypox · 2025-10-02T13:29:29Z

extractor/src/main/java/org/schabi/newpipe/extractor/stream/RefreshableStream.java

+import java.io.IOException;
+
+@SuppressWarnings("checkstyle:LeftCurly")
+public interface RefreshableStream  {


Shouldn't this also somehow store the expiry time? So that the player can prepare in advance if the stream is about to expire

How exactly would the player prepare in advance? What is it supposed to do?

If the currently playing stream expires at 18:50:45 and now it's 18:50:35, there are just 10 more seconds left to fetch data before needing to switch over to the refreshed stream. So instead of waiting until the URL returns 403 and then quitting, the player can notice that only 10 seconds are left and already pre-fetch the refreshed stream and switch to it as soon as it's ready.

ExoPlayer handles buffering of stream data, so I don't think is necessary since stream chunks are already pre-fetched ahead of time.

It fetches chunks ahead of time, so when it gets a 403 it's not for the currently playing chunk.

Oh right, that's true. I'd still add a nullable expiration Instant just in case other clients need it (or maybe NewPipe's Downloader might need it? Or maybe streams in other services that don't throw 403 and where it's more difficult to detect that the stream is now invalid?), since this is information the extractor usually knows. But this is not totally necessary, so do it only if it seems simple to extract such data.

Stypox · 2025-10-02T13:32:12Z

.../src/main/java/org/schabi/newpipe/extractor/services/soundcloud/SoundcloudParsingHelper.java

     *
     * @return the resolved id
     */
+    // TODO: what makes this method different from the others? Don' they all return the same?


Maybe you can get an answer by git blaming?

I've found answer to this question as per #1322 (comment)

absurdlylongusername · 2025-10-20T02:48:29Z

@Stypox

Regarding logging, if I understand correctly the commit structure, it should be possible to extract it in a separate PR, that we can merge much faster. Could you put the logging in a separate PR? Also the

Seems you didn't finish your sentence there lol.

Yeah sure I can make a separate logging PR that can be merged and then rebase this PR on top of it

Stypox · 2025-10-21T16:51:08Z

I have no idea what I wanted to say now, sorry xD

absurdlylongusername changed the title ~~[SoundCloud] Fix soundcloud hls expiry and add extractor logging~~ [SoundCloud] Fix SoundCloud HLS expiry and add extractor logging Jul 8, 2025

absurdlylongusername mentioned this pull request Jul 8, 2025

[SoundCloud] Fix soundcloud regression of expiring HLS streams after 5mins TeamNewPipe/NewPipe#12418

Open

5 tasks

ShareASmile added bug Issue is related to a bug soundcloud service, https://soundcloud.com/ labels Jul 8, 2025

absurdlylongusername added 6 commits October 2, 2025 14:58

[SoundCloud] Validate http response code in SoundcloudParsingHelper

a5e59ce

Add logging functionality to extractor and add some logging

1d553dc

Add StreamingServiceId enum to map service id to name for logging

[SoundCloud] Add some logging to SoundcloudParsingHelper.java

1188011

[SoundCloud] Add some logging to SoundcloudStreamExtractor

95ea906

[SoundCloud] Refactor Soundcloud audio stream extraction code to sepa…

072e977

…rate building Hls and Progressive streams Add HlsAudioStream to facilitate refreshing expired Hls playlists

Add error logging in StreamInfo.extractStreams

578f4f0

Stypox force-pushed the fix-soundcloud-hls-expiry branch from 45df3cd to 578f4f0 Compare October 2, 2025 12:58

TobiGr reviewed Oct 2, 2025

View reviewed changes

Stypox requested changes Oct 2, 2025

View reviewed changes

		@SuppressWarnings("checkstyle:LeftCurly")
		public interface RefreshableStream {


		import javax.annotation.Nonnull;

		public class HlsAudioStream extends AudioStream implements RefreshableStream {

Uh oh!

[SoundCloud] Fix SoundCloud HLS expiry and add extractor logging #1325

Are you sure you want to change the base?

[SoundCloud] Fix SoundCloud HLS expiry and add extractor logging #1325

Uh oh!

Conversation

absurdlylongusername commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fix SoundCloud HLS stream urls expiring

Add Extractor Logging

Please Note

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Stypox left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

absurdlylongusername commented Oct 20, 2025

Uh oh!

Stypox commented Oct 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

absurdlylongusername commented Jul 8, 2025 •

edited

Loading