[SoundCloud] Fix SoundCloud HLS expiry and add extractor logging #1325
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| package org.schabi.newpipe.extractor; | ||
|
|
||
| import java.util.Objects; | ||
|
|
||
| public enum StreamingServiceId { | ||
| NO_SERVICE_ID, | ||
| YOUTUBE, | ||
| SOUNDCLOUD, | ||
| MEDIACCC, | ||
| PEERTUBE, | ||
| BANDCAMP; | ||
|
|
||
|
|
||
| private static final StreamingServiceId[] VALUES = values(); | ||
|
|
||
| public static String nameFromId(final int serviceId) { | ||
| try { | ||
| return VALUES[Objects.checkIndex(serviceId + 1, VALUES.length)].name(); | ||
| } catch (final IndexOutOfBoundsException e) { | ||
| throw new IllegalArgumentException("Invalid serviceId: " + serviceId, e); | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| package org.schabi.newpipe.extractor.exceptions; | ||
|
|
||
| import java.io.IOException; | ||
| import org.schabi.newpipe.extractor.downloader.Response; | ||
|
|
||
| public class HttpResponseException extends IOException { | ||
| public HttpResponseException(final Response response) { | ||
| this("Error in HTTP Response for " + response.latestUrl() + "\n\t" | ||
| + response.responseCode() + " - " + response.responseMessage()); | ||
| } | ||
|
|
||
| public HttpResponseException(final String message) { | ||
| super(message); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,14 +5,14 @@ | |
| import static org.schabi.newpipe.extractor.ServiceList.SoundCloud; | ||
| import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty; | ||
| import static org.schabi.newpipe.extractor.utils.Utils.replaceHttpWithHttps; | ||
| import static org.schabi.newpipe.extractor.utils.HttpUtils.validateResponseCode; | ||
|
|
||
| import com.grack.nanojson.JsonArray; | ||
| import com.grack.nanojson.JsonObject; | ||
| import com.grack.nanojson.JsonParser; | ||
| import com.grack.nanojson.JsonParserException; | ||
| import org.jsoup.Jsoup; | ||
| import org.jsoup.nodes.Document; | ||
| import org.jsoup.select.Elements; | ||
| import org.schabi.newpipe.extractor.MultiInfoItemsCollector; | ||
| import org.schabi.newpipe.extractor.Image; | ||
|
|
@@ -28,6 +28,7 @@ | |
| import org.schabi.newpipe.extractor.services.soundcloud.extractors.SoundcloudLikesInfoItemExtractor; | ||
| import org.schabi.newpipe.extractor.services.soundcloud.extractors.SoundcloudStreamInfoItemExtractor; | ||
| import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector; | ||
| import org.schabi.newpipe.extractor.utils.ExtractorLogger; | ||
| import org.schabi.newpipe.extractor.utils.ImageSuffix; | ||
| import org.schabi.newpipe.extractor.utils.JsonUtils; | ||
| import org.schabi.newpipe.extractor.utils.Parser; | ||
|
|
@@ -86,6 +87,7 @@ public final class SoundcloudParsingHelper { | |
| private static final List<ImageSuffix> VISUALS_IMAGE_SUFFIXES = | ||
| List.of(new ImageSuffix("t1240x260", 1240, 260, MEDIUM), | ||
| new ImageSuffix("t2480x520", 2480, 520, MEDIUM)); | ||
| public static final String TAG = SoundcloudParsingHelper.class.getSimpleName(); | ||
|
|
||
| private static String clientId; | ||
| public static final String SOUNDCLOUD_API_V2_URL = "https://api-v2.soundcloud.com/"; | ||
|
|
@@ -99,13 +101,14 @@ private SoundcloudParsingHelper() { | |
|
|
||
| public static synchronized String clientId() throws ExtractionException, IOException { | ||
| if (!isNullOrEmpty(clientId)) { | ||
| ExtractorLogger.d(TAG, "Returning clientId=" + clientId); | ||
| return clientId; | ||
| } | ||
|
|
||
| final Downloader dl = NewPipe.getDownloader(); | ||
|
|
||
| final Response downloadResponse = dl.get("https://soundcloud.com").validateResponseCode(); | ||
| final String responseBody = downloadResponse.responseBody(); | ||
| final String clientIdPattern = ",client_id:\"(.*?)\""; | ||
|
|
||
| final Document doc = Jsoup.parse(responseBody); | ||
|
|
@@ -116,12 +119,15 @@ public static synchronized String clientId() throws ExtractionException, IOExcep | |
|
|
||
| final var headers = Map.of("Range", List.of("bytes=0-50000")); | ||
|
|
||
| for (final var element : possibleScripts) { | ||
| final String srcUrl = element.attr("src"); | ||
| if (!isNullOrEmpty(srcUrl)) { | ||
| try { | ||
| ExtractorLogger.d(TAG, "Searching for clientId in " + srcUrl); | ||
| clientId = Parser.matchGroup1(clientIdPattern, dl.get(srcUrl, headers) | ||
| .validateResponseCode() | ||
| .responseBody()); | ||
| ExtractorLogger.d(TAG, "Found clientId=" + clientId); | ||
| return clientId; | ||
| } catch (final RegexException ignored) { | ||
| // Ignore it and proceed to try searching other script | ||
|
|
@@ -148,13 +154,16 @@ public static OffsetDateTime parseDateFrom(final String textualUploadDate) | |
| } | ||
| } | ||
|
|
||
| // CHECKSTYLE:OFF | ||
| /** | ||
| * Call the endpoint "/resolve" of the API. | ||
| * <p> | ||
| * See https://web.archive.org/web/20170804051146/https://developers.soundcloud.com/docs/api/reference#resolve | ||
| */ | ||
| // CHECKSTYLE:ON | ||
| public static JsonObject resolveFor(@Nonnull final Downloader downloader, final String url) | ||
| throws IOException, ExtractionException { | ||
| ExtractorLogger.d(TAG, "resolveFor(" + url + ")"); | ||
| final String apiUrl = SOUNDCLOUD_API_V2_URL + "resolve" | ||
| + "?url=" + Utils.encodeUrlUtf8(url) | ||
| + "&client_id=" + clientId(); | ||
|
|
@@ -177,10 +186,11 @@ public static JsonObject resolveFor(@Nonnull final Downloader downloader, final | |
| public static String resolveUrlWithEmbedPlayer(final String apiUrl) throws IOException, | ||
| ReCaptchaException { | ||
|
|
||
| final var response = NewPipe.getDownloader().get("https://w.soundcloud.com/player/?url=" | ||
| + Utils.encodeUrlUtf8(apiUrl), SoundCloud.getLocalization()); | ||
| validateResponseCode(response); | ||
| final var responseBody = response.responseBody(); | ||
| return Jsoup.parse(responseBody).select("link[rel=\"canonical\"]").first() | ||
| .attr("abs:href"); | ||
| } | ||
|
|
||
|
|
@@ -189,6 +199,7 @@ public static String resolveUrlWithEmbedPlayer(final String apiUrl) throws IOExc | |
| * | ||
| * @return the resolved id | ||
| */ | ||
| // TODO: what makes this method different from the others? Don' they all return the same? | ||
|
There was a problem hiding this comment. Maybe you can get an answer by git blaming? There was a problem hiding this comment. I've found answer to this question as per #1322 (comment) |
||
| public static String resolveIdWithWidgetApi(final String urlString) throws IOException, | ||
| ParsingException { | ||
| String fixedUrl = urlString; | ||
|
|
@@ -224,9 +235,12 @@ public static String resolveIdWithWidgetApi(final String urlString) throws IOExc | |
| final String widgetUrl = "https://api-widget.soundcloud.com/resolve?url=" | ||
| + Utils.encodeUrlUtf8(url.toString()) | ||
| + "&format=json&client_id=" + SoundcloudParsingHelper.clientId(); | ||
|
|
||
| final var response = NewPipe.getDownloader().get(widgetUrl, | ||
| SoundCloud.getLocalization()); | ||
|
|
||
| final var responseBody = response.validateResponseCode().responseBody(); | ||
| final JsonObject o = JsonParser.object().from(responseBody); | ||
| return String.valueOf(JsonUtils.getValue(o, "id")); | ||
| } catch (final JsonParserException e) { | ||
| throw new ParsingException("Could not parse JSON response", e); | ||
|
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this file used anywhere? Also, the reason why services are not implemented as an enum is also because it should be possible to dynamically add service implementations (think e.g. of plugins). Obviously this has never happened yet, but iirc that's why the API is like that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's used in the NewPipe PR for logging the names of services.
It's possible to have enums and still allow for dynamic implementations. Since plugins are a long way away, I think it makes a lot more sense to have enums for each service to improve readability, traceability, logging and type safety.
Howbeit, I'm not going to change the entire API to use enums because that's long: I just want the nice id -> name static method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can't you use this utility method instead? https://github.com/TeamNewPipe/NewPipe/blob/2a9c6f05387866bef2c42ab67226c14936917d27/app/src/main/java/org/schabi/newpipe/util/ServiceHelper.java#L138
But yeah I agree this is not the best. We should not have the concept of service ID at all except for saving stuff to database, so every
StreamInfoshould not have anygetServiceId()method but rathergetService()directly. So we don't have to keep switching back and forth betweenintandService