Skip to content

Commit

Permalink
Merge branch 'main' into curl-parse
Browse files Browse the repository at this point in the history
  • Loading branch information
hadley authored Dec 20, 2024
2 parents 260cd74 + 962d50a commit 0e71e83
Show file tree
Hide file tree
Showing 27 changed files with 211 additions and 132 deletions.
5 changes: 2 additions & 3 deletions DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: httr2
Title: Perform HTTP Requests and Process the Responses
Version: 1.0.6.9000
Version: 1.0.7.9000
Authors@R: c(
person("Hadley", "Wickham", , "[email protected]", role = c("aut", "cre")),
person("Posit Software, PBC", role = c("cph", "fnd")),
Expand Down Expand Up @@ -37,7 +37,7 @@ Suggests:
jose,
jsonlite,
knitr,
later (>= 1.3.2.9001),
later (>= 1.4.0),
paws.common,
promises,
rmarkdown,
Expand All @@ -54,4 +54,3 @@ Config/testthat/start-first: resp-stream, req-perform
Encoding: UTF-8
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.3.2
Remotes: r-lib/later
9 changes: 9 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
# httr2 (development version)

* `url_parse()` now uses `curl::curl_parse_url()` which is much faster and more correct (#577).
* `req_retry()` now defaults to `max_tries = 2` with a message.
Set to `max_tries = 1` to disable retries.

* Errors thrown during the parsing of an OAuth response now have a dedicated
`httr2_oauth_parse` error class that includes the original response object
(@atheriel, #596).

# httr2 1.0.7

* `req_perform_promise()` upgraded to use event-driven async based on waiting efficiently on curl socket activity (#579).
* New `req_oauth_token_exchange()` and `oauth_flow_token_exchange()` functions implement the OAuth token exchange protocol from RFC 8693 (@atheriel, #460).

Expand Down
27 changes: 15 additions & 12 deletions R/oauth-flow-auth-code.R
Original file line number Diff line number Diff line change
Expand Up @@ -7,30 +7,33 @@
#' This flow is the most commonly used OAuth flow where the user
#' opens a page in their browser, approves the access, and then returns to R.
#' When possible, it redirects the browser back to a temporary local webserver
#' to capture the authorization code. When this is not possible (e.g. when
#' to capture the authorization code. When this is not possible (e.g., when
#' running on a hosted platform like RStudio Server), provide a custom
#' `redirect_uri` and httr2 will prompt the user to enter the code manually.
#'
#' Learn more about the overall OAuth authentication flow in
#' <https://httr2.r-lib.org/articles/oauth.html>.
#' <https://httr2.r-lib.org/articles/oauth.html>, and more about the motivations
#' behind this flow in
#' <https://stack-auth.com/blog/oauth-from-first-principles>.
#'
#' # Security considerations
#'
#' The authorization code flow is used for both web applications and native
#' applications (which are equivalent to R packages). `r rfc(8252)` spells out
#' important considerations for native apps. Most importantly there's no way
#' for native apps to keep secrets from their users. This means that the
#' server should either not require a `client_secret` (i.e. a public client
#' not an confidential client) or ensure that possession of the `client_secret`
#' doesn't bestow any meaningful rights.
#' server should either not require a `client_secret` (i.e. it should be a
#' public client and not a confidential client) or ensure that possession of
#' the `client_secret` doesn't grant any significant privileges.
#'
#' Only modern APIs from the bigger players (Azure, Google, etc) explicitly
#' native apps. However, in most cases, even for older APIs, possessing the
#' `client_secret` gives you no ability to do anything harmful, so our
#' general principle is that it's fine to include it in an R package, as long
#' as it's mildly obfuscated to protect it from credential scraping. There's
#' no incentive to steal your client credentials if it takes less time to
#' create a new client than find your client secret.
#' Only modern APIs from major providers (like Azure and Google) explicitly
#' support native apps. However, in most cases, even for older APIs, possessing
#' the `client_secret` provides limited ability to perform harmful actions.
#' Therefore, our general principle is that it's acceptable to include it in an
#' R package, as long as it's mildly obfuscated to protect against credential
#' scraping attacks (which aim to acquire large numbers of client secrets by
#' scanning public sites like GitHub). The goal is to ensure that obtaining your
#' client credentials is more work than just creating a new client.
#'
#' @export
#' @family OAuth flows
Expand Down
4 changes: 4 additions & 0 deletions R/oauth-flow.R
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ oauth_flow_parse <- function(resp, source, error_call = caller_env()) {
cli::cli_abort(
"Failed to parse response from {.arg {source}} OAuth url.",
parent = err,
resp = resp,
class = "httr2_oauth_parse",
call = error_call
)
}
Expand Down Expand Up @@ -40,6 +42,8 @@ oauth_flow_parse <- function(resp, source, error_call = caller_env()) {
"Failed to parse response from {.arg {source}} OAuth url.",
"*" = "Did not contain {.code access_token}, {.code device_code}, or {.code error} field."
),
resp = resp,
class = "httr2_oauth_parse",
call = error_call
)
}
Expand Down
3 changes: 0 additions & 3 deletions R/progress-bars.R
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,6 @@
#' By default the same as `format`.
#' * `name`: progress bar name. This is by default the empty string and it
#' is displayed at the beginning of the progress bar.
#' * `show_after`: numeric scalar. Only show the progress bar after this
#' number of seconds. It overrides the `cli.progress_show_after`
#' global option.
#' * `type`: progress bar type. Currently supported types are:
#' * `iterator`: the default, a for loop or a mapping function,
#' * `tasks`: a (typically small) number of tasks,
Expand Down
4 changes: 2 additions & 2 deletions R/req-auth.R
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
#'
#' @inheritParams req_perform
#' @param username User name.
#' @param password Password. You avoid entering the password directly when
#' calling this function as it will be captured by `.Rhistory`. Instead,
#' @param password Password. You should avoid entering the password directly
#' when calling this function as it will be captured by `.Rhistory`. Instead,
#' leave it unset and the default behaviour will prompt you for it
#' interactively.
#' @returns A modified HTTP [request].
Expand Down
3 changes: 2 additions & 1 deletion R/req-body.R
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,8 @@ req_body_json_modify <- function(req, ...) {
#' data in the body.
#'
#' * For `req_body_form()`, the values must be strings (or things easily
#' coerced to strings);
#' coerced to strings). Vectors are convertd to strings using the
#' value of `.multi`.
#' * For `req_body_multipart()` the values must be strings or objects
#' produced by [curl::form_file()]/[curl::form_data()].
#' * For `req_body_json_modify()`, any simple data made from atomic vectors
Expand Down
4 changes: 4 additions & 0 deletions R/req-perform.R
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,10 @@ last_request <- function() {
#' works by sending the real HTTP request to a local webserver, thanks to
#' the magic of [curl::curl_echo()].
#'
#' ## Limitations
#'
#' * The `Host` header is not respected.
#'
#' @inheritParams req_verbose
#' @param quiet If `TRUE` doesn't print anything.
#' @returns Invisibly, a list containing information about the request,
Expand Down
65 changes: 36 additions & 29 deletions R/req-retries.R
Original file line number Diff line number Diff line change
@@ -1,55 +1,55 @@
#' Control when a request will retry, and how long it will wait between tries
#' Automatically retry a request on failure
#'
#' @description
#' `req_retry()` alters [req_perform()] so that it will automatically retry
#' in the case of failure. To activate it, you must specify either the total
#' number of requests to make with `max_tries` or the total amount of time
#' to spend with `max_seconds`. Then `req_perform()` will retry if the error is
#' "transient", i.e. it's an HTTP error that can be resolved by waiting. By
#' default, 429 and 503 statuses are treated as transient, but if the API you
#' are wrapping has other transient status codes (or conveys transient-ness
#' with some other property of the response), you can override the default
#' with `is_transient`.
#' `req_retry()` allows [req_perform()] to automatically retry failing
#' requests. It's particularly important for APIs with rate limiting, but can
#' also be useful when dealing with flaky servers.
#'
#' Additionally, if you set `retry_on_failure = TRUE`, the request will retry
#' if either the HTTP request or HTTP response doesn't complete successfully
#' By default, `req_perform()` will retry if the response is a 429
#' ("too many requests", often used for rate limiting) or 503
#' ("service unavailable"). If the API you are wrapping has other transient
#' status codes (or conveys transience with some other property of the
#' response), you can override the default with `is_transient`. And
#' if you set `retry_on_failure = TRUE`, the request will retry
#' if either the HTTP request or HTTP response doesn't complete successfully,
#' leading to an error from curl, the lower-level library that httr2 uses to
#' perform HTTP request. This occurs, for example, if your wifi is down.
#' perform HTTP requests. This occurs, for example, if your Wi-Fi is down.
#'
#' ## Delay
#'
#' It's a bad idea to immediately retry a request, so `req_perform()` will
#' wait a little before trying again:
#'
#' * If the response contains the `Retry-After` header, httr2 will wait the
#' amount of time it specifies. If the API you are wrapping conveys this
#' information with a different header (or other property of the response)
#' you can override the default behaviour with `retry_after`.
#' information with a different header (or other property of the response),
#' you can override the default behavior with `retry_after`.
#'
#' * Otherwise, httr2 will use "truncated exponential backoff with full
#' jitter", i.e. it will wait a random amount of time between one second and
#' `2 ^ tries` seconds, capped to at most 60 seconds. In other words, it
#' jitter", i.e., it will wait a random amount of time between one second and
#' `2 ^ tries` seconds, capped at a maximum of 60 seconds. In other words, it
#' waits `runif(1, 1, 2)` seconds after the first failure, `runif(1, 1, 4)`
#' after the second, `runif(1, 1, 8)` after the third, and so on. If you'd
#' prefer a different strategy, you can override the default with `backoff`.
#'
#' @inheritParams req_perform
#' @param max_tries,max_seconds Cap the maximum number of attempts with
#' `max_tries` or the total elapsed time from the first request with
#' `max_seconds`. If neither option is supplied (the default), [req_perform()]
#' will not retry.
#' @param max_tries,max_seconds Cap the maximum number of attempts
#' (`max_tries`), the total elapsed time from the first request
#' (`max_seconds`), or both.
#'
#' `max_tries` is the total number of attempts make, so this should always
#' be greater than one.`
#' `max_tries` is the total number of attempts made, so this should always
#' be greater than one.
#' @param is_transient A predicate function that takes a single argument
#' (the response) and returns `TRUE` or `FALSE` specifying whether or not
#' the response represents a transient error.
#' @param retry_on_failure Treat low-level failures as if they are
#' transient errors, and can be retried.
#' transient errors that can be retried.
#' @param backoff A function that takes a single argument (the number of failed
#' attempts so far) and returns the number of seconds to wait.
#' @param after A function that takes a single argument (the response) and
#' returns either a number of seconds to wait or `NULL`, which indicates
#' that a precise wait time is not available that the `backoff` strategy
#' should be used instead..
#' returns either a number of seconds to wait or `NA`. `NA` indicates
#' that a precise wait time is not available and that the `backoff` strategy
#' should be used instead.
#' @returns A modified HTTP [request].
#' @export
#' @seealso [req_throttle()] if the API has a rate-limit but doesn't expose
Expand All @@ -61,7 +61,7 @@
#'
#' # use a constant 10s delay after every failure
#' request("http://example.com") |>
#' req_retry(backoff = ~10)
#' req_retry(backoff = \(resp) 10)
#'
#' # When rate-limited, GitHub's API returns a 403 with
#' # `X-RateLimit-Remaining: 0` and an Unix time stored in the
Expand All @@ -86,9 +86,16 @@ req_retry <- function(req,
is_transient = NULL,
backoff = NULL,
after = NULL) {

check_request(req)
check_number_whole(max_tries, min = 2, allow_null = TRUE)
check_number_whole(max_tries, min = 1, allow_null = TRUE)
check_number_whole(max_seconds, min = 0, allow_null = TRUE)

if (is.null(max_tries) && is.null(max_seconds)) {
max_tries <- 2
cli::cli_inform("Setting {.code max_tries = 2}.")
}

check_bool(retry_on_failure)

req_policies(req,
Expand Down
7 changes: 4 additions & 3 deletions R/req-url.R
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,11 @@ req_url <- function(req, url) {
#' * `"error"`, the default, throws an error.
#' * `"comma"`, separates values with a `,`, e.g. `?x=1,2`.
#' * `"pipe"`, separates values with a `|`, e.g. `?x=1|2`.
#' * `"explode"`, turns each element into its own parameter, e.g. `?x=1&x=2`.
#' * `"explode"`, turns each element into its own parameter, e.g. `?x=1&x=2`
#'
#' If none of these functions work, you can alternatively supply a function
#' that takes a character vector and returns a string.
#' If none of these options work for your needs, you can instead supply a
#' function that takes a character vector of argument values and returns a
#' a single string.
req_url_query <- function(.req,
...,
.multi = c("error", "comma", "pipe", "explode")) {
Expand Down
11 changes: 7 additions & 4 deletions R/sequential.R
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,19 @@
#'
#' @param reqs A list of [request]s.
#' @param paths An optional character vector of paths, if you want to download
#' the request bodies to disk. If supplied, must be the same length as `reqs`.
#' the response bodies to disk. If supplied, must be the same length as
#' `reqs`.
#' @param on_error What should happen if one of the requests fails?
#'
#' * `stop`, the default: stop iterating with an error.
#' * `return`: stop iterating, returning all the successful responses
#' received so far, as well as an error object for the failed request.
#' * `continue`: continue iterating, recording errors in the result.
#' @param progress Display a progress bar? Use `TRUE` to turn on a basic
#' progress bar, use a string to give it a name, or see [progress_bars] to
#' customise it in other ways.
#' @param progress Display a progress bar for the status of all requests? Use
#' `TRUE` to turn on a basic progress bar, use a string to give it a name,
#' or see [progress_bars] to customize it in other ways. Not compatible with
#' [req_progress()], as httr2 can only display a single progress bar at a
#' time.
#' @return
#' A list, the same length as `reqs`, containing [response]s and possibly
#' error objects, if `on_error` is `"return"` or `"continue"` and one of the
Expand Down
2 changes: 1 addition & 1 deletion cran-comments.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@

## revdepcheck results

This is a patch release that only adds new features and fixes a randomly failing test. Compared to the last release I have provided a patch for osmapiR: https://github.com/ropensci/osmapiR/pull/58
This is a patch release that only add a new feature and fixes a rarely used function.
3 changes: 2 additions & 1 deletion man/multi_req_perform.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 0 additions & 3 deletions man/progress_bars.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions man/req_auth_basic.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 6 additions & 4 deletions man/req_body.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions man/req_dry_run.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 0e71e83

Please sign in to comment.