Performance gains when converting xml structure to df

[StefanSchellerer]

While working with the new IMF Data API, I explored its documentation, which references the rsdmx package for use with R: IMF API Documentation. However, using this approach can lead to significant waiting times due to the transformation of XML objects into R data.frames. To address this issue, I developed a more efficient solution. Instead of relying on (nested) loops, which can be time-consuming, my approach leverages the length of the XML objects to efficiently join the "Series" attributes with their associated "Obs" elements.

Feel free to use this code to potentially improve the rsdmx performance.

fetch_IMF_data <- function(agency_id, dataset_id, key = NULL, start_period = NULL, end_period = NULL) {
  
  url <- paste0('https://api.imf.org/external/sdmx/2.1/data/', agency_id, ',', dataset_id, '/',
                if(!is.null(key)) paste0(key, '/'),
                if(!is.null(start_period)) paste0('?startPeriod=', start_period),
                if(!is.null(end_period)) paste0('&endPeriod=', end_period))
  
  resp <- curl::curl_fetch_memory(url)
  content <- resp$content |> rawToChar()
  xml_content <- xml2::read_xml(content)
  series_nodes <- xml2::xml_find_all(xml_content, './/Series[.//Obs]')
  series_attrs <- xml2::xml_attrs(series_nodes)
  obs_nodes <- xml2::xml_find_all(xml_content, './/Series//Obs')
  obs_attrs <- xml2::xml_attrs(obs_nodes)
  num_obs <- xml2::xml_length(series_nodes)

  obs_dt <- data.table::rbindlist(
    lapply(obs_attrs, as.list),
    fill = TRUE
  )[, idx := rep(seq_along(num_obs), num_obs)]
  
  series_dt <- data.table::rbindlist(
    lapply(series_attrs, as.list),
    fill = TRUE
  )[, idx := seq_len(.N)]
  
  series_dt[obs_dt, on = 'idx'][, idx := NULL]
}

# example usage:
data <- fetch_IMF_data(agency_id = 'IMF.STA', dataset_id = 'CPI')


microbenchmark::microbenchmark(
  
  # fast conversion
  fetch_IMF_data(agency_id = 'IMF.STA', dataset_id = 'CPI'),
  
  # rsdmx to df
  as.data.frame(rsdmx::readSDMX(providerId = 'IMF_DATA',
                                resource = 'data',
                                flowRef = 'IMF.STA,CPI')),
  
  times = 1L
)
Unit: seconds
                                                                                                     expr       min        lq
                                                fetch_IMF_data(agency_id = "IMF.STA", dataset_id = "CPI")  118.8345  118.8345
 as.data.frame(rsdmx::readSDMX(providerId = "IMF_DATA", resource = "data",      flowRef = "IMF.STA,CPI")) 2323.8632 2323.8632
      mean    median        uq       max neval
  118.8345  118.8345  118.8345  118.8345     1
 2323.8632 2323.8632 2323.8632 2323.8632     1

Best wishes!

[eblondel]

Thanks for the proposal. There is a ticket open here to suggest move from XML to xml2 #196. Feel free to create a PR to support this, but in that case it needs to be systematic on all classes/methods of the R package, not specific to one type of data and provider.