[BUG] The error capturing mechanism is lossy in an edge case #307

chainsawriot · 2022-03-10T20:21:40Z

Please confirm the following

I have searched the existing issues
The behaviour of the program is deviated from what is described in the documentation.
I can reproduce this problem for more than one time.
This is NOT a 3-digit error -- it does not display an error message like something went wrong. Status code: 400.
This is a 3-digit error and I have consulted the Understanding API errors vignette and the suggestions do not help.

Describe the bug

The error capturing mechanism introduced since 1404bc3 (#264) is lossy in an edge case when at least two batches are all errors. It is due to the so-called "dirty road" problem: in the case of all errors, there is no data slot for df_to_json and thus the file name storing errors content will be errors_.json. When the second batch of all errors needs to be saved, it will also save as errors_.json and overwrite the old one.

academictwitteR/R/utils.R

Lines 152 to 163 in 2809432

    
           df_to_json <- function(df, data_path, errors = FALSE){ 
        
             # check input 
        
             # if data path is supplied and file name given, generate data.frame object within loop and JSONs 
        
             jsonlite::write_json(df$data, 
        
                                  file.path(data_path, paste0("data_", df$data$id[nrow(df$data)], ".json"))) 
        
             jsonlite::write_json(df$includes, 
        
                                  file.path(data_path, paste0("users_", df$data$id[nrow(df$data)], ".json"))) 
        
             if (errors) { # error catcher 
        
               jsonlite::write_json(df$errors, 
        
                                    file.path(data_path, paste0("errors_", df$data$id[nrow(df$data)], ".json"))) 
        
             } 
        
           }

It is related to both #304 #305

Expected Behavior

I don't know. @TimBMK any thought?

Steps To Reproduce

In the following example, the vector dirty contains 200 bogus tweet ids. But the error capturing mechanism can only capture 100.

require(academictwitteR)
#> Loading required package: academictwitteR

FFF <- c("1266876474440761346", "1266868259925737474", "1266867327079002121", "1266866660713127936", "1266864490446012418", "1266860737244336129", "1266859737615826944", "1266859455586676736", "1266858090143588352", "1266857669157097473")

bogus <- rep("1", 200)

dirty <- c(bogus, FFF)
dirty
#>   [1] "1"                   "1"                   "1"                  
#>   [4] "1"                   "1"                   "1"                  
#>   [7] "1"                   "1"                   "1"                  
#>  [10] "1"                   "1"                   "1"                  
#>  [13] "1"                   "1"                   "1"                  
#>  [16] "1"                   "1"                   "1"                  
#>  [19] "1"                   "1"                   "1"                  
#>  [22] "1"                   "1"                   "1"                  
#>  [25] "1"                   "1"                   "1"                  
#>  [28] "1"                   "1"                   "1"                  
#>  [31] "1"                   "1"                   "1"                  
#>  [34] "1"                   "1"                   "1"                  
#>  [37] "1"                   "1"                   "1"                  
#>  [40] "1"                   "1"                   "1"                  
#>  [43] "1"                   "1"                   "1"                  
#>  [46] "1"                   "1"                   "1"                  
#>  [49] "1"                   "1"                   "1"                  
#>  [52] "1"                   "1"                   "1"                  
#>  [55] "1"                   "1"                   "1"                  
#>  [58] "1"                   "1"                   "1"                  
#>  [61] "1"                   "1"                   "1"                  
#>  [64] "1"                   "1"                   "1"                  
#>  [67] "1"                   "1"                   "1"                  
#>  [70] "1"                   "1"                   "1"                  
#>  [73] "1"                   "1"                   "1"                  
#>  [76] "1"                   "1"                   "1"                  
#>  [79] "1"                   "1"                   "1"                  
#>  [82] "1"                   "1"                   "1"                  
#>  [85] "1"                   "1"                   "1"                  
#>  [88] "1"                   "1"                   "1"                  
#>  [91] "1"                   "1"                   "1"                  
#>  [94] "1"                   "1"                   "1"                  
#>  [97] "1"                   "1"                   "1"                  
#> [100] "1"                   "1"                   "1"                  
#> [103] "1"                   "1"                   "1"                  
#> [106] "1"                   "1"                   "1"                  
#> [109] "1"                   "1"                   "1"                  
#> [112] "1"                   "1"                   "1"                  
#> [115] "1"                   "1"                   "1"                  
#> [118] "1"                   "1"                   "1"                  
#> [121] "1"                   "1"                   "1"                  
#> [124] "1"                   "1"                   "1"                  
#> [127] "1"                   "1"                   "1"                  
#> [130] "1"                   "1"                   "1"                  
#> [133] "1"                   "1"                   "1"                  
#> [136] "1"                   "1"                   "1"                  
#> [139] "1"                   "1"                   "1"                  
#> [142] "1"                   "1"                   "1"                  
#> [145] "1"                   "1"                   "1"                  
#> [148] "1"                   "1"                   "1"                  
#> [151] "1"                   "1"                   "1"                  
#> [154] "1"                   "1"                   "1"                  
#> [157] "1"                   "1"                   "1"                  
#> [160] "1"                   "1"                   "1"                  
#> [163] "1"                   "1"                   "1"                  
#> [166] "1"                   "1"                   "1"                  
#> [169] "1"                   "1"                   "1"                  
#> [172] "1"                   "1"                   "1"                  
#> [175] "1"                   "1"                   "1"                  
#> [178] "1"                   "1"                   "1"                  
#> [181] "1"                   "1"                   "1"                  
#> [184] "1"                   "1"                   "1"                  
#> [187] "1"                   "1"                   "1"                  
#> [190] "1"                   "1"                   "1"                  
#> [193] "1"                   "1"                   "1"                  
#> [196] "1"                   "1"                   "1"                  
#> [199] "1"                   "1"                   "1266876474440761346"
#> [202] "1266868259925737474" "1266867327079002121" "1266866660713127936"
#> [205] "1266864490446012418" "1266860737244336129" "1266859737615826944"
#> [208] "1266859455586676736" "1266858090143588352" "1266857669157097473"

emptydir <- academictwitteR:::.gen_random_dir()
hydrate_tweets(ids = dirty, data_path = emptydir, bind_tweets = FALSE, errors = TRUE)
#> Batch 1 out of 3 : ids 1 to 1 
#> Batch 2 out of 3 : ids 1 to 1 
#> Batch 3 out of 3 : ids 1266876474440761346 to 1266857669157097473

list.files(emptydir)
#> [1] "data_.json"                      "data_1266857669157097473.json"  
#> [3] "errors_.json"                    "errors_1266857669157097473.json"
#> [5] "users_.json"                     "users_1266857669157097473.json"

bind_tweets(emptydir)
#> ================================================================================
#>        conversation_id                  id public_metrics.retweet_count
#> 1  1266876474440761346 1266876474440761346                            2
#> 2  1266868259925737474 1266868259925737474                           98
#> 3  1266867327079002121 1266867327079002121                            0
#> 4  1266866660713127936 1266866660713127936                          150
#> 5  1266864490446012418 1266864490446012418                          150
#> 6  1266860737244336129 1266860737244336129                           98
#> 7  1266859737615826944 1266859737615826944                           33
#> 8  1266859455586676736 1266859455586676736                           98
#> 9  1266858090143588352 1266858090143588352                            2
#> 10 1266857669157097473 1266857669157097473                           98
#>    public_metrics.reply_count public_metrics.like_count
#> 1                           0                         3
#> 2                           0                         0
#> 3                           1                         5
#> 4                           0                         0
#> 5                           0                         0
#> 6                           0                         0
#> 7                           0                         0
#> 8                           0                         0
#> 9                           0                         0
#> 10                          0                         0
#>    public_metrics.quote_count lang
#> 1                           0   de
#> 2                           0   de
#> 3                           0   de
#> 4                           0   de
#> 5                           0   de
#> 6                           0   de
#> 7                           0   de
#> 8                           0   de
#> 9                           0   de
#> 10                          0   de
#>                                                                                                                                                                                                                                                                                                                 text
#> 1                     diese Petition mitzeichnen!!\n. \nKonsequente Ausrichtung eines zukünftigen Konjunkturpakets.\nNur soziale und technische Innovationen, #Klimaschutz und gesetzlich verankerte Gemeinwohlorientierung machen Deutschland zukunftsfähig.\n#Datteln4\n#FridaysForFuture\nhttps://t.co/eZ8j61yftj
#> 2                                                                                                                                                                       RT @TheReal32492440: #Soros hat es angekündigt: mit #Geld sorgt er für Unruhen und Zerstörung. Dieser Teufel finanziert #NGOs, Zeitungen un…
#> 3  Diese Rücksichtslosigkeit kotzt mich langsam an: Nein, es ist total uncool, seinen Müll wegzuwerfen. \n\nIch sagte schon zu Zeiten von #FridaysForFuture , dass es sinnvoller wäre, am FR Nachmittag mit Müllbeuteln bewaffnet die REGIONALE Natur zu säubern, um ein Gefühl zu erhalten. https://t.co/lT2YVFpXit
#> 4                                                                                                                                                                        RT @bremenforfuture: Bei d. friedlichen Protesten um #Datteln4 hat d. @polizei_nrw_ge gerade 5 Aktivist*innen aus d. Masse gezogen, nieder…
#> 5                                                                                                                                                                        RT @bremenforfuture: Bei d. friedlichen Protesten um #Datteln4 hat d. @polizei_nrw_ge gerade 5 Aktivist*innen aus d. Masse gezogen, nieder…
#> 6                                                                                                                                                                       RT @TheReal32492440: #Soros hat es angekündigt: mit #Geld sorgt er für Unruhen und Zerstörung. Dieser Teufel finanziert #NGOs, Zeitungen un…
#> 7                                                                                                                                                                       RT @fff_2209: Das Kohleausstiegsgesetz darf so nicht verabschiedet werden. Macht auf das Thema aufmerksam und schreibt Euren Bundestagsabge…
#> 8                                                                                                                                                                       RT @TheReal32492440: #Soros hat es angekündigt: mit #Geld sorgt er für Unruhen und Zerstörung. Dieser Teufel finanziert #NGOs, Zeitungen un…
#> 9                                                                                                                                                                       RT @wlkutsch: Gestern hatten wir ein CO2FFEE mit @ICOS_RI über die Arktis: Wissenschaft: wir müssen Emissionen schnell reduzieren. Politik:…
#> 10                                                                                                                                                                      RT @TheReal32492440: #Soros hat es angekündigt: mit #Geld sorgt er für Unruhen und Zerstörung. Dieser Teufel finanziert #NGOs, Zeitungen un…
#>                 source
#> 1  Twitter for Android
#> 2     Twitter for iPad
#> 3   Twitter for iPhone
#> 4  Twitter for Android
#> 5  Twitter for Android
#> 6   Twitter for iPhone
#> 7  Twitter for Android
#> 8      Twitter Web App
#> 9   Twitter for iPhone
#> 10 Twitter for Android
#>                                                        entities.hashtags
#> 1  133, 229, 239, 145, 238, 256, Klimaschutz, Datteln4, FridaysForFuture
#> 2                            21, 52, 120, 27, 57, 125, Soros, Geld, NGOs
#> 3                                             133, 150, FridaysForFuture
#> 4                                                       53, 62, Datteln4
#> 5                                                       53, 62, Datteln4
#> 6                            21, 52, 120, 27, 57, 125, Soros, Geld, NGOs
#> 7                                                                   NULL
#> 8                            21, 52, 120, 27, 57, 125, Soros, Geld, NGOs
#> 9                                                                   NULL
#> 10                           21, 52, 120, 27, 57, 125, Soros, Geld, NGOs
#>                                                                                                                                                                entities.urls
#> 1  257, 280, https://t.co/eZ8j61yftj, https://epetitionen.bundestag.de/petitionen/_2020/_04/_21/Petition_110043.%24%24%24.a.u.html, epetitionen.bundestag.de/petitionen/_20…
#> 2                                                                                                                                                                       NULL
#> 3                                         280, 303, https://t.co/lT2YVFpXit, https://twitter.com/Mr_Twoflower/status/1266867327079002121/photo/1, pic.twitter.com/lT2YVFpXit
#> 4                                                                                                                                                                       NULL
#> 5                                                                                                                                                                       NULL
#> 6                                                                                                                                                                       NULL
#> 7                                                                                                                                                                       NULL
#> 8                                                                                                                                                                       NULL
#> 9                                                                                                                                                                       NULL
#> 10                                                                                                                                                                      NULL
#>                                                                  entities.mentions
#> 1                                                                             NULL
#> 2                                      3, 19, TheReal32492440, 1155848155893837825
#> 3                                                                             NULL
#> 4  3, 70, 19, 85, bremenforfuture, polizei_nrw_ge, 1080927540422815744, 2389272182
#> 5  3, 70, 19, 85, bremenforfuture, polizei_nrw_ge, 1080927540422815744, 2389272182
#> 6                                      3, 19, TheReal32492440, 1155848155893837825
#> 7                                                                             NULL
#> 8                                      3, 19, TheReal32492440, 1155848155893837825
#> 9                         3, 49, 12, 57, wlkutsch, ICOS_RI, 2816669118, 3216719140
#> 10                                     3, 19, TheReal32492440, 1155848155893837825
#>              author_id possibly_sensitive               created_at
#> 1            282135623              FALSE 2020-05-30T23:37:43.000Z
#> 2  1239879547409006592              FALSE 2020-05-30T23:05:05.000Z
#> 3           3850413016               TRUE 2020-05-30T23:01:22.000Z
#> 4            179629368              FALSE 2020-05-30T22:58:43.000Z
#> 5   728209251190120449              FALSE 2020-05-30T22:50:06.000Z
#> 6           3844151483              FALSE 2020-05-30T22:35:11.000Z
#> 7             61153698              FALSE 2020-05-30T22:31:13.000Z
#> 8   871879419559903232              FALSE 2020-05-30T22:30:05.000Z
#> 9  1180995990284853248              FALSE 2020-05-30T22:24:40.000Z
#> 10 1242872854925975552              FALSE 2020-05-30T22:23:00.000Z
#>                 referenced_tweets            media_keys
#> 1                            NULL                  NULL
#> 2  retweeted, 1266816466512297984                  NULL
#> 3                            NULL 3_1266867313988509697
#> 4  retweeted, 1266704570190790656                  NULL
#> 5  retweeted, 1266704570190790656                  NULL
#> 6  retweeted, 1266816466512297984                  NULL
#> 7  retweeted, 1266817176687779846                  NULL
#> 8  retweeted, 1266816466512297984                  NULL
#> 9  retweeted, 1266779244484575232                  NULL
#> 10 retweeted, 1266816466512297984                  NULL

## just 100 rows; not 200
purrr::map_dfr(list.files(emptydir, "^error", full.names = TRUE), jsonlite::read_json)
#> # A tibble: 100 × 7
#>    value detail                  title resource_type parameter resource_id type 
#>    <chr> <chr>                   <chr> <chr>         <chr>     <chr>       <chr>
#>  1 1     Could not find tweet w… Not … tweet         ids       1           http…
#>  2 1     Could not find tweet w… Not … tweet         ids       1           http…
#>  3 1     Could not find tweet w… Not … tweet         ids       1           http…
#>  4 1     Could not find tweet w… Not … tweet         ids       1           http…
#>  5 1     Could not find tweet w… Not … tweet         ids       1           http…
#>  6 1     Could not find tweet w… Not … tweet         ids       1           http…
#>  7 1     Could not find tweet w… Not … tweet         ids       1           http…
#>  8 1     Could not find tweet w… Not … tweet         ids       1           http…
#>  9 1     Could not find tweet w… Not … tweet         ids       1           http…
#> 10 1     Could not find tweet w… Not … tweet         ids       1           http…
#> # … with 90 more rows

^{Created on 2022-03-10 by the reprex package (v2.0.1)}

Environment

require(academictwitteR)
#> Loading required package: academictwitteR
sessionInfo()
#> R version 4.1.2 (2021-11-01)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 20.04.4 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
#> LAPACK: /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3.10.3
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=de_DE.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=de_DE.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=de_DE.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C       
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] academictwitteR_0.3.1
#> 
#> loaded via a namespace (and not attached):
#>  [1] knitr_1.37        magrittr_2.0.2    usethis_2.1.5     R.cache_0.15.0   
#>  [5] rlang_1.0.2       fastmap_1.1.0     fansi_1.0.2       stringr_1.4.0    
#>  [9] styler_1.6.2      highr_0.9         tools_4.1.2       xfun_0.29        
#> [13] R.oo_1.24.0       utf8_1.2.2        cli_3.2.0         withr_2.5.0      
#> [17] htmltools_0.5.2   ellipsis_0.3.2    yaml_2.3.5        digest_0.6.29    
#> [21] tibble_3.1.6      lifecycle_1.0.1   crayon_1.5.0      purrr_0.3.4      
#> [25] R.utils_2.11.0    vctrs_0.3.8       fs_1.5.2          glue_1.6.2       
#> [29] evaluate_0.14     rmarkdown_2.11    reprex_2.0.1      stringi_1.7.6    
#> [33] compiler_4.1.2    pillar_1.7.0      R.methodsS3_1.8.1 backports_1.4.1  
#> [37] pkgconfig_2.0.3

^{Created on 2022-03-10 by the reprex package (v2.0.1)}

Anything else?

No response

The text was updated successfully, but these errors were encountered:

TimBMK · 2022-03-11T09:26:59Z

I guess the expected behaviour according to current naming conventions (pre #308 ) would be to generate the filename by taking the last ID from the search query rather than the result. This would lead to the files being named correctly regardless of them being empty. If the error files are empty, too, we could even consider adding a dummy message to the error file, indicating that there were no returns.

I think some kind of arbitrary naming convention to prevent these files from overwriting each other should be implemented regardless of whether we keep the current convention or employ hashes. Even if they didn't produce errors, these empty files may lead to confusion on the user side

TimBMK · 2022-03-11T10:51:16Z

I've done some testing and it seems the error_.json files are always empty. This seems strange, since I would expect the Twitter API to respond with some kind of error if no tweets can be retrieved. Any idea what the API response in these cases would be?

chainsawriot · 2022-03-11T11:12:12Z

@TimBMK It is not always empty. For example, in the case above the errors_.json is not empty.

require(academictwitteR)
#> Loading required package: academictwitteR

FFF <- c("1266876474440761346", "1266868259925737474", "1266867327079002121", "1266866660713127936", "1266864490446012418", "1266860737244336129", "1266859737615826944", "1266859455586676736", "1266858090143588352", "1266857669157097473")

bogus <- rep("1", 200)

dirty <- c(bogus, FFF)

emptydir <- academictwitteR:::.gen_random_dir()
hydrate_tweets(ids = dirty, data_path = emptydir, bind_tweets = FALSE, errors = TRUE)
#> Batch 1 out of 3 : ids 1 to 1 
#> Batch 2 out of 3 : ids 1 to 1 
#> Batch 3 out of 3 : ids 1266876474440761346 to 1266857669157097473

jsonlite::read_json(file.path(emptydir, "errors_.json"), simplifyVector = TRUE)
#>     value                              detail           title resource_type
#> 1       1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 2       1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 3       1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 4       1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 5       1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 6       1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 7       1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 8       1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 9       1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 10      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 11      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 12      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 13      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 14      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 15      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 16      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 17      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 18      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 19      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 20      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 21      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 22      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 23      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 24      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 25      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 26      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 27      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 28      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 29      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 30      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 31      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 32      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 33      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 34      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 35      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 36      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 37      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 38      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 39      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 40      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 41      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 42      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 43      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 44      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 45      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 46      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 47      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 48      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 49      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 50      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 51      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 52      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 53      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 54      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 55      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 56      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 57      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 58      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 59      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 60      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 61      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 62      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 63      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 64      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 65      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 66      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 67      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 68      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 69      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 70      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 71      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 72      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 73      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 74      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 75      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 76      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 77      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 78      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 79      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 80      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 81      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 82      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 83      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 84      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 85      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 86      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 87      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 88      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 89      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 90      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 91      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 92      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 93      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 94      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 95      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 96      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 97      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 98      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 99      1 Could not find tweet with ids: [1]. Not Found Error         tweet
#> 100     1 Could not find tweet with ids: [1]. Not Found Error         tweet
#>     parameter resource_id                                                  type
#> 1         ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 2         ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 3         ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 4         ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 5         ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 6         ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 7         ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 8         ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 9         ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 10        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 11        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 12        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 13        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 14        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 15        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 16        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 17        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 18        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 19        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 20        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 21        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 22        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 23        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 24        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 25        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 26        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 27        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 28        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 29        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 30        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 31        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 32        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 33        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 34        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 35        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 36        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 37        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 38        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 39        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 40        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 41        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 42        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 43        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 44        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 45        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 46        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 47        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 48        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 49        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 50        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 51        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 52        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 53        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 54        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 55        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 56        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 57        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 58        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 59        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 60        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 61        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 62        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 63        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 64        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 65        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 66        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 67        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 68        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 69        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 70        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 71        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 72        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 73        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 74        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 75        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 76        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 77        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 78        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 79        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 80        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 81        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 82        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 83        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 84        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 85        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 86        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 87        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 88        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 89        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 90        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 91        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 92        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 93        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 94        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 95        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 96        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 97        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 98        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 99        ids           1 https://api.twitter.com/2/problems/resource-not-found
#> 100       ids           1 https://api.twitter.com/2/problems/resource-not-found

^{Created on 2022-03-11 by the reprex package (v2.0.1)}

TimBMK · 2022-03-11T12:15:03Z

Interesting! In this case I would definetly opt for a stringent naming convention based on either the search query (last ID) or a hash. This would allow users to track issues and find out why their tweets were not retrieved.

I'm looking at implementing the error capture mechanism more stringently into other functions such as get_user_timeline() at the moment. For this one at least, the errors seem to be empty in some cases. But this might be related to the actual API response, then.

chainsawriot mentioned this issue Mar 10, 2022

[FR] Next generation archiving architecture #308

Open

TimBMK mentioned this issue Mar 11, 2022

Update error capturing #309

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] The error capturing mechanism is lossy in an edge case #307

[BUG] The error capturing mechanism is lossy in an edge case #307

chainsawriot commented Mar 10, 2022 •

edited

Loading

TimBMK commented Mar 11, 2022

TimBMK commented Mar 11, 2022

chainsawriot commented Mar 11, 2022

TimBMK commented Mar 11, 2022

[BUG] The error capturing mechanism is lossy in an edge case #307

[BUG] The error capturing mechanism is lossy in an edge case #307

Comments

chainsawriot commented Mar 10, 2022 • edited Loading

Please confirm the following

Describe the bug

Expected Behavior

Steps To Reproduce

Environment

Anything else?

TimBMK commented Mar 11, 2022

TimBMK commented Mar 11, 2022

chainsawriot commented Mar 11, 2022

TimBMK commented Mar 11, 2022

chainsawriot commented Mar 10, 2022 •

edited

Loading