New Yorker API intermittently responds with 403s

[thisisparker]

Specifically, the API endpoint appears to sometimes spit 403s when requested from cloud IP addresses — I am so far consistently getting good responses from my home network. But the status check Action here is ~50% 403s, and it's entirely in clusters: today's automated check was all green, which included 8 requests to that API, but the one triggered by a recent small PR had 8 failures.

I've observed similar behavior from a DigitalOcean IP.

It's this part of the downloader in particular, which already includes a kind of hacky approach to setting a custom User-Agent, so maybe there's more I can do there.

Otherwise, I might try just implementing a set of retries, though the clustering behavior makes me pretty pessimistic about that working.

[thisisparker]

I don't want to jinx it but this appears to have calmed down a bit since opening the issue. Will continue to monitor...

[EG-tech]

FWIW I consistently get 403s on The New Yorker from a DigitalOcean IP. no problems at all from home network, might be able to try from an AWS machine to see if things are any different there!

[thisisparker]

Yeah, similarly I've never had a problem from home but I'm still seeing all failures about half the time from GitHub Action runners which are presumably in a known IP address block.

I think there's not much to do about it, but maybe I should update the error on a 403 to give that context

[danielpunkass]

I was curious so I tested from a Linode (Akamai) hosted IP, and it consistently reproduces there, too.

[danielpunkass]

I set up a SOCKS proxy to route all internet traffic through my Linode, so I could see how the New Yorker web page handles the setup. Indeed, it fails to load the puzzle, and displays an unending "Loading" animation. The web inspector reveals log messages about 403 authorization failures. So this suggests to me there is no "missing key" that would allow xword-dl to avoid this failure when going through one of the affected IP addresses.

[thisisparker]

Yeah, if it's failing in a regular browser then I'm pretty well convinced this is at the IP level. Thanks for checking! I think that means that I put a note in the error message in case of 403s in particular. People who are using xword-dl on a VPS can probably figure out what to do from there on their own!

[thisisparker]

And on a separate note, it may be worth running the Github Action status check through a residential proxy, but I will open another issue for that.

[danielpunkass]

I was thinking along the same lines, and wondered if there are any public, free tier VPN or proxy services that happen to get through to the API.

[danielpunkass]

Please take a look at my PR #303 ... if you don't have a major aversion to running an free-tier AWS instance, this might be a good workaround for the issue at least for testing purposes.