Bug: Dorothy installation can fail if GitHub API Rate Limit is encountered again after the wait period

[edmcman]

This is not a great install experience:

It's a little hard to see, but note the 16 minutes remaining

[edmcman]

After waiting, it failed anyway

Now it's stuck with zero information about what it's doing. I can tell from top that it's compiling rust crates, but more visibility from the installer would be nice.

[edmcman]

Also (sorry for piling on with constructive criticism)... it's really not great that deno is being built in the first place. I'm installing on a raspberry pi. It's been building deno for over 10 minutes now.

Installing Dorothy should ideally be:

1. Fast
2. Minimal -- I don't want really want deno or rust installed.
3. Small. Rust and cargo is using 4.2 GiB and it's not even complete.
4. Transparent. I should know everything that the install process is doing before it happens.

Again, I apologize for all the criticism. But I really just wanted to install dorothy quickly to gain access to one of my custom commands. This was not what I expected or wanted at all! Perhaps speed and disk space used could also be measured by the CI process and considered.

[edmcman]

Eventually the whole thing failed:

error[E0433]: failed to resolve: use of undeclared crate or module `flate2`
  --> /home/ed/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zip-2.2.2/src/write.rs:35:5
   |
35 | use flate2::{write::DeflateEncoder, Compression};
   |     ^^^^^^ use of undeclared crate or module `flate2`

error[E0432]: unresolved import `flate2`
  --> /home/ed/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zip-2.2.2/src/write.rs:35:5
   |
35 | use flate2::{write::DeflateEncoder, Compression};
   |     ^^^^^^ use of undeclared crate or module `flate2`

error[E0433]: failed to resolve: use of undeclared crate or module `flate2`
   --> /home/ed/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zip-2.2.2/src/compression.rs:195:14
    |
195 |     Deflated(flate2::bufread::DeflateDecoder<R>),
    |              ^^^^^^ use of undeclared crate or module `flate2`

error[E0433]: failed to resolve: use of undeclared crate or module `flate2`
   --> /home/ed/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zip-2.2.2/src/compression.rs:234:40
    |
234 |                 Decompressor::Deflated(flate2::bufread::DeflateDecoder::new(reader))
    |                                        ^^^^^^ use of undeclared crate or module `flate2`

Some errors have detailed explanations: E0432, E0433.
For more information about an error, try `rustc --explain E0432`.
error: could not compile `zip` (lib) due to 4 previous errors
error: failed to compile `deno v2.1.4`, intermediate artifacts can be found at `/home/ed/.cargo/target`.
To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
The [deno] required utility was not installed via [cargo]
The [deno] required utility was not installed. ❌

Cargo and rust dirs are using 5.8 GiB.

[balupton]

/ref #147

Sorry for the horrible experience, it shouldn't be happening, and is not intentional, and Dorothy does everything to avoid this. It seems something went wrong with Deno's prebuilt binaries so Dorothy needing Deno tried to compile it itself. Dorothy needs to fix this.

As for why Deno is required: Earlier this year, once Deno provided prebuilt binaries for all our targets except riscv, I decided to use Deno to write Dorothy's echo-regexp command that uses Deno regular expressions to solve the strange configuration corruption edge cases that could occur previously, this eliminatex the dependency on several other rust packages with various cross-platform support: sd, teip, ripgrep.

As for why it was compiling: Deno is meant to support arm64, which is why I took the plunge to adopt it, so it seems somehow something when wrong with their official build and it had to build it itself, which failed. Compilation on raspberry pis is a painful long process, and the setup-util technology of Dorothy is designed to do everything possible to avoid that, able to try multiple options before installing a package system or trying compilation.

Which raspberry pi model was it? and which OS was it? Did you have sudo access in this case?

Our CI tests on arm, and I have several raspberry pi 5s and 4s. I'll see if I can reproduce.

Knowing these, I'll see if we can fix it inside Dorothy or if we have to coordinate with the Deno team.

There is also plunge I can make with Dorothy which is offering our own prebuilt binaries for dependencies for every platform, however that is significant work at this time. And at least in this case, it shouldn't have been needed.

[balupton]

Ohhhh. So the reason the prebuilt binaries failed to fetch was the github rate limit was hit, then for some reason was still exhausted after the wait period, so Dorothy was unable to scan the github repository release assets for a compatible download for the system....

Was this on a shared IP that does a lot of github stuff? As waiting the wait period for the reset should have worked, it's what our CI does when I push too many commits too quickly. However, as it waited then hit the same unauthorised error maybe something else continued to hit the github api during the wait period.

Perhaps we need to cache the github responses on our own cloud, be it as our own server or github actions automation. Or if the github rate limit is set, and if interactive, we present a choose menu with three options (1) authorise with github so we can continue the optimal path, (2) wait the desired time and try again, and if failing present the same menu instead of failing (3) give up on github api and continue to a fallback or failure

[edmcman]

Which raspberry pi model was it? and which OS was it? Did you have sudo access in this case?

Raspberry Pi 5 Model B Rev 1.0
Ubuntu 24.04.1 LTS

I do have sudo access.

Our CI tests on arm, and I have several raspberry pi 5s and 4s. I'll see if I can reproduce.

I'm not sure why the build of deno failed. It's basically a fresh Ubuntu install.

Was this on a shared IP that does a lot of github stuff? As waiting the wait period for the reset should have worked, it's what our CI does when I push too many commits too quickly. However, as it waited then hit the same unauthorised error maybe something else continued to hit the github api during the wait period.

I'm unsure why the GitHub API/download failed. This was on my home network. As far as I know, our external IP is not shared with anyone else. There was not any GitHub API usage that I knew of.

The problem is still occurring. I'll try and determine what is going on there.

[edmcman]

The rate limit seems to be right. I am not sure what is using the API though.. I generally don't use the API unauthenticated.

{"resources":{"core":{"limit":60,"remaining":0,"reset":1735530582,"used":60,"resource":"core"},"graphql":{"limit":0,"remaining":0,"reset":1735531523,"used":0,"resource":"graphql"},"integration_manifest":{"limit":5000,"remaining":5000,"reset":1735531523,"used":0,"resource":"integration_manifest"},"search":{"limit":10,"remaining":10,"reset":1735527983,"used":0,"resource":"search"}},"rate":{"limit":60,"remaining":0,"reset":1735530582,"used":60,"resource":"core"}}

[balupton]

Okay, I'll look into code changes to improve on this situation later this week, as I have other tasks to attend to.

For the meantime, extract https://github.com/denoland/deno/releases/download/v2.1.4/deno-aarch64-unknown-linux-gnu.zip and place the deno executable from it inside your ~/.local/bin then chmod +x ~/.local/bin/deno. To unzip tar -xvf ./deno-aarch64-unknown-linux-gnu.zip.

Then run the Dorothy install again and it should work, as it seems the install of jq went fine.

Besides the dependencies mentioned in the readme, which are essential even for the installer, the use of Deno was able to drop our minimal dependencies down to just jq and deno. jq was probably installed via apt on Debian or via github.com/jqlang/jq if apt failed.

Let me know if that workaround works.

This is a solvable growing pain, I hope to have all the Dorothy growing pains sorted in 2025H1, including a roll out of short videos showcasing everything Dorothy can do.

[edmcman]

That worked, thanks!

[balupton]

That worked, thanks!

Phew!

I've also updated the readme with a new dependency and requisites section:

• https://github.com/bevry/dorothy?tab=readme-ov-file#dependencies
• https://github.com/bevry/dorothy?tab=readme-ov-file#requisites

[balupton]

/ref #225

[balupton]

The solution to this is multifold.

Todos:

• different levels of caching within Dorothy of release asset api responses, such that rate limit is not hit in the first place; caching where request is not attempted, and caching where request is attempted but used if failed
• have setup-util hit a Dorothy cloud which caches the asset api responses, the actual asset files, the prebuilt binaries from the recompiled assets, and the built binaries from source assets
• have the Dorothy installation prompt for logging in with the github api
• have rate limit encounters prompt for logging in with the github api, as an alternative to waiting
• support recurrent wait or login prompts, instead of just one
• fetch the api token from github cli or other tooling if they are logged in

[balupton]

I encountered the rate limit hit while debugging installations on fresh macos. However the wait worked fine for me, with no other machines interacting with Dorothy or github. gh was not logged in.

I will implement caching inside github-download which will be opt out, and will be cleared in setup-system clean.

[balupton]

Two improvements for this is coming in #281

There first is that we will now skip querying the GitHub repositories of utilities in not just quiet mode. Before it was constrained to only quiet mode. However other improvements since the original code for that, and some new modifications has enabled this change. This should assist in preventing the rate limit from being hit in the first place:

The second is that the Dorothy install experience will now be more suggestive of installing and logging in with the GitHub CLI, such that Dorothy can utilise its authentication token for hitting the API.

These two improvements should remove the need for a caching a solution. So I will wait and see after their release to note the impact and decide if any further improvements are justified.