Support nightly CI

[mleonhard]

I use Gitlab CI to build and test with rust latest, beta, and nightly. See the gitlab-ci.yml. The nightly script often fails to download rustfmt or clippy.

How about providing a way to reliably use rustfmt in nightly CI executions?

Related: include rustfmt and clippy in the rustlang/nightly docker image

[mleonhard]

A workaround obtained from #4230:

$ rustup component add rustfmt || cargo +nightly install --git https://github.com/rust-lang/rustfmt.git
 --branch master --force --features "rustfmt cargo-fmt" rustfmt-nightly

It works but increases the nightly CI runtime by 10 minutes!

Pulling docker image rustlang/rust:nightly ...
Using docker image sha256:e40103be369d5e215b45d5756adcef70114e3b47ff2defd1ff8c868535d58291
 for rustlang/rust:nightly with digest
 rustlang/rust@sha256:9efef921a94f268110ed9975a506f74126b5857358ab64bbcef1e515de1d90ed ...
...
$ rustup component add rustfmt || cargo +nightly install --git https://github.com/rust-lang/rustfmt.git
 --branch master --force --features "rustfmt cargo-fmt" rustfmt-nightly
error: component 'rustfmt' for target 'x86_64-unknown-linux-gnu' is unavailable for download for
 channel nightly
Sometimes not all components are available in any given nightly.
    Updating git repository `https://github.com/rust-lang/rustfmt.git`
  Installing rustfmt-nightly v2.0.0-rc.2 (https://github.com/rust-lang/rustfmt.git?branch=master#2d6a968d)
    Updating crates.io index
 Downloading crates ...
  Downloaded aho-corasick v0.7.14
  Downloaded quick-error v1.2.3
  Downloaded lock_api v0.4.1
  Downloaded rustc-ap-rustc_ast v686.0.0
  Downloaded typenum v1.12.0
  Downloaded block-buffer v0.7.3
  Downloaded semver-parser v0.7.0
  Downloaded rustc-ap-rustc_graphviz v686.0.0
  Downloaded opaque-debug v0.2.3
  Downloaded quote v1.0.7
  Downloaded bstr v0.2.14
  Downloaded unicode-xid v0.2.1
  Downloaded rustc-ap-rustc_parse v686.0.0
  Downloaded rustc-rayon v0.3.0
  Downloaded walkdir v2.3.1
  Downloaded tinyvec v0.3.4
  Downloaded unicode_categories v0.1.1
  Downloaded vec_map v0.8.2
  Downloaded toml v0.5.7
  Downloaded tracing-attributes v0.1.11
  Downloaded yansi-term v0.1.2
  Downloaded serde v1.0.117
  Downloaded rustc-ap-rustc_span v686.0.0
  Downloaded unicode-normalization v0.1.13
  Downloaded thiserror v1.0.21
  Downloaded tracing v0.1.21
  Downloaded version_check v0.9.2
  Downloaded unicode-segmentation v1.6.0
  Downloaded same-file v1.0.6
  Downloaded textwrap v0.11.0
  Downloaded syn v1.0.48
  Downloaded parking_lot v0.11.0
  Downloaded cfg-if v0.1.10
  Downloaded rustc-ap-rustc_data_structures v686.0.0
  Downloaded rand v0.7.3
  Downloaded rustc-ap-rustc_expand v686.0.0
  Downloaded libc v0.2.80
  Downloaded regex-syntax v0.6.20
  Downloaded regex v1.4.1
  Downloaded clap v2.33.3
  Downloaded scopeguard v1.1.0
  Downloaded hashbrown v0.9.1
  Downloaded dunce v1.0.1
  Downloaded dirs-sys v0.3.5
  Downloaded itertools v0.9.0
  Downloaded itertools v0.8.2
  Downloaded termcolor v1.1.0
  Downloaded synstructure v0.12.4
  Downloaded smallvec v1.4.2
  Downloaded rustc-workspace-hack v1.0.0
  Downloaded rustc-ap-rustc_target v686.0.0
  Downloaded rustc-ap-rustc_lexer v686.0.0
  Downloaded memoffset v0.5.6
  Downloaded humantime v1.3.0
  Downloaded crossbeam-epoch v0.8.2
  Downloaded unicode-width v0.1.8
  Downloaded thread_local v1.0.1
  Downloaded rustc-hash v1.1.0
  Downloaded rustc-ap-rustc_serialize v686.0.0
  Downloaded cc v1.0.61
  Downloaded cargo_metadata v0.11.4
  Downloaded ansi_term v0.11.0
  Downloaded rustc-ap-rustc_feature v686.0.0
  Downloaded rustc-ap-rustc_ast_pretty v686.0.0
  Downloaded remove_dir_all v0.5.3
  Downloaded stacker v0.1.12
  Downloaded stable_deref_trait v1.2.0
  Downloaded serde_json v1.0.59
  Downloaded rustc-ap-rustc_arena v686.0.0
  Downloaded parking_lot_core v0.8.0
  Downloaded maybe-uninit v2.0.0
  Downloaded lazy_static v1.4.0
  Downloaded dirs v2.0.2
  Downloaded crossbeam-utils v0.6.6
  Downloaded proc-macro2 v1.0.24
  Downloaded rustc-ap-rustc_session v686.0.0
  Downloaded rustc-ap-rustc_errors v686.0.0
  Downloaded rustc-ap-rustc_attr v686.0.0
  Downloaded rustc-ap-rustc_ast_passes v686.0.0
  Downloaded rand_core v0.5.1
  Downloaded rand_chacha v0.2.2
  Downloaded psm v0.1.11
  Downloaded proc-macro-error-attr v1.0.4
  Downloaded proc-macro-error v1.0.4
  Downloaded ppv-lite86 v0.2.9
  Downloaded pin-project-lite v0.1.11
  Downloaded num_cpus v1.13.0
  Downloaded md-5 v0.8.0
  Downloaded indexmap v1.6.0
  Downloaded getrandom v0.1.15
  Downloaded generic-array v0.12.3
  Downloaded fake-simd v0.1.2
  Downloaded env_logger v0.7.1
  Downloaded crossbeam-deque v0.7.3
  Downloaded cfg-if v1.0.0
  Downloaded byte-tools v0.3.1
  Downloaded autocfg v1.0.1
  Downloaded atty v0.2.14
  Downloaded arrayvec v0.5.2
  Downloaded anyhow v1.0.33
  Downloaded thiserror-impl v1.0.21
  Downloaded termize v0.1.1
  Downloaded term v0.6.1
  Downloaded structopt v0.3.20
  Downloaded sha-1 v0.8.2
  Downloaded rustc-ap-rustc_fs_util v686.0.0
  Downloaded ignore v0.4.16
  Downloaded digest v0.8.1
  Downloaded crossbeam-utils v0.7.2
  Downloaded crossbeam-queue v0.1.2
  Downloaded byteorder v1.3.4
  Downloaded log v0.4.11
  Downloaded bytecount v0.6.0
  Downloaded annotate-snippets v0.8.0
  Downloaded tracing-core v0.1.17
  Downloaded tempfile v3.1.0
  Downloaded structopt-derive v0.4.13
  Downloaded strsim v0.8.0
  Downloaded serde_derive v1.0.117
  Downloaded semver v0.10.0
  Downloaded scoped-tls v1.0.0
  Downloaded ryu v1.0.5
  Downloaded rustc-rayon-core v0.3.0
  Downloaded rustc-ap-rustc_index v686.0.0
  Downloaded memchr v2.3.4
  Downloaded measureme v9.0.0
  Downloaded jobserver v0.1.21
  Downloaded itoa v0.4.6
  Downloaded instant v0.1.8
  Downloaded heck v0.3.1
  Downloaded globset v0.4.6
  Downloaded getopts v0.2.21
  Downloaded fnv v1.0.7
  Downloaded ena v0.14.0
  Downloaded either v1.6.1
  Downloaded diff v0.1.12
  Downloaded block-padding v0.1.5
  Downloaded bitflags v1.2.1
  Downloaded rustc-ap-rustc_macros v686.0.0
   Compiling cfg-if v0.1.10
   Compiling libc v0.2.80
   Compiling proc-macro2 v1.0.24
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.48
   Compiling autocfg v1.0.1
   Compiling lazy_static v1.4.0
   Compiling scopeguard v1.1.0
   Compiling getrandom v0.1.15
   Compiling typenum v1.12.0
   Compiling cc v1.0.61
   Compiling log v0.4.11
   Compiling smallvec v1.4.2
   Compiling maybe-uninit v2.0.0
   Compiling byteorder v1.3.4
   Compiling bitflags v1.2.1
   Compiling cfg-if v1.0.0
   Compiling rustc-rayon-core v0.3.0
   Compiling either v1.6.1
   Compiling unicode-width v0.1.8
   Compiling byte-tools v0.3.1
   Compiling hashbrown v0.9.1
   Compiling ppv-lite86 v0.2.9
   Compiling remove_dir_all v0.5.3
   Compiling arrayvec v0.5.2
   Compiling pin-project-lite v0.1.11
   Compiling opaque-debug v0.2.3
   Compiling rustc-hash v1.1.0
   Compiling stable_deref_trait v1.2.0
   Compiling rustc-ap-rustc_graphviz v686.0.0
   Compiling fake-simd v0.1.2
   Compiling scoped-tls v1.0.0
   Compiling version_check v0.9.2
   Compiling memchr v2.3.4
   Compiling yansi-term v0.1.2
   Compiling serde_derive v1.0.117
   Compiling termcolor v1.1.0
   Compiling serde v1.0.117
   Compiling rustc-ap-rustc_fs_util v686.0.0
   Compiling ryu v1.0.5
   Compiling regex-syntax v0.6.20
   Compiling tinyvec v0.3.4
   Compiling serde_json v1.0.59
   Compiling unicode-segmentation v1.6.0
   Compiling quick-error v1.2.3
   Compiling anyhow v1.0.33
   Compiling itoa v0.4.6
   Compiling same-file v1.0.6
   Compiling fnv v1.0.7
   Compiling semver-parser v0.7.0
   Compiling ansi_term v0.11.0
   Compiling vec_map v0.8.2
   Compiling strsim v0.8.0
   Compiling rustfmt-nightly v2.0.0-rc.2 (/builds/leonhard-llc/fixed-buffer-rs/cargo/git/checkouts/rustfmt-c4b65366788d3254/2d6a968)
   Compiling diff v0.1.12
   Compiling unicode_categories v0.1.1
   Compiling bytecount v0.6.0
   Compiling rustc-workspace-hack v1.0.0
   Compiling dunce v1.0.1
   Compiling crossbeam-utils v0.7.2
   Compiling memoffset v0.5.6
   Compiling crossbeam-epoch v0.8.2
   Compiling indexmap v1.6.0
   Compiling crossbeam-utils v0.6.6
   Compiling tracing-core v0.1.17
   Compiling thread_local v1.0.1
   Compiling lock_api v0.4.1
   Compiling psm v0.1.11
   Compiling stacker v0.1.12
   Compiling rustc-ap-rustc_arena v686.0.0
   Compiling instant v0.1.8
   Compiling itertools v0.9.0
   Compiling itertools v0.8.2
   Compiling getopts v0.2.21
   Compiling textwrap v0.11.0
   Compiling block-padding v0.1.5
   Compiling rustc-ap-rustc_lexer v686.0.0
   Compiling proc-macro-error-attr v1.0.4
   Compiling proc-macro-error v1.0.4
   Compiling annotate-snippets v0.8.0
   Compiling unicode-normalization v0.1.13
   Compiling heck v0.3.1
   Compiling humantime v1.3.0
   Compiling walkdir v2.3.1
   Compiling crossbeam-queue v0.1.2
   Compiling parking_lot_core v0.8.0
   Compiling num_cpus v1.13.0
   Compiling jobserver v0.1.21
   Compiling atty v0.2.14
   Compiling termize v0.1.1
   Compiling dirs-sys v0.3.5
   Compiling quote v1.0.7
   Compiling generic-array v0.12.3
   Compiling ena v0.14.0
   Compiling aho-corasick v0.7.14
   Compiling bstr v0.2.14
   Compiling rand_core v0.5.1
   Compiling parking_lot v0.11.0
   Compiling clap v2.33.3
   Compiling dirs v2.0.2
   Compiling rustc-ap-rustc_serialize v686.0.0
   Compiling digest v0.8.1
   Compiling block-buffer v0.7.3
   Compiling regex v1.4.1
   Compiling rand_chacha v0.2.2
   Compiling measureme v9.0.0
   Compiling term v0.6.1
   Compiling synstructure v0.12.4
   Compiling tracing-attributes v0.1.11
   Compiling thiserror-impl v1.0.21
   Compiling rustfmt-config_proc_macro v0.5.1 (/builds/leonhard-llc/fixed-buffer-rs/cargo/git/checkouts/rustfmt-c4b65366788d3254/2d6a968/config_proc_macro)
   Compiling crossbeam-deque v0.7.3
   Compiling md-5 v0.8.0
   Compiling sha-1 v0.8.2
   Compiling globset v0.4.6
   Compiling env_logger v0.7.1
   Compiling rand v0.7.3
   Compiling rustc-ap-rustc_macros v686.0.0
   Compiling tracing v0.1.21
   Compiling thiserror v1.0.21
   Compiling structopt-derive v0.4.13
   Compiling ignore v0.4.16
   Compiling tempfile v3.1.0
   Compiling rustc-ap-rustc_index v686.0.0
   Compiling semver v0.10.0
   Compiling toml v0.5.7
   Compiling structopt v0.3.20
   Compiling rustc-rayon v0.3.0
   Compiling cargo_metadata v0.11.4
   Compiling rustc-ap-rustc_data_structures v686.0.0
   Compiling rustc-ap-rustc_span v686.0.0
   Compiling rustc-ap-rustc_target v686.0.0
   Compiling rustc-ap-rustc_ast v686.0.0
   Compiling rustc-ap-rustc_errors v686.0.0
   Compiling rustc-ap-rustc_feature v686.0.0
   Compiling rustc-ap-rustc_ast_pretty v686.0.0
   Compiling rustc-ap-rustc_session v686.0.0
   Compiling rustc-ap-rustc_attr v686.0.0
   Compiling rustc-ap-rustc_parse v686.0.0
   Compiling rustc-ap-rustc_ast_passes v686.0.0
   Compiling rustc-ap-rustc_expand v686.0.0
    Finished release [optimized] target(s) in 11m 04s
  Installing /builds/leonhard-llc/fixed-buffer-rs/cargo/bin/cargo-fmt
  Installing /builds/leonhard-llc/fixed-buffer-rs/cargo/bin/rustfmt
   Installed package `rustfmt-nightly v2.0.0-rc.2 (https://github.com/rust-lang/rustfmt.git?branch=master#2d6a968d)`
 (executables `cargo-fmt`, `rustfmt`)

[taiki-e]

You can use rustup-components-history, like:

toolchain=nightly-$(curl -sSf https://rust-lang.github.io/rustup-components-history/x86_64-unknown-linux-gnu/rustfmt)
rustup set profile minimal
rustup default "$toolchain"
rustup component add rustfmt

[calebcartwright]

@mleonhard - nightly by its very nature means that things can be broken, so if you've configured your CI to always use the latest bleeding edge nightly then you are inevitably going to encounter breakages.

If you want to use rustfmt in CI via rustup then a common strategy is to use a toolchain file. That's typically pinned to a known nightly that has the desired components available and periodically updated, although @taiki-e's solution sounds like a pretty elegant approach as well.

Broken toolstates for rustfmt are almost exclusively caused by breaking upstream rustc changes that are incompatible with version of the private rustc mods that rustfmt consumes, so in cases where rustfmt is unavailable from rustup it's also not going to be possible to install rustfmt from crates.io nor build rustfmt from source; the same underlying issue surfaces in all cases. The only reason you were able to build from source successfully is because we'd happened to have already updated the versions of the rustc mods in source locally. Please also note that the version of rustfmt on the master branch of this repository is very different (a 2.0 RC version) than what's been officially released and distributed.

We are in the process of extending our release process to additionally include pre-compiled binaries for all the tier 1 platforms and attach them as assets to our GitHub Releases. This should be in place (barring any final config bugs) with the next v1.4.23 release, and could potentially be an option for you to get rustfmt binaries for CI as well.

Hope that helps, but going to close this as there's no action to be taken

[calebcartwright]

https://github.com/rust-lang/rustfmt/releases/tag/v1.4.23, binaries attached to release