Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: install dependencies in .tool-versions order #1723

Open
wants to merge 19 commits into
base: master
Choose a base branch
from

Conversation

chrisjpalmer
Copy link

@chrisjpalmer chrisjpalmer commented Feb 22, 2024

UPDATE

Summary

This PR changes the behaviour of asdf install to install tools in the order they are listed in .tool-versions.

The purpose is to prevent installation failures due to dependencies between plugins. An example is the poetry python plugins. When running asdf install for these, the poetry plugin will typically fail because python is not available (see here). This requires the user to manually run asdf install python xxx first and then continue with the rest of the install.

Rather than maintain the correct plugin order as part of the tool, this responsibility can be forwarded to users of the tool via .tool-versions file. asdf install will respect the order listed in .tool-versions when installing the tools.

Other Information

  1. Tests have been written to verify the new behaviour works. All the old tests are passing locally except for two, but they seem unrelated.
    • asdf update --head should checkout the master branch
    • plugin_list_all should sync repo when check_duration set to 0
  2. The behaviour of the tool is to start in the deepest directory (which is the cwd) and recurse upwards to shallower directories.
    • install tools in the .tool-versions file one by one
    • install any tools from legacy file versions if a plugin is installed for one, and it didn't appear in .tool-versions file
    • continue to recurse to shallower directories until a tool for every plugin has been installed

Example

Given the following .tool-versions:

nodejs 18.16.1
python 3.10.13
poetry 1.4.2

before

asdf install

kubectl 1.24.16 is already installed
nodejs 18.16.1 is already installed
No preset version installed for command python3
Please install a version by running one of the following:

asdf install python 3.10.13

or add one of the following versions in your config file at /Users/xxx/workspace/service-developer-portal/.tool-versions
python 3.10.2

Cleanup: Something went wrong!

48 /Users/xxx/.asdf/plugins/poetry/bin/install: POETRY_HOME=$install_path python3 - --version "$version" $flags

after

asdf install

nodejs 18.16.1 is already installed
python-build 3.10.13 /Users/xxx/.asdf/installs/python/3.10.13
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.13.tar.xz...
-> https://www.python.org/ftp/python/3.10.13/Python-3.10.13.tar.xz
Installing Python-3.10.13...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.10.13 to /Users/xxx/.asdf/installs/python/3.10.13
asdf: Warn: You have configured asdf to preserve downloaded files (with always_keep_download=yes or --keep-download). But
asdf: Warn: the current plugin (python) does not support that. Downloaded files will not be preserved.
Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/Users/xxx/.asdf/installs/poetry/1.4.2/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.4.2): Done

Poetry (1.4.2) is installed now. Great!

To get started you need Poetry's bin directory (/Users/xxx/.asdf/installs/poetry/1.4.2/bin) in your `PATH`
environment variable.

Add `export PATH="/Users/xxx/.asdf/installs/poetry/1.4.2/bin:$PATH"` to your shell configuration file.

Alternatively, you can call Poetry explicitly with `/Users/xxx/.asdf/installs/poetry/1.4.2/bin/poetry`.

You can test that everything is set up by executing:

`poetry --version`

Configuring poetry to behave properly with asdf ...
Running: "poetry config virtualenvs.prefer-active-python true".

('Configuration file exists at /Users/xxx/Library/Application Support/pypoetry, reusing this directory.\n\nConsider moving TOML configuration files to /Users/xxx/Library/Preferences/pypoetry, as support for the legacy directory will be removed in an upcoming release.',)
asdf: Warn: You have configured asdf to preserve downloaded files (with always_keep_download=yes or --keep-download). But
asdf: Warn: the current plugin (poetry) does not support that. Downloaded files will not be preserved.
Either specify a tool & version in the command
OR add .tool-versions file in this directory
or in a parent directory

... old PR description

Summary

This PR allows asdf to install tools in the order they are listed in .tool-versions. It works by the command asdf install --ordered-install to be provided.

The purpose is to prevent installation failures due to dependencies between plugins. An example is the poetry python plugins. When running asdf install for these, the poetry plugin will typically fail because python is not available (see here). This requires the user to manually run asdf install python xxx first and then continue with the rest of the install.

The PR takes a simple approach to resolve this problem. Rather than maintain the correct plugin order as part of the tool, which is much too difficult, allow developers to maintain the order in.tool-versions file. Then make the asdf tool follow that order when --ordered-install is provided.

Other Information

I have not written any tests, nor have I done extensive testing to verify this doesn't break any old functionality.
I also have not addressed this change for multiple levels of .tool-versions files in higher directories.
I can look into these things, if people like this change and we think its worth pursuing. Do let me know.

Our company uses asdf extensively for streamlining developer workflows so we are interested in improving the tool if possible.

Thanks!

Copy link

@jfly jfly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excited to see this pr! Why even add the option? I'd vote for just making this the unconditional behavior.

@chrisjpalmer
Copy link
Author

Excited to see this pr! Why even add the option? I'd vote for just making this the unconditional behavior.

I'm down for that, just thought its better to leave the option in there in case it breaks anything for people using the tool now.

@HashNuke
Copy link
Member

@chrisjpalmer Thanks for this very useful PR + the notes on the tests and the scenarios not handled.

Ordered install can be the default behaviour.

But I have not given any thought to how we could handle order across .tool-versions files (especially $HOME/.tool-versions and ${pwd}/.tool-versions). Open to ideas.

Copy link
Member

@HashNuke HashNuke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added notes in the comments but repeating here.

Very useful PR ❤️

  1. Ordered install can be the default behaviour
  2. Open to ideas to handle scenarios for situations where there is a .tool-versions in the $HOME and the current directory.
  3. Tests can be added once we do the above.

@HashNuke HashNuke marked this pull request as draft February 23, 2024 02:33
@HashNuke
Copy link
Member

Changed this to a draft PR since this is work in progress.

@chrisjpalmer
Copy link
Author

chrisjpalmer commented Feb 27, 2024

Thanks Akash. This sounds good.

I have some initial thoughts on how we can support multiple .tool-versions files in higher directories.
First I wanted to confirm the current behvaviour.

Current Process

Based on some testing and reverse engineering the code, I can see that ASDF does the following steps:

  1. Check $PWD/.tool-versions to see if there are any versions who don't have a corresponding plugin installed on the system. If there are any versions without a corresponding plugin, a message is issued and the command exits early.
    • Note: Higher level directories with .tool-versions are not searched.
  2. Iterate plugins in alpha order. For each plugin:
    • Recursively search .tool-versions files for a corresponding version - starting with $PWD/.tool-versions, going up a directory each time.
    • As soon as a match is found, install that version and move on to the next plugin

Let me know if you agree with the above analysis. Assuming I'm right, then the key takeaway is that the .tool-versions files are recursively searched and the default .tool-versions file is considered last.

New Process

To emulate the same behaviour whilst now considering the version order in .tool-versions I believe we need the following process:

  1. Check $PWD/.tool-versions to see if there are any versions who don't have a corresponding plugin installed... (keep this the same)
  2. Recursively scan .tool-versions files starting with $PWD/.tool-versions, going up a directory each time. For each .tool-versions file:
    • Install versions in the order they appear in the file.
    • Store the plugins we have installed in a variable to keep track of what we have already covered off - this allows us to skip version installs for same plugin versions as we traverse higher directories.
  3. Search ends when one of two conditions is true:
    • there are no more directories to search
    • we have satisfied every plugin that is installed on the system

Just for sanity checking its important to note the following about this new process:

  • versions in earlier .tool-versions take precedence over ones in higher directories - this is the same as the previous process which resolves the version for each plugin in earlier .tool-versions files before searching higher
  • versions without a corresponding plugin will not contribute to the installation process. This is the same as the previous process because, the previous installation process is driven by the list of plugins that do exist and hence will also ignore versions that have no corresponding plugin.

Let me know what you think :) Chris

@chrisjpalmer
Copy link
Author

Hi there. I have actually gone ahead and implemented this. Current status is:

  • basic code is in place
  • replaced old install process with ordered install process
  • no unit tests written
  • hand tested on my machine with .tool-version files
  • have not tested legacy install files

@chrisjpalmer
Copy link
Author

Hey @HashNuke just checking in to see if you are happy with the progress on this PR ?

@chrisjpalmer
Copy link
Author

chrisjpalmer commented Mar 6, 2024

Update

I ran the unit test suite... majority are passing!

I added a new unit test which verifies the ordered install behaviour:

  • install_command installs plugins in order

Note: these are failing (on my local) but I believe they are unrelated:

  • asdf update --head should checkout the master branch
  • plugin_list_all should sync repo when check_duration set to 0

For consideration

I was chatting with a colleague of mine, and we were thinking that in addition to having tools be installed in .tool-version order, it might be good to consider the directory structure as well when installing tools. Suppose you have the following structure:

/
  - .tool-versions: python xxx
       /subdir
          - .tool-versions: poetry yyy

Currently in both the previous and new install process, poetry will be installed before python because asdf install installs tools in the deepest directory and the recurses upwards to shallower directories. However, there is an argument that it might be more useful to do this the opposite way. In a monorepo for example a user may specify a general tool in the root directory, and then sub directories could be free to declare additional tools. It may be possible that those additional tools have a dependency on tools declared in the root directory. Therefore it could make sense to actually modify asdf install's behaviour to recurse from the shallower directory downwards.

Thoughts on this ?

Thanks

@chrisjpalmer chrisjpalmer requested a review from HashNuke March 6, 2024 05:01
@chrisjpalmer
Copy link
Author

@HashNuke or @jfly would you mind taking a look to confirm you are happy with progress ?

@chrisjpalmer chrisjpalmer changed the title feat: support ordered install mode feature: install dependencies in .tool-versions order Mar 14, 2024
@chrisjpalmer chrisjpalmer changed the title feature: install dependencies in .tool-versions order feat: install dependencies in .tool-versions order Mar 14, 2024
@Stratus3D
Copy link
Member

Stratus3D commented Mar 14, 2024

  1. Check $PWD/.tool-versions to see if there are any versions who don't have a corresponding plugin installed on the system. If there are any versions without a corresponding plugin, a message is issued and the command exits early.
    Note: Higher level directories with .tool-versions are not searched.

If this is the case I think this might be a bug in the old process (probably out of scope for this PR).


Recursively scan .tool-versions files starting with $PWD/.tool-versions, going up a directory each time. For each .tool-versions file:

Install versions in the order they appear in the file.
Store the plugins we have installed in a variable to keep track of what we have already covered off - this allows us to skip version installs for same plugin versions as we traverse higher directories.

For the new process we might actually want to start at the uppermost .tool-versions file, installing versions in order, and work our way down to the current directory. I feel like the home/root dir is typically used for system wide stuff that the user just happens to be using asdf for (ie version doesn't vary between dirs). This is definitely a nitpick. No need to implement right now.

Currently in both the previous and new install process, poetry will be installed before python because asdf install installs tools in the deepest directory and the recurses upwards to shallower directories. However, there is an argument that it might be more useful to do this the opposite way. In a monorepo for example a user may specify a general tool in the root directory, and then sub directories could be free to declare additional tools. It may be possible that those additional tools have a dependency on tools declared in the root directory. Therefore it could make sense to actually modify asdf install's behaviour to recurse from the shallower directory downwards.

Yep, exactly!


versions without a corresponding plugin will not contribute to the installation process. This is the same as the previous process because, the previous installation process is driven by the list of plugins that do exist and hence will also ignore versions that have no corresponding plugin.

Interesting, I wonder if that is best. I don't think we should be doing anything until all required plugins (ie all those referenced in all .tool-version files at and above the current dir) are installed. Again, definitely out of scope for this PR but I think it may have ended up this way by accident rather than by design. Any idea why we might want to silently ignore missing plugins at this step?

# install tools from .tool-versions
# install order is the order listed in .tool-versions
file_name=$(asdf_tool_versions_filename)
tools_installed=$(_install_directory_tools "$search_path" "$file_name" "$plugins_installed" "$tools_installed")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm used to wrapping everything in quotes. Is there a reason the $() on these two lines aren't quoted? Like this:

file_name="$(asdf_tool_versions_filename)"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary. Word splitting is not done on values in assignments (see Bash Shell Parameters), i.e., quotes are not needed when assigning command substitutions ($(...)) or variables ($var). E.g.:

foo='a space-filled value'
bar=$foo
echo $bar

a space-filled value

foo=$(echo this is a command substitution with spaces)
echo $foo

this is a command substitution with spaces

return 0
fi

tool_versions=$(strip_tool_version_comments "$search_path/$file_name" | awk '{$1=$1};1')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably want this $() to be quoted right?

Copy link
Author

@chrisjpalmer chrisjpalmer Mar 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure to be honest. I noticed that most of the assignments in the installs.bash file do not put quotes around the $(). When I ran the linter, the linter pulled up times where I had passed a $() to a bash function if I didn't wrap it in quotes. It would give the following warning: SC2046 (warning): Quote this to prevent word splitting.. This makes me think that variable assignment is okay but potentially passing as arguments without wrapping in quotes is the one that isn't right. Not sure though, no expert on bash myself.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typically if the value produced by the expression contains spaces and you want it to stay as a string you should quote - string_with_spaces="$(echo "string with spaces")". I think that is the case here, but I may be wrong. Generally, quoting is safer in Bash (unless you are dealing with an array, in which case it won't work).

Comment on lines 470 to 478
display_debug() {
if [[ $DEBUG = "true" ]]; then
printf "debug: %s\n" "$1" >&2
fi
}

display_none() {
printf ""
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be good to put these in utils.sh.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Member

@Stratus3D Stratus3D left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @chrisjpalmer ! Overall I think this is a good change.

Having been away from Bash for a little while I'm a little paranoid of unquoted variables and don't remember exactly when things should be quoted or unquoted. I'll test out this PR branch on my machine today and see how it goes.

@chrisjpalmer chrisjpalmer requested a review from Stratus3D March 18, 2024 02:57
@chrisjpalmer
Copy link
Author

Thanks @Stratus3D for your review. Appreciate that time you put into it. I have tried to address all your comments and respond to anything that I still wasn't sure about.

  1. Check $PWD/.tool-versions to see if there are any versions who don't have a corresponding plugin installed on the system. If there are any versions without a corresponding plugin, a message is issued and the command exits early.
    Note: Higher level directories with .tool-versions are not searched.

If this is the case I think this might be a bug in the old process (probably out of scope for this PR).

Quite possibly. We can always follow this PR up with another one which hardens this behaviour.

Recursively scan .tool-versions files starting with $PWD/.tool-versions, going up a directory each time. For each .tool-versions file:
Install versions in the order they appear in the file.
Store the plugins we have installed in a variable to keep track of what we have already covered off - this allows us to skip version installs for same plugin versions as we traverse higher directories.

For the new process we might actually want to start at the uppermost .tool-versions file, installing versions in order, and work our way down to the current directory. I feel like the home/root dir is typically used for system wide stuff that the user just happens to be using asdf for (ie version doesn't vary between dirs). This is definitely a nitpick. No need to implement right now.

Currently in both the previous and new install process, poetry will be installed before python because asdf install installs tools in the deepest directory and the recurses upwards to shallower directories. However, there is an argument that it might be more useful to do this the opposite way. In a monorepo for example a user may specify a general tool in the root directory, and then sub directories could be free to declare additional tools. It may be possible that those additional tools have a dependency on tools declared in the root directory. Therefore it could make sense to actually modify asdf install's behaviour to recurse from the shallower directory downwards.

Yep, exactly!

I would be happy to chase this up in a follow up PR.

versions without a corresponding plugin will not contribute to the installation process. This is the same as the previous process because, the previous installation process is driven by the list of plugins that do exist and hence will also ignore versions that have no corresponding plugin.

Interesting, I wonder if that is best. I don't think we should be doing anything until all required plugins (ie all those referenced in all .tool-version files at and above the current dir) are installed. Again, definitely out of scope for this PR but I think it may have ended up this way by accident rather than by design. Any idea why we might want to silently ignore missing plugins at this step?

Ah I think when I said that "a version without a corresponding plugin will not contribute to the installation process" what I meant was, ignoring the sanity check that runs before installation kicks off, the main installation loop ignores unknown plugins. However there is a sanity check that runs before the main installation which lets the user know if it detects an unknown plugin. This is only in place for the .tool-versions file in the current directory. Again suggests to me a follow up PR to harden this behavior.

done
plugins_installed=$(printf "%s" "$plugins_installed" | tr "\n" " " | awk '{$1=$1};1')
display_debug "install_local_tool_versions: plugins_installed='$plugins_installed'"
tools_installed=$(install_directory_tools_recursive "$search_path" "$plugins_installed")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the $() on lines 122 and 124 should be quoted.

while [ "$search_path" != "/" ]; do
# install tools from files in current directory
display_debug_hr
tools_installed=$(install_directory_tools "$search_path" "$plugins_installed" "$tools_installed")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quote this one too I think.

fi

# go up a directory
search_path=$(dirname "$search_path")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one as well. paths can contain spaces but they should be treated like a single string.

local legacy_config
legacy_config=$(get_asdf_config_value "legacy_version_file")
if [ "$legacy_config" = "yes" ]; then
tools_installed=$(_install_directory_tools_legacy "$search_path" "$plugins_installed" "$tools_installed")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quote $() here too I think.

for plugin_version in "${parts[@]:1}"; do
# install the version
display_none "$(install_tool_version "$plugin_name" "$plugin_version")"
tools_installed=$(printf "%s %s" "$tools_installed" "$plugin_name" | awk '{$1=$1};1')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quote $() here too I think.


# install the version
display_none "$(install_tool_version "$plugin_name" "$plugin_version")"
tools_installed=$(printf "%s %s" "$tools_installed" "$plugin_name" | awk '{$1=$1};1')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quote here.


# extract plugin legacy information
local plugin_path
plugin_path=$(get_plugin_path "$plugin_name")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quote here. Path may contain whitespace.


# extract plugin legacy filenames
local legacy_filenames=""
legacy_filenames=$("$legacy_list_filenames_script")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For good measure quote here. I can't imagine any legacy file names would contain whitespace since these are typically filenames chosen for software developers, but it won't hurt.


display_none() {
printf ""
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is display_none for?

@@ -156,6 +156,20 @@ display_error() {
printf "%s\n" "$1" >&2
}

display_debug_hr() {
display_debug "--------------------------------------------------------------------------------------------------------------"
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we only use this function once. Are we certain we need it?

@Stratus3D
Copy link
Member

Stratus3D commented Mar 27, 2024

Thanks @chrisjpalmer! just scanned through the code and left some comments.

I see our build is broken here, but it isn't due to your changes. I'll see about fixing it on master.

@stoutput
Copy link

Just checking in on this as it's been a while @Stratus3D. This would be an extremely useful feature to have implemented especially for python + poetry. Seems like the remaining asks are easy cleanups that could be knocked out in 2 min

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants