How to use a different Python version in a virtual environment to solve version compatibility issues with brownie

[n4n0b1t3]

This is a long post, why should you read it?

I encountered over the last few months many situations, where some version conflicts were at play. The main ingredients here are Python and NodeJS. With the solution described here, you have the flexibility to tune both nobs Python and NodeJS to whatever fits your project. A large number of questions in this forum originate from wrong or at least unclear setups. This here is a long article, but it might help you to overcome or even never face those problems.
I am trying to keep this up-to-date. If you notice mistakes in this article or changes like for example the newest Brownie installs now in the latest Python version, please let me know.

Content overview

1. Why did I write this post? -> Motivation
2. How to set up a virtual environment? Walk-through -> Workflow
3. Installing the payload I want to work with -> Installing-the-payload
4. Configure VSCode to use the virtual environment

EDIT 14. Mar 2022:

• Today I had to do a fresh Windows install and with this I had to setup my virtual environment as well. I noticed, that nodeenv will install the latest version which will cause the ganache installation to fail. The workflow below is taking this change into account.
• I added the installation of Visual C++ build tools as well, it is a necessary prerequisite for brownie.

EDIT 26. Feb 2022:

• After I wrote this article I ran into another problem. A python library has issues with directory names with dots in it like python3.9.venv. This has been taken into account here.
• Most of the error messages I am recently getting are garbage, I don't know if this is related to Python in general or rather a brownie issue. This makes it oftentimes very difficult to find out what's wrong.
• Another thing I changed: I noticed no differences between having a central directory for all virtual environments versus having one in each project folder. Since creating a virtual environment for each and every project is pretty redundant and creates a lot of clutter on my hard drive, I moved to have just one main directory for my virtual environments and simply activate them from within my project directory. It takes a bit of time getting used to the activation though.

Motivation

The current Brownie version v1.18.1 won't install (in a virtual environment) on Python 3.10.2 (Win 10). This causes problems when testing for reverted transactions. More on that here:
https://stackoverflow.com/questions/71126128/brownie-testing-for-reverted-transactions-does-not-work-with-pytest-raises-or
eth-brownie/brownie#1441

Approach to fix this
The latest Brownie version can be installed without issues under 3.9.10. I don't want to downgrade my standard Python so I will create a virtual environment with this specific Python version and install everything in there.

Workflow

The following workflow assumes you have already a running Python version. It is written for Windows but can be easily adapted for Mac. All command line stuff I am doing in Powershell. The setup is my personal preference, there are many ways to organize your code and this is just an example.

Install some of the Visual C++ build tools (if you haven't already)

Download and install https://visualstudio.microsoft.com/visual-cpp-build-tools/

Select only the module ticked off in the screenshot (German system) and continue with the installation.

Prepare your Python version and install virtualenv

In my home directory, I already have a directory called pythonversions. To create one simply do ...

cd $home
mkdir pythonversions

Download the Python version you need, e.g. Python 3.9.10. I downloaded the MSI Installer 64bit

Install the Python as "custom installation". Since I don't want this to become my standard python version I deselected "add to Path" and all other optional features besides pip before proceeding. I will need pip. During the installation process, I create a directory within pythonversions named after the version number, in my case 3.9.10. This step unpacks Python into pythonversions/3.9.10

With my main Python version I install virtualenv like so:
pip install virtualenv

Setup your virtual environment for a specific Python version

Under my home directory, I create a directory that will host all virtual environments.
In PowerShell:

C:\>cd $home
n4n0b1t3>mkdir virtualenvironments
n4n0b1t3>cd .\virtualenvironments\
virtualenvironments>

The syntax for creating a virtual environment is the following:

python -m virtualenv -p="<path to the python executable >" <virtual_environment_directory>

Here is an example.

virtualenvironments>python -m virtualenv -p="C:\Users\n4n0b1t3\pythonversions\3.9.10\python.exe" py39bro

I used py39bro as a directory name because I want it to be short and I want to see the python version and if it contains brownie. Name it to your liking.

Setup your project with node.js

Since my project works with brownie, I will have to install nodeenv for node JS capabilities in order to install ganache.
For one of the next steps (nodeenv -p) I need to open my Powershell as Admin. I keep the other Powershell window in the background because I will come back to it later. Press WindowsKey >enter "PowerShell" > right-click on Powershell in start menu > select "run as administrator" > confirm > navigate to your project directory.

Activate the virtual environment. This can be done from any directory, not necessarily from your project root.
The following are not my actual command in PowerShell, I just want to illustrate that the TAB key is your friend.

myProject>$home\virtualE <-- just type the first few letters and press TAB and it will automatically fill out the rest
don't press Enter
myProject>C:\Users\n4n0b1t3\virtualenvironments\  <-- type the name of your virtual environment
don't press Enter
myProject>C:\Users\n4n0b1t3\virtualenvironments\py39bro\  <-- continue to Scripts\activate.ps1
don't press Enter
myProject>C:\Users\n4n0b1t3\virtualenvironments\py39bro\Scripts\activate.ps1
now press enter!
(py39bro) myProject>

Great your virtual environment is activated. Make sure you are running the correct Python version

(py39bro) myProject>python --version
Python 3.9.10

breaking change with new nodejs version installed by nodeenv

status 14. Mar 2022:
Now we are at the point that we can install nodeenv, which is needed to set up Ganache. Here however there is now a change, because with the newest nodejs version 17.7.1 the Ganache installation fails. Here is a short excerpt from the error log, a German English gibberish, which is again not very helpful.

(py39node17) virtualEnvironments>npm install ganache
npm WARN cleanup Failed to remove some directories [
npm WARN cleanup   [
npm WARN cleanup     'C:\\Users\\n4n0b1te\\virtualEnvironments\\node_modules\\ganache\\node_modules',
npm WARN cleanup     [Error: EBUSY: resource busy or locked, rmdir 'C:\Users\n4n0b1te\virtualEnvironments\node_modules\ganache\node_modules\keccak'] {
npm WARN cleanup       errno: -4082,
npm WARN cleanup       code: 'EBUSY',
npm WARN cleanup       syscall: 'rmdir',
npm WARN cleanup       path: 'C:\\Users\\n4n0b1te\\virtualEnvironments\\node_modules\\ganache\\node_modules\\keccak'
npm WARN cleanup     }
npm WARN cleanup   ]
npm WARN cleanup ]
npm ERR! code 1
npm ERR! path C:\Users\n4n0b1te\virtualEnvironments\node_modules\ganache\node_modules\leveldown
npm ERR! command failed
npm ERR! command C:\Windows\system32\cmd.exe /d /s /c node-gyp rebuild
npm ERR! Die Projekte in dieser Projektmappe werden nacheinander erstellt. Um eine parallele Erstellung zu erm�glichen, m�ssen Sie den Schalter "-m" hinzuf�gen.
npm ERR!   builder.cc
npm ERR!   win_delay_load_hook.cc
npm ERR!   snappy.vcxproj -> C:\Users\n4n0b1te\virtualEnvironments\node_modules\ganache\node_modules\leveldown\build\Release\\snappy.lib
npm ERR! gyp info it worked if it ends with ok

The solution is to select the correct nodejs version for the Python virtual environment. In this case, Ganache is compatible with nodejs version 16.14.10.

These are now the steps which allow us to install Ganache after a successful pairing of Nodeenv with the Python virtual environment.

(py39bro)myProject>pip install nodeenv
Collecting nodeenv
  Downloading nodeenv-1.6.0-py2.py3-none-any.whl (21 kB)
Installing collected packages: nodeenv
Successfully installed nodeenv-1.6.0

Connect nodeenv with your virtual Python environment (this step needs admin rights).

(py39bro)myProject>nodeenv -p

In case you are preparing your virtual environment for hardhat you want to install (24.02.2022) node LTS (lates stable). Here is what worked for me at this date:

nodeenv -p -n 16.14.2

important

1. Find out which nodejs version works with your application (ganache in this case).
2. nodeenv --list will list all available nodejs versions.
3. after you have activated your virtual environment you can switch to another node version with
   nodeenv --node=16.14.0 -p

Installing the payload

(stuff I actually want to work with) At this point, I can close my elevated Powershell window and return to the normal Powershell.

Install ganache
(py39bro)myProject>npm install -g ganache

Install Cython to avoid Cytoolz error
(py39bro)myProject>pip install Cython

Install brownie
(py39bro)myProject>pip install eth-brownie

Now comes the moment of truth. I did all this to get to the newest brownie version, which failed installation on Python 3.10.2. with
ERROR: Could not find a version that satisfies the requirement vyper==0.3.1 (from eth-brownie)

(py39bro)myProject>brownie --version
INFO: Could not find files for the given pattern(s).
Brownie v1.18.1 - Python development framework for Ethereum

Hurray, I finally installed Brownie v1.18.1 ... And yes, brownie introduced the error "INFO: Could not find files for the given pattern(s)." which will accompany me from now on without any impact besides creating a slight annoyance.

Just to finish this preparation I will

(py39bro)myProject>brownie init

Configure VSCode to use the virtual environment

In VSCode open settings with Ctrl+,
Into the search box type: python environment and set your venv folders, here an example:

Disclaimer: I am by no means an expert. In fact, I started a few months ago with learning blockchain development. I hope that this post will be obsolete soon and Python, Virtualenv, nodeenv, Cytoolz, Brownie, Vyper and ... will play nicely with each other. Until then, hope this solved your problem too.

(py39bro)myProject>deactivate

[fuldstaendigvanvittigt]

Thank you for the effort, helps us beginners a lot!