From de25253739f4e5e66a53e1b002368b75178fdf2f Mon Sep 17 00:00:00 2001 From: Amrei <88709270+amrei-bp@users.noreply.github.com> Date: Tue, 21 Jan 2025 15:42:31 +0100 Subject: [PATCH 1/5] added chapter on pixi --- _quarto.yml | 3 + pixi/pixi.qmd | 162 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 165 insertions(+) create mode 100644 pixi/pixi.qmd diff --git a/_quarto.yml b/_quarto.yml index 11f5d33..80347fb 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -18,6 +18,9 @@ book: - part: "Github" chapters: - github/gh-collaboration.qmd + - part: "pixi" + chapters: + - pixi/pixi.qmd - summary.qmd - references.qmd diff --git a/pixi/pixi.qmd b/pixi/pixi.qmd new file mode 100644 index 0000000..e960b7c --- /dev/null +++ b/pixi/pixi.qmd @@ -0,0 +1,162 @@ +## Intro to Pixi + +This section is a brief summary of the concepts discussed in the meeting. + +## Why PIXI? + +PIXI is a package management tool that serves as a replacement for Conda or Mamba. It is designed to be faster, multithreaded, and flexible. Unlike Conda, PIXI environments are not isolated, which allows you to: + +- Interact with third-party tools that are available in your system's PATH. +- Benefit from faster installation of packages. + +### Compatibility + +PIXI is fully compatible with Conda. You can: + +- Use the same Conda channels (e.g., `conda-forge`, `bioconda`). +- Use existing Conda environment files with PIXI. + +### Key Difference + +In PIXI, the environment configuration file (`.pixi.toml`) lives in the project directory. This ensures that the environment is tied to the project and simplifies reproducibility. + +--- + +## Getting Started + +### Installation + +1. Install PIXI and add it to your shell's configuration (e.g., `.bashrc` or `.zshrc`). + +```{.bash} +curl -fsSL https://pixi.sh/install.sh | bash +``` + +2. Initialize a PIXI environment in your project: + +```{.bash} +pixi init -c conda-forge -c bioconda +``` + +Here: +- `-c` specifies the channels to use (e.g., `conda-forge`, `bioconda`). +- By default, PIXI will set the environment for your platform (e.g., `linux-64`). + +You can specify additional platforms if needed: + +```{.bash} +pixi init -p linux-64 -p osx-arm64 +``` + +This will create a `pixi.toml`file, as well as a `pixi.lock`file. + +### Adding Packages + +You can add packages to your environment using the `pixi add` command: + +```{.bash} +pixi add bwa samtools +``` + +You can add Python packages from PyPI using PIXI: + +```{.bash} +pixi add python +pixi add --pypi multiqc +``` + +Alternatively, you can directly modify the `.pixi.toml` file in your project directory. + +--- + +## Managing Environments + +### Files in the Directory + +When you use PIXI, it creates a `.pixi` folder in your project directory. This folder contains cached files and other data. If needed, you can safely delete this folder or configure the cache to use a scratch directory. + +To clean the cache: + +```{.bash} +pixi clean cache +``` + +### Reproducibility + +PIXI uses two key files for environment management: + +- `.pixi.toml`: Specifies the environment configuration. +- `.pixi.lock`: Locks down the specific package versions for reproducibility. + +You should commit both files to your version control system to share the exact environment setup with collaborators. + +--- + +## Tasks in PIXI + +PIXI allows you to define and run tasks directly in the `.pixi.toml` file. For example: + +```toml +[tasks] +name-of-task = "nextflow run main.nf -profile PDC" +``` + +One can also add tasks via the command line: + +```{.bash} +pixi task add hello python hello_world.py +``` + +### Task Features + +- Tasks can be combined or run conditionally using `depends-on`. +- You can chain tasks for complex workflows. + +--- + +## Working with the Shell + +PIXI provides a shell environment based on the Deno shell. Many basic Bash commands still work, allowing for: + +- Chaining tools. +- Command substitution. + +### Activating the Environment + +To activate the environment: + +```{.bash} +pixi shell +``` + +--- + +## Advanced Features + +### Multiple Environments + +- PIXI environments are tied to the project directory and based on the `.pixi.toml` file. +- While sharing the same environment across multiple projects is not currently supported, this feature is in development. + +### Features + +PIXI allows multiple "features" in a single project. Features are isolated from each other, helping to avoid version clashes between tools. + +--- + +## Additional Commands + +### Updating PIXI + +To update PIXI: + +```{.bash} +pixi self-update +``` + + +--- + +## Future Developments + +PIXI plans to include the ability to build packages directly in future releases. From 592e6a2b764dba1c6246973c2fcf5dbe4c059ab1 Mon Sep 17 00:00:00 2001 From: Amrei <88709270+amrei-bp@users.noreply.github.com> Date: Mon, 27 Jan 2025 10:29:43 +0100 Subject: [PATCH 2/5] Decapitalized PIXI --- pixi/pixi.qmd | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/pixi/pixi.qmd b/pixi/pixi.qmd index e960b7c..1864420 100644 --- a/pixi/pixi.qmd +++ b/pixi/pixi.qmd @@ -2,23 +2,23 @@ This section is a brief summary of the concepts discussed in the meeting. -## Why PIXI? +## Why Pixi? -PIXI is a package management tool that serves as a replacement for Conda or Mamba. It is designed to be faster, multithreaded, and flexible. Unlike Conda, PIXI environments are not isolated, which allows you to: +Pixi is a package management tool that serves as a replacement for Conda or Mamba. It is designed to be faster, multithreaded, and flexible. Unlike Conda, Pixi environments are not isolated, which allows you to: - Interact with third-party tools that are available in your system's PATH. - Benefit from faster installation of packages. ### Compatibility -PIXI is fully compatible with Conda. You can: +Pixi is fully compatible with Conda. You can: - Use the same Conda channels (e.g., `conda-forge`, `bioconda`). -- Use existing Conda environment files with PIXI. +- Use existing Conda environment files with Pixi. ### Key Difference -In PIXI, the environment configuration file (`.pixi.toml`) lives in the project directory. This ensures that the environment is tied to the project and simplifies reproducibility. +In Pixi, the environment configuration file (`.pixi.toml`) lives in the project directory. This ensures that the environment is tied to the project and simplifies reproducibility. --- @@ -26,13 +26,13 @@ In PIXI, the environment configuration file (`.pixi.toml`) lives in the project ### Installation -1. Install PIXI and add it to your shell's configuration (e.g., `.bashrc` or `.zshrc`). +1. Install Pixi and add it to your shell's configuration (e.g., `.bashrc` or `.zshrc`). ```{.bash} curl -fsSL https://pixi.sh/install.sh | bash ``` -2. Initialize a PIXI environment in your project: +2. Initialize a Pixi environment in your project: ```{.bash} pixi init -c conda-forge -c bioconda @@ -40,7 +40,7 @@ pixi init -c conda-forge -c bioconda Here: - `-c` specifies the channels to use (e.g., `conda-forge`, `bioconda`). -- By default, PIXI will set the environment for your platform (e.g., `linux-64`). +- By default, Pixi will set the environment for your platform (e.g., `linux-64`). You can specify additional platforms if needed: @@ -58,7 +58,7 @@ You can add packages to your environment using the `pixi add` command: pixi add bwa samtools ``` -You can add Python packages from PyPI using PIXI: +You can add Python packages from PyPI using Pixi: ```{.bash} pixi add python @@ -73,7 +73,7 @@ Alternatively, you can directly modify the `.pixi.toml` file in your project dir ### Files in the Directory -When you use PIXI, it creates a `.pixi` folder in your project directory. This folder contains cached files and other data. If needed, you can safely delete this folder or configure the cache to use a scratch directory. +When you use Pixi, it creates a `.pixi` folder in your project directory. This folder contains cached files and other data. If needed, you can safely delete this folder or configure the cache to use a scratch directory. To clean the cache: @@ -83,7 +83,7 @@ pixi clean cache ### Reproducibility -PIXI uses two key files for environment management: +Pixi uses two key files for environment management: - `.pixi.toml`: Specifies the environment configuration. - `.pixi.lock`: Locks down the specific package versions for reproducibility. @@ -92,9 +92,9 @@ You should commit both files to your version control system to share the exact e --- -## Tasks in PIXI +## Tasks in Pixi -PIXI allows you to define and run tasks directly in the `.pixi.toml` file. For example: +Pixi allows you to define and run tasks directly in the `.pixi.toml` file. For example: ```toml [tasks] @@ -116,7 +116,7 @@ pixi task add hello python hello_world.py ## Working with the Shell -PIXI provides a shell environment based on the Deno shell. Many basic Bash commands still work, allowing for: +Pixi provides a shell environment based on the Deno shell. Many basic Bash commands still work, allowing for: - Chaining tools. - Command substitution. @@ -135,20 +135,20 @@ pixi shell ### Multiple Environments -- PIXI environments are tied to the project directory and based on the `.pixi.toml` file. +- Pixi environments are tied to the project directory and based on the `.pixi.toml` file. - While sharing the same environment across multiple projects is not currently supported, this feature is in development. ### Features -PIXI allows multiple "features" in a single project. Features are isolated from each other, helping to avoid version clashes between tools. +Pixi allows multiple "features" in a single project. Features are isolated from each other, helping to avoid version clashes between tools. --- ## Additional Commands -### Updating PIXI +### Updating Pixi -To update PIXI: +To update Pixi: ```{.bash} pixi self-update @@ -159,4 +159,4 @@ pixi self-update ## Future Developments -PIXI plans to include the ability to build packages directly in future releases. +Pixi plans to include the ability to build packages directly in future releases. From 284c1950f4bbc4485b721f50a837c624fdab9018 Mon Sep 17 00:00:00 2001 From: Amrei <88709270+amrei-bp@users.noreply.github.com> Date: Mon, 27 Jan 2025 10:33:03 +0100 Subject: [PATCH 3/5] changed sentence about conda nad pixi environments and how they are not isolated --- pixi/pixi.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pixi/pixi.qmd b/pixi/pixi.qmd index 1864420..28cdb44 100644 --- a/pixi/pixi.qmd +++ b/pixi/pixi.qmd @@ -4,7 +4,7 @@ This section is a brief summary of the concepts discussed in the meeting. ## Why Pixi? -Pixi is a package management tool that serves as a replacement for Conda or Mamba. It is designed to be faster, multithreaded, and flexible. Unlike Conda, Pixi environments are not isolated, which allows you to: +Pixi is a package management tool that serves as a replacement for Conda or Mamba. It is designed to be faster, multithreaded, and flexible. Like Conda, Pixi environments are not isolated, which allows you to: - Interact with third-party tools that are available in your system's PATH. - Benefit from faster installation of packages. From e7c3ea701a5655b2f402e1030c815b74c496446f Mon Sep 17 00:00:00 2001 From: Amrei <88709270+amrei-bp@users.noreply.github.com> Date: Mon, 27 Jan 2025 10:48:28 +0100 Subject: [PATCH 4/5] incorporated the other changes suggested by mahesh-panchal --- pixi/pixi.qmd | 51 +++++++++++++++++++++++++++------------------------ 1 file changed, 27 insertions(+), 24 deletions(-) diff --git a/pixi/pixi.qmd b/pixi/pixi.qmd index 28cdb44..df3a761 100644 --- a/pixi/pixi.qmd +++ b/pixi/pixi.qmd @@ -4,18 +4,17 @@ This section is a brief summary of the concepts discussed in the meeting. ## Why Pixi? -Pixi is a package management tool that serves as a replacement for Conda or Mamba. It is designed to be faster, multithreaded, and flexible. Like Conda, Pixi environments are not isolated, which allows you to: - -- Interact with third-party tools that are available in your system's PATH. -- Benefit from faster installation of packages. +Pixi is a package management tool that serves as a replacement for Conda or Mamba. It is designed to be faster, multithreaded, and flexible. Like Conda, Pixi environments are not isolated, which allows you to interact with third-party tools that are available in your system's PATH. ### Compatibility -Pixi is fully compatible with Conda. You can: +Pixi is somehwat compatible with Conda. You can: - Use the same Conda channels (e.g., `conda-forge`, `bioconda`). - Use existing Conda environment files with Pixi. +However, you cannot activate a conda environment using Pixi, or vice-versa. + ### Key Difference In Pixi, the environment configuration file (`.pixi.toml`) lives in the project directory. This ensures that the environment is tied to the project and simplifies reproducibility. @@ -35,20 +34,16 @@ curl -fsSL https://pixi.sh/install.sh | bash 2. Initialize a Pixi environment in your project: ```{.bash} -pixi init -c conda-forge -c bioconda +pixi init -c conda-forge -c bioconda -p linux-64 -p osx-arm64 ``` Here: -- `-c` specifies the channels to use (e.g., `conda-forge`, `bioconda`). -- By default, Pixi will set the environment for your platform (e.g., `linux-64`). +- `-c` specifies the channels to use (e.g., `conda-forge`, `bioconda`) +- `-p` sets the platforms -You can specify additional platforms if needed: - -```{.bash} -pixi init -p linux-64 -p osx-arm64 -``` +By default, Pixi will set the environment for your platform (e.g., `linux-64`), but you can specify additional platforms as needed. -This will create a `pixi.toml`file, as well as a `pixi.lock`file. +The `pixi init`command will create a `pixi.toml`file, as well as a `pixi.lock`file. Both should be committed into the git repository. The Pixi environments are tied to these two files and thus the project directory. ### Adding Packages @@ -133,14 +128,23 @@ pixi shell ## Advanced Features -### Multiple Environments +### Features + +PIXI allows multiple "features" in a single project. Features are isolated from each other, helping to avoid version clashes between tools. -- Pixi environments are tied to the project directory and based on the `.pixi.toml` file. -- While sharing the same environment across multiple projects is not currently supported, this feature is in development. +### Intel emulation on ARM Macs -### Features +Although packages are increasingly being built for ARM architecture CPUS, not all tools are built for `osx-arm64`. However, they may have been built for `osx-64` (i.e., intel architecture CPUS), in addition to `linux-64`. MacOS includes the tool Rosetta, which can be used to emulate intel on arm Macs, at the cost of performance. -Pixi allows multiple "features" in a single project. Features are isolated from each other, helping to avoid version clashes between tools. +By supporting only `linux-64` and `osx-64` as platforms, Pixi will automatically run the tools using Rosetta emulation on Mac ARM computers. + +```bash +pixi init \ + --channel "conda-forge" \ + --channel "bioconda" \ + --platform "linux-64" \ + --platform "osx-64" +``` --- @@ -154,9 +158,8 @@ To update Pixi: pixi self-update ``` +To clean the cache: ---- - -## Future Developments - -Pixi plans to include the ability to build packages directly in future releases. +```{.bash} +pixi clean cache +``` From acaf3d19fc084d30ea0352a92399aaa9742b6340 Mon Sep 17 00:00:00 2001 From: Amrei <88709270+amrei-bp@users.noreply.github.com> Date: Mon, 27 Jan 2025 11:25:03 +0100 Subject: [PATCH 5/5] ammended to reflect more comments from mahesh-panchal, which I didn't see before --- pixi/pixi.qmd | 33 ++++++++++++++------------------- 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/pixi/pixi.qmd b/pixi/pixi.qmd index df3a761..bdb3d15 100644 --- a/pixi/pixi.qmd +++ b/pixi/pixi.qmd @@ -43,7 +43,13 @@ Here: By default, Pixi will set the environment for your platform (e.g., `linux-64`), but you can specify additional platforms as needed. -The `pixi init`command will create a `pixi.toml`file, as well as a `pixi.lock`file. Both should be committed into the git repository. The Pixi environments are tied to these two files and thus the project directory. +The `pixi init`command will create two files: + +- `.pixi.toml`: Specifies the environment configuration. +- `.pixi.lock`: Locks down the specific package versions for reproducibility. + +You should commit both files to your version control system to share the exact environment setup with collaborators and increase `reproducibility`. + ### Adding Packages @@ -60,7 +66,9 @@ pixi add python pixi add --pypi multiqc ``` -Alternatively, you can directly modify the `.pixi.toml` file in your project directory. +Alternatively, you can directly modify the `.pixi.toml` file in your project directory: + +Under `[dependencies]`, or platform specific dependencies (such as `[target.linux-64.dependencies]` you can add a line for each package you want to include `nextflow = "24.10.4.*"`for example. --- @@ -68,22 +76,8 @@ Alternatively, you can directly modify the `.pixi.toml` file in your project dir ### Files in the Directory -When you use Pixi, it creates a `.pixi` folder in your project directory. This folder contains cached files and other data. If needed, you can safely delete this folder or configure the cache to use a scratch directory. +When you use Pixi, it creates a `.pixi` folder in your project directory. This folder contains the libraries and binaries needed for the environment. If needed, you can safely delete this folder. -To clean the cache: - -```{.bash} -pixi clean cache -``` - -### Reproducibility - -Pixi uses two key files for environment management: - -- `.pixi.toml`: Specifies the environment configuration. -- `.pixi.lock`: Locks down the specific package versions for reproducibility. - -You should commit both files to your version control system to share the exact environment setup with collaborators. --- @@ -104,8 +98,7 @@ pixi task add hello python hello_world.py ### Task Features -- Tasks can be combined or run conditionally using `depends-on`. -- You can chain tasks for complex workflows. +Tasks can be combined or run conditionally. The example in the [documentation](https://pixi.sh/latest/features/advanced_tasks/) is to specify that an application should be complied before being run. The command `depends-on`can be used here. With this one can chain tasks for complex workflows. --- @@ -158,6 +151,8 @@ To update Pixi: pixi self-update ``` +### Cleaning the cache + To clean the cache: ```{.bash}