Skip to content

Commit

Permalink
add docs for design decisions and update lock file related docs
Browse files Browse the repository at this point in the history
  • Loading branch information
SarthakJariwala committed Feb 10, 2022
1 parent 3a1775f commit 20d7194
Show file tree
Hide file tree
Showing 10 changed files with 203 additions and 33 deletions.
10 changes: 10 additions & 0 deletions docs/design_decisions/intro.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Introduction

**EZconda** takes the approach of keeping two files for managing `conda` environments.

1. **[Specifications file](specfile.md)** contains the packages and channels (and *only those* packages/channels) that the [user requests via the command line](../user_guide/install_packages.md).

2. **[Lock file](lockfile.md)** contains _**all**_ the packages along with the *exact version and build number and channels*. It also contains other metadata such as the system information where it was generated, etc.


> The two files keep environments and corresponding specifications in sync as well as [support reproducible environment creation](../user_guide/recreate_env.md).
63 changes: 63 additions & 0 deletions docs/design_decisions/lockfile.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Lock File

A lock file contains _**packages**_ that the user requested _**as well as all of it dependencies**_.

It contains the exact specifications of all the packages and their dependencies, version, build number, and channels.

Lock files are _**specific to the platform and architecture**_ they were generated on.

!!! Note
Information about the system's platform and architecture is also captured in the lock file and is used to validate the lock file when using it to recreate environments.

## Format

Lock file uses the following format to store the information described above -

```TOML
# minimum version of EZconda required
[version]
ezconda-min-version = "0.4.0"

# system specifications
[system]
platform = "darwin"
architecture = "64bit"
machine = "x86_64"

# name of the environment
[environment]
name = "mlproj"

# all the packages
[[packages]]
base_url = "https://conda.anaconda.org/conda-forge"
build_number = "0"
build_string = "py38h5fc983b_0"
channel = "conda-forge"
dist_name = "pandas-1.0.5-py38h5fc983b_0"
name = "pandas"
platform = "osx-64"
version = "1.0.5"

...
```

!!! Warning
Lock files **are not meant to be edited by the user**. They are managed completely by EZconda.

## Naming Convention

Lock files are named after the environment and the platform and its architecture.

```
<env>-<platform>-<architecture>.lock
```

For instance, if the environment name is "mlproj" and is created on a Mac, the name of the lock file will be -

```
mlproj-darwin-x86_64.lock
```

!!! Note
This automatically ensures that same environment created on a different platform will have a separate lock file.
67 changes: 67 additions & 0 deletions docs/design_decisions/specfile.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Specifications File

This file is similar to the `environment.yml` file in `conda`.

However, the big difference is, it only tracks the dependencies that the [user sepcifies at the command line.](#user-specified-packages) and not its dependencies.

??? Info
An exhaustive description of the pacakges and its dependencies is in the [lock file](lockfile.md).

## Format

The specifications file format is similar to `conda` environment specifications files. This is primarily done to maintain compatibility with `conda`.

In other words, you can use this specifications file generated by `ezconda` with `conda`.

It specifies the packages (and not their dependencies) and channels of interest -

```YAML title="mlproj.yml"
# name of the environment
name: mlproj

# channels to search
channels:
- defaults
- conda-forge

# packages to install
packages:
- python=3.8
- numpy>=1.17
- pandas
```
!!! Note
You _**do not**_ need to add or remove packages manually from the environment.yml file.
This is done **automatically** when you install/remove packages using ezconda. See [user guide](../user_guide/install_packages.md) for more details.
## User Specified Packages
If a user [installs](../user_guide/install_packages.md) `pandas` via the command line, the specifications file will only list `pandas` and not all the specific dependencies of `pandas` for that specific architecture.

> This ensures that specifications file is platform independent whereas the [lock file](lockfile.md) is platform specific.

<div class="termy">

```console
$ ezconda install -n mlproj pandas
// Adds (only) pandas to mlproj.yml specifications file
```

</div>

??? Example "mlproj.yml"
```YAML
# name of the environment
name: mlproj
# channels to search
channels:
- defaults
# packages to install
packages:
- pandas
```
35 changes: 23 additions & 12 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,17 +27,21 @@

## Key Features

- **Environment specifications** : Add & remove packages from the <abbr title="commonly known as environment.yml file">specifications file</abbr> as you install & remove packages. _**No manual file edits!**_
- **Environment Management** : Create and manage `conda` environments with ease.

- **Environment management** : Create & manage `conda` environments with ease.
- **Environment Specifications** : Add and remove packages from the <abbr title="commonly known as environment.yml file">specifications file</abbr> as you install & remove them.

> _**No manual file edits! No exporting entire environments!**_
- **Reproducible environments** : Lock current environment state and re-create it when necessary.
- **Reproducible Environments** : Auto lock current environment state and re-create it anywhere.

- **Easy to use & intuitive** : It very closely mimics `conda` API, so there is no new API to learn for users. Autocomplete for all shells.
- **Fast & Reliable Environment Resolution** : Get fast and reliable environment solves by default.

- **Fast & Reliable Environment resolution** : Get fast and reliable environment solves by default. *EZconda* uses `mamba` by default, but you can easily switch between `mamba` and `conda`.
> *EZconda* uses `mamba` by default, but you can easily switch between `mamba` and `conda`.
- **Best practices built-in** : Enforces the user to follow best `conda` practices.
- **Easy & Intuitive** : Intuitive commands and autocompletions by default.

- **Built-in Best Practices** : Forces the user to follow best `conda` practices.

## Requirements

Expand All @@ -47,7 +51,7 @@ Requires a [conda](https://docs.conda.io/projects/conda/en/latest/user-guide/ins

The recommended way to install **EZconda** is using `conda` or `mamba` in the `base` environment :

### Using `conda`:
### Using `conda`

<div class="termy">

Expand All @@ -59,7 +63,7 @@ Successfully installed ezconda

</div>

### Using `mamba`:
### Using `mamba`

<div class="termy">

Expand All @@ -71,7 +75,7 @@ Successfully installed ezconda

</div>

!!! Info
??? Info "mamba"
If you haven't heard of `mamba`, it offers higher speed and more reliable environment solutions. Learn more about `mamba` on their [website](https://mamba.readthedocs.io/en/latest/).

## A Minimal Example
Expand Down Expand Up @@ -123,6 +127,9 @@ dependencies:
- scipy
```
!!! Note
The `conda-forge` channel was also added to the specifications along with the packages.

### Remove packages

The specifications file is also updated when you remove packages.
Expand All @@ -147,24 +154,28 @@ dependencies:
- scipy
```

??? Info
If you try to remove a package that is a dependency for an installed package, **EZconda** will inform you before removing the package. See [docs](user_guide/remove_packages.md) for more details.

### Recreate environment

As you create, install and remove packages, in addition to the specifications file, **EZconda** also generates and maintains a lock file. You can use this lock file to reproducibly recreate an environment.

<div class="termy">

```console
$ ezconda recreate -n new-env ds-proj-osx-64.lock
$ ezconda recreate -n new-env ds-proj-darwin-x86_64.lock
// Creates a new environment 'new-env' that is identical to 'ds-proj'
```
</div>

!!! Info "Lock file"
You can learn more about environment recreation in the [docs](user_guide/recreate_env.md).
You can learn more about [environment recreation](user_guide/recreate_env.md) and [lock file](design_decisions/lockfile.md) in docs.


## Summary

In summary, **EZconda** creates a specifications file and a lock file for you as you create and manage your conda environment and packages.
In summary, **EZconda** provides a higher level abstraction for creating and managing `conda` environments.

To learn more, check out the [User Guide](user_guide/create_new_env.md)
2 changes: 1 addition & 1 deletion docs/user_guide/create_new_env.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ $ ezconda create -n new-proj python=3.9
💾 Saved specifications to 'new-proj.yml'

// Generates an environment lock file
🔒 Lock file generated ⚠ EXPERIMENTAL ⚠
🔒 Lock file generated

⭐ Done!
```
Expand Down
25 changes: 14 additions & 11 deletions docs/user_guide/install_packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ $ ezconda install -n new-proj numpy pandas
💾 Updated specifications to 'new-proj.yml'

// Updates lock file with new packages
🔒 Lock file updated ⚠ EXPERIMENTAL ⚠
🔒 Lock file updated
⭐ Done!
```
</div>
Expand Down Expand Up @@ -48,23 +48,26 @@ However, the lock file contains `libopenblas` and all other dependencies of `num

Check it:

```JSON title="new-proj lock file"
{
```TOML title="new-proj lock file"
[[packages]]
...
"dist_name": "libopenblas-0.3.13-hf4835c0_1",
"name": "libopenblas",
dist_name = "libopenblas-0.3.13-hf4835c0_1"
name = "libopenblas"
[[packages]]
...
"dist_name": "numpy-1.21.2-py39h6fc94f6_0",
"name": "numpy",
dist_name = "numpy-1.21.2-py39h6fc94f6_0"
name = "numpy"
[[packages]]
...
"dist_name": "numpy-base-1.21.2-py39h6ba5a95_0",
"name": "numpy-base",
dist_name = "numpy-base-1.21.2-py39h6ba5a95_0"
name = "numpy-base"
...
}
```

!!! Tip
You will learn more about these differences later in the docs.
You can learn more about these differences and lock file [here](../design_decisions/lockfile.md).

## From specific channel

Expand Down
2 changes: 1 addition & 1 deletion docs/user_guide/lock_existing_env.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ You can generate a lock file by providing `--name`/`-n`:
```console
$ ezconda lock -n chem-ml

// Generates 'chem-ml.lock' file
// Generates 'chem-ml.lock' file from existing `chem-ml` conda environment
```
</div>

Expand Down
25 changes: 18 additions & 7 deletions docs/user_guide/recreate_env.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ This is in addition to the *environment specifications* file (`environment.yml`
Basically, anytime you perform an action on the conda environment using **EZconda**, it updates the specifications file as well as the _**lock file**_.

!!! Note
The lock file is named after the environment name and the platform for which the environment was solved.
The lock file is named after the environment name and the platform and architecture for which the environment was solved.

## `recreate`

Expand All @@ -16,32 +16,34 @@ To reproduce an environment, use the `recreate` command -
<div class="termy">

```console
$ ezconda recreate new-proj-osx-64.lock
$ ezconda recreate new-proj-darwin-x86_64.lock

// Creates a new environment - 'new-proj-osx-64'
// Creates a new environment - 'new-proj-darwin-x86_64'

// Installs all the packages listed in the file
```
</div>

## Name the environment

You can provide a name for the new environment using the `-n` or `--name` option -
By default, **EZconda** will use the environment name specified in the lock file.

However, you can also provide a name for the new environment using the `-n` or `--name` option.

<div class="termy">

```console
$ ezconda recreate -n iris-2022 new-proj-osx-64.lock
$ ezconda recreate -n iris new-proj-darwin-x86_64.lock

// Creates a new environment - 'iris-2022' from the lock file
// Creates a new environment - 'iris' from the lock file
```
</div>

## Platform Specific Lock Files

Lock files are specific to the platform on which the environment was created.

This means an environment specifications file `env.yml` on Mac and Linux will have two different lock files - `env-osx-64.lock` and `env-linux-aarch64.lock`. This is because packages might have different dependencies for different platforms.
This means an environment specifications file `env.yml` on Mac and Linux will have two different lock files - `env-darwin-x86_64.lock` and `env-linux-aarch64.lock`. This is because packages might have different dependencies for different platforms.

For instance, installing `numpy` and `python=3.9` on *mac-osx-64* and *linux-aarch64* will result in slightly different dependencies.

Expand Down Expand Up @@ -69,3 +71,12 @@ For instance, installing `numpy` and `python=3.9` on *mac-osx-64* and *linux-aar
├─ python[3.9.9]
└─ python_abi[3.9]
```

## Lock file validation

If a lock file generated on a Mac is used to recreate an environment on a Linux machine, **EZconda** will throw a lock file validation error instead of creating broken environments.

!!! Note
This is different from how `conda` recreates environments from *explicit* specifications file.

`conda` does not check for platform or architecture compatibility while installing from *explicit* specifications file (also known as lock files).
2 changes: 1 addition & 1 deletion docs/user_guide/remove_packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ $ ezconda remove -n new-proj numpy
💾 Updated specifications to 'new-proj.yml'
// Updates lock file with new packages
🔒 Lock file updated ⚠ EXPERIMENTAL ⚠
🔒 Lock file updated
⭐ Done!
```
</div>
Expand Down
5 changes: 5 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,10 @@ nav:
- Use different solver: "user_guide/changing_solvers.md"
- Configurations : "user_guide/configuration.md"
- Autocompletions : "user_guide/autocomplete.md"
- Design Decisions :
- Introduction : "design_decisions/intro.md"
- Specifications File : "design_decisions/specfile.md"
- Lock File : "design_decisions/lockfile.md"
- Release Notes : "release_notes.md"
- License : "license.md"

Expand All @@ -65,6 +69,7 @@ markdown_extensions:
- pymdownx.tabbed:
alternate_style: true
- pymdownx.keys
- pymdownx.details
- pymdownx.emoji:
emoji_index: !!python/name:materialx.emoji.twemoji
emoji_generator: !!python/name:materialx.emoji.to_svg
Expand Down

0 comments on commit 20d7194

Please sign in to comment.