Restructuring container docs

_data/htc-guide-menu.yml

@@ -85,56 +85,55 @@
     - text: "Overview"
       url: "/uw-research-computing/software-overview-htc"
       icon: ""
-    - text: "Quickstart"
-      url: "/uw-research-computing/software-overview-htc#quickstart"
-      icon: ""
     - text: "Containers "
       icon: ""
       links:
-        - text: "Build a software environment with containers "
+        - text: "What are containers?"
+          url: "/uw-research-computing/software-overview-htc#container-overview"
+          icon: ""
+        - text: "Using software in Apptainer containers"
           icon: ""
           links:
-            - text: "What are containers?"
-              url: "/uw-research-computing/software-overview-htc#containers"
+            - text: "Use Apptainer containers"
+              url: "/uw-research-computing/apptainer-htc"
               icon: ""
-            - text: "Apptainer/Singularity"
+            - text: "More about Apptainer containers"
               url: "/uw-research-computing/apptainer-build"
-              icon: ""
+              icon: "bi bi-book"
               links:
                 - text: "Practice: Build a container"
                   url: "/uw-research-computing/apptainer-htc-advanced-example"
                   icon: "bi bi-clipboard-check"
-            - text: "Docker"
-              url: "/uw-research-computing/docker-build"
+            - text: "Convert Docker images to Apptainer/Singularity images"
+              url: "/uw-research-computing/htc-docker-to-apptainer"
               icon: ""
-              links:
-                - text: "Explore and test Docker containers"
-                  url: "/uw-research-computing/docker-test"
-                  icon: ""
-            - text: "Recipes repository"
-              url: "https://github.com/CHTC/recipes/tree/main/software"
-              icon: ""
-        - text: "Convert Docker images to Apptainer/Singularity images"
-          url: "/uw-research-computing/htc-docker-to-apptainer"
+        - text: "Using software in Docker containers"
+          icon: ""
+          links:
+            - text: "Use Docker containers"
+              url: "/uw-research-computing/docker-jobs"
+              icon: ""                
+            - text: "More about Docker containers"
+              url: "/uw-research-computing/docker-build"
+              icon: "bi bi-book"
+            - text: "Reference: Explore and test Docker containers"
+              url: "/uw-research-computing/docker-test"
+              icon: "bi bi-book"
+        - text: "Recipes repository"
+          url: "https://github.com/CHTC/recipes/tree/main/software"
+          icon: ""       
+    - text: "Other Modes of Software Use"
+      icon: ""
+      links:
+        - text: "Compile software in an interactive job"
+          url: "/uw-research-computing/inter-submit"
           icon: ""
-        - text: "Use Apptainer containers"
-          url: "/uw-research-computing/apptainer-htc"
+        - text: "Licensed software and modules"
+          url: "/uw-research-computing/licensed-software"
           icon: ""
-        - text: "Use Docker containers"
-          url: "/uw-research-computing/docker-jobs"
+        - text: "Use software available in modules"
+          url: "/uw-research-computing/htc-modules"
           icon: ""
-    - text: "Compile software in an interactive job"
-      url: "/uw-research-computing/inter-submit"
-      icon: ""
-    - text: "Licensed software and modules"
-      url: "/uw-research-computing/licensed-software"
-      icon: ""
-    - text: "Use software available in modules"
-      url: "/uw-research-computing/htc-modules"
-      icon: ""
-    - text: "HTC modules transition to EL9"
-      url: "/uw-research-computing/htc-modules-transition-to-el9"
-      icon: ""
 
 - text: "Workflows"
   icon: "bi bi-diagram-2"

_uw-research-computing/alphafold.md

@@ -0,0 +1,139 @@
+---
+highlighter: none
+layout: markdown-page
+title: Using Alphafold on CHTC's HTC System
+---
+
+
+Second -- to access the database:
+You should be able to submit jobs and use this option in the submit file:
+  requirements = (HasGpulabData == true)
+And when the job starts, you can access the database at:
+  /gpulab_data/alphafold
+
+Software that is packaged in a \"container\" can
+be run on the HPC cluster. This guide assumes that you are starting with 
+an existing Docker container and shows how to use it to run a job on the HPC cluster. 
+
+Note that you may need to install a version of MPI to your container 
+when it is initially created. See the [notes](#notes) about this below. 
+
+The two steps to run a container on the HPC cluster: 
+1.  [Convert the container to a Singularity image file](#image)
+2.  [Run a job that uses the container](#command)
+
+<a name="notes"></a>
+Notes about MPI and Containers
+==================
+
+There are two ways to run a Singularity container integrated with MPI: `hybrid` 
+mode and `bind` mode. 
+
+In `hybrid` mode, the container has its own copy of MPI **that is compatible 
+with a version of MPI already installed on the cluster**. 
+
+In `bind` mode, the code in the container has been compiled with MPI that 
+exists outside the container and there is no MPI installation in the container itself. 
+Again, **the version of MPI used needs to be compatible with one already installed
+on the cluster.**
+
+This will be relevant in how the job is executed later on: [Using Singularity Container Images](#command)
+
+<a name="image"></a>
+
+**1. Convert Container to Singularity Format**
+===================
+
+We assume that there is a Docker container (either found
+or created by you) online that you want to use. To use this container 
+on the HPC cluster, it needs to be converted to a Singularity-format
+image file. To do this: 
+
+1. Log in to one of the HPC cluster log in nodes. 
+1. Start an interactive job: 
+	```
+	[alice@login]$ srun -n4 -N1 -p int --pty bash
+	```
+
+1.  Once the interactive job starts, you'll need to unset a shell environment
+variable that prevents download of the Docker container. 
+	```
+	[alice@int]$ unset HTTPS_PROXY
+	```
+
+1. Then, save the Docker container to a Singularity image. 
+	``` 
+	[alice@int]$ singularity build /software/alice/name.simg docker://user/image:version
+	```
+	{:.term}
+
+	For example, if user \"Alice\" wanted to use the [\"Fenics\" container
+	provided on DockerHub](https://hub.docker.com/r/fenicsproject/stable),
+	and save it to a file named `fenics.simg`, she would run:
+
+	``` 
+	[alice@int]$ singularity build /software/alice/fenics.simg docker://fenicsproject/stable:latest
+	```
+	{:.term}
+
+	> This command will by default, pull the initial Docker container from 
+	> Docker Hub. If your Docker container is stored elsewhere, or you are 
+	> starting with a Singularity image, contact CHTC staff for specific instructions. 
+
+1. Once the Singularity command completes, type `exit` to leave the interactive job. 
+
+<a name="command"></a>
+
+**2. Using Singularity Container Images**
+===================
+
+To use a Singularity container in a job, the SLURM submit file will remain mostly the
+same; what will change is the job's primary command at the end of the
+file. This command will run your primary program inside the container
+file you\'ve downloaded. The main MPI command will still be part of the 
+singularity command: 
+
+``` {.sub}
+#!/bin/sh
+#SBATCH options
+
+module load MPI/version
+mpirun -np ## singularity exec /path/to/container/file command-to-run
+```
+
+For example, if Alice wanted to run a script she had written
+(`poisson.py`) inside the downloaded fenics container, using 40 cores, she would use the
+following command at the end of her submit file:
+
+``` {.sub}
+mpirun -np 40 singularity exec /software/alice/fenics.simg ./poisson.py
+```
+
+The example shown above uses the "hybrid" model for running MPI, which assumes 
+that there is a copy of MPI installed **in the container** that matches what already 
+exists on the cluster. 
+
+If your container does **not** have it's own copy of MPI installed, you need 
+to use the "bind" model for running MPI which requires an additional flag and 
+the location of the main MPI directory: 
+
+
+``` {.sub}
+#!/bin/sh
+#SBATCH options
+
+module load MPI/version
+mpirun -np ## singularity exec --bind /path/to/cluster/MPI/dir/ /path/to/container/file command-to-run
+```
+
+On CHTC's cluster, the GCC based version of OpenMPI is installed at the path:
+` /software/chtc/easybuild/v2/software/OpenMPI/4.0.5-GCC-9.3.0/`
+So the command(s) to run the "Alice" example above would be: 
+
+``` {.sub}
+MPI_DIR=/software/chtc/easybuild/v2/software/OpenMPI/4.0.5-GCC-9.3.0/
+mpirun -np 40 singularity exec --bind $MPI_DIR /software/alice/fenics.simg ./poisson.py
+```
+
+More details on the difference between using the "hybrid" and "bind" model 
+for MPI and Singularity is here: https://sylabs.io/guides/3.8/user-guide/mpi.html

_uw-research-computing/apptainer-build.md

@@ -10,30 +10,15 @@ guide:
 --- 
 
 This guide describes the general process for creating an Apptainer container. 
-Specifically, we discuss the components of the "definition file" and how that file is used to construct or "build" the container itself. 
+Specifically, we discuss the components of the "definition file" and how that file is used to construct or "build" the container itself. For a more step-by-step description of 
+how to build a container, see [Use Custom Software in Jobs Using Apptainer](apptainer-htc.html).
 
-For instructions on using and building Apptainer containers
-
-* on the High Throughput system: [Using Apptainer Containers](apptainer-htc.html).
-* on the High Performance system: [Using Apptainer Containers on HPC](apptainer-hpc.html).
 {% capture content %}
 
-[The Apptainer Definition File](#the-apptainer-definition-file)
-
-* [Header section](#header-section)
-* [Files section](#files-section)
-* [Post section](#post-section)
-* [Environment section](#environment-section)
-* [Labels section](#labels-section)
-* [Help section](#help-section)
-
-[The Apptainer Container Image](#the-apptainer-container-image)
-
-* [Building the container](#building-the-container)
-* [Converting a Docker image to an Apptainer container image](#converting-a-docker-image-to-an-apptainer-container-image)
-* [Testing the container interactively](#testing-the-container-interactively)
-
-[Special Considerations for Building Your Container](#special-considerations-for-building-your-container)
+* [The Apptainer definition file](#the-apptainer-definition-file)
+* [Creating a definition file](#creating-a-definition-file)
+* [The Apptainer container image](#the-apptainer-container-image)
+* [Special considerations for building your container](#special-considerations-for-building-your-container)
 
 {% endcapture %}
 {% include /components/directory.html title="Table of Contents" %}
@@ -236,7 +221,38 @@ For example,
 
 For an existing container, you can inspect the help text with the command `apptainer run-help my-container.sif`.
 
-## The Apptainer Container Image
+## Creating a definition file
+
+Here is general process for creating your own definition file for building your custom container:
+
+1. **Consult your software's documentation** 
+
+   Determine the requirements for installing the software you want to use.
+   In particular you are looking for (a) the operating systems it is compatible with and (b) the prerequisite libraries or packages.
+
+2. **Choose a base container** 
+
+   The base container should at minimum use an operating system compatible with your software. 
+   Ideally the container you choose also has many of the prerequisite libraries/programs already installed.
+
+3. **Customize your definition file**
+
+   In addition to examples in this guide, see our [Advanced Apptainer Example - SUMO](apptainer-htc-advanced-example.html) guide for an example of how to customize
+   a definition file for your software. 
+
+Remember that the `.def` file contains the *instructions* for creating your container and is not itself the container. 
+To use the software defined within the `.def` file, you will need to first "build" the container and create the `.sif` file, as described in the following sections.
+
+> ### 📝 Reference: CHTC's "Recipes" Repository
+{:.tip-header}
+
+> To see multiple examples of Apptainer definition files for different
+> softwares, check out the software section of CHTC's "recipes" repository: 
+> * [CHTC software recipes](https://github.com/CHTC/recipes/tree/main/software)
+{:.tip}
+
+
+## The Apptainer container image
 
 The actual container image, which can be executed by Apptainer as a stand-alone operating system, is stored in a `.sif` file.\* 
 The instructions for constructing the `.sif` file are provided by the `.def` definition file, as described above.
@@ -367,3 +383,12 @@ Usage: units [options] ['from-unit' 'to-unit']
   {:.term}
 
   If this does not address the issue, examine the error messages and consult the program documentation for how configure the program to use an alternate location for cache or temporary directories.
+
+
+## Related Pages
+
+* [Advanced Apptainer Example - SUMO](apptainer-htc-advanced-example.html)
+* [Convert Docker Images to Apptainer Images](htc-docker-to-apptainer.html)
+* [Use Custom Software in Jobs Using Apptainer](apptainer-htc.html)
+* [Using Apptainer Containers on HPC](apptainer-hpc.html)
+