Skip to content

Commit

Permalink
X
Browse files Browse the repository at this point in the history
  • Loading branch information
John Major committed Nov 27, 2024
1 parent c9a9019 commit f41bb4a
Show file tree
Hide file tree
Showing 24 changed files with 62 additions and 62 deletions.
18 changes: 9 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -299,8 +299,8 @@ You can now SSH into the head node with the following command:
ssh -i /Users/daylily/.ssh/omics-analysis-b.pem [email protected]
Once logged in, as the 'ubuntu' user, run the following commands:
cd ~/projects/daylily
source dayinit -h
source dayinit --project PROJECT
source dyinit -h
source dyinit --project PROJECT
dy-a local
dy-r help
Expand Down Expand Up @@ -493,7 +493,7 @@ bin/daylily-ssh-into-headnode
**Is `daylily` CLI Available & Working**
```bash
cd ~/projects/daylily
. dayinit # inisitalizes the daylily cli
. dyinit # inisitalizes the daylily cli
dy-a local # activates the local config
dy-r help # should show the help menu
```
Expand All @@ -502,7 +502,7 @@ This should produce a magenta `WORKFLOW SUCCESS` message and `RETURN CODE: 0` at
###### Headnode Confiugration Incomplete
If there is no `~/projects/daylily` directory, or the `dayinit` command is not found, the headnode configuration is incomplete.
If there is no `~/projects/daylily` directory, or the `dyinit` command is not found, the headnode configuration is incomplete.
**Attempt To Complete Headnode Configuration**
From your remote terminal that you created the cluster with, run the following commands to complete the headnode configuration.
Expand Down Expand Up @@ -530,7 +530,7 @@ drwxrwxrwx 3 root root 33K Sep 26 08:35 resources
##### Run A Local Test Workflow
```bash
. dayinit --project PROJECT
. dyinit --project PROJECT
dy-a local
head -n 2 .test_data/data/giab_30x_hg38_analysis_manifest.csv
Expand Down Expand Up @@ -586,7 +586,7 @@ cd daylily
# prepare to run the test
tmux new -s slurm_test
. dayinit
. dyinit
dy-a slurm
# create a test manifest for one giab sample only, which will run on the 0.01x test dataset
Expand Down Expand Up @@ -642,7 +642,7 @@ cd /fsx/analysis_results/ubuntu/slurm_single
git clone https://github.com/Daylily-Informatics/daylily.git # or, if you have set ssh keys with github and intend to make changes: git clone [email protected]:Daylily-Informatics/daylily.git
cd daylily
. dayinit --project PROJECT
. dyinit --project PROJECT
dy-a slurm
# TO create a single sample manifest
Expand All @@ -665,7 +665,7 @@ cd /fsx/analysis_results/ubuntu/slurm_multi_30x_test
git clone https://github.com/Daylily-Informatics/daylily.git # or, if you have set ssh keys with github and intend to make changes: git clone [email protected]:Daylily-Informatics/daylily.git
cd daylily
. dayinit --project PROJECT
. dyinit --project PROJECT
dy-a slurm
# copy full 30x giab sample template to config/analysis_manifest.csv
Expand Down Expand Up @@ -1007,7 +1007,7 @@ active_aligners:
```
- This will enable running these two tools. You will also need a liscence file from sentieon in order to run these tools.
- Please [contact them](https://www.sentieon.com/company/) to obtain a valid liscense .
- Once you have a lisence file, edit the `dayinit` file to include the */fsx/* relative path to this file where `export SENTIEON_LICENSE=` is found.
- Once you have a lisence file, edit the `dyinit` file to include the */fsx/* relative path to this file where `export SENTIEON_LICENSE=` is found.
- Save the liscence file in the each region specific S3 reference bucket, ie: `s3://PREFIX-omics-analysis-REGION/data/cached_envs/`. When this bucket is mounted to the fsx filesystem, the liscence file will be available to all instances at `/fsx/data/cached_envs/`.
Expand Down
6 changes: 3 additions & 3 deletions bin/day_activate
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ DP=$DAY_PROJECT
# Attempt to deactivate any existing environment
source bin/day_deactivate &> /dev/null
echo "Attempting to deactivate existing environments & re-init with --project $DP ."
. dayinit --project $DP &> /dev/null
. dyinit --project $DP &> /dev/null
export DAY_PROFILE=$dayp

echo
# Check if DAY_ROOT is set
if [[ -z "$DAY_ROOT" ]]; then
echo "ERROR: DAY_ROOT is not set. Have you executed 'source dayinit'? Use -h for help."
echo "ERROR: DAY_ROOT is not set. Have you executed 'source dyinit'? Use -h for help."
return 2
fi

Expand Down Expand Up @@ -74,7 +74,7 @@ if [[ "$pe_ret_code" != "0" ]]; then
source bin/util/col_error.bash
echo "ERROR: Activation failed with return code $pe_ret_code."
echo ""
echo "Have you run 'day-build'? If so, try 'bin/day_deactivate reset; source dayinit; day-activate local;'"
echo "Have you run 'day-build'? If so, try 'bin/day_deactivate reset; source dyinit; day-activate local;'"
return $pe_ret_code
else
conda activate DAY
Expand Down
10 changes: 5 additions & 5 deletions bin/day_activate_o
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ gold_profiles/$DAY_PROFILE/*"
fi

if [[ "$1" == "" ]]; then
. dayinit
. dyinit
fi
envi=$1

Expand Down Expand Up @@ -41,7 +41,7 @@ else

**>> EXPECTED: $DAY_ROOT

You are strongly encouraged to run 'day-deactivate', then 'source dayinit && day-activate' from the intended analysis deployment directory.
You are strongly encouraged to run 'day-deactivate', then 'source dyinit && day-activate' from the intended analysis deployment directory.

**>> DETECTED: $PWD

Expand Down Expand Up @@ -118,7 +118,7 @@ then
# Something is not right....?
echo ""
echo ""
echo "DAY_ROOT is not set. Have you executed 'source dayinit'? Or re-run with -h"
echo "DAY_ROOT is not set. Have you executed 'source dyinit'? Or re-run with -h"
ret_code=2

if [[ "$1" == "h" || "$1" == "-h" || "$1" == "--help" || "$1" == "help" ]]; then
Expand Down Expand Up @@ -180,7 +180,7 @@ if [[ "$c" == "" ]]; then
echo " SUCCESS! But you seem to be missing the pkg colr, just attempted to add it."
else
echo " ---==]]>>>>> ERROR !!!!!!!!!!! "
echo " Returned with code $ex_code, try running a test 'day-activate local; day-run -c 3 -j 2 sleep ' otherwise day-deactivate; bash; source dayinit; day-activate local; If you have a 'colr not found error, pip may not have completed sucessfully, try: 'conda activate DAY; pip install --no-input -i https://pypi.org/simple --extra-index-url https://pypi.org/simple chromedriver-binary-auto rgbw-colorspace-converter colr docopt snakemake pandas xmltodict wrapt
echo " Returned with code $ex_code, try running a test 'day-activate local; day-run -c 3 -j 2 sleep ' otherwise day-deactivate; bash; source dyinit; day-activate local; If you have a 'colr not found error, pip may not have completed sucessfully, try: 'conda activate DAY; pip install --no-input -i https://pypi.org/simple --extra-index-url https://pypi.org/simple chromedriver-binary-auto rgbw-colorspace-converter colr docopt snakemake pandas xmltodict wrapt
"
return 22
fi
Expand All @@ -207,7 +207,7 @@ if [[ "$ret_code" != "0" ]]; then
colr "| ! \ " "turquoise" "debianred" "f"
colr "|_________/ " "turquoise" "debianred" "f"
colr "Something went awry. Check the README for alternate install instructuions. Else, ping JEM. ret code was $ret_code" "orangered" "black" "b"
echo "Have you run day_build? If so this is worth a shot:'bin/day_deactivate reset; source dayinit; day-activate local;'"
echo "Have you run day_build? If so this is worth a shot:'bin/day_deactivate reset; source dyinit; day-activate local;'"
return 0
else
colr " _______ GREAT SUCCESS _______ " "$DY_IT0" "$DY_IB0" "$DY_IS1"
Expand Down
12 changes: 6 additions & 6 deletions bin/day_build
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#!/usr/bin/env bash
# This script is intended to be sourced, not executed directly.

# Check if DAY_ROOT is set; if not, source dayinit
# Check if DAY_ROOT is set; if not, source dyinit
if [[ -z "$DAY_ROOT" ]]; then
source dayinit
source dyinit
fi

envi="$1"
Expand Down Expand Up @@ -33,8 +33,8 @@ else
fi
fi

# Source dayinit for local setup
source dayinit local &> /dev/null
# Source dyinit for local setup
source dyinit local &> /dev/null

# Check if 'colr' command is available
if ! command -v colr &> /dev/null; then
Expand All @@ -59,10 +59,10 @@ else
else
echo "GREAT SUCCESS!"
export PS1="$DYPS1 $PS1"
source dayinit local &> /dev/null
source dyinit local &> /dev/null
echo "Please start a new bash session and test the install:"
echo " bash;"
echo " source dayinit;"
echo " source dyinit;"
echo " dy-a local"
echo " dy-r help"
fi
Expand Down
8 changes: 4 additions & 4 deletions bin/day_build_o
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/echo "this script is intended to only be sourced"

if [[ "$DAY_ROOT" == "" ]]; then
. dayinit
. dyinit
fi

envi=$1
Expand Down Expand Up @@ -58,7 +58,7 @@ fi



source dayinit local > /dev/null 2>&1
source dyinit local > /dev/null 2>&1

c=`which colr` &> /dev/null || ls &> /dev/null

Expand Down Expand Up @@ -96,11 +96,11 @@ else
else
colr " _______ GREAT SUCCESS _______ " "chartreuse" "darkgreen" "b"
export PS1="$DYPS1 $PS1"
source dayinit local > /dev/null 2>&1
source dyinit local > /dev/null 2>&1
colr " please start a new bash session and test the install:

bash ;
source dayinit;
source dyinit;
dy-a local
dy-r help
"
Expand Down
2 changes: 1 addition & 1 deletion bin/day_deactivate
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ usage() {
echo "Usage: day-deactivate [activate|reset|--help]"
echo ""
echo "This script deactivates the Daylily environment by:"
echo " - Clearing the DAY_ROOT, DAY_PROFILE, and DAY_BIOME and other dayinit set environment variables."
echo " - Clearing the DAY_ROOT, DAY_PROFILE, and DAY_BIOME and other dyinit set environment variables."
echo " - Attempting to deactivate any active DAY conda environments."
echo " - Restoring the original PATH and PS1 if they were modified."
echo " - Unaliasing Daylily CLI commands."
Expand Down
4 changes: 2 additions & 2 deletions bin/day_deactivate_o
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ fi

ret_msg=""
if [[ $DAY_ROOT == "" && $DAY_PROFILE == "" ]]; then
ret_msg="No DAY environment was detected as active. However, if this is in error, try: 'source dayinit; day-deactivate reset;'"
ret_msg="No DAY environment was detected as active. However, if this is in error, try: 'source dyinit; day-deactivate reset;'"
fi

if [[ $1 == "activate" ]]; then
Expand Down Expand Up @@ -66,7 +66,7 @@ fi
echo " "
echo ' '

echo " ... if deactivate only partially ran and left a weird shell prompt, try: 'source dayinit; day-deactivate reset;' "
echo " ... if deactivate only partially ran and left a weird shell prompt, try: 'source dyinit; day-deactivate reset;' "

return 0

8 changes: 4 additions & 4 deletions bin/daylily-cfg-headnode
Original file line number Diff line number Diff line change
Expand Up @@ -107,18 +107,18 @@ ssh -i "$pem_file" ubuntu@"$cluster_ip_address" -o StrictHostKeyChecking=no -
# Initialize and configure the Daylily environment on the head node
echo "Configuring Daylily environment on the head node..."
ssh -t -i "$pem_file" ubuntu@"$cluster_ip_address" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
"sudo su - $duser -c 'cd ~/projects/daylily && source dayinit --skip-project-check && source bin/day_build'"
"sudo su - $duser -c 'cd ~/projects/daylily && source dyinit --skip-project-check && source bin/day_build'"

# Run a simple help test for Daylily remotely
echo "Running a simple help test on the head node..."
ssh -t -i "$pem_file" ubuntu@"$cluster_ip_address" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
"sudo su - $duser -c 'cd ~/projects/daylily && source dayinit --skip-project-check && source bin/day_activate local && source bin/day_run help'"
"sudo su - $duser -c 'cd ~/projects/daylily && source dyinit --skip-project-check && source bin/day_activate local && source bin/day_run help'"


# start building the minimal cached envs
echo "Building the minimal cached environments ..."
ssh -t -i "$pem_file" ubuntu@"$cluster_ip_address" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
"sudo su - $duser -c 'cd ~/projects/daylily && source dayinit --skip-project-check && source bin/day_activate local && source bin/day_run --conda-create-envs-only produce_snv_concordances produce_tiddit '"
"sudo su - $duser -c 'cd ~/projects/daylily && source dyinit --skip-project-check && source bin/day_activate local && source bin/day_run --conda-create-envs-only produce_snv_concordances produce_tiddit '"



Expand All @@ -127,7 +127,7 @@ echo "You can now SSH into the head node with the following command:"
echo "ssh -i $pem_file ubuntu@$cluster_ip_address"
echo "Once logged in, as the 'ubuntu' user, run the following commands:"
echo " cd ~/projects/daylily"
echo " source dayinit "
echo " source dyinit "
echo " dy-a local"
echo " dy-r help"
echo " "
Expand Down
4 changes: 2 additions & 2 deletions config/day/day_env_installer.sh
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ else
echo "DAY environment created successfully."
echo ""
echo "Try the following commands to get started:"
echo " source dayinit --project <PROJECT>"
echo " source dyinit --project <PROJECT>"
echo " dy-a local"
echo " dy-r help"
else
Expand All @@ -119,7 +119,7 @@ fi
echo ""
echo "Installation complete."
echo "Please log out and log back in, then run:"
echo " source dayinit --project <PROJECT>"
echo " source dyinit --project <PROJECT>"
echo " dy-a local"
echo " dy-r help"

Expand Down
6 changes: 3 additions & 3 deletions config/day/day_env_installer.sh_o
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ conda config --add channels bioconda

mgcnt=$(ls -d1 "$(dirname $(which conda))"/../envs/DAY | wc -l)
if [[ "$mgcnt" == "0" ]]; then
echo "Installing DAY \(which will be sourced for you when you \'source dayinit --project <PROJECT> \' \(see docs\) you should not need to source MG directly. Beginning install... "
echo "Installing DAY \(which will be sourced for you when you \'source dyinit --project <PROJECT> \' \(see docs\) you should not need to source MG directly. Beginning install... "
echo "..."
sleep 1.4

Expand All @@ -108,7 +108,7 @@ if [[ "$mgcnt" == "0" ]]; then
echo "Install exited with > $? < (if not zero, not the best sign)."
echo "try the following:
source dayinit --project <PROJECT> ; dy-a local; dy-r help;
source dyinit --project <PROJECT> ; dy-a local; dy-r help;
#initialized the day CLI, activate the local env settings, runs day to get basic help info on targets
"
Expand All @@ -127,6 +127,6 @@ fi;

echo "==="
echo "___"
echo 'you should log out of this shell and log back into a fresh shell, run source dayinit --project <PROJECT> , dy-a *****, etc.'
echo 'you should log out of this shell and log back into a fresh shell, run source dyinit --project <PROJECT> , dy-a *****, etc.'

return 0
2 changes: 1 addition & 1 deletion config/day/env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
#conda_bin=$(which conda)
#conda_dir=$(dirname $conda_bin)
#if [[ $conda_dir == "" ]]; then
# echo" >>> ERROR: conda has not been found in your PATH. If you do not have conda installed, you may run the mod installer+build command: source dayinit --project <PROJECT> && mod-activate BUILD_all"
# echo" >>> ERROR: conda has not been found in your PATH. If you do not have conda installed, you may run the mod installer+build command: source dyinit --project <PROJECT> && mod-activate BUILD_all"
#fi


Expand Down
2 changes: 1 addition & 1 deletion config/day/env.sh_o
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ conda activate || echo NOCONDA
conda_bin=$(which conda)
conda_dir=$(dirname $conda_bin)
if [[ $conda_dir == "" ]]; then
echo" >>> ERROR: conda has not been found in your PATH. If you do not have conda installed, you may run the mod installer+build command: source dayinit --project <PROJECT> && mod-activate BUILD_all"
echo" >>> ERROR: conda has not been found in your PATH. If you do not have conda installed, you may run the mod installer+build command: source dyinit --project <PROJECT> && mod-activate BUILD_all"
fi


Expand Down
2 changes: 1 addition & 1 deletion config/day_examples/vanilla_analysis_manifest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ Rclone is a very feature rich tool. You can find the docs here. In addition to

```
cd $DAY_ROOT
source dayinit --project <PROJECT>
source dyinit --project <PROJECT>
mg-a local
rclone -L --config config/day_example_analysis_manifests/rclone_xxx_local_example.conf cat localxxx:/fsx/data/external_data/research_experiments/ManualRuns/EX3636/b37/wgs_data/SQ42092_LS1585266_RE2698_G3/Lib_for_SequenceIndex_SQ42092.aligned.deduped.sort.bam | samtools view -H
```
Expand Down
10 changes: 5 additions & 5 deletions docs/first_ephemeral_cluster_analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ cd daylily
cp .test/

# initialize the DAY conda env + other helpful tools
. dayinit # you can speciy --project AWSBUDGETNAME to change from the default region specific budget. New budgets created in AWS must also be added to the allowed ubuntu projects in /opt/slurm/etc/projects_list.conf to be recognized by the cluster. jobs will not launch if the budget is not recognized.
. dyinit # you can speciy --project AWSBUDGETNAME to change from the default region specific budget. New budgets created in AWS must also be added to the allowed ubuntu projects in /opt/slurm/etc/projects_list.conf to be recognized by the cluster. jobs will not launch if the budget is not recognized.

# move an analysis manifest into place, which haas a tiny toy dataset.
cp .test_data/data/0.01xwgs_HG002_hg38.samplesheet.csv confiig/analysis_manifest.csv # YOU WILL USE THE ./bin/create_analysis_manifest script to create your own manifest for your own data when the time comes.
Expand Down Expand Up @@ -110,7 +110,7 @@ ls results/day/{hg38,b37}/reports/*.html
Copy the pre-configured giab analysis manifest for hg38, and run the analysis. This manifest points to the fastq.gz files for each, as well as the truth data set needed to produce the concordance reports.
```bash

. dayinit
. dyinit
dy-a slurm

vi config/day_profiles/slurm/rule_config.yaml # the default build is hg38, so no changes are needed here
Expand All @@ -130,7 +130,7 @@ ls results/day/hg38/other_reports/*giab*
You will need to modify the rule_config.yaml to use b37, and copy the pre-configured giab analysis manifest for b37.
```bash

. dayinit
. dyinit
dy-a slurm

# you will need to chang ethe reference build to b37 from hg38
Expand Down Expand Up @@ -183,7 +183,7 @@ bash bin/create_analysis_manifest -f fastq_manifest.tsv -o config/analysis_manif
- From the github repo directory on the headnode, in a tmux terminal, with your desired `config/analysis_manifest.csv` in place:
```bash

. dayinit
. dyinit
dy-a slurm

# If you wish to enable an aligner other than bwa mem2 or a variant caller other than deep variant, you will need to edit the config/day_profiles/slurm/rule_config.yaml before proceeding.
Expand All @@ -208,7 +208,7 @@ find results/ | grep -E 't.vcf.gz|bam.bai'
- From the github repo directory on the headnode, in a tmux terminal, with your desired `config/analysis_manifest.csv` in place:
```bash

. dayinit
. dyinit
dy-a slurm

dy-r produce_multiqc_final_wgs -p -k -j 9 -n
Expand Down
4 changes: 2 additions & 2 deletions docs/ops/dycli.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
```bash

# init the dy-cli
source dayinit -h
source dayinit --project <PROJECT>
source dyinit -h
source dyinit --project <PROJECT>

# activate the local or slurm running environment
dy-a local # or slurm # tab complete for avail envs
Expand Down
Loading

0 comments on commit f41bb4a

Please sign in to comment.