diff --git a/README.md b/README.md
index c5c9dd3d..aaa1364d 100644
--- a/README.md
+++ b/README.md
@@ -1,4 +1,4 @@
-# Daylily AWS Ephemeral Cluster Setup (0.7.155)
+# Daylily AWS Ephemeral Cluster Setup (0.7.156)
_(0.7.155)_
**beta release**
@@ -87,7 +87,6 @@ I will be assessing:
> Would you like to lend a hand? [contact me](mailto:john@daylilyinformatics.com)
-
# What's It All About?
## BFAIR: Bioinformatics [FAIR](https://www.go-fair.org/fair-principles/) Principles
@@ -251,22 +250,25 @@ You may run in any region or AZ you wish to try. This said, the majority of test
## Prerequisites (On Your Local Machine)
-Local machine development has been exclusively on a mac, using the `zsh` shell (I am moving everything to bash, so there is a mix of bash and zsh at the moment. All `.` and `sourcing` that happens on your local machine expects `zsh`).
+Local machine development has been carried out exclusively on a mac using the `zsh` shell. `bash` should work as well (if you have issues with conda w/mac+bash, confirm that after miniconda install and conda init, the correct `.bashrc` and `.bash_profile` files were updated by `conda init`).
_suggestion: run things in tmux or screen_
Very good odds this will work on any mac and most Linux distros (ubuntu 22.04 are what the cluster nodes run). Windows, I can't say.
### System Packages
-Install with `brew` or `apt-get` ('):
-- `python3`, tested with `3.12.8`
+Install with `brew` or `apt-get`:
+- `python3`, tested with `3.11.0
- `git`, tested with `2.46.0`
-- `jq`, tested with `jq-1.7.1`
- `wget`, tested with `1.25.0`
- `awscli`, tested with `2.22.4` [AWS CLI docs](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
- `tmux` (optional, but suggested)
- `emacs` (optional, I guess, but I'm not sure how to live without it)
+#### Check if your prereq's meet the min versions required
+```bash
+./bin/check_prereq_sw.sh
+```
### AWS CLI Configuration
#### Opt 2
@@ -323,29 +325,32 @@ git clone https://github.com/Daylily-Informatics/daylily.git # or, if you have
cd daylily
```
-### Miniconda
+### Install Miniconda (homebrew is not advised)
_tested with conda version **`24.11.1`**_
-```bash
Install with:
```bash
-source bin/install_miniconda
+./bin/install_miniconda
```
-- This will leave you in a terminal with conda activated, indicated by `(base)` in the terminal prompt.
+- open a new terminal/shell, and conda should be available: `conda -v`.
-### DAYCLI Environment
+### Install DAYCLI Environment
_from `daylily` root dir_
```bash
-#!/bin/zsh
-source bin/init_daycli
+./bin/init_daycli
+
+conda activate DAYCLI
# DAYCLI should now be active... did it work?
colr 'did it work?' 0,100,255 255,100,0
```
-- This will leave you in a terminal with the conda DAYCLI activated, indicated by `(DAYCLI)` in the terminal prompt.
+
+- You should see:
+
+ > 

@@ -377,17 +382,17 @@ _running in a tmux/screen session is advised as the copy may take 1-many hours_
```bash
conda activate DAYCLI
# help
-bash ./bin/create_daylily_omics_analysis_s3.sh -h
+./bin/create_daylily_omics_analysis_s3.sh -h
-AWS_PROFILE=
+export AWS_PROFILE=
BUCKET_PREFIX=
REGION=us-west-2
# dryrun
-bash ./bin/create_daylily_omics_analysis_s3.sh --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX
+./bin/create_daylily_omics_analysis_s3.sh --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX
# run for real
-bash ./bin/create_daylily_omics_analysis_s3.sh --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX --disable-dryrun
+./bin/create_daylily_omics_analysis_s3.sh --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX --disable-dryrun
```
@@ -406,7 +411,7 @@ _this command will take ~5min to complete, and much longer if you expand to all
```bash
conda activate DAYCLI
-AWS_PROFILE=daylily-service
+export AWS_PROFILE=daylily-service
REGION=us-west-2
OUT_TSV=./init_daylily_cluster.tsv
@@ -474,14 +479,13 @@ Once you have selected an AZ && have a reference bucket ready in the region this
The following script will check a variety of required resources, attempt to create some if missing and then prompt you to select various options which will all be used to create a new parallel cluster yaml config, which in turn is used to create the cluster via `StackFormation`. [The template yaml file can be checked out here](config/day_cluster/prod_cluster.yaml).
-```zsh
-#!/bin/zsh
-AWS_PROFILE=daylily-service
+```bash
+export AWS_PROFILE=daylily-service
REGION_AZ=us-west-2c
-source bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE
+./bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE
# And to bypass the non-critical warnings (which is fine, not all can be resolved )
-source bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE --pass-on-warn # If you created an inline policy with a name other than daylily-service-cluster-policy, you will need to acknowledge the warning to proceed (assuming the policy permissions were granted other ways)
+./bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE --pass-on-warn # If you created an inline policy with a name other than daylily-service-cluster-policy, you will need to acknowledge the warning to proceed (assuming the policy permissions were granted other ways)
```
@@ -558,8 +562,7 @@ Once logged in, as the 'ubuntu' user, run the following commands:
### Run Remote Slurm Tests On Headnode
```bash
-#!/bin/zsh
-source ./bin/daylily-run-ephemeral-cluster-remote-tests $pem_file $region $AWS_PROFILE
+./bin/daylily-run-ephemeral-cluster-remote-tests $pem_file $region $AWS_PROFILE
```
A successful test will look like this:
@@ -579,7 +582,6 @@ pcluster list-clusters --region $REGION
[See the instructions here](#first-time-logging-into-head-node) to confirm the headnode is configured and ready to run the daylily pipeline.
-

# Costs
@@ -777,7 +779,7 @@ ssh -i $pem_file ubuntu@$cluster_ip_address
##### Facilitated
```bash
-AWS_PROFILE=
+export AWS_PROFILE=
bin/daylily-ssh-into-headnode
```
@@ -813,7 +815,7 @@ From your remote terminal that you created the cluster with, run the following c
```bash
conda activate DAYCLI
-source ./bin/daylily-cfg-headnode $PATH_TO_PEM $CLUSTER_AWS_REGION $AWS_PROFILE
+./bin/daylily-cfg-headnode $PATH_TO_PEM $CLUSTER_AWS_REGION $AWS_PROFILE
```
> If the problem persists, ssh into the headnode, and attempt to run the commands as the ubuntu user which are being attempted by the `daylily-cfg-headnode` script.
diff --git a/bin/calc_daylily_aws_cost_estimates.py b/bin/calc_daylily_aws_cost_estimates.py
index dc421eaf..95b8fb14 100755
--- a/bin/calc_daylily_aws_cost_estimates.py
+++ b/bin/calc_daylily_aws_cost_estimates.py
@@ -1,3 +1,5 @@
+#!/bin/env python
+
from daylib.day_cost_ec2 import AWSGenomicsAnalyzer
def main():
diff --git a/bin/calcuate_spotprice_for_cluster_yaml.py b/bin/calcuate_spotprice_for_cluster_yaml.py
index e2b90858..2ee53b66 100755
--- a/bin/calcuate_spotprice_for_cluster_yaml.py
+++ b/bin/calcuate_spotprice_for_cluster_yaml.py
@@ -1,4 +1,5 @@
#!/usr/bin/env python
+
import subprocess
import statistics
import argparse
diff --git a/bin/cat_input_files.sh b/bin/cat_input_files.sh
index 322f1933..72c287ef 100755
--- a/bin/cat_input_files.sh
+++ b/bin/cat_input_files.sh
@@ -1,10 +1,5 @@
#!/bin/bash
-# Ensure compatibility with zsh and bash
-if [ -n "$ZSH_VERSION" ]; then
- emulate -LR bash # Ensure bash-like behavior in zsh
-fi
-
# Display usage information
usage() {
echo "Usage: $0 -s SAMPLENAME -p SAMPLE_MATCH_PATTERN -i INPUT_DIR -o OUTPUT_DIR"
@@ -42,7 +37,7 @@ if [[ ! -d "$OUTPUT_DIR" ]]; then
fi
echo ""
-echo ">>> >> > THIS IS A SIMPLE UTILITY AND NOT PRODUCTION TESTED FOR YOUR USE CASE < << <<<"
+echo ">>> >> > THIS IS A SIMPLE UTILITY AND NOT PRODUCTION TESTED < << <<<"
echo ""
sleep 2
diff --git a/bin/check_aws_permissions.sh b/bin/check_aws_permissions.sh
old mode 100644
new mode 100755
diff --git a/bin/check_current_spot_market_by_zones.py b/bin/check_current_spot_market_by_zones.py
index cb717cdc..e773ef28 100755
--- a/bin/check_current_spot_market_by_zones.py
+++ b/bin/check_current_spot_market_by_zones.py
@@ -466,6 +466,17 @@ def main():
if args.profile == "SETME":
print(f"\n\nERROR:: Your AWS_PROFILE is set to >{os.getenv('AWS_PROFILE')}<, Please set your AWS_PROFILE environment variable and specify the same string with --profile.\n")
raise SystemExit
+ if os.environ.get('AWS_PROFILE','') == '':
+ os.environ['AWS_PROFILE'] = args.profile
+ elif args.profile != os.environ['AWS_PROFILE']:
+ print(f"\n\nERROR:: Your AWS_PROFILE is set to >{os.getenv('AWS_PROFILE')}<, Please set your AWS_PROFILE environment variable and specify the same string with --profile.\n")
+ raise SystemExit
+ elif args.profile == os.environ['AWS_PROFILE']:
+ print(f"\n\nAWS_PROFILE is set to >{os.getenv('AWS_PROFILE')}< and profile passed is >{args.profile}<, all good.\n")
+ else:
+ print(f"\n\nAWS_PROFILE is set to >{os.getenv('AWS_PROFILE')}< and profile passed is >{args.profile}<, there is a problem.\n")
+ raise SystemExit
+
zones = args.zones.split(',')
spot_data = collect_spot_prices(instance_types, zones, args.profile)
diff --git a/bin/check_system_pkgs.sh b/bin/check_prereq_sw.sh
similarity index 83%
rename from bin/check_system_pkgs.sh
rename to bin/check_prereq_sw.sh
index 4e9ce48f..17791e79 100755
--- a/bin/check_system_pkgs.sh
+++ b/bin/check_prereq_sw.sh
@@ -2,9 +2,8 @@
# Set required versions
required_versions=(
- "python3:3.12.8"
+ "python3:3.11.0"
"git:2.46.0"
- "jq:1.7.1"
"wget:1.25.0"
"aws:2.22.4"
)
@@ -14,9 +13,8 @@ pass_on_warn=${pass_on_warn:-0}
# Function to compare versions
check_version() {
- local tool=$1
- local required=$2
- local current=$3
+ local required=$1
+ local current=$2
if [[ "$(printf '%s\n' "$required" "$current" | sort -V | head -n1)" != "$required" ]]; then
return 1 # Current version is less than required
@@ -38,12 +36,13 @@ check_tool() {
fi
# Check version
- if ! check_version "$tool" "$required_version" "$current_version"; then
+ if ! check_version "$required_version" "$current_version"; then
echo "Error: $tool version $current_version does not meet the required version $required_version."
[[ "$pass_on_warn" -eq 1 ]] && return 0 || exit 1
fi
- echo "$tool version $current_version meets the requirement of $required_version or higher."
+ # Print success message if version is sufficient
+ echo "$tool version $current_version meets the required version $required_version or higher."
}
# Loop through required tools and check
@@ -58,9 +57,6 @@ for entry in "${required_versions[@]}"; do
git)
check_tool "git" "$required_version" "git --version | awk '{print \$3}'"
;;
- jq)
- check_tool "jq" "$required_version" "jq --version | sed 's/jq-//g'"
- ;;
wget)
check_tool "wget" "$required_version" "wget --version | head -n1 | awk '{print \$3}'"
;;
diff --git a/bin/create_budget.sh b/bin/create_budget.sh
index c1c231b7..a69827de 100755
--- a/bin/create_budget.sh
+++ b/bin/create_budget.sh
@@ -78,9 +78,9 @@ write_or_append_tags_to_s3() {
echo "Writing or appending tags to S3 file: $S3_BUCKET_PATH" url: $S3_BUCKET_URL
# Check if the file exists in S3
- if aws s3 ls "$S3_BUCKET_PATH" > /dev/null 2>&1; then
+ if aws s3 ls "$S3_BUCKET_PATH" --profile $AWS_PROFILE > /dev/null 2>&1; then
echo "File exists. Downloading the existing file."
- aws s3 cp "$S3_BUCKET_PATH" "$TEMP_LOCAL_FILE" --region "$REGION"
+ aws s3 cp "$S3_BUCKET_PATH" "$TEMP_LOCAL_FILE" --region "$REGION" --profile $AWS_PROFILE
if [[ $? -ne 0 ]]; then
echo "ERROR: Failed to download existing file from S3."
@@ -97,7 +97,7 @@ write_or_append_tags_to_s3() {
# Upload the updated file back to S3
echo "Uploading updated file to S3..."
- aws s3 cp "$TEMP_LOCAL_FILE" "$S3_BUCKET_PATH" --region "$REGION"
+ aws s3 cp "$TEMP_LOCAL_FILE" "$S3_BUCKET_PATH" --region "$REGION" --profile $AWS_PROFILE
if [[ $? -eq 0 ]]; then
echo "Successfully updated S3 file: $S3_BUCKET_PATH"
@@ -153,12 +153,12 @@ BUDGET_JSON=$(echo "$BUDGET_TEMPLATE" | \
TEMP_BUDGET_JSON=$(mktemp)
echo "$BUDGET_JSON" > "$TEMP_BUDGET_JSON"
-ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
+ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text --profile $AWS_PROFILE)
# Create the budget using AWS CLI
echo "Creating budget for project: $PROJECT_NAME with amount: $AMOUNT USD in region: $REGION"
aws budgets create-budget --account-id "$ACCOUNT_ID" \
- --budget file://"$TEMP_BUDGET_JSON" --region "$REGION"
+ --budget file://"$TEMP_BUDGET_JSON" --region "$REGION" --profile $AWS_PROFILE
# Check if the budget creation succeeded
if [[ $? -eq 0 ]]; then
@@ -169,7 +169,7 @@ if [[ $? -eq 0 ]]; then
IFS=',' read -ra THRESHOLD_ARRAY <<< "$THRESHOLDS"
for threshold in "${THRESHOLD_ARRAY[@]}"; do
echo P:$PROJECT_NAME T:$threshold E:$EMAIL R:$REGION A:$AMOUNT AI:$ACCOUNT_ID
- aws budgets create-notification --account-id "$ACCOUNT_ID" \
+ aws budgets create-notification --account-id "$ACCOUNT_ID" --profile $AWS_PROFILE \
--budget-name "$PROJECT_NAME" \
--notification '{"ComparisonOperator":"GREATER_THAN","NotificationType":"ACTUAL","Threshold":'"$threshold"',"ThresholdType":"PERCENTAGE"}' \
--region "$REGION" --subscribers '[{"Address":"'"$EMAIL"'","SubscriptionType":"EMAIL"}]'
diff --git a/bin/daylily-analysis-samples-to-manifest b/bin/daylily-analysis-samples-to-manifest
deleted file mode 100755
index 63d4a088..00000000
--- a/bin/daylily-analysis-samples-to-manifest
+++ /dev/null
@@ -1,202 +0,0 @@
-#!/usr/bin/env bash
-
-# Neon color scheme
-INFO="\033[1;36m"
-WARN="\033[1;33m"
-ERROR="\033[1;31m"
-SUCCESS="\033[1;32m"
-RESET="\033[0m"
-
-info() { echo -e "${INFO}[INFO]${RESET} $1"; }
-warn() { echo -e "${WARN}[WARN]${RESET} $1"; }
-error() { echo -e "${ERROR}[ERROR]${RESET} $1"; exit 1; }
-success() { echo -e "${SUCCESS}[SUCCESS] $1"; }
-
-AM_OUT_FILE="na"
-
-check_dependencies() {
- for cmd in ssh; do
- if ! command -v $cmd &>/dev/null; then
- error "Required command '$cmd' is not installed."
- fi
- done
-}
-
-validate_args() {
- if [[ "$1" != "local" && "$1" != "remote" ]]; then
- error "First argument must be 'local' or 'remote'."
- fi
- if [[ ! -f "$2" ]]; then
- error "Sample sheet '$2' does not exist."
- fi
-}
-
-determine_sex() {
- local n_x=$1
- local n_y=$2
- if [[ "$n_x" -eq 2 && "$n_y" -eq 0 ]]; then
- echo "female"
- elif [[ "$n_x" -eq 1 && "$n_y" -eq 1 ]]; then
- echo "male"
- else
- echo "na"
- fi
-}
-
-validate_stage_target() {
- local stage_target="$1"
- local mode="$2"
- local cluster_ip="$3"
- local pem_file="$4"
- local cluster_user="$5"
-
- info "Validating STAGE_TARGET: $mode $stage_target "
- if [[ ! "$stage_target" =~ ^/fsx ]]; then
- error "STAGE_TARGET '$stage_target' must begin with '/fsx'."
- fi
-
- if [[ "$mode" == "remote" ]]; then
- info "Validating STAGE_TARGET on remote cluster: $stage_target"
- ssh -i "$pem_file" "$cluster_user@$cluster_ip" "if [[ ! -d \"$stage_target\" ]]; then mkdir -p \"$stage_target\"; fi" \
- || error "Failed to create STAGE_TARGET directory on remote: $stage_target"
- else
- if [[ ! -d "$stage_target" ]]; then
- info "Creating STAGE_TARGET directory locally: $stage_target"
- mkdir -p "$stage_target" || error "Failed to create STAGE_TARGET directory: $stage_target"
- fi
- fi
-}
-
-stage_data_files() {
- local r1_src="$1"
- local r2_src="$2"
- local stage_target="$3"
- local sample_prefix="$4"
- local mode="$5"
- local cluster_ip="$6"
- local pem_file="$7"
- local cluster_user="$8"
-
- local r1_dest="${stage_target}/${sample_prefix}.R1.fastq.gz"
- local r2_dest="${stage_target}/${sample_prefix}.R2.fastq.gz"
-
- if [[ "$mode" == "remote" ]]; then
- info "Staging R1 to remote: $r1_dest"
-
- ssh -i "$pem_file" "$cluster_user@$cluster_ip" "if [[ -f '$r1_dest' ]]; then exit 1; fi" && \
- echo "File does not exist on remote" || \
- { echo "Error: File '$r1_dest' already exists on remote."; exit 1; }
-
- scp -i "$pem_file" "$r1_src" "${cluster_user}@${cluster_ip}:${r1_dest}" || error "Failed to stage R1 to remote."
- success "R1 staged to remote: $r1_dest"
-
- ssh -i "$pem_file" "$cluster_user@$cluster_ip" "if [[ -f '$r2_dest' ]]; then exit 1; fi" && \
- echo "File does not exist on remote" || \
- { echo "Error: File '$r2_dest' already exists on remote."; exit 1; }
- info "Staging R2 to remote: $r2_dest"
- scp -i "$pem_file" "$r2_src" "${cluster_user}@${cluster_ip}:${r2_dest}" || error "Failed to stage R2 to remote."
- success "R2 staged to remote: $r2_dest"
- else
- [[ ! -e "$r1_src" ]] && error "R1 file '$r1_src' does not exist."
- [[ ! -e "$r2_src" ]] && error "R2 file '$r2_src' does not exist."
-
- cp "$r1_src" "$r1_dest" || error "Failed to copy R1: $r1_src to $r1_dest"
- success "R1 staged locally: $r1_dest"
-
- cp "$r2_src" "$r2_dest" || error "Failed to copy R2: $r2_src to $r2_dest"
- success "R2 staged locally: $r2_dest"
- fi
-}
-
-parse_and_validate_tsv() {
- local input_file="$1"
- local mode="$2"
- local cluster_ip="$3"
- local pem_file="$4"
- local cluster_user="$5"
-
-
- while IFS=$'\t' read -r RUN_ID SAMPLE_ID SAMPLE_TYPE LIB_PREP SEQ_PLATFORM LANE SEQBC_ID PATH_TO_CONCORDANCE_DATA_DIR R1_FQ R2_FQ STAGE_DIRECTIVE STAGE_TARGET SUBSAMPLE_PCT IS_POS_CTRL IS_NEG_CTRL N_X N_Y; do
- [[ "$LANE" != "0" ]] && error "Invalid LANE value '$LANE'. Only '0' is allowed."
- [[ "$IS_POS_CTRL" != "na" || "$IS_NEG_CTRL" != "na" ]] && error "Positive or Negative Control columns are not enabled yet."
-
- echo "Processing: $RUN_ID $SAMPLE_ID $SAMPLE_TYPE $LIB_PREP $SEQ_PLATFORM $LANE $SEQBC_ID $PATH_TO_CONCORDANCE_DATA_DIR $R1_FQ $R2_FQ $STAGE_DIRECTIVE $STAGE_TARGET $SUBSAMPLE_PCT $IS_POS_CTRL $IS_NEG_CTRL $N_X $N_Y"
-
- AS_IN_FILE="${STAGE_TARGET}/${RUN_ID}.analysis_samples.tsv"
- tmp_am_out_file="./${RUN_ID}.analysis_manifest.csv.tmp"
- touch "$tmp_am_out_file"
- AM_OUT_FILE="${STAGE_TARGET}/${RUN_ID}.analysis_manifest.csv"
- done < <(tail -n +2 "$input_file")
-
- echo "ASIO $AS_IN_FILE ... $AM_OUT_FILE ... $tmp_am_out_file"
-
- echo "samp,sample,sample_lane,SQ,RU,EX,LANE,r1_path,r2_path,biological_sex,iddna_uid,concordance_control_path,is_positive_control,is_negative_control,sample_type,merge_single,external_sample_id,instrument,lib_prep,bwa_kmer,subsample_pct" >"$tmp_am_out_file"
-
- while IFS=$'\t' read -r RUN_ID SAMPLE_ID SAMPLE_TYPE LIB_PREP SEQ_PLATFORM LANE SEQBC_ID PATH_TO_CONCORDANCE_DATA_DIR R1_FQ R2_FQ STAGE_DIRECTIVE STAGE_TARGET SUBSAMPLE_PCT IS_POS_CTRL IS_NEG_CTRL N_X N_Y; do
- [[ "$LANE" != "0" ]] && error "Invalid LANE value '$LANE'. Only '0' is allowed."
- [[ "$IS_POS_CTRL" != "na" || "$IS_NEG_CTRL" != "na" ]] && error "Positive or Negative Control columns are not enabled yet."
-
- echo "Processing: $RUN_ID $SAMPLE_ID $SAMPLE_TYPE $LIB_PREP $SEQ_PLATFORM $LANE $SEQBC_ID $PATH_TO_CONCORDANCE_DATA_DIR $R1_FQ $R2_FQ $STAGE_DIRECTIVE $STAGE_TARGET $SUBSAMPLE_PCT $IS_POS_CTRL $IS_NEG_CTRL $N_X $N_Y"
- local biological_sex=$(determine_sex "$N_X" "$N_Y")
- local sample_prefix="${RUN_ID}_${SAMPLE_ID}_${SAMPLE_TYPE}_${LANE}-${SEQBC_ID}"
-
- echo "----SD $STAGE_DIRECTIVE"
- validate_stage_target "$STAGE_TARGET" "$mode" "$cluster_ip" "$pem_file" "$cluster_user"
-
- if [[ "$STAGE_DIRECTIVE" == "stage_data" ]]; then
- stage_data_files "$R1_FQ" "$R2_FQ" "$STAGE_TARGET" "$sample_prefix" "$mode" "$cluster_ip" "$pem_file" "$cluster_user"
- fi
-
- local staged_r1="${STAGE_TARGET}/${sample_prefix}.R1.fastq.gz"
- local staged_r2="${STAGE_TARGET}/${sample_prefix}.R2.fastq.gz"
- echo "${sample_prefix},${sample_prefix},${sample_prefix},${SAMPLE_TYPE},${RUN_ID},${SAMPLE_ID},${LANE}-${SEQBC_ID},${staged_r1},${staged_r2},${biological_sex},na,${PATH_TO_CONCORDANCE_DATA_DIR},na,na,${SAMPLE_TYPE},merge,${SAMPLE_ID},${SEQ_PLATFORM},${LIB_PREP},19,${SUBSAMPLE_PCT}" >>"$tmp_am_out_file"
- done < <(tail -n +2 "$input_file")
-
-
- if [[ "$mode" == "remote" ]]; then
-
- ssh -i "$pem_file" "$cluster_user@$cluster_ip" "if [[ -f '$AS_IN_FILE' ]]; then exit 1; fi" && \
- echo "File does not exist on remote" || \
- { echo "Error: File '$AS_IN_FILE' already exists on remote."; exit 1; }
-
- scp -i "$pem_file" "$input_file" "${cluster_user}@${cluster_ip}:${AS_IN_FILE}" || error "Failed to stage R1 to remote."
- success "Analysis samples $input_file moved to remote: scp -i $pem_file $input_file ${cluster_user}@${cluster_ip}:$AS_IN_FILE"
-
-
- ssh -i "$pem_file" "$cluster_user@$cluster_ip" "if [[ -f '$AM_OUT_FILE' ]]; then exit 1; fi" && \
- echo "File does not exist on remote" || \
- { echo "Error: File '$AM_OUT_FILE' already exists on remote."; exit 1; }
-
- scp -i "$pem_file" "$tmp_am_out_file" "${cluster_user}@${cluster_ip}:${AM_OUT_FILE}" || error "Failed to stage R1 to remote."
- success "$tmp_am_out_file moved to remote: scp -i $pem_file $input_file ${cluster_user}@${cluster_ip}:$AM_OUT_FILE"
-
-
-
- else
- echo "pass on copy $AS_IN_FILE"
- fi
-}
-
-main() {
- local mode="$1"
- local input_file="$2"
- local cluster_ip="$3"
- local pem_file="$4"
- local cluster_user="$5"
-
- check_dependencies
- validate_args "$@"
-
- #if [[ "$mode" == "remote" ]]; then
- # ssh -i "$pem_file" "$cluster_user@$cluster_ip" \
- # "if [[ ! -d '/fsx/staged_sample_data' ]]; then mkdir '/fsx/staged_sample_data'; fi" \
- # || error "Failed to create '/fsx/staged_sample_data' directory on remote cluster."
- #else
- # mkdir "/fsx/staged_sample_data" || error "Failed to create local '/fsx/staged_sample_data'."
- #fi
-
- parse_and_validate_tsv "$input_file" "$mode" "$cluster_ip" "$pem_file" "$cluster_user"
- success "Manifest generated: $AM_OUT_FILE"
-}
-
-main "$@"
diff --git a/bin/daylily-cfg-headnode b/bin/daylily-cfg-headnode
old mode 100644
new mode 100755
index 278eddb5..841cdfc6
--- a/bin/daylily-cfg-headnode
+++ b/bin/daylily-cfg-headnode
@@ -1,13 +1,7 @@
-#!/bin/zsh
+#!/bin/bash
# Daylily Headnode Configuration Script
-# Ensure the script is sourced, not executed directly
-if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
- echo "This script should be sourced, not executed directly. Please use 'source THISSCRIPT' to run."
- exit 1
-fi
-
# Capture arguments for PEM file and region
pem_file=$1
region=$2
@@ -142,7 +136,7 @@ echo " "
echo "___bonus round___"
echo "Would you like to start building various caches needed to run jobs? [y/n]"
-timeout 120 bash -c 'read -p "Press y to start or n to skip: " REPLY && echo "$REPLY"' && export REPLY=$REPLY
+timeout 30 bash -c 'read -p "Press y to start or n to skip: " REPLY && echo "$REPLY"' && export REPLY=$REPLY
if [[ "$REPLY" == "y" ]]; then
# start building the minimal cached envs
@@ -161,6 +155,6 @@ echo "And, you may now access the headnode via the PCUI, via 'source bin/daylily
echo -e " ssh -i $pem_file ubuntu@$cluster_ip_address\n"
echo " "
echo "If you wish to run some remote tests, you can do so with the following command:"
-echo -e " . bin/daylily-run-ephemeral-cluster-remote-tests $pem_file $region $aws_profile\n"
+echo -e " ./bin/daylily-run-ephemeral-cluster-remote-tests $pem_file $region $aws_profile\n"
echo " "
echo "...fin"
diff --git a/bin/daylily-create-ephemeral-cluster b/bin/daylily-create-ephemeral-cluster
old mode 100644
new mode 100755
index c57e4983..2e39d1b2
--- a/bin/daylily-create-ephemeral-cluster
+++ b/bin/daylily-create-ephemeral-cluster
@@ -1,38 +1,15 @@
-#!/bin/zsh
+#!/bin/bash
-# Function to check if the script is sourced
-is_sourced() {
- # If the script is sourced, BASH_SOURCE[0] != $0
- [[ "${BASH_SOURCE[0]}" != "${0}" ]]
-}
-
-CURR_SHELL=$(ps -p $$ -o comm=)
-echo $CURR_SHELL
-
-if [[ "$CURR_SHELL" != "zsh" ]]; then
- echo "ERROR: This script must be executed from a zsh shell. please run 'zsh' and re-run this."
- ~/miniconda3/bin/conda init zsh || echo "conda init failed, possibly legit if conda is not installed yet."
-
- if is_sourced; then
- # If sourced, use 'return' to avoid exiting the shell
- return 3
- else
- # If executed, use 'exit' to terminate the script
- exit 3
- fi
-fi
-
-echo "Now running in zsh. Continuing script execution..."
-# Your script logic here
-emulate -L bash
+CURR_SHELL=$(. bin/retshell)
+echo "$CURR_SHELL"
# Default configuration
-pass_on_warn='' # Default for warnings causing failures
+pass_on_warn=0 # Default for warnings causing failures
# Function to display help
usage() {
- echo "Usage: source $0 [--region-az REGION-AZ] [--pass-on-warn]"
+ echo "Usage: $0 --profile AWS_PROFILE [--region-az REGION-AZ] [--pass-on-warn]"
echo " --region-az REGION-AZ Specify the AWS region and availability zone (tested w us-west-2c/d)"
echo " --pass-on-warn Allow warnings to pass without failure (default: false)"
echo " --profile aws profile to use"
@@ -42,12 +19,12 @@ usage() {
# Check if script is sourced with no arguments
if [[ $# -eq 0 ]]; then
usage
- return 0
+ exit 0
fi
if [ -z "$AWS_PROFILE" ]; then
echo "Error: AWS_PROFILE is not set."
- return 1 # Exit the function with an error status
+ exit 1 # Exit the function with an error status
fi
# Parse command-line arguments
@@ -55,16 +32,16 @@ while [[ $# -gt 0 ]]; do
case "$1" in
--region-az)
region_az="$2"
- region="${region_az::-1}" # Remove the last character from the AZ
+ region="${region_az:0:${#region_az}-1}"
shift 2
;;
--pass-on-warn)
- pass_on_warn=true
+ pass_on_warn="1"
shift
;;
-h|--help)
usage
- return 0 # Use exit instead of return
+ exit 0
;;
--profile)
AWS_PROFILEIN="$2"
@@ -73,31 +50,24 @@ while [[ $# -gt 0 ]]; do
*)
echo "Error: Unknown option: $1"
usage
- return 3 # Use exit instead of return
+ exit 3
;;
esac
done
-
-# Ensure the script is sourced
-if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
- echo "Error: This script must be sourced, not executed directly. Use 'source $0' to run."
- exit 3
-fi
-
if [[ -z "$AWS_PROFILE" ]]; then
echo "Error: --profile flag not set"
- return 3 # Exit the function with an error status
+ exit 3 # Exit the function with an error status
fi
if [[ "$AWS_PROFILE" != "$AWS_PROFILEIN" ]]; then
echo "Error: AWS_PROFILE and --profile must match: $AWS_PROFILE and $AWS_PROFILEIN"
- return 3 # Exit the function with an error status
+ exit 3 # Exit the function with an error status
fi
if [[ -z "$region_az" ]]; then
echo "Error: --region-az flag not set"
- return 3 # Exit the function with an error status
+ exit 3 # Exit the function with an error status
fi
# Extract the last character
@@ -108,12 +78,12 @@ if [[ "$az_char" =~ [a-zA-Z] ]]; then
echo "The last character of --region-az ${region_az} '$az_char' is a letter."
else
echo "ERROR: The last character of --region-az ${region_az} '$az_char' is a NOT letter."
- return 3 # Exit the function with an error status
+ exit 3 # Exit the function with an error status
fi
if [[ "$AWS_PROFILEIN" != "$AWS_PROFILE" ]]; then
echo "Error: AWS_PROFILE and --profile must match: $AWS_PROFILE and $AWS_PROFILEIN"
- return 3 # Exit the function with an error status
+ exit 3 # Exit the function with an error status
fi
AWS_PROFILE=$AWS_PROFILE
@@ -123,58 +93,71 @@ echo "YOUR AWS_PROFILE IS NOW SET TO: $AWS_PROFILE"
handle_warning() {
local message="$1"
echo -e "WARNING: $message"
- echo "If appropriate, consider setting --pass-on-warn to continue without failure. -$pass_on_warn-"
- if [[ "$pass_on_warn" == "" ]]; then
- echo "exiting"
- return 3
+ echo "If appropriate, consider setting --pass-on-warn to continue without failure. Current setting: $pass_on_warn"
+ if [[ "$pass_on_warn" != "1" ]]; then
+ echo "Exiting due to warning."
+ exit 3
fi
}
validate_region() {
echo "REGION:" $1
}
+echo "DEBUG: pass_on_warn after parsing: $pass_on_warn"
-echo "Welcome to the Daylily CLI Setup"
-echo "Daylily is tested to run in region $region (AZ: $region_az)."
-echo "Please ensure your ~/.aws/config file matches this region."
-
-### Conda Environment Setup
-
-
-# Ensure Conda exists
-conda --version &> /dev/null
-if [[ $? -ne 0 ]]; then
- echo "Error: Conda is not available in this shell. Attempting to determine if autoinstall is possible."
-
- source bin/install_miniconda
+# Check if system packages are installed
+./bin/check_prereq_sw.sh
+if [ $? -ne 0 ]; then
+ handle_warning "There were problems detected with pre-requisites, and the warnings bypassed. Run './bin/check_prereq_sw.sh' for more information ."
+ if [[ $? -ne 0 ]]; then
+ echo "Error: System package check failed, investigate or run again with or use --pass-on-warn."
+ exit 3
+ fi
fi
+# Ensure Conda exists
conda --version &> /dev/null
if [[ $? -ne 0 ]]; then
- echo "Error: Conda is not available in this shell despite attempting an autoinstall. Exiting."
- return 3
+ echo "> > Error < <"
+ echo "Conda is not available in this shell (version >= 24.0.0)."
+ echo "Please run the following command to install miniconda. *warning*, if you are using homebrew conda, things might get wobbly."
+ echo ""
+ echo "To install miniconda, run:"
+ echo " ./bin/install_miniconda # and once installed, open a new shell and run "
+ exit 1
fi
+# Check conda version is sufficent
# Required version
-required_version="24.11.1"
-
-# Check if conda is installed
-if ! command -v conda &> /dev/null; then
- echo "Error: conda is not installed."
- return 3
-fi
+required_version="24.0.0"
# Get the current conda version
current_version=$(conda --version | awk '{print $2}')
# Compare versions
if [[ "$(printf '%s\n' "$required_version" "$current_version" | sort -V | head -n1)" != "$required_version" ]]; then
- echo "Error: Conda version $current_version is less than the required version $required_version."
- return 3
+ echo "Error: Conda version $current_version is less than the required version $required_version ."
+ exit 3
fi
echo "Conda version $current_version meets the requirement of $required_version or higher."
+
+# Check if the Conda environment DAYCLI is active
+if [[ "$CONDA_DEFAULT_ENV" == "DAYCLI" ]]; then
+ echo "The Conda environment DAYCLI is active, proceeding."
+else
+ echo "The Conda environment DAYCLI is not active or missing."
+ echo ""
+ echo "Please activate the DAYCLI environment by running:"
+ echo " conda activate DAYCLI"
+ echo ""
+ echo "If the environment does not exist, create it by running:"
+ echo " ./bin/init_daycli"
+ exit 3
+fi
+
+
# Activate or create the Daylily CLI conda environment
if conda env list | grep -q "^DAYCLI "; then
echo "Conda environment 'DAYCLI' already exists. Activating it."
@@ -183,7 +166,7 @@ else
source bin/init_daycli
if [[ $? -ne 0 ]]; then
echo "Error: Failed to create the 'DAYCLI' environment. Exiting."
- return 3
+ exit 3
fi
fi
conda activate DAYCLI
@@ -198,7 +181,7 @@ if [[ "$pcluster_version" != "$expected_pcluster_version" ]]; then
handle_warning "Warning: Expected pcluster version $expected_pcluster_version, but found version $pcluster_version."
if [[ $? -ne 0 ]]; then
echo "Error: pcluster version mis-match: expected:$expected_pcluster_version , detected:$pcluster_version ."
- return 3
+ exit 3
fi
fi
@@ -207,13 +190,13 @@ fi
echo "Verifying AWS credentials..."
if ! aws sts get-caller-identity --region "$region" --profile $AWS_PROFILE &> /dev/null; then
echo "Error: AWS credentials are invalid or do not have access to the AWS account in region $region."
- return 3
+ exit 3
else
echo "AWS credentials verified successfully."
fi
# Call the region validation function
-validate_region "$region" || return 3
+validate_region "$region" || exit 3
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text --profile $AWS_PROFILE )
@@ -242,10 +225,10 @@ else
echo "Proceeding anyway..."
elif [ "$CHOICE" -eq 2 ]; then
echo "Exiting."
- return 3 2>/dev/null || exit 3
+ exit 3 2>/dev/null || exit 3
else
echo "Invalid choice. Exiting."
- return 3 2>/dev/null || exit 3
+ exit 3 2>/dev/null || exit 3
fi
fi
@@ -263,7 +246,7 @@ check_quota() {
if [[ $? -ne 0 ]]; then
handle_warning "Error: Unable to retrieve quota for $resource_name."
- return 3
+ exit 3
fi
quota_value=$(echo "$quota" | jq -r '.Quota.Value')
@@ -272,7 +255,7 @@ check_quota() {
handle_warning "Warning: $resource_name quota is below the recommended $recommended_count. Current quota: $quota_value."
if [[ $? -ne 0 ]]; then
echo "quota check failed"
- return 3
+ exit 3
fi
else
echo "$resource_name quota is sufficient: $quota_value."
@@ -286,42 +269,42 @@ echo "Performing AWS quota checks..."
# Dedicated Instances
check_quota "Spot vCPU Max" "L-1216C47A" 20 ec2
if [[ $? -ne 0 ]]; then
- return 3
+ exit 3
fi
echo ""
# Spot vCPU Max Quota
check_quota "Spot vCPU Max" "L-34B43A08" 208 ec2
if [[ $? -ne 0 ]]; then
- return 3
+ exit 3
fi
echo ""
# VPC Quota (Default: 5 per region)
check_quota "VPCs" "L-F678F1CE" 4 vpc
if [[ $? -ne 0 ]]; then
- return 3
+ exit 3
fi
echo ""
# Elastic IP Quota (Default: 5 per region)
check_quota "Elastic IPs" "L-0263D0A3" 4 ec2
if [[ $? -ne 0 ]]; then
- return 3
+ exit 3
fi
echo ""
# NAT Gateway Quota (Default: 5 per AZ)
check_quota "NAT Gateways" "L-FE5A380F" 4 vpc
if [[ $? -ne 0 ]]; then
- return 3
+ exit 3
fi
echo ""
# Internet Gateway Quota (Default: 5 per region)
check_quota "Internet Gateways" "L-A4707A72" 4 vpc
if [[ $? -ne 0 ]]; then
- return 3
+ exit 3
fi
echo ""
@@ -389,7 +372,7 @@ echo ""
if [[ ${#valid_keys[@]} -eq 0 ]]; then
echo "No valid ED25519 keys with matching PEM files were found."
echo "Please ensure the required keys are available in the $HOME/.ssh directory AND that the keypair name includes 'omics' in the name."
- return 3
+ exit 3
fi
# Present valid keys to the user using 'select'
echo "Select a valid ED25519 key with a matching PEM file:"
@@ -459,7 +442,7 @@ done <<< "$buckets"
if [[ ${#matching_buckets[@]} -eq 0 ]]; then
echo "No matching buckets found in region $region."
- return 3
+ exit 3
fi
echo "Select a matching S3 bucket:"
@@ -481,13 +464,13 @@ check_s3_folder() {
echo "Folder s3://$bucket/$folder/ exists."
else
echo "ERROR: Folder s3://$bucket/$folder/ does not exist."
- return 1
+ exit 1
fi
}
echo "Validating required folders in bucket: $selected_bucket"
-check_s3_folder "$bucket" "data" || return 3
-check_s3_folder "$bucket" "cluster_boot_config" || return 3
+check_s3_folder "$bucket" "data" || exit 3
+check_s3_folder "$bucket" "cluster_boot_config" || exit 3
bucket_url="s3://$selected_bucket"
@@ -520,7 +503,7 @@ if [[ -z "$public_subnets" && -z "$private_subnets" ]]; then
bin/init_cloudstackformation.sh ./config/day_cluster/pcluster_env.yml "$res_prefix" "$region_az" "$region" $AWS_PROFILE
elif [[ -z "$public_subnets" || -z "$private_subnets" ]]; then
echo "Error: Incomplete setup. Ensure public and private subnets and the required ARN are available. You might try running : bin/init_cloudstackformation.sh ./config/day_cluster/pcluster_env.yml $res_prefix $region_az $region $AWS_PROFILE"
- return 1
+ exit 1
else
echo "All required resources are available."
fi
@@ -577,7 +560,7 @@ echo ""
# Function to validate cluster name
validate_cluster_name() {
if [[ ! "$1" =~ ^[a-zA-Z0-9\-]+$ ]] || [[ ${#1} -gt 25 ]]; then
- return 1
+ exit 1
else
return 0
fi
@@ -636,7 +619,7 @@ if [[ -z "$global_budget_exists" || "$global_budget_exists" == "None" ]]; then
if [[ $? -ne 0 ]]; then
echo "Error: Failed to create the budget. Exiting."
- return 3
+ exit 3
else
echo "Budget '$global_budget_name' created successfully."
fi
@@ -685,7 +668,7 @@ if [[ -z "$budget_exists" || "$budget_exists" == "None" ]]; then
if [[ $? -ne 0 ]]; then
echo "Error: Failed to create the budget. Exiting."
- return 3
+ exit 3
else
echo "Budget '$budget_name' created successfully."
fi
@@ -719,7 +702,7 @@ fi
if [[ ! -f "$cluster_yaml" ]]; then
echo "Error: YAML file '$cluster_yaml' does not exist."
- return 3
+ exit 3
fi
echo ""
@@ -852,7 +835,7 @@ if [[ $? -ne 0 ]]; then
echo "Error: Failed to calculate spot bid prices."
echo "This is likely due to a policy/permissions issue in AWS."
echo "Please refer to the quickstart guide for more information on required policies."
- return 3 # Use 'exit 3' if outside a function or script
+ exit 3 # Use 'exit 3' if outside a function or script
fi
echo ""
@@ -870,10 +853,10 @@ if [[ "$message" == "Request would have succeeded, but DryRun flag is set." ]];
echo "Dry run successful. Proceeding with cluster creation."
elif [[ "$DAY_BREAK" == "1" ]]; then
echo "DAY_BREAK == '1'. Exiting without creating the cluster."
- return 0
+ exit 0
else
echo "Error: Dry run failed ( pcluster create-cluster -n $cluster_name -c $target_conf_fin --dryrun true --region $region ). Exiting."
- return 3
+ exit 3
fi
echo ""
diff --git a/bin/daylily-get-ephemperal-cluster-deets b/bin/daylily-get-ephemperal-cluster-deets
old mode 100644
new mode 100755
diff --git a/bin/daylily-run-ephemeral-cluster-remote-tests b/bin/daylily-run-ephemeral-cluster-remote-tests
old mode 100644
new mode 100755
index d9c57a2e..27c4e005
--- a/bin/daylily-run-ephemeral-cluster-remote-tests
+++ b/bin/daylily-run-ephemeral-cluster-remote-tests
@@ -1,13 +1,7 @@
-#!/bin/zsh
+#!/bin/bash
# Daylily Headnode Configuration Script
-# Ensure the script is sourced, not executed directly
-if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
- echo "This script should be sourced, not executed directly. Please use 'source THISSCRIPT' to run."
- exit 1
-fi
-
# Capture arguments for PEM file and region
pem_file=$1
region=$2
@@ -91,12 +85,12 @@ read REPLY
if [[ "$REPLY" == "y" ]]; then
ssh -i "$pem_file" ubuntu@"$cluster_ip_address" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
- "sudo su - $duser -c 'mkdir -p /fsx/analysis_results/ubuntu/daylily_remote_test && cd /fsx/analysis_results/ubuntu/daylily_remote_test && git clone https://github.com/Daylily-Informatics/daylily.git daylily_remote_test'"
+ "sudo su - $duser -c 'mkdir -p /fsx/analysis_results/ubuntu/daylily_remote_test && cd /fsx/analysis_results/ubuntu/daylily_remote_test && git clone https://github.com/Daylily-Informatics/daylily.git daylily'"
session_name="cluster_test_$(date +%s)"
ssh -t -i "$pem_file" ubuntu@"$cluster_ip_address" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
"tmux new-session -d -s \"$session_name\" \
- \"bash -c 'source ~/.bashrc && cd /fsx/analysis_results/ubuntu/daylily_remote_test && source dyinit && source bin/day_activate slurm hg38 remote && ./bin/day_run produce_snv_concordances -p -k -j 2 --config aligners=[\\\"strobe\\\",\\\"bwa2a\\\"] dedupers=[\\\"dppl\\\"] genome_build=\\\"hg38\\\" snv_callers=[\\\"deep\\\"]; bash'\"" && \
+ \"bash -c 'source ~/.bashrc && cd /fsx/analysis_results/ubuntu/daylily_remote_test/daylily && source dyinit && source bin/day_activate slurm hg38 remote && ./bin/day_run produce_snv_concordances -p -k -j 2 --config aligners=[\\\"strobe\\\",\\\"bwa2a\\\"] dedupers=[\\\"dppl\\\"] genome_build=\\\"hg38\\\" snv_callers=[\\\"deep\\\"]; bash'\"" && \
echo "Tmux session >>> '$session_name' <<< started on the remote server."
echo "You can monitor the session using the following command:"
diff --git a/bin/init_daycli b/bin/init_daycli
index 7fd96c5e..faa5db7a 100755
--- a/bin/init_daycli
+++ b/bin/init_daycli
@@ -1,77 +1,39 @@
#!/bin/bash
-skip_warn=$1
-warn_msg=""
-
mkdir -p ~/.config/daylily
-# Check if the script is running in Zsh and emulate Zsh behavior
-if [ -n "$ZSH_VERSION" ]; then
- emulate -L zsh # Ensure Zsh behaves like Zsh (if required)
-
- # Check if the script is sourced or executed
- if [ -n "$ZSH_EVAL_CONTEXT" ] && [[ "$ZSH_EVAL_CONTEXT" =~ :file$ ]]; then
- echo "This script is sourced."
- else
- echo "Error: This script must be sourced, not executed directly. Use 'source $0' to run."
- exit 3
- fi
-fi
-
-# Ensure the script is sourced
-if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
- echo "Error: This script must be sourced, not executed directly. Use 'source $0' to run."
- exit 3
-fi
-# Check if system packages are installed
-bash bin/check_system_pkgs.sh
-if [ $? -ne 0 ]; then
-
- if [ "$skip_warn" != "skip_warn" ]; then
- warn_msg="There were problems detected with pre-requisites, and the warnings bypassed. Run 'bash bin/check_system_pkgs.sh' for more information."
- else
- echo "Some system packages are missing or are the wrong version which may cause problems. Please review them by running: 'bash bin/check_system_pkgs.sh' for more information. You may proceed past this warning by passing the first argument as 'skip_warn'."
- return 1
- fi
-fi
# Ensure Conda exists
if ! command -v conda &> /dev/null; then
- echo "Error: Conda is not available in this shell. Attempting to install Miniconda..."
- echo ""
+ echo "Error: Conda is not available in this shell. Please install Miniconda."
+ echo "Install with:"
- echo "source ./bin/install_miniconda"
- source ./bin/install_miniconda
+ echo " ./bin/install_miniconda "
+ exit 3
fi
-CURR_SHELL=$(ps -p $$ -o comm=)
-echo $CURR_SHELL
if ! command -v conda &> /dev/null; then
- echo "Error: Conda is not available in this shell. Autoinstall failed, please see the miniconda docs."
- echo "*** If your shell (login:$SHELL , current:$CURR_SHELL) is bash, please check if conda init correctly updated both your .bashrc and .bash_profile with the conda initialization block. If present in one but not the other, copy it, open a new bash shell and try again (most commonly a problem with MACs). ***"
- return 1
+ echo "Error: Conda is not available in this shell. "
+ echo "Install with:"
+ echo " ./bin/install_miniconda "
+ exit 1
fi
# Activate or create the Daylily CLI conda environment
if conda env list | grep -q "^DAYCLI "; then
- echo "Conda environment 'DAYCLI' already exists."
+ echo "Conda environment 'DAYCLI' already exists!"
+ echo "Activate with: "
+ echo " conda activate DAYCLI"
+ exit 3
else
echo "Creating 'DAYCLI' environment."
conda env create -y -n DAYCLI -f config/day/daycli.yaml
- #conda create -y -n DAYCLI -c conda-forge python parallel jq nodejs==18.15.0 aws-parallelcluster==3.11.1 flask=2.2.5 pyyaml
-fi
-echo "Activating 'DAYCLI' environment."
-conda activate DAYCLI
-# Function to check if a Python package is installed
-check_and_install() {
- PACKAGE=$1
- VERSION=$2
- if ! python -c "import $PACKAGE" &> /dev/null; then
- echo "$PACKAGE not found. Installing $PACKAGE==$VERSION..."
- pip install "$PACKAGE==$VERSION"
+ if [ $? -ne 0 ]; then
+ echo "Error: Failed to create 'DAYCLI' environment."
+ exit 1
fi
-}
-
+fi
-echo "Daylily CLI environment is ready ($warn_msg)."
\ No newline at end of file
+echo "Daylily CLI environment is ready! Activate with: "
+echo " conda activate DAYCLI"
\ No newline at end of file
diff --git a/bin/install_miniconda b/bin/install_miniconda
old mode 100644
new mode 100755
index 33b033c3..60a043ad
--- a/bin/install_miniconda
+++ b/bin/install_miniconda
@@ -1,23 +1,5 @@
-# Ensure the script is sourced and not executed directly
-if [[ -n "$ZSH_VERSION" ]]; then
- # Check if the script is being sourced in zsh
- # Check if the script is sourced or executed
- if [ -n "$ZSH_EVAL_CONTEXT" ] && [[ "$ZSH_EVAL_CONTEXT" =~ :file$ ]]; then
- echo "This script is sourced, proceed"
- else
- echo "Error: This script must be sourced, not executed directly. Use 'source $0' to run."
- exit 3
- fi
-elif [[ -n "$BASH_VERSION" ]]; then
- # Check if the script is being sourced in bash
- if [[ "$0" == "${BASH_SOURCE[0]}" ]]; then
- echo "This script must be sourced. Use: source $0 or . $0"
- return 1
- fi
-else
- echo "Unsupported shell. Please use bash or zsh to run this script."
- return 1
-fi
+
+
# Check if Miniconda is already installed
conda --version &> /dev/null
@@ -25,7 +7,7 @@ if [[ $? -ne 0 ]]; then
echo "Miniconda is not installed. Proceeding with installation..."
else
echo "Miniconda is already installed. Please uninstall it before proceeding."
- return 0
+ exit 0
fi
@@ -49,7 +31,7 @@ if [[ -z "$MACHINE" ]]; then
;;
*)
echo "Unsupported architecture: $uname_output"
- return 1
+ exit 1
;;
esac
fi
@@ -82,29 +64,68 @@ elif [[ "$MACHINE" == "linux_arm" ]]; then
else
echo "Unsupported machine type: $MACHINE"
echo "Please visit https://docs.conda.io/en/latest/miniconda.html for manual installation instructions."
- return 2
+ exit 2
fi
# Initialize Conda
if [[ -x ~/miniconda3/bin/conda ]]; then
echo "Initializing Conda..."
- ~/miniconda3/bin/conda init "$(basename "$SHELL")"
~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh
- echo "Reloading the shell..."
- if [[ $- == *i* ]]; then
- source ~/.bashrc 2>/dev/null || source ~/.zshrc 2>/dev/null
- else
- echo "Please restart your terminal or run: source ~/.bashrc or source ~/.zshrc"
+ # Define file paths
+ BASHRC="$HOME/.bashrc"
+ BASH_PROFILE="$HOME/.bash_profile"
+
+ touch $BASHRC $BASH_PROFILE
+
+ # Function to extract and copy the Conda block
+ copy_conda_block() {
+ local source_file=$1
+ local target_file=$2
+
+ # Extract the Conda block from the source file
+ CONDA_BLOCK=$(sed -n '/# >>> conda initialize >>>/,/# <<< conda initialize <<> $target_file
+ echo "$CONDA_BLOCK" >> "$target_file"
+ echo "Conda initialization block added to $target_file."
+ }
+
+ # Check for Conda block in both files
+ BASHRC_HAS_BLOCK=false
+ BASH_PROFILE_HAS_BLOCK=false
+
+ if grep -q "# >>> conda initialize >>>" "$BASHRC"; then
+ BASHRC_HAS_BLOCK=true
+ fi
+
+ if grep -q "# >>> conda initialize >>>" "$BASH_PROFILE"; then
+ BASH_PROFILE_HAS_BLOCK=true
fi
- if command -v conda &>/dev/null; then
- echo "Miniconda installed and initialized successfully!"
+ # Handle the cases
+ if $BASHRC_HAS_BLOCK && ! $BASH_PROFILE_HAS_BLOCK; then
+ echo "Conda initialization block found in .bashrc but missing in .bash_profile."
+ copy_conda_block "$BASHRC" "$BASH_PROFILE"
+ elif $BASH_PROFILE_HAS_BLOCK && ! $BASHRC_HAS_BLOCK; then
+ echo "Conda initialization block found in .bash_profile but missing in .bashrc."
+ copy_conda_block "$BASH_PROFILE" "$BASHRC"
+ elif $BASHRC_HAS_BLOCK && $BASH_PROFILE_HAS_BLOCK; then
+ echo "Conda initialization block already exists in both .bashrc and .bash_profile. No changes made."
else
- echo "Miniconda installation succeeded but initialization failed. Please check your shell configuration."
+ echo "No Conda initialization block found in either .bashrc or .bash_profile."
+ echo "From a bash shell, please run 'conda init bash' to initialize Conda."
+ exit 1
fi
+
+ echo "Miniconda installation successful."
+ echo "Open a new shell, and conda should be available (try: 'conda -v')."
else
- echo "Miniconda installation failed. Please try again."
- return 3
+ echo "Miniconda installation failed. Please try again and/or atttempt the commands in this script manually."
+ exit 3
fi
diff --git a/bin/other/regsub_yaml.sh b/bin/other/regsub_yaml.sh
index d3f174e9..ccc57b0f 100755
--- a/bin/other/regsub_yaml.sh
+++ b/bin/other/regsub_yaml.sh
@@ -33,4 +33,6 @@ while IFS='=' read -r key value; do
sed -i "" "s/$key/$escaped_value/g" "$cluster_cfg_yaml"
done < "$cluster_init_values"
-echo "\n\n\tSubstitutions completed in $cluster_cfg_yaml.\n"
+echo ""
+echo "Substitutions completed in $cluster_cfg_yaml."
+echo ""
diff --git a/bin/retshell b/bin/retshell
new file mode 100644
index 00000000..5a43d324
--- /dev/null
+++ b/bin/retshell
@@ -0,0 +1,2 @@
+CURR_SHELL=$(ps -p $$ -o comm=)
+echo "$CURR_SHELL"
\ No newline at end of file
diff --git a/config/day/daycli.yaml b/config/day/daycli.yaml
index f90c4d05..e1b87905 100755
--- a/config/day/daycli.yaml
+++ b/config/day/daycli.yaml
@@ -19,6 +19,8 @@ dependencies:
- tabix
- fd-find
- graphviz
+ - nodejs
+ - jq
- pip:
- docopt==0.6.2
- colr==0.9.1
diff --git a/docs/quickest_start.md b/docs/quickest_start.md
index 171c5818..c7ba410f 100644
--- a/docs/quickest_start.md
+++ b/docs/quickest_start.md
@@ -24,10 +24,10 @@ _on the laptop you'll create ephemeral cluster, where you have saved the daylily
_One is needed per region you intend to run in._
```bash
-AWS_PROFILE=daylily-service
+export AWS_PROFILE=daylily-service
BUCKET_PREFIX=yocorp
REGION=us-west-2
-bash ./bin/create_daylily_omics_analysis_s3.sh --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX --disable-dryrun
+./bin/create_daylily_omics_analysis_s3.sh --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX --disable-dryrun
```
- This will take ~20min if you remain in `us-west-2`, longer if cross region.
@@ -45,7 +45,7 @@ rclone copy $sentieon_lic yocorp_remote:yocorp-daylily-omics-analysis-us-west-2/
### Check Spot Market Pricing (optional, but suggested to get best pricing over time)
```bash
-AWS_PROFILE=daylily-service
+export AWS_PROFILE=daylily-service
REGION=us-west-2
OUT_TSV=./init_daylily_cluster.tsv
@@ -57,11 +57,11 @@ OUT_TSV=./init_daylily_cluster.tsv
### Create Ephemeral Cluster
_this script will check and install a number of prerequsites and attempt to install_
-```zsh
-AWS_PROFILE=daylily-service
+```bash
+export AWS_PROFILE=daylily-service
REGION_AZ=us-west-2c
-source bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE
+./bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE
```
You will be prompted for a variety of config settings. Some resources, if missing, will be created. Other resources if missing will trigger failures. If all is well, you'll be left with a success message with a suggested command to start a remote test on the headnode.
diff --git a/setup.py b/setup.py
index 9637008c..6bfd9a6d 100755
--- a/setup.py
+++ b/setup.py
@@ -2,7 +2,7 @@
setup(
name="daylily",
- version="0.7.155",
+ version="0.7.156",
packages=find_packages(),
install_requires=[
# Add dependencies here