diff --git a/README.md b/README.md index c5c9dd3d..aaa1364d 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Daylily AWS Ephemeral Cluster Setup (0.7.155) +# Daylily AWS Ephemeral Cluster Setup (0.7.156) _(0.7.155)_ **beta release** @@ -87,7 +87,6 @@ I will be assessing: > Would you like to lend a hand? [contact me](mailto:john@daylilyinformatics.com) - # What's It All About? ## BFAIR: Bioinformatics [FAIR](https://www.go-fair.org/fair-principles/) Principles @@ -251,22 +250,25 @@ You may run in any region or AZ you wish to try. This said, the majority of test ## Prerequisites (On Your Local Machine) -Local machine development has been exclusively on a mac, using the `zsh` shell (I am moving everything to bash, so there is a mix of bash and zsh at the moment. All `.` and `sourcing` that happens on your local machine expects `zsh`). +Local machine development has been carried out exclusively on a mac using the `zsh` shell. `bash` should work as well (if you have issues with conda w/mac+bash, confirm that after miniconda install and conda init, the correct `.bashrc` and `.bash_profile` files were updated by `conda init`). _suggestion: run things in tmux or screen_ Very good odds this will work on any mac and most Linux distros (ubuntu 22.04 are what the cluster nodes run). Windows, I can't say. ### System Packages -Install with `brew` or `apt-get` ('): -- `python3`, tested with `3.12.8` +Install with `brew` or `apt-get`: +- `python3`, tested with `3.11.0 - `git`, tested with `2.46.0` -- `jq`, tested with `jq-1.7.1` - `wget`, tested with `1.25.0` - `awscli`, tested with `2.22.4` [AWS CLI docs](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) - `tmux` (optional, but suggested) - `emacs` (optional, I guess, but I'm not sure how to live without it) +#### Check if your prereq's meet the min versions required +```bash +./bin/check_prereq_sw.sh +``` ### AWS CLI Configuration #### Opt 2 @@ -323,29 +325,32 @@ git clone https://github.com/Daylily-Informatics/daylily.git # or, if you have cd daylily ``` -### Miniconda +### Install Miniconda (homebrew is not advised) _tested with conda version **`24.11.1`**_ -```bash Install with: ```bash -source bin/install_miniconda +./bin/install_miniconda ``` -- This will leave you in a terminal with conda activated, indicated by `(base)` in the terminal prompt. +- open a new terminal/shell, and conda should be available: `conda -v`. -### DAYCLI Environment +### Install DAYCLI Environment _from `daylily` root dir_ ```bash -#!/bin/zsh -source bin/init_daycli +./bin/init_daycli + +conda activate DAYCLI # DAYCLI should now be active... did it work? colr 'did it work?' 0,100,255 255,100,0 ``` -- This will leave you in a terminal with the conda DAYCLI activated, indicated by `(DAYCLI)` in the terminal prompt. + +- You should see: + + > ![](docs/images/diw.png)

@@ -377,17 +382,17 @@ _running in a tmux/screen session is advised as the copy may take 1-many hours_ ```bash conda activate DAYCLI # help -bash ./bin/create_daylily_omics_analysis_s3.sh -h +./bin/create_daylily_omics_analysis_s3.sh -h -AWS_PROFILE= +export AWS_PROFILE= BUCKET_PREFIX= REGION=us-west-2 # dryrun -bash ./bin/create_daylily_omics_analysis_s3.sh --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX +./bin/create_daylily_omics_analysis_s3.sh --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX # run for real -bash ./bin/create_daylily_omics_analysis_s3.sh --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX --disable-dryrun +./bin/create_daylily_omics_analysis_s3.sh --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX --disable-dryrun ``` @@ -406,7 +411,7 @@ _this command will take ~5min to complete, and much longer if you expand to all ```bash conda activate DAYCLI -AWS_PROFILE=daylily-service +export AWS_PROFILE=daylily-service REGION=us-west-2 OUT_TSV=./init_daylily_cluster.tsv @@ -474,14 +479,13 @@ Once you have selected an AZ && have a reference bucket ready in the region this The following script will check a variety of required resources, attempt to create some if missing and then prompt you to select various options which will all be used to create a new parallel cluster yaml config, which in turn is used to create the cluster via `StackFormation`. [The template yaml file can be checked out here](config/day_cluster/prod_cluster.yaml). -```zsh -#!/bin/zsh -AWS_PROFILE=daylily-service +```bash +export AWS_PROFILE=daylily-service REGION_AZ=us-west-2c -source bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE +./bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE # And to bypass the non-critical warnings (which is fine, not all can be resolved ) -source bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE --pass-on-warn # If you created an inline policy with a name other than daylily-service-cluster-policy, you will need to acknowledge the warning to proceed (assuming the policy permissions were granted other ways) +./bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE --pass-on-warn # If you created an inline policy with a name other than daylily-service-cluster-policy, you will need to acknowledge the warning to proceed (assuming the policy permissions were granted other ways) ``` @@ -558,8 +562,7 @@ Once logged in, as the 'ubuntu' user, run the following commands: ### Run Remote Slurm Tests On Headnode ```bash -#!/bin/zsh -source ./bin/daylily-run-ephemeral-cluster-remote-tests $pem_file $region $AWS_PROFILE +./bin/daylily-run-ephemeral-cluster-remote-tests $pem_file $region $AWS_PROFILE ``` A successful test will look like this: @@ -579,7 +582,6 @@ pcluster list-clusters --region $REGION [See the instructions here](#first-time-logging-into-head-node) to confirm the headnode is configured and ready to run the daylily pipeline. -

# Costs @@ -777,7 +779,7 @@ ssh -i $pem_file ubuntu@$cluster_ip_address ##### Facilitated ```bash -AWS_PROFILE= +export AWS_PROFILE= bin/daylily-ssh-into-headnode ``` @@ -813,7 +815,7 @@ From your remote terminal that you created the cluster with, run the following c ```bash conda activate DAYCLI -source ./bin/daylily-cfg-headnode $PATH_TO_PEM $CLUSTER_AWS_REGION $AWS_PROFILE +./bin/daylily-cfg-headnode $PATH_TO_PEM $CLUSTER_AWS_REGION $AWS_PROFILE ``` > If the problem persists, ssh into the headnode, and attempt to run the commands as the ubuntu user which are being attempted by the `daylily-cfg-headnode` script. diff --git a/bin/calc_daylily_aws_cost_estimates.py b/bin/calc_daylily_aws_cost_estimates.py index dc421eaf..95b8fb14 100755 --- a/bin/calc_daylily_aws_cost_estimates.py +++ b/bin/calc_daylily_aws_cost_estimates.py @@ -1,3 +1,5 @@ +#!/bin/env python + from daylib.day_cost_ec2 import AWSGenomicsAnalyzer def main(): diff --git a/bin/calcuate_spotprice_for_cluster_yaml.py b/bin/calcuate_spotprice_for_cluster_yaml.py index e2b90858..2ee53b66 100755 --- a/bin/calcuate_spotprice_for_cluster_yaml.py +++ b/bin/calcuate_spotprice_for_cluster_yaml.py @@ -1,4 +1,5 @@ #!/usr/bin/env python + import subprocess import statistics import argparse diff --git a/bin/cat_input_files.sh b/bin/cat_input_files.sh index 322f1933..72c287ef 100755 --- a/bin/cat_input_files.sh +++ b/bin/cat_input_files.sh @@ -1,10 +1,5 @@ #!/bin/bash -# Ensure compatibility with zsh and bash -if [ -n "$ZSH_VERSION" ]; then - emulate -LR bash # Ensure bash-like behavior in zsh -fi - # Display usage information usage() { echo "Usage: $0 -s SAMPLENAME -p SAMPLE_MATCH_PATTERN -i INPUT_DIR -o OUTPUT_DIR" @@ -42,7 +37,7 @@ if [[ ! -d "$OUTPUT_DIR" ]]; then fi echo "" -echo ">>> >> > THIS IS A SIMPLE UTILITY AND NOT PRODUCTION TESTED FOR YOUR USE CASE < << <<<" +echo ">>> >> > THIS IS A SIMPLE UTILITY AND NOT PRODUCTION TESTED < << <<<" echo "" sleep 2 diff --git a/bin/check_aws_permissions.sh b/bin/check_aws_permissions.sh old mode 100644 new mode 100755 diff --git a/bin/check_current_spot_market_by_zones.py b/bin/check_current_spot_market_by_zones.py index cb717cdc..e773ef28 100755 --- a/bin/check_current_spot_market_by_zones.py +++ b/bin/check_current_spot_market_by_zones.py @@ -466,6 +466,17 @@ def main(): if args.profile == "SETME": print(f"\n\nERROR:: Your AWS_PROFILE is set to >{os.getenv('AWS_PROFILE')}<, Please set your AWS_PROFILE environment variable and specify the same string with --profile.\n") raise SystemExit + if os.environ.get('AWS_PROFILE','') == '': + os.environ['AWS_PROFILE'] = args.profile + elif args.profile != os.environ['AWS_PROFILE']: + print(f"\n\nERROR:: Your AWS_PROFILE is set to >{os.getenv('AWS_PROFILE')}<, Please set your AWS_PROFILE environment variable and specify the same string with --profile.\n") + raise SystemExit + elif args.profile == os.environ['AWS_PROFILE']: + print(f"\n\nAWS_PROFILE is set to >{os.getenv('AWS_PROFILE')}< and profile passed is >{args.profile}<, all good.\n") + else: + print(f"\n\nAWS_PROFILE is set to >{os.getenv('AWS_PROFILE')}< and profile passed is >{args.profile}<, there is a problem.\n") + raise SystemExit + zones = args.zones.split(',') spot_data = collect_spot_prices(instance_types, zones, args.profile) diff --git a/bin/check_system_pkgs.sh b/bin/check_prereq_sw.sh similarity index 83% rename from bin/check_system_pkgs.sh rename to bin/check_prereq_sw.sh index 4e9ce48f..17791e79 100755 --- a/bin/check_system_pkgs.sh +++ b/bin/check_prereq_sw.sh @@ -2,9 +2,8 @@ # Set required versions required_versions=( - "python3:3.12.8" + "python3:3.11.0" "git:2.46.0" - "jq:1.7.1" "wget:1.25.0" "aws:2.22.4" ) @@ -14,9 +13,8 @@ pass_on_warn=${pass_on_warn:-0} # Function to compare versions check_version() { - local tool=$1 - local required=$2 - local current=$3 + local required=$1 + local current=$2 if [[ "$(printf '%s\n' "$required" "$current" | sort -V | head -n1)" != "$required" ]]; then return 1 # Current version is less than required @@ -38,12 +36,13 @@ check_tool() { fi # Check version - if ! check_version "$tool" "$required_version" "$current_version"; then + if ! check_version "$required_version" "$current_version"; then echo "Error: $tool version $current_version does not meet the required version $required_version." [[ "$pass_on_warn" -eq 1 ]] && return 0 || exit 1 fi - echo "$tool version $current_version meets the requirement of $required_version or higher." + # Print success message if version is sufficient + echo "$tool version $current_version meets the required version $required_version or higher." } # Loop through required tools and check @@ -58,9 +57,6 @@ for entry in "${required_versions[@]}"; do git) check_tool "git" "$required_version" "git --version | awk '{print \$3}'" ;; - jq) - check_tool "jq" "$required_version" "jq --version | sed 's/jq-//g'" - ;; wget) check_tool "wget" "$required_version" "wget --version | head -n1 | awk '{print \$3}'" ;; diff --git a/bin/create_budget.sh b/bin/create_budget.sh index c1c231b7..a69827de 100755 --- a/bin/create_budget.sh +++ b/bin/create_budget.sh @@ -78,9 +78,9 @@ write_or_append_tags_to_s3() { echo "Writing or appending tags to S3 file: $S3_BUCKET_PATH" url: $S3_BUCKET_URL # Check if the file exists in S3 - if aws s3 ls "$S3_BUCKET_PATH" > /dev/null 2>&1; then + if aws s3 ls "$S3_BUCKET_PATH" --profile $AWS_PROFILE > /dev/null 2>&1; then echo "File exists. Downloading the existing file." - aws s3 cp "$S3_BUCKET_PATH" "$TEMP_LOCAL_FILE" --region "$REGION" + aws s3 cp "$S3_BUCKET_PATH" "$TEMP_LOCAL_FILE" --region "$REGION" --profile $AWS_PROFILE if [[ $? -ne 0 ]]; then echo "ERROR: Failed to download existing file from S3." @@ -97,7 +97,7 @@ write_or_append_tags_to_s3() { # Upload the updated file back to S3 echo "Uploading updated file to S3..." - aws s3 cp "$TEMP_LOCAL_FILE" "$S3_BUCKET_PATH" --region "$REGION" + aws s3 cp "$TEMP_LOCAL_FILE" "$S3_BUCKET_PATH" --region "$REGION" --profile $AWS_PROFILE if [[ $? -eq 0 ]]; then echo "Successfully updated S3 file: $S3_BUCKET_PATH" @@ -153,12 +153,12 @@ BUDGET_JSON=$(echo "$BUDGET_TEMPLATE" | \ TEMP_BUDGET_JSON=$(mktemp) echo "$BUDGET_JSON" > "$TEMP_BUDGET_JSON" -ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) +ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text --profile $AWS_PROFILE) # Create the budget using AWS CLI echo "Creating budget for project: $PROJECT_NAME with amount: $AMOUNT USD in region: $REGION" aws budgets create-budget --account-id "$ACCOUNT_ID" \ - --budget file://"$TEMP_BUDGET_JSON" --region "$REGION" + --budget file://"$TEMP_BUDGET_JSON" --region "$REGION" --profile $AWS_PROFILE # Check if the budget creation succeeded if [[ $? -eq 0 ]]; then @@ -169,7 +169,7 @@ if [[ $? -eq 0 ]]; then IFS=',' read -ra THRESHOLD_ARRAY <<< "$THRESHOLDS" for threshold in "${THRESHOLD_ARRAY[@]}"; do echo P:$PROJECT_NAME T:$threshold E:$EMAIL R:$REGION A:$AMOUNT AI:$ACCOUNT_ID - aws budgets create-notification --account-id "$ACCOUNT_ID" \ + aws budgets create-notification --account-id "$ACCOUNT_ID" --profile $AWS_PROFILE \ --budget-name "$PROJECT_NAME" \ --notification '{"ComparisonOperator":"GREATER_THAN","NotificationType":"ACTUAL","Threshold":'"$threshold"',"ThresholdType":"PERCENTAGE"}' \ --region "$REGION" --subscribers '[{"Address":"'"$EMAIL"'","SubscriptionType":"EMAIL"}]' diff --git a/bin/daylily-analysis-samples-to-manifest b/bin/daylily-analysis-samples-to-manifest deleted file mode 100755 index 63d4a088..00000000 --- a/bin/daylily-analysis-samples-to-manifest +++ /dev/null @@ -1,202 +0,0 @@ -#!/usr/bin/env bash - -# Neon color scheme -INFO="\033[1;36m" -WARN="\033[1;33m" -ERROR="\033[1;31m" -SUCCESS="\033[1;32m" -RESET="\033[0m" - -info() { echo -e "${INFO}[INFO]${RESET} $1"; } -warn() { echo -e "${WARN}[WARN]${RESET} $1"; } -error() { echo -e "${ERROR}[ERROR]${RESET} $1"; exit 1; } -success() { echo -e "${SUCCESS}[SUCCESS] $1"; } - -AM_OUT_FILE="na" - -check_dependencies() { - for cmd in ssh; do - if ! command -v $cmd &>/dev/null; then - error "Required command '$cmd' is not installed." - fi - done -} - -validate_args() { - if [[ "$1" != "local" && "$1" != "remote" ]]; then - error "First argument must be 'local' or 'remote'." - fi - if [[ ! -f "$2" ]]; then - error "Sample sheet '$2' does not exist." - fi -} - -determine_sex() { - local n_x=$1 - local n_y=$2 - if [[ "$n_x" -eq 2 && "$n_y" -eq 0 ]]; then - echo "female" - elif [[ "$n_x" -eq 1 && "$n_y" -eq 1 ]]; then - echo "male" - else - echo "na" - fi -} - -validate_stage_target() { - local stage_target="$1" - local mode="$2" - local cluster_ip="$3" - local pem_file="$4" - local cluster_user="$5" - - info "Validating STAGE_TARGET: $mode $stage_target " - if [[ ! "$stage_target" =~ ^/fsx ]]; then - error "STAGE_TARGET '$stage_target' must begin with '/fsx'." - fi - - if [[ "$mode" == "remote" ]]; then - info "Validating STAGE_TARGET on remote cluster: $stage_target" - ssh -i "$pem_file" "$cluster_user@$cluster_ip" "if [[ ! -d \"$stage_target\" ]]; then mkdir -p \"$stage_target\"; fi" \ - || error "Failed to create STAGE_TARGET directory on remote: $stage_target" - else - if [[ ! -d "$stage_target" ]]; then - info "Creating STAGE_TARGET directory locally: $stage_target" - mkdir -p "$stage_target" || error "Failed to create STAGE_TARGET directory: $stage_target" - fi - fi -} - -stage_data_files() { - local r1_src="$1" - local r2_src="$2" - local stage_target="$3" - local sample_prefix="$4" - local mode="$5" - local cluster_ip="$6" - local pem_file="$7" - local cluster_user="$8" - - local r1_dest="${stage_target}/${sample_prefix}.R1.fastq.gz" - local r2_dest="${stage_target}/${sample_prefix}.R2.fastq.gz" - - if [[ "$mode" == "remote" ]]; then - info "Staging R1 to remote: $r1_dest" - - ssh -i "$pem_file" "$cluster_user@$cluster_ip" "if [[ -f '$r1_dest' ]]; then exit 1; fi" && \ - echo "File does not exist on remote" || \ - { echo "Error: File '$r1_dest' already exists on remote."; exit 1; } - - scp -i "$pem_file" "$r1_src" "${cluster_user}@${cluster_ip}:${r1_dest}" || error "Failed to stage R1 to remote." - success "R1 staged to remote: $r1_dest" - - ssh -i "$pem_file" "$cluster_user@$cluster_ip" "if [[ -f '$r2_dest' ]]; then exit 1; fi" && \ - echo "File does not exist on remote" || \ - { echo "Error: File '$r2_dest' already exists on remote."; exit 1; } - info "Staging R2 to remote: $r2_dest" - scp -i "$pem_file" "$r2_src" "${cluster_user}@${cluster_ip}:${r2_dest}" || error "Failed to stage R2 to remote." - success "R2 staged to remote: $r2_dest" - else - [[ ! -e "$r1_src" ]] && error "R1 file '$r1_src' does not exist." - [[ ! -e "$r2_src" ]] && error "R2 file '$r2_src' does not exist." - - cp "$r1_src" "$r1_dest" || error "Failed to copy R1: $r1_src to $r1_dest" - success "R1 staged locally: $r1_dest" - - cp "$r2_src" "$r2_dest" || error "Failed to copy R2: $r2_src to $r2_dest" - success "R2 staged locally: $r2_dest" - fi -} - -parse_and_validate_tsv() { - local input_file="$1" - local mode="$2" - local cluster_ip="$3" - local pem_file="$4" - local cluster_user="$5" - - - while IFS=$'\t' read -r RUN_ID SAMPLE_ID SAMPLE_TYPE LIB_PREP SEQ_PLATFORM LANE SEQBC_ID PATH_TO_CONCORDANCE_DATA_DIR R1_FQ R2_FQ STAGE_DIRECTIVE STAGE_TARGET SUBSAMPLE_PCT IS_POS_CTRL IS_NEG_CTRL N_X N_Y; do - [[ "$LANE" != "0" ]] && error "Invalid LANE value '$LANE'. Only '0' is allowed." - [[ "$IS_POS_CTRL" != "na" || "$IS_NEG_CTRL" != "na" ]] && error "Positive or Negative Control columns are not enabled yet." - - echo "Processing: $RUN_ID $SAMPLE_ID $SAMPLE_TYPE $LIB_PREP $SEQ_PLATFORM $LANE $SEQBC_ID $PATH_TO_CONCORDANCE_DATA_DIR $R1_FQ $R2_FQ $STAGE_DIRECTIVE $STAGE_TARGET $SUBSAMPLE_PCT $IS_POS_CTRL $IS_NEG_CTRL $N_X $N_Y" - - AS_IN_FILE="${STAGE_TARGET}/${RUN_ID}.analysis_samples.tsv" - tmp_am_out_file="./${RUN_ID}.analysis_manifest.csv.tmp" - touch "$tmp_am_out_file" - AM_OUT_FILE="${STAGE_TARGET}/${RUN_ID}.analysis_manifest.csv" - done < <(tail -n +2 "$input_file") - - echo "ASIO $AS_IN_FILE ... $AM_OUT_FILE ... $tmp_am_out_file" - - echo "samp,sample,sample_lane,SQ,RU,EX,LANE,r1_path,r2_path,biological_sex,iddna_uid,concordance_control_path,is_positive_control,is_negative_control,sample_type,merge_single,external_sample_id,instrument,lib_prep,bwa_kmer,subsample_pct" >"$tmp_am_out_file" - - while IFS=$'\t' read -r RUN_ID SAMPLE_ID SAMPLE_TYPE LIB_PREP SEQ_PLATFORM LANE SEQBC_ID PATH_TO_CONCORDANCE_DATA_DIR R1_FQ R2_FQ STAGE_DIRECTIVE STAGE_TARGET SUBSAMPLE_PCT IS_POS_CTRL IS_NEG_CTRL N_X N_Y; do - [[ "$LANE" != "0" ]] && error "Invalid LANE value '$LANE'. Only '0' is allowed." - [[ "$IS_POS_CTRL" != "na" || "$IS_NEG_CTRL" != "na" ]] && error "Positive or Negative Control columns are not enabled yet." - - echo "Processing: $RUN_ID $SAMPLE_ID $SAMPLE_TYPE $LIB_PREP $SEQ_PLATFORM $LANE $SEQBC_ID $PATH_TO_CONCORDANCE_DATA_DIR $R1_FQ $R2_FQ $STAGE_DIRECTIVE $STAGE_TARGET $SUBSAMPLE_PCT $IS_POS_CTRL $IS_NEG_CTRL $N_X $N_Y" - local biological_sex=$(determine_sex "$N_X" "$N_Y") - local sample_prefix="${RUN_ID}_${SAMPLE_ID}_${SAMPLE_TYPE}_${LANE}-${SEQBC_ID}" - - echo "----SD $STAGE_DIRECTIVE" - validate_stage_target "$STAGE_TARGET" "$mode" "$cluster_ip" "$pem_file" "$cluster_user" - - if [[ "$STAGE_DIRECTIVE" == "stage_data" ]]; then - stage_data_files "$R1_FQ" "$R2_FQ" "$STAGE_TARGET" "$sample_prefix" "$mode" "$cluster_ip" "$pem_file" "$cluster_user" - fi - - local staged_r1="${STAGE_TARGET}/${sample_prefix}.R1.fastq.gz" - local staged_r2="${STAGE_TARGET}/${sample_prefix}.R2.fastq.gz" - echo "${sample_prefix},${sample_prefix},${sample_prefix},${SAMPLE_TYPE},${RUN_ID},${SAMPLE_ID},${LANE}-${SEQBC_ID},${staged_r1},${staged_r2},${biological_sex},na,${PATH_TO_CONCORDANCE_DATA_DIR},na,na,${SAMPLE_TYPE},merge,${SAMPLE_ID},${SEQ_PLATFORM},${LIB_PREP},19,${SUBSAMPLE_PCT}" >>"$tmp_am_out_file" - done < <(tail -n +2 "$input_file") - - - if [[ "$mode" == "remote" ]]; then - - ssh -i "$pem_file" "$cluster_user@$cluster_ip" "if [[ -f '$AS_IN_FILE' ]]; then exit 1; fi" && \ - echo "File does not exist on remote" || \ - { echo "Error: File '$AS_IN_FILE' already exists on remote."; exit 1; } - - scp -i "$pem_file" "$input_file" "${cluster_user}@${cluster_ip}:${AS_IN_FILE}" || error "Failed to stage R1 to remote." - success "Analysis samples $input_file moved to remote: scp -i $pem_file $input_file ${cluster_user}@${cluster_ip}:$AS_IN_FILE" - - - ssh -i "$pem_file" "$cluster_user@$cluster_ip" "if [[ -f '$AM_OUT_FILE' ]]; then exit 1; fi" && \ - echo "File does not exist on remote" || \ - { echo "Error: File '$AM_OUT_FILE' already exists on remote."; exit 1; } - - scp -i "$pem_file" "$tmp_am_out_file" "${cluster_user}@${cluster_ip}:${AM_OUT_FILE}" || error "Failed to stage R1 to remote." - success "$tmp_am_out_file moved to remote: scp -i $pem_file $input_file ${cluster_user}@${cluster_ip}:$AM_OUT_FILE" - - - - else - echo "pass on copy $AS_IN_FILE" - fi -} - -main() { - local mode="$1" - local input_file="$2" - local cluster_ip="$3" - local pem_file="$4" - local cluster_user="$5" - - check_dependencies - validate_args "$@" - - #if [[ "$mode" == "remote" ]]; then - # ssh -i "$pem_file" "$cluster_user@$cluster_ip" \ - # "if [[ ! -d '/fsx/staged_sample_data' ]]; then mkdir '/fsx/staged_sample_data'; fi" \ - # || error "Failed to create '/fsx/staged_sample_data' directory on remote cluster." - #else - # mkdir "/fsx/staged_sample_data" || error "Failed to create local '/fsx/staged_sample_data'." - #fi - - parse_and_validate_tsv "$input_file" "$mode" "$cluster_ip" "$pem_file" "$cluster_user" - success "Manifest generated: $AM_OUT_FILE" -} - -main "$@" diff --git a/bin/daylily-cfg-headnode b/bin/daylily-cfg-headnode old mode 100644 new mode 100755 index 278eddb5..841cdfc6 --- a/bin/daylily-cfg-headnode +++ b/bin/daylily-cfg-headnode @@ -1,13 +1,7 @@ -#!/bin/zsh +#!/bin/bash # Daylily Headnode Configuration Script -# Ensure the script is sourced, not executed directly -if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then - echo "This script should be sourced, not executed directly. Please use 'source THISSCRIPT' to run." - exit 1 -fi - # Capture arguments for PEM file and region pem_file=$1 region=$2 @@ -142,7 +136,7 @@ echo " " echo "___bonus round___" echo "Would you like to start building various caches needed to run jobs? [y/n]" -timeout 120 bash -c 'read -p "Press y to start or n to skip: " REPLY && echo "$REPLY"' && export REPLY=$REPLY +timeout 30 bash -c 'read -p "Press y to start or n to skip: " REPLY && echo "$REPLY"' && export REPLY=$REPLY if [[ "$REPLY" == "y" ]]; then # start building the minimal cached envs @@ -161,6 +155,6 @@ echo "And, you may now access the headnode via the PCUI, via 'source bin/daylily echo -e " ssh -i $pem_file ubuntu@$cluster_ip_address\n" echo " " echo "If you wish to run some remote tests, you can do so with the following command:" -echo -e " . bin/daylily-run-ephemeral-cluster-remote-tests $pem_file $region $aws_profile\n" +echo -e " ./bin/daylily-run-ephemeral-cluster-remote-tests $pem_file $region $aws_profile\n" echo " " echo "...fin" diff --git a/bin/daylily-create-ephemeral-cluster b/bin/daylily-create-ephemeral-cluster old mode 100644 new mode 100755 index c57e4983..2e39d1b2 --- a/bin/daylily-create-ephemeral-cluster +++ b/bin/daylily-create-ephemeral-cluster @@ -1,38 +1,15 @@ -#!/bin/zsh +#!/bin/bash -# Function to check if the script is sourced -is_sourced() { - # If the script is sourced, BASH_SOURCE[0] != $0 - [[ "${BASH_SOURCE[0]}" != "${0}" ]] -} - -CURR_SHELL=$(ps -p $$ -o comm=) -echo $CURR_SHELL - -if [[ "$CURR_SHELL" != "zsh" ]]; then - echo "ERROR: This script must be executed from a zsh shell. please run 'zsh' and re-run this." - ~/miniconda3/bin/conda init zsh || echo "conda init failed, possibly legit if conda is not installed yet." - - if is_sourced; then - # If sourced, use 'return' to avoid exiting the shell - return 3 - else - # If executed, use 'exit' to terminate the script - exit 3 - fi -fi - -echo "Now running in zsh. Continuing script execution..." -# Your script logic here -emulate -L bash +CURR_SHELL=$(. bin/retshell) +echo "$CURR_SHELL" # Default configuration -pass_on_warn='' # Default for warnings causing failures +pass_on_warn=0 # Default for warnings causing failures # Function to display help usage() { - echo "Usage: source $0 [--region-az REGION-AZ] [--pass-on-warn]" + echo "Usage: $0 --profile AWS_PROFILE [--region-az REGION-AZ] [--pass-on-warn]" echo " --region-az REGION-AZ Specify the AWS region and availability zone (tested w us-west-2c/d)" echo " --pass-on-warn Allow warnings to pass without failure (default: false)" echo " --profile aws profile to use" @@ -42,12 +19,12 @@ usage() { # Check if script is sourced with no arguments if [[ $# -eq 0 ]]; then usage - return 0 + exit 0 fi if [ -z "$AWS_PROFILE" ]; then echo "Error: AWS_PROFILE is not set." - return 1 # Exit the function with an error status + exit 1 # Exit the function with an error status fi # Parse command-line arguments @@ -55,16 +32,16 @@ while [[ $# -gt 0 ]]; do case "$1" in --region-az) region_az="$2" - region="${region_az::-1}" # Remove the last character from the AZ + region="${region_az:0:${#region_az}-1}" shift 2 ;; --pass-on-warn) - pass_on_warn=true + pass_on_warn="1" shift ;; -h|--help) usage - return 0 # Use exit instead of return + exit 0 ;; --profile) AWS_PROFILEIN="$2" @@ -73,31 +50,24 @@ while [[ $# -gt 0 ]]; do *) echo "Error: Unknown option: $1" usage - return 3 # Use exit instead of return + exit 3 ;; esac done - -# Ensure the script is sourced -if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then - echo "Error: This script must be sourced, not executed directly. Use 'source $0' to run." - exit 3 -fi - if [[ -z "$AWS_PROFILE" ]]; then echo "Error: --profile flag not set" - return 3 # Exit the function with an error status + exit 3 # Exit the function with an error status fi if [[ "$AWS_PROFILE" != "$AWS_PROFILEIN" ]]; then echo "Error: AWS_PROFILE and --profile must match: $AWS_PROFILE and $AWS_PROFILEIN" - return 3 # Exit the function with an error status + exit 3 # Exit the function with an error status fi if [[ -z "$region_az" ]]; then echo "Error: --region-az flag not set" - return 3 # Exit the function with an error status + exit 3 # Exit the function with an error status fi # Extract the last character @@ -108,12 +78,12 @@ if [[ "$az_char" =~ [a-zA-Z] ]]; then echo "The last character of --region-az ${region_az} '$az_char' is a letter." else echo "ERROR: The last character of --region-az ${region_az} '$az_char' is a NOT letter." - return 3 # Exit the function with an error status + exit 3 # Exit the function with an error status fi if [[ "$AWS_PROFILEIN" != "$AWS_PROFILE" ]]; then echo "Error: AWS_PROFILE and --profile must match: $AWS_PROFILE and $AWS_PROFILEIN" - return 3 # Exit the function with an error status + exit 3 # Exit the function with an error status fi AWS_PROFILE=$AWS_PROFILE @@ -123,58 +93,71 @@ echo "YOUR AWS_PROFILE IS NOW SET TO: $AWS_PROFILE" handle_warning() { local message="$1" echo -e "WARNING: $message" - echo "If appropriate, consider setting --pass-on-warn to continue without failure. -$pass_on_warn-" - if [[ "$pass_on_warn" == "" ]]; then - echo "exiting" - return 3 + echo "If appropriate, consider setting --pass-on-warn to continue without failure. Current setting: $pass_on_warn" + if [[ "$pass_on_warn" != "1" ]]; then + echo "Exiting due to warning." + exit 3 fi } validate_region() { echo "REGION:" $1 } +echo "DEBUG: pass_on_warn after parsing: $pass_on_warn" -echo "Welcome to the Daylily CLI Setup" -echo "Daylily is tested to run in region $region (AZ: $region_az)." -echo "Please ensure your ~/.aws/config file matches this region." - -### Conda Environment Setup - - -# Ensure Conda exists -conda --version &> /dev/null -if [[ $? -ne 0 ]]; then - echo "Error: Conda is not available in this shell. Attempting to determine if autoinstall is possible." - - source bin/install_miniconda +# Check if system packages are installed +./bin/check_prereq_sw.sh +if [ $? -ne 0 ]; then + handle_warning "There were problems detected with pre-requisites, and the warnings bypassed. Run './bin/check_prereq_sw.sh' for more information ." + if [[ $? -ne 0 ]]; then + echo "Error: System package check failed, investigate or run again with or use --pass-on-warn." + exit 3 + fi fi +# Ensure Conda exists conda --version &> /dev/null if [[ $? -ne 0 ]]; then - echo "Error: Conda is not available in this shell despite attempting an autoinstall. Exiting." - return 3 + echo "> > Error < <" + echo "Conda is not available in this shell (version >= 24.0.0)." + echo "Please run the following command to install miniconda. *warning*, if you are using homebrew conda, things might get wobbly." + echo "" + echo "To install miniconda, run:" + echo " ./bin/install_miniconda # and once installed, open a new shell and run " + exit 1 fi +# Check conda version is sufficent # Required version -required_version="24.11.1" - -# Check if conda is installed -if ! command -v conda &> /dev/null; then - echo "Error: conda is not installed." - return 3 -fi +required_version="24.0.0" # Get the current conda version current_version=$(conda --version | awk '{print $2}') # Compare versions if [[ "$(printf '%s\n' "$required_version" "$current_version" | sort -V | head -n1)" != "$required_version" ]]; then - echo "Error: Conda version $current_version is less than the required version $required_version." - return 3 + echo "Error: Conda version $current_version is less than the required version $required_version ." + exit 3 fi echo "Conda version $current_version meets the requirement of $required_version or higher." + +# Check if the Conda environment DAYCLI is active +if [[ "$CONDA_DEFAULT_ENV" == "DAYCLI" ]]; then + echo "The Conda environment DAYCLI is active, proceeding." +else + echo "The Conda environment DAYCLI is not active or missing." + echo "" + echo "Please activate the DAYCLI environment by running:" + echo " conda activate DAYCLI" + echo "" + echo "If the environment does not exist, create it by running:" + echo " ./bin/init_daycli" + exit 3 +fi + + # Activate or create the Daylily CLI conda environment if conda env list | grep -q "^DAYCLI "; then echo "Conda environment 'DAYCLI' already exists. Activating it." @@ -183,7 +166,7 @@ else source bin/init_daycli if [[ $? -ne 0 ]]; then echo "Error: Failed to create the 'DAYCLI' environment. Exiting." - return 3 + exit 3 fi fi conda activate DAYCLI @@ -198,7 +181,7 @@ if [[ "$pcluster_version" != "$expected_pcluster_version" ]]; then handle_warning "Warning: Expected pcluster version $expected_pcluster_version, but found version $pcluster_version." if [[ $? -ne 0 ]]; then echo "Error: pcluster version mis-match: expected:$expected_pcluster_version , detected:$pcluster_version ." - return 3 + exit 3 fi fi @@ -207,13 +190,13 @@ fi echo "Verifying AWS credentials..." if ! aws sts get-caller-identity --region "$region" --profile $AWS_PROFILE &> /dev/null; then echo "Error: AWS credentials are invalid or do not have access to the AWS account in region $region." - return 3 + exit 3 else echo "AWS credentials verified successfully." fi # Call the region validation function -validate_region "$region" || return 3 +validate_region "$region" || exit 3 AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text --profile $AWS_PROFILE ) @@ -242,10 +225,10 @@ else echo "Proceeding anyway..." elif [ "$CHOICE" -eq 2 ]; then echo "Exiting." - return 3 2>/dev/null || exit 3 + exit 3 2>/dev/null || exit 3 else echo "Invalid choice. Exiting." - return 3 2>/dev/null || exit 3 + exit 3 2>/dev/null || exit 3 fi fi @@ -263,7 +246,7 @@ check_quota() { if [[ $? -ne 0 ]]; then handle_warning "Error: Unable to retrieve quota for $resource_name." - return 3 + exit 3 fi quota_value=$(echo "$quota" | jq -r '.Quota.Value') @@ -272,7 +255,7 @@ check_quota() { handle_warning "Warning: $resource_name quota is below the recommended $recommended_count. Current quota: $quota_value." if [[ $? -ne 0 ]]; then echo "quota check failed" - return 3 + exit 3 fi else echo "$resource_name quota is sufficient: $quota_value." @@ -286,42 +269,42 @@ echo "Performing AWS quota checks..." # Dedicated Instances check_quota "Spot vCPU Max" "L-1216C47A" 20 ec2 if [[ $? -ne 0 ]]; then - return 3 + exit 3 fi echo "" # Spot vCPU Max Quota check_quota "Spot vCPU Max" "L-34B43A08" 208 ec2 if [[ $? -ne 0 ]]; then - return 3 + exit 3 fi echo "" # VPC Quota (Default: 5 per region) check_quota "VPCs" "L-F678F1CE" 4 vpc if [[ $? -ne 0 ]]; then - return 3 + exit 3 fi echo "" # Elastic IP Quota (Default: 5 per region) check_quota "Elastic IPs" "L-0263D0A3" 4 ec2 if [[ $? -ne 0 ]]; then - return 3 + exit 3 fi echo "" # NAT Gateway Quota (Default: 5 per AZ) check_quota "NAT Gateways" "L-FE5A380F" 4 vpc if [[ $? -ne 0 ]]; then - return 3 + exit 3 fi echo "" # Internet Gateway Quota (Default: 5 per region) check_quota "Internet Gateways" "L-A4707A72" 4 vpc if [[ $? -ne 0 ]]; then - return 3 + exit 3 fi echo "" @@ -389,7 +372,7 @@ echo "" if [[ ${#valid_keys[@]} -eq 0 ]]; then echo "No valid ED25519 keys with matching PEM files were found." echo "Please ensure the required keys are available in the $HOME/.ssh directory AND that the keypair name includes 'omics' in the name." - return 3 + exit 3 fi # Present valid keys to the user using 'select' echo "Select a valid ED25519 key with a matching PEM file:" @@ -459,7 +442,7 @@ done <<< "$buckets" if [[ ${#matching_buckets[@]} -eq 0 ]]; then echo "No matching buckets found in region $region." - return 3 + exit 3 fi echo "Select a matching S3 bucket:" @@ -481,13 +464,13 @@ check_s3_folder() { echo "Folder s3://$bucket/$folder/ exists." else echo "ERROR: Folder s3://$bucket/$folder/ does not exist." - return 1 + exit 1 fi } echo "Validating required folders in bucket: $selected_bucket" -check_s3_folder "$bucket" "data" || return 3 -check_s3_folder "$bucket" "cluster_boot_config" || return 3 +check_s3_folder "$bucket" "data" || exit 3 +check_s3_folder "$bucket" "cluster_boot_config" || exit 3 bucket_url="s3://$selected_bucket" @@ -520,7 +503,7 @@ if [[ -z "$public_subnets" && -z "$private_subnets" ]]; then bin/init_cloudstackformation.sh ./config/day_cluster/pcluster_env.yml "$res_prefix" "$region_az" "$region" $AWS_PROFILE elif [[ -z "$public_subnets" || -z "$private_subnets" ]]; then echo "Error: Incomplete setup. Ensure public and private subnets and the required ARN are available. You might try running : bin/init_cloudstackformation.sh ./config/day_cluster/pcluster_env.yml $res_prefix $region_az $region $AWS_PROFILE" - return 1 + exit 1 else echo "All required resources are available." fi @@ -577,7 +560,7 @@ echo "" # Function to validate cluster name validate_cluster_name() { if [[ ! "$1" =~ ^[a-zA-Z0-9\-]+$ ]] || [[ ${#1} -gt 25 ]]; then - return 1 + exit 1 else return 0 fi @@ -636,7 +619,7 @@ if [[ -z "$global_budget_exists" || "$global_budget_exists" == "None" ]]; then if [[ $? -ne 0 ]]; then echo "Error: Failed to create the budget. Exiting." - return 3 + exit 3 else echo "Budget '$global_budget_name' created successfully." fi @@ -685,7 +668,7 @@ if [[ -z "$budget_exists" || "$budget_exists" == "None" ]]; then if [[ $? -ne 0 ]]; then echo "Error: Failed to create the budget. Exiting." - return 3 + exit 3 else echo "Budget '$budget_name' created successfully." fi @@ -719,7 +702,7 @@ fi if [[ ! -f "$cluster_yaml" ]]; then echo "Error: YAML file '$cluster_yaml' does not exist." - return 3 + exit 3 fi echo "" @@ -852,7 +835,7 @@ if [[ $? -ne 0 ]]; then echo "Error: Failed to calculate spot bid prices." echo "This is likely due to a policy/permissions issue in AWS." echo "Please refer to the quickstart guide for more information on required policies." - return 3 # Use 'exit 3' if outside a function or script + exit 3 # Use 'exit 3' if outside a function or script fi echo "" @@ -870,10 +853,10 @@ if [[ "$message" == "Request would have succeeded, but DryRun flag is set." ]]; echo "Dry run successful. Proceeding with cluster creation." elif [[ "$DAY_BREAK" == "1" ]]; then echo "DAY_BREAK == '1'. Exiting without creating the cluster." - return 0 + exit 0 else echo "Error: Dry run failed ( pcluster create-cluster -n $cluster_name -c $target_conf_fin --dryrun true --region $region ). Exiting." - return 3 + exit 3 fi echo "" diff --git a/bin/daylily-get-ephemperal-cluster-deets b/bin/daylily-get-ephemperal-cluster-deets old mode 100644 new mode 100755 diff --git a/bin/daylily-run-ephemeral-cluster-remote-tests b/bin/daylily-run-ephemeral-cluster-remote-tests old mode 100644 new mode 100755 index d9c57a2e..27c4e005 --- a/bin/daylily-run-ephemeral-cluster-remote-tests +++ b/bin/daylily-run-ephemeral-cluster-remote-tests @@ -1,13 +1,7 @@ -#!/bin/zsh +#!/bin/bash # Daylily Headnode Configuration Script -# Ensure the script is sourced, not executed directly -if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then - echo "This script should be sourced, not executed directly. Please use 'source THISSCRIPT' to run." - exit 1 -fi - # Capture arguments for PEM file and region pem_file=$1 region=$2 @@ -91,12 +85,12 @@ read REPLY if [[ "$REPLY" == "y" ]]; then ssh -i "$pem_file" ubuntu@"$cluster_ip_address" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ - "sudo su - $duser -c 'mkdir -p /fsx/analysis_results/ubuntu/daylily_remote_test && cd /fsx/analysis_results/ubuntu/daylily_remote_test && git clone https://github.com/Daylily-Informatics/daylily.git daylily_remote_test'" + "sudo su - $duser -c 'mkdir -p /fsx/analysis_results/ubuntu/daylily_remote_test && cd /fsx/analysis_results/ubuntu/daylily_remote_test && git clone https://github.com/Daylily-Informatics/daylily.git daylily'" session_name="cluster_test_$(date +%s)" ssh -t -i "$pem_file" ubuntu@"$cluster_ip_address" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ "tmux new-session -d -s \"$session_name\" \ - \"bash -c 'source ~/.bashrc && cd /fsx/analysis_results/ubuntu/daylily_remote_test && source dyinit && source bin/day_activate slurm hg38 remote && ./bin/day_run produce_snv_concordances -p -k -j 2 --config aligners=[\\\"strobe\\\",\\\"bwa2a\\\"] dedupers=[\\\"dppl\\\"] genome_build=\\\"hg38\\\" snv_callers=[\\\"deep\\\"]; bash'\"" && \ + \"bash -c 'source ~/.bashrc && cd /fsx/analysis_results/ubuntu/daylily_remote_test/daylily && source dyinit && source bin/day_activate slurm hg38 remote && ./bin/day_run produce_snv_concordances -p -k -j 2 --config aligners=[\\\"strobe\\\",\\\"bwa2a\\\"] dedupers=[\\\"dppl\\\"] genome_build=\\\"hg38\\\" snv_callers=[\\\"deep\\\"]; bash'\"" && \ echo "Tmux session >>> '$session_name' <<< started on the remote server." echo "You can monitor the session using the following command:" diff --git a/bin/init_daycli b/bin/init_daycli index 7fd96c5e..faa5db7a 100755 --- a/bin/init_daycli +++ b/bin/init_daycli @@ -1,77 +1,39 @@ #!/bin/bash -skip_warn=$1 -warn_msg="" - mkdir -p ~/.config/daylily -# Check if the script is running in Zsh and emulate Zsh behavior -if [ -n "$ZSH_VERSION" ]; then - emulate -L zsh # Ensure Zsh behaves like Zsh (if required) - - # Check if the script is sourced or executed - if [ -n "$ZSH_EVAL_CONTEXT" ] && [[ "$ZSH_EVAL_CONTEXT" =~ :file$ ]]; then - echo "This script is sourced." - else - echo "Error: This script must be sourced, not executed directly. Use 'source $0' to run." - exit 3 - fi -fi - -# Ensure the script is sourced -if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then - echo "Error: This script must be sourced, not executed directly. Use 'source $0' to run." - exit 3 -fi -# Check if system packages are installed -bash bin/check_system_pkgs.sh -if [ $? -ne 0 ]; then - - if [ "$skip_warn" != "skip_warn" ]; then - warn_msg="There were problems detected with pre-requisites, and the warnings bypassed. Run 'bash bin/check_system_pkgs.sh' for more information." - else - echo "Some system packages are missing or are the wrong version which may cause problems. Please review them by running: 'bash bin/check_system_pkgs.sh' for more information. You may proceed past this warning by passing the first argument as 'skip_warn'." - return 1 - fi -fi # Ensure Conda exists if ! command -v conda &> /dev/null; then - echo "Error: Conda is not available in this shell. Attempting to install Miniconda..." - echo "" + echo "Error: Conda is not available in this shell. Please install Miniconda." + echo "Install with:" - echo "source ./bin/install_miniconda" - source ./bin/install_miniconda + echo " ./bin/install_miniconda " + exit 3 fi -CURR_SHELL=$(ps -p $$ -o comm=) -echo $CURR_SHELL if ! command -v conda &> /dev/null; then - echo "Error: Conda is not available in this shell. Autoinstall failed, please see the miniconda docs." - echo "*** If your shell (login:$SHELL , current:$CURR_SHELL) is bash, please check if conda init correctly updated both your .bashrc and .bash_profile with the conda initialization block. If present in one but not the other, copy it, open a new bash shell and try again (most commonly a problem with MACs). ***" - return 1 + echo "Error: Conda is not available in this shell. " + echo "Install with:" + echo " ./bin/install_miniconda " + exit 1 fi # Activate or create the Daylily CLI conda environment if conda env list | grep -q "^DAYCLI "; then - echo "Conda environment 'DAYCLI' already exists." + echo "Conda environment 'DAYCLI' already exists!" + echo "Activate with: " + echo " conda activate DAYCLI" + exit 3 else echo "Creating 'DAYCLI' environment." conda env create -y -n DAYCLI -f config/day/daycli.yaml - #conda create -y -n DAYCLI -c conda-forge python parallel jq nodejs==18.15.0 aws-parallelcluster==3.11.1 flask=2.2.5 pyyaml -fi -echo "Activating 'DAYCLI' environment." -conda activate DAYCLI -# Function to check if a Python package is installed -check_and_install() { - PACKAGE=$1 - VERSION=$2 - if ! python -c "import $PACKAGE" &> /dev/null; then - echo "$PACKAGE not found. Installing $PACKAGE==$VERSION..." - pip install "$PACKAGE==$VERSION" + if [ $? -ne 0 ]; then + echo "Error: Failed to create 'DAYCLI' environment." + exit 1 fi -} - +fi -echo "Daylily CLI environment is ready ($warn_msg)." \ No newline at end of file +echo "Daylily CLI environment is ready! Activate with: " +echo " conda activate DAYCLI" \ No newline at end of file diff --git a/bin/install_miniconda b/bin/install_miniconda old mode 100644 new mode 100755 index 33b033c3..60a043ad --- a/bin/install_miniconda +++ b/bin/install_miniconda @@ -1,23 +1,5 @@ -# Ensure the script is sourced and not executed directly -if [[ -n "$ZSH_VERSION" ]]; then - # Check if the script is being sourced in zsh - # Check if the script is sourced or executed - if [ -n "$ZSH_EVAL_CONTEXT" ] && [[ "$ZSH_EVAL_CONTEXT" =~ :file$ ]]; then - echo "This script is sourced, proceed" - else - echo "Error: This script must be sourced, not executed directly. Use 'source $0' to run." - exit 3 - fi -elif [[ -n "$BASH_VERSION" ]]; then - # Check if the script is being sourced in bash - if [[ "$0" == "${BASH_SOURCE[0]}" ]]; then - echo "This script must be sourced. Use: source $0 or . $0" - return 1 - fi -else - echo "Unsupported shell. Please use bash or zsh to run this script." - return 1 -fi + + # Check if Miniconda is already installed conda --version &> /dev/null @@ -25,7 +7,7 @@ if [[ $? -ne 0 ]]; then echo "Miniconda is not installed. Proceeding with installation..." else echo "Miniconda is already installed. Please uninstall it before proceeding." - return 0 + exit 0 fi @@ -49,7 +31,7 @@ if [[ -z "$MACHINE" ]]; then ;; *) echo "Unsupported architecture: $uname_output" - return 1 + exit 1 ;; esac fi @@ -82,29 +64,68 @@ elif [[ "$MACHINE" == "linux_arm" ]]; then else echo "Unsupported machine type: $MACHINE" echo "Please visit https://docs.conda.io/en/latest/miniconda.html for manual installation instructions." - return 2 + exit 2 fi # Initialize Conda if [[ -x ~/miniconda3/bin/conda ]]; then echo "Initializing Conda..." - ~/miniconda3/bin/conda init "$(basename "$SHELL")" ~/miniconda3/bin/conda init bash ~/miniconda3/bin/conda init zsh - echo "Reloading the shell..." - if [[ $- == *i* ]]; then - source ~/.bashrc 2>/dev/null || source ~/.zshrc 2>/dev/null - else - echo "Please restart your terminal or run: source ~/.bashrc or source ~/.zshrc" + # Define file paths + BASHRC="$HOME/.bashrc" + BASH_PROFILE="$HOME/.bash_profile" + + touch $BASHRC $BASH_PROFILE + + # Function to extract and copy the Conda block + copy_conda_block() { + local source_file=$1 + local target_file=$2 + + # Extract the Conda block from the source file + CONDA_BLOCK=$(sed -n '/# >>> conda initialize >>>/,/# <<< conda initialize <<> $target_file + echo "$CONDA_BLOCK" >> "$target_file" + echo "Conda initialization block added to $target_file." + } + + # Check for Conda block in both files + BASHRC_HAS_BLOCK=false + BASH_PROFILE_HAS_BLOCK=false + + if grep -q "# >>> conda initialize >>>" "$BASHRC"; then + BASHRC_HAS_BLOCK=true + fi + + if grep -q "# >>> conda initialize >>>" "$BASH_PROFILE"; then + BASH_PROFILE_HAS_BLOCK=true fi - if command -v conda &>/dev/null; then - echo "Miniconda installed and initialized successfully!" + # Handle the cases + if $BASHRC_HAS_BLOCK && ! $BASH_PROFILE_HAS_BLOCK; then + echo "Conda initialization block found in .bashrc but missing in .bash_profile." + copy_conda_block "$BASHRC" "$BASH_PROFILE" + elif $BASH_PROFILE_HAS_BLOCK && ! $BASHRC_HAS_BLOCK; then + echo "Conda initialization block found in .bash_profile but missing in .bashrc." + copy_conda_block "$BASH_PROFILE" "$BASHRC" + elif $BASHRC_HAS_BLOCK && $BASH_PROFILE_HAS_BLOCK; then + echo "Conda initialization block already exists in both .bashrc and .bash_profile. No changes made." else - echo "Miniconda installation succeeded but initialization failed. Please check your shell configuration." + echo "No Conda initialization block found in either .bashrc or .bash_profile." + echo "From a bash shell, please run 'conda init bash' to initialize Conda." + exit 1 fi + + echo "Miniconda installation successful." + echo "Open a new shell, and conda should be available (try: 'conda -v')." else - echo "Miniconda installation failed. Please try again." - return 3 + echo "Miniconda installation failed. Please try again and/or atttempt the commands in this script manually." + exit 3 fi diff --git a/bin/other/regsub_yaml.sh b/bin/other/regsub_yaml.sh index d3f174e9..ccc57b0f 100755 --- a/bin/other/regsub_yaml.sh +++ b/bin/other/regsub_yaml.sh @@ -33,4 +33,6 @@ while IFS='=' read -r key value; do sed -i "" "s/$key/$escaped_value/g" "$cluster_cfg_yaml" done < "$cluster_init_values" -echo "\n\n\tSubstitutions completed in $cluster_cfg_yaml.\n" +echo "" +echo "Substitutions completed in $cluster_cfg_yaml." +echo "" diff --git a/bin/retshell b/bin/retshell new file mode 100644 index 00000000..5a43d324 --- /dev/null +++ b/bin/retshell @@ -0,0 +1,2 @@ +CURR_SHELL=$(ps -p $$ -o comm=) +echo "$CURR_SHELL" \ No newline at end of file diff --git a/config/day/daycli.yaml b/config/day/daycli.yaml index f90c4d05..e1b87905 100755 --- a/config/day/daycli.yaml +++ b/config/day/daycli.yaml @@ -19,6 +19,8 @@ dependencies: - tabix - fd-find - graphviz + - nodejs + - jq - pip: - docopt==0.6.2 - colr==0.9.1 diff --git a/docs/quickest_start.md b/docs/quickest_start.md index 171c5818..c7ba410f 100644 --- a/docs/quickest_start.md +++ b/docs/quickest_start.md @@ -24,10 +24,10 @@ _on the laptop you'll create ephemeral cluster, where you have saved the daylily _One is needed per region you intend to run in._ ```bash -AWS_PROFILE=daylily-service +export AWS_PROFILE=daylily-service BUCKET_PREFIX=yocorp REGION=us-west-2 -bash ./bin/create_daylily_omics_analysis_s3.sh --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX --disable-dryrun +./bin/create_daylily_omics_analysis_s3.sh --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX --disable-dryrun ``` - This will take ~20min if you remain in `us-west-2`, longer if cross region. @@ -45,7 +45,7 @@ rclone copy $sentieon_lic yocorp_remote:yocorp-daylily-omics-analysis-us-west-2/ ### Check Spot Market Pricing (optional, but suggested to get best pricing over time) ```bash -AWS_PROFILE=daylily-service +export AWS_PROFILE=daylily-service REGION=us-west-2 OUT_TSV=./init_daylily_cluster.tsv @@ -57,11 +57,11 @@ OUT_TSV=./init_daylily_cluster.tsv ### Create Ephemeral Cluster _this script will check and install a number of prerequsites and attempt to install_ -```zsh -AWS_PROFILE=daylily-service +```bash +export AWS_PROFILE=daylily-service REGION_AZ=us-west-2c -source bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE +./bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE ``` You will be prompted for a variety of config settings. Some resources, if missing, will be created. Other resources if missing will trigger failures. If all is well, you'll be left with a success message with a suggested command to start a remote test on the headnode. diff --git a/setup.py b/setup.py index 9637008c..6bfd9a6d 100755 --- a/setup.py +++ b/setup.py @@ -2,7 +2,7 @@ setup( name="daylily", - version="0.7.155", + version="0.7.156", packages=find_packages(), install_requires=[ # Add dependencies here