0.7.147

CHANGELOG.md

@@ -1,10 +1,13 @@
 ## Changelog
 
-### [0.7.78] - 2023-11-17
+### [0.7.147] - 2024-11-30
+- The [README](README.md) has been walked through with a fresh AWS user, and instructions through the end of confirming the headnode can submit and run jobs works! Including running tests and the PCUI functionality works.  Whoop!
+
+### [0.7.78] - 2024-11-17
 - Repeated end-to-end cluster creation and giab test data running to completion. About to bless a beta release (which will have associated data).
 - My intention is to stop further dev on 0.7.* and move to 0.8.* for the next round of changes.
 
-### [0.7.37] - 2023-10-15
+### [0.7.37] - 2024-10-15
 - Initial pre-beta release.
 - Added support for pcui and running daylily in multiple AZs.
 - Included scripts for cloud formation and cluster setup.

README.md

@@ -1,4 +1,4 @@
-# Daylily AWS Ephemeral Cluster Setup (0.7.144)
+# Daylily AWS Ephemeral Cluster Setup (0.7.147)
 
 
 **beta release**
@@ -431,7 +431,10 @@ The script will take all of this info, and proceed to:
   - First, a dryrun cluster creation is attempted.  If successful, creation proceeds.  If unsuccessful, the process will terminate.
   - The ephemeral cluster creation will begin and a monitoring script will watch for its completion. **this can take from 20m to an hour to complete**, depending on the region, size of Fsx requested, S3 size, etc.  There is a max timeout set in the cluster config yaml of 1hr, which will cause a failure if the cluster is not up in that time. 
 
-The terminal will block, and if all is well, you will see the following message:
+The terminal will block, a status message will slowly scroll by, and after ~20m, if successful, the headnode config will begin (you may be prompted to select the cluster to config if there are multiple in the AZ.  The headnode confiig will setup a few final bits, and then run a few tests (you should see a few magenta success bars during this process).
+
+
+If all is well, you will see the following message:
 
 ```text
 You can now SSH into the head node with the following command:
@@ -445,22 +448,25 @@ Once logged in, as the 'ubuntu' user, run the following commands:
 
   dy-r help
  
-Setup complete. You can now start working with Daylily on the head node.
-```
+"Would you like to start building various caches needed to run jobs? [y/n]"
 
-  - (optional), you may select `y` or `n` to begin building the cached environments on the cluster.
+```
+  - (optional), you may select `y` or `n` to begin building the cached environments on the cluster. The caches will be automatically created if missing whenever a job is submitted. They should only need to be created _once_ per ephemeral cluster (the compute nodes all share the caches w/the headnode). The build can take 15-30m the first time.
   - You are ready to roll.
 
 
 > During cluster creation, and especially if you need to debug a failure, please go to the `CloudFormation` console and look at the `CLUSTER-NAME` stack.  The `Events` tab will give you a good idea of what is happening, and the `Outputs` tab will give you the IP of the headnode, and the `Resources` tab will give you the ARN of the FSx filesystem, which you can use to look at the FSx console to see the status of the filesystem creation.
 
-### (untested) Remote Test New Cluster
+### Run Remote Slurm Tests On Headnode
 
 ```bash
 #!/bin/zsh
 source ./bin/daylily-run-ephemeral-cluster-remote-tests $pem_file $region $AWS_PROFILE
-
 ```
+A successful test will look like this:
+  
+  > ![](docs/images/daylily_remote_test_success.png)
+
 
 ### Review Clusters
 You may confirm the cluster creation was successful with the following command (alternatively, use the PCUI console).
@@ -547,7 +553,9 @@ You will need to enter the following (all other params may be left as default):
 - `ImageBuilderSubnetId`: the *subnetID of the public subnet* created in your cluster creation, visit the VPC console to find this (possibly navigate from the EC2 headnode to this).
 - Check the 2 acknowledgement boxes, and click `Create Stack`.
 
-This will boot you to cloudformation, and will take ~10m to complete.  Once complete, you will receive an email with the password to the PCUI console. To find the PCUI url, visit the `Outputs` tab of the `parallelcluster-ui` stack in the cloudformation console and look for the https:// link (which will work once the stack completes). You use the entered email and the password emailed to it to login the first time.
+This will boot you to cloudformation, and will take ~10m to complete.  Once complete, you will receive an email with the password to the PCUI console. 
+
+To find the PCUI url, visit the `Outputs` tab of the `parallelcluster-ui` stack in the cloudformation console, the url for `ParallelClusterUIUrl` is the one you should use. You use the entered email and the password emailed to login the first time.
 
 > The PCUI stuff is not required, but very VERY awesome.
 

bin/daylily-cfg-headnode

@@ -148,7 +148,10 @@ if [[ "$REPLY" == "y" ]]; then
     # start building the minimal cached envs
     echo "Building some cached analysis environments to save a little time ... this can take a little time"
     ssh -t -i "$pem_file" ubuntu@"$cluster_ip_address"  -o StrictHostKeyChecking=no   -o UserKnownHostsFile=/dev/null \
-        "source ~/.bashrc && cd ~/projects/daylily && source dyinit --skip-project-check && source bin/day_activate local hg38 remote && ./bin/day_run  produce_snv_concordances -p -k -j 2 --config aligners=['strobe','bwa2a'] dedupers=['dppl'] snv_callers=['oct','deep']  --conda-create-envs-only  "
+        "source ~/.bashrc && cd ~/projects/daylily && source dyinit --skip-project-check && source bin/day_activate local hg38 remote &&  ./bin/day_run  produce_snv_concordances -p -k -j 2 --config aligners=['strobe','bwa2a'] dedupers=['dppl'] snv_callers=['oct','deep']  --conda-create-envs-only    "
+    echo " "
+    echo "Caches have built for the following tools: strobe, bwa2a, dppl, oct, deep"
+    echo " "
 else
     # Print the command that would have been run
     echo "Skipping cache building. Command would have been:"

bin/daylily-run-ephemeral-cluster-remote-tests

@@ -85,19 +85,6 @@ if [[ ! -f "$pem_file" ]]; then
 fi
 
 
-
-
-# a quick test to confirm things are running as expected locally on the headnode
-echo "Run a quick test to ensure things are working as expected... [local run]"
-ssh -t -i "$pem_file" ubuntu@"$cluster_ip_address"  -o StrictHostKeyChecking=no   -o UserKnownHostsFile=/dev/null \
-    "source ~/.bashrc && cd ~/projects/daylily && source dyinit --project daylily-global && source bin/day_activate local hg38 remote && ./bin/day_run  produce_deduplicated_bams -p -k -j 2 --config aligners=['strobe'] dedupers=['dppl']  "
-
-
-# Provide final instructions for SSH access to the head node
-echo "Reminder::: You can now SSH into the head node with the following command:"
-echo "        ssh -i $pem_file ubuntu@$cluster_ip_address"
-echo " "
-
 echo "Would you like to remotely launch a workflow w/in a tmux session on the headnode? [y/n]"
 
 read REPLY
@@ -114,15 +101,12 @@ if [[ "$REPLY" == "y" ]]; then
 fi
 
 
-echo " "
-echo "PLACEHOLDER -- EXAMPLE OF REMOTE SUBMITTING A WORFLOW FROM A LOCAL SAMPLE SHEET"
-echo " "
 
 echo ""
-echo "again, you may now access the headnode via the PCUI, via 'source bin/daylily-ssh-into-headnode', or SSH into the head node with the following command:"
+echo " >>>>>> you may now access the headnode via the PCUI, via 'source bin/daylily-ssh-into-headnode', or SSH into the head node with the following command:"
 echo "        ssh -i $pem_file ubuntu@$cluster_ip_address"
+echo "        tmux ls"
+echo "        tmux a -t c"
 echo " "
-echo "ALERT! - REMEMBER TO EXPORT YOUR FSX LUSTRE FILESYSTEM analysis_results BACK TO S3 BEFORE DELETING THE CLUSTER - "
-echo "to help w/this, you can run ( bin/daylily-export-fsx-to-s3 ) "
 echo " "
 echo "...fin"

config/aws/daylily-service-cluster-policy.json

@@ -24,7 +24,8 @@
 				"iam:AttachRolePolicy",
 				"iam:DetachRolePolicy",
 				"iam:TagRole",
-				"iam:PutRolePolicy"
+				"iam:PutRolePolicy",
+				"iam:DeleteRole*"
 			],
 			"Resource": "*"
 		},
@@ -33,16 +34,16 @@
                 "iam:PassRole"
             ],
             "Resource": [
-                "arn:aws:iam::<AWS_ACCOUNT_ID>:role/parallelcluster/*",
-				"arn:aws:iam::<AWS_ACCOUNT_ID>:role/ApiGatewayLogRole-2c4ca6e0/*"
+				"arn:aws:iam::<AWS_ACCOUNT_ID>:role/*"
             ],
             "Effect": "Allow"
         },
 		{
 			"Effect": "Allow",
 			"Action": [
 				"cognito-idp:*",
-				"servicequotas:GetServiceQuota"
+				"servicequotas:GetServiceQuota",
+				"ssm:*"
 			],
 			"Resource": "*"
 		},
@@ -94,7 +95,8 @@
 			"Action": [
 				"route53:*",
 				"apigateway:*",
-				"secretsmanager:*"
+				"secretsmanager:*",
+				"ecr:*"
 			],
 			"Resource": "*"
 		},

setup.py

@@ -2,7 +2,7 @@
 
 setup(
     name="daylily",
-    version="0.7.144",
+    version="0.7.147",
     packages=find_packages(),
     install_requires=[
         # Add dependencies here,