CHESS Quantum Beamline Processing Workflow

A Pegasus workflow to do processing for CHESS Quantum Materials Beamline

Running the Sample Workflow on CHESS SGE Cluster

Login to Classe

$ ssh lnx201.classe.cornell.edu
$ cd ~/CLASSE_shortcuts/chess_[username]
$ git clone https://github.com/pegasus-isi/chess-qmb-workflow.git

Check out the codes for the individual jobs

$ cd chess-qmb-workflow
$ git clone https://gitlab01.classe.cornell.edu/ss3428/pegasus.git code

The above repository is in CLASSE GitLab and you need to do the checkout using your Classe account.

Submit the workflow

$ cd chess-qmb-workflow 
$ ./chess-qmb.py --execution_site sge

Note that when Pegasus plans/submits a workflow, a work directory is created and presented in the output. This directory is the handle to the workflow instance and used by Pegasus command line tools. Some useful tools to know about:

• pegasus-status -v [wfdir] Provides status on a currently running workflow. (more)
• pegasus-analyzer [wfdir] Provides debugging clues why a workflow failed. Run this after a workflow has failed. (more)
• pegasus-statistics [wfdir] Provides statistics, such as walltimes, on a workflow after it has completed. (more)
• pegasus-remove [wfdir] Removes a workflow from the system. (more)

During the workflow planning, Pegasus transforms the workflow to make it work well in the target execution environment.

The executable workflow has a set of additional tasks added by Pegasus: create scratch dir, data staging in and out, and data cleanup jobs.

Check the status of the workflow:

$ pegasus-status [wfdir]

You can keep checking the status periodically to see that the workflow is making progress.

Keep checking progress with pegasus-status. Once the workflow is done, display statistics with pegasus-statistics:

$ pegasus-status [wfdir]
$ pegasus-statistics [wfdir]
...