README.org

TOC

• Installation
• Usage
  • Setup
    • Select a host to run the code
    • Prepare to run WITCH for the first time
  • Commands
    • Sync
      • Download a remote folder
      • Upload a local folder
    • WITCH
      • Generate data folder
      • Calibrate
      • Run witch non-interactively
      • Check non-interactive WITCH runs
      • Run with starting point
      • Run with double-dash parameters
      • Calibrate in a local folder
    • Post-process
      • Show double-dash parameters of a past run
      • Run database on a past run results
    • Misc
      • Run a command via ssh

Installation

All commands below are to be run on your local computer. If you use Windows, make sure you work in a BASH shell prompt, e.g. via the Windows Subsystem for Linux.

1. Clone this repo, e.g. in ~/dotfiles/wbash

   cd ~/dotfiles
   git clone git@github.com:witch-team/wbash.git
       

2. Source wbash.sh in your ~/.bashrc

   echo 'source ~/dotfiles/wbash/wbash.sh' >> ~/.bashrc
       

   Open a new shell to make the new commands available.

3. Follow the setup wizard:

   wsetup-wizard
       

Usage

Setup

Select a host to run the code

export WHOST=zeus
export WHOST=athena
export WHOST=local

3 hosts are supported: zeus and athena will run the code on the respective supercomputer. local will run code locally on your computer. You can change the default (currently zeus) at the beginning of wbash.sh.

The directory used on WHOST depends on

1. WHOST
2. your current local directory name (WD)
3. the git branch you are in (BRANCH)

as follows:

WHOST	WD	BRANCH	work dir on WHOST
local			current dir
athena/zeus	witch-BRANCH		WHOST:~/work/witch-BRANCH
	witch-master		WHOST:~/work/witch
	witch	BRANCH==master	WHOST:~/work/witch
	witch	BRANCH!=master	WHOST:~/work/witch-BRANCH

Prepare to run WITCH for the first time

git clone git@github.com:witch-team/witch.git
cd witch
wsetup

This will:

• clone ../witch-data if it does not exist, pull it, then sync it to WHOST
• clone ../witchtools if it does not exist, pull it, then sync it to WHOST
• pull current dir, then sync it to WHOST
• run tools/R/setup.R on WHOST, which will install basic R dependencies

Commands

Sync

Download a remote folder

To download the remote folder xxx from the remost working dir, use:

wdown xxx

Use wdown -a xxx to download also the all_data_temp*.gdx.

Upload a local folder

To upload the local folder yyy, use:

wup yyy

By default:

• w/o arguments the current dir is uploaded;
• only versioned files are uploaded; to include non-git files, use -a before yyy;
• target folder name on WHOST is the same, to use a different name, e.g. zzz, use -t zzz before yyy.

WITCH

Generate data folder

To generate data_witch17:

wdata

This will:

• sync ../witch-data and ./ to host
• run translate_witch_data.R interactively (drop -i to launch a job non-interactively)
• download the resulting data_witch17 folder (needs to be done manually if non-interactive).

Use -r xxx to use regional setup xxx.

Calibrate

wrun -j calib -i -c

This will:

• sync ../witch-data and ./ to host
• submit a job on WHOST named calib that runs gams run_witch.gms set up for calibration
• at the end download data_witch17 and a calib folder with these contents:

  calib
  ├── calib.err
  ├── calib.lst
  ├── calib.out
  ├── errors_calib.txt
  └── results_calib.gdx
      

File names follow the job name (i.e. the one specified after -j).

Run witch non-interactively

Drop the -i after wrun (this won’t affect local runs).

Check non-interactive WITCH runs

To list all running jobs:

wcheck

To attach to the console output of a running job named bau:

wcheck bau

To check the errors_*.txt file of job bau:

werr bau

Run with starting point

To run a bau using calib/results_calib.gdx as starting gdx:

wrun -j bau -s calib

Notice that it’s enough to specify calib. This will be translated into calib/results_calib. You can also specify a local gdx file after -s. If so, this will be moved to the current folder, synced to host, and used as a startgdx. The same mechanism holds for:

-f

for gdxfix

-b

for baugdx

Add -S for startboost.

Run with double-dash parameters

To pass double-dash parameters to gams, just specify them at the end:

wrun -j wb600 --policy=innopaths --scen=wb600

Calibrate in a local folder

wrun -j calib_xxx -C

Notice the upper-case -C instead of -c. To later use this calibration in a bau for example, run:

wrun -j bau_xxx -i -u calib_xxx

This will also set start and bau gdx to calib_xxx/results_calib_xxx.

Post-process

Show double-dash parameters of a past run

If the run was named bau, use:

wshow bau

Run database on a past run results

If the run was named ctax, use:

wdb ctax

After running the command above, the ctax/ local folder will be updated with the generated db files. Use -o switch to specify a gdx out db name, and -b to specify a bau gdx different than bau/results_bau.

Misc

Run a command via ssh

To run a command, e.g. ls -clt, in WHOST working dir, use:

wssh ls -clt