Baseline training configuration

[ealerskans]

This is the baseline configuration we discussed today for the training_params.yaml file

num_workers: 8 
batch_size: 1
hidden_dim: 128
processor_layers: 4
model: hi_lam
graph: hierarchical
epochs: 30
ar_steps_train: 1
lr: 0.001
val_interval: 1
ar_steps_eval: 4
val_steps_to_log: 1 2 4
metrics_watch: val_rmse, mse, random_metric
logger: mlflow
num_nodes: 2

This requires an update of train_wrapper.py since there has been an addition of variables.

Since we dicussed about using the hierarchical graph instead the graph-creation stage in dvc.yaml also needs to be updated:

mllam-exps/dvc.yaml

Lines 10 to 12 in 8f940d6

	create_graph-slurm:
	cmd: python -m neural_lam.create_graph --config_path data/config.yaml --name 1level
	--levels 1

We discussed about using a hierarchical graph of 4 levels so not sure if that would be something like this

  create_graph:
    cmd: python -m neural_lam.create_graph --config_path data/config.yaml --hierarchical --name hierarch_4lev
      --levels 4

In the config above named the new graph "hierarch_4lev" but we could perhaps use something more descriptive. @leifdenby I saw that you had a suggestion for this.

The evaluate_params.yaml would also need to be updated accordingly.

The slurm training script would also need to be modified since we agreed upon using 2 nodes instead of 1 and currently the number of nodes supplied to the training wrapper is hard-coded to 1, as well as the numebr of nodes to use in the slurm header.

[leifdenby]

TODO:

• graph creation needs changing to hierarchical with 4 levels @SimonKamuk Feat/hilam 4lvl #6
• number of nodes as a param, not hardcoded in submission script @observingClouds Simplify params #7
• slurm script for dataset generation @ealerskans Add slurm script for dataset-preparation stage #2
• instructions on secrets needs updating because secrets file is not tracked, need to double check env vars propagate down when running queued experiments that spawn their own shell @leifdenby
• common params.yaml for common train/eval params @leifdenby Create common params file #5 and/or Simplify params #7
• move gpu stats write directory to logs/ folder @SimonKamuk
• datastore config for Baseline A @observingClouds First common DANRA baseline dataset #1

[observingClouds]

@ealerskans, and others: we also agreed upon using --precision bf16-mixed, right?
and what about the graph? I see Kapser uses --model graph_lam while here we mention --model hi_lam

[observingClouds]

In the config above named the new graph "hierarch_4lev" but we could perhaps use something more descriptive. @leifdenby I saw that you had a suggestion for this.

It actually needs to be hierachical as the name is used for the subfolder in graph and is expected to be hierachical

[ealerskans]

@ealerskans, and others: we also agreed upon using --precision bf16-mixed, right?

Yes, we agreed upon that. I just forgot to add it.

and what about the graph? I see Kapser uses --model graph_lam while here we mention --model hi_lam

I thought that Kasper was using both to compare them? I think we agreed upon the hierarchical one (can't remember why)

[ealerskans]

In the config above named the new graph "hierarch_4lev" but we could perhaps use something more descriptive. @leifdenby I saw that you had a suggestion for this.

It actually needs to be hierachical as the name is used for the subfolder in graph and is expected to be hierachical

But if we supply --name then it should work?
https://github.com/mllam/neural-lam/blob/ada9c1644beb56de832127cba46971b1be1bdd98/neural_lam/create_graph.py#L561-L592

Isn't that also why we have the "name" (graph key) here?

mllam-exps/data/training_params.yaml

Lines 1 to 2 in d8a473e

	model: graph_lam
	graph: 1level

[observingClouds]

@leifdenby , regarding the propagation of the mlflow secrets, would a solution be to advise people to create a secrets.sh in the home directory and refer to that in slurm.neural-lam.sh with ~/secrets.sh. In this case the secrets are for sure always available and one will not be able to commit them by accident as they are outside the git repo

[leifdenby]

@leifdenby , regarding the propagation of the mlflow secrets, would a solution be to advise people to create a secrets.sh in the home directory and refer to that in slurm.neural-lam.sh with ~/secrets.sh. In this case the secrets are for sure always available and one will not be able to commit them by accident as they are outside the git repo

Yes we could do that, although is that needed? Can't people just be instructed to set these envirionment variables like they have to set MLFLOW_TRACKING_URI? As in, put them in their ~/.bashrc

[leifdenby]

Isn't that also why we have the "name" (graph key) here?

It is quite confusing, the name simply refers to the filename (or more correctly the path suffix) the graph is saved in. We have to both provide a "name" for the graph and tell neural-lam what kind of graph that is (with --model graph_lam), or rather what kind of model to construct (that needs to match the kind of graph). This should be changed in my opinion since I think that from a given graph you only construct one particular kind of neural network pytorch nn.Module model. @matschreiner suggested the same to me the other day I think :)