DAG-like workflows

[JoshKarpel]

We should implement some kind of DAG workflow support, likely piggy-backing off of HTCondor's DAG support.

One possible API to model this off of: https://keras.io/models/model/

[stsievert]

Dask has a good API for DAGs: https://docs.dask.org/en/latest/graphs.html. The specification is not at all Dask specific. For example, here's a computation and it's DAG representation:

Computation:

def inc(i):
    return i + 1

def add(a, b):
    return a + b

x = 1
y = inc(x)
z = add(y, 10)

DAG representation:

d = {
    'x': 1,
    'y': (inc, 'x'),
    'z': (add, 'y', 10)
}

Incorporating this DAG-specification might mean better integration with Dask (better than the current HTCondorCluster (I would guess this is the case, though I'm not familiar with the internals of Dask).

[keith6014]

My suggestion would be to use DAGman for everything, including vanilla type jobs. Then use the DASK like setup to build the DAGs.