Currently a node can join the master node in a cluster but once it is shut down there is a panic. This issue aims to make this part of the system more robust.
Useful Additions
-
Re-establish connection: Currently on both sides there is a panic if a connection is terminated. This should be fixed as follows:
- If a worker node looses connection to the master node or does not manage to establish a connection in the first place, it should periodically retry connecting to the master node again.
- The master should be able to accept a connection to the same node again if it has previously left the cluster (this might already supported but should be checked).
-
Context cleanup: Currently the worker awaits a delete message to delete a composition context. If it looses the connection to the master node that manages those contexts it should automatically delete all those stored contexts.
-
Logging: The master node should log a newly established connection and lost connection to a worker node (probably with info level). The worker node should log the same (probably also with info level) + the retries with debug level.
Currently a node can join the master node in a cluster but once it is shut down there is a panic. This issue aims to make this part of the system more robust.
Useful Additions
Re-establish connection: Currently on both sides there is a panic if a connection is terminated. This should be fixed as follows:
Context cleanup: Currently the worker awaits a delete message to delete a composition context. If it looses the connection to the master node that manages those contexts it should automatically delete all those stored contexts.
Logging: The master node should log a newly established connection and lost connection to a worker node (probably with
infolevel). The worker node should log the same (probably also withinfolevel) + the retries withdebuglevel.