generalized data lineage of plugin instances

[jennydaman]

This is an idea of how ChRIS can be overhauled to both simplify its functionality and add new features.

Need

This idea addresses all the shortcomings of CUBE described in #481, as well as adding features, improving performance, understandability, and "elegance". These qualities are important to users.

Definitions

• root plugin instance: a plugin instance which has 0 previous plugin instances
• plugin type: one of: "fs", "ds", "ts"

Generalized Multiple Inheritance

• ChRIS plugin spec SHALL be simplified to not have a type
• The spec of a ChRIS plugin SHALL be commandname [--parameters ...] inputdir/ outputdir/
• A feed SHALL be a set of [0, n) plugin instances, and may have multiple roots
• If a plugin instance has [2, n) previous plugin instances, its inputs SHALL be the files of all its previous plugin instances collected in one directory. If there are any file name collisions, the plugin instance MUST fail before being sent to the remote compute resource.

Implementation Pseudocode

Python pseudocode showing how plugin instance scheduling logic can be achieved using the observer pattern.

class PluginInstance:

    def finished(self):
        return self.status in ('cancelled', 'finishedSuccessfully', 'finishedWithError')

    def errored(self):
        return self.status in ('finishedWithError', 'cancelled')

    def on_create(self):
        if all(p.status == 'finishedSuccessfully' for p in self.previous):
            return self.schedule_now()
        if any(p.errored() in for p in self.previous):
            self.status = 'cancelled'
            return
        for previous in self.previous:
            if not previous.finished():
                previous.add_subscriber(self)
        # waiting for previous to finish...

    def on_finished(self):
        for subscriber in self.subscriber:
            subscriber.notify(self)

    def notify(self, finished_previous):
        if finished_previous != 'finishedSuccessfully':
            self.status = 'cancelled'
            return
        if all(p.status == 'finishedSuccessfully' for p in self.previous):
            return self.schedule_now()
        # else, keep waiting...

Additionally, on start (and restart) of CUBE: search for orphans, log them, and start them. Where an "orphan" is a plugin instance whose previous are all 'finishedSuccessfully' but its own status is 'waiting'. (This would only happen in the event of database corruption, bug, being killed, improper shutdown, ...)

Comparison with Current System

• A plugin instance with 0 previous plugin instances is a root, akin to a fs-type plugin. If a plugin instance has 1 previous, it is akin to a ds-type plugin. If a plugin instance has 2 or more plugin instances, it is akin to a ts-type plugin followed by a ds-type plugin.
• A feed is merely for organizational purposes. Currently, a feed is a connected single-root DAG. In this proposal, a feed is [0, N) DAGs which may or may not be connected.

Improvements

Features

• plugin instances may have multiple previous plugin instances
• feeds may have multiple roots

Performance

Currently, the scheduling of plugin instances is implemented using busy waiting, which is slow and expensive.

ChRIS_ultron_backEnd/chris_backend/plugininstances/tasks.py

Lines 72 to 84 in 5d3c2e2

	@shared_task(bind=True)
	@skip_if_running
	def schedule_waiting_plugin_instances(self): # task is passed info about itself
	"""
	Schedule the apps corresponding to all plugin instances in 'waiting' DB status
	and whom previous plugin instance is in 'finishedSuccessfully' DB status.
	"""
	instances = PluginInstance.objects.filter(status='waiting',
	previous__status='finishedSuccessfully')
	for plg_inst in instances:
	plg_inst.status = 'scheduled'
	plg_inst.save()
	run_plugin_instance.delay(plg_inst.id) # call async task

In this proposal, a more efficient implementation based on the observer pattern is described.

Understandability and Elegance

This proposal would resolve these points of confusion:

• The difference between "fs", "ds", and "ts" plugins
• IMO "feed syntheses," "data synthesis," and "topology synthesis" are unnecessary jargon
• When running pl-topologicalcopy, it is necessary to specify its parent in both previous_id and also plugininstances
• When using pl-topologicalcopy to do feed joins, the data provenance is not represented