Checking for removed layers

[djtfmartin]

We currently don't check for layers that have been removed from system.
This could mean the sample cache stored in HDFS at a dataset level or for all datasets continues to have layers in it that are no longer required making indexes bigger than necessary.
The remedy for this currently is to delete the sample cache manually (we could this an option to the Jenkins job).

To implement this, we will need to store a listing of the layers in sampling service and check against the service for changes.
A better solution might be to support soft deletes in the layers service. This is done to an extent already with the enabled flag, but there is no associated timestamp so there isn't a way of determining when a layer was deleted.

As the removal of layers isn't something that is that frequent, this is probably low priority.