Need better way to pause between jobs

[keith-turner]

I was running a long running stress test for beta release. I configured the test to load 100K random integers and then sleep for 3min. The sleep time was too long and it was not keeping the cluster busy. Below is a graphite plot of notifications queued over time.

Need to do something besides just sleep. Could possibly wait for the # of notifications to drop below a certain amount before running the next load job. This could be a function of the amount we are loading. For example when loading 100K, could wait until the # of notifications drops below 5x100K before starting the next load job.