Proper way to implement processRecord retries

[gjkliewer]

I am wondering what the proper way to implement retry logic in the processRecords function that will attempt (indefinitely) to reprocess records from a checkpointed sequence number.

For example, in our KCL application, in the processRecord function, we do some processing on the record data and then send the result via TCP to another service. If for some reason that service is down, we do not want to continue processing subsequent records but rather continue to retry processing the same record until it succeeds. In this scenario there is no issue with the data, it's just that we aren't able to push the processed data to it's desired destination.

From testing I've discovered that within the processRecord function, even if completeCallback() is called without calling checkpoint(), the KCL will still attempt to grab the next record and not retry the previous record that failed to be "processed". But, I've also noticed that if completeCallback() is called without calling checkpoint() and the KCL application is restarted, then it will start again from after the last checkpoint, thus retrying the records that failed to "process".

So by this, it seems that in order to implement retry logic, we should be stopping the KCL application and restarting it. Is there a recommended way to do that? In the java KCL library I noticed there is a worker.requestShutdown() method, but I don't see that in the Nodejs library, is it just a feature that hasn't been implemented yet? Either way, is the suggested work around to just do a process.exit() call when we want to retry (and having something watch the process that restarts it)? That seems like it may not be the best in regards to graceful a shutdown, is there a better way?

I believe this is related to the conversation here awslabs/amazon-kinesis-client#10