A high performance rust-rdkafka service

[barkanido]

hey folks, I plan to write a service that will consume from Kafka and write a few keys to a key-value data store for each message consumed. I plan to use rust-rdkafka. currently, this Kafka topic has around 250K messages per second, so performance counts 😉
My database client is blocking (https://github.com/aerospike/aerospike-client-rust).
The processing outline is:

1. consume a message
2. parse JSON
3. validation + possible filtering
4. possibly producing some data into another kafka topic
5. write a few records into aerospike
6. GOTO 1

Currently, I see 2 options:

1. use the nonblocking mode for the Kafka consumer and do everything on Tokio thread pools? (tokio::spawn+tokio::task::spawn_blocking)
2. consume in a serial blocking way (with while let Some(message) = message_stream.next().await) or somehow feed this into rayon's par_iter.
   my biggest concern is the ability to back-pressure the message consuming when the database starts to fail/respond slowly.

What do you guys think should work best?

[Darksonn]

Answered on the user's forum.