in_kafka: optimize poll timeout handling for threaded and main event loop modes

[nareshku]

Summary

Set the Kafka poll timeout based on execution mode:
(Note: The fix is based on what's suggested in #9625)

1. Running in the main event loop (non-threaded):

   • Use a minimal timeout (1ms) to avoid blocking other inputs.

2. Running in a dedicated thread:

   • Optimize for throughput by allowing Kafka's internal batching.
   • Align with 'librdkafka.fetch.wait.max.ms' (default: 500ms) to maximize batch efficiency.
   • Set timeout slightly higher than 'librdkafka.fetch.wait.max.ms' (e.g., 1.5x - 2x) to ensure it does not interfere with Kafka’s fetch behavior, while still keeping the consumer responsive.

Fixes #8030.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Tested locally with the config poll_timeout_ms option.

[INPUT]
    Name        kafka
    Tag         kafka-test
    Brokers     <broker endpoint>
    Topics      <topic-name>
    poll_ms     100
    poll_timeout_ms 1000
    threaded    true

Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.