Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Opensearch output not erroring when opensearch can't be reached #3007

Open
Hargo-W opened this issue Nov 14, 2024 · 0 comments
Open

Opensearch output not erroring when opensearch can't be reached #3007

Hargo-W opened this issue Nov 14, 2024 · 0 comments
Labels
bug needs investigation It looks as though have all the information needed but investigation is required outputs Any tasks or issues relating specifically to outputs

Comments

@Hargo-W
Copy link

Hargo-W commented Nov 14, 2024

The opensearch output will not throw an error if it cannot reach opensearch. The consumed message will be acked.

running this config in a docker container using the latest RPC image:

input:
  generate:
    count: 1
    mapping: 'root = "Error please"'

output:
  opensearch:
    urls: ["foo"]
    index: "bar"
    action: "index"
    id: "123"

logger:
  level: TRACE

I get these logs:

2024-11-14 14:56:06 benthos  | level=info msg="Running main config from specified file" @service=redpanda-connect benthos_version=v4.33.0 path=/benthos.yaml
2024-11-14 14:56:06 benthos  | level=info msg="Listening for HTTP requests at: http://0.0.0.0:4195" @service=redpanda-connect
2024-11-14 14:56:06 benthos  | level=info msg="Input type generate is now active" @service=redpanda-connect label="" path=root.input
2024-11-14 14:56:06 benthos  | level=trace msg="Consumed 1 messages from 'generate'." @service=redpanda-connect label="" path=root.input
2024-11-14 14:56:06 benthos  | level=info msg="Launching a Redpanda Connect instance, use CTRL+C to close" @service=redpanda-connect
2024-11-14 14:56:06 benthos  | level=info msg="Output type opensearch is now active" @service=redpanda-connect label="" path=root.output
2024-11-14 14:56:06 benthos  | level=trace msg="Attempting to write 1 messages to 'opensearch'." @service=redpanda-connect label="" path=root.output
2024-11-14 14:56:06 benthos  | level=debug msg="Successfully dispatched [%!s(uint64=0)] documents in 7ms (%!s(int64=0) docs/sec)" @service=redpanda-connect label="" path=root.output
2024-11-14 14:56:06 benthos  | level=trace msg="Successfully wrote 1 messages to 'opensearch'." @service=redpanda-connect label="" path=root.output
2024-11-14 14:56:06 benthos  | level=debug msg="Waiting for pending acks to resolve before shutting down." @service=redpanda-connect label="" path=root.input
2024-11-14 14:56:06 benthos  | level=debug msg="Pending acks resolved." @service=redpanda-connect label="" path=root.input
2024-11-14 14:56:06 benthos  | level=info msg="Pipeline has terminated. Shutting down the service" @service=redpanda-connect
2036-01-01 00:00:00 
benthos exited with code 0

These logs state that the message was written successfully, even though there is no opensearch cluster to connect to.

I tested this setup with a kafka output as well as a redis output instead. Both of them throw errors and retry as expected.

Thank you!

@mihaitodor mihaitodor added bug outputs Any tasks or issues relating specifically to outputs needs investigation It looks as though have all the information needed but investigation is required labels Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug needs investigation It looks as though have all the information needed but investigation is required outputs Any tasks or issues relating specifically to outputs
Projects
None yet
Development

No branches or pull requests

2 participants