MM2 trying to migrate from kafka to redpanda

[klinux]

Hello,

I'm trying to use MM2 to mirroring topics from Kafka to Redpanda, but when mirroring starts, Redpanda simply dies, with this error:

INFO  2023-08-03 08:24:10,552 [shard 0] raft - [group_id:242, {kafka/commerce-event-created/0}] consensus.cc:1317 - Starting with voted_for {id: {-2147483648}, revision: {-9223372036854775808}} term 0 initial_state true
2023-08-03T08:24:10.561437688Z ERROR 2023-08-03 08:24:10,561 [shard 0] assert - Assert failure: (/var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0f5ff99c2b18b4005-1/redpanda/redpanda/src/v/raft/heartbeat_manager.cc:346) 'ret.second' double registration of group: {kafka/commerce-event-paid/1}:1001
2023-08-03T08:24:10.561745682Z ERROR 2023-08-03 08:24:10,561 [shard 0] assert - Backtrace below:
2023-08-03T08:24:10.561774236Z 0x6fd29b3 0x2bb67bb 0x2bb1886 0x2b3717e 0x4526568 0x453166a 0x6d1980f 0x6d1ceb1 0x6d1a076 0x6c1eb40 0x6c1cf38 0x213f845 0x701380b /opt/redpanda/lib/libc.so.6+0x2d58f /opt/redpanda/lib/libc.so.6+0x2d648 0x21394a4
   --------
2023-08-03T08:24:10.561790593Z    seastar::internal::coroutine_traits_base<seastar::lw_shared_ptr<raft::consensus>>::promise_type
2023-08-03T08:24:10.561798333Z    --------
2023-08-03T08:24:10.561806971Z    seastar::internal::coroutine_traits_base<std::__1::error_code>::promise_type
   --------
2023-08-03T08:24:10.561842002Z    seastar::internal::coroutine_traits_base<void>::promise_type
2023-08-03T08:24:10.561848636Z    --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::max_concurrent_for_each<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, cluster::controller_backend::do_bootstrap()::$_0>(absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_inte

After that error, the cluster dies, and even restarting the error persists and clusters don't come back.

It's is possible to do mirroring between Kafka and Redpanda with MM2?

version: 23.2.3
environment: GKE

Thank you

[mmaslankaprv]

Hello, thank you for reporting this, is this happening every time ?

[klinux]

Hello @mmaslankaprv

Yes, I tried some times and the error persist, after increase memory it was possible to use mirrormake to create mirror between kafka and redpanda, but the error is crazy, the cluster simply dead and don't come back even after restarts.

[mmaslankaprv]

would it be possible to get full logs from the failed run ?

[klinux]

Here the all stack

INFO  2023-08-03 08:24:10,552 [shard 0] raft - [group_id:242, {kafka/commerce-event-invoice-created/0}] consensus.cc:1317 - Starting with voted_for {id: {-2147483648}, revision: {-9223372036854775808}} term 0 initial_state true
2023-08-03T08:24:10.561437688Z ERROR 2023-08-03 08:24:10,561 [shard 0] assert - Assert failure: (/var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0f5ff99c2b18b4005-1/redpanda/redpanda/src/v/raft/heartbeat_manager.cc:346) 'ret.second' double registration of group: {kafka/commerce-event-invoice-paid/1}:1001
2023-08-03T08:24:10.561745682Z ERROR 2023-08-03 08:24:10,561 [shard 0] assert - Backtrace below:
2023-08-03T08:24:10.561774236Z 0x6fd29b3 0x2bb67bb 0x2bb1886 0x2b3717e 0x4526568 0x453166a 0x6d1980f 0x6d1ceb1 0x6d1a076 0x6c1eb40 0x6c1cf38 0x213f845 0x701380b /opt/redpanda/lib/libc.so.6+0x2d58f /opt/redpanda/lib/libc.so.6+0x2d648 0x21394a4
   --------
2023-08-03T08:24:10.561790593Z    seastar::internal::coroutine_traits_base<seastar::lw_shared_ptr<raft::consensus>>::promise_type
2023-08-03T08:24:10.561798333Z    --------
2023-08-03T08:24:10.561806971Z    seastar::internal::coroutine_traits_base<std::__1::error_code>::promise_type
   --------
2023-08-03T08:24:10.561842002Z    seastar::internal::coroutine_traits_base<void>::promise_type
2023-08-03T08:24:10.561848636Z    --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::max_concurrent_for_each<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, cluster::controller_backend::do_bootstrap()::$_0>(absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_inte
rnal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, unsigned long, cluster::controller_backend::do_bootstrap()::$_0&&)::'lambda'(seastar::future<void> seastar::max_concurrent_for_each<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pai
r<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, cluster::controller_backend::do_bootstrap()::$_0>(absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_ba
ckend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, unsigned long, cluster::controller_backend::do_bootstrap()::$_0&&)::state&)::operator()(seastar::future<void> seastar::max_concurrent_for_each<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_202206
23::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, cluster::controller_backend::do_bootstrap()::$_0>(absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<mode
l::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, unsigned long, cluster::controller_backend::do_bootstrap()::$_0&&)::state&) const::'lambda0'()::operator()() const::'lambda'()::operator()()::'lambda'(seastar::future<void>), seastar::futurize<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>>::type seastar::future<void>::then_wrapped_nrvo<void, seastar::future<void> seastar::max_concurrent_for_each<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model
::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, cluster::controller_backend::do_bootstrap()::$_0>(absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::
delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, unsigned long, cluster::controller_backend::do_bootstrap()::$_0&&)::'lambda'(seastar::future<void> seastar::max_concurrent_for_each<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_inte
rnal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, cluster::controller_backend::do_bootstrap()::$_0>(absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1:
:allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, unsigned long, cluster::controller_backend::do_bootstrap()::$_0&&)::state&)::operator()(seastar::future<void> seastar::max_concurrent_for_each<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<mode
l::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, cluster::controller_backend::do_bootstrap()::$_0>(absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadat
a>>>*>, unsigned long, cluster::controller_backend::do_bootstrap()::$_0&&)::state&) const::'lambda0'()::operator()() const::'lambda'()::operator()()::'lambda'(seastar::future<void>)>(absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void> seastar::max_concurrent_for_each<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_ite
rator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, cluster::controller_backend::do_bootstrap()::$_0>(absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std:
:__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, unsigned long, cluster::controller_backend::do_bootstrap()::$_0&&)::'lambda'(seastar::future<void> seastar::max_concurrent_for_each<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::de
que<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, cluster::controller_backend::do_bootstrap()::$_0>(absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, unsigned long, clu
ster::controller_backend::do_bootstrap()::$_0&&)::state&)::operator()(seastar::future<void> seastar::max_concurrent_for_each<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, cluster::controller_backend::do_bootstrap()::$_0>(absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::_
_1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::deque<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, unsigned long, cluster::controller_backend::do_bootstrap()::$_0&&)::state&) const::'lambda0'()::operator()() const::'lambda'()::operator()()::'lambda'(seastar::future<void>)&, seastar::future_state<seastar::internal::monostate>&&), void>

[mmaslankaprv]

Would it be possible for you to send us a content of <redpanda_data_directory>/redpnada/controller directory please

[emaxerrno]

@mmaslankaprv - we should make this part of rpk debug bundle so we ask users to only use one command. cc: @r-vasquez we should think about adding the controller dir to rpk debug bundle w/ a flag rpk debug bundle --with-controller-directory

[r-vasquez]

It is part of the debug bundle now by default 😄.

Please keep in mind that we trim the directory if the size is bigger than 20MB but you can change that with the --controller-logs-size-limit

[emaxerrno]

@r-vasquez amazing 🙌🏽 - @mmaslankaprv maybe we ask @klinux to run rpk debug bundle if he's on a later release

[mmaslankaprv]

@klinux can you run rpk debug bundle on a failing cluster ?

[klinux]

@mmaslankaprv after increase memory the cluster is running now, but I can recreate the cluster with the parameters that I was trying and configure mm2 with the new cluster and try debug in new cluster. I'm back with the information

[mmaslankaprv]

thank you