Fresh Docker image fails due to feature flags on macOS

[lucasgelfond]

Starting up on the tutorial, have no issue installing rabbitMQ via Homebrew but when I try to run with Docker I get this error about feature flags.

I uninstalled and reinstalled Docker Desktop, currently running 4.31.0 (153195). When I run docker --version I get Docker version 26.1.4, build 5650f9b.

Also uninstalled and reinstalled my version of RabbitMQ locally. I'm up to date, rabbitmqctl --version yields 3.13.3. During local setup, when starting one of the containers, I also ran /opt/homebrew/sbin/rabbitmqctl enable_feature_flag all as per the instal instructions to enable all feature flags. I'm a RabbitMQ newbie but seems like feature flags are used to make breaking changes in ways that allow for older versions to be updated—not sure what's going on (I believe this is a totally fresh install! as is the Docker image!) so figure something might be broken.

The command I'm running specifically is directly from the tutorial — I should be fine just running locally, but wanted to flag because others likely will have:

docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.13-management

BOOT FAILED
===========
Error during startup: {error,
                       {incompatible_feature_flags,
                        {exception,
                         {badmap,rabbitmq_management},
                         [{maps,filter,
                           [#Fun<rabbit_feature_flags.0.3681879>,
                            rabbitmq_management],
                           [{file,"maps.erl"},
                            {line,611},
                            {error_info,#{module => erl_stdlib_errors}}]},
                          {rabbit_feature_flags,mark_as_enabled_locally,2,
                           [{file,"rabbit_feature_flags.erl"},
                            {line,1202}]}]}}}

2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>   crasher:
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     initial call: application_master:init/4
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     pid: <0.253.0>
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     registered_name: []
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     exception exit: {{incompatible_feature_flags,
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                          {exception,
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                              {badmap,rabbitmq_management},
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                              [{maps,filter,
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                                   [#Fun<rabbit_feature_flags.0.3681879>,
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                                    rabbitmq_management],
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                                   [{file,"maps.erl"},
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                                    {line,611},
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                                    {error_info,
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                                        #{module => erl_stdlib_errors}}]},
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                               {rabbit_feature_flags,mark_as_enabled_locally,
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                                   2,
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                                   [{file,"rabbit_feature_flags.erl"},
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                                    {line,1202}]}]}},
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>                      {rabbit,start,[normal,[]]}}
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>       in function  application_master:init/4 (application_master.erl, line 142)
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     ancestors: [<0.252.0>]
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     message_queue_len: 1
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     messages: [{'EXIT',<0.254.0>,normal}]
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     links: [<0.252.0>,<0.44.0>]
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     dictionary: []
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     trap_exit: true
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     status: running
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     heap_size: 2586
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     stack_size: 28
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>     reductions: 216
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>   neighbours:
2024-07-02 00:40:40.382486+00:00 [error] <0.253.0>
2024-07-02 00:40:40.387862+00:00 [notice] <0.44.0> Application rabbit exited with reason: {{incompatible_feature_flags,{exception,{badmap,rabbitmq_management},[{maps,filter,[#Fun<rabbit_feature_flags.0.3681879>,rabbitmq_management],[{file,"maps.erl"},{line,611},{error_info,#{module => erl_stdlib_errors}}]},{rabbit_feature_flags,mark_as_enabled_locally,2,[{file,"rabbit_feature_flags.erl"},{line,1202}]}]}},{rabbit,start,[normal,[]]}}
Runtime terminating during boot (terminating)

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
[os_mon] memory supervisor port (memsup): Erlang has close

[lucasgelfond]

The default image still doesn't work for me so I'll leave this open, but a few things that I did, at least one of which led to the fix:

• cleared a ton of disk space (although, didn't work after doing this so unlikely this was deciding factor)
• downloading the Linux build (per this note on StackOverflow)
• disabling the feature flags that caused issues

The way I ended up doing this was:

• Download rabbitmq image for linux

docker pull --platform linux/arm64/v8 rabbitmq:3.13.3-management

• Create a rabbitmq.conf file with the following:

# rabbitmq.conf
feature_flags.classic_mirrored_queue_version = false
feature_flags.quorum_queue = false
feature_flags.stream_queue = false
feature_flags.implicit_default_bindings = false
feature_flags.virtual_host_metadata = false
feature_flags.maintenance_mode_status = false
feature_flags.user_limits = false
feature_flags.stream_single_active_consumer = false
feature_flags.feature_flags_v2 = false
feature_flags.direct_exchange_routing_v2 = false
feature_flags.listener_records_in_ets = false
feature_flags.tracking_records_in_ets = false
feature_flags.classic_queue_type_delivery_support = false
feature_flags.empty_basic_get_metric = false
feature_flags.drop_unroutable_metric = false

Run:

docker run -d --name rabbitmq \
  --platform linux/arm64/v8 \
  -p 5672:5672 -p 15672:15672 \
  -v /path/to/your/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf \
  rabbitmq:3.13.3-management

Hit a quick docker ps to make sure it's running (it should be!). This should do most of it!

[michaelklishin]

cleared a ton of disk space (although, didn't work after doing this so unlikely this was deciding factor)

If the disk volume used by a RabbitMQ node was completely full at some point, it easily could have resulted in some data never making it to disk.

Team RabbitMQ strongly recommends treating nodes that have run into this condition as nodes that have failing disks: they must be replaced. In development environments this means that they must be stopped, reset and restarted (as brand new blank nodes).

[lucasgelfond]

You had some existing on disk data that was incorrect, rabbitmq_management is not a feature flag but it was listed as such. Without having access to node data directories, we can only guess as to what'd happened.

FYI, disabling feature flags is wrong and is guaranteed to prevent you from upgrading RabbitMQ in the future. The Version Upgradeability table further demonstrates how graduated feature flags become mandatory over time.

Ah, thanks, not sure how I had this. Possibly through initial brew install.

Odd that would've happened, don't think I've used rmq myself before, no clue where it would be.

Thanks for notes re clearing disks and disk space - ended up taking a chance to clear out everything more generally and freed up about 50GB.

Thanks for your help!

[lucasgelfond]

(And apologies for true newbie-dom — seems like this was a reasonable fix, thanks for helping!)

[adamsousa]

Just ran into this exact same error on my M3 Macbook. This issue only started when I changed rabbitmq to arch linux/amd64. Amd64 is required in my project, and i get mismatch archs when attempting to run rabbitmq in docker compose without it.

Only solution for me was disabling all feature flags, that allow it to correctly boot. Not ideal. Looking for better solutions.

[adamsousa]

My error output did point to a specific feature flag: failed to enable: stream_update_config_command

[lukebakken]

Hi @lucasgelfond - what does this statement mean?

Also uninstalled and reinstalled my version of RabbitMQ locally

Did you have RabbitMQ installed by some other means before trying to use docker?

[lucasgelfond]

Sorry, missed this notif somehow! Yeah, I had it via Homebrew!