Bug 2050624 - Standalone deployment on scenario07 failing to start rabbitmq_wait_bundle container, rabbitmq crash with error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}
Summary: Standalone deployment on scenario07 failing to start rabbitmq_wait_bundle con...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: erlang
Version: 17.0 (Wallaby)
Hardware: x86_64
OS: Linux
medium
urgent
Target Milestone: ---
: ---
Assignee: Peter Lemenkov
QA Contact: Arik Chernetsky
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-02-04 11:04 UTC by Sandeep Yadav
Modified: 2025-01-17 19:53 UTC (History)
6 users (show)

Fixed In Version: erlang-24.3.4.2-1.el9ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-09-21 12:18:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github erlang otp issues 5762 0 None closed epmd: don't fail if non-critical interfaces fail to bind 2022-06-22 05:46:18 UTC
Red Hat Issue Tracker OSP-12465 0 None None None 2022-02-04 11:07:04 UTC
Red Hat Product Errata RHEA-2022:6543 0 None None None 2022-09-21 12:19:23 UTC

Description Sandeep Yadav 2022-02-04 11:04:47 UTC
Description of problem:

We are trying RHOSP17 on RHEL9,  Standalone deployment on scenario07 failing to start rabbitmq_wait_bundle container, rabbitmq crash with error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}


Version-Release number of selected component (if applicable):

RHOSP17 (on RHEL9)


How reproducible:

Everytime

Steps to Reproduce:
1.  Deploy standalone

Actual results:

Deployment failing with below error:-

standalone_deploy.log:-

~~~
Error: 'rabbitmqctl eval "lists:keymember(rabbit, 1, application:which_applications())." | grep -q true' returned 1 instead of one of [0]
Error: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Exec[rabbitmq-ready]/returns: change from 'notrun' to ['0'] failed: 'rabbitmqctl eval "lists:keymember(rabbit, 1, application:which_applications())." | grep -q true' returned 1 instead of one of [0]
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Rabbitmq_policy[ha-all@/]: Skipping because of failed dependencies
+ rc=6
+ set -e
+ set +ux
2022-02-03 08:44:11.062094 | fa163ea7-a348-2f62-06ce-000000001785 |      FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | standalone | error={"changed": false, "msg": "Failed containers: rabbitmq_wait_bundle"}
2022-02-03 08:44:11.062980 | fa163ea7-a348-2f62-06ce-000000001785 |     TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | standalone | 0:51:31.236681 | 2328.71s
~~~

var/log/containers/rabbitmq/startup_err
~~~
BOOT FAILED
===========
Exception during startup:

error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[[rabbit_prelaunch_750@localhost,shortnames],false,net_sup_dynamic]},permanent,false,1000,supervisor,[erl_distribution]}}}}

    rabbit_prelaunch_dist:duplicate_node_check/1, line 78
    rabbit_prelaunch_dist:setup/1, line 23
    rabbit_prelaunch:do_run/0, line 115
    rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
    supervisor:do_start_child_i/3, line 414
    supervisor:do_start_child/2, line 400
    supervisor:-start_children/2-fun-0-/3, line 384
    supervisor:children_map/4, line 1250

Kernel pid terminated (application_controller) ({application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[[rabbit_prelaunch_750@localhost,shortnames],false,net_sup_dynamic]},permanent,false,1000,supervisor,[erl_distribution]}}}}}},{rabbit_prelaunch_app,start,[normal,[]]}}})

Crash dump is being written to: erl_crash.dump...done
~~~

Expected results:

Deployment should work


Additional info:


Erlang and rabbitmq version we are using:-

~~~
erlang-asn1                      x86_64  24.2.1-1.el9osttrunk     osptrunk-candidate-deps  896 k
rabbitmq-server                  x86_64  3.9.10-2.el9osttrunk     osptrunk-candidate-deps   14 M
~~~

Comment 4 John Eckersberg 2022-03-02 21:07:52 UTC
I have submitted https://github.com/erlang/otp/pull/5762 which makes it possible for epmd to work correctly even if disable_ipv6 = 1.

Comment 5 John Eckersberg 2022-04-21 13:29:58 UTC
(In reply to John Eckersberg from comment #4)
> I have submitted https://github.com/erlang/otp/pull/5762 which makes it
> possible for epmd to work correctly even if disable_ipv6 = 1.

This was merged and will be available for OTP 25, so I am going to re-open this to track getting OTP 25 in for OSP17 once it is added to the rabbitmq support matrix at https://www.rabbitmq.com/which-erlang.html

Comment 6 Peter Lemenkov 2022-05-04 12:34:12 UTC
Addressed in 23.3.4.14 and 24.3.4. I'm going to make a build

Comment 7 Yaniv Kaul 2022-05-11 13:00:31 UTC
(In reply to Peter Lemenkov from comment #6)
> Addressed in 23.3.4.14 and 24.3.4. I'm going to make a build

When is the build going to be available?

Comment 8 Peter Lemenkov 2022-06-27 15:05:02 UTC
(In reply to Yaniv Kaul from comment #7)
> (In reply to Peter Lemenkov from comment #6)
> > Addressed in 23.3.4.14 and 24.3.4. I'm going to make a build
> 
> When is the build going to be available?

I've just tagged erlang-24.3.4.2-1.el9s into messaging9s-rabbitmq-38-testing. Should be available quite soon

Comment 15 errata-xmlrpc 2022-09-21 12:18:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543

Comment 16 Elise Campbell 2022-12-27 06:18:13 UTC Comment hidden (spam)
Comment 17 brandon555 2024-05-30 10:18:42 UTC Comment hidden (spam)
Comment 18 brandon555 2024-05-31 04:59:06 UTC Comment hidden (spam)
Comment 19 Bonnie Church 2024-09-03 05:45:11 UTC Comment hidden (spam)
Comment 20 cegeja6622 2024-09-27 06:51:53 UTC Comment hidden (spam)
Comment 21 Bella 2024-12-17 09:39:49 UTC Comment hidden (spam)

Note You need to log in before you can comment on or make changes to this bug.