Bug 2050624

Summary: Standalone deployment on scenario07 failing to start rabbitmq_wait_bundle container, rabbitmq crash with error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}
Product: Red Hat OpenStack Reporter: Sandeep Yadav <sandyada>
Component: erlangAssignee: Peter Lemenkov <plemenko>
Status: CLOSED ERRATA QA Contact: Arik Chernetsky <achernet>
Severity: urgent Docs Contact:
Priority: medium    
Version: 17.0 (Wallaby)CC: apevec, jeckersb, jschluet, lhh, lmiccini, plemenko
Target Milestone: ---Keywords: Reopened, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: erlang-24.3.4.2-1.el9ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-21 12:18:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sandeep Yadav 2022-02-04 11:04:47 UTC
Description of problem:

We are trying RHOSP17 on RHEL9,  Standalone deployment on scenario07 failing to start rabbitmq_wait_bundle container, rabbitmq crash with error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}


Version-Release number of selected component (if applicable):

RHOSP17 (on RHEL9)


How reproducible:

Everytime

Steps to Reproduce:
1.  Deploy standalone

Actual results:

Deployment failing with below error:-

standalone_deploy.log:-

~~~
Error: 'rabbitmqctl eval "lists:keymember(rabbit, 1, application:which_applications())." | grep -q true' returned 1 instead of one of [0]
Error: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Exec[rabbitmq-ready]/returns: change from 'notrun' to ['0'] failed: 'rabbitmqctl eval "lists:keymember(rabbit, 1, application:which_applications())." | grep -q true' returned 1 instead of one of [0]
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Rabbitmq_policy[ha-all@/]: Skipping because of failed dependencies
+ rc=6
+ set -e
+ set +ux
2022-02-03 08:44:11.062094 | fa163ea7-a348-2f62-06ce-000000001785 |      FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | standalone | error={"changed": false, "msg": "Failed containers: rabbitmq_wait_bundle"}
2022-02-03 08:44:11.062980 | fa163ea7-a348-2f62-06ce-000000001785 |     TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | standalone | 0:51:31.236681 | 2328.71s
~~~

var/log/containers/rabbitmq/startup_err
~~~
BOOT FAILED
===========
Exception during startup:

error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[[rabbit_prelaunch_750@localhost,shortnames],false,net_sup_dynamic]},permanent,false,1000,supervisor,[erl_distribution]}}}}

    rabbit_prelaunch_dist:duplicate_node_check/1, line 78
    rabbit_prelaunch_dist:setup/1, line 23
    rabbit_prelaunch:do_run/0, line 115
    rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
    supervisor:do_start_child_i/3, line 414
    supervisor:do_start_child/2, line 400
    supervisor:-start_children/2-fun-0-/3, line 384
    supervisor:children_map/4, line 1250

Kernel pid terminated (application_controller) ({application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[[rabbit_prelaunch_750@localhost,shortnames],false,net_sup_dynamic]},permanent,false,1000,supervisor,[erl_distribution]}}}}}},{rabbit_prelaunch_app,start,[normal,[]]}}})

Crash dump is being written to: erl_crash.dump...done
~~~

Expected results:

Deployment should work


Additional info:


Erlang and rabbitmq version we are using:-

~~~
erlang-asn1                      x86_64  24.2.1-1.el9osttrunk     osptrunk-candidate-deps  896 k
rabbitmq-server                  x86_64  3.9.10-2.el9osttrunk     osptrunk-candidate-deps   14 M
~~~

Comment 4 John Eckersberg 2022-03-02 21:07:52 UTC
I have submitted https://github.com/erlang/otp/pull/5762 which makes it possible for epmd to work correctly even if disable_ipv6 = 1.

Comment 5 John Eckersberg 2022-04-21 13:29:58 UTC
(In reply to John Eckersberg from comment #4)
> I have submitted https://github.com/erlang/otp/pull/5762 which makes it
> possible for epmd to work correctly even if disable_ipv6 = 1.

This was merged and will be available for OTP 25, so I am going to re-open this to track getting OTP 25 in for OSP17 once it is added to the rabbitmq support matrix at https://www.rabbitmq.com/which-erlang.html

Comment 6 Peter Lemenkov 2022-05-04 12:34:12 UTC
Addressed in 23.3.4.14 and 24.3.4. I'm going to make a build

Comment 7 Yaniv Kaul 2022-05-11 13:00:31 UTC
(In reply to Peter Lemenkov from comment #6)
> Addressed in 23.3.4.14 and 24.3.4. I'm going to make a build

When is the build going to be available?

Comment 8 Peter Lemenkov 2022-06-27 15:05:02 UTC
(In reply to Yaniv Kaul from comment #7)
> (In reply to Peter Lemenkov from comment #6)
> > Addressed in 23.3.4.14 and 24.3.4. I'm going to make a build
> 
> When is the build going to be available?

I've just tagged erlang-24.3.4.2-1.el9s into messaging9s-rabbitmq-38-testing. Should be available quite soon

Comment 15 errata-xmlrpc 2022-09-21 12:18:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543

Comment 16 Elise Campbell 2022-12-27 06:18:13 UTC Comment hidden (spam)
Comment 17 brandon555 2024-05-30 10:18:42 UTC Comment hidden (spam)
Comment 18 brandon555 2024-05-31 04:59:06 UTC Comment hidden (spam)
Comment 19 Bonnie Church 2024-09-03 05:45:11 UTC Comment hidden (spam)
Comment 20 cegeja6622 2024-09-27 06:51:53 UTC Comment hidden (spam)
Comment 21 Bella 2024-12-17 09:39:49 UTC Comment hidden (spam)