Bug 1667439

Summary: Failed to install RHOS-9 undercloud: Starting proxy rabbitmq: cannot bind socket
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: instack-undercloudAssignee: James Slagle <jslagle>
Status: CLOSED DUPLICATE QA Contact: Arik Chernetsky <achernet>
Severity: urgent Docs Contact:
Priority: high    
Version: 9.0 (Mitaka)CC: aschultz, augol, ccamacho, mburns, sgolovat, yprokule
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-21 17:29:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yurii Prokulevych 2019-01-18 14:03:55 UTC
Description of problem:
-----------------------
Undercloud installation fails
...
ESC[mNotice: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user_role[ironic@service]/ensure: createdESC[0m
ESC[1;31mError: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/bin/openstack token issue --format value' returned 1: Unable to establish connection to https://192.168.24.2:13000/v3/auth/tokens (tried 48, for a total of 170 seconds)ESC[0m
ESC[mNotice: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Dependency Keystone_user[admin] has failures: trueESC[0m
ESC[mNotice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'192.168.24.1' (using password: YES)")ESC[0m
ESC[1;31mError: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0]ESC[0m
ESC[1;31mError: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0]ESC[0m
ESC[mNotice: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::end]: Dependency Keystone_user[admin] has failures: trueESC[0m
ESC[1;31mWarning: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::end]: Skipping because of failed dependenciesESC[0m
ESC[mNotice: /Stage[main]/Heat::Deps/Anchor[heat::service::begin]: Dependency Keystone_user[admin] has failures: trueESC[0m
ESC[1;31mWarning: /Stage[main]/Heat::Deps/Anchor[heat::service::begin]: Skipping because of failed dependenciesESC[0m
ESC[mNotice: /Stage[main]/Heat::Deps/Anchor[heat::service::begin]: Triggered 'refresh' from 2 eventsESC[0m
ESC[mNotice: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Dependency Keystone_user[admin] has failures: trueESC[0m
ESC[1;31mWarning: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Skipping because of failed dependenciesESC[0m
ESC[mNotice: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Triggered 'refresh' from 1 eventsESC[0m
ESC[mNotice: /Stage[main]/Heat::Engine/Service[heat-engine]: Dependency Keystone_user[admin] has failures: trueESC[0m
ESC[1;31mWarning: /Stage[main]/Heat::Engine/Service[heat-engine]: Skipping because of failed dependenciesESC[0m
ESC[mNotice: /Stage[main]/Heat::Engine/Service[heat-engine]: Triggered 'refresh' from 1 eventsESC[0m
ESC[mNotice: /Stage[main]/Heat::Api/Service[heat-api]: Dependency Keystone_user[admin] has failures: trueESC[0m
ESC[1;31mWarning: /Stage[main]/Heat::Api/Service[heat-api]: Skipping because of failed dependenciesESC[0m
ESC[mNotice: /Stage[main]/Heat::Api/Service[heat-api]: Triggered 'refresh' from 1 eventsESC[0m
ESC[mNotice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Dependency Keystone_user[admin] has failures: trueESC[0m
ESC[1;31mWarning: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Skipping because of failed dependenciesESC[0m
ESC[mNotice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 3 eventsESC[0m
ESC[mNotice: Finished catalog run in 561.29 secondsESC[0m
+ rc=6
+ set -e
+ echo 'puppet apply exited with exit code 6'
puppet apply exited with exit code 6
+ '[' 6 '!=' 2 -a 6 '!=' 0 ']'
+ exit 6
[2019-01-18 08:57:36,897] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 6]

[2019-01-18 08:57:36,898] (os-refresh-config) [ERROR] Aborting...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 864, in install
    _run_orc(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 741, in _run_orc
    _run_live_command(args, instack_env, 'os-refresh-config')
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 412, in _run_live_command
    raise RuntimeError('%s failed. See log for details.' % name)
RuntimeError: os-refresh-config failed. See log for details.

Problem is that haproxy service fails to start:
-----------------------------------------------
[root@undercloud-0 ~]# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2019-01-18 08:52:32 EST; 8min ago
 Main PID: 2348 (code=exited, status=1/FAILURE)

Jan 18 08:52:32 undercloud-0.redhat.local haproxy[2349]: Proxy glance_api started.
Jan 18 08:52:32 undercloud-0.redhat.local haproxy[2349]: Proxy glance_registry started.
Jan 18 08:52:32 undercloud-0.redhat.local haproxy[2349]: Proxy haproxy.stats started.
Jan 18 08:52:32 undercloud-0.redhat.local haproxy[2349]: Proxy heat_api started.
Jan 18 08:52:32 undercloud-0.redhat.local haproxy[2349]: Proxy ironic started.
Jan 18 08:52:32 undercloud-0.redhat.local haproxy-systemd-wrapper[2348]: [ALERT] 017/085232 (2349) : Starting proxy rabbitmq: cannot bind socket [192.168.24.3:5672]
Jan 18 08:52:32 undercloud-0.redhat.local haproxy-systemd-wrapper[2348]: haproxy-systemd-wrapper: exit, haproxy RC=1
Jan 18 08:52:32 undercloud-0.redhat.local systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE
Jan 18 08:52:32 undercloud-0.redhat.local systemd[1]: Unit haproxy.service entered failed state.
Jan 18 08:52:32 undercloud-0.redhat.local systemd[1]: haproxy.service failed.


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
haproxy-1.5.18-8.el7.x86_64
openstack-tripleo-heat-templates-2.0.0-69.el7ost.noarch
openstack-heat-templates-0-0.3.96a0b0bgit.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-69.el7ost.noarch
rabbitmq-server-3.6.3-6.el7ost.noarch
instack-undercloud-4.0.0-20.el7ost.noarch
instack-0.0.8-3.el7ost.noarch

How reproducible:
-----------------


Steps to Reproduce:
-------------------
1. Setup 2018-11-15.2(director) and 2018-11-15.1(core) repos
2. Prepare undercloud.conf
3. Start installation

Actual results:
---------------
Undercloud installation fails


Expected results:
-----------------
Undercloud is installed

Comment 1 Alex Schultz 2019-01-18 15:43:42 UTC
Please provide an sosreport with the full logs.

Comment 3 Alex Schultz 2019-01-21 17:29:15 UTC

*** This bug has been marked as a duplicate of bug 1651355 ***