Bug 1609963

Summary: 'openstack undercloud install' fails during 'Check for unhealthy containers after step 5' : openstack-heat-api-cfn and openstack-heat-api are unhealthy
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-tripleo-heat-templatesAssignee: Emilien Macchi <emacchi>
Status: CLOSED ERRATA QA Contact: Gurenko Alex <agurenko>
Severity: medium Docs Contact:
Priority: medium    
Version: 14.0 (Rocky)CC: ahrechan, aschultz, dbecker, emacchi, jslagle, mburns, mcornea, mlammon, morazi
Target Milestone: betaKeywords: Reopened, Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-9.0.0-0.20180919080945.0rc1.0rc1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-11 11:51:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2018-07-30 21:53:40 UTC
'openstack undercloud install' fails during 'Check for unhealthy containers after step 5' : openstack-heat-api-cfn and openstack-heat-api are unhealthy

Environment:
heat-cfntools-1.2.6-5.el7.noarch
python-heat-agent-docker-cmd-1.6.1-0.20180709100740.fdd6a5f.el7ost.noarch
ansible-role-redhat-subscription-1.0.1-1.el7ost.noarch
openstack-heat-agents-1.6.1-0.20180709100740.fdd6a5f.el7ost.noarch
openstack-heat-common-12.0.0-0.20180604085325.7d878a8.el7ost.noarch
python-tripleoclient-heat-installer-10.3.1-0.20180716131225.300dfb3.el7ost.noarch
python-heat-agent-json-file-1.6.1-0.20180709100740.fdd6a5f.el7ost.noarch
python-heat-agent-apply-config-1.6.1-0.20180709100740.fdd6a5f.el7ost.noarch
ansible-pacemaker-1.0.4-0.20180220234310.0e4d7c0.el7ost.noarch
ansible-role-tripleo-modify-image-0.0.1-0.20180718224444.c2ef865.el7ost.noarch
python-heat-agent-ansible-1.6.1-0.20180709100740.fdd6a5f.el7ost.noarch
openstack-heat-api-12.0.0-0.20180604085325.7d878a8.el7ost.noarch
python2-heatclient-1.16.0-0.20180619123220.f095a0f.el7ost.noarch
python-heat-agent-1.6.1-0.20180709100740.fdd6a5f.el7ost.noarch
python-heat-agent-puppet-1.6.1-0.20180709100740.fdd6a5f.el7ost.noarch
ansible-role-container-registry-0.0.1-0.20180718101745.9fcedac.el7ost.noarch
instack-undercloud-9.1.1-0.20180716115151.bc82d48.el7ost.noarch
openstack-heat-monolith-12.0.0-0.20180604085325.7d878a8.el7ost.noarch
puppet-heat-13.1.1-0.20180719053944.8ef3646.el7ost.noarch
python-heat-agent-hiera-1.6.1-0.20180709100740.fdd6a5f.el7ost.noarch
ansible-2.5.4-1.el7ae.noarch
ansible-tripleo-ipsec-8.1.1-0.20180405121919.325d233.el7ost.noarch
openstack-tripleo-heat-templates-9.0.0-0.20180720154239.959e1d7.el7ost.noarch
openstack-heat-engine-12.0.0-0.20180604085325.7d878a8.el7ost.noarch


Steps to reproduce:
Edit undercloud.conf to include:
[stack@undercloud75 ~]$ cat undercloud.conf 
[DEFAULT]
# Network interface on the Undercloud that will be handling the PXE
# boots and DHCP for Overcloud instances. (string value)
local_interface = eth1
local_ip = 192.168.0.1/24
#undercloud_public_host = 192.168.0.2
#undercloud_admin_host = 192.168.0.3
generate_service_certificate = false
container_images_file=/home/stack/containers-prepare-parameter.yaml
undercloud_ntp_servers=clock.redhat.com
docker_insecure_registries=docker-registry.engineering.redhat.com
# BZ#1608508
undercloud_enable_selinux=False
[ctlplane-subnet]
local_subnet = ctlplane-subnet
cidr = 192.168.0.0/24
dhcp_start = 192.168.0.5
dhcp_end = 192.168.0.24
gateway = 192.168.0.1
inspection_iprange = 192.168.0.100,192.168.0.120
masquerade = true

Note the attempt to use nonSSL undercloud.

Run 'openstack undercloud install'.

Encountered https://bugzilla.redhat.com/show_bug.cgi?id=1609896

Re-ran the command 'openstack undercloud install'

Result:
}

TASK [Check for unhealthy containers after step 5] ****************************************************************************************************************************************************************
ok: [undercloud75]

TASK [Debug output for task which failed: Check for unhealthy containers after step 5] ****************************************************************************************************************************
fatal: [undercloud75]: FAILED! => {
    "failed_when_result": true,
    "outputs.stdout_lines|default([])|union(outputs.stderr_lines|default([]))": [
        "0a29b766d34c        192.168.24.1:8787/rhosp14/openstack-heat-api-cfn:2018-07-26.6   \"kolla_start\"       2 minutes ago       Up 2 minutes (unhealthy)                       heat_api_cfn",
        "a0ea4705e67d        192.168.24.1:8787/rhosp14/openstack-heat-api:2018-07-26.6       \"kolla_start\"       2 minutes ago       Up 2 minutes (unhealthy)                       heat_api"
    ]   
}   

NO MORE HOSTS LEFT ************************************************************************************************************************************************************************************************

PLAY RECAP ********************************************************************************************************************************************************************************************************
undercloud75               : ok=216  changed=58   unreachable=0    failed=1

Not cleaning working directory /home/stack/tripleo-heat-installer-templates
Not cleaning ansible directory /home/stack/undercloud-ansible-Cm4Mv9
Install artifact is located at /home/stack/undercloud-install-20180730214110.tar.bzip2

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Deployment Failed!

ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-yfWWeD



The issue reproduces.

Comment 2 Emilien Macchi 2018-07-30 22:02:55 UTC

*** This bug has been marked as a duplicate of bug 1608903 ***

Comment 3 Marius Cornea 2018-07-31 15:49:30 UTC
Per our discussion on IRC I'm reopening this bug to keep track of the health checks validation removal in https://review.openstack.org/#/c/587006/

Comment 4 Alex Schultz 2018-07-31 18:35:55 UTC
We've reverted this logic upstream, the real issue being that the container health checks were changed but we've seen intermittent problems with container health during the deploy which we'll be working towards understanding resolving in the next release.

Comment 9 Artem Hrechanychenko 2019-01-08 14:15:37 UTC
VERIFIED

openstack-tripleo-heat-templates-9.0.1-0.20181013060907.el7ost.noarch.rpm

Comment 11 errata-xmlrpc 2019-01-11 11:51:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045