Bug 1628908

Summary: [UPGRADES][14] OSP 13 to 14 upgrade fails due to openvswitch package rename in ControllerAllNodesValidationDeployment' step
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: openstack-tripleo-heat-templatesAssignee: Brent Eagles <beagles>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: amuller, augol, bcafarel, beagles, bhaley, chrisw, dbecker, jjoyce, mbracho, mburns, morazi, nyechiel, ralongi, sathlang, skaplons, srevivo, tvignaud, yprokule
Target Milestone: betaKeywords: Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-9.0.1-0.20181013060892 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-11 11:53:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yurii Prokulevych 2018-09-14 10:16:38 UTC
Description of problem:
-----------------------
Upgrade controllers from RHOS-13 to RHOS-14 failed while running 'Run deployment ControllerAllNodesValidationDeployment' step. It failed to reach IP address from ExternalNetwork/

"[2018-09-13 15:07:14,299] (heat-config) [INFO] deploy_signal_transport=NO_SIGNAL", ',
"[2018-09-13 15:07:14,299] (heat-config) [DEBUG] Running /var/lib/heat-config/heat-config-script/db67c5d6-e65b-4e5a-ba84-8cafec50496d", ',
"[2018-09-13 15:17:51,437] (heat-config) [INFO] Trying to ping 10.0.0.112 for local network 10.0.0.0/24.", ',
"Ping to 10.0.0.112 failed. Retrying...", ',
"Ping to 10.0.0.112 failed. Retrying...", ',
"Ping to 10.0.0.112 failed. Retrying...", ',
"Ping to 10.0.0.112 failed. Retrying...", ',
"Ping to 10.0.0.112 failed. Retrying...", ',
"Ping to 10.0.0.112 failed. Retrying...", ',
"Ping to 10.0.0.112 failed. Retrying...", ',
"Ping to 10.0.0.112 failed. Retrying...", ',
"Ping to 10.0.0.112 failed. Retrying...", ',
"Ping to 10.0.0.112 failed. Retrying...", ',
"FAILURE", ',
"[2018-09-13 15:17:51,437] (heat-config) [DEBUG] 10.0.0.112 is not pingable. Local Network: 10.0.0.0/24", ',
"[2018-09-13 15:17:51,437] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-script/db67c5d6-e65b-4e5a-ba84-8cafec50496d. [1]", ',


Also other IPs on br-isolated bridge were not reachable between controllers.

After reboot connectivity got restored.

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-neutron-common-12.0.3-2.el7ost.noarch
openstack-neutron-fwaas-12.0.1-0.20180409052822.384366b.el7ost.noarch
openstack-neutron-openvswitch-12.0.3-2.el7ost.noarch
puppet-neutron-12.4.1-1.3aa3109git.el7ost.noarch
python-neutron-12.0.3-2.el7ost.noarch
openstack-neutron-lbaas-12.0.1-0.20180424200349.cdbf25c.el7ost.noarch
python-neutron-fwaas-12.0.1-0.20180409052822.384366b.el7ost.noarch
openstack-neutron-12.0.3-2.el7ost.noarch
openstack-neutron-ml2-12.0.3-2.el7ost.noarch
python-neutron-lbaas-12.0.1-0.20180424200349.cdbf25c.el7ost.noarch
python2-neutronclient-6.7.0-1.el7ost.noarch
python2-neutron-lib-1.13.0-1.el7ost.noarch

openstack-neutron-openvswitch-12.0.3-2.el7ost.noarch
openvswitch-2.9.0-56.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-5.el7fdp.noarch
puppet-vswitch-8.4.0-0.20180329150818.fd94e91.el7ost.noarch
python-openvswitch-2.9.0-56.el7fdp.noarch

Steps to Reproduce:
-------------------
1. Upgrade RHOS-13 uc to RHOS-14
2. Prepare images/repos for RHOS-14 overcloud
3. Prepare upgrade playbooks
4. Run upgrade of controller nodes:
    openstack overcloud upgrade run \
        --stack overcloud \
        --roles Controller --playbook all

Actual results:
---------------
Post upgrade playbook failed

Expected results:
-----------------
Controllers upgrade succeeds

Additional info:
----------------
Virtual setup: 3controllers + 2computes + 3ceph

Comment 2 Assaf Muller 2018-09-14 12:48:44 UTC
Any initial troubleshooting?

Comment 3 Yurii Prokulevych 2018-09-14 12:57:16 UTC
nope

Comment 4 Brian Haley 2018-09-17 13:45:02 UTC
Hi Yuri,

Can you reproduce this?  If so can we login and look around once you do?  It would be the easiest way to diagnose what the issue might be.  Thanks.

Comment 25 Bernard Cafarelli 2018-11-12 13:34:01 UTC
Master patch merged, I created backport at https://review.openstack.org/#/c/617335/

Comment 50 Yurii Prokulevych 2018-12-17 14:42:21 UTC
Verified with openstack-tripleo-heat-templates-9.0.1-0.20181013060906.el7ost.noarch

On overcloud:
-------------
openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch
openvswitch2.10-2.10.0-28.el7fdp.1.x86_64
python-rhosp-openvswitch-2.10-0.1.el7ost.noarch
python-openvswitch2.10-2.10.0-28.el7fdp.1.x86_64
rhosp-openvswitch-2.10-0.1.el7ost.noarch
openstack-neutron-openvswitch-13.0.2-0.20180929022427.c7f970c.el7ost.noarch

[root@controller-0 ~]# docker ps | grep openvsw
629eacd0c45a        192.168.24.1:8787/rhosp14/openstack-neutron-openvswitch-agent:2018-12-14.1   "kolla_start"            2 hours ago         Up 2 hours (healthy)                          neutron_ovs_agent

Comment 53 errata-xmlrpc 2019-01-11 11:53:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045