Bug 1322065

Summary: rhel-osp-director: After completing the upgrade from 7.3 ->8.0, unable to launch an overcloud instance.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-neutronAssignee: Ihar Hrachyshka <ihrachys>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: high Docs Contact:
Priority: high    
Version: 8.0 (Liberty)CC: amuller, chrisw, dbecker, jschluet, mandreou, mburns, morazi, nyechiel, oblaut, rhel-osp-director-maint, sasha, therve, yeylon
Target Milestone: ga   
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-7.0.1-15.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-07 21:36:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
nova-compute log crom a compute
none
Patch for liberty backport none

Description Alexander Chuzhoy 2016-03-29 17:18:56 UTC
rhel-osp-director: After completing the upgrade from 7.3 ->8.0, unable to launch an overcloud instance. 


Environment:
openstack-tripleo-heat-templates-kilo-0.8.14-1.el7ost.noarch
openstack-nova-scheduler-12.0.2-4.el7ost.noarch
openstack-nova-conductor-12.0.2-4.el7ost.noarch
openstack-nova-compute-12.0.2-4.el7ost.noarch
openstack-tripleo-heat-templates-0.8.14-1.el7ost.noarch
python-novaclient-3.1.0-2.el7ost.noarch
openstack-nova-common-12.0.2-4.el7ost.noarch
openstack-nova-novncproxy-12.0.2-4.el7ost.noarch
python-nova-12.0.2-4.el7ost.noarch
openstack-nova-cert-12.0.2-4.el7ost.noarch
openstack-nova-console-12.0.2-4.el7ost.noarch
instack-undercloud-2.2.7-1.el7ost.noarch
openstack-nova-api-12.0.2-4.el7ost.noarch



Steps to reproduce:
1. Deploy overcloud 7.3 HA + 1 compute + 1 ceph.
2. Populate the overcloud with instance.
3. Upgrade the setup to 8.0
4. Attempt to launch another instance.

Result:

The instance ends up with status error:
Checking the instance itself:
{"message": "No valid host was found. There are not enough hosts available.

nova-compute appears down in "nova service-list".
On compute, the openstack-nova-compute.service is ActiveState=active







Expected result:
The instance should be launched with no issues after the upgrade.

Comment 1 Alexander Chuzhoy 2016-03-29 17:19:53 UTC
Created attachment 1141351 [details]
nova-compute log crom a compute

Comment 2 Thomas Hervé 2016-03-29 19:11:30 UTC
The issue talking to Neutron looks like https://bugs.launchpad.net/neutron/+bug/1509312

Comment 3 Alexander Chuzhoy 2016-03-29 20:05:17 UTC
Launching an instance without attaching it to networks works.

Comment 4 Thomas Hervé 2016-03-29 20:31:05 UTC
Created attachment 1141441 [details]
Patch for liberty backport

Joining a patch doing backport of https://review.openstack.org/294132/ for liberty

Comment 5 Marios Andreou 2016-03-30 12:07:40 UTC
sasha did you include/try the latest fix from emilien to restart openstack-neutron-agent on the compute node? I see in your attached nova-compute logs there is also this: 

2016-03-29 16:54:45.226 19103 ERROR oslo.messaging._drivers.impl_rabbit [req-8ba77eba-915f-4d42-92dc-70b6fc4207fc - - - - -] AMQP server on 192.168.100.15:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds.

(the agent didn't restart and so didn't get the new password it now has in /etc/neutron/neutron.conf )

I was able to run the tripleo-ci pingtest locally after upgrade ok

Comment 6 Ihar Hrachyshka 2016-03-30 12:16:17 UTC
Mario, I think you confuse issues. It's not related to any config changes.

Comment 7 Marios Andreou 2016-03-30 13:34:19 UTC
(In reply to Ihar Hrachyshka from comment #6)
> Mario, I think you confuse issues. It's not related to any config changes.

thanks ihar I was wondering if it was related to the (puppet-neutron) workaround we carry for the neutron-openvswitch-agent restart on config change at  https://review.openstack.org/#/c/298946/

but after the call just now it seems not

Comment 11 Alexander Chuzhoy 2016-04-05 20:15:27 UTC
Verified:

Environment:
openstack-neutron-7.0.1-15.el7ost.noarch


Was able to launch an instance with network connectivity post upgrade.

Comment 12 errata-xmlrpc 2016-04-07 21:36:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0603.html