Bug 1344682

Summary: packstack failed to install with nova network
Product: Red Hat OpenStack Reporter: Attila Fazekas <afazekas>
Component: openstack-puppet-modulesAssignee: Emilien Macchi <emacchi>
Status: CLOSED ERRATA QA Contact: Prasanth Anbalagan <panbalag>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0 (Mitaka)CC: aortega, apevec, ichavero, jjoyce, jpena, jschluet, panbalag, srevivo
Target Milestone: gaKeywords: Automation, AutomationBlocker, Triaged
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-puppet-modules-8.1.1-1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-11 12:25:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Attila Fazekas 2016-06-10 10:42:00 UTC 
Description of problem:
192.168.1.13_nova.pp:                             [ ERROR ]
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 192.168.1.13_nova.pp
Error: Execution of '/usr/bin/nova network-create novanetwork --fixed-range-v4 192.168.32.0/22' returned 1: ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
You will find full trace in log /var/tmp/packstack/20160610-061323-blmaSI/manifests/192.168.1.13_nova.pp.log


Packstack fails to configure the network because the nova-network service is not active when the request made.

Retrying the command in a loop, or waiting for the service to start probably is expected to help.

Version-Release number of selected component (if applicable):
rpm -qa |egrep '(nova|packstack|puppet)'
openstack-nova-api-13.0.0-7.el7ost.noarch
openstack-nova-network-13.0.0-7.el7ost.noarch
python-novaclient-3.3.0-1.el7ost.noarch
openstack-packstack-puppet-8.0.0-1.el7ost.noarch
openstack-nova-console-13.0.0-7.el7ost.noarch
openstack-nova-compute-13.0.0-7.el7ost.noarch
openstack-puppet-modules-8.0.4-1.el7ost.noarch
openstack-packstack-8.0.0-1.el7ost.noarch
openstack-nova-cert-13.0.0-7.el7ost.noarch
openstack-nova-novncproxy-13.0.0-7.el7ost.noarch
puppet-3.6.2-2.el7.noarch
openstack-nova-conductor-13.0.0-7.el7ost.noarch
openstack-nova-common-13.0.0-7.el7ost.noarch
openstack-nova-scheduler-13.0.0-7.el7ost.noarch
python-nova-13.0.0-7.el7ost.noarch


How reproducible:
In my current env it is reproducible.  (m1.xlarge)

Steps to Reproduce:
1. try to install openstack with nova network 


The first log line in the nova-network.log service (watch the time):
2016-06-10 06:22:25.456 8898 WARNING oslo_reports.guru_meditation_report [-] Guru mediation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.

nova-api.log:
2016-06-10 06:21:21.365 6663 DEBUG oslo.messaging._drivers.impl_rabbit [req-7c31a405-29ce-4169-93ac-54302a41cfa8 70299aae526d4cbb976b3aa2520585f3 b01e0d5af292467e8db39bfe16b65e48 - - -] Connected to AMQP server 
on 192.168.1.13:5672 via [amqp] client __init__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:561
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions [req-7c31a405-29ce-4169-93ac-54302a41cfa8 70299aae526d4cbb976b3aa2520585f3 b01e0d5af292467e8db39bfe16b65e48 - - -] Unexpected exception in API method
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/networks.py", line 148, in create
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions     network = self.network_api.create(context, **params)[0]
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 49, in wrapped
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions     return func(self, context, *args, **kwargs)
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 103, in create
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions     return self.network_rpcapi.create_networks(context, **kwargs)
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 140, in create_networks
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions     return self.client.call(ctxt, 'create_networks', **kwargs)
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 413, in call
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions     return self.prepare().call(ctxt, method, **kwargs)
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions     retry=self.retry)
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions     retry=retry)
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions     result = self._waiter.wait(msg_id, timeout)
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions     message = self.waiters.get(msg_id, timeout=timeout)
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions     'to message ID %s' % msg_id)
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID 4b885b24e12441dfa35f26fa4b74b9b5
2016-06-10 06:22:21.367 6663 ERROR nova.api.openstack.extensions 
2016-06-10 06:22:21.369 6663 INFO nova.api.openstack.wsgi [req-7c31a405-29ce-4169-93ac-54302a41cfa8 70299aae526d4cbb976b3aa2520585f3 b01e0d5af292467e8db39bfe16b65e48 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
2016-06-10 06:22:21.369 6663 DEBUG nova.api.openstack.wsgi [req-7c31a405-29ce-4169-93ac-54302a41cfa8 70299aae526d4cbb976b3aa2520585f3 b01e0d5af292467e8db39bfe16b65e48 - - -] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1070



Workaround:
If you rerun packstack , it will work.
Comment 3 Alan Pevec 2016-06-16 13:48:57 UTC 
Ivan, there seems to be a race which should be avoided with proper deps in the puppet manifests?
Comment 4 Javier Peña 2016-06-20 16:27:03 UTC 
I think this is related to https://review.openstack.org/318195 and its backports to Mitaka and Liberty
Comment 5 Ivan Chavero 2016-06-20 22:22:48 UTC 
The problem addressed in this bug does not appear anymore

But i got this ther error while running prescript

ERROR : Error appeared during Puppet run: 192.168.100.143_provision_bridge.pp
Error: Could not find data item CONFIG_NEUTRON_KS_PW in any Hiera data file and no default supplied at /var/tmp/packstack/4a4de68e55a54df384d2a49f8ea6478a/manifests/192.168.100.143_provision_bridge.pp:29 on node rhel7.2

It's because neutron is disabled so we don't have any neutron stuff in the hiera file.

I think this bug should be closed and create a new bug for this new problem, what do you think Alan?
Comment 6 Alan Pevec 2016-06-20 22:37:56 UTC 
Yes, reported issue should've been fixed by OPM 8.1.1 update, as suggested in comment 4:
https://github.com/redhat-openstack/openstack-puppet-modules/commit/49ff3ec82b63880ffcc0bab0e1c0c32f1a6fe71a
Comment 8 Prasanth Anbalagan 2016-07-05 21:31:23 UTC 
Verified packstack installation with following version

[root@serverA ~(keystone_admin)]# rpm -qa |egrep '(nova|packstack|puppet)'
puppet-3.6.2-3.el7.noarch
python-nova-13.1.0-1.el7ost.noarch
python-novaclient-3.3.0-1.el7ost.noarch
openstack-nova-common-13.1.0-1.el7ost.noarch
openstack-nova-network-13.1.0-1.el7ost.noarch
openstack-nova-compute-13.1.0-1.el7ost.noarch
openstack-nova-novncproxy-13.1.0-1.el7ost.noarch
openstack-packstack-8.0.0-3.el7ost.noarch
openstack-nova-api-13.1.0-1.el7ost.noarch
openstack-nova-console-13.1.0-1.el7ost.noarch
openstack-puppet-modules-8.1.2-1.el7ost.noarch
openstack-nova-conductor-13.1.0-1.el7ost.noarch
openstack-nova-scheduler-13.1.0-1.el7ost.noarch
openstack-packstack-puppet-8.0.0-3.el7ost.noarch
openstack-nova-cert-13.1.0-1.el7ost.noarch
Comment 10 errata-xmlrpc 2016-08-11 12:25:09 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1597.html