Bug 1614817

Summary: Error: Systemd start for neutron-destroy-patch-ports failed! No module named compat
Product: Red Hat OpenStack Reporter: Filip Hubík <fhubik>
Component: openvswitchAssignee: Flavio Leitner <fleitner>
Status: CLOSED ERRATA QA Contact: Filip Hubík <fhubik>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: apevec, bcafarel, bhaley, chrisw, fhubik, fleitner, jschluet, mburns, rhos-maint, srevivo, tredaelli, tvignaud, twilson
Target Milestone: betaKeywords: Automation, AutomationBlocker, Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-11 11:51:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/tmp/packstack/20180810-085725-cBTrNd/manifests/172.16.1.19_network.pp.log
none
/var/tmp/packstack/20180810-085725-cBTrNd/openstack-setup.log
none
Add compat module to ovs package in setup.py none

Description Filip Hubík 2018-08-10 13:35:28 UTC 
Created attachment 1475020 [details]
/var/tmp/packstack/20180810-085725-cBTrNd/manifests/172.16.1.19_network.pp.log

Description of problem:

Packstack deployment fails with error:
...
Preparing Puppet manifests                           [ DONE ]
Copying Puppet modules and manifests                 [ DONE ]
Applying 172.16.1.19_controller.pp
172.16.1.19_controller.pp:                           [ DONE ]
Applying 172.16.1.19_network.pp
172.16.1.19_network.pp:                           [ ERROR ]
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 172.16.1.19_network.pp
Error: Systemd start for neutron-destroy-patch-ports failed!
You will find full trace in log /var/tmp/packstack/20180810-085725-cBTrNd/manifests/172.16.1.19_network.pp.log
Please check log file /var/tmp/packstack/20180810-085725-cBTrNd/openstack-setup.log for more information
Additional information:
 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.
 * Warning: NetworkManager is active on 172.16.1.19. OpenStack networking currently does not work on systems that have the Network Manager service enabled.
 * File /root/keystonerc_admin has been created on OpenStack client host 172.16.1.19. To use the command line tools you need to source the file.
 * To access the OpenStack Dashboard browse to http://172.16.1.19/dashboard .
Please, find your login credentials stored in the keystonerc_admin in your home directory.
 * Note temporary directory /var/tmp/packstack/0ebbed1dbe2e4207991645dacbcf5238 on host 172.16.1.19 was not deleted for debugging purposes.

MSG:

non-zero return code

Error(s) in /var/tmp/packstack/20180810-085725-cBTrNd/manifests/172.16.1.19_network.pp.log:
...
debug: Runing journalctl command to get logs for systemd start failure: journalctl -n 50 --since '5 minutes ago' -u neutron-destroy-patch-ports --no-pager
Debug: Executing: 'journalctl -n 50 --since '5 minutes ago' -u neutron-destroy-patch-ports --no-pager'
Error: Systemd start for neutron-destroy-patch-ports failed!
journalctl log for neutron-destroy-patch-ports:
-- Logs begin at Fri 2018-08-10 08:51:57 EDT, end at Fri 2018-08-10 09:14:56 EDT. --
systemd[1]: Starting OpenStack Neutron Destroy Patch Ports...
python[5110]: /usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.21.1) or chardet (2.2.1) doesn't match a supported version!
python[5110]: RequestsDependencyWarning)
python[5110]: PMD: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
python[5110]: PMD: net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)
python[5110]: PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
python[5110]: PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4)
python[5110]: Traceback (most recent call last):
python[5110]: File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
python[5110]: "__main__", fname, loader, pkg_name)
python[5110]: File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
python[5110]: exec code in run_globals
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/cmd/destroy_patch_ports.py", line 83, in <module>
python[5110]: main()
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/cmd/destroy_patch_ports.py", line 78, in main
python[5110]: port_cleaner = PatchPortCleaner(cfg.CONF)
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/cmd/destroy_patch_ports.py", line 44, in __init__
python[5110]: for bridge in mappings.values()]
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 218, in __init__
python[5110]: super(OVSBridge, self).__init__()
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 117, in __init__
python[5110]: self.ovsdb = ovsdb_api.from_config(self)
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/api.py", line 30, in from_config
python[5110]: ovsdb_api.interface_map[iface_name or cfg.CONF.OVS.ovsdb_interface])
python[5110]: File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 73, in import_module
python[5110]: __import__(import_str)
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 17, in <module>
python[5110]: from ovsdbapp.backend.ovs_idl import command
python[5110]: File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 16, in <module>
python[5110]: from ovsdbapp.backend.ovs_idl import command as cmd
python[5110]: File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 21, in <module>
python[5110]: from ovsdbapp.backend.ovs_idl import idlutils
python[5110]: File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 21, in <module>
python[5110]: from ovs.db import idl
python[5110]: File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 25, in <module>
python[5110]: from ovs.db import custom_index
python[5110]: File "/usr/lib64/python2.7/site-packages/ovs/db/custom_index.py", line 12, in <module>
python[5110]: from ovs.compat import sortedcontainers
python[5110]: ImportError: No module named compat
systemd[1]: neutron-destroy-patch-ports.service: main process exited, code=exited, status=1/FAILURE
systemd[1]: Failed to start OpenStack Neutron Destroy Patch Ports.
systemd[1]: Unit neutron-destroy-patch-ports.service entered failed state.
systemd[1]: neutron-destroy-patch-ports.service failed.
...

Version-Release number of selected component (if applicable):
2018-08-10.2

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP14 using packstack, InfraRed, topology AIO
2. Packstack deployments fails

Additional info:
openstack-packstack-13.0.0-0.20180731101322.73e0b07.el7ost.noarch
openstack-packstack-puppet-13.0.0-0.20180731101322.73e0b07.el7ost.noarch
python-neutron-13.0.0-0.20180803061300.765fd4d.el7ost.noarch
openstack-neutron-openvswitch-13.0.0-0.20180803061300.765fd4d.el7ost.noarch
python2-neutron-lib-1.18.0-0.20180725152803.67865c7.el7ost.noarch
puppet-neutron-13.2.0-0.20180805213727.2a42bb2.el7ost.noarch
openstack-neutron-common-13.0.0-0.20180803061300.765fd4d.el7ost.noarch
openstack-neutron-13.0.0-0.20180803061300.765fd4d.el7ost.noarch
python2-neutronclient-6.9.0-0.20180803115753.d090ea2.el7ost.noarch
Comment 1 Filip Hubík 2018-08-10 13:36:12 UTC 
Created attachment 1475021 [details]
/var/tmp/packstack/20180810-085725-cBTrNd/openstack-setup.log
Comment 2 Filip Hubík 2018-08-10 13:38:51 UTC 
Created attachment 1475022 [details]
answer file
Comment 5 Terry Wilson 2018-08-10 15:03:59 UTC 
Created attachment 1475105 [details]
Add compat module to ovs package in setup.py

This is happening because the python/setup.py forgot to include the compat/compat.sortedcontainers directory. I tested a quick build with the patch applied and it added the appropriate directories to the RPM. I'll go ahead and propose the patch upstream as well.
Comment 6 Terry Wilson 2018-08-10 15:14:42 UTC 
Patch proposed upstream at https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/350966.html
Comment 11 Terry Wilson 2018-08-10 20:17:26 UTC 
The patch has been merged upstream in both master and branch-2.10.
Comment 12 Flavio Leitner 2018-08-10 21:54:40 UTC 
openvswitch2.0 package including the package is available:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=17734563

Please verify.
fbl
Comment 13 Terry Wilson 2018-08-10 23:36:47 UTC 
After installing the fixed rpm from, doing from ovs.compat import sortedcontainers succeeds.
Comment 15 Filip Hubík 2018-08-14 11:52:36 UTC 
I can not confirm that this issue is fixed by openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp (it can be kept in version openvswitch2.10-2.10.0-0.20180808gitfaf64fb.el7fdp)
but it is fixed by update of python-openvswitch2.10 to version python-openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp.

I adjusted "Fixed in Version" field.
Comment 16 Assaf Muller 2018-08-14 14:28:18 UTC 
*** Bug 1615892 has been marked as a duplicate of this bug. ***
Comment 18 Filip Hubík 2018-08-20 09:46:51 UTC 
I can verify this is not issue since puddle 2018-08-17.2 where fixed package is included:

packtack-aio $ rpm -qa | grep openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fd
python-openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64
openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64
Comment 24 errata-xmlrpc 2019-01-11 11:51:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045