Bug 1722033

Summary:	OSP15 UC neutron_ovs_agent fails - No module named neutron.cmd.destroy_patch_ports
Product:	Red Hat OpenStack	Reporter:	Pavel Sedlák <psedlak>
Component:	openstack-neutron	Assignee:	Bernard Cafarelli <bcafarel>
Status:	CLOSED ERRATA	QA Contact:	Eran Kuris <ekuris>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	15.0 (Stein)	CC:	amuller, bcafarel, chrisw, scohen
Target Milestone:	beta	Keywords:	Triaged
Target Release:	15.0 (Stein)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-neutron-14.0.2-0.20190619122701.3733721.el8	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-09-21 11:23:21 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Pavel Sedlák 2019-06-19 12:03:59 UTC

In OSP15 CI phase1 OC deployment fails on OC nodes not having ip addresses.
Issue seems to be with neutron_ovs_agent service is not running on Undercloud,
due to it failing on importing neutron.cmd.destroy_patch_ports python3 module.

Container image used in this build is rhosp15/openstack-neutron-openvswitch-agent:20190618.1
with rpms:
> network-scripts-openvswitch2.11-2.11.0-9.el8fdp.x86_64
> openstack-neutron-openvswitch-14.0.2-0.20190614231136.3733721.el8ost.noarch
> openvswitch2.11-2.11.0-9.el8fdp.x86_64
> openvswitch-selinux-extra-policy-1.0-12.el8fdp.noarch
> python3-openvswitch2.11-2.11.0-9.el8fdp.x86_64
> python3-ovsdbapp-0.15.0-0.20190312195947.bc06517.el8ost.noarch
> python3-rhosp-openvswitch-2.11-0.3.el8ost.noarch
> rhosp-openvswitch-2.11-0.3.el8ost.noarch
>
> openstack-neutron-14.0.2-0.20190614231136.3733721.el8ost.noarch
> openstack-neutron-common-14.0.2-0.20190614231136.3733721.el8ost.noarch
> openstack-neutron-lbaas-14.0.1-0.20190614170521.30bdd86.el8ost.noarch
> openstack-neutron-ml2-14.0.2-0.20190614231136.3733721.el8ost.noarch
> openstack-neutron-openvswitch-14.0.2-0.20190614231136.3733721.el8ost.noarch
> puppet-neutron-14.4.1-0.20190531220405.ff3610d.el8ost.noarch
> python3-neutron-14.0.2-0.20190614231136.3733721.el8ost.noarch
> python3-neutronclient-6.12.0-0.20190312100012.680b417.el8ost.noarch
> python3-neutron-dynamic-routing-14.0.1-0.20190426180400.f313f0e.el8ost.noarch
> python3-neutron-lbaas-14.0.1-0.20190614170521.30bdd86.el8ost.noarch
> python3-neutron-lib-1.25.0-0.20190521130309.fc2a810.el8ost.noarch


systemctl
> ● tripleo_neutron_ovs_agent.service                loaded failed     failed        neutron_ovs_agent container
journalctl (snip, repeated)
> Jun 19 10:30:12 undercloud-0.redhat.local systemd[1]: Starting neutron_ovs_agent container...
> Jun 19 10:30:12 undercloud-0.redhat.local podman[69452]: neutron_ovs_agent
> Jun 19 10:30:12 undercloud-0.redhat.local systemd[1]: Started neutron_ovs_agent container.
> Jun 19 10:30:13 undercloud-0.redhat.local podman[69559]: 4b2dcf900ccbc98e3c5a3e97f27c357983323e2a78bbf8d44699b88b1c71>
> Jun 19 10:30:13 undercloud-0.redhat.local systemd[1]: tripleo_neutron_ovs_agent.service: Service RestartSec=100ms exp>
> Jun 19 10:30:13 undercloud-0.redhat.local systemd[1]: tripleo_neutron_ovs_agent.service: Scheduled restart job, resta>
> Jun 19 10:30:13 undercloud-0.redhat.local systemd[1]: Stopped neutron_ovs_agent container.
> Jun 19 10:30:13 undercloud-0.redhat.local systemd[1]: Starting neutron_ovs_agent container...
> Jun 19 10:30:13 undercloud-0.redhat.local podman[69574]: neutron_ovs_agent
> Jun 19 10:30:13 undercloud-0.redhat.local systemd[1]: Started neutron_ovs_agent container.
> Jun 19 10:30:14 undercloud-0.redhat.local podman[69693]: 4b2dcf900ccbc98e3c5a3e97f27c357983323e2a78bbf8d44699b88b1c71>
> Jun 19 10:30:14 undercloud-0.redhat.local systemd[1]: tripleo_neutron_ovs_agent.service: Service RestartSec=100ms exp>
> Jun 19 10:30:14 undercloud-0.redhat.local systemd[1]: tripleo_neutron_ovs_agent.service: Scheduled restart job, resta>
> Jun 19 10:30:14 undercloud-0.redhat.local systemd[1]: Stopped neutron_ovs_agent container.

podman ps
> 4b2dcf900ccb  192.168.24.1:8787/rhosp15/openstack-neutron-openvswitch-agent:20190618.1  dumb-init --singl...  40 minutes ago     Exited (1) 20 minutes ago            neutron_ovs_agent
podman logs 4b2dcf900ccb (end of, this is repeated there again and again)
> + sudo -E kolla_set_configs
> INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
> INFO:__main__:Validating config file
> INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
> INFO:__main__:Copying service configuration files
> INFO:__main__:Deleting /etc/default/neutron-server
> INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/default/neutron-server to /etc/default/neutron-server
> INFO:__main__:Deleting /etc/my.cnf.d/tripleo.cnf
> INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/tripleo.cnf to /etc/my.cnf.d/tripleo.cnf
> INFO:__main__:Deleting /etc/neutron/plugin.ini
> INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/plugin.ini to /etc/neutron/plugin.ini
> INFO:__main__:Deleting /etc/neutron/plugins/ml2/ironic_neutron_agent.ini
> INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/plugins/ml2/ironic_neutron_agent.ini to /etc/neutron/plugins/ml2/ironic_neutron_agent.ini
> INFO:__main__:Deleting /etc/neutron/plugins/ml2/ml2_conf.ini
> INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/plugins/ml2/ml2_conf.ini to /etc/neutron/plugins/ml2/ml2_conf.ini
> INFO:__main__:Deleting /etc/neutron/plugins/ml2/openvswitch_agent.ini
> INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/plugins/ml2/openvswitch_agent.ini to /etc/neutron/plugins/ml2/openvswitch_agent.ini
> INFO:__main__:Deleting /etc/neutron/dhcp_agent.ini
> INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/dhcp_agent.ini to /etc/neutron/dhcp_agent.ini
> INFO:__main__:Deleting /etc/neutron/l3_agent.ini
> INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/l3_agent.ini to /etc/neutron/l3_agent.ini
> INFO:__main__:Deleting /etc/neutron/neutron.conf
> INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/neutron.conf to /etc/neutron/neutron.conf
> INFO:__main__:Writing out command to execute
> INFO:__main__:Setting permission for /var/log/neutron
> INFO:__main__:Setting permission for /var/log/neutron/server.log
> INFO:__main__:Setting permission for /var/log/neutron/dhcp-agent.log
> INFO:__main__:Setting permission for /var/log/neutron/l3-agent.log
> INFO:__main__:Setting permission for /var/log/neutron/privsep-helper.log
> INFO:__main__:Setting permission for /var/log/neutron/ironic-neutron-agent.log
> ++ cat /run_command
> + CMD=/neutron_ovs_agent_launcher.sh
> + ARGS=
> + [[ ! -n '' ]]
> + . kolla_extend_start
> ++ [[ ! -d /var/log/kolla/neutron ]]
> +++ stat -c %a /var/log/kolla/neutron
> ++ [[ 2755 != \7\5\5 ]]
> ++ chmod 755 /var/log/kolla/neutron
> ++ . /usr/local/bin/kolla_neutron_extend_start
> Running command: '/neutron_ovs_agent_launcher.sh'
> + echo 'Running command: '\''/neutron_ovs_agent_launcher.sh'\'''
> + exec /neutron_ovs_agent_launcher.sh
> + /usr/bin/python3 -m neutron.cmd.destroy_patch_ports --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-openvswitch-agent
> /usr/bin/python3: No module named neutron.cmd.destroy_patch_ports


Btw not sure if it is important or not, seems the destroy_patch_ports is mentioned explicitely in
> [root@undercloud-0 ~]# cat /var/lib/container-config-scripts/neutron_ovs_agent_launcher.sh
> #!/bin/bash
> set -xe
> /usr/bin/python3 -m neutron.cmd.destroy_patch_ports --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-openvswitch-agent
> /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --log-file=/var/log/neutron/openvswitch-agent.log

Comment 9 errata-xmlrpc 2019-09-21 11:23:21 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811