Hi, In OSP 11, there's an annoying error message (which likely won't cause any issues though), concerning the fact that neutron openvswitch agent creates multiple rows in the Manager table. Switched on debugging in /etc/neutron/neutron.conf on the controller node. Then, cleared up the OVS database: ~~~ sudo systemctl stop neutron-openvswitch-agent sudo systemctl stop openvswitch sudo rm -rf /etc/openvswitch/conf.db sudo systemctl start openvswitch sudo systemctl restart network ~~~ ~~~ [root@-overcloud-controller-0 conf.d]# ovs-vsctl show 7d558fb0-ea2c-4d0d-8508-f31ddc44d41d Bridge br-ex fail_mode: standalone Port "em2" Interface "em2" Port br-ex Interface br-ex type: internal Bridge br-tenant fail_mode: standalone Port "em3" Interface "em3" Port br-tenant Interface br-tenant type: internal ovs_version: "2.6.1" [root@-overcloud-controller-0 conf.d]# ovs-vsctl list Manager [root@-overcloud-controller-0 conf.d]# [root@-overcloud-controller-0 conf.d]# [root@-overcloud-controller-0 conf.d]# journalctl -u neutron-openvswitch-agent -f & [1] 673967 [root@-overcloud-controller-0 conf.d]# -- Logs begin at Mon 2017-10-23 10:04:19 UTC. -- Oct 24 22:58:54 -overcloud-controller-0 neutron-openvswitch-agent[654518]: Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. Oct 24 22:58:54 -overcloud-controller-0 neutron-openvswitch-agent[654518]: Option "rpc_backend" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. Oct 24 22:58:55 -overcloud-controller-0 neutron-openvswitch-agent[654518]: Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications". Oct 24 22:59:00 -overcloud-controller-0 neutron-openvswitch-agent[654518]: /usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py:200: FutureWarning: The access_policy argument is changing its default value to <class 'oslo_messaging.rpc.dispatcher.DefaultRPCAccessPolicy'> in version '?', please update the code to explicitly set None as the value: access_policy defaults to LegacyRPCAccessPolicy which exposes private methods. Explicitly set access_policy to DefaultRPCAccessPolicy or ExplicitRPCAccessPolicy. Oct 24 22:59:00 -overcloud-controller-0 neutron-openvswitch-agent[654518]: access_policy) Oct 24 22:59:01 -overcloud-controller-0 sudo[656436]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf Oct 24 22:59:05 -overcloud-controller-0 ovs-vsctl[657009]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --id=@manager create Manager "target=\"ptcp:6640:127.0.0.1\"" -- add Open_vSwitch . manager_options @manager Oct 24 22:59:05 -overcloud-controller-0 ovs-vsctl[657009]: ovs|00002|db_ctl_base|ERR|transaction error: {"details":"Transaction causes multiple rows in \"Manager\" table to have identical values (\"ptcp:6640:127.0.0.1\") for index on column \"target\". First row, with UUID 62bdd797-b9b5-4324-bc27-36a50917d570, existed in the database before this transaction and was not modified by the transaction. Second row, with UUID 9fcfd4d0-d75c-4f4e-a6ef-37997863cb43, was inserted by this transaction.","error":"constraint violation"} Oct 24 23:02:14 -overcloud-controller-0 systemd[1]: Stopping OpenStack Neutron Open vSwitch Agent... Oct 24 23:02:15 -overcloud-controller-0 systemd[1]: Stopped OpenStack Neutron Open vSwitch Agent. [root@-overcloud-controller-0 conf.d]# [root@-overcloud-controller-0 conf.d]# [root@-overcloud-controller-0 conf.d]# [root@-overcloud-controller-0 conf.d]# [root@-overcloud-controller-0 conf.d]# systemctl restart neutron-openvswitch-agent Oct 24 23:03:54 -overcloud-controller-0 systemd[1]: Starting OpenStack Neutron Open vSwitch Agent... Oct 24 23:03:54 -overcloud-controller-0 neutron-enable-bridge-firewall.sh[674028]: net.bridge.bridge-nf-call-arptables = 1 Oct 24 23:03:54 -overcloud-controller-0 neutron-enable-bridge-firewall.sh[674028]: net.bridge.bridge-nf-call-iptables = 1 Oct 24 23:03:54 -overcloud-controller-0 neutron-enable-bridge-firewall.sh[674028]: net.bridge.bridge-nf-call-ip6tables = 1 Oct 24 23:03:54 -overcloud-controller-0 systemd[1]: Started OpenStack Neutron Open vSwitch Agent. [root@-overcloud-controller-0 conf.d]# Oct 24 23:03:55 -overcloud-controller-0 neutron-openvswitch-agent[674035]: Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports. Oct 24 23:03:55 -overcloud-controller-0 neutron-openvswitch-agent[674035]: Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. Oct 24 23:03:55 -overcloud-controller-0 neutron-openvswitch-agent[674035]: Option "rpc_backend" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. Oct 24 23:03:56 -overcloud-controller-0 neutron-openvswitch-agent[674035]: Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications". Oct 24 23:03:57 -overcloud-controller-0 sudo[674062]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf Oct 24 23:03:57 -overcloud-controller-0 ovs-vsctl[674070]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --id=@manager create Manager "target=\"ptcp:6640:127.0.0.1\"" -- add Open_vSwitch . manager_options @manager -- set Manager ptcp:6640:127.0.0.1 inactivity_probe=10000 Oct 24 23:03:58 -overcloud-controller-0 neutron-openvswitch-agent[674035]: /usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py:200: FutureWarning: The access_policy argument is changing its default value to <class 'oslo_messaging.rpc.dispatcher.DefaultRPCAccessPolicy'> in version '?', please update the code to explicitly set None as the value: access_policy defaults to LegacyRPCAccessPolicy which exposes private methods. Explicitly set access_policy to DefaultRPCAccessPolicy or ExplicitRPCAccessPolicy. Oct 24 23:03:58 -overcloud-controller-0 neutron-openvswitch-agent[674035]: access_policy) Oct 24 23:03:59 -overcloud-controller-0 ovs-vsctl[674192]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --id=@manager create Manager "target=\"ptcp:6640:127.0.0.1\"" -- add Open_vSwitch . manager_options @manager Oct 24 23:03:59 -overcloud-controller-0 ovs-vsctl[674192]: ovs|00002|db_ctl_base|ERR|transaction error: {"details":"Transaction causes multiple rows in \"Manager\" table to have identical values (\"ptcp:6640:127.0.0.1\") for index on column \"target\". First row, with UUID a7471d14-1793-458e-b54d-ca6b7c1dd3a7, existed in the database before this transaction and was not modified by the transaction. Second row, with UUID 246ffd91-9a49-4cdf-843b-b840789ad42e, was inserted by this transaction.","error":"constraint violation"} [root@-overcloud-controller-0 conf.d]# ^C [root@-overcloud-controller-0 conf.d]# f -bash: f: command not found [root@-overcloud-controller-0 conf.d]# fg journalctl -u neutron-openvswitch-agent -f ^C So the problem is with neutron-openvswitch-agent which tries to insert the Manager twice.
This trace happens when multiple agents are started and native ovs interface sets the manager for itself. It doesn't cause any failure just logs the error that the manager already exists. If you check the manager table, can you see only a single manager there with correct connection uri?
Hi, It does look ok. So we can safely ignore this error message? In that case, I'm going to write a KCS which documents this. - Andreas +++++ [root@overcloud-controller-0 ~]# journalctl -u neutron-openvswitch-agent -f & [1] 906189 [root@overcloud-controller-0 ~]# -- Logs begin at Tue 2017-10-24 01:32:42 UTC. -- Oct 24 17:39:20 overcloud-controller-0.localdomain neutron-openvswitch-agent[648103]: Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports. Oct 24 17:39:21 overcloud-controller-0.localdomain neutron-openvswitch-agent[648103]: Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. Oct 24 17:39:21 overcloud-controller-0.localdomain neutron-openvswitch-agent[648103]: Option "rpc_backend" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. Oct 24 17:39:21 overcloud-controller-0.localdomain neutron-openvswitch-agent[648103]: Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications". Oct 24 17:39:21 overcloud-controller-0.localdomain neutron-openvswitch-agent[648103]: Could not load neutron.openstack.common.notifier.rpc_notifier Oct 24 17:39:26 overcloud-controller-0.localdomain neutron-openvswitch-agent[648103]: /usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py:200: FutureWarning: The access_policy argument is changing its default value to <class 'oslo_messaging.rpc.dispatcher.DefaultRPCAccessPolicy'> in version '?', please update the code to explicitly set None as the value: access_policy defaults to LegacyRPCAccessPolicy which exposes private methods. Explicitly set access_policy to DefaultRPCAccessPolicy or ExplicitRPCAccessPolicy. Oct 24 17:39:26 overcloud-controller-0.localdomain neutron-openvswitch-agent[648103]: access_policy) Oct 24 17:39:27 overcloud-controller-0.localdomain sudo[648766]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf Oct 24 17:39:28 overcloud-controller-0.localdomain ovs-vsctl[648874]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --id=@manager create Manager "target=\"ptcp:6640:127.0.0.1\"" -- add Open_vSwitch . manager_options @manager Oct 24 17:39:28 overcloud-controller-0.localdomain ovs-vsctl[648874]: ovs|00002|db_ctl_base|ERR|transaction error: {"details":"Transaction causes multiple rows in \"Manager\" table to have identical values (\"ptcp:6640:127.0.0.1\") for index on column \"target\". First row, with UUID 3dba7961-9267-4551-8a55-42ae74a0902c, existed in the database before this transaction and was not modified by the transaction. Second row, with UUID dae3fb03-ddef-44fb-a4ee-54ef256eef0a, was inserted by this transaction.","error":"constraint violation"} [root@overcloud-controller-0 ~]# [root@overcloud-controller-0 ~]# systemctl restart neutron-openvswitch-agent Oct 25 14:43:34 overcloud-controller-0.localdomain systemd[1]: Stopping OpenStack Neutron Open vSwitch Agent... Oct 25 14:43:35 overcloud-controller-0.localdomain systemd[1]: Starting OpenStack Neutron Open vSwitch Agent... Oct 25 14:43:35 overcloud-controller-0.localdomain neutron-enable-bridge-firewall.sh[906643]: net.bridge.bridge-nf-call-arptables = 1 Oct 25 14:43:35 overcloud-controller-0.localdomain neutron-enable-bridge-firewall.sh[906643]: net.bridge.bridge-nf-call-iptables = 1 Oct 25 14:43:35 overcloud-controller-0.localdomain neutron-enable-bridge-firewall.sh[906643]: net.bridge.bridge-nf-call-ip6tables = 1 Oct 25 14:43:35 overcloud-controller-0.localdomain systemd[1]: Started OpenStack Neutron Open vSwitch Agent. [root@overcloud-controller-0 ~]# Oct 25 14:43:36 overcloud-controller-0.localdomain neutron-openvswitch-agent[906649]: Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports. Oct 25 14:43:36 overcloud-controller-0.localdomain neutron-openvswitch-agent[906649]: Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. Oct 25 14:43:36 overcloud-controller-0.localdomain neutron-openvswitch-agent[906649]: Option "rpc_backend" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. Oct 25 14:43:36 overcloud-controller-0.localdomain neutron-openvswitch-agent[906649]: Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications". Oct 25 14:43:36 overcloud-controller-0.localdomain neutron-openvswitch-agent[906649]: Could not load neutron.openstack.common.notifier.rpc_notifier Oct 25 14:43:38 overcloud-controller-0.localdomain neutron-openvswitch-agent[906649]: /usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py:200: FutureWarning: The access_policy argument is changing its default value to <class 'oslo_messaging.rpc.dispatcher.DefaultRPCAccessPolicy'> in version '?', please update the code to explicitly set None as the value: access_policy defaults to LegacyRPCAccessPolicy which exposes private methods. Explicitly set access_policy to DefaultRPCAccessPolicy or ExplicitRPCAccessPolicy. Oct 25 14:43:38 overcloud-controller-0.localdomain neutron-openvswitch-agent[906649]: access_policy) Oct 25 14:43:39 overcloud-controller-0.localdomain sudo[907145]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf Oct 25 14:43:39 overcloud-controller-0.localdomain ovs-vsctl[907157]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --id=@manager create Manager "target=\"ptcp:6640:127.0.0.1\"" -- add Open_vSwitch . manager_options @manager Oct 25 14:43:39 overcloud-controller-0.localdomain ovs-vsctl[907157]: ovs|00002|db_ctl_base|ERR|transaction error: {"details":"Transaction causes multiple rows in \"Manager\" table to have identical values (\"ptcp:6640:127.0.0.1\") for index on column \"target\". First row, with UUID 13742552-2c38-4af7-b7b7-da093ea8fdc5, was inserted by this transaction. Second row, with UUID 3dba7961-9267-4551-8a55-42ae74a0902c, existed in the database before this transaction and was not modified by the transaction.","error":"constraint violation"} [root@overcloud-controller-0 ~]# [root@overcloud-controller-0 ~]# [root@overcloud-controller-0 ~]# fg journalctl -u neutron-openvswitch-agent -f ^C [root@overcloud-controller-0 ~]# ^C [root@overcloud-controller-0 ~]# ovs-vsctl list Manager _uuid : 3dba7961-9267-4551-8a55-42ae74a0902c connection_mode : [] external_ids : {} inactivity_probe : 10000 is_connected : true max_backoff : [] other_config : {} status : {bound_port="6640", n_connections="3", sec_since_connect="0", sec_since_disconnect="0"} target : "ptcp:6640:127.0.0.1" [root@overcloud-controller-0 ~]#
Guys I am running on CentOS 7 and have a seperate network node and I;m hitting this bug. It happens after I create a port to my br-ex interface: ovs-vsctl add-port br-ex eth0; systemctl restart network This is where I loose access to the box. I have executed all the above commands in this bug and see that its a dupe of this bug but the steps above still do not prevent the box from loosing network connection when we do a network restart. Is there a fix? Or will Openstack.org fix the neutron-agent? thanks David
This bug is linked to a harmless log message that we don't intend to change/fix for this release. David, you're referring to a separate issue. Can you please file a new bug report and include steps to reproduce and a full sosreport with DEBUG level logs that includes the timestamp for the event that causes the network interruption?