Bug 1157776 - ovs failures while setting gateway for a router
Summary: ovs failures while setting gateway for a router
Keywords:
Status: CLOSED DUPLICATE of bug 1160340
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 6.0 (Juno)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 6.0 (Juno)
Assignee: Nir Magnezi
QA Contact: Ofer Blaut
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-10-27 16:53 UTC by Marko Myllynen
Modified: 2019-07-11 08:18 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-05-07 12:48:14 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
osp-logs.tar (630.00 KB, application/octet-stream)
2015-04-08 11:36 UTC, Marko Myllynen
no flags Details

Description Marko Myllynen 2014-10-27 16:53:25 UTC
Description of problem:
When configuring networks in a br-ex using setup with the stanzas below I see failures in /var/log/openvswitch/ovs-vswitchd.log (neutron output omitted for brevity):

# . /root/keystonerc_admin
# tid=$(keystone tenant-list | awk '/services/ {print $2}')
# neutron net-create --tenant-id $tid --router:external=true net-ext0
# neutron subnet-create --tenant-id $tid --name subnet-ext0 --disable-dhcp --gateway 192.168.122.1 --allocation-pool start=192.168.122.200,end=192.168.122.254 net-ext0 192.168.122.0/24
# . /root/keystonerc_demo
# neutron net-create net-demo0
# neutron subnet-create --name subnet-demo0 --enable-dhcp --gateway 192.168.1.1 --allocation-pool start=192.168.1.2,end=192.168.1.254 net-demo0 192.168.1.0/24
# neutron router-create router-demo0
# . /root/keystonerc_admin
# neutron router-gateway-set router-demo0 net-ext0
# tail -n 3 /var/log/openvswitch/ovs-vswitchd.log
2014-10-27T16:05:46.537Z|00033|bridge|INFO|bridge br-ex: added interface qg-eb8b8a0d-42 on port 2
2014-10-27T16:05:47.413Z|00034|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qg-eb8b8a0d-42 device failed: No such device
2014-10-27T16:05:51.254Z|00035|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on qg-eb8b8a0d-42 device failed: No such device
# ovs-vsctl list-ports br-ex
eth0
qg-eb8b8a0d-42

And a bit later:

# . /root/keystonerc_demo
# neutron router-interface-add router-demo0 subnet-demo0
# tail -n 7 /var/log/openvswitch/ovs-vswitchd.log
2014-10-27T16:39:56.153Z|00036|bridge|INFO|bridge br-int: added interface qr-920e5c0b-62 on port 2
2014-10-27T16:39:56.153Z|00037|netdev_linux|WARN|qg-eb8b8a0d-42: removing policing failed: No such device
2014-10-27T16:39:56.693Z|00038|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qr-920e5c0b-62 device failed: No such device
2014-10-27T16:39:59.569Z|00039|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on qr-920e5c0b-62 device failed: No such device
2014-10-27T16:39:59.569Z|00040|netdev_linux|WARN|qr-920e5c0b-62: removing policing failed: No such device
2014-10-27T16:40:09.040Z|00041|ofproto|INFO|br-tun: 2 flow_mods 10 s ago (1 adds, 1 modifications)
2014-10-27T16:40:09.843Z|00042|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes)

Later on everything seems to work properly despite of the above. If the failures are expected perhaps they should be logged less prominently.

Version-Release number of selected component (if applicable):
openstack-neutron-2014.1.2-2.el7ost.noarch

Comment 4 Yoni Shafrir 2015-03-02 07:05:37 UTC
Can you please attach full logs of the following:

1. neutron server
2. neutron l2 agent (ovs agent)
3. neutron l3 agent

Comment 6 Marko Myllynen 2015-04-08 11:36:13 UTC
I reproduced this on OSP 6 and I'm attaching a tar ball containing all the requested logs from the ctrl/network node and two compute nodes.

After installation I booted up the machines around 14:10 local time and then did the above steps listed in comment 0 starting around 14:15.

On OSP 6 I also see these kinds of messages being flooded to logs:

2015-04-08T11:17:09.329Z|00003|dpif(handler_5)|WARN|system@ovs-system: failed to
 put[create][modify] (No such file or directory) key20(bad key length 4, expecte
d -1)(00 00 00 00/(bad mask length 0, expected -1)(),key19(bad key length 4, exp
ected -1)(00 00 00 00/(bad mask length 0, expected -1)(),skb_priority(0),in_port
(1),skb_mark(0/0),eth(src=1e:b3:ff:f5:89:4c,dst=76:1b:43:d7:fc:4e),eth_type(0x08
00),ipv4(src=192.168.122.101/0.0.0.0,dst=192.168.122.103/0.0.0.0,proto=6/0,tos=0
/0,ttl=64/0,frag=no/0xff),tcp(src=5672/0,dst=37345/0),tcp_flags(0x018/0x000), ac
tions:2

Thanks.

Comment 7 Marko Myllynen 2015-04-08 11:36:56 UTC
Created attachment 1012174 [details]
osp-logs.tar

Comment 8 Nir Magnezi 2015-05-07 12:48:14 UTC
I managed to reproduce this with the latest upstream code.
The error is triggered each time an interface moves across namespaces. Hence, other scenarios such as a creation of a dhcp port for a subnet will produce the same error.

There's a broader issue with ovs here, which is fully described in bug #1160340 comment #3.

In short:
The net interface disappeared from the namespace where ovs-vswitchd is running.
Due to the fact that ovs-vswitchd is unaware of namespaces, it just cannot access it anymore. Therefore it produces those warnings.
According to Jiri, The kernel part of ovs is somehow namespace-aware so the packets keep flowing even after the interface is moved to a different namespace.

*** This bug has been marked as a duplicate of bug 1160340 ***


Note You need to log in before you can comment on or make changes to this bug.