Bug 1843475

Summary:	OSP 13 dpdk sriov overcloud deployment failed with ovs internal error parsing flow key
Product:	Red Hat OpenStack	Reporter:	Maxim Babushkin <mbabushk>
Component:	openvswitch	Assignee:	Open vSwitch development team <ovs-team>
Status:	CLOSED ERRATA	QA Contact:	Sanjay Upadhyay <supadhya>
Severity:	high	Docs Contact:
Priority:	high
Version:	13.0 (Queens)	CC:	aconole, amuller, apevec, bcafarel, chrisw, dalvarez, fiezzi, mgarciac, mjozefcz, oblaut, ovs-team, rhos-maint, scohen, spower, supadhya
Target Milestone:	z12	Keywords:	Triaged, ZStream
Target Release:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openvswitch2.11-2.11.0-54.20200327gita4efc59.el7fdp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-06-24 11:41:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Maxim Babushkin 2020-06-03 11:40:50 UTC

Description of problem:
Recent osp 13 puddle 2020-05-28.2 failed to deploy dpdk sriov overcloud with the following error:

ovs-vswitchd[7490]: ovs|00001|dpif_netdev(revalidator6)|ERR|internal error parsing flow key skb_priority(0),skb_mark(0),ct
_state(0),ct_zone(0),ct_mark(0),ct_label(0),recirc_id(0),dp_hash(0),in_port(2),packet_type(ns=0,id=0),eth(src=e4:fc:82:34:4b:48,dst=01:00:5e:00:00:01),eth_type(
0x8100),vlan(vid=134,pcp=0),encap(eth_type(0x0800),ipv4(src=10.20.155.5,dst=224.0.0.1,proto=2,tos=0xc0,ttl=1,frag=no))

Note - Previous puddle successfully deployed.


Version-Release number of selected component (if applicable):
osp 13
Puddle 2020-05-28.2


How reproducible:
Overcloud deploy of dpdk sriov based environment

Additional info:
From /var/log/messages

Jun  2 12:00:51 computeovsdpdksriov-0 ovs-vswitchd[7490]: ovs|00001|dpif_netdev(revalidator6)|ERR|internal error parsing flow key skb_priority(0),skb_mark(0),ct
_state(0),ct_zone(0),ct_mark(0),ct_label(0),recirc_id(0),dp_hash(0),in_port(2),packet_type(ns=0,id=0),eth(src=e4:fc:82:34:4b:48,dst=01:00:5e:00:00:01),eth_type(
0x8100),vlan(vid=134,pcp=0),encap(eth_type(0x0800),ipv4(src=10.20.155.5,dst=224.0.0.1,proto=2,tos=0xc0,ttl=1,frag=no))
Jun  2 12:01:01 computeovsdpdksriov-0 systemd: Created slice User Slice of root.
Jun  2 12:01:01 computeovsdpdksriov-0 systemd: Started Session 1 of user root.
Jun  2 12:04:46 computeovsdpdksriov-0 ovs-vswitchd[7490]: ovs|00003|dpif_netdev(revalidator6)|ERR|internal error parsing flow key skb_priority(0),skb_mark(0),ct
_state(0),ct_zone(0),ct_mark(0),ct_label(0),recirc_id(0),dp_hash(0),in_port(3),packet_type(ns=0,id=0),eth(src=e4:fc:82:34:4b:47,dst=01:00:5e:00:00:01),eth_type(
0x8100),vlan(vid=134,pcp=0),encap(eth_type(0x0800),ipv4(src=10.20.155.5,dst=224.0.0.1,proto=2,tos=0xc0,ttl=1,frag=no))
Jun  2 12:08:57 computeovsdpdksriov-0 ovs-vswitchd[7490]: ovs|00005|dpif_netdev(revalidator6)|ERR|internal error parsing flow key skb_priority(0),skb_mark(0),ct
_state(0),ct_zone(0),ct_mark(0),ct_label(0),recirc_id(0),dp_hash(0),in_port(3),packet_type(ns=0,id=0),eth(src=e4:fc:82:34:4b:47,dst=01:00:5e:00:00:01),eth_type(
0x8100),vlan(vid=134,pcp=0),encap(eth_type(0x0800),ipv4(src=10.20.155.5,dst=224.0.0.1,proto=2,tos=0xc0,ttl=1,frag=no))


From /var/log/openvswitch/ovs-vswitchd.log

2020-06-02T16:00:51.877Z|00001|dpif_netdev(revalidator6)|ERR|internal error parsing flow key skb_priority(0),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),recirc_id(0),dp_hash(0),in_port(2),packet_type(ns=0,id=0),eth(src=e4:fc:82:34:4b:48,dst=01:00:5e:00:00:01),eth_type(0x8100),vlan(vid=134,pcp=0),encap(eth_type(0x0800),ipv4(src=10.20.155.5,dst=224.0.0.1,proto=2,tos=0xc0,ttl=1,frag=no))
2020-06-02T16:00:51.877Z|00002|dpif(revalidator6)|WARN|netdev@ovs-netdev: failed to put[modify] (Invalid argument) ufid:480746e3-742f-4b34-94a3-1cbb3099bfd3 skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(2),packet_type(ns=0,id=0),eth(src=e4:fc:82:34:4b:48,dst=01:00:5e:00:00:01),eth_type(0x8100),vlan(vid=134,pcp=0/0x0),encap(eth_type(0x0800),ipv4(src=10.20.155.5/0.0.0.0,dst=224.0.0.1/0.0.0.0,proto=2/0,tos=0xc0/0,ttl=1/0,frag=no)), actions:userspace(pid=0,slow_path(match))
2020-06-02T16:04:46.216Z|00003|dpif_netdev(revalidator6)|ERR|internal error parsing flow key skb_priority(0),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),recirc_id(0),dp_hash(0),in_port(3),packet_type(ns=0,id=0),eth(src=e4:fc:82:34:4b:47,dst=01:00:5e:00:00:01),eth_type(0x8100),vlan(vid=134,pcp=0),encap(eth_type(0x0800),ipv4(src=10.20.155.5,dst=224.0.0.1,proto=2,tos=0xc0,ttl=1,frag=no))
2020-06-02T16:04:46.216Z|00004|dpif(revalidator6)|WARN|netdev@ovs-netdev: failed to put[modify] (Invalid argument) ufid:a2786594-e54a-43a8-8371-29af98cc36ba skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(3),packet_type(ns=0,id=0),eth(src=e4:fc:82:34:4b:47,dst=01:00:5e:00:00:01),eth_type(0x8100),vlan(vid=134,pcp=0/0x0),encap(eth_type(0x0800),ipv4(src=10.20.155.5/0.0.0.0,dst=224.0.0.1/0.0.0.0,proto=2/0,tos=0xc0/0,ttl=1/0,frag=no)), actions:userspace(pid=0,slow_path(match))
2020-06-02T16:08:57.216Z|00005|dpif_netdev(revalidator6)|ERR|internal error parsing flow key skb_priority(0),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),recirc_id(0),dp_hash(0),in_port(3),packet_type(ns=0,id=0),eth(src=e4:fc:82:34:4b:47,dst=01:00:5e:00:00:01),eth_type(0x8100),vlan(vid=134,pcp=0),encap(eth_type(0x0800),ipv4(src=10.20.155.5,dst=224.0.0.1,proto=2,tos=0xc0,ttl=1,frag=no))
2020-06-02T16:08:57.216Z|00006|dpif(revalidator6)|WARN|netdev@ovs-netdev: failed to put[modify] (Invalid argument) ufid:a2786594-e54a-43a8-8371-29af98cc36ba skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(3),packet_type(ns=0,id=0),eth(src=e4:fc:82:34:4b:47,dst=01:00:5e:00:00:01),eth_type(0x8100),vlan(vid=134,pcp=0/0x0),encap(eth_type(0x0800),ipv4(src=10.20.155.5/0.0.0.0,dst=224.0.0.1/0.0.0.0,proto=2/0,tos=0xc0/0,ttl=1/0,frag=no)), actions:userspace(pid=0,slow_path(match))

Comment 1 Maxim Babushkin 2020-06-03 11:42:24 UTC

Sosreports link - http://rhos-release.virt.bos.redhat.com/log/bz1843475/

Comment 12 Maxim Babushkin 2020-06-10 06:41:10 UTC

*** Bug 1843472 has been marked as a duplicate of this bug. ***

Comment 14 Maxim Babushkin 2020-06-11 16:20:41 UTC

Tested with 2020-06-09.2 puddle.
Deployment and testing passed.

Comment 16 errata-xmlrpc 2020-06-24 11:41:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2719