1779854 – Error with large openflow messages

The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1779854 - Error with large openflow messages

Summary: Error with large openflow messages

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux Fast Datapath
Classification:	Red Hat
Component:	ovn2.11
Sub Component:
Version:	RHEL 7.7
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	Mark Michelson
QA Contact:	ying xu
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1837593 1838166
TreeView+	depends on / blocked

Reported:	2019-12-04 21:02 UTC by Mark Michelson
Modified:	2023-10-06 18:56 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1837593 1838166 (view as bug list)
Environment:
Last Closed:	2020-07-15 13:00:54 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	FD-341	0	None	None	None	2023-10-06 18:56:02 UTC
Red Hat Product Errata	RHBA-2020:2941	0	None	None	None	2020-07-15 13:01:30 UTC

Description Mark Michelson 2019-12-04 21:02:59 UTC

Description of problem:
OVN can potentially create overly large OpenFlow flows depending on configuration. An example that can do this is to create a load balancer that has a very large number of backends. Other ways to potentially trigger this is with extra large ACLs or very large multicast groups.

When the generated OpenFlow message is in excess of 65535, the length cannot be expressed in an OpenFlow header. The message length is then truncated to the final 16 bits of the length of the flow. This results in an incomplete flow being installed.

How reproducible:
Every time.

Steps to Reproduce:
Open the OVN sandbox and run the following script

for i in `seq 1 5` ; do for j in `seq 1 254` ; do echo 192.169.$i.$j:80 >> /tmp/ips; done ; done
IPS=`cat /tmp/ips |xargs |tr ' ' ','`
ovn-nbctl ls-add ls1
ovn-nbctl lsp-add ls1 ls1-p1
ovs-vsctl add-port br-int p1 -- set Interface p1 external_ids:iface-id=ls1-p1
ovn-nbctl lb-add lb0 172.172.0.1:8080 ${IPS}
ovn-nbctl ls-lb-add ls1 lb0

Then check sandbox/ovn-controller.log to see the errors generated.

Actual results:
You'll see errors in ovn-controller's log such as:
2019-12-04T20:51:52.556Z|00015|ofp_msgs|WARN|unknown OpenFlow message (version 0, type 64)
2019-12-04T20:51:52.556Z|00016|ofctrl|INFO|OpenFlow error: OFPT_ERROR (xid=0xffffffff): OFPBRC_BAD_VERSION
***decode error: OFPBRC_BAD_TYPE***
00000000  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000010  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000020  00 0f 13 00 00 00 00 00-ff ff 00 18 00 00 23 20 |..............# |
00000030  00 24 00 00 00 02 00 11-c0 a9 04 df 00 50 00 00 |.$...........P..|
00000040  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000050  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000060  00 0f 13 00                                     |....            |
2019-12-04T20:51:52.556Z|00017|vconn_stream|ERR|send: Broken pipe
2019-12-04T20:51:52.556Z|00018|rconn|WARN|unix:/home/putnopvut/ovn-copy/tutorial/sandbox/br-int.mgmt: connection dropped (Broken pipe)
2019-12-04T20:51:53.557Z|00019|rconn|INFO|unix:/home/putnopvut/ovn-copy/tutorial/sandbox/br-int.mgmt: connecting...
2019-12-04T20:51:53.558Z|00020|rconn|INFO|unix:/home/putnopvut/ovn-copy/tutorial/sandbox/br-int.mgmt: connected

Expected results:
I would expect that the openflow message would be split into chunks and sent to ovs-vswitch, allowing for the large flow to be installed if possible. If flows in excess of 65535 bytes are not possible, then the flow itself should be split into multiple flows.

Comment 1 Dan Williams 2020-05-12 16:34:25 UTC

Upstream patch: http://patchwork.ozlabs.org/project/openvswitch/patch/20200507212442.315956-1-mmichels@redhat.com/

Comment 2 Mark Michelson 2020-05-15 20:52:47 UTC

Patch has been backported to ovn2.13 FDN.

Comment 6 ying xu 2020-05-21 03:52:56 UTC

use the reproducer provided in the bug ,I reproduced on version:
# rpm -qa|grep ovn
ovn2.13-2.13.0-21.el7fdp.x86_64
ovn2.13-central-2.13.0-21.el7fdp.x86_64
ovn2.13-host-2.13.0-21.el7fdp.x86_64

2020-05-21T03:34:36.048Z|00023|ofctrl|INFO|OpenFlow error: OFPT_ERROR (xid=0xffffffff): OFPBRC_BAD_VERSION
***decode error: OFPBRC_BAD_TYPE***
00000000  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000010  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000020  00 0f 13 00 00 00 00 00-ff ff 00 18 00 00 23 20 |..............# |
00000030  00 24 00 00 00 02 00 11-c0 a9 02 ef 00 50 00 00 |.$...........P..|
00000040  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000050  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000060  00 0f 13 00                                     |....            |
2020-05-21T03:34:36.049Z|00024|vconn_stream|ERR|send: Broken pipe
2020-05-21T03:34:36.049Z|00025|rconn|WARN|unix:/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2020-05-21T03:34:37.050Z|00026|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
2020-05-21T03:34:37.051Z|00027|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected
2020-05-21T03:34:37.280Z|00028|ofctrl|INFO|OpenFlow error: OFPT_ERROR (xid=0xffffffff): OFPBRC_BAD_VERSION
***decode error: OFPBRC_BAD_TYPE***
00000000  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000010  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000020  00 0f 13 00 00 00 00 00-ff ff 00 18 00 00 23 20 |..............# |
00000030  00 24 00 00 00 02 00 11-c0 a9 02 ef 00 50 00 00 |.$...........P..|
00000040  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000050  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000060  00 0f 13 00                                     |....            |
2020-05-21T03:34:37.280Z|00029|vconn_stream|ERR|send: Broken pipe
2020-05-21T03:34:37.280Z|00030|rconn|WARN|unix:/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2020-05-21T03:34:37.281Z|00031|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: waiting 2 seconds before reconnect
2020-05-21T03:34:39.283Z|00032|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
2020-05-21T03:34:39.283Z|00033|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected
2020-05-21T03:34:39.512Z|00034|ofctrl|INFO|OpenFlow error: OFPT_ERROR (xid=0xffffffff): OFPBRC_BAD_VERSION
***decode error: OFPBRC_BAD_TYPE***
00000000  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000010  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000020  00 0f 13 00 00 00 00 00-ff ff 00 18 00 00 23 20 |..............# |
00000030  00 24 00 00 00 02 00 11-c0 a9 02 ef 00 50 00 00 |.$...........P..|
00000040  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000050  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000060  00 0f 13 00                                     |....            |
2020-05-21T03:34:39.512Z|00035|rconn|WARN|unix:/run/openvswitch/br-int.mgmt: connection dropped (Connection reset by peer)
2020-05-21T03:34:39.513Z|00036|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: waiting 4 seconds before reconnect
2020-05-21T03:34:43.517Z|00037|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
2020-05-21T03:34:43.518Z|00038|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected
2020-05-21T03:34:43.749Z|00039|ofctrl|INFO|OpenFlow error: OFPT_ERROR (xid=0xffffffff): OFPBRC_BAD_VERSION
***decode error: OFPBRC_BAD_TYPE***
00000000  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000010  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000020  00 0f 13 00 00 00 00 00-ff ff 00 18 00 00 23 20 |..............# |
00000030  00 24 00 00 00 02 00 11-c0 a9 02 ef 00 50 00 00 |.$...........P..|
00000040  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000050  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000060  00 0f 13 00                                     |....            |
2020-05-21T03:34:43.749Z|00040|vconn_stream|ERR|send: Broken pipe
2020-05-21T03:34:43.749Z|00041|rconn|WARN|unix:/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2020-05-21T03:34:43.750Z|00042|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: continuing to retry connections in the background but suppressing further logging
2020-05-21T03:34:51.753Z|00043|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected
2020-05-21T03:34:51.983Z|00044|ofctrl|INFO|OpenFlow error: OFPT_ERROR (xid=0xffffffff): OFPBRC_BAD_VERSION
***decode error: OFPBRC_BAD_TYPE***
00000000  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000010  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000020  00 0f 13 00 00 00 00 00-ff ff 00 18 00 00 23 20 |..............# |
00000030  00 24 00 00 00 02 00 11-c0 a9 02 ef 00 50 00 00 |.$...........P..|
00000040  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000050  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000060  00 0f 13 00                                     |....            |
2020-05-21T03:34:51.983Z|00045|vconn_stream|ERR|send: Broken pipe
2020-05-21T03:34:51.983Z|00046|rconn|WARN|unix:/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2020-05-21T03:34:59.984Z|00047|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected
2020-05-21T03:35:00.216Z|00048|ofctrl|INFO|OpenFlow error: OFPT_ERROR (xid=0xffffffff): OFPBRC_BAD_VERSION
***decode error: OFPBRC_BAD_TYPE***
00000000  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000010  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000020  00 0f 13 00 00 00 00 00-ff ff 00 18 00 00 23 20 |..............# |
00000030  00 24 00 00 00 02 00 11-c0 a9 02 ef 00 50 00 00 |.$...........P..|
00000040  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000050  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000060  00 0f 13 00                                     |....            |
2020-05-21T03:35:00.216Z|00049|vconn_stream|ERR|send: Broken pipe
2020-05-21T03:35:00.216Z|00050|rconn|WARN|unix:/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2020-05-21T03:35:08.221Z|00051|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected
2020-05-21T03:35:08.460Z|00052|ofctrl|INFO|OpenFlow error: OFPT_ERROR (xid=0xffffffff): OFPBRC_BAD_VERSION
***decode error: OFPBRC_BAD_TYPE***
00000000  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000010  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000020  00 0f 13 00 00 00 00 00-ff ff 00 18 00 00 23 20 |..............# |
00000030  00 24 00 00 00 02 00 11-c0 a9 02 ef 00 50 00 00 |.$...........P..|
00000040  00 40 00 64 ff ff ff ff-ff ff ff ff 00 00 00 00 |.@.d............|
00000050  ff ff 00 30 00 00 23 20-00 23 00 01 00 01 1a 04 |...0..# .#......|
00000060  00 0f 13 00                                     |....            |
2020-05-21T03:35:08.460Z|00053|vconn_stream|ERR|send: Broken pipe
2020-05-21T03:35:08.460Z|00054|rconn|WARN|unix:/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)

I verified on version:
# rpm -qa|grep ovn
ovn2.13-2.13.0-31.el7fdp.x86_64
ovn2.13-central-2.13.0-31.el7fdp.x86_64
ovn2.13-host-2.13.0-31.el7fdp.x86_64

# cat /var/log/ovn/ovn-controller.log 
2020-05-21T03:46:19.172Z|00001|vlog|INFO|opened log file /var/log/ovn/ovn-controller.log
2020-05-21T03:46:19.175Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting...
2020-05-21T03:46:19.175Z|00003|reconnect|INFO|unix:/run/openvswitch/db.sock: connected
2020-05-21T03:46:19.179Z|00004|main|INFO|OVS IDL reconnected, force recompute.
2020-05-21T03:46:19.179Z|00005|reconnect|INFO|tcp:20.0.10.25:6642: connecting...
2020-05-21T03:46:19.179Z|00006|main|INFO|OVNSB IDL reconnected, force recompute.
2020-05-21T03:46:19.180Z|00007|reconnect|INFO|tcp:20.0.10.25:6642: connected
2020-05-21T03:46:19.184Z|00008|ofctrl|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch
2020-05-21T03:46:19.184Z|00009|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
2020-05-21T03:46:19.184Z|00010|rconn|WARN|unix:/run/openvswitch/br-int.mgmt: connection failed (No such file or directory)
2020-05-21T03:46:19.184Z|00011|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: waiting 1 seconds before reconnect
2020-05-21T03:46:20.185Z|00012|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
2020-05-21T03:46:20.186Z|00013|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected
2020-05-21T03:46:20.191Z|00014|dpif_netlink|INFO|The kernel module does not support meters.
2020-05-21T03:46:20.194Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch
2020-05-21T03:46:20.194Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
2020-05-21T03:46:20.195Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connected
2020-05-21T03:46:40.793Z|00004|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connection closed by peer
2020-05-21T03:46:40.793Z|00015|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connection closed by peer
2020-05-21T03:46:41.040Z|00016|binding|INFO|Claiming lport ls1p1 for this chassis.
2020-05-21T03:46:41.040Z|00017|binding|INFO|ls1p1: Claiming 00:01:02:01:01:01
2020-05-21T03:46:41.610Z|00018|binding|INFO|Claiming lport ls1p2 for this chassis.
2020-05-21T03:46:41.610Z|00019|binding|INFO|ls1p2: Claiming 00:01:02:01:01:02
2020-05-21T03:46:41.793Z|00020|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
2020-05-21T03:46:41.793Z|00005|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
2020-05-21T03:46:41.794Z|00021|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected
2020-05-21T03:46:41.794Z|00006|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connected
[root@dell-per730-19 load_balance]# cat /var/log/ovn/ovn-controller.log |grep "OpenFlow error"

Comment 9 errata-xmlrpc 2020-07-15 13:00:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2941

Comment 11 Red Hat Bugzilla 2023-09-14 05:48:08 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days

Note You need to log in before you can comment on or make changes to this bug.