Description of problem: The BZ https://bugzilla.redhat.com/show_bug.cgi?id=1757086 disabled conjunctions as it was buggy. We need to reenable conjunctions and make it work properly. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Conjunctive flows helps dramatically reducing the number of OpenFlow rules in certain scenarios in RHOSP, in particular where a lot of Neutron Security Group rules are used (referencing remote security groups and/or using port ranges filtering).
Hi Mark, as I know, conjunctions is a kind of flow in ovn, but how does it work? Could you tell us how we can verify the function of conjunctions in ovn? Thanks & Best Regards, Jianlin Shi
Sure thing. The way that OVN can generate conjunctive matches is through ACLs. Here's a quick example. Let's first set up a logical switch, and add ports to it. ovn-nbctl ls-add sw ovn-nbctl lsp-add sw p1 ovn-nbctl lsp-set-addresses p1 "00:00:00:00:00:02 192.168.0.2" ovn-nbctl lsp-add sw p2 ovn-nbctl lsp-set-addresses p2 "00:00:00:00:00:03 192.168.0.3" ovn-nbctl lsp-add sw p3 ovn-nbctl lsp-set-addresses p3 "00:00:00:00:00:04 192.168.0.4" Now bind the ports on the hypervisor ovs-vsctl add-port br-int p1 -- set Interface p1 external_ids:iface-id=p1 ovs-vsctl add-port br-int p2 -- set Interface p2 external_ids:iface-id=p2 ovs-vsctl add-port br-int p3 -- set Interface p3 external_ids:iface-id=p3 Now we create an address set and port group. Add the switch port addresses and ports to the address set and port group. ovn-nbctl create Address_Set name=set1 addresses="\"192.168.0.2\",\"192.168.0.3\",\"192.168.0.4\"" ovn-nbctl pg-add pg1 p1 p2 p3 Now create an ACL that references the address set and port group ovn-nbctl acl-add sw to-lport 1000 "inport==@pg1 && ip4 && ip4.src==\$set1" allow On the hypervisor, run ovs-ofctl dump-flows br-int | grep "table=44" Table 44 is the table with egress ACLs. You should see flows with "conjunction" actions and a flow with a "conj_id" action. On my system, it looks like this (note that the flows may not appear in the exact same order for you): cookie=0x0, duration=1.726s, table=44, n_packets=0, n_bytes=0, idle_age=1, priority=2000,ip,metadata=0x1,nw_src=192.168.0.3 actions=conjunction(2,2/2) cookie=0x0, duration=1.726s, table=44, n_packets=0, n_bytes=0, idle_age=1, priority=2000,ip,metadata=0x1,nw_src=192.168.0.4 actions=conjunction(2,2/2) cookie=0x0, duration=1.726s, table=44, n_packets=0, n_bytes=0, idle_age=1, priority=2000,ip,metadata=0x1,nw_src=192.168.0.2 actions=conjunction(2,2/2) cookie=0x0, duration=1.726s, table=44, n_packets=0, n_bytes=0, idle_age=1, priority=2000,ip,reg14=0x2,metadata=0x1 actions=conjunction(2,1/2) cookie=0x0, duration=1.726s, table=44, n_packets=0, n_bytes=0, idle_age=1, priority=2000,ip,reg14=0x1,metadata=0x1 actions=conjunction(2,1/2) cookie=0x0, duration=1.726s, table=44, n_packets=0, n_bytes=0, idle_age=1, priority=2000,ip,reg14=0x3,metadata=0x1 actions=conjunction(2,1/2) cookie=0xe6470355, duration=1.726s, table=44, n_packets=0, n_bytes=0, idle_age=1, priority=2000,conj_id=2,ip,metadata=0x1 actions=resubmit(,45) cookie=0xa3ccf6ef, duration=1.755s, table=44, n_packets=0, n_bytes=0, idle_age=1, priority=0,metadata=0x1 actions=resubmit(,45) Conjunctions work by breaking apart the individual portions of the ACL match and assigning each to a portion of the conjunction. In this example, we create a conjunction with ID 2. The port match (reg14) is part 1/2 of conjunction 2. The IP address match (nw_src) is part 2/2 of conjunction 2. If each part of the conjunction is matched, then the action is to resubmit to table 45. If conjunctions were disabled, you would probably see something like this instead: cookie=0x5e66fd41, duration=2.982s, table=44, n_packets=0, n_bytes=0, idle_age=2, priority=2000,ip,reg14=0x1,metadata=0x1,nw_src=192.168.0.2 actions=resubmit(,45) cookie=0x5e66fd41, duration=2.982s, table=44, n_packets=0, n_bytes=0, idle_age=2, priority=2000,ip,reg14=0x3,metadata=0x1,nw_src=192.168.0.2 actions=resubmit(,45) cookie=0x5e66fd41, duration=2.982s, table=44, n_packets=0, n_bytes=0, idle_age=2, priority=2000,ip,reg14=0x2,metadata=0x1,nw_src=192.168.0.3 actions=resubmit(,45) cookie=0x5e66fd41, duration=2.982s, table=44, n_packets=0, n_bytes=0, idle_age=2, priority=2000,ip,reg14=0x1,metadata=0x1,nw_src=192.168.0.3 actions=resubmit(,45) cookie=0x5e66fd41, duration=2.982s, table=44, n_packets=0, n_bytes=0, idle_age=2, priority=2000,ip,reg14=0x3,metadata=0x1,nw_src=192.168.0.3 actions=resubmit(,45) cookie=0x5e66fd41, duration=2.982s, table=44, n_packets=0, n_bytes=0, idle_age=2, priority=2000,ip,reg14=0x2,metadata=0x1,nw_src=192.168.0.4 actions=resubmit(,45) cookie=0x5e66fd41, duration=2.982s, table=44, n_packets=0, n_bytes=0, idle_age=2, priority=2000,ip,reg14=0x2,metadata=0x1,nw_src=192.168.0.2 actions=resubmit(,45) cookie=0x5e66fd41, duration=2.982s, table=44, n_packets=0, n_bytes=0, idle_age=2, priority=2000,ip,reg14=0x1,metadata=0x1,nw_src=192.168.0.4 actions=resubmit(,45) cookie=0x5e66fd41, duration=2.982s, table=44, n_packets=0, n_bytes=0, idle_age=2, priority=2000,ip,reg14=0x3,metadata=0x1,nw_src=192.168.0.4 actions=resubmit(,45) cookie=0x93f27b, duration=3.012s, table=44, n_packets=0, n_bytes=0, idle_age=3, priority=0,metadata=0x1 actions=resubmit(,45) Notice how a flow must be installed for each combination of port and IP address. In our case, we have only 3 ports and 3 IP addresses. If there are more ports and IP addresses, then the number of flows gets much larger when conjunctive matches are disabled. I hope this explanation helps.
I tried to run on ovn2.11-2.11.1-24.el7fdp.x86_64, no conjunction found: [root@dell-per740-12 bz1764032]# bash -x try.sh + systemctl start ovn-northd + systemctl start openvswitch + ovn-nbctl set-connection ptcp:6641 + ovn-sbctl set-connection ptcp:6642 + ovs-vsctl set Open_vSwitch . external-ids:system-id=hv0 external-ids:ovn-remote=tcp:20.0.30.26:6642 external-ids:ovn-encap-type=geneve external-ids:ovn-encap-ip=20.0.30.26 + systemctl start ovn-controller + ovn-nbctl ls-add sw + ovn-nbctl lsp-add sw p1 + ovn-nbctl lsp-set-addresses p1 '00:00:00:00:00:02 192.168.0.2' + ovn-nbctl lsp-add sw p2 + ovn-nbctl lsp-set-addresses p2 '00:00:00:00:00:03 192.168.0.3' + ovn-nbctl lsp-add sw p3 + ovn-nbctl lsp-set-addresses p3 '00:00:00:00:00:04 192.168.0.4' + ovs-vsctl add-port br-int p1 -- set Interface p1 type=internal external_ids:iface-id=p1 + ovs-vsctl add-port br-int p2 -- set Interface p2 type=internal external_ids:iface-id=p2 + ovs-vsctl add-port br-int p3 -- set Interface p3 type=internal external_ids:iface-id=p3 + ovn-nbctl create Address_Set name=set1 'addresses="192.168.0.2","192.168.0.3","192.168.0.4"' daae0868-7f03-4227-8605-267ee44e9e47 + ovn-nbctl pg-add pg1 p1 p2 p3 + ovn-nbctl acl-add sw to-lport 1000 'inport==@pg1 && ip4 && ip4.src==$set1' allow [root@dell-per740-12 bz1764032]# ovs-ofctl dump-flows br-int | grep "table=44" cookie=0xf532b46b, duration=18.804s, table=44, n_packets=0, n_bytes=0, idle_age=18, priority=2000,ip,reg14=0x1,metadata=0x1,nw_src=192.168.0.2 actions=resubmit(,45) cookie=0xf532b46b, duration=18.804s, table=44, n_packets=0, n_bytes=0, idle_age=18, priority=2000,ip,reg14=0x3,metadata=0x1,nw_src=192.168.0.2 actions=resubmit(,45) cookie=0xf532b46b, duration=18.804s, table=44, n_packets=0, n_bytes=0, idle_age=18, priority=2000,ip,reg14=0x2,metadata=0x1,nw_src=192.168.0.3 actions=resubmit(,45) cookie=0xf532b46b, duration=18.804s, table=44, n_packets=0, n_bytes=0, idle_age=18, priority=2000,ip,reg14=0x1,metadata=0x1,nw_src=192.168.0.3 actions=resubmit(,45) cookie=0xf532b46b, duration=18.804s, table=44, n_packets=0, n_bytes=0, idle_age=18, priority=2000,ip,reg14=0x3,metadata=0x1,nw_src=192.168.0.3 actions=resubmit(,45) cookie=0xf532b46b, duration=18.804s, table=44, n_packets=0, n_bytes=0, idle_age=18, priority=2000,ip,reg14=0x2,metadata=0x1,nw_src=192.168.0.4 actions=resubmit(,45) cookie=0xf532b46b, duration=18.804s, table=44, n_packets=0, n_bytes=0, idle_age=18, priority=2000,ip,reg14=0x2,metadata=0x1,nw_src=192.168.0.2 actions=resubmit(,45) cookie=0xf532b46b, duration=18.804s, table=44, n_packets=0, n_bytes=0, idle_age=18, priority=2000,ip,reg14=0x1,metadata=0x1,nw_src=192.168.0.4 actions=resubmit(,45) cookie=0xf532b46b, duration=18.804s, table=44, n_packets=0, n_bytes=0, idle_age=18, priority=2000,ip,reg14=0x3,metadata=0x1,nw_src=192.168.0.4 actions=resubmit(,45) cookie=0x189785ba, duration=18.804s, table=44, n_packets=0, n_bytes=0, idle_age=18, priority=0,metadata=0x1 actions=resubmit(,45) [root@dell-per740-12 bz1764032]# rpm -qa | grep -E "openvswitch|ovn" openvswitch2.11-2.11.0-35.el7fdp.x86_64 ovn2.11-2.11.1-24.el7fdp.x86_64 openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch ovn2.11-host-2.11.1-24.el7fdp.x86_64 ovn2.11-central-2.11.1-24.el7fdp.x86_64 please help to check
I can confirm that with ovn2.11-2.11.1-24, I see the same misbehavior. I did not see this problem with OVN master. It appears the necessary code is present, so I will need to figure out what is going wrong.
It appears that one of the patches that contributed towards disabling conjunctions in ovn2.11 was not reverted when re-enabling them. I have applied the patch locally and confirmed that this now works. I will let you know when this has been applied.
I have created a scratch build that fixed the issue for me. Can you check that this also fixed the issue for you? https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=25476831
(In reply to Mark Michelson from comment #10) > I have created a scratch build that fixed the issue for me. Can you check > that this also fixed the issue for you? > > https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=25476831 the build state is failed: State failed Result BuildError: error building package (arch ppc64le), mock exited with status 1; see root.log for more information
I performed another scratch build and it succeeded this time: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=25706093
testes with kernel provided in comment 12: [root@hp-dl380pg8-12 bz1764032]# bash -x rep.sh + systemctl start ovn-northd + systemctl start openvswitch + ovn-nbctl set-connection ptcp:6641 + ovn-sbctl set-connection ptcp:6642 + ovs-vsctl set Open_vSwitch . external-ids:system-id=hv0 external-ids:ovn-remote=tcp:20.0.68.26:6642 external-ids:ovn-encap-type=geneve external-ids:ovn-encap-ip=20.0.68.26 + systemctl start ovn-controller + ovn-nbctl ls-add sw + ovn-nbctl lsp-add sw p1 + ovn-nbctl lsp-set-addresses p1 '00:00:00:00:00:02 192.168.0.2' + ovn-nbctl lsp-add sw p2 + ovn-nbctl lsp-set-addresses p2 '00:00:00:00:00:03 192.168.0.3' + ovn-nbctl lsp-add sw p3 + ovn-nbctl lsp-set-addresses p3 '00:00:00:00:00:04 192.168.0.4' + ovs-vsctl add-port br-int p1 -- set Interface p1 type=internal external_ids:iface-id=p1 + ovs-vsctl add-port br-int p2 -- set Interface p2 type=internal external_ids:iface-id=p2 + ovs-vsctl add-port br-int p3 -- set Interface p3 type=internal external_ids:iface-id=p3 + ovn-nbctl create Address_Set name=set1 'addresses="192.168.0.2","192.168.0.3","192.168.0.4"' 295dd0ea-7242-4074-b352-652e4c0ff06a + ovn-nbctl pg-add pg1 p1 p2 p3 + ovn-nbctl acl-add sw to-lport 1000 'inport==@pg1 && ip4 && ip4.src==$set1' allow [root@hp-dl380pg8-12 bz1764032]# ovs-ofctl dump-flows br-int | grep "table=44" cookie=0x0, duration=8.079s, table=44, n_packets=0, n_bytes=0, idle_age=8, priority=2000,ip,metadata=0x1,nw_src=192.168.0.3 actions=conjunction(2,2/2) cookie=0x0, duration=8.079s, table=44, n_packets=0, n_bytes=0, idle_age=8, priority=2000,ip,metadata=0x1,nw_src=192.168.0.4 actions=conjunction(2,2/2) cookie=0x0, duration=8.078s, table=44, n_packets=0, n_bytes=0, idle_age=8, priority=2000,ip,metadata=0x1,nw_src=192.168.0.2 actions=conjunction(2,2/2) cookie=0x0, duration=8.079s, table=44, n_packets=0, n_bytes=0, idle_age=8, priority=2000,ip,reg14=0x2,metadata=0x1 actions=conjunction(2,1/2) cookie=0x0, duration=8.078s, table=44, n_packets=0, n_bytes=0, idle_age=8, priority=2000,ip,reg14=0x3,metadata=0x1 actions=conjunction(2,1/2) cookie=0x0, duration=8.078s, table=44, n_packets=0, n_bytes=0, idle_age=8, priority=2000,ip,reg14=0x1,metadata=0x1 actions=conjunction(2,1/2) <==== conjunction is re-enabled cookie=0x833b6fe9, duration=8.078s, table=44, n_packets=0, n_bytes=0, idle_age=8, priority=2000,conj_id=2,ip,metadata=0x1 actions=resubmit(,45) cookie=0x9940df01, duration=8.078s, table=44, n_packets=0, n_bytes=0, idle_age=8, priority=0,metadata=0x1 actions=resubmit(,45) [root@hp-dl380pg8-12 bz1764032]# rpm -qa | grep -E "openvswitch|ovn" ovn2.11-2.11.1-25.el7fdn.x86_64 openvswitch2.11-2.11.0-35.el7fdp.x86_64 ovn2.11-central-2.11.1-25.el7fdn.x86_64 openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch ovn2.11-host-2.11.1-25.el7fdn.x86_64
Verified on ovn2.11.1-33: [root@dell-per740-30 bz1764032]# bash -x rep.sh + systemctl start ovn-northd + systemctl start openvswitch + ovn-nbctl set-connection ptcp:6641 + ovn-sbctl set-connection ptcp:6642 + ovs-vsctl set Open_vSwitch . external-ids:system-id=hv0 external-ids:ovn-remote=tcp:20.0.43.26:6642 external-ids:ovn-encap-type=geneve external-ids:ovn-encap-ip=20.0.68.43 + systemctl start ovn-controller + ovn-nbctl ls-add sw + ovn-nbctl lsp-add sw p1 + ovn-nbctl lsp-set-addresses p1 '00:00:00:00:00:02 192.168.0.2' + ovn-nbctl lsp-add sw p2 + ovn-nbctl lsp-set-addresses p2 '00:00:00:00:00:03 192.168.0.3' + ovn-nbctl lsp-add sw p3 + ovn-nbctl lsp-set-addresses p3 '00:00:00:00:00:04 192.168.0.4' + ovs-vsctl add-port br-int p1 -- set Interface p1 type=internal external_ids:iface-id=p1 + ovs-vsctl add-port br-int p2 -- set Interface p2 type=internal external_ids:iface-id=p2 + ovs-vsctl add-port br-int p3 -- set Interface p3 type=internal external_ids:iface-id=p3 + ovn-nbctl create Address_Set name=set1 'addresses="192.168.0.2","192.168.0.3","192.168.0.4"' e731cc68-d1f2-43cb-83e7-fd84d6e6db88 + ovn-nbctl pg-add pg1 p1 p2 p3 + ovn-nbctl acl-add sw to-lport 1000 'inport==@pg1 && ip4 && ip4.src==$set1' allow [root@dell-per740-30 bz1764032]# ovs-ofctl dump-flows br-int | grep "table=44" cookie=0xbdef8349, duration=3.317s, table=44, n_packets=0, n_bytes=0, idle_age=3, priority=34000,metadata=0x1,dl_src=a6:86:91:42:04:8f actions=resubmit(,45) cookie=0x0, duration=3.323s, table=44, n_packets=0, n_bytes=0, idle_age=3, priority=2000,ip,metadata=0x1,nw_src=192.168.0.3 actions=conjunction(2,2/2) cookie=0x0, duration=3.323s, table=44, n_packets=0, n_bytes=0, idle_age=3, priority=2000,ip,metadata=0x1,nw_src=192.168.0.4 actions=conjunction(2,2/2) cookie=0x0, duration=3.317s, table=44, n_packets=0, n_bytes=0, idle_age=3, priority=2000,ip,metadata=0x1,nw_src=192.168.0.2 actions=conjunction(2,2/2) cookie=0x0, duration=3.323s, table=44, n_packets=0, n_bytes=0, idle_age=3, priority=2000,ip,reg14=0x2,metadata=0x1 actions=conjunction(2,1/2) cookie=0x0, duration=3.317s, table=44, n_packets=0, n_bytes=0, idle_age=3, priority=2000,ip,reg14=0x3,metadata=0x1 actions=conjunction(2,1/2) cookie=0x0, duration=3.317s, table=44, n_packets=0, n_bytes=0, idle_age=3, priority=2000,ip,reg14=0x1,metadata=0x1 actions=conjunction(2,1/2) <=== conjunction cookie=0x71aebeba, duration=3.317s, table=44, n_packets=0, n_bytes=0, idle_age=3, priority=2000,conj_id=2,ip,metadata=0x1 actions=resubmit(,45) cookie=0xdb711e2a, duration=3.317s, table=44, n_packets=0, n_bytes=0, idle_age=3, priority=0,metadata=0x1 actions=resubmit(,45) [root@dell-per740-30 bz1764032]# rpm -qa | grep -E "openvswitch|ovn" openvswitch2.11-2.11.0-47.el7fdp.x86_64 ovn2.11-2.11.1-33.el7fdp.x86_64 openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch ovn2.11-host-2.11.1-33.el7fdp.x86_64 ovn2.11-central-2.11.1-33.el7fdp.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0750
*** Bug 1768310 has been marked as a duplicate of this bug. ***