Bug 1840556

Summary: Bad conjunctions in ovn when using default security groups
Product: Red Hat OpenStack Reporter: Slawek Kaplonski <skaplons>
Component: openstack-neutronAssignee: Jakub Libosvar <jlibosva>
Status: CLOSED CURRENTRELEASE QA Contact: Roman Safronov <rsafrono>
Severity: high Docs Contact:
Priority: high    
Version: 16.1 (Train)CC: camorris, chrisw, dalvarez, jhardee, jlibosva, scohen
Target Milestone: ---Keywords: TestOnly, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1871931 1889282 (view as bug list) Environment:
Last Closed: 2022-05-16 14:46:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1871931    
Bug Blocks: 1889282    

Description Slawek Kaplonski 2020-05-27 08:12:03 UTC 
I spawned VM with Centos 8 (CentOS Linux release 8.1.1911 (Core)) and then installed tripleo-standalone on it.
After that, when I created 2 vms plugged to the private network, those VMs are not reachable. I'm using default security group with 1 additional rule to allow all ingress traffic:

openstack security group show 95f4dba8-dd53-410d-bc9e-a3daef7aa29b
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                                                                                                                                          |
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at      | 2020-05-27T07:55:58Z                                                                                                                                                                                           |
| description     | Default security group                                                                                                                                                                                         |
| id              | 95f4dba8-dd53-410d-bc9e-a3daef7aa29b                                                                                                                                                                           |
| location        | cloud='standalone', project.domain_id=, project.domain_name='Default', project.id='128cfb9fd9c445e4b84531fa301467a8', project.name='admin', region_name='regionOne', zone=                                     |
| name            | default                                                                                                                                                                                                        |
| project_id      | 128cfb9fd9c445e4b84531fa301467a8                                                                                                                                                                               |
| revision_number | 2                                                                                                                                                                                                              |
| rules           | created_at='2020-05-27T07:55:58Z', direction='ingress', ethertype='IPv4', id='14aee5d7-d5e0-49d7-a94f-70e561636e1e', remote_group_id='95f4dba8-dd53-410d-bc9e-a3daef7aa29b', updated_at='2020-05-27T07:55:58Z' |
|                 | created_at='2020-05-27T08:01:40Z', direction='ingress', ethertype='IPv4', id='7ab518b6-7147-45d1-b56b-d593b09c0ed6', updated_at='2020-05-27T08:01:40Z'                                                         |
|                 | created_at='2020-05-27T07:55:58Z', direction='egress', ethertype='IPv6', id='dc7d044c-7546-4ce3-ae57-123f74cf88b8', updated_at='2020-05-27T07:55:58Z'                                                          |
|                 | created_at='2020-05-27T07:55:58Z', direction='egress', ethertype='IPv4', id='e9923534-a000-4190-969f-0747c721ffda', updated_at='2020-05-27T07:55:58Z'                                                          |
|                 | created_at='2020-05-27T07:55:58Z', direction='ingress', ethertype='IPv6', id='ea4bc94e-09c9-4026-85b5-d5a40398f75c', remote_group_id='95f4dba8-dd53-410d-bc9e-a3daef7aa29b', updated_at='2020-05-27T07:55:58Z' |
| stateful        | True                                                                                                                                                                                                           |
| tags            | []                                                                                                                                                                                                             |
| updated_at      | 2020-05-27T08:01:40Z                                                                                                                                                                                           |
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

It seems that problem is caused by the SG rules with remote_group_id because when I removed them, all worked fine for me.

In OVN controller logs I see errors like:

2020-05-27T08:01:41.254Z|00032|ofp_actions|WARN|"conjunction" actions may be used along with "note" but not any other kind of action (such as the "set_field" action used here)
2020-05-27T08:01:41.254Z|00033|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x233): NXBAC_BAD_CONJUNCTION
OFPT_FLOW_MOD (OF1.3) (xid=0x233): ***decode error: NXBAC_BAD_CONJUNCTION***
00000000  04 0e 00 c0 00 00 02 33-00 00 00 00 00 00 00 00 |.......3........|
00000010  00 00 00 00 00 00 00 00-2c 02 00 00 00 00 07 d2 |........,.......|
00000020  ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|
00000030  00 01 00 4e 80 00 0a 02-08 00 00 01 1e 04 00 00 |...N............|
00000040  00 03 00 01 d3 08 00 00-00 22 00 00 00 2b 00 01 |........."...+..|
00000050  d9 20 00 00 00 00 00 00-00 00 00 00 00 00 00 00 |. ..............|
00000060  00 01 00 00 00 00 00 00-00 00 00 00 00 00 00 00 |................|
00000070  00 01 80 00 04 08 00 00-00 00 00 00 00 01 00 00 |................|
00000080  00 04 00 40 00 00 00 00-ff ff 00 18 00 00 23 20 |...@..........# |
00000090  00 07 18 40 00 01 de 10-00 00 00 00 00 00 00 01 |...@............|
000000a0  ff ff 00 10 00 00 23 20-00 0e ff f8 2d 00 00 00 |......# ....-...|
000000b0  ff ff 00 10 00 00 23 20-00 22 01 02 00 00 00 05 |......# ."......|


In ovs-vswitchd log:

2020-05-27T08:01:41.235Z|00047|ofp_actions|WARN|"conjunction" actions may be used along with "note" but not any other kind of action (such as the "set_field" action used here)
2020-05-27T08:01:41.235Z|00048|connmgr|INFO|br-int<->unix#0: sending NXBAC_BAD_CONJUNCTION error reply to OFPT_FLOW_MOD message
2020-05-27T08:01:41.236Z|00049|ofp_actions|WARN|"conjunction" actions may be used along with "note" but not any other kind of action (such as the "resubmit" action used here)
2020-05-27T08:01:41.236Z|00050|connmgr|INFO|br-int<->unix#0: sending NXBAC_BAD_CONJUNCTION error reply to OFPT_FLOW_MOD message
2020-05-27T08:01:41.236Z|00051|ofp_actions|WARN|"conjunction" actions may be used along with "note" but not any other kind of action (such as the "resubmit" action used here)
2020-05-27T08:01:41.236Z|00052|connmgr|INFO|br-int<->unix#0: sending NXBAC_BAD_CONJUNCTION error reply to OFPT_FLOW_MOD message
2020-05-27T08:01:41.236Z|00053|ofp_actions|WARN|"conjunction" actions may be used along with "note" but not any other kind of action (such as the "resubmit" action used here)
2020-05-27T08:01:41.236Z|00054|connmgr|INFO|br-int<->unix#0: sending NXBAC_BAD_CONJUNCTION error reply to OFPT_FLOW_MOD message
2020-05-27T08:01:41.236Z|00055|ofp_actions|WARN|"conjunction" actions may be used along with "note" but not any other kind of action (such as the "resubmit" action used here)
2020-05-27T08:01:41.236Z|00056|connmgr|INFO|br-int<->unix#0: sending NXBAC_BAD_CONJUNCTION error reply to OFPT_FLOW_MOD message
2020-05-27T08:01:41.236Z|00057|ofp_actions|WARN|"conjunction" actions may be used along with "note" but not any other kind of action (such as the "set_field" action used here)
2020-05-27T08:01:41.236Z|00058|connmgr|INFO|br-int<->unix#0: sending NXBAC_BAD_CONJUNCTION error reply to OFPT_FLOW_MOD message
2020-05-27T08:01:41.236Z|00059|ofp_actions|WARN|"conjunction" actions may be used along with "note" but not any other kind of action (such as the "set_field" action used here)
2020-05-27T08:01:41.236Z|00060|connmgr|INFO|br-int<->unix#0: sending NXBAC_BAD_CONJUNCTION error reply to OFPT_FLOW_MOD message
2020-05-27T08:01:41.236Z|00061|ofp_actions|WARN|"conjunction" actions may be used along with "note" but not any other kind of action (such as the "set_field" action used here)
2020-05-27T08:01:41.236Z|00062|connmgr|INFO|br-int<->unix#0: sending NXBAC_BAD_CONJUNCTION error reply to OFPT_FLOW_MOD message


===============
Versions of packages which I have installed:

[root@tripleo-centos vagrant]# podman images | grep neutron
docker.io/tripleomaster/centos-binary-neutron-server               current-tripleo   f19e459a94fd   9 days ago   1.19 GB
docker.io/tripleomaster/centos-binary-neutron-metadata-agent-ovn   current-tripleo   270685cbdbcd   9 days ago   1.01 GB

[root@tripleo-centos vagrant]# podman exec -it -u root neutron_api rpm -qa | grep neutron
python3-neutronclient-7.1.1-0.20200420111911.ac04e5d.el8.noarch
openstack-neutron-fwaas-16.0.0-0.20200428162227.94c0d54.el8.noarch
python3-neutron-lib-2.3.0-0.20200415130217.4f787b8.el8.noarch
python3-neutron-16.1.0-0.20200516222418.be8cab9.el8.noarch
openstack-neutron-16.1.0-0.20200516222418.be8cab9.el8.noarch
python3-neutron-fwaas-16.0.0-0.20200428162227.94c0d54.el8.noarch
python3-neutron-vpnaas-16.1.0-0.20200504032456.72cee57.el8.noarch
openstack-neutron-ml2-16.1.0-0.20200516222418.be8cab9.el8.noarch
puppet-neutron-17.0.0-0.20200516013831.9fb7b74.el8.noarch
openstack-neutron-common-16.1.0-0.20200516222418.be8cab9.el8.noarch
python3-neutron-dynamic-routing-16.0.0-0.20200505153928.44e77ea.el8.noarch
openstack-neutron-vpnaas-16.1.0-0.20200504032456.72cee57.el8.noarch

[root@tripleo-centos vagrant]# podman images | grep ovn
docker.io/tripleomaster/centos-binary-nova-novncproxy              current-tripleo   544acd4346da   9 days ago   1.22 GB
docker.io/tripleomaster/centos-binary-neutron-metadata-agent-ovn   current-tripleo   270685cbdbcd   9 days ago   1.01 GB
docker.io/tripleomaster/centos-binary-ovn-northd                   current-tripleo   8291433d7448   9 days ago   852 MB
docker.io/tripleomaster/centos-binary-ovn-northd                   pcmklatest        8291433d7448   9 days ago   852 MB
docker.io/tripleomaster/centos-binary-ovn-controller               current-tripleo   e8efc9a55bb2   9 days ago   734 MB

[root@tripleo-centos vagrant]# podman exec -it -u root ovn_controller rpm -aq | grep ovn
ovn-20.03.0-2.el8.x86_64
puppet-ovn-17.0.0-0.20200515234945.1d4c0ad.el8.noarch
ovn-host-20.03.0-2.el8.x86_64

[root@tripleo-centos vagrant]# podman exec -it -u root ovn_controller rpm -aq | grep openvswitch
openvswitch-2.12.0-1.el8.x86_64
python3-openvswitch-2.12.0-1.el8.x86_64
Comment 1 Jakub Libosvar 2020-05-27 09:22:30 UTC 
As discussed with Slawek over IRC, the integration bridge didn't have protocols set to required OpenFlow versions 13 and 15.

Other thing is to try out OVS 2.13 with OVN-20.03
Comment 3 Daniel Alvarez Sanchez 2020-09-18 14:19:09 UTC 
Patch WIP here:

http://patchwork.ozlabs.org/project/ovn/patch/1599773812-28259-1-git-send-email-dceara@redhat.com/
Comment 5 Daniel Alvarez Sanchez 2020-12-16 10:46:53 UTC 
The fix landed already: https://errata.devel.redhat.com/advisory/65530
Comment 7 Roman Safronov 2021-01-27 10:02:42 UTC 
Verified that the described problem does not happen on RHOS-16.1-RHEL-8-20210120.n.1. With ingress rule in default security group VMs are accessible and no errors in logs. Tried also after restarting OVN controller.