2051954 – Allow changing of policyAuditConfig ratelimit post-deployment

Bug 2051954 - Allow changing of policyAuditConfig ratelimit post-deployment

Summary: Allow changing of policyAuditConfig ratelimit post-deployment

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.11
Hardware:	Unspecified
OS:	Unspecified
Priority:	low
Severity:	low
Target Milestone:	---
Target Release:	4.11.0
Assignee:	Andreas Karis
QA Contact:	Arti Sood
Docs Contact:
URL:
Whiteboard:
Depends On:	2073462 1939524
Blocks:
TreeView+	depends on / blocked

Reported:	2022-02-08 11:55 UTC by Andreas Karis
Modified:	2022-08-10 10:48 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:	policyAuditConfig ratelimit can now be changed post-deployment and changes are passed down from OVN-Kubernetes to the OVN layer. However, a bug in OVN (https://bugzilla.redhat.com/show_bug.cgi?id=1939524) must be resolved for this for work correctly. As a workaround, administrators must restart all ovnkube-node pods for ratelimit updates to take effect.
Clone Of:
Environment:
Last Closed:	2022-08-10 10:47:58 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	openshift ovn-kubernetes pull 1029	None	open	Bump OVN to ovn-2021-21.12.0-42.el8fdp	2022-04-08 09:46:01 UTC
Github	openshift ovn-kubernetes pull 947	None	Merged	Bug 2011525: [DownstreamMerge] Downstream merge 08-02-2022	2022-04-08 09:06:21 UTC
Github	ovn-org ovn-kubernetes pull 2806	None	Merged	ovnkube master: Reflect changes to acl-logging-rate-limit in ovn db	2022-04-08 09:06:24 UTC
Red Hat Product Errata	RHSA-2022:5069	None	None	None	2022-08-10 10:48:20 UTC

Description Andreas Karis 2022-02-08 11:55:46 UTC

Description of problem:
When changing the policyAuditConfig ratelimit post-deployment:
~~~
oc edit  network.operator cluster
(...)
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  annotations:
    networkoperator.openshift.io/ovn-cluster-initiator: 10.0.157.13
  creationTimestamp: "2022-02-08T09:40:34Z"
  generation: 83
  name: cluster
  resourceVersion: "54659"
  uid: 8887f6c6-d077-475d-b9ce-cf1ed39f2d1c
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  defaultNetwork:
    ovnKubernetesConfig:
      gatewayConfig:
        routingViaHost: false
      genevePort: 6081
      mtu: 8901
      policyAuditConfig:
        destination: "null"
        maxFileSize: 50
        rateLimit: 1
        syslogFacility: local0
~~~

Even though ovnkube master is restarted with --acl-logging-rate-limit 1:
~~~
sh-4.4# ps aux | grep ovnkube | grep master
root           1  0.1  0.6 1645428 100900 ?      Ssl  11:13   0:02 /usr/bin/ovnkube --init-master ip-10-0-157-13.us-west-2.compute.internal --config-file=/run/ovnkube-config/ovnkube.conf --ovn-empty-lb-events --loglevel 4 --metrics-bind-address 127.0.0.1:29102 --metrics-enable-pprof --gateway-mode shared --gateway-interface br-ex --sb-address ssl:10.0.157.13:9642,ssl:10.0.191.160:9642,ssl:10.0.254.62:9642 --sb-client-privkey /ovn-cert/tls.key --sb-client-cert /ovn-cert/tls.crt --sb-client-cacert /ovn-ca/ca-bundle.crt --sb-cert-common-name ovn --nb-address ssl:10.0.157.13:9641,ssl:10.0.191.160:9641,ssl:10.0.254.62:9641 --nb-client-privkey /ovn-cert/tls.key --nb-client-cert /ovn-cert/tls.crt --nb-client-cacert /ovn-ca/ca-bundle.crt --nbctl-daemon-mode --nb-cert-common-name ovn --enable-multicast --disable-snat-multiple-gws --acl-logging-rate-limit 1
~~~

The meter band is not updated:
~~~
[akaris@linux 2048352]$ oc rsh -n openshift-ovn-kubernetes $(get_leader_pod)
Defaulted container "northd" out of: northd, nbdb, kube-rbac-proxy, sbdb, ovnkube-master, ovn-dbchecker
sh-4.4# ovn-nbctl list meter 
_uuid               : 857bbc5c-6638-4ce2-a2b6-20827ae4d915
bands               : [cbf1280d-4549-4cad-8742-1ddc4c623466]
external_ids        : {}
fair                : true
name                : acl-logging
unit                : pktps
sh-4.4# ovn-nbctl list meter_band
_uuid               : cbf1280d-4549-4cad-8742-1ddc4c623466
action              : drop
burst_size          : 0
external_ids        : {}
rate                : 20
~~~

In theory, according to the documentation, these parameters can only be set on installation:
https://github.com/openshift/cluster-network-operator/blob/ee498c60dbb91dd06d274e1521c97419362499ff/README.md#configuring-ovnkubernetes

But we do change some of these parameters, and the fact that the rate is not updated is not consistent as we *do* for example accept changes to maxFileSize and we reconfigure the pods accordingly.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

That can easily be added here: https://github.com/openshift/ovn-kubernetes/blob/c0a3ddb51dfdf8916350714003dfcd3fcabc9e87/go-controller/pkg/ovn/master.go#L283

Comment 2 Surya Seetharaman 2022-02-09 19:35:04 UTC

Question, like you have stated these configs are install-time only changes, did we have a use case where we needed to change the rate-limiting at run time? Feels off to document something and suddenly change a field half way to be able to do it post-install time.

I guess real question is is this a bug?

Comment 3 Andreas Karis 2022-02-09 20:41:33 UTC

The observation actually came from QA. 

The cluster operator allows changing this field:
~~~
oc edit  network.operator cluster
(...)
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  annotations:
    networkoperator.openshift.io/ovn-cluster-initiator: 10.0.157.13
  creationTimestamp: "2022-02-08T09:40:34Z"
  generation: 83
  name: cluster
  resourceVersion: "54659"
  uid: 8887f6c6-d077-475d-b9ce-cf1ed39f2d1c
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  defaultNetwork:
    ovnKubernetesConfig:
      gatewayConfig:
        routingViaHost: false
      genevePort: 6081
      mtu: 8901
      policyAuditConfig:
        destination: "null"
        maxFileSize: 50
        rateLimit: 1  # <----------- this here
(...)
~~~

And it will in turn restart all ovnkube master nodes, and it will also set:
~~~
/usr/bin/ovnkube --init-master (...) --acl-logging-rate-limit 1
~~~

Just that the ovnkube-master code lacked the capability to actually reflect that parameter update in the ovn nbdb (it now works upstream).

Changing the following also always worked:
~~~
      policyAuditConfig:
        maxFileSize: 50   # <---- that can be updated and it works
~~~

So no it's technically not a bug (it would have been better to open a jira issue), but a tiny feature / RFE to make the settings more consistent (as in why would updating maxFileSize work, but not rateLimit).

With that said, our downstream doc is not in line with the fact that (some of?) the fields under policyAuditConfig: can actually be changed post-installation. But can customers even see https://github.com/openshift/cluster-network-operator/blob/master/README.md (it's a private repo, isn't it?)

So in an ideal world, we'd:
- either update the documentation to reflect that values can be changed
- or not accept changes to policyAuditConfig in cluster-network-operator, and emit a warning.

What disturbed/disturbs me is the in-between state where CNO accepts a change, acts upon it and restarts the pods with a CLI parameter with a new value but with no effect :-)

Comment 9 Andreas Karis 2022-03-02 15:40:38 UTC

I think this might be https://bugzilla.redhat.com/show_bug.cgi?id=1939524

Comment 11 Andreas Karis 2022-03-03 11:14:50 UTC

With that said, that's for the allow logs. For the deny logs, I indeed run into https://bugzilla.redhat.com/show_bug.cgi?id=1939524

See meter 4, which is not updated:

sh-4.4# ovs-ofctl -O OpenFlow15 meter-stats br-int
OFPST_METER reply (OF1.5) (xid=0x2):
meter:6 flow_count:0 packet_in_count:159767 byte_in_count:12887874 duration:1012.615s bands:
0: packet_count:159547 byte_count:12871842

meter:1 flow_count:0 packet_in_count:0 byte_in_count:0 duration:4235.298s bands:
0: packet_count:0 byte_count:0

meter:4 flow_count:0 packet_in_count:3 byte_in_count:288 duration:4235.298s bands:
0: packet_count:0 byte_count:0

meter:5 flow_count:0 packet_in_count:3372 byte_in_count:249528 duration:2882.281s bands:
0: packet_count:3018 byte_count:223332

meter:2 flow_count:0 packet_in_count:6 byte_in_count:252 duration:4235.298s bands:
0: packet_count:0 byte_count:0

meter:3 flow_count:0 packet_in_count:0 byte_in_count:0 duration:4235.298s bands:
0: packet_count:0 byte_count:0
sh-4.4# 


sh-4.4# ovs-ofctl -O OpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=6 pktps stats bands=
type=drop rate=2

meter=1 pktps stats bands=
type=drop rate=25

meter=4 pktps stats bands=
type=drop rate=25

meter=5 pktps stats bands=
type=drop rate=20

meter=2 pktps stats bands=
type=drop rate=25

meter=3 pktps stats bands=
type=drop rate=25



I have to delete the ovnkube-node pod:
oc delete pod -n openshift-ovn-kubernetes ovnkube-node-zx7nc

And after that:

sh-4.4# ovs-ofctl -O OpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=4 pktps stats bands=
type=drop rate=25

meter=6 pktps stats bands=
type=drop rate=25

meter=1 pktps stats bands=
type=drop rate=25

meter=2 pktps stats bands=
type=drop rate=2

meter=5 pktps stats bands=
type=drop rate=2

meter=3 pktps stats bands=
type=drop rate=25
sh-4.4# ovs-ofctl -O OpenFlow15 meter-stats br-int
OFPST_METER reply (OF1.5) (xid=0x2):
meter:4 flow_count:0 packet_in_count:0 byte_in_count:0 duration:228.147s bands:
0: packet_count:0 byte_count:0

meter:6 flow_count:0 packet_in_count:0 byte_in_count:0 duration:228.147s bands:
0: packet_count:0 byte_count:0

meter:1 flow_count:0 packet_in_count:0 byte_in_count:0 duration:228.147s bands:
0: packet_count:0 byte_count:0

meter:2 flow_count:0 packet_in_count:2621 byte_in_count:193954 duration:228.147s bands:
0: packet_count:2484 byte_count:183816

meter:5 flow_count:0 packet_in_count:6006 byte_in_count:484484 duration:228.147s bands:
0: packet_count:5995 byte_count:483630

meter:3 flow_count:0 packet_in_count:0 byte_in_count:0 duration:228.147s bands:
0: packet_count:0 byte_count:0



And now, when I test with:

for i in `seq 0 1000`; do curl --connect-timeout 1 10.129.2.10:9376 & sleep 0.1; done

I indeed get the correct rate:

2022-03-03T11:13:23.074Z|00130|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace", verdict=drop, severity=alert: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:0b,dl_dst=0a:58:0a:81:02:0a,nw_src=10.129.2.11,nw_dst=10.129.2.10,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=43170,tp_dst=9376,tcp_flags=syn
2022-03-03T11:13:23.579Z|00131|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace", verdict=drop, severity=alert: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:0b,dl_dst=0a:58:0a:81:02:0a,nw_src=10.129.2.11,nw_dst=10.129.2.10,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=43182,tp_dst=9376,tcp_flags=syn
2022-03-03T11:13:24.083Z|00132|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace", verdict=drop, severity=alert: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:0b,dl_dst=0a:58:0a:81:02:0a,nw_src=10.129.2.11,nw_dst=10.129.2.10,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=43194,tp_dst=9376,tcp_flags=syn
2022-03-03T11:13:24.586Z|00133|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace", verdict=drop, severity=alert: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:0b,dl_dst=0a:58:0a:81:02:0a,nw_src=10.129.2.11,nw_dst=10.129.2.10,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=43206,tp_dst=9376,tcp_flags=syn

Comment 16 Andreas Karis 2022-05-02 14:40:29 UTC

Create project, namespaces, deployments:
~~~
oc new-project test
oc create deployment hello-node --image=k8s.gcr.io/e2e-test-images/agnhost:2.33 -- /agnhost serve-hostname
oc expose deployment hello-node --port=9376
oc expose svc hello-node
oc scale deployment -n test hello-node --replicas=2
oc new-project test2
oc create deployment hello-node --image=k8s.gcr.io/e2e-test-images/agnhost:2.33 -- /agnhost serve-hostname
oc scale deployment -n test2 hello-node --replicas=2
oc annotate namespace test k8s.ovn.org/acl-logging='{ "deny": "alert", "allow": "notice" }'
oc annotate namespace test2 k8s.ovn.org/acl-logging='{ "deny": "alert", "allow": "notice" }'
~~~

Create the networkpolicy:
~~~
cat <<'EOF' | oc apply -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-same-namespace
  namespace: test
spec:
  podSelector: {}
  ingress:
  - from:
    - podSelector: {}
EOF
~~~

Verify the policies:
~~~
sh-4.4# ovn-nbctl find acl "name>=test_allow-from-openshift-ingress"
_uuid               : 3d738c88-ebc3-4e54-be4e-ee5f002a4477
action              : drop
direction           : to-lport
external_ids        : {default-deny-policy-type=Ingress}
label               : 0
log                 : true
match               : "outport == @a18007334074686647077_ingressDefaultDeny"
meter               : acl-logging
name                : test_allow-same-namespace
options             : {}
priority            : 1000
severity            : alert

_uuid               : 31cf9e1c-a961-4ac5-944b-5402bdedbc84
action              : allow-related
direction           : to-lport
external_ids        : {Ingress_num="0", ipblock_cidr="false", l4Match=None, namespace=test, policy=allow-same-namespace, policy_type=Ingress}
label               : 0
log                 : true
match               : "ip4.src == {$a3938097884066350661} && outport == @a17042190824695124102"
meter               : acl-logging
name                : test_allow-same-namespace_0
options             : {}
priority            : 1001
severity            : notice

_uuid               : c8371ff4-e611-41e9-8573-18cf928d7b0b
action              : drop
direction           : from-lport
external_ids        : {default-deny-policy-type=Egress}
label               : 0
log                 : true
match               : "inport == @a18007334074686647077_egressDefaultDeny"
meter               : acl-logging
name                : test_allow-same-namespace
options             : {apply-after-lb="true"}
priority            : 1000
severity            : alert
~~~

Verify the current meter band:
~~~
[akaris@linux origin (egressip-tests-option3)]$ oc rsh -n openshift-ovn-kubernetes $(get_leader_pod)
Defaulted container "northd" out of: northd, nbdb, kube-rbac-proxy, sbdb, ovnkube-master, ovn-dbchecker
sh-4.4# ovn-nbctl list meter_band
_uuid               : 85176842-696d-4cb7-9bdd-6a34b874c16e
action              : drop
burst_size          : 0
external_ids        : {}
rate                : 25

_uuid               : 0187f28f-9e4c-443a-9052-6e927b94d6cc
action              : drop
burst_size          : 0
external_ids        : {}
rate                : 20
~~~

Update the cluster config:
~~~
oc edit  network.operator cluster
(...)
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  annotations:
    networkoperator.openshift.io/ovn-cluster-initiator: 10.0.157.13
  creationTimestamp: "2022-02-08T09:40:34Z"
  generation: 83
  name: cluster
  resourceVersion: "54659"
  uid: 8887f6c6-d077-475d-b9ce-cf1ed39f2d1c
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  defaultNetwork:
    ovnKubernetesConfig:
      gatewayConfig:
        routingViaHost: false
      genevePort: 6081
      mtu: 8901
      policyAuditConfig:
        destination: "null"
        maxFileSize: 50
        rateLimit: 1  # <----------- this here
(...)
~~~

Verify the current meter_band after the ovnkube-master pods restarted (oc get pods -n openshift-ovn-kubernetes):
~~~
[akaris@linux origin (egressip-tests-option3)]$  oc rsh -n openshift-ovn-kubernetes $(get_leader_pod)
Defaulted container "northd" out of: northd, nbdb, kube-rbac-proxy, sbdb, ovnkube-master, ovn-dbchecker
sh-4.4# ovn-nbctl list meter_band
_uuid               : 633400e8-8cec-48e2-b027-bb3e51b356c5
action              : drop
burst_size          : 0
external_ids        : {}
rate                : 1

_uuid               : 85176842-696d-4cb7-9bdd-6a34b874c16e
action              : drop
burst_size          : 0
external_ids        : {}
rate                : 25
~~~

Verify the current meter rate - updated:
~~~
[akaris@linux origin (egressip-tests-option3)]$ oc get pods -n test -o wide
NAME                          READY   STATUS    RESTARTS   AGE     IP            NODE                                         NOMINATED NODE   READINESS GATES
hello-node-78bd88f59b-9cb75   1/1     Running   0          4m57s   10.129.2.10   ip-10-0-210-88.us-east-2.compute.internal    <none>           <none>
hello-node-78bd88f59b-wsqkn   1/1     Running   0          24m     10.131.0.14   ip-10-0-151-112.us-east-2.compute.internal   <none>           <none>
[akaris@linux origin (egressip-tests-option3)]$ 
[akaris@linux origin (egressip-tests-option3)]$ oc debug node/ip-10-0-151-112.us-east-2.compute.internal
sh-4.4# chroot /host
sh-4.4#  ovs-ofctl -O OpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=4 pktps stats bands=
type=drop rate=1

meter=1 pktps stats bands=
type=drop rate=25

meter=5 pktps stats bands=
type=drop rate=1

meter=2 pktps stats bands=
type=drop rate=25

meter=3 pktps stats bands=
type=drop rate=25
~~~

Update the cluster config:
~~~
oc edit  network.operator cluster
(...)
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  annotations:
    networkoperator.openshift.io/ovn-cluster-initiator: 10.0.157.13
  creationTimestamp: "2022-02-08T09:40:34Z"
  generation: 83
  name: cluster
  resourceVersion: "54659"
  uid: 8887f6c6-d077-475d-b9ce-cf1ed39f2d1c
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  defaultNetwork:
    ovnKubernetesConfig:
      gatewayConfig:
        routingViaHost: false
      genevePort: 6081
      mtu: 8901
      policyAuditConfig:
        destination: "null"
        maxFileSize: 50
        rateLimit: 3  # <----------- this here
(...)
~~~

Wait for the ovnkube-master pods to restart:
~~~
oc get pods -n openshift-ovn-kubernetes --watch
~~~

After all 3 ovnkube-master pods restarted, verify the meter rate:
~~~
[akaris@linux origin (egressip-tests-option3)]$ oc debug node/ip-10-0-151-112.us-east-2.compute.internal
sh-4.4# chroot /host
sh-4.4#  ovs-ofctl -O OpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=4 pktps stats bands=
type=drop rate=3

meter=1 pktps stats bands=
type=drop rate=25

meter=5 pktps stats bands=
type=drop rate=3

meter=2 pktps stats bands=
type=drop rate=25

meter=3 pktps stats bands=
type=drop rate=25
~~~

Now, check the average meter rate - account for burst, so this **must** be a sustained load (hence the seq and the 100000 requests, o.k., perhaps that's a bit much ;-) ):
~~~
[akaris@linux origin (egressip-tests-option3)]$ oc get pods -n test -o wide
NAME                          READY   STATUS    RESTARTS   AGE   IP            NODE                                         NOMINATED NODE   READINESS GATES
hello-node-78bd88f59b-9cb75   1/1     Running   0          23m   10.129.2.10   ip-10-0-210-88.us-east-2.compute.internal    <none>           <none>
hello-node-78bd88f59b-wsqkn   1/1     Running   0          42m   10.131.0.14   ip-10-0-151-112.us-east-2.compute.internal   <none>           <none>
hello-node-78bd88f59b-wsqkn~ $ for i in `seq 0 100000`; do curl --connect-timeout 1 10.131.0.14:9376 ; done
~~~

At the same time, observe on the host:
~~~
[akaris@linux origin (egressip-tests-option3)]$  oc debug node/ip-10-0-151-112.us-east-2.compute.internal
Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), hostPath volumes (volume "host"), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Starting pod/ip-10-0-151-112us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.151.112
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-4.4# tail -f /var/log/ovn/acl-audit-log.log  -n0
~~~

The result - note that I filtered unrelated messages:
~~~
2022-05-02T14:36:57.123Z|01153|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=42230,tp_dst=9376,tcp_flags=syn
2022-05-02T14:36:57.456Z|01154|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=42374,tp_dst=9376,tcp_flags=ack
2022-05-02T14:36:57.790Z|01155|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=42524,tp_dst=9376,tcp_flags=syn

2022-05-02T14:36:58.124Z|01156|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=42676,tp_dst=9376,tcp_flags=syn
2022-05-02T14:36:58.458Z|01157|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=42828,tp_dst=9376,tcp_flags=syn
(...)
2022-05-02T14:36:58.792Z|01160|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=42978,tp_dst=9376,tcp_flags=syn

2022-05-02T14:36:59.123Z|01161|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=43128,tp_dst=9376,tcp_flags=ack
2022-05-02T14:36:59.456Z|01162|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=43270,tp_dst=9376,tcp_flags=ack
2022-05-02T14:36:59.791Z|01163|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=43418,tp_dst=9376,tcp_flags=ack

2022-05-02T14:37:00.123Z|01164|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=43570,tp_dst=9376,tcp_flags=ack
2022-05-02T14:37:00.456Z|01165|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=43718,tp_dst=9376,tcp_flags=ack
2022-05-02T14:37:00.790Z|01166|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=43866,tp_dst=9376,tcp_flags=ack

2022-05-02T14:37:01.124Z|01167|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=44010,tp_dst=9376,tcp_flags=syn
2022-05-02T14:37:01.456Z|01168|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=44152,tp_dst=9376,tcp_flags=ack
2022-05-02T14:37:01.790Z|01169|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=44290,tp_dst=9376,tcp_flags=ack

2022-05-02T14:37:02.123Z|01170|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=44430,tp_dst=9376,tcp_flags=ack
2022-05-02T14:37:02.457Z|01171|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=44560,tp_dst=9376,tcp_flags=syn
2022-05-02T14:37:02.792Z|01172|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=44708,tp_dst=9376,tcp_flags=syn

2022-05-02T14:37:03.124Z|01173|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=44866,tp_dst=9376,tcp_flags=syn
2022-05-02T14:37:03.457Z|01174|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=45028,tp_dst=9376,tcp_flags=syn
2022-05-02T14:37:03.792Z|01175|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=45176,tp_dst=9376,tcp_flags=syn

2022-05-02T14:37:04.123Z|01176|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=45322,tp_dst=9376,tcp_flags=ack
2022-05-02T14:37:04.456Z|01177|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=45468,tp_dst=9376,tcp_flags=ack
~~~

Comment 17 Andreas Karis 2022-05-02 14:56:00 UTC

I then changed the meter rate to 2:
~~~
sh-4.4# ovs-ofctl -O OpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=4 pktps stats bands=
type=drop rate=2

meter=1 pktps stats bands=
type=drop rate=25

meter=5 pktps stats bands=
type=drop rate=2

meter=2 pktps stats bands=
type=drop rate=25

meter=3 pktps stats bands=
type=drop rate=25
~~~

Same result:
~~~
022-05-02T14:52:59.077Z|01815|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=38836,tp_dst=9376,tcp_flags=fin|ack
2022-05-02T14:52:59.577Z|01816|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=39062,tp_dst=9376,tcp_flags=syn

2022-05-02T14:53:00.077Z|01817|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=39298,tp_dst=9376,tcp_flags=ack
2022-05-02T14:53:00.577Z|01818|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=39524,tp_dst=9376,tcp_flags=syn

2022-05-02T14:53:01.077Z|01819|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=39742,tp_dst=9376,tcp_flags=ack
2022-05-02T14:53:01.577Z|01820|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=39956,tp_dst=9376,tcp_flags=ack

2022-05-02T14:53:02.077Z|01821|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=40176,tp_dst=9376,tcp_flags=ack
2022-05-02T14:53:02.579Z|01822|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=40366,tp_dst=9376,tcp_flags=syn

2022-05-02T14:53:03.077Z|01823|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=40584,tp_dst=9376,tcp_flags=ack
2022-05-02T14:53:03.577Z|01824|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=40812,tp_dst=9376,tcp_flags=syn

2022-05-02T14:53:04.077Z|01825|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=41036,tp_dst=9376,tcp_flags=syn
2022-05-02T14:53:04.577Z|01826|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=41256,tp_dst=9376,tcp_flags=ack

2022-05-02T14:53:05.079Z|01827|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=41478,tp_dst=9376,tcp_flags=syn
2022-05-02T14:53:05.579Z|01828|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=41690,tp_dst=9376,tcp_flags=syn
(...)

2022-05-02T14:53:06.078Z|01831|acl_log(ovn_pinctrl0)|INFO|name="test_allow-same-namespace_0", verdict=allow, severity=notice, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:01,dl_dst=0a:58:0a:83:00:0e,nw_src=10.129.2.10,nw_dst=10.131.0.14,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=41918,tp_dst=9376,tcp_flags=syn

~~~

Comment 18 Andreas Karis 2022-05-02 14:58:48 UTC

According to my tests, this looks good. As I said earlier, it's important to sustain that traffic for a while so that we avoid  the impact of bursts on the rate limiter, but after creating a few connections for a sustained time, it looks good to me.

Comment 21 errata-xmlrpc 2022-08-10 10:47:58 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069

Note You need to log in before you can comment on or make changes to this bug.