Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1976241

Summary: [4.8.z backport] Upgrade from Openshift 4.5 -> 4.6 Results in Orphaned Address sets
Product: OpenShift Container Platform Reporter: Antoni Segura Puimedon <asegurap>
Component: NetworkingAssignee: Jaime Caamaño Ruiz <jcaamano>
Networking sub component: ovn-kubernetes QA Contact: zhaozhanqi <zzhao>
Status: CLOSED ERRATA Docs Contact: Jeana Routh <jrouth>
Severity: high    
Priority: unspecified CC: akaris, anusaxen, arajapa, astoycos, bhershbe, jcaamano, pablo.iranzo, sgordon, yjoseph, zzhao
Version: 4.8   
Target Milestone: ---   
Target Release: 4.8.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
* The address set naming convention used in OVN-Kubernetes for {product-title} 4.5 was changed in {product-title} 4.6, but the migration of existing address sets to the new naming convention was not handled as part of the upgrade. Network policies that were created in version 4.5 with namespace selector criteria for their ingress or egress sections rely on matching old address sets that were not kept up-to-date with the pod IP addresses within such namespaces. These policies might not work correctly in 4.6 or later releases and might allow or drop unexpected traffic. + Previously, the workaround was to remove and recreate these policies. With this release, address sets with the old naming convention are removed, and policy ACLs referencing the old address sets are updated to reference the address sets following the new naming convention during the OVN-Kubernetes upgrade. Affected network policies created in version 4.5 work correctly again after upgrade. (link:https://bugzilla.redhat.com/show_bug.cgi?id=1976241[*BZ#1976241*])
 Story Points: ---
Clone Of: 1962387
: 1976242 (view as bug list) Environment:
Last Closed: 2021-08-31 16:17:10 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1962387    
Bug Blocks: 1976242    

Comment 3 zhaozhanqi 2021-08-12 13:18:30 UTC 
Tested this on 4.8.0-0.nightly-2021-08-11-223756 

steps:

1. setup cluster with ovn plugin
2. new namespace z1 and create test pod

oc create -f https://raw.githubusercontent.com/openshift/verification-tests/master/testdata/networking/list_for_pods.json

3. create network policy in this namespace
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-same-namespace
spec:
  podSelector:
  ingress:
  - from:
    - podSelector: {}

4. rsh into master pod and check the acl add address

 #ovn-nbctl list acl | grep namespace=z1 -C 4

_uuid               : ebf54c0c-1f16-42e4-9701-d6c2951185bb
action              : allow-related
direction           : to-lport
external_ids        : {Ingress_num="0", ipblock_cidr="false", l4Match=None, namespace=z1, policy=allow-same-namespace, policy_type=Ingress}
log                 : false
match               : "ip4.src == {$a14577698995162000448} && outport == @a17264831995179924701"
meter               : acl-logging
name                : z1_allow-same-namespace_0
priority            : 1001
severity            : info


sh-4.4# ovn-nbctl list address | grep a14577698995162000448 -B 4

_uuid               : 08ebe2c9-d96f-495e-a08d-b544c69d0c23
addresses           : ["10.129.2.26", "10.131.0.48", "172.30.141.98"]
external_ids        : {name=z1.allow-same-namespace.ingress.0_v4}
name                : a14577698995162000448


5. Create fake record, update above acl with another reference from a14577698995162000448 to a14577698995162000449

# ovn-nbctl set acl ebf54c0c-1f16-42e4-9701-d6c2951185bb 'match="ip4.src == {$a14577698995162000449} && outport == @a17264831995179924701"'



6. Create another address 

#ovn-nbctl create Address_Set name=a14577698995162000449 addresses=172.16.1.3 external_ids=name=z1.allow-same-namespace.ingress.0


sh-4.4# ovn-nbctl list acl | grep a14577698995162000449 -C 6

_uuid               : ebf54c0c-1f16-42e4-9701-d6c2951185bb
action              : allow-related
direction           : to-lport
external_ids        : {Ingress_num="0", ipblock_cidr="false", l4Match=None, namespace=z1, policy=allow-same-namespace, policy_type=Ingress}
log                 : false
match               : "ip4.src == {$a14577698995162000449} && outport == @a17264831995179924701"
meter               : acl-logging
name                : z1_allow-same-namespace_0
priority            : 1001
severity            : info

_uuid               : 1e406bed-2443-4035-9a4d-2af022336cce
sh-4.4# ovn-nbctl list address | grep a14577698995162000449 -B 4

_uuid               : 208b1229-89cd-4f19-bfc9-3f742ad1d970
addresses           : ["172.16.1.3"]
external_ids        : {name=z1.allow-same-namespace.ingress.0}
name                : a14577698995162000449

7. Restart ovnkube master pods. 

8. Check the ovn acl already be updated to correct reference a14577698995162000448

#ovn-nbctl list acl | grep namespace=z1 -C 4

_uuid               : ebf54c0c-1f16-42e4-9701-d6c2951185bb
action              : allow-related
direction           : to-lport
external_ids        : {Ingress_num="0", ipblock_cidr="false", l4Match=None, namespace=z1, policy=allow-same-namespace, policy_type=Ingress}
log                 : false
match               : "ip4.src == {$a14577698995162000448} && outport == @a17264831995179924701"
meter               : acl-logging
name                : z1_allow-same-namespace_0
priority            : 1001
severity            : info



However the fake address record still exit. 

sh-4.4# ovn-nbctl list address | grep a14577698995162000449 -B 4

_uuid               : 208b1229-89cd-4f19-bfc9-3f742ad1d970
addresses           : ["172.16.1.3"]
external_ids        : {name=z1.allow-same-namespace.ingress.0}
name                : a14577698995162000449
Comment 4 zhaozhanqi 2021-08-12 13:22:58 UTC 
I checked PR https://github.com/openshift/ovn-kubernetes/pull/635/files#diff-10e844883fb71e6b364c9039e12867797bf643a9189a82fdd2715d2ecc9b2fd8R12 the old address-set should be removed. please confirm this? thanks
Comment 6 Jaime Caamaño Ruiz 2021-08-16 11:34:22 UTC 
@zzhao the correct hashed name for z1.allow-same-namespace.ingress.0 is a9512858319328250577 not a14577698995162000449. See https://play.golang.org/p/-UOyx0RHkQb. Can you try with a9512858319328250577?
Comment 9 ximhan 2021-08-20 07:26:57 UTC 
OpenShift engineering has decided to NOT ship 4.8.6 on 8/23 due to the following issue.
https://bugzilla.redhat.com/show_bug.cgi?id=1995785
All the fixes part will be now included in 4.8.7 on 8/30.
Comment 13 errata-xmlrpc 2021-08-31 16:17:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.8.9 bug fix), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3247
Comment 14 Jeana Routh 2021-09-01 20:00:27 UTC 
Supplying doc text from the 4.8.9 Release Notes to test RN query steps.