+++ This bug was initially created as a clone of Bug #1883419 +++ +++ This bug was initially created as a clone of Bug #1883166 +++ Due to the use of pools, there may be ports that belongs to pools associated to deleted nodes. Thus those ports are wasted as they cannot be used.
Verified on 4.4.0-0.nightly-2020-10-17-051433 over OSP13 with OVS (2020-09-16.1) and on 4.4.0-0.nightly-2020-10-20-061440 over OSP16.1 with OVN (RHOS-16.1-RHEL-8-20201007.n.0) # OSP13 with OVS: (shiftstack) [stack@undercloud-0 ~]$ cat core_puddle_version 2020-09-16.1 (shiftstack) [stack@undercloud-0 ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.0-0.nightly-2020-10-17-051433 True False 128m Cluster version is 4.4.0-0.nightly-2020-10-17-051433 # Given below status: (shiftstack) [stack@undercloud-0 ~]$ openstack network trunk list +--------------------------------------+-----------------------------+--------------------------------------+-------------+ | ID | Name | Parent Port | Description | +--------------------------------------+-----------------------------+--------------------------------------+-------------+ | 49c8aa0c-19b0-4f77-9500-8fa929f0a85f | ostest-jtbvm-master-trunk-1 | c14ef1b1-7cdd-4e93-8229-9da00795c511 | | | 9b38b271-61de-48a5-ad00-2d435fc2d752 | ostest-jtbvm-master-trunk-0 | 556a8497-72df-489e-a9ea-e767dd1ef974 | | | 9c05c56f-602b-49dd-a589-516bbf2a2add | ostest-jtbvm-worker-cs8zv | f6cb4568-fbd8-4c7f-bc37-e2f182ccf961 | | | c4f5bce1-f1ff-440c-a4dc-7948879384b6 | ostest-jtbvm-master-trunk-2 | 1533d36f-1d83-43d9-b6a0-3565dd192f2f | | | cee09a38-c452-4fd6-af9e-e9f4a86dccb4 | ostest-jtbvm-worker-mdrvl | a0633388-4d51-49f8-bc9b-1dd64f0d0977 | | | e8b93144-79f2-4cfd-8027-3998113ab064 | ostest-jtbvm-worker-ks9zl | e9e4f8d8-d2fb-41ec-8dfd-ce546bd2fe69 | | +--------------------------------------+-----------------------------+--------------------------------------+-------------+ # where ostest-jtbvm-worker-cs8zv has: (shiftstack) [stack@undercloud-0 ~]$ openstack network subport list --trunk 9c05c56f-602b-49dd-a589-516bbf2a2add +--------------------------------------+-------------------+-----------------+ | Port | Segmentation Type | Segmentation ID | +--------------------------------------+-------------------+-----------------+ | c32ccfca-a9f2-4952-8ef9-71f318a325de | vlan | 22 | | 48fe941d-6843-4824-896b-22b7a60dc1c9 | vlan | 84 | | 4c429ffe-c64a-433c-b47a-1a51ab5b6013 | vlan | 383 | | ff60382f-084b-470f-8189-21f4ee376896 | vlan | 709 | | af5bb1cf-db99-4dad-884e-a9c1bf278342 | vlan | 3860 | | 5281dc0f-0234-4e5c-a388-7d84d0be4faf | vlan | 4042 | +--------------------------------------+-------------------+-----------------+ # setting replica to 2. ostest-jtbvm-worker-cs8zv dissapears together with its trunk: (shiftstack) [stack@undercloud-0 ~]$ openstack network trunk list +--------------------------------------+-----------------------------+--------------------------------------+-------------+ | ID | Name | Parent Port | Description | +--------------------------------------+-----------------------------+--------------------------------------+-------------+ | 49c8aa0c-19b0-4f77-9500-8fa929f0a85f | ostest-jtbvm-master-trunk-1 | c14ef1b1-7cdd-4e93-8229-9da00795c511 | | | 9b38b271-61de-48a5-ad00-2d435fc2d752 | ostest-jtbvm-master-trunk-0 | 556a8497-72df-489e-a9ea-e767dd1ef974 | | | c4f5bce1-f1ff-440c-a4dc-7948879384b6 | ostest-jtbvm-master-trunk-2 | 1533d36f-1d83-43d9-b6a0-3565dd192f2f | | | cee09a38-c452-4fd6-af9e-e9f4a86dccb4 | ostest-jtbvm-worker-mdrvl | a0633388-4d51-49f8-bc9b-1dd64f0d0977 | | | e8b93144-79f2-4cfd-8027-3998113ab064 | ostest-jtbvm-worker-ks9zl | e9e4f8d8-d2fb-41ec-8dfd-ce546bd2fe69 | | +--------------------------------------+-----------------------------+--------------------------------------+-------------+ (shiftstack) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+---------------------------+--------+-------------------------------------+--------------------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+---------------------------+--------+-------------------------------------+--------------------+-----------+ | 469002f9-3b70-4322-8bdc-5ef8acbd0806 | ostest-jtbvm-worker-ks9zl | ACTIVE | ostest-jtbvm-openshift=10.196.3.28 | ostest-jtbvm-rhcos | m4.xlarge | | 0944e3fa-1a55-4c4c-8f7f-f5bf73b20f24 | ostest-jtbvm-worker-mdrvl | ACTIVE | ostest-jtbvm-openshift=10.196.0.123 | ostest-jtbvm-rhcos | m4.xlarge | | 5af57c72-6369-479d-b560-5d0cf251de81 | ostest-jtbvm-master-2 | ACTIVE | ostest-jtbvm-openshift=10.196.3.44 | ostest-jtbvm-rhcos | m4.xlarge | | 3deca1a4-8abe-4c48-99bb-743c08904769 | ostest-jtbvm-master-1 | ACTIVE | ostest-jtbvm-openshift=10.196.0.213 | ostest-jtbvm-rhcos | m4.xlarge | | b3bc17cb-8684-4a79-a5ae-12be291e438d | ostest-jtbvm-master-0 | ACTIVE | ostest-jtbvm-openshift=10.196.0.75 | ostest-jtbvm-rhcos | m4.xlarge | +--------------------------------------+---------------------------+--------+-------------------------------------+--------------------+-----------+ # After waiting 600 *2 (2 cycles), the leftover ports are also deleted: (shiftstack) [stack@undercloud-0 ~]$ openstack port show 48fe941d-6843-4824-896b-22b7a60dc1c9 No Port found for 48fe941d-6843-4824-896b-22b7a60dc1c9 (shiftstack) [stack@undercloud-0 ~]$ openstack port show 4c429ffe-c64a-433c-b47a-1a51ab5b6013 No Port found for 4c429ffe-c64a-433c-b47a-1a51ab5b6013 (shiftstack) [stack@undercloud-0 ~]$ openstack port show ff60382f-084b-470f-8189-21f4ee376896 No Port found for ff60382f-084b-470f-8189-21f4ee376896 (shiftstack) [stack@undercloud-0 ~]$ openstack port show af5bb1cf-db99-4dad-884e-a9c1bf278342 No Port found for af5bb1cf-db99-4dad-884e-a9c1bf278342 (shiftstack) [stack@undercloud-0 ~]$ openstack port show 5281dc0f-0234-4e5c-a388-7d84d0be4faf No Port found for 5281dc0f-0234-4e5c-a388-7d84d0be4faf (shiftstack) [stack@undercloud-0 ~]$ openstack port show c32ccfca-a9f2-4952-8ef9-71f318a325de No Port found for c32ccfca-a9f2-4952-8ef9-71f318a325de # kuryr pods remain stable during this process: NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kuryr-cni-5d4nf 1/1 Running 0 4h38m 10.196.0.75 ostest-jtbvm-master-0 <none> <none> kuryr-cni-9v967 1/1 Running 1 4h38m 10.196.3.44 ostest-jtbvm-master-2 <none> <none> kuryr-cni-jgqv7 1/1 Running 0 4h22m 10.196.0.123 ostest-jtbvm-worker-mdrvl <none> <none> kuryr-cni-srmn6 1/1 Running 1 3h 10.196.3.28 ostest-jtbvm-worker-ks9zl <none> <none> kuryr-cni-zswpp 1/1 Running 1 4h38m 10.196.0.213 ostest-jtbvm-master-1 <none> <none> kuryr-controller-764749d486-vkbl7 1/1 Running 0 4h38m 10.196.0.213 ostest-jtbvm-master-1 <none> <none> ################################################################################################################################## # OSP16.1 with OVN [stack@undercloud-0 ~]$ cat core_puddle_version RHOS-16.1-RHEL-8-20201007.n.0 [stack@undercloud-0 ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.0-0.nightly-2020-10-20-061440 True False 51m Cluster version is 4.4.0-0.nightly-2020-10-20-061440 # Given below status: (shiftstack) [stack@undercloud-0 ~]$ openstack network trunk list +--------------------------------------+-----------------------------+--------------------------------------+-------------+ | ID | Name | Parent Port | Description | +--------------------------------------+-----------------------------+--------------------------------------+-------------+ | 434b9022-3aa1-4a43-8b08-f701664c67a6 | ostest-fr28b-worker-5xwcx | 7166f37e-dee2-4eeb-a807-52229cf1123a | | | 582ef98c-fb59-41aa-b4e6-de008d60f971 | ostest-fr28b-master-trunk-1 | 69b6059f-a9d5-45d2-8700-f9eda679221d | | | 86126679-2981-4061-ba58-6dc6a9421857 | ostest-fr28b-master-trunk-0 | 1e2863b6-a915-4c61-88ea-69af5342ecea | | | 8e012c62-9efc-47d1-994a-4682f4b4ea5b | ostest-fr28b-master-trunk-2 | 0870b56d-c3dd-4810-b7f6-7f644c72c977 | | | d44bfabf-81e0-46e8-9096-45a061158395 | ostest-fr28b-worker-qc222 | aaad4be0-d5b8-4594-8a09-95414cc0981b | | | db39c3f7-4469-4261-a89e-27f4937b4580 | ostest-fr28b-worker-5whxm | 3056ab89-de42-484b-8a17-55b81f8038dd | | +--------------------------------------+-----------------------------+--------------------------------------+-------------+ # setting replica to 2. ostest-fr28b-worker-5whxm dissapears together with its trunk: (shiftstack) [stack@undercloud-0 ~]$ openstack network trunk list +--------------------------------------+-----------------------------+--------------------------------------+-------------+ | ID | Name | Parent Port | Description | +--------------------------------------+-----------------------------+--------------------------------------+-------------+ | 434b9022-3aa1-4a43-8b08-f701664c67a6 | ostest-fr28b-worker-5xwcx | 7166f37e-dee2-4eeb-a807-52229cf1123a | | | 582ef98c-fb59-41aa-b4e6-de008d60f971 | ostest-fr28b-master-trunk-1 | 69b6059f-a9d5-45d2-8700-f9eda679221d | | | 86126679-2981-4061-ba58-6dc6a9421857 | ostest-fr28b-master-trunk-0 | 1e2863b6-a915-4c61-88ea-69af5342ecea | | | 8e012c62-9efc-47d1-994a-4682f4b4ea5b | ostest-fr28b-master-trunk-2 | 0870b56d-c3dd-4810-b7f6-7f644c72c977 | | | d44bfabf-81e0-46e8-9096-45a061158395 | ostest-fr28b-worker-qc222 | aaad4be0-d5b8-4594-8a09-95414cc0981b | | +--------------------------------------+-----------------------------+--------------------------------------+-------------+ (shiftstack) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+---------------------------+--------+-------------------------------------+--------------------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+---------------------------+--------+-------------------------------------+--------------------+--------+ | 4eb42d6f-c248-4643-9043-f876e9169ae5 | ostest-fr28b-worker-5xwcx | ACTIVE | ostest-fr28b-openshift=10.196.3.90 | ostest-fr28b-rhcos | | | 580b93e0-40e4-4c1d-8d65-ad6ad6841b26 | ostest-fr28b-worker-qc222 | ACTIVE | ostest-fr28b-openshift=10.196.0.158 | ostest-fr28b-rhcos | | | 25a96b3e-7f7a-4900-9b72-8059487ad08e | ostest-fr28b-master-0 | ACTIVE | ostest-fr28b-openshift=10.196.2.139 | ostest-fr28b-rhcos | | | 77bf719c-b437-4eb7-b74e-9ee2de5b216e | ostest-fr28b-master-2 | ACTIVE | ostest-fr28b-openshift=10.196.3.121 | ostest-fr28b-rhcos | | | 4228852b-1277-49c8-a5a7-0fcabb2c41da | ostest-fr28b-master-1 | ACTIVE | ostest-fr28b-openshift=10.196.2.41 | ostest-fr28b-rhcos | | +--------------------------------------+---------------------------+--------+-------------------------------------+--------------------+--------+ # Ports marked to be deleted (shiftstack) [stack@undercloud-0 ~]$ openstack port list --tags="openshiftClusterID=ostest-fr28b" --device-owner="" | grep -v ostest +--------------------------------------+---------------------------+-------------------+-------------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+---------------------------+-------------------+-------------------------------------------------------------------------------+--------+ | 086723fc-98db-4281-9d17-1d96c19ce21d | | fa:16:3e:e4:8b:35 | ip_address='10.128.32.69', subnet_id='e86560b0-770a-4f4a-8984-33d5d17903c2' | DOWN | | 16a6ce0a-7830-40eb-b9a0-38f4e5c463ed | | fa:16:3e:91:79:57 | ip_address='10.128.57.109', subnet_id='be9a3cd7-0dd7-423d-b530-2c8626a5e7d1' | DOWN | | 18e70522-b3d7-4ffa-a2bf-1087c0263728 | | fa:16:3e:e2:0c:16 | ip_address='10.128.103.187', subnet_id='b973bc34-b860-4ba0-8142-f1055624ef35' | DOWN | | 1d036efe-c82a-4b08-993b-db7923e6b092 | | fa:16:3e:8c:ee:cc | ip_address='10.128.33.120', subnet_id='e86560b0-770a-4f4a-8984-33d5d17903c2' | DOWN | | 258482fe-cc8a-4800-a7e9-ebc7e8c10abc | | fa:16:3e:6d:8d:b7 | ip_address='10.128.99.109', subnet_id='13b66cf2-7efc-4094-9e2f-88b45b657b57' | DOWN | | 25c68e40-2a98-40da-b1b3-ee7addd3ed60 | | fa:16:3e:a5:0e:de | ip_address='10.128.6.83', subnet_id='98f41419-3b35-48f5-8cb2-d202b2146bd3' | DOWN | | 2eca92ac-7bea-44d4-94cb-6a00029226d4 | | fa:16:3e:f9:57:80 | ip_address='10.128.57.89', subnet_id='be9a3cd7-0dd7-423d-b530-2c8626a5e7d1' | DOWN | | 34fcf964-1bf2-461d-a6d4-8c0f71285caa | | fa:16:3e:c9:5d:92 | ip_address='10.128.57.185', subnet_id='be9a3cd7-0dd7-423d-b530-2c8626a5e7d1' | DOWN | | 3688ed0b-d689-4bfe-b977-492e38607120 | | fa:16:3e:6f:8b:9b | ip_address='10.128.7.10', subnet_id='98f41419-3b35-48f5-8cb2-d202b2146bd3' | DOWN | | 3ec08438-d558-46c1-bc6f-1c5589fa0f36 | | fa:16:3e:cc:8e:0a | ip_address='10.128.7.3', subnet_id='98f41419-3b35-48f5-8cb2-d202b2146bd3' | DOWN | | 42735a65-7f0a-40ca-8853-c66f1df94b79 | | fa:16:3e:4f:40:5a | ip_address='10.128.7.2', subnet_id='98f41419-3b35-48f5-8cb2-d202b2146bd3' | DOWN | | 6071adce-b0e8-412c-8523-40141f689180 | | fa:16:3e:d3:38:1f | ip_address='10.128.56.137', subnet_id='be9a3cd7-0dd7-423d-b530-2c8626a5e7d1' | DOWN | | 6fb1f159-2af6-4d26-99e7-92e7672b5f09 | | fa:16:3e:39:df:d3 | ip_address='10.128.33.158', subnet_id='e86560b0-770a-4f4a-8984-33d5d17903c2' | DOWN | | 7016188b-027a-4017-bfe3-35bfb0f76ad9 | | fa:16:3e:46:d7:d0 | ip_address='10.128.9.61', subnet_id='9a357bff-5844-4198-9043-c1c74bffb93a' | DOWN | | 72c3c245-0e57-450b-ae2b-11d5308aef77 | | fa:16:3e:a9:d9:cd | ip_address='10.128.32.7', subnet_id='e86560b0-770a-4f4a-8984-33d5d17903c2' | DOWN | | 807ee728-9ff5-465a-9b6a-716bfef57d1a | | fa:16:3e:f8:4f:5b | ip_address='10.128.8.50', subnet_id='9a357bff-5844-4198-9043-c1c74bffb93a' | DOWN | | 85c7509e-2de6-41b6-88cc-33e5d6071e5d | | fa:16:3e:ba:ec:b3 | ip_address='10.128.103.131', subnet_id='b973bc34-b860-4ba0-8142-f1055624ef35' | DOWN | | 8b90e915-6a78-4240-a21d-4daa9b21ec74 | | fa:16:3e:d1:97:f0 | ip_address='10.128.56.144', subnet_id='be9a3cd7-0dd7-423d-b530-2c8626a5e7d1' | DOWN | | 94997ec6-e9d2-481a-a7ab-5978604d3d6c | | fa:16:3e:a0:24:a8 | ip_address='10.128.6.59', subnet_id='98f41419-3b35-48f5-8cb2-d202b2146bd3' | DOWN | | 9bb5d89d-a055-432b-a0fd-807124097486 | | fa:16:3e:91:5b:3e | ip_address='10.128.33.184', subnet_id='e86560b0-770a-4f4a-8984-33d5d17903c2' | DOWN | | a34834d0-d1af-469b-a6e1-9cd9897dfbbe | | fa:16:3e:cc:d9:12 | ip_address='10.128.6.14', subnet_id='98f41419-3b35-48f5-8cb2-d202b2146bd3' | DOWN | | a34e3ad9-d315-4187-939a-cb66136b8ff1 | | fa:16:3e:18:a8:c4 | ip_address='10.128.8.241', subnet_id='9a357bff-5844-4198-9043-c1c74bffb93a' | DOWN | | d49eae2b-6e80-404c-8f00-e30d3f62d446 | | fa:16:3e:27:64:1a | ip_address='10.128.103.56', subnet_id='b973bc34-b860-4ba0-8142-f1055624ef35' | DOWN | | d731bc34-23b4-46a7-9f46-265db7ee0272 | | fa:16:3e:6e:2a:a4 | ip_address='10.128.6.91', subnet_id='98f41419-3b35-48f5-8cb2-d202b2146bd3' | DOWN | | df145a00-7454-457c-9976-8346a7c60d76 | | fa:16:3e:c5:fd:82 | ip_address='10.128.98.178', subnet_id='13b66cf2-7efc-4094-9e2f-88b45b657b57' | DOWN | | e706960e-cd39-4a9a-b7df-422c43295264 | | fa:16:3e:4a:f7:9d | ip_address='10.128.7.90', subnet_id='98f41419-3b35-48f5-8cb2-d202b2146bd3' | DOWN | | e8f26a92-c239-4acc-9c06-d4c383113c82 | | fa:16:3e:23:06:51 | ip_address='10.128.6.161', subnet_id='98f41419-3b35-48f5-8cb2-d202b2146bd3' | DOWN | | ee478e5f-18f5-4d83-89d7-4ccb4e14ae0f | | fa:16:3e:9e:40:e5 | ip_address='10.128.32.100', subnet_id='e86560b0-770a-4f4a-8984-33d5d17903c2' | DOWN | | f67678d9-75a8-48fd-b0ec-d62b174a8bd8 | | fa:16:3e:94:93:eb | ip_address='10.128.99.185', subnet_id='13b66cf2-7efc-4094-9e2f-88b45b657b57' | DOWN | | fa39ac51-198a-47db-a96f-02e109e12930 | | fa:16:3e:1e:ae:9a | ip_address='10.128.57.10', subnet_id='be9a3cd7-0dd7-423d-b530-2c8626a5e7d1' | DOWN | +--------------------------------------+---------------------------+-------------------+-------------------------------------------------------------------------------+--------+ # After waiting 600 *2 (2 cycles), the leftover ports are also deleted: (shiftstack) [stack@undercloud-0 ~]$ openstack port list --tags="openshiftClusterID=ostest-fr28b" --device-owner="" | grep -v ostest +--------------------------------------+---------------------------+-------------------+---------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+---------------------------+-------------------+---------------------------------------------------------------------------+--------+ +--------------------------------------+---------------------------+-------------------+---------------------------------------------------------------------------+--------+ # kuryr pods remain stable during this process: NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kuryr-cni-28w65 1/1 Running 1 99m 10.196.3.90 ostest-fr28b-worker-5xwcx <none> <none> kuryr-cni-b89pk 1/1 Running 0 99m 10.196.0.158 ostest-fr28b-worker-qc222 <none> <none> kuryr-cni-frc9f 1/1 Running 1 122m 10.196.2.139 ostest-fr28b-master-0 <none> <none> kuryr-cni-mw22s 1/1 Running 0 122m 10.196.2.41 ostest-fr28b-master-1 <none> <none> kuryr-cni-wpf4v 1/1 Running 0 122m 10.196.3.121 ostest-fr28b-master-2 <none> <none> kuryr-controller-78df6444b-nnmcl 1/1 Running 1 122m 10.196.3.121 ostest-fr28b-master-2 <none> <none>
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.4.30 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4321