Bug 1729276 - Using AWS when a secondary IP address is added the order of the list of InternalIPs is not preserved.
Summary: Using AWS when a secondary IP address is added the order of the list of Inter...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.1.z
Assignee: Robert Krawitz
QA Contact: Weinan Liu
URL:
Whiteboard: 4.1.8
Depends On: 1696628
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-11 18:17 UTC by Robert Krawitz
Modified: 2019-07-31 02:45 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Merging algorithm for additional IP addresses added to a node was incorrect. Consequence: After adding an additional IP address to a node, the list of addresses was out of order, resulting in the node being unable to talk to the api server. Fix: Change the merge algorithm for addresses to not reorder the addresses. Result: Adding secondary IP addresses to a node no longer changes the ordering and the node is able to continue communication with the api server.
Clone Of: 1696628
Environment:
Last Closed: 2019-07-31 02:44:55 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:1866 None None None 2019-07-31 02:45:02 UTC
Github openshift origin pull 23367 None None None 2019-07-19 17:14:46 UTC

Comment 1 Wei Sun 2019-07-24 07:06:29 UTC
The fix is merged to 4.1.0-0.nightly-2019-07-24-051320 ,please check if we could verify it.

Comment 2 Weinan Liu 2019-07-24 08:49:52 UTC
Verified.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-07-24-051320   True        False         12m     Cluster version is 4.1.0-0.nightly-2019-07-24-051320

Added 4th secondary IP addresses to node ip-10-0-145-136.us-east-2.compute.internal from AWS console . Then get the node  rebooted, it is still using original IP address.


$ oc get nodes -o wide
NAME                                         STATUS   ROLES    AGE    VERSION             INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                   KERNEL-VERSION               CONTAINER-RUNTIME
ip-10-0-137-203.us-east-2.compute.internal   Ready    master   145m   v1.14.0+bbfcbc8ac   10.0.137.203   <none>        Red Hat Enterprise Linux CoreOS 420.8.20190718.1 (Ootpa)   4.18.0-80.4.2.el8_0.x86_64   cri-o://1.14.8-3.rhaos4.2.el8
ip-10-0-137-237.us-east-2.compute.internal   Ready    worker   136m   v1.14.0+bbfcbc8ac   10.0.137.237   <none>        Red Hat Enterprise Linux CoreOS 420.8.20190718.1 (Ootpa)   4.18.0-80.4.2.el8_0.x86_64   cri-o://1.14.8-3.rhaos4.2.el8
ip-10-0-145-136.us-east-2.compute.internal   Ready    worker   136m   v1.14.0+bbfcbc8ac   10.0.145.136   <none>        Red Hat Enterprise Linux CoreOS 420.8.20190718.1 (Ootpa)   4.18.0-80.4.2.el8_0.x86_64   cri-o://1.14.8-3.rhaos4.2.el8
ip-10-0-154-217.us-east-2.compute.internal   Ready    master   145m   v1.14.0+bbfcbc8ac   10.0.154.217   <none>        Red Hat Enterprise Linux CoreOS 420.8.20190718.1 (Ootpa)   4.18.0-80.4.2.el8_0.x86_64   cri-o://1.14.8-3.rhaos4.2.el8
ip-10-0-173-64.us-east-2.compute.internal    Ready    master   145m   v1.14.0+bbfcbc8ac   10.0.173.64    <none>        Red Hat Enterprise Linux CoreOS 420.8.20190718.1 (Ootpa)   4.18.0-80.4.2.el8_0.x86_64   cri-o://1.14.8-3.rhaos4.2.el8

Added 4 secondary IPs

sh-4.4# curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/06:8b:1b:6c:bc:24/local-ipv4s
10.0.145.136
10.0.150.98
10.0.156.131
10.0.146.68
10.0.155.234

$ aws ec2 describe-instances --instance-ids i-0c27102bad9011ea1 --query 'Reservations[].Instances[].NetworkInterfaces[].PrivateIpAddresses'
[
    [
        {
            "Primary": true,
            "PrivateDnsName": "ip-10-0-145-136.us-east-2.compute.internal",
            "PrivateIpAddress": "10.0.145.136"
        },
        {
            "Primary": false,
            "PrivateDnsName": "ip-10-0-150-98.us-east-2.compute.internal",
            "PrivateIpAddress": "10.0.150.98"
        },
        {
            "Primary": false,
            "PrivateDnsName": "ip-10-0-156-131.us-east-2.compute.internal",
            "PrivateIpAddress": "10.0.156.131"
        },
        {
            "Primary": false,
            "PrivateDnsName": "ip-10-0-146-68.us-east-2.compute.internal",
            "PrivateIpAddress": "10.0.146.68"
        },
        {
            "Primary": false,
            "PrivateDnsName": "ip-10-0-155-234.us-east-2.compute.internal",
            "PrivateIpAddress": "10.0.155.234"
        }
    ]
]

$ oc describe node ip-10-0-145-136.us-east-2.compute.internal
Name:               ip-10-0-145-136.us-east-2.compute.internal
...
Addresses:
  InternalIP:   10.0.145.136
  InternalIP:   10.0.150.98
  InternalIP:   10.0.156.131
  InternalIP:   10.0.146.68
  InternalIP:   10.0.155.234
  InternalDNS:  ip-10-0-145-136.us-east-2.compute.internal
  Hostname:     ip-10-0-145-136.us-east-2.compute.internal

$ oc get nodes -o wide
NAME                                         STATUS   ROLES    AGE     VERSION             INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                   KERNEL-VERSION               CONTAINER-RUNTIME
ip-10-0-137-203.us-east-2.compute.internal   Ready    master   4h35m   v1.14.0+bbfcbc8ac   10.0.137.203   <none>        Red Hat Enterprise Linux CoreOS 420.8.20190718.1 (Ootpa)   4.18.0-80.4.2.el8_0.x86_64   cri-o://1.14.8-3.rhaos4.2.el8
ip-10-0-137-237.us-east-2.compute.internal   Ready    worker   4h26m   v1.14.0+bbfcbc8ac   10.0.137.237   <none>        Red Hat Enterprise Linux CoreOS 420.8.20190718.1 (Ootpa)   4.18.0-80.4.2.el8_0.x86_64   cri-o://1.14.8-3.rhaos4.2.el8
ip-10-0-145-136.us-east-2.compute.internal   Ready    worker   4h26m   v1.14.0+bbfcbc8ac   10.0.145.136   <none>        Red Hat Enterprise Linux CoreOS 420.8.20190718.1 (Ootpa)   4.18.0-80.4.2.el8_0.x86_64   cri-o://1.14.8-3.rhaos4.2.el8
ip-10-0-154-217.us-east-2.compute.internal   Ready    master   4h35m   v1.14.0+bbfcbc8ac   10.0.154.217   <none>        Red Hat Enterprise Linux CoreOS 420.8.20190718.1 (Ootpa)   4.18.0-80.4.2.el8_0.x86_64   cri-o://1.14.8-3.rhaos4.2.el8
ip-10-0-173-64.us-east-2.compute.internal    Ready    master   4h35m   v1.14.0+bbfcbc8ac   10.0.173.64    <none>        Red Hat Enterprise Linux CoreOS 420.8.20190718.1 (Ootpa)   4.18.0-80.4.2.el8_0.x86_64   cri-o://1.14.8-3.rhaos4.2.el8

Comment 5 errata-xmlrpc 2019-07-31 02:44:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1866


Note You need to log in before you can comment on or make changes to this bug.