Bug 1559271 - AWS nodes does not keep persistent labels after shut down
Summary: AWS nodes does not keep persistent labels after shut down
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.6.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.11.0
Assignee: Seth Jennings
QA Contact: weiwei jiang
URL:
Whiteboard:
: 1435401 1569043 1608912 1608971 1619599 (view as bug list)
Depends On:
Blocks: 1435401
TreeView+ depends on / blocked
 
Reported: 2018-03-22 07:15 UTC by Suhaas Bhat
Modified: 2019-05-29 09:55 UTC (History)
20 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2018-10-11 07:19:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3570631 0 None None None 2018-09-06 03:06:41 UTC
Red Hat Product Errata RHBA-2018:2652 0 None None None 2018-10-11 07:20:11 UTC

Description Suhaas Bhat 2018-03-22 07:15:01 UTC
Description of problem:
OCP nodes on AWS do not keep persistent labels when given from "oc label" command after shutting down the node.

Version-Release number of selected component (if applicable):
oc v3.6.173.0.21
kubernetes v1.6.1+5115d708d7


Actual results: Labels are not persistent after node shutdown using "oc command"


Expected results: Should be persistent

Additional Info :
Correct entries in /etc/origin/cloudprovider/aws.conf
Same Kubernetes Cluster Tags across all cluster

Comment 2 Seth Jennings 2018-03-22 14:09:32 UTC
Recently had a similiar issue with Openstack cloudprovider in that the node controller was getting a filtered list of instances, only getting the active ones.
https://bugzilla.redhat.com/show_bug.cgi?id=1558422

Comment 3 Seth Jennings 2018-04-11 03:29:47 UTC
More relevant upstream discussion/work:
https://github.com/kubernetes/kubernetes/issues/58635
https://github.com/kubernetes/kubernetes/pull/60009
https://github.com/kubernetes/kubernetes/issues/46442
https://github.com/kubernetes/kubernetes/issues/45118

There is also an old open PR to remove the instance filtering from the AWS cloudprovider, which would address is bug:
https://github.com/kubernetes/kubernetes/pull/45986

Comment 4 Seth Jennings 2018-04-18 14:08:06 UTC
The effort has stalled upstream.  I am pushing for acceptance of a targeted fix for just this case (one line change in the AWS cloudprovider) as a stopgap while a more general solution is developed.

Comment 5 Seth Jennings 2018-05-02 16:39:04 UTC
*** Bug 1569043 has been marked as a duplicate of this bug. ***

Comment 10 Seth Jennings 2018-07-30 20:16:00 UTC
*** Bug 1608971 has been marked as a duplicate of this bug. ***

Comment 11 Seth Jennings 2018-07-30 20:54:51 UTC
*** Bug 1608912 has been marked as a duplicate of this bug. ***

Comment 13 Seth Jennings 2018-07-31 18:55:14 UTC
I posted a more concise PR upstream to avoid friction:
https://github.com/kubernetes/kubernetes/pull/66835

Comment 15 Seth Jennings 2018-08-15 20:52:44 UTC
Origin PR:
https://github.com/openshift/origin/pull/20663

Comment 16 Seth Jennings 2018-08-22 14:05:39 UTC
*** Bug 1619599 has been marked as a duplicate of this bug. ***

Comment 21 weiwei jiang 2018-09-06 07:20:26 UTC
Checked on 
# oc version 
oc v3.11.0-0.28.0
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-7-85.ec2.internal:8443
openshift v3.11.0-0.28.0
kubernetes v1.11.0+d4cacc0

And this issue has been fixed.
1. labels from `oc label` is persistent now.
2. labels from node-config.yaml(from 3.10, should be configmap in openshift-node namespace) is also applied after reboot system
3. labels already applied to node from node-config.yaml will still keep after reboot the system

Comment 23 errata-xmlrpc 2018-10-11 07:19:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2652

Comment 24 Niels de Vos 2019-02-05 10:40:41 UTC
*** Bug 1435401 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.