Bug 1714659 - SSH'ing in to node does not result in annotation being applied to node
Summary: SSH'ing in to node does not result in annotation being applied to node
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RHCOS
Version: 4.1.z
Hardware: x86_64
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.1.z
Assignee: Steve Milner
QA Contact: Micah Abbott
URL:
Whiteboard: 4.1.2
Depends On:
Blocks: 1718956
TreeView+ depends on / blocked
 
Reported: 2019-05-28 14:18 UTC by rvanderp
Modified: 2019-11-05 13:51 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-19 06:45:34 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3801571 0 Learn more None How to tell if an OpenShift 4.x Node is annotated 2019-05-28 14:37:31 UTC
Red Hat Product Errata RHBA-2019:1382 0 None None None 2019-06-19 06:45:44 UTC

Description rvanderp 2019-05-28 14:18:45 UTC
Description of problem:

On a bare metal install, when ssh'ing in to the master or worker nodes, RHCOS reports that the node has been annotated with `machineconfiguration.openshift.io/ssh=accessed` but no annotation is applied to nodes.


Version-Release number of selected component (if applicable):
4.1.0-rc.7

How reproducible:
Consistently reproducible

Steps to Reproduce:
1. ssh to a node
2. run `oc describe node <node>`
3. check node annotations 

Actual results:
The annotation `machineconfiguration.openshift.io/ssh=accessed` is not applied to the node

Expected results:


Additional info:

Comment 1 rvanderp 2019-05-28 14:28:01 UTC
I think we need a test case associated with this for bare metal installs.

Comment 2 rvanderp 2019-05-28 14:30:36 UTC
Attached are the commands I ran to reproduce the problem -->

[root@rwv3-develop-laptop ocp-4-libvirt-lab]# oc get nodes -o 'custom-columns=Node Name:.metadata.name,Machine Name:.metadata.annotations.machine\.openshift\.io/machine,SSHAccessed:.metadata.annotations.machineconfiguration\.openshift\.io/ssh'
Node Name   Machine Name   SSHAccessed
master-0    <none>         <none>
worker-0    <none>         <none>
[root@rwv3-develop-laptop ocp-4-libvirt-lab]# ssh core@192.168.100.10
The authenticity of host '192.168.100.10 (192.168.100.10)' can't be established.
ECDSA key fingerprint is SHA256:3pIid7khIPgN/ZnNE7atJSw4B1OczgfpbL4iG6kIFss.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.100.10' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 410.8.20190520.0
WARNING: Direct SSH access to machines is not recommended.
This node has been annotated with machineconfiguration.openshift.io/ssh=accessed

---
[core@master-0 ~]$ exit
logout
Connection to 192.168.100.10 closed.
[root@rwv3-develop-laptop ocp-4-libvirt-lab]# oc get nodes -o 'custom-columns=Node Name:.metadata.name,Machine Name:.metadata.annotations.machine\.openshift\.io/machine,SSHAccessed:.metadata.annotations.machineconfiguration\.openshift\.io/ssh'
Node Name   Machine Name   SSHAccessed
master-0    <none>         <none>
worker-0    <none>         <none>




Attached is the output of `oc describe node` -->

[root@rwv3-develop-laptop ocp-4-libvirt-lab]# oc describe node master-0
Name:			master-0
Role:			
Labels:			beta.kubernetes.io/arch=amd64
			beta.kubernetes.io/os=linux
			kubernetes.io/hostname=master-0
			node-role.kubernetes.io/master=
			node.openshift.io/os_id=rhcos
			node.openshift.io/os_version=4.1
Annotations:		machineconfiguration.openshift.io/currentConfig=rendered-master-163cb29e01cc8a21f205eb80613d317a
			machineconfiguration.openshift.io/desiredConfig=rendered-master-163cb29e01cc8a21f205eb80613d317a
			machineconfiguration.openshift.io/state=Done
			volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:			node-role.kubernetes.io/master:NoSchedule
CreationTimestamp:	Sat, 25 May 2019 17:46:29 -0400
Conditions:
  Type			Status	LastHeartbeatTime			LastTransitionTime			Reason				Message
  ----			------	-----------------			------------------			------				-------
  MemoryPressure 	False 	Tue, 28 May 2019 10:20:43 -0400 	Tue, 28 May 2019 09:52:09 -0400 	KubeletHasSufficientMemory 	kubelet has sufficient memory available
  DiskPressure 		False 	Tue, 28 May 2019 10:20:43 -0400 	Tue, 28 May 2019 09:52:09 -0400 	KubeletHasNoDiskPressure 	kubelet has no disk pressure
  PIDPressure 		False 	Tue, 28 May 2019 10:20:43 -0400 	Tue, 28 May 2019 09:52:09 -0400 	KubeletHasSufficientPID 	kubelet has sufficient PID available
  Ready 		True 	Tue, 28 May 2019 10:20:43 -0400 	Tue, 28 May 2019 09:52:09 -0400 	KubeletReady 			kubelet is posting ready status
Addresses:
  InternalIP:	192.168.100.10
  Hostname:	master-0
Capacity:
 cpu:		4
 hugepages-1Gi:	0
 hugepages-2Mi:	0
 memory:	14016976Ki
 pods:		250
Allocatable:
 cpu:		3500m
 hugepages-1Gi:	0
 hugepages-2Mi:	0
 memory:	13402576Ki
 pods:		250
System Info:
 Machine ID:						db5e43d2f78647899c3a1661a2533287
 System UUID:						db5e43d2-f786-4789-9c3a-1661a2533287
 Boot ID:						18722c64-02cf-43e0-a12c-765dbe3ed97d
 Kernel Version:					4.18.0-80.1.2.el8_0.x86_64
 OS Image:						Red Hat Enterprise Linux CoreOS 410.8.20190520.0 (Ootpa)
 Operating System:					linux
 Architecture:						amd64
 Container Runtime Version:				cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8
 Kubelet Version:					v1.13.4+cb455d664
 Kube-Proxy Version:					v1.13.4+cb455d664
ExternalID:						master-0
Non-terminated Pods:					(56 in total)
  Namespace						Name									CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ---------						----									------------	----------	---------------	-------------
  openshift-apiserver-operator				openshift-apiserver-operator-659785c78c-hv9cd				10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-apiserver					apiserver-pcvnh								150m (4%)	0 (0%)		200Mi (1%)	0 (0%)
  openshift-authentication-operator			authentication-operator-86bd89cb64-rs5z5				10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-authentication				oauth-openshift-5b57f6fc78-2xx6p					10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-authentication				oauth-openshift-5b57f6fc78-7tl62					10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-cloud-credential-operator			cloud-credential-operator-6cff76c999-tbgpc				10m (0%)	0 (0%)		150Mi (1%)	500Mi (3%)
  openshift-cluster-machine-approver			machine-approver-7cd7f97455-m2ld4					10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-cluster-node-tuning-operator		cluster-node-tuning-operator-795c9dfc96-8dvtg				10m (0%)	0 (0%)		20Mi (0%)	0 (0%)
  openshift-cluster-node-tuning-operator		tuned-nw4qj								0 (0%)		0 (0%)		0 (0%)		0 (0%)
  openshift-cluster-samples-operator			cluster-samples-operator-67b8c675c4-mbpls				10m (0%)	0 (0%)		0 (0%)		0 (0%)
  openshift-cluster-storage-operator			cluster-storage-operator-754c849f5c-c8pht				0 (0%)		0 (0%)		0 (0%)		0 (0%)
  openshift-cluster-version				cluster-version-operator-697b4645d9-x7fhd				20m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-console-operator				console-operator-646b694cfb-vcbtz					10m (0%)	0 (0%)		100Mi (0%)	0 (0%)
  openshift-console					console-7bb5c68f8b-94rzb						10m (0%)	0 (0%)		100Mi (0%)	0 (0%)
  openshift-console					console-7bb5c68f8b-gkxrf						10m (0%)	0 (0%)		100Mi (0%)	0 (0%)
  openshift-console					downloads-65877c7d-c6mmp						0 (0%)		0 (0%)		0 (0%)		0 (0%)
  openshift-console					downloads-65877c7d-r52xb						0 (0%)		0 (0%)		0 (0%)		0 (0%)
  openshift-controller-manager-operator			openshift-controller-manager-operator-74855c96bd-twdcs			10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-controller-manager				controller-manager-jqh4t						100m (2%)	0 (0%)		100Mi (0%)	0 (0%)
  openshift-dns-operator				dns-operator-67cb8f9f94-s2sn8						10m (0%)	0 (0%)		0 (0%)		0 (0%)
  openshift-dns						dns-default-bngrm							110m (3%)	0 (0%)		70Mi (0%)	512Mi (3%)
  openshift-etcd					etcd-member-master-0							300m (8%)	0 (0%)		600Mi (4%)	0 (0%)
  openshift-image-registry				cluster-image-registry-operator-644dfbf98b-w86p9			10m (0%)	0 (0%)		0 (0%)		0 (0%)
  openshift-image-registry				node-ca-kf5cf								0 (0%)		0 (0%)		0 (0%)		0 (0%)
  openshift-ingress-operator				ingress-operator-c6d68c8cb-gzxhf					10m (0%)	0 (0%)		0 (0%)		0 (0%)
  openshift-kube-apiserver-operator			kube-apiserver-operator-6bcfbb68d9-9zc8f				10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-kube-apiserver				kube-apiserver-master-0							160m (4%)	0 (0%)		1074Mi (8%)	0 (0%)
  openshift-kube-controller-manager-operator		kube-controller-manager-operator-5565c768b6-8rdpt			10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-kube-controller-manager			kube-controller-manager-master-0					110m (3%)	0 (0%)		250Mi (1%)	0 (0%)
  openshift-kube-scheduler-operator			openshift-kube-scheduler-operator-7f4779f978-gq9l6			0 (0%)		0 (0%)		50Mi (0%)	0 (0%)
  openshift-kube-scheduler				openshift-kube-scheduler-master-0					0 (0%)		0 (0%)		50Mi (0%)	0 (0%)
  openshift-machine-api					cluster-autoscaler-operator-7988df7df4-bsn6p				20m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-machine-api					machine-api-operator-66f55c847-scdsp					10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-machine-config-operator			etcd-quorum-guard-66b78568d6-6s5zg					10m (0%)	0 (0%)		5Mi (0%)	0 (0%)
  openshift-machine-config-operator			machine-config-controller-69876f9fc5-fdp8l				20m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-machine-config-operator			machine-config-daemon-8mn7v						20m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-machine-config-operator			machine-config-operator-fbff896bc-89f6z					20m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-machine-config-operator			machine-config-server-krpvr						20m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-marketplace					marketplace-operator-c568c789f-dt4pj					0 (0%)		0 (0%)		0 (0%)		0 (0%)
  openshift-monitoring					cluster-monitoring-operator-56d4684966-kd797				10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-monitoring					node-exporter-sxnzm							10m (0%)	0 (0%)		20Mi (0%)	0 (0%)
  openshift-multus					multus-zrwng								0 (0%)		0 (0%)		0 (0%)		0 (0%)
  openshift-network-operator				network-operator-5dd57dd7bd-7vkxk					10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-operator-lifecycle-manager			catalog-operator-6998c87fc5-8n9xk					0 (0%)		0 (0%)		0 (0%)		0 (0%)
  openshift-operator-lifecycle-manager			olm-operator-6bc4bc64f8-9h8wx						0 (0%)		0 (0%)		0 (0%)		0 (0%)
  openshift-operator-lifecycle-manager			packageserver-86866ccd8-f9f7l						0 (0%)		0 (0%)		0 (0%)		0 (0%)
  openshift-operator-lifecycle-manager			packageserver-86866ccd8-jgpnh						0 (0%)		0 (0%)		0 (0%)		0 (0%)
  openshift-sdn						ovs-8n667								200m (5%)	0 (0%)		400Mi (3%)	0 (0%)
  openshift-sdn						sdn-controller-2zx7r							10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-sdn						sdn-p7wm6								100m (2%)	0 (0%)		200Mi (1%)	0 (0%)
  openshift-service-ca-operator				service-ca-operator-987fc4cdd-zkmwq					10m (0%)	0 (0%)		80Mi (0%)	0 (0%)
  openshift-service-ca					apiservice-cabundle-injector-5bf59477d7-nlb4b				10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-service-ca					configmap-cabundle-injector-6cfcd498d7-4qtwx				10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
  openshift-service-ca					service-serving-cert-signer-7789d64745-7bwq7				10m (0%)	0 (0%)		120Mi (0%)	0 (0%)
  openshift-service-catalog-apiserver-operator		openshift-service-catalog-apiserver-operator-5dbfcf7bc5-zlq8l		0 (0%)		0 (0%)		50Mi (0%)	0 (0%)
  openshift-service-catalog-controller-manager-operator	openshift-service-catalog-controller-manager-operator-5bf65dp2g		10m (0%)	0 (0%)		50Mi (0%)	0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ------------	----------	---------------	-------------
  1630m (46%)	0 (0%)		4789Mi (36%)	1012Mi (7%)
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----			-------------	--------	------			-------
  1d		1d		1	kubelet, master-0			Normal		Starting		Starting kubelet.
  1d		1d		8	kubelet, master-0			Normal		NodeHasSufficientMemory	Node master-0 status is now: NodeHasSufficientMemory
  1d		1d		8	kubelet, master-0			Normal		NodeHasNoDiskPressure	Node master-0 status is now: NodeHasNoDiskPressure
  1d		1d		7	kubelet, master-0			Normal		NodeHasSufficientPID	Node master-0 status is now: NodeHasSufficientPID
  1d		1d		1	kubelet, master-0			Normal		NodeAllocatableEnforced	Updated Node Allocatable limit across pods
  30m		30m		1	kubelet, master-0			Normal		Starting		Starting kubelet.
  30m		30m		8	kubelet, master-0			Normal		NodeHasSufficientMemory	Node master-0 status is now: NodeHasSufficientMemory
  30m		30m		8	kubelet, master-0			Normal		NodeHasNoDiskPressure	Node master-0 status is now: NodeHasNoDiskPressure
  30m		30m		7	kubelet, master-0			Normal		NodeHasSufficientPID	Node master-0 status is now: NodeHasSufficientPID
  30m		30m		1	kubelet, master-0			Normal		NodeAllocatableEnforced	Updated Node Allocatable limit across pods

Comment 3 Antonio Murdaca 2019-05-28 14:32:11 UTC
We're looking into this...The thing with this annotation is that we now have "oc debug node/nodename" which defeats the annotation altogether since anyone can oc debug, chroot and do whatever. I'm not super sure we want this at all now and we can't really do anything right now other than reporting "accessed" (i.e. we're not reconciling or doing anything else if someone messes on the host).

Comment 4 Colin Walters 2019-05-28 15:01:30 UTC
Yeah, I think we either need to:

 - Drop this attempt at annotating
 - Add an annotation if any privileged pod that e.g. has an interactive TTY landed on a node, as well as SSH

Short term I think we should just drop the bit from the MOTD in RHCOS.

Comment 13 errata-xmlrpc 2019-06-19 06:45:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1382


Note You need to log in before you can comment on or make changes to this bug.