Description of problem: On a bare metal install, when ssh'ing in to the master or worker nodes, RHCOS reports that the node has been annotated with `machineconfiguration.openshift.io/ssh=accessed` but no annotation is applied to nodes. Version-Release number of selected component (if applicable): 4.1.0-rc.7 How reproducible: Consistently reproducible Steps to Reproduce: 1. ssh to a node 2. run `oc describe node <node>` 3. check node annotations Actual results: The annotation `machineconfiguration.openshift.io/ssh=accessed` is not applied to the node Expected results: Additional info:
I think we need a test case associated with this for bare metal installs.
Attached are the commands I ran to reproduce the problem --> [root@rwv3-develop-laptop ocp-4-libvirt-lab]# oc get nodes -o 'custom-columns=Node Name:.metadata.name,Machine Name:.metadata.annotations.machine\.openshift\.io/machine,SSHAccessed:.metadata.annotations.machineconfiguration\.openshift\.io/ssh' Node Name Machine Name SSHAccessed master-0 <none> <none> worker-0 <none> <none> [root@rwv3-develop-laptop ocp-4-libvirt-lab]# ssh core.100.10 The authenticity of host '192.168.100.10 (192.168.100.10)' can't be established. ECDSA key fingerprint is SHA256:3pIid7khIPgN/ZnNE7atJSw4B1OczgfpbL4iG6kIFss. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.168.100.10' (ECDSA) to the list of known hosts. Red Hat Enterprise Linux CoreOS 410.8.20190520.0 WARNING: Direct SSH access to machines is not recommended. This node has been annotated with machineconfiguration.openshift.io/ssh=accessed --- [core@master-0 ~]$ exit logout Connection to 192.168.100.10 closed. [root@rwv3-develop-laptop ocp-4-libvirt-lab]# oc get nodes -o 'custom-columns=Node Name:.metadata.name,Machine Name:.metadata.annotations.machine\.openshift\.io/machine,SSHAccessed:.metadata.annotations.machineconfiguration\.openshift\.io/ssh' Node Name Machine Name SSHAccessed master-0 <none> <none> worker-0 <none> <none> Attached is the output of `oc describe node` --> [root@rwv3-develop-laptop ocp-4-libvirt-lab]# oc describe node master-0 Name: master-0 Role: Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/hostname=master-0 node-role.kubernetes.io/master= node.openshift.io/os_id=rhcos node.openshift.io/os_version=4.1 Annotations: machineconfiguration.openshift.io/currentConfig=rendered-master-163cb29e01cc8a21f205eb80613d317a machineconfiguration.openshift.io/desiredConfig=rendered-master-163cb29e01cc8a21f205eb80613d317a machineconfiguration.openshift.io/state=Done volumes.kubernetes.io/controller-managed-attach-detach=true Taints: node-role.kubernetes.io/master:NoSchedule CreationTimestamp: Sat, 25 May 2019 17:46:29 -0400 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Tue, 28 May 2019 10:20:43 -0400 Tue, 28 May 2019 09:52:09 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Tue, 28 May 2019 10:20:43 -0400 Tue, 28 May 2019 09:52:09 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Tue, 28 May 2019 10:20:43 -0400 Tue, 28 May 2019 09:52:09 -0400 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Tue, 28 May 2019 10:20:43 -0400 Tue, 28 May 2019 09:52:09 -0400 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.100.10 Hostname: master-0 Capacity: cpu: 4 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 14016976Ki pods: 250 Allocatable: cpu: 3500m hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 13402576Ki pods: 250 System Info: Machine ID: db5e43d2f78647899c3a1661a2533287 System UUID: db5e43d2-f786-4789-9c3a-1661a2533287 Boot ID: 18722c64-02cf-43e0-a12c-765dbe3ed97d Kernel Version: 4.18.0-80.1.2.el8_0.x86_64 OS Image: Red Hat Enterprise Linux CoreOS 410.8.20190520.0 (Ootpa) Operating System: linux Architecture: amd64 Container Runtime Version: cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8 Kubelet Version: v1.13.4+cb455d664 Kube-Proxy Version: v1.13.4+cb455d664 ExternalID: master-0 Non-terminated Pods: (56 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits --------- ---- ------------ ---------- --------------- ------------- openshift-apiserver-operator openshift-apiserver-operator-659785c78c-hv9cd 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-apiserver apiserver-pcvnh 150m (4%) 0 (0%) 200Mi (1%) 0 (0%) openshift-authentication-operator authentication-operator-86bd89cb64-rs5z5 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-authentication oauth-openshift-5b57f6fc78-2xx6p 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-authentication oauth-openshift-5b57f6fc78-7tl62 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-cloud-credential-operator cloud-credential-operator-6cff76c999-tbgpc 10m (0%) 0 (0%) 150Mi (1%) 500Mi (3%) openshift-cluster-machine-approver machine-approver-7cd7f97455-m2ld4 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-cluster-node-tuning-operator cluster-node-tuning-operator-795c9dfc96-8dvtg 10m (0%) 0 (0%) 20Mi (0%) 0 (0%) openshift-cluster-node-tuning-operator tuned-nw4qj 0 (0%) 0 (0%) 0 (0%) 0 (0%) openshift-cluster-samples-operator cluster-samples-operator-67b8c675c4-mbpls 10m (0%) 0 (0%) 0 (0%) 0 (0%) openshift-cluster-storage-operator cluster-storage-operator-754c849f5c-c8pht 0 (0%) 0 (0%) 0 (0%) 0 (0%) openshift-cluster-version cluster-version-operator-697b4645d9-x7fhd 20m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-console-operator console-operator-646b694cfb-vcbtz 10m (0%) 0 (0%) 100Mi (0%) 0 (0%) openshift-console console-7bb5c68f8b-94rzb 10m (0%) 0 (0%) 100Mi (0%) 0 (0%) openshift-console console-7bb5c68f8b-gkxrf 10m (0%) 0 (0%) 100Mi (0%) 0 (0%) openshift-console downloads-65877c7d-c6mmp 0 (0%) 0 (0%) 0 (0%) 0 (0%) openshift-console downloads-65877c7d-r52xb 0 (0%) 0 (0%) 0 (0%) 0 (0%) openshift-controller-manager-operator openshift-controller-manager-operator-74855c96bd-twdcs 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-controller-manager controller-manager-jqh4t 100m (2%) 0 (0%) 100Mi (0%) 0 (0%) openshift-dns-operator dns-operator-67cb8f9f94-s2sn8 10m (0%) 0 (0%) 0 (0%) 0 (0%) openshift-dns dns-default-bngrm 110m (3%) 0 (0%) 70Mi (0%) 512Mi (3%) openshift-etcd etcd-member-master-0 300m (8%) 0 (0%) 600Mi (4%) 0 (0%) openshift-image-registry cluster-image-registry-operator-644dfbf98b-w86p9 10m (0%) 0 (0%) 0 (0%) 0 (0%) openshift-image-registry node-ca-kf5cf 0 (0%) 0 (0%) 0 (0%) 0 (0%) openshift-ingress-operator ingress-operator-c6d68c8cb-gzxhf 10m (0%) 0 (0%) 0 (0%) 0 (0%) openshift-kube-apiserver-operator kube-apiserver-operator-6bcfbb68d9-9zc8f 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-kube-apiserver kube-apiserver-master-0 160m (4%) 0 (0%) 1074Mi (8%) 0 (0%) openshift-kube-controller-manager-operator kube-controller-manager-operator-5565c768b6-8rdpt 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-kube-controller-manager kube-controller-manager-master-0 110m (3%) 0 (0%) 250Mi (1%) 0 (0%) openshift-kube-scheduler-operator openshift-kube-scheduler-operator-7f4779f978-gq9l6 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-kube-scheduler openshift-kube-scheduler-master-0 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-machine-api cluster-autoscaler-operator-7988df7df4-bsn6p 20m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-machine-api machine-api-operator-66f55c847-scdsp 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-machine-config-operator etcd-quorum-guard-66b78568d6-6s5zg 10m (0%) 0 (0%) 5Mi (0%) 0 (0%) openshift-machine-config-operator machine-config-controller-69876f9fc5-fdp8l 20m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-machine-config-operator machine-config-daemon-8mn7v 20m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-machine-config-operator machine-config-operator-fbff896bc-89f6z 20m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-machine-config-operator machine-config-server-krpvr 20m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-marketplace marketplace-operator-c568c789f-dt4pj 0 (0%) 0 (0%) 0 (0%) 0 (0%) openshift-monitoring cluster-monitoring-operator-56d4684966-kd797 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-monitoring node-exporter-sxnzm 10m (0%) 0 (0%) 20Mi (0%) 0 (0%) openshift-multus multus-zrwng 0 (0%) 0 (0%) 0 (0%) 0 (0%) openshift-network-operator network-operator-5dd57dd7bd-7vkxk 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-operator-lifecycle-manager catalog-operator-6998c87fc5-8n9xk 0 (0%) 0 (0%) 0 (0%) 0 (0%) openshift-operator-lifecycle-manager olm-operator-6bc4bc64f8-9h8wx 0 (0%) 0 (0%) 0 (0%) 0 (0%) openshift-operator-lifecycle-manager packageserver-86866ccd8-f9f7l 0 (0%) 0 (0%) 0 (0%) 0 (0%) openshift-operator-lifecycle-manager packageserver-86866ccd8-jgpnh 0 (0%) 0 (0%) 0 (0%) 0 (0%) openshift-sdn ovs-8n667 200m (5%) 0 (0%) 400Mi (3%) 0 (0%) openshift-sdn sdn-controller-2zx7r 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-sdn sdn-p7wm6 100m (2%) 0 (0%) 200Mi (1%) 0 (0%) openshift-service-ca-operator service-ca-operator-987fc4cdd-zkmwq 10m (0%) 0 (0%) 80Mi (0%) 0 (0%) openshift-service-ca apiservice-cabundle-injector-5bf59477d7-nlb4b 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-service-ca configmap-cabundle-injector-6cfcd498d7-4qtwx 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-service-ca service-serving-cert-signer-7789d64745-7bwq7 10m (0%) 0 (0%) 120Mi (0%) 0 (0%) openshift-service-catalog-apiserver-operator openshift-service-catalog-apiserver-operator-5dbfcf7bc5-zlq8l 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) openshift-service-catalog-controller-manager-operator openshift-service-catalog-controller-manager-operator-5bf65dp2g 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) CPU Requests CPU Limits Memory Requests Memory Limits ------------ ---------- --------------- ------------- 1630m (46%) 0 (0%) 4789Mi (36%) 1012Mi (7%) Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1d 1d 1 kubelet, master-0 Normal Starting Starting kubelet. 1d 1d 8 kubelet, master-0 Normal NodeHasSufficientMemory Node master-0 status is now: NodeHasSufficientMemory 1d 1d 8 kubelet, master-0 Normal NodeHasNoDiskPressure Node master-0 status is now: NodeHasNoDiskPressure 1d 1d 7 kubelet, master-0 Normal NodeHasSufficientPID Node master-0 status is now: NodeHasSufficientPID 1d 1d 1 kubelet, master-0 Normal NodeAllocatableEnforced Updated Node Allocatable limit across pods 30m 30m 1 kubelet, master-0 Normal Starting Starting kubelet. 30m 30m 8 kubelet, master-0 Normal NodeHasSufficientMemory Node master-0 status is now: NodeHasSufficientMemory 30m 30m 8 kubelet, master-0 Normal NodeHasNoDiskPressure Node master-0 status is now: NodeHasNoDiskPressure 30m 30m 7 kubelet, master-0 Normal NodeHasSufficientPID Node master-0 status is now: NodeHasSufficientPID 30m 30m 1 kubelet, master-0 Normal NodeAllocatableEnforced Updated Node Allocatable limit across pods
We're looking into this...The thing with this annotation is that we now have "oc debug node/nodename" which defeats the annotation altogether since anyone can oc debug, chroot and do whatever. I'm not super sure we want this at all now and we can't really do anything right now other than reporting "accessed" (i.e. we're not reconciling or doing anything else if someone messes on the host).
Yeah, I think we either need to: - Drop this attempt at annotating - Add an annotation if any privileged pod that e.g. has an interactive TTY landed on a node, as well as SSH Short term I think we should just drop the bit from the MOTD in RHCOS.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1382