Description of problem: When specifying the install section for a systemd service, the enabled service should be symlinked there. However, this seems inconsistent with MCO. I see some nodes the unit is enabled in the wrong place. Thanks Weibin. Looking at your cluster I can see why this failed. The broken node is on 4.6. However: [root@weliang52-9pll7-compute-0 ~]# stat /host/etc/systemd/system/network-online.target.wants/ovs-configuration.service stat: cannot stat '/host/etc/systemd/system/network-online.target.wants/ovs-configuration.service': No such file or directory ^We dont find the ovs-configuration service so we start OVS containers. Now OVS is running in container and systemd and things break. However, I see ovs-configuration was placed into: /host/etc/systemd/system/multi-user.target.wants/ovs-configuration.service Even though the service specifies install into network-online.target: [Unit] Description=Configures OVS with proper host networking configuration # Removal of this file signals firstboot completion ConditionPathExists=!/etc/ignition-machine-config-encapsulated.json # This service is used to move a physical NIC into OVS and reconfigure OVS to use the host IP Requires=openvswitch.service Wants=NetworkManager-wait-online.service After=NetworkManager-wait-online.service openvswitch.service network.service Before=network-online.target kubelet.service crio.service node-valid-hostname.service [Service] # Need oneshot to delay kubelet Type=oneshot ExecStart=/usr/local/bin/configure-ovs.sh OVNKubernetes StandardOutput=journal+console StandardError=journal+console [Install] WantedBy=network-online.target sh-4.4# systemctl disable ovs-configuration.service Removed /etc/systemd/system/multi-user.target.wants/ovs-configuration.service. sh-4.4# systemctl enable ovs-configuration.service Created symlink /etc/systemd/system/network-online.target.wants/ovs-configuration.service → /etc/systemd/system/ovs-configuration.service. sh-4.4# stat /etc/systemd/system/network-online.target.wants/ovs-configuration.service File: /etc/systemd/system/network-online.target.wants/ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service Size: 45 Blocks: 0 IO Block: 4096 symbolic link Device: fd00h/64768d Inode: 180451495 Links: 1 Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:container_file_t:s0 Access: 2020-10-05 17:49:31.598094614 +0000 Modify: 2020-10-05 17:49:31.497090653 +0000 Change: 2020-10-05 17:49:31.498090692 +0000 Birth: - sh-4.4# ^ By simply disabling/re-enabling we can see the file is now installed into the right place.
It looks like this is happening because the path is hardcoded: https://github.com/openshift/machine-config-operator/blob/bddec155817fb91a1c7fc1d1bed0fa488034e167/pkg/daemon/daemon.go#L136 wantsPathSystemd = "/etc/systemd/system/multi-user.target.wants/" https://github.com/openshift/machine-config-operator/blob/81430e1653280317c7f586bda0683a468e557cf4/pkg/daemon/update.go#L1169 Instead of reading it from the unit file.
*** Bug 1866865 has been marked as a duplicate of this bug. ***
A couple of things: 1. The MCO always did enabling/disabling to multi-user.target.wants so this isn't a regression See https://github.com/openshift/machine-config-operator/blob/81430e1653280317c7f586bda0683a468e557cf4/pkg/daemon/update.go#L1169 and line 1189 this I think just never worked properly but somehow nobody really did it post-install so it was never caught. 2. this is actually inconsistent with ignition behaviour, which writes systemd-presets to do the enabling/disabling with that in mind, could you tell me when you added this config and how? Because if this was during installation, ignition should have added it via systemd-preset, and if that linked wrong then it might be a bug somewhere else. If this was post-installation or e.g. RHEL UPI (which uses MCD onceFrom to write directly) then its likely due to the above.
Targeting 4.7 is fine. I believe Weibin saw this on bare metal IPI: https://bugzilla.redhat.com/show_bug.cgi?id=1880591#c42 Weibin can you confirm please? This is service is installed via: https://github.com/openshift/machine-config-operator/blob/master/templates/common/_base/units/ovs-configuration.service.yaml#L22 The problem is some nodes have the right symlink and others dont. I'm guessing something disables/renables the unit later and it gets linked into the right place in some cases.
(In reply to Tim Rozet from comment #5) > Targeting 4.7 is fine. I believe Weibin saw this on bare metal IPI: > https://bugzilla.redhat.com/show_bug.cgi?id=1880591#c42 Weibin can you > confirm please? The problem can be reproduced in UPI-on-baremetal, not IPI > This is service is installed via: > https://github.com/openshift/machine-config-operator/blob/master/templates/ > common/_base/units/ovs-configuration.service.yaml#L22 > > The problem is some nodes have the right symlink and others dont. I'm > guessing something disables/renables the unit later and it gets linked into > the right place in some cases.
I also observed that during the replace master node procedure the new master node was missing the multi-user.target.wants/ symlink original master 241173104 0 lrwxrwxrwx 1 root root 45 Oct 4 13:58 /etc/systemd/system/multi-user.target.wants/ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service 243270161 0 lrwxrwxrwx 1 root root 45 Oct 4 13:32 /etc/systemd/system/network-online.target.wants/ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service 234882135 4 -rw-r--r-- 1 root root 738 Oct 4 13:58 /etc/systemd/system/ovs-configuration.service new master 243270161 0 lrwxrwxrwx 1 root root 45 Oct 5 02:21 /etc/systemd/system/network-online.target.wants/ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service 234882136 4 -rw-r--r-- 1 root root 738 Oct 5 02:20 /etc/systemd/system/ovs-configuration.service
The proposed fix in https://github.com/openshift/machine-config-operator/pull/2142 should fix this issue but it is currently blocked on some performance slowdowns. I am trying to figure that out still.
Raising severity as this blocks UPI on RHEL with OVN.
Marking at as a testblocker as per https://bugzilla.redhat.com/show_bug.cgi?id=1882667#c24
Team was focused on feature completion. Will prioritize this for next sprint
Verified on 4.7.0-0.nightly-2021-01-05-055003. Verified /etc/systemd/system for unit files in the correct directories after adding the systemd service and then verified them again after enabling the service. test.service is symlinked under network-online.target.Wants $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2021-01-05-055003 True False 152m Cluster version is 4.7.0-0.nightly-2021-01-05-055003 $ oc get mc NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 00-worker ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 01-master-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 01-master-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 01-worker-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 01-worker-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 99-master-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 99-master-ssh 3.1.0 118m 99-worker-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 99-worker-ssh 3.1.0 118m rendered-master-67e43dd971e9c2aa62c313cf04414565 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m rendered-worker-d4723287e9b9741f455c4182a64f9739 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m $ cat 1885365-systemd-unit-install.yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: 98-test-service spec: config: ignition: version: 3.1.0 systemd: units: - name: test.service contents: | [Unit] Description=Hello Test Service After=network-online.target [Service] Type=simple ExecStart=/usr/bin/echo hello [Install] WantedBy=network-online.target $ oc create -f 1885365-systemd-unit-install.yaml machineconfig.machineconfiguration.openshift.io/98-test-service created $ oc get mc NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 00-worker ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 01-master-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 01-master-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 01-worker-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 01-worker-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 98-test-service 3.1.0 8s 99-master-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 99-master-ssh 3.1.0 118m 99-worker-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m 99-worker-ssh 3.1.0 118m rendered-master-67e43dd971e9c2aa62c313cf04414565 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m rendered-worker-d4723287e9b9741f455c4182a64f9739 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 110m rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 2s $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-d4723287e9b9741f455c4182a64f9739 False True False 3 0 0 0 112m $ watch oc get mcp/worker $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988 True False False 3 3 3 0 125m $ oc get nodes NAME STATUS ROLES AGE VERSION ip-10-0-138-209.us-west-2.compute.internal Ready worker 114m v1.20.0+87544c5 ip-10-0-148-72.us-west-2.compute.internal Ready master 122m v1.20.0+87544c5 ip-10-0-160-130.us-west-2.compute.internal Ready worker 113m v1.20.0+87544c5 ip-10-0-161-209.us-west-2.compute.internal Ready master 122m v1.20.0+87544c5 ip-10-0-201-77.us-west-2.compute.internal Ready worker 113m v1.20.0+87544c5 ip-10-0-211-131.us-west-2.compute.internal Ready master 126m v1.20.0+87544c5 $ oc debug node/ip-10-0-138-209.us-west-2.compute.internal Starting pod/ip-10-0-138-209us-west-2computeinternal-debug ... To use host binaries, run `chroot /host` If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# cd /etc/systemd/system system/ system-preset/ system.conf system.conf.d/ sh-4.4# cd /etc/systemd/system/ basic.target.wants/ multi-user.target.wants/ crio.service.d/ network-online.target.requires/ crio.service.requires/ network-online.target.wants/ ctrl-alt-del.target node-valid-hostname.service dbus-org.freedesktop.nm-dispatcher.service nodeip-configuration.service dbus-org.freedesktop.timedate1.service ovs-configuration.service dev-virtio\x2dports-org.qemu.guest_agent.0.device.wants/ ovs-vswitchd.service.d/ docker.socket.d/ ovsdb-server.service.d/ getty.target.wants/ pivot.service.d/ kubelet.service sockets.target.wants/ kubelet.service.d/ sysinit.target.wants/ kubelet.service.requires/ test.service local-fs.target.wants/ timers.target.wants/ machine-config-daemon-firstboot.service vmtoolsd.service.requires/ machine-config-daemon-firstboot.service.requires/ zincati.service.d/ machine-config-daemon-pull.service sh-4.4# cd /etc/systemd/system/network-online.target.wants/ sh-4.4# ls NetworkManager-wait-online.service ovs-configuration.service sh-4.4# ls -la total 4 drwxr-xr-x. 2 root root 81 Jan 5 19:09 . drwxr-xr-x. 23 root root 4096 Jan 5 20:52 .. lrwxrwxrwx. 1 root root 58 Jan 5 19:09 NetworkManager-wait-online.service -> /usr/lib/systemd/system/NetworkManager-wait-online.service lrwxrwxrwx. 1 root root 45 Jan 5 19:09 ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service sh-4.4# cat ../test.service [Unit] Description=Hello Test Service After=network-online.target [Service] Type=simple ExecStart=/usr/bin/echo hello [Install] WantedBy=network-online.target sh-4.4# systemctl status test.service ● test.service - Hello Test Service Loaded: loaded (/etc/systemd/system/test.service; disabled; vendor preset: disabled) Active: inactive (dead) sh-4.4# exkt sh: exkt: command not found sh-4.4# exit exit sh-4.2# exit exit Removing debug pod ... $ cat << EOF > 1885365-enable-test-service.yaml > apiVersion: machineconfiguration.openshift.io/v1 > kind: MachineConfig > metadata: > labels: > machineconfiguration.openshift.io/role: worker > name: 98-test-service-2 > spec: > config: > ignition: > version: 3.1.0 > systemd: > units: > - name: test.service > enabled: true > EOF $ oc create -f 1885365-enable-test-service.yaml machineconfig.machineconfiguration.openshift.io/98-test-service-2 created $ oc get mc NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 139m 00-worker ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 139m 01-master-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 139m 01-master-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 139m 01-worker-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 139m 01-worker-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 139m 98-test-service 3.1.0 28m 98-test-service-2 3.1.0 5s 99-master-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 139m 99-master-ssh 3.1.0 147m 99-worker-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 139m 99-worker-ssh 3.1.0 147m rendered-master-67e43dd971e9c2aa62c313cf04414565 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 139m rendered-worker-d4723287e9b9741f455c4182a64f9739 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 139m rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 28m $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988 False True False 3 0 0 0 141m $ watch oc get mcp/worker $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-9930bad8115020c22867d05d819226c7 True False False 3 3 3 0 151m $ oc debug node/ip-10-0-138-209.us-west-2.compute.internal Starting pod/ip-10-0-138-209us-west-2computeinternal-debug ... To use host binaries, run `chroot /host` If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# systemctl status test.service ● test.service - Hello Test Service Loaded: loaded (/etc/systemd/system/test.service; enabled; vendor preset: disabled) Active: inactive (dead) since Tue 2021-01-05 21:26:46 UTC; 4min 8s ago Process: 1406 ExecStart=/usr/bin/echo hello (code=exited, status=0/SUCCESS) Main PID: 1406 (code=exited, status=0/SUCCESS) CPU: 1ms Jan 05 21:26:46 ip-10-0-138-209 systemd[1]: Started Hello Test Service. Jan 05 21:26:46 ip-10-0-138-209 echo[1406]: hello Jan 05 21:26:46 ip-10-0-138-209 systemd[1]: test.service: Succeeded. Jan 05 21:26:46 ip-10-0-138-209 systemd[1]: test.service: Consumed 1ms CPU time sh-4.4# cd /etc/systemd/system sh-4.4# ls -la network-online.target.wants/ total 4 drwxr-xr-x. 2 root root 101 Jan 5 21:26 . drwxr-xr-x. 23 root root 4096 Jan 5 21:26 .. lrwxrwxrwx. 1 root root 58 Jan 5 19:09 NetworkManager-wait-online.service -> /usr/lib/systemd/system/NetworkManager-wait-online.service lrwxrwxrwx. 1 root root 45 Jan 5 19:09 ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service lrwxrwxrwx. 1 root root 32 Jan 5 21:26 test.service -> /etc/systemd/system/test.service sh-4.4# cat /etc/systemd/system/test.service [Unit] Description=Hello Test Service After=network-online.target [Service] Type=simple ExecStart=/usr/bin/echo hello [Install] WantedBy=network-online.target sh-4.4# exit exit sh-4.2# exit exit Removing debug pod ... $ oc get mc NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 00-worker ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 01-master-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 01-master-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 01-worker-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 01-worker-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 98-test-service 3.1.0 41m 98-test-service-2 3.1.0 12m 99-master-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 99-master-ssh 3.1.0 159m 99-worker-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 99-worker-ssh 3.1.0 159m rendered-master-67e43dd971e9c2aa62c313cf04414565 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m rendered-worker-9930bad8115020c22867d05d819226c7 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 12m rendered-worker-d4723287e9b9741f455c4182a64f9739 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 40m $ oc delete mc/98-test-service-2 machineconfig.machineconfiguration.openshift.io "98-test-service-2" deleted $ oc get mc NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 00-worker ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 01-master-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 01-master-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 01-worker-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 01-worker-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 98-test-service 3.1.0 41m 99-master-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 99-master-ssh 3.1.0 159m 99-worker-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m 99-worker-ssh 3.1.0 159m rendered-master-67e43dd971e9c2aa62c313cf04414565 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m rendered-worker-9930bad8115020c22867d05d819226c7 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 12m rendered-worker-d4723287e9b9741f455c4182a64f9739 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 151m rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 41m $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-9930bad8115020c22867d05d819226c7 True False False 3 3 3 0 153m $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-9930bad8115020c22867d05d819226c7 False True False 3 0 0 0 153m $ watch oc get mcp/worker $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988 True False False 3 3 3 0 167m $ oc debug node/ip-10-0-138-209.us-west-2.compute.internal Starting pod/ip-10-0-138-209us-west-2computeinternal-debug ... To use host binaries, run `chroot /host` If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# systemctl status test.service ● test.service - Hello Test Service Loaded: loaded (/etc/systemd/system/test.service; disabled; vendor preset: disabled) Active: inactive (dead) sh-4.4# cd /etc/systemd/system sh-4.4# ls -la network-online.target.wants/ total 4 drwxr-xr-x. 2 root root 81 Jan 5 21:32 . drwxr-xr-x. 23 root root 4096 Jan 5 21:32 .. lrwxrwxrwx. 1 root root 58 Jan 5 19:09 NetworkManager-wait-online.service -> /usr/lib/systemd/system/NetworkManager-wait-online.service lrwxrwxrwx. 1 root root 45 Jan 5 19:09 ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service sh-4.4# exit exit sh-4.2# exit exit Removing debug pod ... $ oc get mc NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 00-worker ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 01-master-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 01-master-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 01-worker-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 01-worker-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 98-test-service 3.1.0 55m 99-master-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 99-master-ssh 3.1.0 173m 99-worker-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 99-worker-ssh 3.1.0 173m rendered-master-67e43dd971e9c2aa62c313cf04414565 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m rendered-worker-9930bad8115020c22867d05d819226c7 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 26m rendered-worker-d4723287e9b9741f455c4182a64f9739 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 55m $ oc delete mc/98-test-service machineconfig.machineconfiguration.openshift.io "98-test-service" deleted $ oc get mc NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 00-worker ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 01-master-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 01-master-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 01-worker-container-runtime ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 01-worker-kubelet ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 99-master-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 99-master-ssh 3.1.0 173m 99-worker-generated-registries ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m 99-worker-ssh 3.1.0 173m rendered-master-67e43dd971e9c2aa62c313cf04414565 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m rendered-worker-9930bad8115020c22867d05d819226c7 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 26m rendered-worker-d4723287e9b9741f455c4182a64f9739 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 166m rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988 ca283c2500df8cdc787600e8fcbd311b99859538 3.2.0 55m $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988 False True False 3 0 0 0 168m $ watch oc get mcp/worker $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-d4723287e9b9741f455c4182a64f9739 True False False 3 3 3 0 178m $ oc debug node/ip-10-0-138-209.us-west-2.compute.internal Starting pod/ip-10-0-138-209us-west-2computeinternal-debug ... To use host binaries, run `chroot /host` If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# systemctl status test.service Unit test.service could not be found. sh-4.4# cd /etc/systemd/system sh-4.4# ls -la network-online.target.wants/ total 4 drwxr-xr-x. 2 root root 81 Jan 5 21:32 . drwxr-xr-x. 23 root root 4096 Jan 5 21:47 .. lrwxrwxrwx. 1 root root 58 Jan 5 19:09 NetworkManager-wait-online.service -> /usr/lib/systemd/system/NetworkManager-wait-online.service lrwxrwxrwx. 1 root root 45 Jan 5 19:09 ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service sh-4.4# exit exit sh-4.2# exit exit Removing debug pod ...
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633