Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1885365

Summary: MCO does not respect the install section of systemd files when enabling
Product: OpenShift Container Platform Reporter: Tim Rozet <trozet>
Component: Machine Config OperatorAssignee: Yu Qi Zhang <jerzhang>
Machine Config Operator sub component: Machine Config Operator QA Contact: Michael Nguyen <mnguyen>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: high CC: cpassare, mkrejci, rbrattai, rioliu, weliang, wking, yanyang
Version: 4.6   
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Since the beginning of OCP4, the MCO never properly respected the Install section for enabling/disabling units, instead hardcoding to multi-user.target.wants Consequence: Any unit not targetting multi-user.target.wants wrongly targets multi-user.target.wants Fix: now uses systemd directly to determine where the service should be linked to, so install section of a unit is properly evaluated Result: services should be enabled/disabled correctly
 Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:23:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1882667    

Description Tim Rozet 2020-10-05 18:08:08 UTC 
Description of problem:
When specifying the install section for a systemd service, the enabled service should be symlinked there. However, this seems inconsistent with MCO. I see some nodes the unit is enabled in the wrong place.

Thanks Weibin. Looking at your cluster I can see why this failed. The broken node is on 4.6. However:

[root@weliang52-9pll7-compute-0 ~]# stat /host/etc/systemd/system/network-online.target.wants/ovs-configuration.service
stat: cannot stat '/host/etc/systemd/system/network-online.target.wants/ovs-configuration.service': No such file or directory

^We dont find the ovs-configuration service so we start OVS containers. Now OVS is running in container and systemd and things break. However, I see ovs-configuration was placed into:

/host/etc/systemd/system/multi-user.target.wants/ovs-configuration.service


Even though the service specifies install into network-online.target:

[Unit]
Description=Configures OVS with proper host networking configuration
# Removal of this file signals firstboot completion
ConditionPathExists=!/etc/ignition-machine-config-encapsulated.json
# This service is used to move a physical NIC into OVS and reconfigure OVS to use the host IP
Requires=openvswitch.service
Wants=NetworkManager-wait-online.service
After=NetworkManager-wait-online.service openvswitch.service network.service
Before=network-online.target kubelet.service crio.service node-valid-hostname.service

[Service]
# Need oneshot to delay kubelet
Type=oneshot
ExecStart=/usr/local/bin/configure-ovs.sh OVNKubernetes
StandardOutput=journal+console
StandardError=journal+console

[Install]
WantedBy=network-online.target

sh-4.4# systemctl disable ovs-configuration.service
Removed /etc/systemd/system/multi-user.target.wants/ovs-configuration.service.
sh-4.4# systemctl enable ovs-configuration.service
Created symlink /etc/systemd/system/network-online.target.wants/ovs-configuration.service → /etc/systemd/system/ovs-configuration.service.
sh-4.4# stat /etc/systemd/system/network-online.target.wants/ovs-configuration.service
  File: /etc/systemd/system/network-online.target.wants/ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service
  Size: 45              Blocks: 0          IO Block: 4096   symbolic link
Device: fd00h/64768d    Inode: 180451495   Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:container_file_t:s0
Access: 2020-10-05 17:49:31.598094614 +0000
Modify: 2020-10-05 17:49:31.497090653 +0000
Change: 2020-10-05 17:49:31.498090692 +0000
 Birth: -
sh-4.4#

^ By simply disabling/re-enabling we can see the file is now installed into the right place.
Comment 1 Tim Rozet 2020-10-05 18:13:00 UTC 
It looks like this is happening because the path is hardcoded:

https://github.com/openshift/machine-config-operator/blob/bddec155817fb91a1c7fc1d1bed0fa488034e167/pkg/daemon/daemon.go#L136
	wantsPathSystemd = "/etc/systemd/system/multi-user.target.wants/"

https://github.com/openshift/machine-config-operator/blob/81430e1653280317c7f586bda0683a468e557cf4/pkg/daemon/update.go#L1169

Instead of reading it from the unit file.
Comment 3 Yu Qi Zhang 2020-10-06 22:53:42 UTC 
*** Bug 1866865 has been marked as a duplicate of this bug. ***
Comment 4 Yu Qi Zhang 2020-10-06 22:54:12 UTC 
A couple of things:

1. The MCO always did enabling/disabling to multi-user.target.wants so this isn't a regression
See https://github.com/openshift/machine-config-operator/blob/81430e1653280317c7f586bda0683a468e557cf4/pkg/daemon/update.go#L1169 and line 1189
this I think just never worked properly but somehow nobody really did it post-install so it was never caught.

2. this is actually inconsistent with ignition behaviour, which writes systemd-presets to do the enabling/disabling

with that in mind, could you tell me when you added this config and how? Because if this was during installation, ignition should have added it via systemd-preset, and if that linked wrong then it might be a bug somewhere else. If this was post-installation or e.g. RHEL UPI (which uses MCD onceFrom to write directly) then its likely due to the above.
Comment 5 Tim Rozet 2020-10-07 00:45:14 UTC 
Targeting 4.7 is fine. I believe Weibin saw this on bare metal IPI: https://bugzilla.redhat.com/show_bug.cgi?id=1880591#c42 Weibin can you confirm please?

This is service is installed via:
https://github.com/openshift/machine-config-operator/blob/master/templates/common/_base/units/ovs-configuration.service.yaml#L22

The problem is some nodes have the right symlink and others dont. I'm guessing something disables/renables the unit later and it gets linked into the right place in some cases.
Comment 6 Weibin Liang 2020-10-07 13:16:31 UTC 
(In reply to Tim Rozet from comment #5)
> Targeting 4.7 is fine. I believe Weibin saw this on bare metal IPI:
> https://bugzilla.redhat.com/show_bug.cgi?id=1880591#c42 Weibin can you
> confirm please?


The problem can be reproduced in UPI-on-baremetal, not IPI 



> This is service is installed via:
> https://github.com/openshift/machine-config-operator/blob/master/templates/
> common/_base/units/ovs-configuration.service.yaml#L22
> 
> The problem is some nodes have the right symlink and others dont. I'm
> guessing something disables/renables the unit later and it gets linked into
> the right place in some cases.
Comment 7 Ross Brattain 2020-10-20 11:38:10 UTC 
I also observed that during the replace master node procedure the new master node was missing the multi-user.target.wants/ symlink

original master

241173104      0 lrwxrwxrwx   1  root     root           45 Oct  4 13:58 /etc/systemd/system/multi-user.target.wants/ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service
243270161      0 lrwxrwxrwx   1  root     root           45 Oct  4 13:32 /etc/systemd/system/network-online.target.wants/ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service
234882135      4 -rw-r--r--   1  root     root          738 Oct  4 13:58 /etc/systemd/system/ovs-configuration.service

new master

243270161      0 lrwxrwxrwx   1  root     root           45 Oct  5 02:21 /etc/systemd/system/network-online.target.wants/ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service
234882136      4 -rw-r--r--   1  root     root          738 Oct  5 02:20 /etc/systemd/system/ovs-configuration.service
Comment 8 Yu Qi Zhang 2020-10-26 13:27:08 UTC 
The proposed fix in https://github.com/openshift/machine-config-operator/pull/2142 should fix this issue but it is currently blocked on some performance slowdowns. I am trying to figure that out still.
Comment 9 Tim Rozet 2020-12-02 19:57:16 UTC 
Raising severity as this blocks UPI on RHEL with OVN.
Comment 10 Anurag saxena 2020-12-02 20:03:18 UTC 
Marking at as a testblocker as per https://bugzilla.redhat.com/show_bug.cgi?id=1882667#c24
Comment 11 Yu Qi Zhang 2020-12-04 21:35:43 UTC 
Team was focused on feature completion. Will prioritize this for next sprint
Comment 14 Michael Nguyen 2021-01-05 22:03:30 UTC 
Verified on 4.7.0-0.nightly-2021-01-05-055003.  Verified /etc/systemd/system for unit files in the correct directories after adding the systemd service and then verified them again after enabling the service.  test.service is symlinked under network-online.target.Wants

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2021-01-05-055003   True        False         152m    Cluster version is 4.7.0-0.nightly-2021-01-05-055003

$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
00-worker                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
01-master-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
01-master-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
01-worker-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
01-worker-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
99-master-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
99-master-ssh                                                                                 3.1.0             118m
99-worker-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
99-worker-ssh                                                                                 3.1.0             118m
rendered-master-67e43dd971e9c2aa62c313cf04414565   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
rendered-worker-d4723287e9b9741f455c4182a64f9739   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m

$ cat 1885365-systemd-unit-install.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 98-test-service
spec:
  config:
    ignition:
      version: 3.1.0
    systemd:
      units:
        - name: test.service
          contents: |
            [Unit]
            Description=Hello Test Service
            After=network-online.target
            [Service]
            Type=simple
            ExecStart=/usr/bin/echo hello
            [Install]
            WantedBy=network-online.target
$ oc create -f 1885365-systemd-unit-install.yaml 
machineconfig.machineconfiguration.openshift.io/98-test-service created
$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
00-worker                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
01-master-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
01-master-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
01-worker-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
01-worker-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
98-test-service                                                                               3.1.0             8s
99-master-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
99-master-ssh                                                                                 3.1.0             118m
99-worker-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
99-worker-ssh                                                                                 3.1.0             118m
rendered-master-67e43dd971e9c2aa62c313cf04414565   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
rendered-worker-d4723287e9b9741f455c4182a64f9739   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             110m
rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             2s
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-d4723287e9b9741f455c4182a64f9739   False     True       False      3              0                   0                     0                      112m
$ watch oc get mcp/worker
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988   True      False      False      3              3                   3                     0                      125m


$ oc get nodes
NAME                                         STATUS   ROLES    AGE    VERSION
ip-10-0-138-209.us-west-2.compute.internal   Ready    worker   114m   v1.20.0+87544c5
ip-10-0-148-72.us-west-2.compute.internal    Ready    master   122m   v1.20.0+87544c5
ip-10-0-160-130.us-west-2.compute.internal   Ready    worker   113m   v1.20.0+87544c5
ip-10-0-161-209.us-west-2.compute.internal   Ready    master   122m   v1.20.0+87544c5
ip-10-0-201-77.us-west-2.compute.internal    Ready    worker   113m   v1.20.0+87544c5
ip-10-0-211-131.us-west-2.compute.internal   Ready    master   126m   v1.20.0+87544c5
$ oc debug node/ip-10-0-138-209.us-west-2.compute.internal
Starting pod/ip-10-0-138-209us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# cd /etc/systemd/system
system/        system-preset/ system.conf    system.conf.d/ 
sh-4.4# cd /etc/systemd/system/ 
basic.target.wants/                                      multi-user.target.wants/
crio.service.d/                                          network-online.target.requires/
crio.service.requires/                                   network-online.target.wants/
ctrl-alt-del.target                                      node-valid-hostname.service
dbus-org.freedesktop.nm-dispatcher.service               nodeip-configuration.service
dbus-org.freedesktop.timedate1.service                   ovs-configuration.service
dev-virtio\x2dports-org.qemu.guest_agent.0.device.wants/ ovs-vswitchd.service.d/
docker.socket.d/                                         ovsdb-server.service.d/
getty.target.wants/                                      pivot.service.d/
kubelet.service                                          sockets.target.wants/
kubelet.service.d/                                       sysinit.target.wants/
kubelet.service.requires/                                test.service
local-fs.target.wants/                                   timers.target.wants/
machine-config-daemon-firstboot.service                  vmtoolsd.service.requires/
machine-config-daemon-firstboot.service.requires/        zincati.service.d/
machine-config-daemon-pull.service                       
sh-4.4# cd /etc/systemd/system/network-online.target.wants/
sh-4.4# ls
NetworkManager-wait-online.service  ovs-configuration.service
sh-4.4# ls -la
total 4
drwxr-xr-x.  2 root root   81 Jan  5 19:09 .
drwxr-xr-x. 23 root root 4096 Jan  5 20:52 ..
lrwxrwxrwx.  1 root root   58 Jan  5 19:09 NetworkManager-wait-online.service -> /usr/lib/systemd/system/NetworkManager-wait-online.service
lrwxrwxrwx.  1 root root   45 Jan  5 19:09 ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service
sh-4.4# cat ../test.service 
[Unit]
Description=Hello Test Service
After=network-online.target
[Service]
Type=simple
ExecStart=/usr/bin/echo hello
[Install]
WantedBy=network-online.target
sh-4.4# systemctl status test.service
● test.service - Hello Test Service
   Loaded: loaded (/etc/systemd/system/test.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
sh-4.4# exkt
sh: exkt: command not found
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...




$ cat << EOF > 1885365-enable-test-service.yaml
> apiVersion: machineconfiguration.openshift.io/v1
> kind: MachineConfig
> metadata:
>   labels:
>     machineconfiguration.openshift.io/role: worker
>   name: 98-test-service-2
> spec:
>   config:
>     ignition:
>       version: 3.1.0
>     systemd:
>       units:
>         - name: test.service
>           enabled: true
> EOF
$ oc create -f 1885365-enable-test-service.yaml 
machineconfig.machineconfiguration.openshift.io/98-test-service-2 created
$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             139m
00-worker                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             139m
01-master-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             139m
01-master-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             139m
01-worker-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             139m
01-worker-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             139m
98-test-service                                                                               3.1.0             28m
98-test-service-2                                                                             3.1.0             5s
99-master-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             139m
99-master-ssh                                                                                 3.1.0             147m
99-worker-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             139m
99-worker-ssh                                                                                 3.1.0             147m
rendered-master-67e43dd971e9c2aa62c313cf04414565   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             139m
rendered-worker-d4723287e9b9741f455c4182a64f9739   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             139m
rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             28m
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988   False     True       False      3              0                   0                     0                      141m
$ watch oc get mcp/worker
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-9930bad8115020c22867d05d819226c7   True      False      False      3              3                   3                     0                      151m
$ oc debug node/ip-10-0-138-209.us-west-2.compute.internal
Starting pod/ip-10-0-138-209us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# systemctl status test.service
● test.service - Hello Test Service
   Loaded: loaded (/etc/systemd/system/test.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Tue 2021-01-05 21:26:46 UTC; 4min 8s ago
  Process: 1406 ExecStart=/usr/bin/echo hello (code=exited, status=0/SUCCESS)
 Main PID: 1406 (code=exited, status=0/SUCCESS)
      CPU: 1ms

Jan 05 21:26:46 ip-10-0-138-209 systemd[1]: Started Hello Test Service.
Jan 05 21:26:46 ip-10-0-138-209 echo[1406]: hello
Jan 05 21:26:46 ip-10-0-138-209 systemd[1]: test.service: Succeeded.
Jan 05 21:26:46 ip-10-0-138-209 systemd[1]: test.service: Consumed 1ms CPU time
sh-4.4# cd /etc/systemd/system
sh-4.4# ls -la network-online.target.wants/
total 4
drwxr-xr-x.  2 root root  101 Jan  5 21:26 .
drwxr-xr-x. 23 root root 4096 Jan  5 21:26 ..
lrwxrwxrwx.  1 root root   58 Jan  5 19:09 NetworkManager-wait-online.service -> /usr/lib/systemd/system/NetworkManager-wait-online.service
lrwxrwxrwx.  1 root root   45 Jan  5 19:09 ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service
lrwxrwxrwx.  1 root root   32 Jan  5 21:26 test.service -> /etc/systemd/system/test.service
sh-4.4# cat /etc/systemd/system/test.service
[Unit]
Description=Hello Test Service
After=network-online.target
[Service]
Type=simple
ExecStart=/usr/bin/echo hello
[Install]
WantedBy=network-online.target
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...


$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
00-worker                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
01-master-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
01-master-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
01-worker-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
01-worker-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
98-test-service                                                                               3.1.0             41m
98-test-service-2                                                                             3.1.0             12m
99-master-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
99-master-ssh                                                                                 3.1.0             159m
99-worker-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
99-worker-ssh                                                                                 3.1.0             159m
rendered-master-67e43dd971e9c2aa62c313cf04414565   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
rendered-worker-9930bad8115020c22867d05d819226c7   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             12m
rendered-worker-d4723287e9b9741f455c4182a64f9739   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             40m
$ oc delete mc/98-test-service-2
machineconfig.machineconfiguration.openshift.io "98-test-service-2" deleted
$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
00-worker                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
01-master-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
01-master-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
01-worker-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
01-worker-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
98-test-service                                                                               3.1.0             41m
99-master-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
99-master-ssh                                                                                 3.1.0             159m
99-worker-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
99-worker-ssh                                                                                 3.1.0             159m
rendered-master-67e43dd971e9c2aa62c313cf04414565   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
rendered-worker-9930bad8115020c22867d05d819226c7   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             12m
rendered-worker-d4723287e9b9741f455c4182a64f9739   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             151m
rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             41m
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-9930bad8115020c22867d05d819226c7   True      False      False      3              3                   3                     0                      153m
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-9930bad8115020c22867d05d819226c7   False     True       False      3              0                   0                     0                      153m
$ watch oc get mcp/worker
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988   True      False      False      3              3                   3                     0                      167m
$ oc debug node/ip-10-0-138-209.us-west-2.compute.internal
Starting pod/ip-10-0-138-209us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# systemctl status test.service
● test.service - Hello Test Service
   Loaded: loaded (/etc/systemd/system/test.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
sh-4.4# cd /etc/systemd/system
sh-4.4# ls -la network-online.target.wants/
total 4
drwxr-xr-x.  2 root root   81 Jan  5 21:32 .
drwxr-xr-x. 23 root root 4096 Jan  5 21:32 ..
lrwxrwxrwx.  1 root root   58 Jan  5 19:09 NetworkManager-wait-online.service -> /usr/lib/systemd/system/NetworkManager-wait-online.service
lrwxrwxrwx.  1 root root   45 Jan  5 19:09 ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...



$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
00-worker                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
01-master-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
01-master-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
01-worker-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
01-worker-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
98-test-service                                                                               3.1.0             55m
99-master-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
99-master-ssh                                                                                 3.1.0             173m
99-worker-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
99-worker-ssh                                                                                 3.1.0             173m
rendered-master-67e43dd971e9c2aa62c313cf04414565   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
rendered-worker-9930bad8115020c22867d05d819226c7   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             26m
rendered-worker-d4723287e9b9741f455c4182a64f9739   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             55m
$ oc delete mc/98-test-service
machineconfig.machineconfiguration.openshift.io "98-test-service" deleted
$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
00-worker                                          ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
01-master-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
01-master-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
01-worker-container-runtime                        ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
01-worker-kubelet                                  ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
99-master-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
99-master-ssh                                                                                 3.1.0             173m
99-worker-generated-registries                     ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
99-worker-ssh                                                                                 3.1.0             173m
rendered-master-67e43dd971e9c2aa62c313cf04414565   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
rendered-worker-9930bad8115020c22867d05d819226c7   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             26m
rendered-worker-d4723287e9b9741f455c4182a64f9739   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             166m
rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988   ca283c2500df8cdc787600e8fcbd311b99859538   3.2.0             55m
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-f32d2e2ab4c0bfc0bdb68eb8b305b988   False     True       False      3              0                   0                     0                      168m
$ watch oc get mcp/worker
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-d4723287e9b9741f455c4182a64f9739   True      False      False      3              3                   3                     0                      178m
$ oc debug node/ip-10-0-138-209.us-west-2.compute.internal
Starting pod/ip-10-0-138-209us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# systemctl status test.service
Unit test.service could not be found.
sh-4.4# cd /etc/systemd/system
sh-4.4# ls -la network-online.target.wants/
total 4
drwxr-xr-x.  2 root root   81 Jan  5 21:32 .
drwxr-xr-x. 23 root root 4096 Jan  5 21:47 ..
lrwxrwxrwx.  1 root root   58 Jan  5 19:09 NetworkManager-wait-online.service -> /usr/lib/systemd/system/NetworkManager-wait-online.service
lrwxrwxrwx.  1 root root   45 Jan  5 19:09 ovs-configuration.service -> /etc/systemd/system/ovs-configuration.service
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...
Comment 16 errata-xmlrpc 2021-02-24 15:23:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633