Bug 1895360 - Machine Config Daemon removes a file although its defined in the dropin
Summary: Machine Config Daemon removes a file although its defined in the dropin
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.0
Assignee: Antonio Murdaca
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On:
Blocks: 1899735
TreeView+ depends on / blocked
 
Reported: 2020-11-06 13:23 UTC by Vadim Rutkovsky
Modified: 2021-02-24 15:32 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:31:26 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2196 0 None closed Bug 1895360: pkg/daemon: don't delete a file if its replaced with a dropin 2021-02-14 10:55:35 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:32:00 UTC

Description Vadim Rutkovsky 2020-11-06 13:23:29 UTC
Steps To Reproduce:

1. Create systemd dropin via a file
2.  Realize that its suboptimal, convert into a dropin
3.  Ignition will rewrite the file, but MCD's deleteStaleData would remove it file since its no longer in .Storage.Files
4.  Machine would reboot and MCD would complain that file is not found.

This is blocking OKD 4.5 -> 4.6 upgrade, as in 4.5 we placed kubelet MCO dropin for proxy setup via storage.files and in 4.6 its replaced by a unit dropin.

Comment 2 Michael Nguyen 2020-11-24 22:46:07 UTC
Verified on 4.7.0-0.nightly-2020-11-24-113830



$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2020-11-24-113830   True        False         128m    Cluster version is 4.7.0-0.nightly-2020-11-24-113830
$ cat drop-in-file.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: drop-in-file
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf;base64,W1VuaXRdCg==
        filesystem: root
        mode: 0644
        path: /etc/systemd/system/machine-config-daemon-firstboot.service.d/override.conf
$ oc create -f drop-in-file.yaml 
machineconfig.machineconfiguration.openshift.io/drop-in-file created
$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
00-worker                                          5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-master-container-runtime                        5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-master-kubelet                                  5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-worker-container-runtime                        5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-worker-kubelet                                  5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
99-master-generated-registries                     5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
99-master-ssh                                                                                 3.1.0             161m
99-worker-generated-registries                     5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
99-worker-ssh                                                                                 3.1.0             161m
drop-in-file                                                                                  3.1.0             3s
rendered-master-06fc8e27c85447046c0bf7a773d9ab64   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
rendered-worker-4e48ec375c1e899ef0d4d8a9b6123ac1   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             32m
rendered-worker-f28ffee0e4a09287edecc8431995df67   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
00-worker                                          5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-master-container-runtime                        5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-master-kubelet                                  5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-worker-container-runtime                        5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-worker-kubelet                                  5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
99-master-generated-registries                     5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
99-master-ssh                                                                                 3.1.0             161m
99-worker-generated-registries                     5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
99-worker-ssh                                                                                 3.1.0             161m
drop-in-file                                                                                  3.1.0             8s
rendered-master-06fc8e27c85447046c0bf7a773d9ab64   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
rendered-worker-40dfad2479e4cbc293f887a8296183f4   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             3s
rendered-worker-4e48ec375c1e899ef0d4d8a9b6123ac1   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             32m
rendered-worker-f28ffee0e4a09287edecc8431995df67   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-40dfad2479e4cbc293f887a8296183f4   True      False      False      3              3                   3                     0                      3h20m
$ oc debug node/ip-10-0-134-48.us-west-2.compute.internal
Starting pod/ip-10-0-134-48us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# cat /etc/systemd/system/machine-config-daemon-firstboot.service.d/override.conf 
[Unit]
sh-4.4# exit
exit
sh-4.2# exit 
exit

Removing debug pod ...
$ oc edit mc/drop-in-file
machineconfig.machineconfiguration.openshift.io/drop-in-file edited
$ oc get mc/drop-in-file -o yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  creationTimestamp: "2020-11-24T21:10:34Z"
  generation: 3
  labels:
    machineconfiguration.openshift.io/role: worker
  managedFields:
  - apiVersion: machineconfiguration.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:machineconfiguration.openshift.io/role: {}
      f:spec:
        .: {}
        f:config:
          .: {}
          f:ignition:
            .: {}
            f:version: {}
          f:systemd:
            .: {}
            f:units: {}
    manager: oc
    operation: Update
    time: "2020-11-24T22:32:00Z"
  name: drop-in-file
  resourceVersion: "94477"
  selfLink: /apis/machineconfiguration.openshift.io/v1/machineconfigs/drop-in-file
  uid: c79aa973-f417-48d3-9359-77c1ec0ac54f
spec:
  config:
    ignition:
      version: 3.1.0
    systemd:
      units:
      - dropins:
        - contents: |
            [Unit]
          name: override.conf
        enabled: true
        name: machine-config-daemon-firstboot.service
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-f28ffee0e4a09287edecc8431995df67   False     True       False      3              0                   0                     0                      3h53m
$ oc debug node/ip-10-0-134-48.us-west-2.compute.internal
Starting pod/ip-10-0-134-48us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# cat /etc/systemd/system/machine-config-daemon-firstboot.service.d/override.conf 
[Unit]
sh-4.4# exit
exit
sh-4.2#

Comment 5 errata-xmlrpc 2021-02-24 15:31:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.