Bug 2032565 - Deploying compressed files with a MachineConfig resource degrades the MachineConfigPool
Summary: Deploying compressed files with a MachineConfig resource degrades the Machine...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.10
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.10.0
Assignee: Zack Zlotnik
QA Contact: Sergio
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-14 17:54 UTC by Sergio
Modified: 2022-03-16 07:16 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:33:57 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2874 0 None open Bug 2032565: makes config drift monitor aware of compressed files 2021-12-16 13:30:58 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:34:19 UTC

Description Sergio 2021-12-14 17:54:00 UTC
Description of problem:
When we create a MachineConfig using a compressed file, the MachineConfigPool is degraded. The pool cannot be recovered by editing the desiderconfig and currentconfig information in the degraded node.


Version-Release number of MCO (Machine Config Operator) (if applicable):
4.10.0-0.nightly-2021-12-14-083101

Platform (AWS, VSphere, Metal, etc.):
AWS

Are you certain that the root cause of the issue being reported is the MCO (Machine Config Operator)?
(Y/N/Not sure): Yes

How reproducible:
Always

Did you catch this issue by running a Jenkins job? If yes, please list:
1. Jenkins job:

2. Profile:


Steps to Reproduce:
1. Create this machine config to deploy a file in compressed format

cat << EOF | oc create -f -
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: gzip-compression-test-mc
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - contents:
          compression: gzip
          source: data:;base64,H4sIAAAAAAACAzWQQW5DMQhE9/8Uc4Aop2h33fYA1CYpko0dA1GOX9zf7owwM/PmYyzukGnRUUcbCyYO6uwXlKHGxdljgapMsSJ6BzfJpXHNg4MlrI8K5z7zWLRIlRrqCEejr5QH+ynN6HRXAjV5BF2BTz9Ypac4uuzHM0fqFzxCDDrMV1Twi1cRJ5ehiNaol3FKix0qJtvqV1NmfgZTJu8ZapwE6eVp9rY1KZwhKzLLSSt6LJ6Lv1krr0QXxXO0mOnHmSdZwWaMIq39d5RIgVvchfzQnQiTVg6x0uf9VXg6x24yaxilEBdylJhSKWUTTDHXkMrql8N2Welaok3a5Bi3mxQhVDZee9tH2zloVyRZiP1VG/16/ACjjAk6wQEAAA==
        path: /etc/test-file
EOF


Actual results:
The worker pool is degraded

$ oc get mcp worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-7c56e617174ffbdd5168f67d4156687c   False     True       True       2              1                   1                     1                      93m

The degraded node is reporting this reason for the degrade status

$ oc get node $NODE -o yaml
...
    machineconfiguration.openshift.io/reason: 'unexpected on-disk state validating
      against rendered-worker-f3c8ed833206d614ba32ba6e7896c18c: content mismatch for
      file "/etc/test-file"'
    machineconfiguration.openshift.io/state: Degraded


Expected results:
The file described in compressed format should be deployed without problems


Additional info:

It could be a regression due to proactive config drift monitor https://github.com/openshift/machine-config-operator/pull/2795. Compressed files will always error and cannot recover.

Comment 1 Zack Zlotnik 2021-12-14 21:54:46 UTC
Thanks for reporting this. This was indeed caused by https://github.com/openshift/machine-config-operator/pull/2795.

What's happening is:
- The MachineConfig specifies a file which is compressed. The MCD applies the config to disk.
- While it applies the config to disk, the MCD decompresses the file before writing it.
- The Config Drift Monitor attempts to compare the decompressed on-disk file contents to the compressed contents specified in the MachineConfig.
- Because the decompressed contents do not match the compressed contents, the Config Drift Monitor marks the node Degraded.

I've implemented a fix: https://github.com/openshift/machine-config-operator/pull/2874

Comment 7 errata-xmlrpc 2022-03-10 16:33:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.