Bug 1817458 - Node goes to degraded status when machine-config-daemon moves a file across filesystems
Summary: Node goes to degraded status when machine-config-daemon moves a file across f...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.3.z
Assignee: Antonio Murdaca
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On: 1817455
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-26 11:38 UTC by Antonio Murdaca
Modified: 2020-05-20 13:48 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1817455
Environment:
Last Closed: 2020-05-20 13:47:53 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift machine-config-operator pull 1686 None closed Bug 1817458: [release-4.3] fix wrongful backup of files not originally on the system 2020-05-12 18:19:17 UTC
Red Hat Product Errata RHBA-2020:2129 None None None 2020-05-20 13:48:08 UTC

Comment 1 W. Trevor King 2020-04-07 20:28:36 UTC
Bugzilla bot won't set this POST until the 4.4 bug 1817455 is verified, so I'm manually linking the PR to make it easier to find.

Comment 4 Michael Nguyen 2020-05-14 16:10:22 UTC
Verified on 4.3.0-0.nightly-2020-05-12-224733

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.0-0.nightly-2020-05-12-224733   True        False         24m     Cluster version is 4.3.0-0.nightly-2020-05-12-224733
$ oc get mc | grep rendered-worker
rendered-worker-7aa2e43cfd629c043d31415f6043f436            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             64m
$ cat << EOF > file.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: test-1
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
      - contents:
          source: data:;base64,dGVzdCBzdHJpbmcK
        filesystem: root
        mode: 0644
        path: /etc/test
EOF
$ oc apply -f file.yaml 
machineconfig.machineconfiguration.openshift.io/test-1 created
$ oc get mc
NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
00-master                                                   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             65m
00-worker                                                   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             65m
01-master-container-runtime                                 860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             65m
01-master-kubelet                                           860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             65m
01-worker-container-runtime                                 860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             65m
01-worker-kubelet                                           860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             65m
99-master-b9bb9be5-8dfc-4093-9f7f-0b0b6e94ccb3-registries   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             65m
99-master-ssh                                                                                          2.2.0             66m
99-worker-b004156f-0ad3-4210-b9c9-6365a15cdcf6-registries   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             65m
99-worker-ssh                                                                                          2.2.0             66m
rendered-master-7e79f6f1b7d307d77046ea3c64e7e14b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             65m
rendered-worker-7aa2e43cfd629c043d31415f6043f436            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             65m
rendered-worker-f16e588bda90b80462dda7bbd8c43a7b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             5s
test-1                                                                                                 2.2.0             10s
$ echo "rendered-worker-f16e588bda90b80462dda7bbd8c43a7b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             5s"
rendered-worker-f16e588bda90b80462dda7bbd8c43a7b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             5s
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT
worker   rendered-worker-7aa2e43cfd629c043d31415f6043f436   False     True       False      3              0                   0                     0
$ watch oc get nodes
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT
worker   rendered-worker-f16e588bda90b80462dda7bbd8c43a7b   True      False      False      3              3                   3                     0
$ for i in $(oc get node -l node-role.kubernetes.io/worker -o name); do oc debug $i -- cat /host/etc/test; done
Starting pod/ip-10-0-132-187us-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
test string

Removing debug pod ...
Starting pod/ip-10-0-143-126us-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
test string

Removing debug pod ...
Starting pod/ip-10-0-145-238us-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
test string

Removing debug pod ...
$ cat << EOF > file2.yaml 
> apiVersion: machineconfiguration.openshift.io/v1
> kind: MachineConfig
> metadata:
>   labels:
>     machineconfiguration.openshift.io/role: worker
>   name: test-2
> spec:
>   config:
>     ignition:
>       version: 2.2.0
>     storage:
>       files:
>       - contents:
>           source: data:;base64,dGVzdCBzdHJpbmcgMgo=
>         filesystem: root
>         mode: 0644
>         path: /etc/test
> EOF
$ oc apply -f file2.yaml 
machineconfig.machineconfiguration.openshift.io/test-2 created
$ oc get mc
NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
00-master                                                   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             72m
00-worker                                                   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             72m
01-master-container-runtime                                 860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             72m
01-master-kubelet                                           860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             72m
01-worker-container-runtime                                 860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             72m
01-worker-kubelet                                           860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             72m
99-master-b9bb9be5-8dfc-4093-9f7f-0b0b6e94ccb3-registries   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             72m
99-master-ssh                                                                                          2.2.0             73m
99-worker-b004156f-0ad3-4210-b9c9-6365a15cdcf6-registries   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             72m
99-worker-ssh                                                                                          2.2.0             73m
rendered-master-7e79f6f1b7d307d77046ea3c64e7e14b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             72m
rendered-worker-7aa2e43cfd629c043d31415f6043f436            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             72m
rendered-worker-c05f53a919c7f98797187a6f20caf0a0            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             5s
rendered-worker-f16e588bda90b80462dda7bbd8c43a7b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             6m49s
test-1                                                                                                 2.2.0             6m54s
test-2                                                                                                 2.2.0             10s
$ echo "rendered-worker-c05f53a919c7f98797187a6f20caf0a0            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             5s"
rendered-worker-c05f53a919c7f98797187a6f20caf0a0            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             5s
$ watch oc get nodes
$  oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT
worker   rendered-worker-c05f53a919c7f98797187a6f20caf0a0   True      False      False      3              3                   3                     0
$ for i in $(oc get node -l node-role.kubernetes.io/worker -o name); do oc debug $i -- cat /host/etc/test; done
Starting pod/ip-10-0-132-187us-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
test string 2

Removing debug pod ...
Starting pod/ip-10-0-143-126us-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
test string 2

Removing debug pod ...
Starting pod/ip-10-0-145-238us-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
test string 2

Removing debug pod ...
$ oc delete mc/test-2
machineconfig.machineconfiguration.openshift.io "test-2" deleted
$ oc get mc
NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
00-master                                                   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
00-worker                                                   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
01-master-container-runtime                                 860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
01-master-kubelet                                           860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
01-worker-container-runtime                                 860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
01-worker-kubelet                                           860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
99-master-b9bb9be5-8dfc-4093-9f7f-0b0b6e94ccb3-registries   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
99-master-ssh                                                                                          2.2.0             79m
99-worker-b004156f-0ad3-4210-b9c9-6365a15cdcf6-registries   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
99-worker-ssh                                                                                          2.2.0             79m
rendered-master-7e79f6f1b7d307d77046ea3c64e7e14b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
rendered-worker-7aa2e43cfd629c043d31415f6043f436            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
rendered-worker-c05f53a919c7f98797187a6f20caf0a0            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             6m15s
rendered-worker-f16e588bda90b80462dda7bbd8c43a7b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             12m
test-1                                                                                                 2.2.0             13m
$ oc get mc
NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
00-master                                                   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
00-worker                                                   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
01-master-container-runtime                                 860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
01-master-kubelet                                           860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
01-worker-container-runtime                                 860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
01-worker-kubelet                                           860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
99-master-b9bb9be5-8dfc-4093-9f7f-0b0b6e94ccb3-registries   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
99-master-ssh                                                                                          2.2.0             79m
99-worker-b004156f-0ad3-4210-b9c9-6365a15cdcf6-registries   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
99-worker-ssh                                                                                          2.2.0             79m
rendered-master-7e79f6f1b7d307d77046ea3c64e7e14b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
rendered-worker-7aa2e43cfd629c043d31415f6043f436            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             78m
rendered-worker-c05f53a919c7f98797187a6f20caf0a0            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             6m26s
rendered-worker-f16e588bda90b80462dda7bbd8c43a7b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             13m
test-1                                                                                                 2.2.0             13m
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT
worker   rendered-worker-c05f53a919c7f98797187a6f20caf0a0   False     True       False      3              0                   0                     0
$ watch oc get node
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT
worker   rendered-worker-f16e588bda90b80462dda7bbd8c43a7b   True      False      False      3              3                   3                     0
$ for i in $(oc get node -l node-role.kubernetes.io/worker -o name); do oc debug $i -- cat /host/etc/test; done
Starting pod/ip-10-0-132-187us-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
test string

Removing debug pod ...
Starting pod/ip-10-0-143-126us-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
test string

Removing debug pod ...
Starting pod/ip-10-0-145-238us-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
test string

Removing debug pod ...
$ oc get mc
NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
00-master                                                   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             90m
00-worker                                                   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             90m
01-master-container-runtime                                 860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             90m
01-master-kubelet                                           860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             90m
01-worker-container-runtime                                 860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             90m
01-worker-kubelet                                           860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             90m
99-master-b9bb9be5-8dfc-4093-9f7f-0b0b6e94ccb3-registries   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             90m
99-master-ssh                                                                                          2.2.0             91m
99-worker-b004156f-0ad3-4210-b9c9-6365a15cdcf6-registries   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             90m
99-worker-ssh                                                                                          2.2.0             91m
rendered-master-7e79f6f1b7d307d77046ea3c64e7e14b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             90m
rendered-worker-7aa2e43cfd629c043d31415f6043f436            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             90m
rendered-worker-c05f53a919c7f98797187a6f20caf0a0            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             18m
rendered-worker-f16e588bda90b80462dda7bbd8c43a7b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             25m
test-1                                                                                                 2.2.0             25m
$ oc delete mc/test-1
machineconfig.machineconfiguration.openshift.io "test-1" deleted
$ oc get mc
NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
00-master                                                   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             91m
00-worker                                                   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             91m
01-master-container-runtime                                 860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             91m
01-master-kubelet                                           860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             91m
01-worker-container-runtime                                 860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             91m
01-worker-kubelet                                           860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             91m
99-master-b9bb9be5-8dfc-4093-9f7f-0b0b6e94ccb3-registries   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             91m
99-master-ssh                                                                                          2.2.0             92m
99-worker-b004156f-0ad3-4210-b9c9-6365a15cdcf6-registries   860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             91m
99-worker-ssh                                                                                          2.2.0             92m
rendered-master-7e79f6f1b7d307d77046ea3c64e7e14b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             91m
rendered-worker-7aa2e43cfd629c043d31415f6043f436            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             91m
rendered-worker-c05f53a919c7f98797187a6f20caf0a0            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             19m
rendered-worker-f16e588bda90b80462dda7bbd8c43a7b            860382c905f4358418c6513a9ab55fdd6dcc4f2d   2.2.0             25m
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT
worker   rendered-worker-f16e588bda90b80462dda7bbd8c43a7b   False     True       False      3              0                   0                     0
$ watch oc get nodes
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT
worker   rendered-worker-7aa2e43cfd629c043d31415f6043f436   True      False      False      3              3                   3                     0
$ for i in $(oc get node -l node-role.kubernetes.io/worker -o name); do oc debug $i -- cat /host/etc/test; done
Starting pod/ip-10-0-132-187us-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
cat: /host/etc/test: No such file or directory

Removing debug pod ...
Starting pod/ip-10-0-143-126us-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
cat: /host/etc/test: No such file or directory

Removing debug pod ...
Starting pod/ip-10-0-145-238us-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
cat: /host/etc/test: No such file or directory

Removing debug pod ...

Comment 6 errata-xmlrpc 2020-05-20 13:47:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2129


Note You need to log in before you can comment on or make changes to this bug.