Bug 1897575

Summary: Scaling new nodes to during configuration change uses old configuration
Product: OpenShift Container Platform Reporter: OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component: Machine Config OperatorAssignee: Antonio Murdaca <amurdaca>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: high Docs Contact:
Priority: high    
Version: 4.5CC: aaleman, erich, mkrejci, walters, wking
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-12-14 13:50:52 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1873288    
Bug Blocks:    

Comment 3 Michael Nguyen 2020-11-30 22:23:57 UTC
Verified on 4.6.0-0.nightly-2020-11-28-204928
$ cat file.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: test-file
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf;base64,c2VydmVyIGZvby5leGFtcGxlLm5ldCBtYXhkZWxheSAwLjQgb2ZmbGluZQpzZXJ2ZXIgYmFyLmV4YW1wbGUubmV0IG1heGRlbGF5IDAuNCBvZmZsaW5lCnNlcnZlciBiYXouZXhhbXBsZS5uZXQgbWF4ZGVsYXkgMC40IG9mZmxpbmUK
        filesystem: root
        mode: 0644
        path: /etc/test
$ oc create -f file.yaml 
machineconfig.machineconfiguration.openshift.io/test-file created
$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          5a9e6b4eedaf72ecc2534355173843e011f19765   3.1.0             3h34m
00-worker                                          5a9e6b4eedaf72ecc2534355173843e011f19765   3.1.0             3h34m
01-master-container-runtime                        5a9e6b4eedaf72ecc2534355173843e011f19765   3.1.0             3h34m
01-master-kubelet                                  5a9e6b4eedaf72ecc2534355173843e011f19765   3.1.0             3h34m
01-worker-container-runtime                        5a9e6b4eedaf72ecc2534355173843e011f19765   3.1.0             3h34m
01-worker-kubelet                                  5a9e6b4eedaf72ecc2534355173843e011f19765   3.1.0             3h34m
99-master-generated-registries                     5a9e6b4eedaf72ecc2534355173843e011f19765   3.1.0             3h34m
99-master-ssh                                                                                 3.1.0             3h44m
99-worker-generated-registries                     5a9e6b4eedaf72ecc2534355173843e011f19765   3.1.0             3h34m
99-worker-ssh                                                                                 3.1.0             3h44m
rendered-master-68cfb783ca9fea66121a140662b1eecd   5a9e6b4eedaf72ecc2534355173843e011f19765   3.1.0             3h34m
rendered-worker-e2825f140e84a200a54950101efa3790   5a9e6b4eedaf72ecc2534355173843e011f19765   3.1.0             5s
rendered-worker-eb2b3c116a90d6d1cc32431a7d060211   5a9e6b4eedaf72ecc2534355173843e011f19765   3.1.0             3h34m
test-file                                                                                     3.1.0             5s
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-eb2b3c116a90d6d1cc32431a7d060211   False     True       False      3              0                   0                     0                      3h35m
$ oc -n openshift-machine-api get machinesets
NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
mnguyen46-mqt96-worker-us-west-2a   1         1         1       1           3h44m
mnguyen46-mqt96-worker-us-west-2b   1         1         1       1           3h44m
mnguyen46-mqt96-worker-us-west-2c   1         1         1       1           3h44m
mnguyen46-mqt96-worker-us-west-2d   0         0                             3h44m
$ watch oc get mcp/worker
$ oc get nodes
NAME                                         STATUS                     ROLES    AGE     VERSION
ip-10-0-137-98.us-west-2.compute.internal    Ready                      master   3h37m   v1.19.0+6d3423a
ip-10-0-158-198.us-west-2.compute.internal   Ready                      worker   3h27m   v1.19.0+6d3423a
ip-10-0-166-100.us-west-2.compute.internal   Ready                      worker   3h28m   v1.19.0+6d3423a
ip-10-0-183-163.us-west-2.compute.internal   Ready                      master   3h36m   v1.19.0+6d3423a
ip-10-0-207-100.us-west-2.compute.internal   Ready                      master   3h37m   v1.19.0+6d3423a
ip-10-0-219-169.us-west-2.compute.internal   Ready,SchedulingDisabled   worker   3h29m   v1.19.0+6d3423a
$ watch oc get mcp/worker
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-eb2b3c116a90d6d1cc32431a7d060211   False     True       False      3              1                   1                     0                      3h38m
$  oc -n openshift-machine-api scale --replicas=2 machinesets/mnguyen46-mqt96-worker-us-west-2a
machineset.machine.openshift.io/mnguyen46-mqt96-worker-us-west-2a scaled
$ oc -n openshift-machine-api get machinesets
NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
mnguyen46-mqt96-worker-us-west-2a   2         2         1       1           3h47m
mnguyen46-mqt96-worker-us-west-2b   1         1         1       1           3h47m
mnguyen46-mqt96-worker-us-west-2c   1         1         1       1           3h47m
mnguyen46-mqt96-worker-us-west-2d   0         0                             3h47m
$ oc get nodes
NAME                                         STATUS                     ROLES    AGE     VERSION
ip-10-0-137-98.us-west-2.compute.internal    Ready                      master   3h39m   v1.19.0+6d3423a
ip-10-0-158-198.us-west-2.compute.internal   Ready                      worker   3h28m   v1.19.0+6d3423a
ip-10-0-166-100.us-west-2.compute.internal   Ready,SchedulingDisabled   worker   3h30m   v1.19.0+6d3423a
ip-10-0-183-163.us-west-2.compute.internal   Ready                      master   3h38m   v1.19.0+6d3423a
ip-10-0-207-100.us-west-2.compute.internal   Ready                      master   3h39m   v1.19.0+6d3423a
ip-10-0-219-169.us-west-2.compute.internal   Ready                      worker   3h31m   v1.19.0+6d3423a
$ watch oc -n openshift-machine-api get machinesets
$  oc -n openshift-machine-api get machinesets
NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
mnguyen46-mqt96-worker-us-west-2a   2         2         2       2           3h52m
mnguyen46-mqt96-worker-us-west-2b   1         1         1       1           3h52m
mnguyen46-mqt96-worker-us-west-2c   1         1         1       1           3h52m
mnguyen46-mqt96-worker-us-west-2d   0         0                             3h52m
$ oc get nodes
NAME                                         STATUS                     ROLES    AGE     VERSION
ip-10-0-128-247.us-west-2.compute.internal   Ready                      worker   81s     v1.19.0+6d3423a
ip-10-0-137-98.us-west-2.compute.internal    Ready                      master   3h45m   v1.19.0+6d3423a
ip-10-0-158-198.us-west-2.compute.internal   Ready,SchedulingDisabled   worker   3h34m   v1.19.0+6d3423a
ip-10-0-166-100.us-west-2.compute.internal   Ready                      worker   3h35m   v1.19.0+6d3423a
ip-10-0-183-163.us-west-2.compute.internal   Ready                      master   3h44m   v1.19.0+6d3423a
ip-10-0-207-100.us-west-2.compute.internal   Ready                      master   3h45m   v1.19.0+6d3423a
ip-10-0-219-169.us-west-2.compute.internal   Ready                      worker   3h36m   v1.19.0+6d3423a
$ oc debug node/ip-10-0-128-247.us-west-2.compute.internal
Starting pod/ip-10-0-128-247us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# journalctl --list-boots
-1 94a7faccd8194371bf3ab16398008c00 Mon 2020-11-30 22:15:10 UTC—Mon 2020-11-30 22:18:23 UTC
 0 41f8682d5a104ca4baa4f1e17e30f5d2 Mon 2020-11-30 22:18:43 UTC—Mon 2020-11-30 22:21:35 UTC
sh-4.4# cat /etc/test 
server foo.example.net maxdelay 0.4 offline
server bar.example.net maxdelay 0.4 offline
server baz.example.net maxdelay 0.4 offline
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...

Comment 5 errata-xmlrpc 2020-12-14 13:50:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.6.8 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5259