Bug 1718726

Ryan, can you take a look?  Not sure if this is a KubeletConfig controller issue or an MC issue.

From what I can tell after looking at this on Friday, it looks like there was an RHCOS upgrade.

Next time this issue comes up we need the MCC logs and `oc get kubeletconfigs` output.

Offhand I think the most likely thing here is a race in the kubelet/render controller which caused the rendered MC to transiently drop the kubelet config.

I'd look at:
`oc -n openshift-machine-config-operator logs deploy/machine-config-controller`
to check.
Or you can look at `oc describe machineconfig/rendered-blah` to see what MCs it used as input.  If it's missing the kubeletconfig one, there's your smoking gun.

Did not happen after install of 4.2.0-0.nightly-2019-07-26-234318 and upgrade to 4.2.0-0.nightly-2019-07-28-222114.

Reproduced upgrading from 4.2.0-0.okd-2019-07-28-045558 to 4.2.0-0.okd-2019-07-29-205431 (which crosses an OS boundary).

The node is labeled correctly as a worker, but has a capacity of only 250 pods:

$ oc describe node ip-10-0-152-211.us-east-2.compute.internal
Name:               ip-10-0-152-211.us-east-2.compute.internal
Roles:              worker
...
Capacity:
 attachable-volumes-aws-ebs:  39
 cpu:                         2
 hugepages-1Gi:               0
 hugepages-2Mi:               0
 memory:                      8162900Ki
 pods:                        250


The machineconfigpool is labeled correctly:

$ oc describe machineconfigpool worker
Name:         worker
Namespace:    
Labels:       custom-kubelet=large-pods
              machineconfiguration.openshift.io/mco-built-in=
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1
Kind:         MachineConfigPool


The KubeletConfig is correct:

$ oc describe kubeletconfig set-max-pods
Name:         set-max-pods
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1
Kind:         KubeletConfig
Metadata:
  Creation Timestamp:  2019-07-29T21:31:53Z
  Finalizers:
    99-worker-cff08ded-b23d-11e9-804f-02ab32c68f4a-kubelet
    99-worker-cff08ded-b23d-11e9-804f-02ab32c68f4a-kubelet
    99-worker-cff08ded-b23d-11e9-804f-02ab32c68f4a-kubelet
  Generation:        1
  Resource Version:  56071
  Self Link:         /apis/machineconfiguration.openshift.io/v1/kubeletconfigs/set-max-pods
  UID:               48a1237e-b248-11e9-a6e4-0a033c674f6c
Spec:
  Kubelet Config:
    Kube API Burst:  30
    Kube APIQPS:     15
    Max Pods:        750
  Machine Config Pool Selector:
    Match Labels:
      Custom - Kubelet:  large-pods
Status:
  Conditions:
    Last Transition Time:  2019-07-29T21:31:53Z
    Message:               Success
    Status:                True
    Type:                  Success
    Last Transition Time:  2019-07-29T22:17:26Z
    Message:               Success
    Status:                True
    Type:                  Success
    Last Transition Time:  2019-07-29T22:23:52Z
    Message:               Success
    Status:                True
    Type:                  Success
Events:                    <none>

Created attachment 1594427 [details]
Upgrade trace to 4.2.0-0.okd-2019-07-29-205431

Robert, can you grab the machine-config-controller pod logs as Colin suggested in https://bugzilla.redhat.com/show_bug.cgi?id=1718726#c4 - that could tell us if the render controller ran w/o the kubelet config (even if it should at some point generate a new rendered with that and start a roll out afaict)

The kubelet config MC seems to be there tho


      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-worker-kubelet

The MCC logs would tell us more for sure

From Robert's output tho, there seem to be rendered-MC w/o the kubelet config indeed.

Created attachment 1594428 [details]
Node log from worker node corresponding to upgrade trace.

Note that the transition from 750 to 250 pods occurred across the reboot around 22:36 UTC.

Created attachment 1594429 [details]
journalctl -u kubelet

Created attachment 1594430 [details]
MCO log

Created attachment 1594431 [details]
machineconfig for the worker

```
1:40:07 [~] oc get machineconfigs
NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
00-master                                                   093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             3h24m
00-worker                                                   093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             3h24m
01-master-container-runtime                                 093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             3h24m
01-master-kubelet                                           093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             3h24m
01-worker-container-runtime                                 093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             3h24m
01-worker-kubelet                                           093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             3h24m
99-master-cfef54fd-b23d-11e9-804f-02ab32c68f4a-registries   093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             3h24m
99-master-ssh                                                                                          2.2.0             3h25m
99-worker-cff08ded-b23d-11e9-804f-02ab32c68f4a-registries   093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             3h24m
99-worker-ssh                                                                                          2.2.0             3h24m
rendered-master-28f01d3937cd8a942520c72df62f36ea            093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             3h24m
rendered-master-2ff3158a4ce7ed781e4a0cfda3f429c3            093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             84m
rendered-worker-37078352b9543dc667ea867ea2a3a65c            093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             129m
rendered-worker-a046d622eb6041e65084db6dcbeb9ae2            093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             3h24m
rendered-worker-cf3ba9c821b86b58407223b59e539b48            093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             73m
rendered-worker-e456022d72042350c47c3828e4797f2b            093e96ef4cdbd15ecda18323dadf4d552fcfd327   2.2.0             84m
```

There's no `-kubelet` postfixed MC generated by the kubelet config controller it seems (why).

The worker MCP is targeting the resulting MC which doesn't contain the expected kubelet config MC.

it looks like from the other rendered-worker-MC that the maxPod setting was indeed there but somehow the MC for it got lost

Fixed via https://github.com/openshift/machine-config-operator/pull/1022.

To test:

1) Install an image with underlying OS _older_ than the OS in the current release.

2) Check the OS version of this release:

$ oc image info -o json $(oc adm release info --image-for=machine-os-content) |jq '.config.config.Labels.version'
"42.80.20190731.2"

3) Increase the max pods (for example) of some or all nodes:

$ oc label machineconfigpool worker custom-kubelet=large-pods
$ oc create -f - <<'EOF'
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: set-max-pods
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom-kubelet: large-pods
  kubeletConfig:
    maxPods: 750
    KubeAPIBurst: 30
    KubeAPIQPS: 15
EOF

4) Wait for the number of pods on the relevant nodes to increase, and for the pods to be ready:

$ oc describe node |grep pods:
 pods:                        750
 pods:                        750
 pods:                        250
 pods:                        250
 pods:                        750
 pods:                        750
 pods:                        250
 pods:                        250
 pods:                        750
 pods:                        750
 pods:                        250
 pods:                        250
$ oc get nodes
NAME                                         STATUS   ROLES    AGE    VERSION
ip-10-0-138-155.us-east-2.compute.internal   Ready    worker   160m   v1.14.0+ff340d215
ip-10-0-140-107.us-east-2.compute.internal   Ready    master   165m   v1.14.0+ff340d215
ip-10-0-144-221.us-east-2.compute.internal   Ready    worker   160m   v1.14.0+ff340d215
ip-10-0-147-172.us-east-2.compute.internal   Ready    master   165m   v1.14.0+ff340d215
ip-10-0-163-193.us-east-2.compute.internal   Ready    worker   160m   v1.14.0+ff340d215
ip-10-0-174-195.us-east-2.compute.internal   Ready    master   165m   v1.14.0+ff340d215

5) Perform the upgrade.

6) Wait for the upgrade to complete.  Verify that the OS is different from the old one:

$ oc image info -o json $(oc adm release info --image-for=machine-os-content) |jq '.config.config.Labels.version'
"42.80.20190801.0"

*If the image you upgraded to has the same OS build as the previous image (hence the node OS will not be upgraded), it will not correctly test the bug.*

7) Repeat step (3) -- if the bug is fixed, you'll still see the adjusted max pods.

I'm still working on this, looking for "OS _older_ than the OS in the current release", and it can get built successfully would take some time

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922