Bug 1766346
| Summary: | kargs not applied to new nodes created by machine-api | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Erica von Buelow <evb> |
| Component: | Machine Config Operator | Assignee: | Sinny Kumari <skumari> |
| Status: | CLOSED ERRATA | QA Contact: | Antonio Murdaca <amurdaca> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.2.z | CC: | amurdaca, cnegus, kgarriso, skumari, smilner, walters |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | 4.5.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-07-13 17:12:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Erica von Buelow
2019-10-28 19:49:29 UTC
@erica was this covered by the epic? not sure if it needs to be updated. lmk! (In reply to Kirsten Garrison from comment #1) The epic addressed it for new 4.3+ installed clusters that have day-1 kargs support now. For clusters with installed in 4.1 or 4.2 (even those updated to 4.3 after install) that need to use day-2 kargs configuration even for new nodes, this is an issue. My observation while looking at this bug in detail: Affected OCP release version: Only 4.2 based cluster The way Karg support works: - In OCP 4.2, we added support in MCO to update KernelArguments(kargs) as Day 2 operation - In 4.2, during the firstboot service machine-config-daemon-host.service calls `machine-config-daemon pivot` from the machine-config-daemon binary available on RHCOS host. pivot calls rpm-ostree to rebase host with latest machine-os-content and reboots the host. It is not aware of processing kargs. While, in 4.3 we also have another service machine-config-daemon-firstboot.service which runs `machine-config-daemon firstboot-complete-machineconfig` that processes kernelArgs during firstboot. As of now since we can't update bootimage, we can't add capability to process MachineConfig during firstboot. Why this bug is happening? Consider we have an existing OCP 4.2 cluster and we applied a MachineConfig to add kargs, for example: the one mentioned in comment0 (custom-worker-00) . MCO geneartes a new rendered config, creates file /etc/kubernetes/tmp.txt and adds karg 'evb' on all the worker nodes. Now we add a new worker node (w1) into cluster using machine-api, worker w1 gets created containing all files/units defined from newly rendered config and service machine-config-daemon-firstboot.service runs `machine-config-daemon pivot`, w1 rebases to latest machine-os-content and reboots. kargs available in rendered config gets ignored as MCO doesn't know how to read it during firstboot. Newly created w1 worker node joins the cluster. What happens when we create another MachineConfig adding another karg? Suppose we create a new MachineConfig for worker node with kernelArgument: 'foo'. MCO generates a new rendered config appending 'foo' in kernelArguments, all worker nodes will be appended with 'foo' in kargs. All worker nodes except w1 will have 'evb' and 'foo' kargs while w1 will only have 'foo'. This is because of the way MCO checks for what kargs should be applied (see https://github.com/openshift/machine-config-operator/blob/db561314c7afae1d77c16cfdb95f0f0ce6b8977d/pkg/daemon/update.go#L582) . We look for diff into kargs available in oldconfig and newconfig. Since, oldconfig already has 'evb', it doesn't know that some of the worker node like w1 doesn't have 'evb' karg already applied. The only difference it sees between oldconfig and newconfig for karg is 'foo'. What happens when we try to remove MachineConfig custom-worker-00? Let's remove custom-worker-00 MachineConfig to fix the problem. MCO will compare oldconfig and newconfig and will try to delete karg 'evb'. All the worker node except w1 will delete 'evb' from kargs list. For w1, when it will try to delete 'evb' using `rpm-ostree kargs --delet='evb'` it fails because has never been applied. This will lead w1 worker node to degrade. Solution: Solving this issue is tricky because we will need to know what all kargs is already been applied on the RHCOS node. Some of the kargs is already been set during install time which we don't know. We have an open issue in ostree to identify the base kargs https://github.com/ostreedev/ostree/issues/479 Since this issue happens only on OCP 4.2 based cluster, perhaps we can document in the MCO doc with this known issue with workaround. There are two possible workaround to avoid this situation while adding a RHCOS node: 1. First delete all MachineConfig which applies any Kargs and then add node and then apply the deleted MachineConifgs 2. Once a machine is scaled-up, user checks the latest rendered config applied, fetches kargs list and then rsh into the created node and applies the kargs manually https://github.com/openshift/enhancements/pull/201 is intended to fix this by ensuring that "aleph version 4.2"¹ clusters have their bootimages upgraded. This would allow us to drop the complicated technical debt you're talking about that we carry in the MCO and ensure kargs just work the same as 4.3+. Now... https://github.com/openshift/machine-config-operator/pull/926 was intending to address one of the huge problems that occurs if this *does* happen on an "aleph 4.2" cluster today. You're going down the path of trying to make this work on 4.2...might be viable but the complexity is high. It seems easier to ask people to upgrade to 4.3 in place, and manually do a bootimage update once that's done. I can write up the manual steps in the enhancement if that'd help. ¹ By this I mean, clusters which were originally installed as 4.2. Why "aleph"? See https://github.com/coreos/coreos-assembler/pull/768 Thanks Colin for the input! (In reply to Colin Walters from comment #7) > https://github.com/openshift/enhancements/pull/201 is intended to fix this > by ensuring that "aleph version 4.2"¹ clusters have their bootimages > upgraded. +1, Can't wait to see bootimage update being implemented. > It seems easier to ask people to upgrade to 4.3 in place, and manually do a bootimage update once that's done. Can we already do this manually today or it will be possible once enhancement https://github.com/openshift/enhancements/pull/201 gets implemented? > I can write up the manual steps in the enhancement if that'd help. Yes, that will be great. I'm thinking this BZ isn't something beneficial to keep it open forever - we do have a card https://issues.redhat.com/browse/GRPA-1739 which we can use to track this and can refer to that whenever this arises - I'm going to close this as the fix is something that will come as a feature (bootimage updates likely). Let me know if anybody feels otherwise but this BZ is being pushed at every release and it's somehow 6 months old. (Closing Deferred also as I think it's the best status for this) Reopening this bug to document the fix by manually updating bootimage https://github.com/openshift/machine-config-operator/pull/1748 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409 |