Bug 1797687

Summary:	Can't apply Machine Configs to Masters with roles different from "master" and machineconfigpools different from "master"
Product:	OpenShift Container Platform	Reporter:	Jose Kaz C R <jcasimir>
Component:	Machine Config Operator	Assignee:	Sinny Kumari <skumari>
Status:	CLOSED ERRATA	QA Contact:	Michael Nguyen <mnguyen>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.2.z	CC:	amurdaca, walters
Target Milestone:	---
Target Release:	4.5.0
Hardware:	x86_64
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	Known Issue
Doc Text:	Cause: Custom pool is not supported on master node. Consequence: Custom pool on a node having master role is not supported. `oc label node` will apply the new custom role to the target master node but MCO will not apply changes specific to the custom pool. Error can be seen in the Machine Config Controller pod logs. Workaround (if any): User is advised to not apply multiple roles on master. This behavior is to make sure that control plane nodes remain stable.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-07-13 17:13:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jose Kaz C R 2020-02-03 16:29:40 UTC

Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:

Create a MachineConfig for the Masters that require a different node role, e.g. master-0 and try to apply it to the masters.


Steps to Reproduce:
1. Create a Machine Config and Machine Config Pools for the master that are different from "master". Possible Use Case: Add a fixed IP to a new network card to one of the masters. E.g. MachineConfigPool: 01-Master-mcp, MachineConfig: 01-Master-mc

oc create -f 01-Master-mcp.yaml
oc create -f 01-Master-mc.yaml

2. Label the MachineConfigPool with the respective new label.

oc label machineconfigpool master 0-master=""  

3. Add the tag to the master node with a new role that belongs to the new Machine Config Pools. E.g. 0-master

oc label node <node_name> node-role.kubernetes.io/0-master=""


Actual results:
The configuration is never applied to the masters.


Expected results:
The node receives the configuration from the MachineConfig


Additional info:
One possible use case is to have network cards that have fixed IPs that are added post install to the masters, for example, when you are using a Storage Solution like Solidfire and the trident driver requires to reach it from the masters too.

Comment 1 Antonio Murdaca 2020-02-05 10:21:47 UTC

Did you follow https://github.com/openshift/machine-config-operator/blob/master/docs/custom-pools.md ?

Comment 2 Jose Kaz C R 2020-02-05 16:05:23 UTC

Yes, this does not work with the masters for some unknown reason. It works flawlessly with the workers, but not the masters.

Comment 3 Antonio Murdaca 2020-02-05 16:11:10 UTC

(In reply to Jose Kaz C R from comment #2)
> Yes, this does not work with the masters for some unknown reason. It works
> flawlessly with the workers, but not the masters.

uhm, we can take a look at it I guess and see what's happening - and asses if masters can be assigned a custom pool at all (I see the use case tho)

Comment 4 Sinny Kumari 2020-02-07 08:36:55 UTC

When a node has both master and custom role (say 0-master), MCC would be logging error message: 

E0207 08:02:20.931403       1 node_controller.go:415] error finding pool for node: node <node> has both master role and custom role 01-master


By looking at https://github.com/cgwalters/machine-config-operator/blob/b9d6c62de34d1fe01aa3b98a3559339c73c09d57/pkg/controller/node/node_controller.go#L301 it seems MCO currently doesn't support custom role being applied on master.

@Colin @Antonio Any idea if there is a reason behind not supporting custom pool for master node?

Comment 5 Colin Walters 2020-02-11 23:07:29 UTC

I think what you really want for this is machine-specific machineconfigs (which makes sense when written that way ;) )

This would be an RFE for the MCO, and actually the CNTO https://github.com/openshift/cluster-node-tuning-operator needs it too.

Comment 8 Michael Nguyen 2020-02-25 00:03:45 UTC

Verified documentation in github now has warnings about custom pools with master nodes.

Comment 10 errata-xmlrpc 2020-07-13 17:13:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409