Bug 1772490

Summary:	Creating a custom pool results in controller error "failed to read dir /etc/mcc/templates/custom-pool"
Product:	OpenShift Container Platform	Reporter:	Jason Kincl <kincljc>
Component:	Machine Config Operator	Assignee:	Kirsten Garrison <kgarriso>
Status:	CLOSED WONTFIX	QA Contact:	Michael Nguyen <mnguyen>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.1.z	CC:	amurdaca, jtudelag, kgarriso, ltitov, skumari
Target Milestone:	---
Target Release:	4.1.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-01-30 16:16:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1772680
Bug Blocks:

Description Jason Kincl 2019-11-14 13:33:43 UTC

Description of problem:

Creating a custom pool creates errors in the controller which is looking for a set of templates that do not exist.

How reproducible:

Following https://github.com/openshift/machine-config-operator/blob/master/docs/custom-pools.md


Steps to Reproduce:
1. create custom pool
2. check logs for MCC

Actual results:

results in MCC logs:

I1114 13:26:25.344351       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:25.374172       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:25.392505       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:25.412723       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:25.452989       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:25.533219       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:25.718971       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:26.063772       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:26.730793       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:28.011163       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:30.571547       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:35.718176       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:45.984266       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:27:06.491769       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:27:47.452091       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
E1114 13:29:09.372392       1 container_runtime_config_controller.go:341] could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:29:09.372419       1 container_runtime_config_controller.go:342] Dropping image config "openshift-config" out of the queue: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory


Expected results:

MCC shouldn't error on this, there will never be templates for custom pools in the MCC image.

Comment 1 Sinny Kumari 2019-11-21 14:54:56 UTC

Similar issue was seen upstream https://github.com/openshift/machine-config-operator/issues/429#issuecomment-524092635, but looks like this shouldn't be causing issue with new pool to be created.

Do you see an issue with custom pool being created?
What is the output of 'oc get machineconfigpool' , 'oc get machineconfig' and 'oc get nodes' ?


Maybe unrelated but I think backporting fix on handling multiple roles from https://github.com/openshift/machine-config-operator/pull/828 in 4.1.z would be useful.

Comment 2 Jason Kincl 2019-11-21 17:22:56 UTC

(In reply to Sinny Kumari from comment #1)
> Similar issue was seen upstream
> https://github.com/openshift/machine-config-operator/issues/429#issuecomment-
> 524092635, but looks like this shouldn't be causing issue with new pool to
> be created.
> 
> Do you see an issue with custom pool being created?
> What is the output of 'oc get machineconfigpool' , 'oc get machineconfig'
> and 'oc get nodes' ?
> 
> 
> Maybe unrelated but I think backporting fix on handling multiple roles from
> https://github.com/openshift/machine-config-operator/pull/828 in 4.1.z would
> be useful.

The custom pool did get created but the reconcile queue of the MCO was flooded with retry failures trying to sync the templates which seemed to considerably slow down the MCO ability to reconcile nodes to a new desired configuration for the existing worker pool.

Comment 4 Kirsten Garrison 2019-12-18 21:16:08 UTC

@Leonid see: https://bugzilla.redhat.com/show_bug.cgi?id=1772680

for the 4.2 fix

Comment 5 Kirsten Garrison 2019-12-18 21:17:37 UTC

For 4.1 looks like this needs a backport of https://bugzilla.redhat.com/show_bug.cgi?id=1772680, right @Antonio?

Comment 6 Antonio Murdaca 2020-01-30 16:16:12 UTC

The fix for this is https://github.com/openshift/machine-config-operator/pull/963 which went into OCP >= 4.2

I'm not sure at this point how strong I feel about backporting this as easy at it looks like.

I'm leaning towards documenting these warnings aren't serious but just spammy. If anybody else feels otherwise, please reopen and we'll attemp a backport.