Bug 1772490 - Creating a custom pool results in controller error "failed to read dir /etc/mcc/templates/custom-pool"
Summary: Creating a custom pool results in controller error "failed to read dir /etc/m...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.1.z
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 4.1.z
Assignee: Kirsten Garrison
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On: 1772680
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-14 13:33 UTC by Jason Kincl
Modified: 2020-02-24 16:45 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-30 16:16:12 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Jason Kincl 2019-11-14 13:33:43 UTC
Description of problem:

Creating a custom pool creates errors in the controller which is looking for a set of templates that do not exist.

How reproducible:

Following https://github.com/openshift/machine-config-operator/blob/master/docs/custom-pools.md


Steps to Reproduce:
1. create custom pool
2. check logs for MCC

Actual results:

results in MCC logs:

I1114 13:26:25.344351       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:25.374172       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:25.392505       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:25.412723       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:25.452989       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:25.533219       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:25.718971       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:26.063772       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:26.730793       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:28.011163       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:30.571547       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:35.718176       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:26:45.984266       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:27:06.491769       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:27:47.452091       1 container_runtime_config_controller.go:336] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
E1114 13:29:09.372392       1 container_runtime_config_controller.go:341] could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory
I1114 13:29:09.372419       1 container_runtime_config_controller.go:342] Dropping image config "openshift-config" out of the queue: could not Create/Update MachineConfig: could not generate origin ContainerRuntime Configs: generateMachineConfigsforRole failed with error failed to read dir "/etc/mcc/templates/custom-pool": open /etc/mcc/templates/custom-pool: no such file or directory


Expected results:

MCC shouldn't error on this, there will never be templates for custom pools in the MCC image.

Comment 1 Sinny Kumari 2019-11-21 14:54:56 UTC
Similar issue was seen upstream https://github.com/openshift/machine-config-operator/issues/429#issuecomment-524092635, but looks like this shouldn't be causing issue with new pool to be created.

Do you see an issue with custom pool being created?
What is the output of 'oc get machineconfigpool' , 'oc get machineconfig' and 'oc get nodes' ?


Maybe unrelated but I think backporting fix on handling multiple roles from https://github.com/openshift/machine-config-operator/pull/828 in 4.1.z would be useful.

Comment 2 Jason Kincl 2019-11-21 17:22:56 UTC
(In reply to Sinny Kumari from comment #1)
> Similar issue was seen upstream
> https://github.com/openshift/machine-config-operator/issues/429#issuecomment-
> 524092635, but looks like this shouldn't be causing issue with new pool to
> be created.
> 
> Do you see an issue with custom pool being created?
> What is the output of 'oc get machineconfigpool' , 'oc get machineconfig'
> and 'oc get nodes' ?
> 
> 
> Maybe unrelated but I think backporting fix on handling multiple roles from
> https://github.com/openshift/machine-config-operator/pull/828 in 4.1.z would
> be useful.

The custom pool did get created but the reconcile queue of the MCO was flooded with retry failures trying to sync the templates which seemed to considerably slow down the MCO ability to reconcile nodes to a new desired configuration for the existing worker pool.

Comment 4 Kirsten Garrison 2019-12-18 21:16:08 UTC
@Leonid see: https://bugzilla.redhat.com/show_bug.cgi?id=1772680

for the 4.2 fix

Comment 5 Kirsten Garrison 2019-12-18 21:17:37 UTC
For 4.1 looks like this needs a backport of https://bugzilla.redhat.com/show_bug.cgi?id=1772680, right @Antonio?

Comment 6 Antonio Murdaca 2020-01-30 16:16:12 UTC
The fix for this is https://github.com/openshift/machine-config-operator/pull/963 which went into OCP >= 4.2

I'm not sure at this point how strong I feel about backporting this as easy at it looks like.

I'm leaning towards documenting these warnings aren't serious but just spammy. If anybody else feels otherwise, please reopen and we'll attemp a backport.


Note You need to log in before you can comment on or make changes to this bug.