Bug 1809018

Summary: [OCP 4.2] metadata.annotations: Too long: must have at most 262144 characters
Product: OpenShift Container Platform Reporter: Mario Abajo <mabajodu>
Component: Machine Config OperatorAssignee: Antonio Murdaca <amurdaca>
Status: CLOSED DUPLICATE QA Contact: Michael Nguyen <mnguyen>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.2.zCC: igreen
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-04 13:44:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mario Abajo 2020-03-02 09:42:32 UTC
Description of problem:
Machine-config-daemon refuses to run because of the following error:
E0301 12:58:27.535629 3165818 writer.go:127] Marking Degraded due to: machineconfiguration.openshift.io/desiredConfig annotation not found in &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name: .......
E0301 16:08:36.224319 3165818 writer.go:142] Error setting Degraded annotation for node <<node>>: unable to update node "&Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:,DoNotUse_ExternalID:,ProviderID:,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{},Allocatable:ResourceList{},Phase:,Conditions:[],Addresses:[],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:0,},},NodeInfo:NodeSystemInfo{MachineID:,SystemUUID:,BootID:,KernelVersion:,OSImage:,ContainerRuntimeVersion:,KubeletVersion:,KubeProxyVersion:,OperatingSystem:,Architecture:,},Images:[],VolumesInUse:[],VolumesAttached:[],Config:nil,},}": Node "<<node>>" is invalid: metadata.annotations: Too long: must have at most 262144 characters

Inside the nodes definitions (not all of them) the field "machineconfiguration.openshift.io/reason" is unexpectedly big (attaching file).

The problem is why is this field filled with so much data?

Version-Release number of selected component (if applicable):
4.2.8

How reproducible:
Not yet tested in lab, but i think that what triggers this is an edition of the file "/etc/containers/registries.conf" outside of the cluster scope (manual editing). But restoring the file is not enough as the daemon refuses to work with such a big metadata field.


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:
"machineconfiguration.openshift.io/reason" field having a manageable size.

Additional info:
Attaching some files

Comment 2 Mario Abajo 2020-03-02 10:15:48 UTC
I used this command to clean the "machineconfiguration.openshift.io/reason" field:

$ oc patch node <node> --type merge --patch '{"metadata": {"annotations": {"machineconfiguration.openshift.io/reason": ""}}}'