Bug 2088428

Description of problem:

After installing a cluster on GEN 10 plus hardware, the customer(Nokia-NOM) finds that the baremetal operator is sporadically stuck in 'progressing' state. 

From our analysis, the installation is actually complete. The nodes and rest of the cluster operators are Available and not Degraded. 

- The cluster-baremetal-opeator(top-level) doesn't have any errors as such except for some warnings.(worth looking at as they might have caused the status to NOT be updated) [1]
- The metal3-baremetal-operator(2nd-level) pod also doesn't have any errors if I grep out info logs.  
- I checked other resources like machines, BMH, and provisioning CR that are managed by the bare metal operator and do not see any problem there. [2]
- It is possible that the cluster-baremetal-opeator has synced the resources but failed to report it back to the cluster-version-operator. 
- Logs from the cluster version operator indicate that the manifest resource sync is complete. I believe it is unlikely that an issue is here within CVO. [3]

Due to nothing in particular standing out from the logs, asked them to roll out a new bare-metal operator pod. 
  
  # oc delete pod cluster-baremetal-operator-xxxxxx -n openshift-machine-api
  
This got the stale condition to be pushed out and the baremetal operator was in progressing=false state now. 




----
Logs:

[1]
```
2022-05-17T06:35:23.879134688Z E0517 06:35:23.879105       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:225: Failed to watch *v1.Proxy: the server could not find the requested resource (get proxies.config.openshift.io)
2022-05-17T06:35:23.890165361Z E0517 06:35:23.890113       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:137: Failed to watch *v1alpha1.Provisioning: the server could not find the requested resource (get provisionings.metal3.io)
2022-05-17T06:39:41.825418177Z W0517 06:39:41.825142       1 reflector.go:436] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:137: watch of *v1alpha1.Provisioning ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
2022-05-17T06:39:41.825418177Z W0517 06:39:41.825171       1 reflector.go:436] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:225: watch of *v1.DaemonSet ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
2022-05-17T06:39:41.825418177Z W0517 06:39:41.825175       1 reflector.go:436] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:225: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
2
```

[2] 
```
[simore@supportshell-1 03221561]$ omg get machine 
NAME                        PHASE    TYPE  REGION  ZONE  AGE
sl732-fw8cs-master-0        Running                      6h17m
sl732-fw8cs-master-1        Running                      6h17m
sl732-fw8cs-master-2        Running                      6h17m
sl732-fw8cs-worker-0-ctsgj  Running                      5h38m
sl732-fw8cs-worker-0-j5shn  Running                      5h38m

...
[simore@supportshell-1 03221561]$ omg get bmh
NAME           AGE
sl732compute1  6h17m
sl732compute2  6h17m
sl732control1  6h17m
sl732control2  6h17m
sl732control3  6h17m

...
[simore@supportshell-1 03221561]$ omg get provisioning -oyaml 
apiVersion: metal3.io/v1alpha1
kind: Provisioning
metadata:
  creationTimestamp: '2022-05-17T05:53:51Z'
  finalizers:
  - provisioning.metal3.io
  generation: 2
  name: provisioning-configuration
  resourceVersion: '16277'
  uid: 7efe44a3-1e28-468e-afa9-9bbef0fbdf02
spec:
  provisioningNetwork: Disabled
  provisioningOSDownloadURL: http://10.92.117.51:8080/rhcos-openstack.x86_64.qcow2.gz?sha256=e0a1d8a99c5869150a56b8de475ea7952ca2fa3aacad7ca48533d1176df503ab
status:
  generations:
  - group: apps
    hash: ''
    lastGeneration: 1
    name: metal3
    namespace: openshift-machine-api
    resource: deployments
  - group: apps
    hash: ''
    lastGeneration: 1
    name: metal3-image-cache
    namespace: openshift-machine-api
    resource: daemonsets
  observedGeneration: 2
  readyReplicas: 0
```

[3]
```
2022-05-17T12:11:03.852498096Z I0517 12:11:03.852421       1 cvo.go:483] Finished syncing cluster version "openshift-cluster-version/version" (136.141µs)
2022-05-17T12:11:05.346829912Z I0517 12:11:05.346694       1 sync_worker.go:283] Wait finished
2022-05-17T12:11:05.346829912Z I0517 12:11:05.346721       1 sync_worker.go:337] Previous sync status: &cvo.SyncWorkerStatus{Generation:1, Step:"", Failure:error(nil), Done:681, Total:681, Completed:87, Reconcil
ing:true, Initial:false, VersionHash:"PPyjn2NyQf4=", LastProgress:time.Time{wall:0xc09901eb9736078c, ext:17469120453094, loc:(*time.Location)(0x2ad3e80)}, Actual:v1.Release{Version:"4.8.29-assembly.art3875", Ima
ge:"quay.io/openshift-release-dev/ocp-release@sha256:5719c6cdcac7f8517d00d11fe07a571b357d88a9c0462565cc61adcf50cd2a66", URL:"", Channels:[]string(nil)}, Verified:false}
```


Version-Release number of selected component (if applicable):
- This is the hotfix version of OCP being installed 4.8.29-assembly.art3875


How reproducible:
- The issue is not consistently reproduced on each new installation. 
- Customer mentions it has reoccurred a few times.


Steps to Reproduce:
1. Install RHOCP on GEN 10 plus hardware.
2. Hope the issue occurs 


Actual results:
- The baremetal operator is sporadically stuck in progressing state after a fresh install. 


Expected results:
- After the install the operator is supposed to NOT be in a progressing state. 


Additional info:
- Customer is unsure whether to trust the status if all the resources are in fact synced from the operator conditions. 
- Customer is requesting an RCA as to why this happens. 

Support understands due to the inconsistent nature of the reproducer, it is difficult to find the root cause of this issue. It might also depend on various underlying factors like network gitch, etc. We need engineering assistance in understanding if there is a way to increase log level of this operator in a way that could point further at the root cause. Additionally, if such an issue is seen again, what info/logs should be captured that could help in finding the root cause.