Bug 1839612

Summary:	[rhv]machine controller continuously reconciling status
Product:	OpenShift Container Platform	Reporter:	sunzhaohua <zhsun>
Component:	Cloud Compute	Assignee:	Roy Golan <rgolan>
Cloud Compute sub component:	oVirt Provider	QA Contact:	Lucie Leistnerova <lleistne>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	low
Priority:	unspecified
Version:	4.5
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-07-09 11:30:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description sunzhaohua 2020-05-25 03:22:31 UTC

Description of problem:
machine controller continuously reconciling status

Version-Release number of selected component (if applicable):
4.5.0-0.nightly-2020-05-20-053050

How reproducible:
Always

Steps to Reproduce:
1. Check mahine-controller logs
2.
3.

Actual results:
I0525 02:50:22.531518       1 actuator.go:261] Updating machine status
I0525 02:50:22.700312       1 controller.go:164] Reconciling Machine "primary-t7984-master-0"
I0525 02:50:22.700337       1 controller.go:376] Machine "primary-t7984-master-0" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0525 02:50:22.743780       1 controller.go:284] Reconciling machine "primary-t7984-master-0" triggers idempotent update
I0525 02:50:22.783194       1 actuator.go:261] Updating machine status
I0525 02:50:22.816680       1 controller.go:164] Reconciling Machine "primary-t7984-master-1"
I0525 02:50:22.816704       1 controller.go:376] Machine "primary-t7984-master-1" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0525 02:50:22.859063       1 controller.go:284] Reconciling machine "primary-t7984-master-1" triggers idempotent update
I0525 02:50:22.900111       1 actuator.go:261] Updating machine status
I0525 02:50:22.937277       1 controller.go:164] Reconciling Machine "primary-t7984-master-2"
I0525 02:50:22.937300       1 controller.go:376] Machine "primary-t7984-master-2" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0525 02:50:22.977132       1 controller.go:284] Reconciling machine "primary-t7984-master-2" triggers idempotent update
I0525 02:50:23.015794       1 actuator.go:261] Updating machine status
I0525 02:50:23.056099       1 controller.go:164] Reconciling Machine "primary-t7984-worker-0-9t8kq"
I0525 02:50:23.056127       1 controller.go:376] Machine "primary-t7984-worker-0-9t8kq" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0525 02:50:23.102849       1 controller.go:284] Reconciling machine "primary-t7984-worker-0-9t8kq" triggers idempotent update
I0525 02:50:23.148105       1 actuator.go:261] Updating machine status
I0525 02:50:23.377514       1 controller.go:164] Reconciling Machine "primary-t7984-worker-0-lsd9c"
I0525 02:50:23.377536       1 controller.go:376] Machine "primary-t7984-worker-0-lsd9c" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0525 02:50:23.423202       1 controller.go:284] Reconciling machine "primary-t7984-worker-0-lsd9c" triggers idempotent update
I0525 02:50:23.468307       1 actuator.go:261] Updating machine status
I0525 02:50:23.529990       1 controller.go:164] Reconciling Machine "primary-t7984-worker-0-nx52p"
I0525 02:50:23.530330       1 controller.go:376] Machine "primary-t7984-worker-0-nx52p" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0525 02:50:23.576036       1 controller.go:284] Reconciling machine "primary-t7984-worker-0-nx52p" triggers idempotent update
I0525 02:50:23.617063       1 actuator.go:261] Updating machine status
I0525 02:50:23.656483       1 controller.go:164] Reconciling Machine "primary-t7984-worker-0-rdr2r"
I0525 02:50:23.656512       1 controller.go:376] Machine "primary-t7984-worker-0-rdr2r" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0525 02:50:23.696554       1 controller.go:284] Reconciling machine "primary-t7984-worker-0-rdr2r" triggers idempotent update
I0525 02:50:23.735190       1 actuator.go:261] Updating machine status
I0525 02:50:23.775800       1 controller.go:164] Reconciling Machine "primary-t7984-master-0"
I0525 02:50:23.775824       1 controller.go:376] Machine "primary-t7984-master-0" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0525 02:50:23.823266       1 controller.go:284] Reconciling machine "primary-t7984-master-0" triggers idempotent update
I0525 02:50:23.869944       1 actuator.go:261] Updating machine status
I0525 02:50:23.908538       1 controller.go:164] Reconciling Machine "primary-t7984-master-1"
I0525 02:50:23.908589       1 controller.go:376] Machine "primary-t7984-master-1" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0525 02:50:23.955151       1 controller.go:284] Reconciling machine "primary-t7984-master-1" triggers idempotent update
I0525 02:50:23.997074       1 actuator.go:261] Updating machine status


Expected results:
If no event is triggered no reconciling logs

Additional info:

Comment 1 sunzhaohua 2020-05-25 03:30:15 UTC

machine-controller full log:
http://file.rdu.redhat.com/~zhsun/machine-controller.log

Comment 2 Roy Golan 2020-05-25 05:38:14 UTC

I just handled that on https://bugzilla.redhat.com/show_bug.cgi?id=1827670

Essentially the machine object will be updated only if a change was discovered in provider (rhv)

For example look the machine controller log from the this CI run after the fix - https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-ovirt-4.5/73/artifacts/e2e-ovirt/pods/openshift-machine-api_machine-api-controllers-6756dd6588-bmxkm_machine-controller.log

Comment 3 sunzhaohua 2020-05-25 06:47:28 UTC

Thanks Roy Golan, seems this bug is a dup of https://bugzilla.redhat.com/show_bug.cgi?id=1827670, will check it when I have a new cluster.

Comment 4 Roy Golan 2020-07-09 11:30:08 UTC


*** This bug has been marked as a duplicate of bug 1827670 ***