Description of problem: The instance-state annotation and the providerStatus.instanceState occasionally showing different values [1]. This should never happen as they consume from the same source. This is likely to be an issue on the client failing to patch the status or patching it wrongly. [1] https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.4/866/artifacts/e2e-aws/machines.json Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
This a build where this happened: https://gcsweb-ci.svc.ci.openshift.org/gcs/origin-ci-test/pr-logs/pull/openshift_openshift-apiserver/9/pull-ci-openshift-openshift-apiserver-master-e2e-aws-serial/17/artifacts/e2e-aws-serial/ See machine controller https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_openshift-apiserver/9/pull-ci-openshift-openshift-apiserver-master-e2e-aws-serial/17/artifacts/e2e-aws-serial/pods/openshift-machine-api_machine-api-controllers-7d6c9dcb86-9tcnr_machine-controller.log See machine "ci-op-syr3smwj-19c40-zmjmj-worker-us-east-1b-5x5hn" https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_openshift-apiserver/9/pull-ci-openshift-openshift-apiserver-master-e2e-aws-serial/17/artifacts/e2e-aws-serial/must-gather/registry-svc-ci-openshift-org-ci-op-syr3smwj-stable-sha256-dd03a9e0ac7b6e710037d8f0d0a5001b47c27bf859e49b0256e8ac54f5bd8198/namespaces/openshift-machine-api/machine.openshift.io/machines/ci-op-syr3smwj-19c40-zmjmj-worker-us-east-1b-5x5hn.yaml
This is a regression introduced by https://github.com/openshift/cluster-api-provider-aws/pull/285/commits/e6a5716b217977d28ca76d4eabff063077d4b4e4#diff-acdcd15b24a848e93a682408b101755fR638-R651 We'll put a fix for it soon.
Verified in 4.4.0-0.nightly-2020-02-05-181112. Added regression test for this scenario, this will be covered in future tests. oc get machines -n openshift-machine-api -o jsonpath='{range .items[*]}{.metadata.annotations}{"\t"}{.status.providerStatus.instanceState}{"\n"}{end}' map[machine.openshift.io/instance-state:running] running map[machine.openshift.io/instance-state:running] running map[machine.openshift.io/instance-state:running] running map[machine.openshift.io/instance-state:running] running map[machine.openshift.io/instance-state:running] running map[machine.openshift.io/instance-state:running] running map[machine.openshift.io/instance-state:running] running