Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1975975

Summary: periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-aws-upgrade-rollback is permfailing
Product: OpenShift Container Platform Reporter: Ben Parees <bparees>
Component: Machine Config OperatorAssignee: Yu Qi Zhang <jerzhang>
Status: CLOSED DUPLICATE QA Contact: Michael Nguyen <mnguyen>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.7CC: sippy, wking
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
job=periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-aws-upgrade-rollback=all
Last Closed: 2021-06-24 21:35:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ben Parees 2021-06-24 20:41:04 UTC 
job:
periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-aws-upgrade-rollback 

is always failing in CI, see testgrid results:
https://testgrid.k8s.io/redhat-openshift-ocp-release-4.7-informing#periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-aws-upgrade-rollback

sample job:
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-aws-upgrade-rollback/1407830411454713856

The log seems to indicate nodes are failing to update, probably during the rollback from 4.7 to 4.6:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-aws-upgrade-rollback/1407830411454713856/artifacts/e2e-aws-upgrade-rollback/openshift-e2e-test/artifacts/e2e.log


Jun 24 02:34:28.388 E clusteroperator/machine-config changed Degraded to True: RequiredPoolsFailed: Unable to apply 4.6.36: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a5d35055682d2a1fffc88aa3399d922f expected 093319e36578d1d40f02e4d8ba939dee1f536448 has 8530c27d3d9b6155923d348058bc025a6a98ec3c: pool is degraded because nodes fail with "3 nodes are reporting degraded status on sync": "Node ip-10-0-183-137.ec2.internal is reporting: \"unexpected on-disk state validating against rendered-master-a5d35055682d2a1fffc88aa3399d922f: Failed to parse Ignition for validation: parsing Ignition config failed: unknown version. Supported spec versions: 2.2, 3.0, 3.1\", Node ip-10-0-185-215.ec2.internal is reporting: \"unexpected on-disk state validating against rendered-master-a5d35055682d2a1fffc88aa3399d922f: Failed to parse Ignition for validation: parsing Ignition config failed: unknown version. Supported spec versions: 2.2, 3.0, 3.1\", Node ip-10-0-193-173.ec2.internal is reporting: \"unexpected on-disk state validating against rendered-master-a5d35055682d2a1fffc88aa3399d922f: Failed to parse Ignition for validation: parsing Ignition config failed: unknown version. Supported spec versions: 2.2, 3.0, 3.1\"", retrying
Comment 3 W. Trevor King 2021-06-25 05:42:34 UTC 
Err, maybe a Sippy / CI-search indexing bug?  Because bug 1947477 grew the Environment job=... entry that should have had created the Sippy association back in April [1].

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1947477#c5
Comment 4 Ben Parees 2021-06-25 13:20:43 UTC 
> Err, maybe a Sippy / CI-search indexing bug?  Because bug 1947477 grew the Environment job=... entry that should have had created the Sippy association back in April

not sure, but https://sippy.ci.openshift.org/?release=4.7#InfrequentJobPassRatesByJobName definitely has the association now (maybe it had it before and i missed it when i opened this?)