job: periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-aws-upgrade-rollback is always failing in CI, see testgrid results: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.7-informing#periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-aws-upgrade-rollback sample job: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-aws-upgrade-rollback/1407830411454713856 The log seems to indicate nodes are failing to update, probably during the rollback from 4.7 to 4.6: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-aws-upgrade-rollback/1407830411454713856/artifacts/e2e-aws-upgrade-rollback/openshift-e2e-test/artifacts/e2e.log Jun 24 02:34:28.388 E clusteroperator/machine-config changed Degraded to True: RequiredPoolsFailed: Unable to apply 4.6.36: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a5d35055682d2a1fffc88aa3399d922f expected 093319e36578d1d40f02e4d8ba939dee1f536448 has 8530c27d3d9b6155923d348058bc025a6a98ec3c: pool is degraded because nodes fail with "3 nodes are reporting degraded status on sync": "Node ip-10-0-183-137.ec2.internal is reporting: \"unexpected on-disk state validating against rendered-master-a5d35055682d2a1fffc88aa3399d922f: Failed to parse Ignition for validation: parsing Ignition config failed: unknown version. Supported spec versions: 2.2, 3.0, 3.1\", Node ip-10-0-185-215.ec2.internal is reporting: \"unexpected on-disk state validating against rendered-master-a5d35055682d2a1fffc88aa3399d922f: Failed to parse Ignition for validation: parsing Ignition config failed: unknown version. Supported spec versions: 2.2, 3.0, 3.1\", Node ip-10-0-193-173.ec2.internal is reporting: \"unexpected on-disk state validating against rendered-master-a5d35055682d2a1fffc88aa3399d922f: Failed to parse Ignition for validation: parsing Ignition config failed: unknown version. Supported spec versions: 2.2, 3.0, 3.1\"", retrying
Err, maybe a Sippy / CI-search indexing bug? Because bug 1947477 grew the Environment job=... entry that should have had created the Sippy association back in April [1]. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1947477#c5
> Err, maybe a Sippy / CI-search indexing bug? Because bug 1947477 grew the Environment job=... entry that should have had created the Sippy association back in April not sure, but https://sippy.ci.openshift.org/?release=4.7#InfrequentJobPassRatesByJobName definitely has the association now (maybe it had it before and i missed it when i opened this?)