Description of problem: We see failures with "[sig-arch] events should not repeat pathologically expand_less failures" in recent CI upgrade runs of OCP 4.9 to 4.10 and 4.8 to 4.9 for s390x. : [sig-arch] events should not repeat pathologically expand_less 0s 2 events happened too frequently event happened 34 times, something is wrong: ns/openshift-machine-api machine/libvirt-s390x-0-0-708-wltbm-worker-0-ck7tg - reason/Updated Updated Machine libvirt-s390x-0-0-708-wltbm-worker-0-ck7tg event happened 34 times, something is wrong: ns/openshift-machine-api machine/libvirt-s390x-0-0-708-wltbm-worker-0-trzd6 - reason/Updated Updated Machine libvirt-s390x-0-0-708-wltbm-worker-0-trzd6 1) 4.9 to 4.10 upgrade --> https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.10-upgrade-from-nightly-4.9-ocp-remote-libvirt-s390x/1452575786342027264 2) 4.8 to 4.9 upgrade --> https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.9-upgrade-from-nightly-4.8-ocp-remote-libvirt-s390x/1452288892060307456 Search results --> https://search.ci.openshift.org/?search=reason%2FUpdated+Updated+Machine&maxAge=48h&context=1&type=bug%2Bjunit&name=s390x&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job Version-Release number of selected component (if applicable): ocp 4.10, 4.9 How reproducible: Monitor the s390x CI jobs here https://prow.ci.openshift.org/?job=periodic*remote*libvirt*s390x for the upgrade jobs. Steps to Reproduce: - Actual results: The tests should pass. Expected results: Additional info: - Observing this on libvirt platform - related BZ - https://bugzilla.redhat.com/show_bug.cgi?id=1988992 where it was earlier reported and noticed on Azure platform. - based on slack triage, looks like the event creation should happen after the update succeeded at https://github.com/openshift/cluster-api-provider-libvirt/blob/master/pkg/cloud/libvirt/actuators/machine/actuator.go#L209
no ": [sig-arch] events should not repeat pathologically" failure now. https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.10-upgrade-from-nightly-4.9-ocp-remote-libvirt-s390x/1459959316772556800 https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.10-upgrade-from-nightly-4.9-ocp-remote-libvirt-s390x/1459778372287598592
We still see failures with "[sig-arch] events should not repeat pathologically" in recent CI upgrade runs of OCP 4.10 to 4.11, 4.9 to 4.10 and 4.8 to 4.9 on s390x. : [sig-arch] events should not repeat pathologically 2 events happened too frequently event happened 51 times, something is wrong: ns/openshift-machine-api machine/libvirt-s390x-1-1-1e5-btbz4-worker-0-jbj7n - reason/Updated Updated Machine libvirt-s390x-1-1-1e5-btbz4-worker-0-jbj7n event happened 47 times, something is wrong: ns/openshift-machine-api machine/libvirt-s390x-1-1-1e5-btbz4-worker-0-sxf6q - reason/Updated Updated Machine libvirt-s390x-1-1-1e5-btbz4-worker-0-sxf6q 1) 4.8 to 4.9 upgrade --> 2) 4.9 to 4.10 upgrade --> 3) 4.10 to 4.11 upgrade --> Search results --> https://search.ci.openshift.org/?search=reason%2FUpdated+Updated+Machine&maxAge=336h&context=1&type=bug%2Bjunit&name=s390x&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job Version-Release number of selected component (if applicable): ocp 4.11, 4.10, 4.9 How reproducible: Monitor the s390x CI jobs here https://prow.ci.openshift.org/?job=periodic*remote*libvirt*s390x for the upgrade jobs. Steps to Reproduce: - Actual results: The tests should pass. Expected results: We still see failures with "[sig-arch] events should not repeat pathologically" in recent CI upgrade runs of OCP 4.10 to 4.11, 4.9 to 4.10 and 4.8 to 4.9 on s390x. : [sig-arch] events should not repeat pathologically 2 events happened too frequently event happened 51 times, something is wrong: ns/openshift-machine-api machine/libvirt-s390x-1-1-1e5-btbz4-worker-0-jbj7n - reason/Updated Updated Machine libvirt-s390x-1-1-1e5-btbz4-worker-0-jbj7n event happened 47 times, something is wrong: ns/openshift-machine-api machine/libvirt-s390x-1-1-1e5-btbz4-worker-0-sxf6q - reason/Updated Updated Machine libvirt-s390x-1-1-1e5-btbz4-worker-0-sxf6q 1) 4.8 to 4.9 upgrade --> https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.9-upgrade-from-nightly-4.8-ocp-remote-libvirt-s390x/1491245642264088576 2) 4.9 to 4.10 upgrade --> https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.10-upgrade-from-nightly-4.9-ocp-remote-libvirt-s390x/1491487255926149120 3) 4.10 to 4.11 upgrade --> https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.11-upgrade-from-nightly-4.10-ocp-remote-libvirt-s390x/1491532562751819776 Search results --> https://search.ci.openshift.org/?search=reason%2FUpdated+Updated+Machine&maxAge=336h&context=1&type=bug%2Bjunit&name=s390x&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job Version-Release number of selected component (if applicable): ocp 4.11, 4.10, 4.9 How reproducible: Monitor the s390x CI jobs here https://prow.ci.openshift.org/?job=periodic*remote*libvirt*s390x for the upgrade jobs. Steps to Reproduce: - Actual results: The test fails Expected results: The tests should pass.
no ": [sig-arch] events should not repeat pathologically" failure for 4.10 to 4.11 upgrade these days after code merge, move this to Verified. https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.11-upgrade-from-nightly-4.10-ocp-remote-libvirt-s390x/1508187230869393408 https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.11-upgrade-from-nightly-4.10-ocp-remote-libvirt-s390x/1508006035464392704 but still have failures with "[sig-arch] events should not repeat pathologically" for 4.8 to 4.9 upgrade and 4.9 to 4.10 upgrade, seems need backport. 4.9 to 4.10 upgrade: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.10-upgrade-from-nightly-4.9-ocp-remote-libvirt-s390x/1508202337959677952 4.8 to 4.9 upgrade: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.9-upgrade-from-nightly-4.8-ocp-remote-libvirt-s390x/1507553107687510016
Hi Joel, do you think this should be backport to 4.10 and 4.9 to solve the failures with "[sig-arch] events should not repeat pathologically" for 4.9 to 4.10 upgrade and 4.8 to 4.9 upgrade?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069