Bug 2033426 - release-openshift-origin-installer-e2e-aws-upgrade-4.6-to-4.7-to-4.8-to-4.9-ci fails due to missing admin ack
Summary: release-openshift-origin-installer-e2e-aws-upgrade-4.6-to-4.7-to-4.8-to-4.9-c...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Test Framework
Version: 4.8
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 4.8.z
Assignee: W. Trevor King
QA Contact:
URL:
Whiteboard:
Depends On: 2027929
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-16 19:24 UTC by OpenShift BugZilla Robot
Modified: 2021-12-22 01:54 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-12-21 20:29:24 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 26704 0 None Merged [release-4.8] Bug 2033426: test/e2e/upgrade/adminack: Poll gates for duration of update 2021-12-21 20:02:21 UTC

Comment 5 W. Trevor King 2021-12-21 20:29:24 UTC
Motivation for this 4.8 backport was failures like [1]:

  : [sig-cluster-lifecycle] TestAdminAck should succeed [Suite:openshift/conformance/parallel]
    Run #0: Failed	9s
    fail [runtime/map_faststr.go:204]: Test Panicked: assignment to entry in nil map

Pull request landed over 3d ago [2].  Checking:

  $ w3m -dump -cols 200 'https://search.ci.openshift.org/?maxAge=48h&type=junit&search=Test+Panicked:+assignment+to+entry+in+nil+map' | grep 'failures match' | sort
  periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-ovn-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
  periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade (all) - 11 runs, 55% failed, 17% of failures match = 9% impact
  periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-azure-ovn-upgrade (all) - 2 runs, 100% failed, 100% of failures match = 100% impact
  periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-gcp-upgrade (all) - 11 runs, 45% failed, 20% of failures match = 9% impact
  release-openshift-origin-installer-e2e-aws-shared-vpc-4.8 (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
  release-openshift-origin-installer-e2e-azure-shared-vpc-4.8 (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
  release-openshift-origin-installer-e2e-gcp-shared-vpc-4.8 (all) - 1 runs, 100% failed, 100% of failures match = 100% impact

That's better than the aggregate over the past 2 weeks:

  $ w3m -dump -cols 200 'https://search.ci.openshift.org/?maxAge=336h&type=junit&search=Test+Panicked:+assignment+to+entry+in+nil+map' | grep 'failures match' | sort
  periodic-ci-openshift-multiarch-master-nightly-4.8-ocp-e2e-remote-libvirt-ppc64le (all) - 20 runs, 90% failed, 94% of failures match = 85% impact
  periodic-ci-openshift-multiarch-master-nightly-4.8-ocp-e2e-remote-libvirt-s390x (all) - 20 runs, 90% failed, 89% of failures match = 80% impact
  periodic-ci-openshift-multiarch-master-nightly-4.9-upgrade-from-nightly-4.8-ocp-remote-libvirt-ppc64le (all) - 28 runs, 36% failed, 10% of failures match = 4% impact
  periodic-ci-openshift-multiarch-master-nightly-4.9-upgrade-from-nightly-4.8-ocp-remote-libvirt-s390x (all) - 28 runs, 57% failed, 6% of failures match = 4% impact
  periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-ovn-upgrade (all) - 7 runs, 100% failed, 86% of failures match = 86% impact
  periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade (all) - 101 runs, 81% failed, 76% of failures match = 61% impact
  periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-azure-ovn-upgrade (all) - 14 runs, 100% failed, 86% of failures match = 86% impact
  periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-azure-upgrade (all) - 14 runs, 100% failed, 64% of failures match = 64% impact
  periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-gcp-ovn-upgrade (all) - 7 runs, 100% failed, 86% of failures match = 86% impact
  periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-gcp-upgrade (all) - 101 runs, 81% failed, 76% of failures match = 61% impact
  periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-openstack-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
  periodic-ci-openshift-release-master-nightly-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade (all) - 26 runs, 88% failed, 83% of failures match = 73% impact
  periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.8-upgrade-from-stable-4.7-e2e-openstack-upgrade (all) - 5 runs, 100% failed, 40% of failures match = 40% impact
  pull-ci-openshift-ovn-kubernetes-release-4.8-4.8-upgrade-from-stable-4.7-e2e-aws-ovn-upgrade (all) - 2 runs, 100% failed, 100% of failures match = 100% impact
  rehearse-24376-periodic-ci-openshift-multiarch-master-nightly-4.8-ocp-e2e-remote-libvirt-ppc64le (all) - 8 runs, 100% failed, 25% of failures match = 25% impact
  release-openshift-origin-installer-e2e-aws-disruptive-4.8 (all) - 7 runs, 100% failed, 43% of failures match = 43% impact
  release-openshift-origin-installer-e2e-aws-shared-vpc-4.8 (all) - 7 runs, 100% failed, 100% of failures match = 100% impact
  release-openshift-origin-installer-e2e-aws-upgrade (all) - 46 runs, 35% failed, 13% of failures match = 4% impact
  release-openshift-origin-installer-e2e-azure-shared-vpc-4.8 (all) - 7 runs, 100% failed, 100% of failures match = 100% impact
  release-openshift-origin-installer-e2e-gcp-shared-vpc-4.8 (all) - 7 runs, 100% failed, 100% of failures match = 100% impact

And [3] shows a bunch of 4.8-upgrade-from-stable-4.7 jobs over the past day with no hits for the panic.  That's enough for me to close this bug CURRENTRELEASE, because we don't usually attach origin CI-suite fixes to shipped errata.  Bug 2030972 is tracking some follow-up work for this test-case, and if we find other issues we can open more dev-release bugs to track those changes back through the backport process.

[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-gcp-upgrade/1470777392988426240
[2]: https://github.com/openshift/origin/pull/26704#event-5787378598
[3]: https://search.ci.openshift.org/chart?maxAge=24h&type=junit&name=4.8-upgrade-from-stable-4.7&search=Test+Panicked%3A+assignment+to+entry+in+nil+map

Comment 6 wewang 2021-12-22 01:54:16 UTC
Thanks @wking


Note You need to log in before you can comment on or make changes to this bug.