Created attachment 1697905 [details]
upgrade-test-log

Description of problem:
Upgrade test suite fails on ppc64le environment - Frontends were down more than toleration level

Version-Release number of selected component (if applicable):
4.3.z

How reproducible:
Consistently

Steps to Reproduce:
1. Install 4.3.23
2. Run openshift-tests run-upgrade all --to-image=<OCP4.4_image>

Actual results:
Jun  5 04:04:58.996: INFO: Waiting up to 7m0s for all (but 100) nodes to be ready

STEP: Destroying namespace "e2e-k8s-sig-apps-job-upgrade-617" for this suite.

Jun  5 04:04:59.005: INFO: Running AfterSuite actions on node 1

fail [github.com/openshift/origin/test/extended/util/disruption/disruption.go:226]: Jun  5 04:04:58.773: Frontends were unreachable during disruption for at least 16m30s of 47m16s (35%):


Expected results:
No errors, and tests should pass

Additional info:
Part of the upgrade tests &frontends.AvailableTest{} is failing on power architecture using libvirt IPI method.. 

Test fails at :

https://github.com/openshift/origin/blob/master/test/extended/util/disruption/frontends/frontends.go#L106

Test fails because the sum of the duration of all events exceeds toleratation(0.2) mentioned in the above test code... Tested with the higher value of toleration 0.4 upgrade tests get passed as expected on power hardware..

[root@osp115 upgrade]# git diff ../../extended/util/disruption/frontends/frontends.go
diff --git a/test/extended/util/disruption/frontends/frontends.go b/test/extended/util/disruption/frontends/frontends.go
index a5e673ac07..6195d0cf45 100644
— a/test/extended/util/disruption/frontends/frontends.go
+++ b/test/extended/util/disruption/frontends/frontends.go
@@ -100,7 +100,7 @@ func (t *AvailableTest) Test(f *framework.Framework, done <-chan struct{}, upgra
cancel()
end := time.Now()

disruption.ExpectNoDisruption(f, 0.20, end.Sub(start), m.Events(time.Time{}, time.Time{}), "Frontends were unreachable during disruption")
+ disruption.ExpectNoDisruption(f, 0.40, end.Sub(start), m.Events(time.Time{}, time.Time{}), "Frontends were unreachable during disruption")
}
// Teardown cleans up any remaining resources.