Description of problem: Constantly failing job: https://prow.svc.ci.openshift.org/job-history/origin-ci-test/pr-logs/directory/pull-ci-openshift-openshift-ansible-release-3.11-e2e-gcp [Fail] [Feature:DeploymentConfig] deploymentconfigs with test deployments [Conformance] [It] should run a deployment to completion and then scale to zero [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs with multiple image change triggers [Conformance] [It] should run a successful deployment with a trigger used by different containers [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs with enhanced status [Conformance] [It] should include various info in status [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs adoption [Conformance] [It] will orphan all RCs and adopt them back when recreated [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs won't deploy RC with unresolved images [Conformance] [It] when patched with empty image [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs with failing hook [Conformance] [It] should get all logs from retried hooks [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs keep the deployer pod invariant valid [Conformance] [It] should deal with cancellation after deployer pod succeeded [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs viewing rollout history [Conformance] [It] should print the rollout history [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs with minimum ready seconds set [Conformance] [It] should not transition the deployment to Complete before satisfied [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs when changing image change trigger [Conformance] [It] should successfully trigger from an updated image [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs generation [Conformance] [It] should deploy based on a status version bump [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs initially [Conformance] [It] should not deploy if pods never transition to ready [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs rolled back [Conformance] [It] should rollback to an older deployment [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs when tagging images [Conformance] [It] should successfully tag the deployed image [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs keep the deployer pod invariant valid [Conformance] [It] should deal with config change in case the deployment is still running [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs with custom deployments [Conformance] [It] should run the custom deployment steps [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs with multiple image change triggers [Conformance] [It] should run a successful deployment with multiple triggers [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs [It] should adhere to Three Laws of Controllers [Conformance] [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs paused [Conformance] [It] should disable actions on deployments [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs when run iteratively [Conformance] [It] should only deploy the last deployment [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs keep the deployer pod invariant valid [Conformance] [It] should deal with cancellation of running deployment [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs with revision history limits [Conformance] [It] should never persist more old deployments than acceptable after being observed by the controller [Suite:openshift/conformance/parallel/minimal] [Fail] [Feature:DeploymentConfig] deploymentconfigs with env in params referencing the configmap [Conformance] [It] should expand the config map key to a value [Suite:openshift/conformance/parallel/minimal] Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Nothing has changed in deployment related tests for over a year, this problem looks like it's related to GCP installation because the cluster which the tests are running on is 4 x master, and since the deployer pod requires node-role.kubernetes.io/compute: "true" it can't run and thus the test will fail. I don't see anything particular that stands out since November in the CI configuration either so I'm sending this over to ansible installer team, maybe something changed there wrt GCP cluster.
These failures have reduced and are now just flakey instead of many failures per job. https://search.apps.build01.ci.devcluster.openshift.com/?search=Fail.*Feature%3ADeploymentConfig&maxAge=48h&context=1&type=bug%2Bjunit&name=3.11&maxMatches=5&maxBytes=20971520&groupBy=job Across 83 runs and 6 jobs (83.13% failed), matched 8.70% of failing runs and 16.67% of jobs in 45ms pull-ci-openshift-openshift-ansible-release-3.11-e2e-aws - 16 runs, 88% failed, 43% of failures match #2818 junit 30 minutes ago [Fail] [Feature:DeploymentConfig] deploymentconfigs adoption [Conformance] [It] will orphan all RCs and adopt them back when recreated [Suite:openshift/conformance/parallel/minimal] /tmp/openshift/build-rpms/rpm/BUILD/origin-3.11.0/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:1657 #2816 junit 7 hours ago [Fail] [Feature:DeploymentConfig] deploymentconfigs paused [Conformance] [It] should disable actions on deployments [Suite:openshift/conformance/parallel/minimal] /tmp/openshift/build-rpms/rpm/BUILD/origin-3.11.0/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:793 [Fail] [Feature:DeploymentConfig] deploymentconfigs with revision history limits [Conformance] [It] should never persist more old deployments than acceptable after being observed by the controller [Suite:openshift/conformance/parallel/minimal] /tmp/openshift/build-rpms/rpm/BUILD/origin-3.11.0/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:969 [Fail] [Feature:DeploymentConfig] deploymentconfigs rolled back [Conformance] [It] should rollback to an older deployment [Suite:openshift/conformance/parallel/minimal] /tmp/openshift/build-rpms/rpm/BUILD/origin-3.11.0/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:872 #2815 junit 7 hours ago [Fail] [Feature:DeploymentConfig] deploymentconfigs rolled back [Conformance] [It] should rollback to an older deployment [Suite:openshift/conformance/parallel/minimal] /tmp/openshift/build-rpms/rpm/BUILD/origin-3.11.0/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:878 #2813 junit 9 hours ago [Fail] [Feature:DeploymentConfig] deploymentconfigs when changing image change trigger [Conformance] [It] should successfully trigger from an updated image [Suite:openshift/conformance/parallel/minimal] /tmp/openshift/build-rpms/rpm/BUILD/origin-3.11.0/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:436 #2813 junit 9 hours ago [Fail] [Feature:DeploymentConfig] deploymentconfigs rolled back [Conformance] [It] should rollback to an older deployment [Suite:openshift/conformance/parallel/minimal] /tmp/openshift/build-rpms/rpm/BUILD/origin-3.11.0/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:878 #2811 junit 23 hours ago [Fail] [Feature:DeploymentConfig] deploymentconfigs adoption [Conformance] [It] will orphan all RCs and adopt them back when recreated [Suite:openshift/conformance/parallel/minimal] /tmp/openshift/build-rpms/rpm/BUILD/origin-3.11.0/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:1657 #2810 junit 27 hours ago [Fail] [Feature:DeploymentConfig] deploymentconfigs adoption [Conformance] [It] will orphan all RCs and adopt them back when recreated [Suite:openshift/conformance/parallel/minimal] /tmp/openshift/build-rpms/rpm/BUILD/origin-3.11.0/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:1657