Description of problem: See https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_installer/2822/pull-ci-openshift-installer-master-e2e-aws-scaleup-rhel7/3718 and https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.4/203 This test fails as MEMORY is expected to be "419430400", but its actually MEMORY=9223372036854771712. This seems to happen often on RHEL7 workers (in scaleup-rhel7 job) and on FCOS (openshift-origin-installer job), so I assume there is some RHEL8/RHCOS-specific tweak which doesn't get applied there?
Seems that its upstream issue https://github.com/cri-o/cri-o/issues/2996, fixed by https://github.com/cri-o/cri-o/pull/2997 4.3 RHEL7 workers pick up fixed cri-o version [1] - cri-o-1.16.2-3.dev.rhaos4.3.gitcccde39.el7.x86_64, but 4.4 nightlies still use older version [2] - cri-o.x86_64 0:1.16.1-3.dev.rhaos4.3.git7b04b62.el7 Perhaps the package didn't get necessary 4.4 tags? [1] https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-openshift-ansible-e2e-aws-scaleup-rhel7-4.3/407 [2] https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_installer/2822/pull-ci-openshift-installer-master-e2e-aws-scaleup-rhel7/3718
This will also be fixed once 1.17 is picked up for 4.4. in the meantime, Jindrich can you PTAL?
OKD pulls in crio 1.17 rc0 (and soon released version), this indeed makes the test pass. Not sure if it currently affects OCP 4.4
jnovy@localhost .../cri-o/rhaos-4.4-rhel-8 (rhaos-4.4-rhel-8)$ latestpkg Build Tag Built by ---------------------------------------- -------------------- ---------------- cri-o-1.17.0-1.dev.rhaos4.4.git3afd333.el8 rhaos-4.4-rhel-8-candidate lmandvek Lokesh, PTAL at comment #1 - assuming we're good here? Assuming builds off rhaos-4.4-rhel-8 get automatically tagged correctly for 4.4 nightlies?
(In reply to Jindrich Novy from comment #4) > jnovy@localhost .../cri-o/rhaos-4.4-rhel-8 (rhaos-4.4-rhel-8)$ latestpkg > Build Tag Built by > ---------------------------------------- -------------------- > ---------------- > cri-o-1.17.0-1.dev.rhaos4.4.git3afd333.el8 rhaos-4.4-rhel-8-candidate > lmandvek > > Lokesh, PTAL at comment #1 - assuming we're good here? Assuming builds off > rhaos-4.4-rhel-8 get automatically tagged correctly for 4.4 nightlies? yes, that's right.
Test no longer fails when cri-o 1.17 is used
Checked with 4.4.0-0.nightly-2020-02-28-000349 $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME wjos16i2275-zwmbd-master-0 Ready master 133m v1.17.1 192.168.2.160 <none> Red Hat Enterprise Linux CoreOS 44.81.202002271930-0 (Ootpa) 4.18.0-147.5.1.el8_1.x86_64 cri-o://1.17.0-4.dev.rhaos4.4.gitc3436cc.el8 wjos16i2275-zwmbd-master-1 Ready master 133m v1.17.1 192.168.2.50 <none> Red Hat Enterprise Linux CoreOS 44.81.202002271930-0 (Ootpa) 4.18.0-147.5.1.el8_1.x86_64 cri-o://1.17.0-4.dev.rhaos4.4.gitc3436cc.el8 wjos16i2275-zwmbd-master-2 Ready master 133m v1.17.1 192.168.2.21 <none> Red Hat Enterprise Linux CoreOS 44.81.202002271930-0 (Ootpa) 4.18.0-147.5.1.el8_1.x86_64 cri-o://1.17.0-4.dev.rhaos4.4.gitc3436cc.el8 wjos16i2275-zwmbd-worker-fz2zl Ready worker 123m v1.17.1 192.168.3.62 <none> Red Hat Enterprise Linux CoreOS 44.81.202002271930-0 (Ootpa) 4.18.0-147.5.1.el8_1.x86_64 cri-o://1.17.0-4.dev.rhaos4.4.gitc3436cc.el8 wjos16i2275-zwmbd-worker-mkn87 Ready worker 125m v1.17.1 192.168.2.232 <none> Red Hat Enterprise Linux CoreOS 44.81.202002271930-0 (Ootpa) 4.18.0-147.5.1.el8_1.x86_64 cri-o://1.17.0-4.dev.rhaos4.4.gitc3436cc.el8 wjos16i2275-zwmbd-worker-sxhbs Ready worker 125m v1.17.1 192.168.1.10 <none> Red Hat Enterprise Linux CoreOS 44.81.202002271930-0 (Ootpa) 4.18.0-147.5.1.el8_1.x86_64 cri-o://1.17.0-4.dev.rhaos4.4.gitc3436cc.el8 $ _output/local/bin/linux/amd64/openshift-tests run-test "[Feature:Builds][Conformance] s2i build with a quota Building from a template [Top Level] [Feature:Builds][Conformance] s2i build with a quota Building from a template should create an s2i build with a quota and run it [Suite:openshift/conformance/parallel/minimal]" I0228 13:57:42.240480 788732 test_context.go:419] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready Feb 28 13:57:42.298: INFO: Waiting up to 30m0s for all (but 100) nodes to be schedulable Feb 28 13:57:42.898: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready Feb 28 13:57:43.814: INFO: 0 / 0 pods in namespace 'kube-system' are running and ready (0 seconds elapsed) Feb 28 13:57:43.814: INFO: expected 0 pod replicas in namespace 'kube-system', 0 are Running and Ready. Feb 28 13:57:43.814: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start Feb 28 13:57:44.118: INFO: e2e test version: v1.17.1 Feb 28 13:57:44.415: INFO: kube-apiserver version: v1.17.1 Feb 28 13:57:44.717: INFO: Cluster IP family: ipv4 [BeforeEach] [Top Level] /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:57 [BeforeEach] [Feature:Builds][Conformance] s2i build with a quota /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:154 STEP: Creating a kubernetes client [BeforeEach] [Feature:Builds][Conformance] s2i build with a quota /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/client.go:109 Feb 28 13:57:47.249: INFO: configPath is now "/tmp/configfile012251761" Feb 28 13:57:47.249: INFO: The user is now "e2e-test-s2i-build-quota-b58pd-user" Feb 28 13:57:47.249: INFO: Creating project "e2e-test-s2i-build-quota-b58pd" Feb 28 13:57:47.755: INFO: Waiting on permissions in project "e2e-test-s2i-build-quota-b58pd" ... Feb 28 13:57:48.054: INFO: Waiting for ServiceAccount "default" to be provisioned... Feb 28 13:57:48.455: INFO: Waiting for ServiceAccount "deployer" to be provisioned... Feb 28 13:57:48.856: INFO: Waiting for ServiceAccount "builder" to be provisioned... Feb 28 13:57:49.258: INFO: Waiting for RoleBinding "system:image-pullers" to be provisioned... Feb 28 13:57:49.862: INFO: Waiting for RoleBinding "system:image-builders" to be provisioned... Feb 28 13:57:50.458: INFO: Waiting for RoleBinding "system:deployers" to be provisioned... Feb 28 13:57:51.057: INFO: Project "e2e-test-s2i-build-quota-b58pd" has been fully provisioned. [BeforeEach] /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/builds/s2i_quota.go:28 [It] [Top Level] [Feature:Builds][Conformance] s2i build with a quota Building from a template should create an s2i build with a quota and run it [Suite:openshift/conformance/parallel/minimal] /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/builds/s2i_quota.go:41 STEP: calling oc create -f "/tmp/fixture-testdata-dir225897178/test/extended/testdata/builds/test-s2i-build-quota.json" Feb 28 13:57:51.057: INFO: Running 'oc --namespace=e2e-test-s2i-build-quota-b58pd --kubeconfig=/tmp/configfile012251761 create -f /tmp/fixture-testdata-dir225897178/test/extended/testdata/builds/test-s2i-build-quota.json' buildconfig.build.openshift.io/s2i-build-quota created STEP: starting a test build Feb 28 13:57:53.724: INFO: Running 'oc --namespace=e2e-test-s2i-build-quota-b58pd --kubeconfig=/tmp/configfile012251761 start-build s2i-build-quota --from-dir /tmp/fixture-testdata-dir225897178/test/extended/testdata/builds/build-quota -o=name' Feb 28 13:58:08.967: INFO: start-build output with args [s2i-build-quota --from-dir /tmp/fixture-testdata-dir225897178/test/extended/testdata/builds/build-quota -o=name]: Error><nil> StdOut> build.build.openshift.io/s2i-build-quota-1 StdErr> Uploading directory "/tmp/fixture-testdata-dir225897178/test/extended/testdata/builds/build-quota" as binary input for the build ... ... Uploading finished Feb 28 13:58:08.968: INFO: Waiting for s2i-build-quota-1 to complete Feb 28 13:59:25.585: INFO: Done waiting for s2i-build-quota-1: util.BuildResult{BuildPath:"build.build.openshift.io/s2i-build-quota-1", BuildName:"s2i-build-quota-1", StartBuildStdErr:"Uploading directory \"/tmp/fixture-testdata-dir225897178/test/extended/testdata/builds/build-quota\" as binary input for the build ...\n...\nUploading finished", StartBuildStdOut:"build.build.open shift.io/s2i-build-quota-1", StartBuildErr:error(nil), BuildConfigName:"", Build:(*v1.Build)(0xc000d9d500), BuildAttempt:true, BuildSuccess:true, BuildFailure:false, BuildCancelled:false, BuildTimeout:false, LogDumper:(util.LogDumperFunc)(nil), Oc:(*util.CLI)(0xc000cd7000)} with error: <nil> STEP: expecting the build logs to contain the correct cgroups values Feb 28 13:59:25.585: INFO: Running 'oc --namespace=e2e-test-s2i-build-quota-b58pd --kubeconfig=/tmp/configfile012251761 logs -f build.build.openshift.io/s2i-build-quota-1' Feb 28 13:59:29.480: INFO: Found event v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"s2i-build-quota-1.15f77b6986b426bd", GenerateName:"", Namespace:"e2e-test-s2i-build-quota-b58pd", SelfLink:"/api/v1/namespaces/e2e-test-s2i-build-quota-b58pd/events/s2i-build-quota-1.15f77b6986b426bd", UID:"c219cb61-2fbb-4e85-bc1e-916afd7e1c0e", ResourceVe rsion:"52742", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466287, loc:(*time.Location)(0x97203e0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.Ma nagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Build", Namespace:"e2e-test-s2i-build-quota-b58pd", Name:"s2i-build-quota-1", UID:"2246caca-271f-47c0-bdc7-896a2ae56ed2", APIVersion:"build.openshift.io/v1", ResourceVersion:"52741", FieldPath:""}, Reason:"BuildStarted", Message:"Build e2e-test-s2i-build-quota-b58pd/s2i-build-quota-1 is now running", Source:v1.Event Source{Component:"build-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466287, loc:(*time.Location)(0x97203e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466287, loc:(*time.Location)(0x97203e0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(ni l), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""} Feb 28 13:59:29.782: INFO: Found event v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"s2i-build-quota-1.15f77b6986b426bd", GenerateName:"", Namespace:"e2e-test-s2i-build-quota-b58pd", SelfLink:"/api/v1/namespaces/e2e-test-s2i-build-quota-b58pd/events/s2i-build-quota-1.15f77b6986b426bd", UID:"c219cb61-2fbb-4e85-bc1e-916afd7e1c0e", ResourceVe rsion:"52742", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466287, loc:(*time.Location)(0x97203e0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.Ma nagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Build", Namespace:"e2e-test-s2i-build-quota-b58pd", Name:"s2i-build-quota-1", UID:"2246caca-271f-47c0-bdc7-896a2ae56ed2", APIVersion:"build.openshift.io/v1", ResourceVersion:"52741", FieldPath:""}, Reason:"BuildStarted", Message:"Build e2e-test-s2i-build-quota-b58pd/s2i-build-quota-1 is now running", Source:v1.Event Source{Component:"build-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466287, loc:(*time.Location)(0x97203e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466287, loc:(*time.Location)(0x97203e0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(ni l), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""} Feb 28 13:59:29.782: INFO: Found event v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"s2i-build-quota-1.15f77b7b05f936e5", GenerateName:"", Namespace:"e2e-test-s2i-build-quota-b58pd", SelfLink:"/api/v1/namespaces/e2e-test-s2i-build-quota-b58pd/events/s2i-build-quota-1.15f77b7b05f936e5", UID:"2fabc9c8-9748-40b7-9bbb-bc41c432a547", ResourceVe rsion:"53144", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466362, loc:(*time.Location)(0x97203e0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.Ma nagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Build", Namespace:"e2e-test-s2i-build-quota-b58pd", Name:"s2i-build-quota-1", UID:"2246caca-271f-47c0-bdc7-896a2ae56ed2", APIVersion:"build.openshift.io/v1", ResourceVersion:"53141", FieldPath:""}, Reason:"BuildCompleted", Message:"Build e2e-test-s2i-build-quota-b58pd/s2i-build-quota-1 completed successfully", Sourc e:v1.EventSource{Component:"build-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466362, loc:(*time.Location)(0x97203e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466362, loc:(*time.Location)(0x97203e0)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.Event Series)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""} [AfterEach] /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/builds/s2i_quota.go:32 [AfterEach] [Feature:Builds][Conformance] s2i build with a quota /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/client.go:101 Feb 28 13:59:30.766: INFO: Deleted {user.openshift.io/v1, Resource=users e2e-test-s2i-build-quota-b58pd-user}, err: <nil> Feb 28 13:59:31.071: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthclients e2e-client-e2e-test-s2i-build-quota-b58pd}, err: <nil> Feb 28 13:59:31.375: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthaccesstokens h28Glxv5Sk6jJnYjWvG17wAAAAAAAAAA}, err: <nil> [AfterEach] [Feature:Builds][Conformance] s2i build with a quota /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:155 Feb 28 13:59:31.375: INFO: Waiting up to 7m0s for all (but 100) nodes to be ready STEP: Destroying namespace "e2e-test-s2i-build-quota-b58pd" for this suite. Feb 28 13:59:32.585: INFO: Running AfterSuite actions on all nodes Feb 28 13:59:32.585: INFO: Running AfterSuite actions on node 1
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581