Bug 1788935 - Test "[Feature:Builds][Conformance] s2i build with a quota Building from a template should create an s2i build with a quota and run it" fails on OKD and RHEL7 workers
Summary: Test "[Feature:Builds][Conformance] s2i build with a quota Building from a te...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 4.4
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
: 4.4.0
Assignee: Peter Hunt
QA Contact: weiwei jiang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-01-08 12:04 UTC by Vadim Rutkovsky
Modified: 2020-05-04 11:23 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-04 11:23:07 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1764323 0 unspecified CLOSED "s2i build with a quota Building from a template" test is failing with cri-o 1.16 and OpenShift 4.3 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-04 11:23:40 UTC

Description Vadim Rutkovsky 2020-01-08 12:04:53 UTC
Description of problem:

See 
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_installer/2822/pull-ci-openshift-installer-master-e2e-aws-scaleup-rhel7/3718

and

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.4/203

This test fails as MEMORY is expected to be "419430400", but its actually

MEMORY=9223372036854771712.

This seems to happen often on RHEL7 workers (in scaleup-rhel7 job) and on FCOS (openshift-origin-installer job), so I assume there is some RHEL8/RHCOS-specific tweak which doesn't get applied there?

Comment 1 Vadim Rutkovsky 2020-01-08 13:01:00 UTC
Seems that its upstream issue https://github.com/cri-o/cri-o/issues/2996, fixed by https://github.com/cri-o/cri-o/pull/2997

4.3 RHEL7 workers pick up fixed cri-o version [1] - cri-o-1.16.2-3.dev.rhaos4.3.gitcccde39.el7.x86_64, 
but 4.4 nightlies still use older version [2] - cri-o.x86_64 0:1.16.1-3.dev.rhaos4.3.git7b04b62.el7

Perhaps the package didn't get necessary 4.4 tags?

[1] https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-openshift-ansible-e2e-aws-scaleup-rhel7-4.3/407
[2] https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_installer/2822/pull-ci-openshift-installer-master-e2e-aws-scaleup-rhel7/3718

Comment 2 Peter Hunt 2020-02-11 13:56:12 UTC
This will also be fixed once 1.17 is picked up for 4.4. in the meantime, Jindrich can you PTAL?

Comment 3 Vadim Rutkovsky 2020-02-11 14:39:23 UTC
OKD pulls in crio 1.17 rc0 (and soon released version), this indeed makes the test pass. Not sure if it currently affects OCP 4.4

Comment 4 Jindrich Novy 2020-02-12 16:25:17 UTC
jnovy@localhost .../cri-o/rhaos-4.4-rhel-8 (rhaos-4.4-rhel-8)$ latestpkg
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
cri-o-1.17.0-1.dev.rhaos4.4.git3afd333.el8  rhaos-4.4-rhel-8-candidate  lmandvek

Lokesh, PTAL at comment #1 - assuming we're good here? Assuming builds off rhaos-4.4-rhel-8 get automatically tagged correctly for 4.4 nightlies?

Comment 5 Lokesh Mandvekar 2020-02-12 17:09:08 UTC
(In reply to Jindrich Novy from comment #4)
> jnovy@localhost .../cri-o/rhaos-4.4-rhel-8 (rhaos-4.4-rhel-8)$ latestpkg
> Build                                     Tag                   Built by
> ----------------------------------------  -------------------- 
> ----------------
> cri-o-1.17.0-1.dev.rhaos4.4.git3afd333.el8  rhaos-4.4-rhel-8-candidate 
> lmandvek
> 
> Lokesh, PTAL at comment #1 - assuming we're good here? Assuming builds off
> rhaos-4.4-rhel-8 get automatically tagged correctly for 4.4 nightlies?

yes, that's right.

Comment 8 Vadim Rutkovsky 2020-02-25 23:55:33 UTC
Test no longer fails when cri-o 1.17 is used

Comment 9 weiwei jiang 2020-02-28 06:03:29 UTC
Checked with 4.4.0-0.nightly-2020-02-28-000349

$ oc get nodes -o wide
NAME                             STATUS   ROLES    AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                CONTAINER-RUNTIME
wjos16i2275-zwmbd-master-0       Ready    master   133m   v1.17.1   192.168.2.160   <none>        Red Hat Enterprise Linux CoreOS 44.81.202002271930-0 (Ootpa)   4.18.0-147.5.1.el8_1.x86_64   cri-o://1.17.0-4.dev.rhaos4.4.gitc3436cc.el8
wjos16i2275-zwmbd-master-1       Ready    master   133m   v1.17.1   192.168.2.50    <none>        Red Hat Enterprise Linux CoreOS 44.81.202002271930-0 (Ootpa)   4.18.0-147.5.1.el8_1.x86_64   cri-o://1.17.0-4.dev.rhaos4.4.gitc3436cc.el8
wjos16i2275-zwmbd-master-2       Ready    master   133m   v1.17.1   192.168.2.21    <none>        Red Hat Enterprise Linux CoreOS 44.81.202002271930-0 (Ootpa)   4.18.0-147.5.1.el8_1.x86_64   cri-o://1.17.0-4.dev.rhaos4.4.gitc3436cc.el8
wjos16i2275-zwmbd-worker-fz2zl   Ready    worker   123m   v1.17.1   192.168.3.62    <none>        Red Hat Enterprise Linux CoreOS 44.81.202002271930-0 (Ootpa)   4.18.0-147.5.1.el8_1.x86_64   cri-o://1.17.0-4.dev.rhaos4.4.gitc3436cc.el8
wjos16i2275-zwmbd-worker-mkn87   Ready    worker   125m   v1.17.1   192.168.2.232   <none>        Red Hat Enterprise Linux CoreOS 44.81.202002271930-0 (Ootpa)   4.18.0-147.5.1.el8_1.x86_64   cri-o://1.17.0-4.dev.rhaos4.4.gitc3436cc.el8
wjos16i2275-zwmbd-worker-sxhbs   Ready    worker   125m   v1.17.1   192.168.1.10    <none>        Red Hat Enterprise Linux CoreOS 44.81.202002271930-0 (Ootpa)   4.18.0-147.5.1.el8_1.x86_64   cri-o://1.17.0-4.dev.rhaos4.4.gitc3436cc.el8

 $ _output/local/bin/linux/amd64/openshift-tests run-test "[Feature:Builds][Conformance] s2i build with a quota  Building from a template [Top Level] [Feature:Builds][Conformance] s2i build with a quota  Building from a template should create an s2i build with a quota and run it [Suite:openshift/conformance/parallel/minimal]"                               
I0228 13:57:42.240480  788732 test_context.go:419] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready                                                                                                                                                                                                                                                    
Feb 28 13:57:42.298: INFO: Waiting up to 30m0s for all (but 100) nodes to be schedulable                                                                                                                                                                                                                                                                                                     
Feb 28 13:57:42.898: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready                                                                                                                                                                                                                                                             
Feb 28 13:57:43.814: INFO: 0 / 0 pods in namespace 'kube-system' are running and ready (0 seconds elapsed)                                                                             
Feb 28 13:57:43.814: INFO: expected 0 pod replicas in namespace 'kube-system', 0 are Running and Ready.                                                                                                                                                                                                                                                                                      
Feb 28 13:57:43.814: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start                                                                                                                                                                                                                                                                                         
Feb 28 13:57:44.118: INFO: e2e test version: v1.17.1                                                                                                                                                                                                                                                                                                                                         
Feb 28 13:57:44.415: INFO: kube-apiserver version: v1.17.1                                                                                                                                                                                                                                                                                                                                   
Feb 28 13:57:44.717: INFO: Cluster IP family: ipv4                                                                                                                                                                                                                                                                                                                                           
[BeforeEach] [Top Level]                                                                                                                                                                                                                                                                                                                                                                     
  /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:57                                                       
[BeforeEach] [Feature:Builds][Conformance] s2i build with a quota                                                                                                                                                                                                                                                                                                                            
  /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:154                                                                                                                                                                                                                              
STEP: Creating a kubernetes client                                                                                                                                                                                                                                                                                                                                                           
[BeforeEach] [Feature:Builds][Conformance] s2i build with a quota                                                                                                                                                                                                                                                                                                                            
  /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/client.go:109                                                                                                                                                                                                                                                          
Feb 28 13:57:47.249: INFO: configPath is now "/tmp/configfile012251761"                                                                                                                                                                                                                                                                                                                      
Feb 28 13:57:47.249: INFO: The user is now "e2e-test-s2i-build-quota-b58pd-user"                                                                                                                                                                                                                                                                                                             
Feb 28 13:57:47.249: INFO: Creating project "e2e-test-s2i-build-quota-b58pd"                                                                                                                                                                                                                                                                                                                 
Feb 28 13:57:47.755: INFO: Waiting on permissions in project "e2e-test-s2i-build-quota-b58pd" ...                                                                                      
Feb 28 13:57:48.054: INFO: Waiting for ServiceAccount "default" to be provisioned...                                                                        
Feb 28 13:57:48.455: INFO: Waiting for ServiceAccount "deployer" to be provisioned...                                                                                                                                                                                                                                                                                                        
Feb 28 13:57:48.856: INFO: Waiting for ServiceAccount "builder" to be provisioned...                                                                                                                                                                                                                                                                                                         
Feb 28 13:57:49.258: INFO: Waiting for RoleBinding "system:image-pullers" to be provisioned...                                                                                                
Feb 28 13:57:49.862: INFO: Waiting for RoleBinding "system:image-builders" to be provisioned...                                                                                               
Feb 28 13:57:50.458: INFO: Waiting for RoleBinding "system:deployers" to be provisioned...                                                                                                    
Feb 28 13:57:51.057: INFO: Project "e2e-test-s2i-build-quota-b58pd" has been fully provisioned.                                                                                               
[BeforeEach]                                                                                   
  /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/builds/s2i_quota.go:28                                                       
[It] [Top Level] [Feature:Builds][Conformance] s2i build with a quota  Building from a template should create an s2i build with a quota and run it [Suite:openshift/conformance/parallel/minimal]
  /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/builds/s2i_quota.go:41                                                       
STEP: calling oc create -f "/tmp/fixture-testdata-dir225897178/test/extended/testdata/builds/test-s2i-build-quota.json"                                                                       
Feb 28 13:57:51.057: INFO: Running 'oc --namespace=e2e-test-s2i-build-quota-b58pd --kubeconfig=/tmp/configfile012251761 create -f /tmp/fixture-testdata-dir225897178/test/extended/testdata/builds/test-s2i-build-quota.json'
buildconfig.build.openshift.io/s2i-build-quota created                                                                                                                                        
STEP: starting a test build                                                                    
Feb 28 13:57:53.724: INFO: Running 'oc --namespace=e2e-test-s2i-build-quota-b58pd --kubeconfig=/tmp/configfile012251761 start-build s2i-build-quota --from-dir /tmp/fixture-testdata-dir225897178/test/extended/testdata/builds/build-quota -o=name'
Feb 28 13:58:08.967: INFO:                                                                     

start-build output with args [s2i-build-quota --from-dir /tmp/fixture-testdata-dir225897178/test/extended/testdata/builds/build-quota -o=name]:                                               
Error><nil>                                                                                    
StdOut>                                                                                        
build.build.openshift.io/s2i-build-quota-1                                                     
StdErr>                                                                                        
Uploading directory "/tmp/fixture-testdata-dir225897178/test/extended/testdata/builds/build-quota" as binary input for the build ...                                                          
...                                                                                            
Uploading finished                                                                             


Feb 28 13:58:08.968: INFO: Waiting for s2i-build-quota-1 to complete                                                                                                                          

Feb 28 13:59:25.585: INFO: Done waiting for s2i-build-quota-1: util.BuildResult{BuildPath:"build.build.openshift.io/s2i-build-quota-1", BuildName:"s2i-build-quota-1", StartBuildStdErr:"Uploading directory \"/tmp/fixture-testdata-dir225897178/test/extended/testdata/builds/build-quota\" as binary input for the build ...\n...\nUploading finished", StartBuildStdOut:"build.build.open
shift.io/s2i-build-quota-1", StartBuildErr:error(nil), BuildConfigName:"", Build:(*v1.Build)(0xc000d9d500), BuildAttempt:true, BuildSuccess:true, BuildFailure:false, BuildCancelled:false, BuildTimeout:false, LogDumper:(util.LogDumperFunc)(nil), Oc:(*util.CLI)(0xc000cd7000)}
 with error: <nil>                                                                             

STEP: expecting the build logs to contain the correct cgroups values                                                                                                                          
Feb 28 13:59:25.585: INFO: Running 'oc --namespace=e2e-test-s2i-build-quota-b58pd --kubeconfig=/tmp/configfile012251761 logs -f build.build.openshift.io/s2i-build-quota-1'                   
Feb 28 13:59:29.480: INFO: Found event v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"s2i-build-quota-1.15f77b6986b426bd", GenerateName:"", Namespace:"e2e-test-s2i-build-quota-b58pd", SelfLink:"/api/v1/namespaces/e2e-test-s2i-build-quota-b58pd/events/s2i-build-quota-1.15f77b6986b426bd", UID:"c219cb61-2fbb-4e85-bc1e-916afd7e1c0e", ResourceVe
rsion:"52742", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466287, loc:(*time.Location)(0x97203e0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.Ma
nagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Build", Namespace:"e2e-test-s2i-build-quota-b58pd", Name:"s2i-build-quota-1", UID:"2246caca-271f-47c0-bdc7-896a2ae56ed2", APIVersion:"build.openshift.io/v1", ResourceVersion:"52741", FieldPath:""}, Reason:"BuildStarted", Message:"Build e2e-test-s2i-build-quota-b58pd/s2i-build-quota-1 is now running", Source:v1.Event
Source{Component:"build-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466287, loc:(*time.Location)(0x97203e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466287, loc:(*time.Location)(0x97203e0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(ni
l), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}                                                                                              
Feb 28 13:59:29.782: INFO: Found event v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"s2i-build-quota-1.15f77b6986b426bd", GenerateName:"", Namespace:"e2e-test-s2i-build-quota-b58pd", SelfLink:"/api/v1/namespaces/e2e-test-s2i-build-quota-b58pd/events/s2i-build-quota-1.15f77b6986b426bd", UID:"c219cb61-2fbb-4e85-bc1e-916afd7e1c0e", ResourceVe
rsion:"52742", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466287, loc:(*time.Location)(0x97203e0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.Ma
nagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Build", Namespace:"e2e-test-s2i-build-quota-b58pd", Name:"s2i-build-quota-1", UID:"2246caca-271f-47c0-bdc7-896a2ae56ed2", APIVersion:"build.openshift.io/v1", ResourceVersion:"52741", FieldPath:""}, Reason:"BuildStarted", Message:"Build e2e-test-s2i-build-quota-b58pd/s2i-build-quota-1 is now running", Source:v1.Event
Source{Component:"build-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466287, loc:(*time.Location)(0x97203e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466287, loc:(*time.Location)(0x97203e0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(ni
l), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}                                                                                              
Feb 28 13:59:29.782: INFO: Found event v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"s2i-build-quota-1.15f77b7b05f936e5", GenerateName:"", Namespace:"e2e-test-s2i-build-quota-b58pd", SelfLink:"/api/v1/namespaces/e2e-test-s2i-build-quota-b58pd/events/s2i-build-quota-1.15f77b7b05f936e5", UID:"2fabc9c8-9748-40b7-9bbb-bc41c432a547", ResourceVe
rsion:"53144", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466362, loc:(*time.Location)(0x97203e0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.Ma
nagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Build", Namespace:"e2e-test-s2i-build-quota-b58pd", Name:"s2i-build-quota-1", UID:"2246caca-271f-47c0-bdc7-896a2ae56ed2", APIVersion:"build.openshift.io/v1", ResourceVersion:"53141", FieldPath:""}, Reason:"BuildCompleted", Message:"Build e2e-test-s2i-build-quota-b58pd/s2i-build-quota-1 completed successfully", Sourc
e:v1.EventSource{Component:"build-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466362, loc:(*time.Location)(0x97203e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63718466362, loc:(*time.Location)(0x97203e0)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.Event
Series)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}                                                                                    
[AfterEach]                                                                                    
  /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/builds/s2i_quota.go:32                                                       
[AfterEach] [Feature:Builds][Conformance] s2i build with a quota                                                                                                                              
  /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/client.go:101                                                           
Feb 28 13:59:30.766: INFO: Deleted {user.openshift.io/v1, Resource=users  e2e-test-s2i-build-quota-b58pd-user}, err: <nil>                                                                    
Feb 28 13:59:31.071: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthclients  e2e-client-e2e-test-s2i-build-quota-b58pd}, err: <nil>                                                      
Feb 28 13:59:31.375: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthaccesstokens  h28Glxv5Sk6jJnYjWvG17wAAAAAAAAAA}, err: <nil>                                                          
[AfterEach] [Feature:Builds][Conformance] s2i build with a quota                                                                                                                              
  /home/wjiang/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:155                               
Feb 28 13:59:31.375: INFO: Waiting up to 7m0s for all (but 100) nodes to be ready                                                                                                             
STEP: Destroying namespace "e2e-test-s2i-build-quota-b58pd" for this suite.                                                                                                                   
Feb 28 13:59:32.585: INFO: Running AfterSuite actions on all nodes                                                                                                                            
Feb 28 13:59:32.585: INFO: Running AfterSuite actions on node 1

Comment 12 errata-xmlrpc 2020-05-04 11:23:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.