Bug 1443187
| Summary: | Global build default resource requests do not override empty resource config in bc | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Mike Fiedler <mifiedle> | ||||||||||||
| Component: | Build | Assignee: | Ben Parees <bparees> | ||||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Mike Fiedler <mifiedle> | ||||||||||||
| Severity: | high | Docs Contact: | |||||||||||||
| Priority: | unspecified | ||||||||||||||
| Version: | 3.5.0 | CC: | aos-bugs, bparees, haowang, jrosenta, mifiedle, smunilla, vlaad, xiuwang | ||||||||||||
| Target Milestone: | --- | ||||||||||||||
| Target Release: | --- | ||||||||||||||
| Hardware: | x86_64 | ||||||||||||||
| OS: | Linux | ||||||||||||||
| Whiteboard: | |||||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||||
| Doc Text: |
Cause: When resources were specified in the build default configuration, the resource values were not applied to the build pods, they were only applied to the build object.
Consequence: Builds ran without the default resource limits being applied to them because the pod was created before the build was updated with the default resource limits.
Fix: The build resource defaults will be applied to the build pod.
Result: Build pods will have the default resource limits applied, if they do not already specify resource limits.
|
Story Points: | --- | ||||||||||||
| Clone Of: | Environment: | ||||||||||||||
| Last Closed: | 2017-08-10 05:20:02 UTC | Type: | Bug | ||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||
| Documentation: | --- | CRM: | |||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||
| Embargoed: | |||||||||||||||
| Attachments: |
|
||||||||||||||
Created attachment 1272390 [details]
Build config
can you provide master logs w/ loglevel 5 enabled? in particular we're looking for this output: "Initialized build defaults plugin with config <config>" "Setting default resource limit" "Setting default resource request" Created attachment 1272617 [details]
Master logs
- set master-controllers and master-api loglevel to 5
- restarted all 3 masters
- collected logs
I don't see the messages you are looking for though. My build nodeSelector and toleration annotation (added since original master-config attached) are being honored, though.
@vlaad took a look at my setup and we compared logfiles he collected at loglevel 5 with mine. Mine are definitely missing the JSON snipped with the build config. The only difference we can see is that he had a single master (combined api/controller) whereas mine are split services on 3 masters. I see this in your log: Apr 19 09:14:43 svt-m-3 journal: Suppressed 1681 messages from /system.slice/atomic-openshift-master-controllers.service is it possible systemd is throwing away log messages on us? Can you log direct to disk instead? I was able to capture a log with the messages. Attaching.
Apr 19 10:33:55 svt-m-1 atomic-openshift-master-controllers: I0419 10:33:55.828105 11798 admission.go:29] Initialized build defaults plugin with config: api.BuildDefaultsConfig{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, GitHTTPProxy:"", GitHTTPSProxy:"", GitNoProxy:"", Env:[]api.EnvVar(nil), SourceStrategyDefaults:(*api.SourceStrategyDefaultsConfig)(nil), ImageLabels:[]api.ImageLabel(nil), NodeSelector:map[string]string{"builder":"yes", "region":"primary"}, Annotations:map[string]string{"scheduler.alpha.kubernetes.io/tolerations":"[{\"key\":\"builder\", \"value\":\"yes\"}]"}, Resources:api.ResourceRequirements{Limits:api.ResourceList(nil), Requests:api.ResourceList{"memory":resource.Quantity{i:resource.int64Amount{value:268435456, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}, "cpu":resource.Quantity{i:resource.int64Amount{value:150, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"150m", Format:"DecimalSI"}}}}
Created attachment 1272652 [details]
Master at loglevel 5 showing BuildDefaultsConfig
Created attachment 1272665 [details]
New master logs with rate limiting disabled
Fresh logs with rsyslog and journald rate limiting disabled. master3 is the controller and has the BuildDefaultsConfig messages. I do not see "applying defaults" in any of the logs
thanks I think i see what is happening here. we are applying the defaults to the build object but we need to also apply them to the pod object at this point in the flow. Global build default resources still don't override empty resource config in bc Tried in below envs, both don't work. Server https://host-8-174-95.host.centralci.eng.rdu2.redhat.com:8443 openshift v3.6.62 kubernetes v1.6.1+5115d708d7 and openshift origin env devenv-rhel7-6193 > Global build default resources still don't override empty resource config in bc
how did you validate it? The global build default resource values will not be seen in the BC, nor in the build object. The only way to confirm they were applied is to look at the resource constraints of the build pod itself, or look at build json definition in the BUILD env variable defined on the build pod.
One way to verify this is to trigger a build (or a bunch of builds) and then oc describe the node the build pod is running on. That should show the cpu and memory requests for the build pod. You can assign this bz to me for QA if you wish. I had checked the build pod using #oc get po build-pod -o yaml | grep -i resource -A 3 The result please see [1] resources field in the BUILD env variable is empty. [1] http://pastebin.test.redhat.com/481015 @Mike So sweet,thank for your support. I will assign this to you. Also I will keep an eye on this in case you are busy. can you provide the full content of your master-config.yaml? alternatively can you add some global build default env variables to your config, so we can confirm that the build defaults config is otherwise being picked up and applied? (again you won't see the build env variables on the build, but you will see them in the build definition within the BUILD env of the build pod) level 5 logs from the master would also be useful. XiuJuan, i'd still like to understand why it does not appear to be working in your environment so we can sort out any test configuration issues or doc problems, please see my comment 18 above. Created attachment 1276092 [details]
Master Config
I set a env TEST=new in the build default config. The env has set in the BUILD env of the build pod, you can refer the attachment. it looks like there may have been a regression in this area since my other fix, the new issue is described here: https://github.com/openshift/origin/issues/14039 i'll work on a fix. Can't reproduce this bug at Server https://ip-172-18-6-64.ec2.internal:8443 openshift v3.6.74 kubernetes v1.6.1+5115d708d7 Global build default resources override empty resource config in bc Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1716 |
Description of problem: When a buildconfig is created from a template, it is created with a resource request/limit of: resources: {} Trying to override this in the global build default section of master-config.yaml with the following (and restarting all masters): BuildDefaults: configuration: apiVersion: v1 env: [] kind: BuildDefaultsConfig nodeSelector: builder: "yes" region: "primary" resources: limits: {} requests: cpu: "150m" memory: "256Mi" does not work. When the build runs, the nodeSelector is honored but the resource request is not honored. The build pod gets a CPU request of 0 and a memory request of 0. If the request values are specified in the build config, then they are honored and the build pod gets the appropriate request values. Full bc.yaml and master-config.yaml attached. Version-Release number of selected component (if applicable): 3.5.5 How reproducible: Always Steps to Reproduce: 1. Create a new project and a new app from cakephp-mysql-example 2. Edit master-config.yaml on all masters and set build defaults for resources to: resources: limits: {} requests: cpu: "150m" memory: "256Mi" 3. Restart all masters 4. Start a new build of the build config 5. oc get pods -o wide to display the build node 6. oc describe node <node> to view the requests for the build pod Actual results: Requests for the build pod are 0. If the requests are added to the bc, they will be honored. Expected results: Build defaults requests override empty or absent bc requests. Additional info: