Bug 1506073 - when a user has an explicit value for cpu_limit that is lower than the default for the cpu_request, we should be setting the cpu_request to be the same value as the cpu_limit
Summary: when a user has an explicit value for cpu_limit that is lower than the defaul...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
high
low
Target Milestone: ---
: 3.7.0
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-25 03:47 UTC by Peter Ruan
Modified: 2017-12-18 13:22 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Ansible role accepted cpu request and limit without additional evaluation Consequence: Creating resource would fail if request was greater then limit Fix: Evaluate the facts, inventory, and defaults and lower the request to the limit value if needed Result: Resource is created
Clone Of:
Environment:
Last Closed: 2017-12-18 13:22:48 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift openshift-ansible pull 5897 0 'None' closed bug 1506073. Lower cpu request for logging when it exceeds limit 2021-02-17 20:45:36 UTC
Red Hat Product Errata RHBA-2017:3464 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.7 bug fix and enhancement update 2017-12-18 18:22:05 UTC

Description Peter Ruan 2017-10-25 03:47:50 UTC
Description of problem:
when a user has an explicit value for cpu_limit that is lower than the default for the cpu_request, we should be setting the cpu_request to be the same value as the cpu_limit instead of using the default value.

Version-Release number of selected component (if applicable):


How reproducible:
alway.  

Steps to Reproduce:
1. in the inventory file add these entries w/o specifing 'openshift_logging_elasticsearch_cpu_request'


openshift_logging_elasticsearch_cpu_limit=300m
openshift_logging_elasticsearch_memory_limit=512Mi
openshift_logging_es_cpu_limit=300m
openshift_logging_es_ops_cpu_limit=300m
openshift_logging_es_memory_limit=512Mi
openshift_logging_es_ops_memory_limit=512Mi
openshift_logging_kibana_memory_limit=512Mi
openshift_logging_kibana_ops_memory_limit=512Mi

2. run ansible installation.

Actual results:

TASK [openshift_logging_elasticsearch : Set ES dc] *****************************
      task path: /usr/share/ansible/openshift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:365
      Tuesday 24 October 2017  20:46:59 +0000 (0:00:01.916)       0:01:22.022 ******* 
      fatal: [host-8-241-44.host.centralci.eng.rdu2.redhat.com]: FAILED! => {"changed": false, "failed": true, "msg": {"cmd": "/usr/bin/oc create -f /tmp/openshift-logging-ansible-HRXHT5/templates/logging-es-dc.yml -n obhlh", "results": {}, "returncode": 1, "stderr": "The DeploymentConfig \"logging-es-data-master-kj9rl9yo\" is invalid: spec.template.spec.containers[1].resources.limits: Invalid value: \"300m\": must be greater than or equal to cpu request\n", "stdout": ""}}


Expected results:
 installation successful

Additional info:

Comment 1 Jeff Cantrill 2017-10-25 15:00:58 UTC
This mitigated by https://bugzilla.redhat.com/show_bug.cgi?id=1501960 which introduced https://github.com/openshift/openshift-ansible/pull/5748 to remove cpu_limits.

Can you also please specify a specific version of ansible and logging so we may identify where you are seeing the issue.  The referenced mitigation was backported to 3.6 as well

Comment 2 Peter Ruan 2017-10-25 17:57:27 UTC
Sure,

ansible-2.4.0.0-1.fc25.noarch

kibana-4.6.4-3.el7.x86_64.rpm

Comment 3 Jeff Cantrill 2017-10-25 19:53:41 UTC
I believe this issue is related to improperly handling of the cpu_limits, possibly releated to the PR referenced about.  The work around it to set your cpu limit to be greater then your request.

Comment 4 Peter Ruan 2017-10-26 16:29:54 UTC
@Jeff, yes, you are correct, Peter P. is aware of this issue and he pointed out the same solution you did (he asked me to file a bug).  My test passed after I update my inventory to include the parameter xxx_cpu_request like the following

openshift_logging_use_ops=true
openshift_logging_elasticsearch_cpu_limit=300m
openshift_logging_elasticsearch_cpu_request=300m
openshift_logging_elasticsearch_memory_limit=512Mi
openshift_logging_es_cpu_limit=300m
openshift_logging_es_cpu_request=300m
openshift_logging_es_ops_cpu_limit=300m
openshift_logging_es_ops_cpu_request=300m
openshift_logging_es_memory_limit=512Mi
openshift_logging_es_ops_memory_limit=512Mi
openshift_logging_kibana_memory_limit=512Mi
openshift_logging_kibana_ops_memory_limit=512Mi

Comment 5 openshift-github-bot 2017-10-27 22:11:42 UTC
Commits pushed to master at https://github.com/openshift/openshift-ansible

https://github.com/openshift/openshift-ansible/commit/205d03455cf37d66f3629de7951463d755b72b16
bug 1506073. Lower cpu request for logging when it exceeds limit

https://github.com/openshift/openshift-ansible/commit/2c81304511afcf03b35583f85040fb7d0b83cad5
Merge pull request #5897 from jcantrill/1506073_cpu_request_match_limit

Automatic merge from submit-queue.

bug 1506073. Lower cpu request for logging when it exceeds limit

This PR fixes https://bugzilla.redhat.com/show_bug.cgi?id=1506073 by:
* Lowering the CPU request to match the limit when the request is greater then a specified limit

I have an open question on if this is an acceptable change of if it makes the outcome unexpected.  Should we prefer to exit during the deployment and advise the operator to correct their inventory?

Comment 6 Peter Ruan 2017-10-30 20:23:13 UTC
@Jeff, before the bz 1506073 was introduced, we don't need to worry about setting this parameter in the inventory.  Therefore, my preference is to have the system given a warning that it's changing the _request parameter instead of exit deployment.  Otherwise the user will need to update the inventory file(s) to include this parameter specifically.

Comment 7 Anping Li 2017-11-10 05:28:01 UTC
Verified and pass with openshift-ansible-3.7.4-1. the resource are set as openshift_logging_es_cpu_limit when cpu_request > cpu_limit

1) inventory variable
openshift_logging_es_cpu_limit=200m
openshift_logging_es_cpu_request=300m

2) the resource value in DC.
        resources:
          limits:
            cpu: 200m
            memory: 1Gi
          requests:
            cpu: 200m
            memory: 1Gi

Comment 10 errata-xmlrpc 2017-12-18 13:22:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3464


Note You need to log in before you can comment on or make changes to this bug.