Bug 1530125 - AnsibleHelper is consuming too much cpu time
Summary: AnsibleHelper is consuming too much cpu time
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-hosted-engine-setup
Classification: oVirt
Component: Plugins.General
Version: 2.2.1
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ovirt-4.2.1
: 2.2.5
Assignee: Yedidyah Bar David
QA Contact: Nikolai Sednev
URL:
Whiteboard:
Depends On:
Blocks: 1458709
TreeView+ depends on / blocked
 
Reported: 2018-01-02 06:35 UTC by Yedidyah Bar David
Modified: 2018-02-22 10:01 UTC (History)
3 users (show)

Fixed In Version: ovirt-hosted-engine-setup-2.2.5
Clone Of:
Environment:
Last Closed: 2018-02-22 10:01:49 UTC
oVirt Team: Integration
Embargoed:
rule-engine: ovirt-4.2+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 85890 0 master MERGED Make AnsibleHelper sleep between reads 2018-09-03 09:37:05 UTC

Description Yedidyah Bar David 2018-01-02 06:35:55 UTC
Description of problem:

While running ansible, hosted-engine --deploy uses lots of cpu time in the python code, where it's supposed to only wait for ansible.

Most likely this is in src/ovirt_hosted_engine_setup/ansible_utils.py:

        with open(out_path, 'r') as out_fh:
            buffer = ''
            proc = subprocess.Popen(
                ansible_playbook_cmd,
                env=env,
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
            )
            while True:
                output = out_fh.readline()

We should replace readline with something more efficient.

Comment 1 Yaniv Kaul 2018-01-02 07:05:34 UTC
(In reply to Yedidyah Bar David from comment #0)
> Description of problem:
> 
> While running ansible, hosted-engine --deploy uses lots of cpu time in the
> python code, where it's supposed to only wait for ansible.
> 
> Most likely this is in src/ovirt_hosted_engine_setup/ansible_utils.py:
> 
>         with open(out_path, 'r') as out_fh:
>             buffer = ''
>             proc = subprocess.Popen(
>                 ansible_playbook_cmd,
>                 env=env,
>                 stdout=subprocess.PIPE,
>                 stderr=subprocess.PIPE,
>             )
>             while True:
>                 output = out_fh.readline()
> 
> We should replace readline with something more efficient.

Is readline() the issue or the fact we don't have a sleep somewhere in between? From a bit of Googling, seems like the latter? perhaps even time.sleep(0.1) would suffice.
As is, it's consuming more CPU than the VM itself. I doubt it's vastly slowing the process, as a host has other CPUs to do the rest of the work.

Comment 2 Yedidyah Bar David 2018-01-02 07:22:28 UTC
sleep(0.1) will certainly help, but I am not sure it's the best solution.

Comment 3 Nikolai Sednev 2018-02-12 15:11:49 UTC
Please provide detailed steps for reproduction.
Its unclear what should be done here.

Comment 4 Yedidyah Bar David 2018-02-13 07:06:32 UTC
Reproduction/Verification:

Run hosted-engine --deploy --ansible

Reply to all questions, reach the point in which it runs ansible. You will see in the console 'TASK...'.

On the same host, run 'top' (or some more advanced utils if you wish).

With a broken version, hosted-engine process (might show as python or otopi) will use close to %100 CPU time (assuming you have more than one core).

With a fixed version, it will use much less CPU time.

For extra points, patch ansible_utils.py (should be in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup ), changing 'time.sleep(0.1)' to other values - can be lower or higher - and find the best one. Higher numbers will use less CPU time, but will cause the output to lag a bit behind and seem "choppy". Lower numbers will use more CPU time and output will seem more "responsive". If you do this, please send a patch to change it :-) (or at least comment here). Thanks.

Comment 5 Nikolai Sednev 2018-02-13 14:28:05 UTC
Failed to reproduce on these components:
rhvm-appliance-4.2-20180202.0.el7.noarch
ovirt-hosted-engine-ha-2.2.4-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.9-1.el7ev.noarch
Linux 3.10.0-693.17.1.el7.x86_64 #1 SMP Sun Jan 14 10:36:03 EST 2018 x86_64 x86_64 x86_64 GNU/Linux

Moving to verified.

Comment 6 Sandro Bonazzola 2018-02-22 10:01:49 UTC
This bugzilla is included in oVirt 4.2.1 release, published on Feb 12th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.