Bug 972457 - agetty idle I/O polling causes elevated CPU usage
agetty idle I/O polling causes elevated CPU usage
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: util-linux (Show other bugs)
19
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Karel Zak
Fedora Extras Quality Assurance
:
: 973501 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-09 11:03 EDT by Alexander Demenshin
Modified: 2013-06-14 00:46 EDT (History)
5 users (show)

See Also:
Fixed In Version: util-linux-2.23.1-3.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-14 00:46:06 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Alexander Demenshin 2013-06-09 11:03:44 EDT
Description of problem:

When running F19 (and, most likely, all previous versions - see below) as KVM guest, active agetty constantly consumes a bit of CPU (1% in my case), which is elevated to ca. 4-5% of host CPU usage. Given that most virtualized systems are running headless but will have at least one agetty running, several of them will consume significant amount of CPU time at the host even when idling.

Version-Release number of selected component (if applicable):

util-linux-2.23.1-2.fc19.x86_64

How reproducible:

100% (Proxmox 3.0 KVM, though it may affect any QEMU-KVM or even other VMs/hypervisors).

Steps to Reproduce:
1. Run F19 inside KVM
2. Login and logout on console (though in my case login is not necessary)
3. Make sure that the guest is idle
4. Observe CPU usage in host and guest.

Actual results:

Checking CPU usage reveals that agetty in guest system is constantly consuming ca. 1% of CPU time (may depend on CPU type, it is Core i5-3570T in my case), host system (KVM process) is consuming ca. 4-5% of CPU. Running 6 active agetty processes leads to ca. 12-13% CPU usage in host (each still consuming ca. 1% in guest).

Expected results:

CPU usage should be very close to 0% while idling (as it is when agetty is not running).

Additional info:

ltrace of running agetty reveals the following loop:

16:36:30.110604 usleep(1000)                                                                                  = <void>
16:36:30.112047 read(0 <no return ...>
error: maximum array length seems negative
, "\n", 1)                                                                                                    = -1
16:36:30.112494 __errno_location()                                                                            = 0x7f02ac6756c0
16:36:30.112829 usleep(1000)                                                                                  = <void>
16:36:30.114235 read(0 <no return ...>
error: maximum array length seems negative
, "\n", 1)                                                                                                    = -1
16:36:30.114748 __errno_location()                                                                            = 0x7f02ac6756c0

This loop (function get_logname in agetty.c) is the cause of elevated CPU usage, as it is repeated every 1ms - actually, just running usleep(1000) in a loop (without any I/O) will lead to this result.

Probably, this can (and should) be fixed outside of agetty (at hypervisor/VM/kernel level, as it is related to nanosleep implementation), but for the time being it makes sense to get rid of this loop in agetty to avoid unnecessary CPU consumption in virtualized environments.

PS: There are many reports about Linux KVM guests consuming up to 10% of host CPU when idle even on tickless kernels, so I believe that usleep() loop on agetty may be the cause.
Comment 1 Zdenek Kabelac 2013-06-11 14:37:22 EDT
Yep - agetty is getting very visible in the 'top' list.
It can actually beat even Firefox now with the amount of used CPU time.
(And my fan rotates unnecessarily)
Comment 2 Karel Zak 2013-06-12 15:07:39 EDT
*** Bug 973501 has been marked as a duplicate of this bug. ***
Comment 3 Florian Weimer 2013-06-12 15:12:43 EDT
It's not related to KVM, it happens on bare metal, too.
Comment 4 Florian Weimer 2013-06-12 15:30:40 EDT
Probably introduced by:

commit 3aa6b68f7e19fa3e1c2bba75bee921a98b7b46af
Author: Werner Fink <werner@suse.de>
Date:   Mon May 9 15:52:36 2011 +0200

    agetty: proper session on the terminal line
    
    Ensure a proper session on the terminal line, that is do a
    vhangup() and become the controlling terminal.  After this
    determine if the terminal line a virtual console by using
    the ioctl TIOCMGET to get the status modem bits of a serial
    line which is a invalid argument on a virtual console.
    
    Signed-off-by: Werner Fink <werner@suse.de>

I have verified that the triggering error code is EAGAIN, and the descriptor has indeed the O_NONBLOCK flag set.
Comment 5 Alexander Demenshin 2013-06-12 16:17:34 EDT
(In reply to Florian Weimer from comment #3)

> It's not related to KVM, it happens on bare metal, too.

Indeed... But the effect is most noticeable in KVM, though.
Comment 6 Karel Zak 2013-06-13 05:46:22 EDT
(In reply to Florian Weimer from comment #4)
> Probably introduced by:

No, it was my mistake introduced by --local-line patch (backported to f19). 

I have fixed the code, so O_NONBLOCK should be used only for serial lines with -L (CLOCAL) flags. I have also increased the usleep() to 0.25s to make it less expensive on systems with ugly serial lines.
Comment 7 Fedora Update System 2013-06-13 06:01:08 EDT
util-linux-2.23.1-3.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/util-linux-2.23.1-3.fc19
Comment 8 Fedora Update System 2013-06-13 14:02:56 EDT
Package util-linux-2.23.1-3.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing util-linux-2.23.1-3.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-10745/util-linux-2.23.1-3.fc19
then log in and leave karma (feedback).
Comment 9 Fedora Update System 2013-06-14 00:46:06 EDT
util-linux-2.23.1-3.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.