Bug 239654

Summary: Enhancement/backport: Expose I/O accounting stats in /proc
Product: Red Hat Enterprise Linux 5 Reporter: Kenn Humborg <kenn>
Component: kernelAssignee: RHEL Program Management <pm-rhel>
Status: CLOSED WONTFIX QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.0CC: apiemont, asereda, bill-bugzilla.redhat.com, dkovalsk, dwilson, eric, eteo, ghristov, gianluca.cecchi, gilboad, gregswift, k.georgiou, luke, madko, matt, mozilla_bugs
Target Milestone: ---Keywords: FutureFeature
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-10-25 07:02:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kenn Humborg 2007-05-10 10:25:33 UTC
Description of problem:

Total bytes read/written and number of read/write syscalls are
maintained in task_struct (rchar, wchar, syscr, syscw fields).

However, these are not exposed to userspace.

Version-Release number of selected component (if applicable):

kernel-2.6.18-8.1.1.el5PAE

How reproducible:

100%

Steps to Reproduce:
1. cd /proc/self
2. ls
  
Actual results:

No file called "io"

Expected results:

A file called "io" with contents like:

rchar: 59719
wchar: 47113
syscr: 982
syscw: 434

Additional info:

This is achievable with a back-port of this patch:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=aba76fdb8a5fefba73d3490563bf7c4da37b1a34

The reason I am looking for this is to help me identify which processes
are generating I/O load.

Comment 1 Bill McGonigle 2009-04-22 21:36:37 UTC
This link may also be helpful:

  http://kernelnewbies.org/Linux_2_6_20#head-bc56a9724daca612776dded22c281ca9ab54bb12

It seems to list the patches required for 2.6.20-style IO accounting.

This is needed to get e.g. iotop running on RHEL, so admins can see who's eating their disks.

I'm not sure about the timing of 2.6.20 vs. Kenn's request, but this may be the preferred implementation at this point, to get us through until RHEL6.

Comment 2 Eric Gearhart 2009-09-08 07:09:57 UTC
To get iotop running on RHEL, in addition to the kernel /proc backports we'd need to either update python to 2.5 (probably not going to happen in RHEL 5) or somehow we figure out why iotop specifically needs python to be >= 2.5, right?

It'd be great if iotop was backported to RHEL 5! I really miss it (I was running Debian in the past and used it extensively)

Comment 3 Greg Swift 2009-09-08 16:50:36 UTC
The RHEL5.4 release states the following:

Blktrace – block IO layer tracing mechanism – providing info on request
queue operations

I/O accounting – per-process I/O accounting data /proc/<pid>/io – not
impacted by cached data, also added to iostat

So does that mean this bug got handled?

Comment 4 Antonello Piemonte 2009-09-24 11:40:53 UTC
seems indeed available in 5.4

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.4 (Tikanga)
# cat /proc/8882/io
rchar: 1179045
wchar: 10619
syscr: 40646
syscw: 123
read_bytes: 360448
write_bytes: 45056
cancelled_write_bytes: 0

Comment 5 Matt Dainty 2010-02-01 16:59:23 UTC
Although the statistics do appear to be added and available through the /proc filesystem, iotop at least still won't work as it seems the netlink taskstats interface hasn't also been updated. Poking around /usr/include/linux/taskstats.h doesn't show the extra fields.

Incidentally, getting iotop to work is straightforward without needing to upgrade python, I just installed the python-ctypes package available in EPEL, (the homepage for iotop mentions this).

Comment 6 Georgi Hristov 2010-02-22 02:26:58 UTC
So, did IOTOP work for you with RHEL5.4 and Pytohon 2.4 + python-ctypes ? Or does it still need some kernel stats not exposed yet?

Comment 7 Georgi Hristov 2010-02-22 02:54:35 UTC
I just wanted to come back and note that IOTOP did not work on mine RHEL5.4 and Pytohon 2.4 + python-ctypes system.  I got an error that "CONFIG_TASK_DELAY_ACCT not enabled in kernel". I do have /proc/<pid>/io properly exposed.

BTW, IOTOP rpm did not properly install (download from freshmeat.net). After rebuilding the source RPM, it installed with no problems.

Comment 8 Gianluca Cecchi 2010-02-23 08:11:34 UTC
Hello,
should this message give a solution?
http://www.redhat.com/archives/epel-devel-list/2010-January/msg00015.html

Did you try iotop-0.4 too?

Comment 9 Greg Swift 2010-02-23 14:30:35 UTC
I tried building the fedora 0.4-1 version on RHEL 5.  Was able to build the RPM but I get the same error when executing iotop.

CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine SWAPIN and IO %

Comment 10 Matt Dainty 2010-02-23 15:06:21 UTC
From reading the source, iotop doesn't use the /proc filesystem to gather I/O statistics apart from checking the existence of /proc/self/io as a crude test to see if the kernel has the necessary support enabled.

iotop uses the netlink taskstats interface to gather I/O statistics and from what I can tell, the changes to that interface have not been backported to the RHEL 5 kernel. There may be a reason as to why, it could break something else.

Compare /usr/include/linux/taskstats.h from the kernel-headers package between RHEL and Fedora for example, you'll see the extra I/O counters in the Fedora version.

Comment 11 Bill McGonigle 2010-02-23 16:53:35 UTC
This looks like the commit:

http://repo.or.cz/w/iotop.git/commitdiff
/bed5a7472b8452b667a9d5e2a51a5ada345aaccc

>CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine SWAPIN and IO 

It looks like it is enabled:
 
  http://rhkernel.org/RHEL5+2.6.18-8.el5/configs/

but the new requirements say:

 # Check for requirements:
-#   o Python >= 2.5 for AF_NETLINK sockets
 #   o Linux >= 2.6.20 with I/O accounting

and RHEL has 2.6.18, so there must be some delta to 2.6.20 that's still needed.

The initial comment on this bug wasn't iotop specific, but if that's the ultimate goal, the summary/request should be updated, or a new bug posted.  Kenn, your call I think.

Comment 12 Gianluca Cecchi 2010-02-23 19:19:03 UTC
There are:
1) https://bugzilla.redhat.com/show_bug.cgi?id=546266
with python-ctypes inclusion that now is an option for iotop upstream use if you have python 2.4 such as in rh el 5
(see also home page at http://guichaz.free.fr/iotop/)

2) https://bugzilla.redhat.com/show_bug.cgi?id=557062
 Review Request: iotop - Top like utility for I/O  

That seems to be targeted to 5.6.... any hope for 5.5 inclusion that is now in beta?
From this bug it seems there are not quite serious BAD things right now...

Many users are waiting for one utility such as iotop and I think it could be a nice thing to be put into release notes for 5.5.

There are also systemtap+scripts to accomodate similar results, but probably iotop would be better and easier to use.


See also older EPEL bugzilla entry https://bugzilla.redhat.com/show_bug.cgi?id=465795

Thanks in advance for attention

Comment 13 Kenn Humborg 2010-02-24 18:04:14 UTC
The particular problem I was trying track down is no longer an
issue, so I don't have any strong feelings on this anymore.

Certainly, iotop looks like a very useful tool and probably would have
helped me a lot back then, so I'd agree with re-casting this bug 
as a request to get iotop working.

In addition, I don't have a machine running 5.4 that I can play with
right now, so I can't help out much with testing iotop, so I'll leave 
it up to you guys whether you want to change this bug's summary or 
create a new one.

Thanks all!

Comment 14 Eric Gearhart 2010-02-24 19:44:04 UTC
(In reply to comment #13)
> I'll leave 
> it up to you guys whether you want to change this bug's summary or 
> create a new one.

I very much would like to see iotop make it into RHEL... should create a new feature request asking specifically for iotop? I can and will if that's the best path.

Letting this ticket "roll over" to being an iotop request would be OK with me as well

Eric

Comment 15 Gianluca Cecchi 2010-02-24 21:51:58 UTC
As I wrote in comment #12, there is already a bug open for iotop inclusion as a review request: it is number 557062.
So I propose to close this bug as its subject has been solved with rh el 5.4 release; and being this a requirement, I'm proposing to stress change of target version for 557062 solution into 5.5.
Let the ones here sponsoring iotop inclusion go and add their comments to 
https://bugzilla.redhat.com/show_bug.cgi?id=557062 too

Comment 20 RHEL Program Management 2012-10-10 11:50:42 UTC
Thank you for submitting this issue for consideration. Red Hat Enterprise Linux 5 has reached the end of Production 1 Phase of its Life Cycle.  Red Hat does not plan to incorporate the suggested capability in a future Red Hat Enterprise Linux 5 minor release. If you would like Red  Hat to re-consider this feature request and the requested functionality is not currently in Red Hat Enterprise Linux 6, please re-open the request via appropriate support channels and provide additional supporting details about the importance of this issue.