RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1719991 - Add nfsclient.ops.<opname>.errors - a new count of RPC ops completing with errors in /proc/self/mountstats for NFS
Summary: Add nfsclient.ops.<opname>.errors - a new count of RPC ops completing with er...
Keywords:
Status: CLOSED DUPLICATE of bug 1723598
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: pcp
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.2
Assignee: Nathan Scott
QA Contact: Jan Kurik
URL:
Whiteboard:
Depends On: 1636572
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-12 21:48 UTC by Dave Wysochanski
Modified: 2021-09-17 12:47 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-10 23:48:32 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Normal RHEL8 mountstats with NFSv3, 4.0, 4.1, and 4.2 shares mounted (11.16 KB, text/plain)
2019-06-13 11:36 UTC, Dave Wysochanski 		no flags Details
Patched RHEL8 mountstats showing RPC iostat version 1.1 and 9th column of numbers, with NFSv3, 4.0, 4.1, and 4.2 shares mounted (11.58 KB, text/plain)
2019-06-13 11:37 UTC, Dave Wysochanski 		no flags Details
View All

Description Dave Wysochanski 2019-06-12 21:48:33 UTC 
Description of problem:
We're adding a per-op error completion count to the kernel's /proc/self/mountstats file.  This will help NFS client troubleshooting.

Version-Release number of selected component (if applicable):
pcp-4.3.0-3.el8.x86_64

How reproducible:
easy

Steps to Reproduce:
See other bugs, specifically nfs-utils to generate non-zero counts.  You also need kernel patches not yet in RHEL8 (hopefully soon upstream as they are in maintainer tree)

Actual results:

Expected results:
nfsclient.ops.<opname>.errors exists and counts the last number in /proc/self/mountstats when RPC iostats version >= 1.1.  

We should see these error counts in pminfo and they should reflect the values in mountstats


Additional info:
kernel: https://bugzilla.redhat.com/show_bug.cgi?id=1636572#c27
nfs-utils: https://bugzilla.redhat.com/show_bug.cgi?id=1719983
Comment 1 Nathan Scott 2019-06-12 22:14:11 UTC 
Thanks for the heads-up Dave.  Could you attach a /proc/mountstats file with the new output format please, if you have a machine running a patched kernel with some NFS mounts handy?  (we'll use this for regression testing)

cheers.
Comment 2 Dave Wysochanski 2019-06-13 11:36:32 UTC 
Created attachment 1580252 [details]
Normal RHEL8 mountstats with NFSv3, 4.0, 4.1, and 4.2 shares mounted
Comment 3 Dave Wysochanski 2019-06-13 11:37:11 UTC 
Created attachment 1580253 [details]
Patched RHEL8 mountstats showing RPC iostat version 1.1 and 9th column of numbers, with NFSv3, 4.0, 4.1, and 4.2 shares mounted
Comment 4 Nathan Scott 2019-06-14 09:45:03 UTC 
Thanks Dave, I'll get onto implementing this shortly.
Comment 5 Nathan Scott 2019-06-14 21:20:27 UTC 
Dave, I see in your mountstats.py patch in the nfs-utils BZ (1719983) ...

-                print('\t%d ops (%d%%)' % \
-                    (count, ((count * 100) / sends)), end=' ')
+                print('\t%d ops (%d%%) %d completions %d xmits %d errors' % \
+                    (count, ((count * 100) / sends), stats[2], stats[10], stats[9]), end=' ')

This suggests there's another metric patch lurking somewhere for the metric reported above as 'xmits'.  However I don't see this in Tronds tree (and mainline has neither yet).  Is this tenth field going to be added to the kernel, as this userspace patch suggests?

Thanks.
Comment 6 Dave Wysochanski 2019-06-14 23:04:08 UTC 
Nathan,

No for now I abandoned the work to try to split out "completions" and "transmissions" and I never submitted those kernel patches, only the error count patches.  Also the one nfs-utils patch that has been merged is for the extra error count so there's only one extra count being added:
http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;h=73491ef272f9131888ef9f45207abbc2055d6aae

I also have a few more nfs-utils patches for other tools that are waiting to be merged.  Those just fixup other tools though for the 'errors' count:
https://bugzilla.redhat.com/show_bug.cgi?id=1719983#c1
Comment 7 Nathan Scott 2019-06-24 21:21:26 UTC 
Fixed upstream, will aim for inclusion in 8.2

commit c81f8ee0fd0166b0427fb6aaa9963f4f3c095288
Author: Nathan Scott <nathans>
Date:   Tue Jun 18 08:16:39 2019 +1000

    pmdanfsclient: add per-op, per-mount rpc errors metrics
    
    Resolves Red Hat BZ #1719991
Comment 8 Nathan Scott 2019-10-10 23:48:32 UTC 
This commit is part of the PCP rebase for 8.2 and does not require special QE attention - marking it as a duplicate.

*** This bug has been marked as a duplicate of bug 1723598 ***
Note You need to log in before you can comment on or make changes to this bug.