Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1719991

Summary: Add nfsclient.ops.<opname>.errors - a new count of RPC ops completing with errors in /proc/self/mountstats for NFS
Product: Red Hat Enterprise Linux 8 Reporter: Dave Wysochanski <dwysocha>
Component: pcpAssignee: Nathan Scott <nathans>
Status: CLOSED DUPLICATE QA Contact: Jan Kurik <jkurik>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.0CC: agerstmayr, jkurik, mgoodwin, nathans, patrickm
Target Milestone: rc   
Target Release: 8.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-10 23:48:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1636572    
Bug Blocks:    
Attachments:
Description Flags
Normal RHEL8 mountstats with NFSv3, 4.0, 4.1, and 4.2 shares mounted
none
Patched RHEL8 mountstats showing RPC iostat version 1.1 and 9th column of numbers, with NFSv3, 4.0, 4.1, and 4.2 shares mounted none

Description Dave Wysochanski 2019-06-12 21:48:33 UTC
Description of problem:
We're adding a per-op error completion count to the kernel's /proc/self/mountstats file.  This will help NFS client troubleshooting.

Version-Release number of selected component (if applicable):
pcp-4.3.0-3.el8.x86_64

How reproducible:
easy

Steps to Reproduce:
See other bugs, specifically nfs-utils to generate non-zero counts.  You also need kernel patches not yet in RHEL8 (hopefully soon upstream as they are in maintainer tree)

Actual results:

Expected results:
nfsclient.ops.<opname>.errors exists and counts the last number in /proc/self/mountstats when RPC iostats version >= 1.1.  

We should see these error counts in pminfo and they should reflect the values in mountstats


Additional info:
kernel: https://bugzilla.redhat.com/show_bug.cgi?id=1636572#c27
nfs-utils: https://bugzilla.redhat.com/show_bug.cgi?id=1719983

Comment 1 Nathan Scott 2019-06-12 22:14:11 UTC
Thanks for the heads-up Dave.  Could you attach a /proc/mountstats file with the new output format please, if you have a machine running a patched kernel with some NFS mounts handy?  (we'll use this for regression testing)

cheers.

Comment 2 Dave Wysochanski 2019-06-13 11:36:32 UTC
Created attachment 1580252 [details]
Normal RHEL8 mountstats with NFSv3, 4.0, 4.1, and 4.2 shares mounted

Comment 3 Dave Wysochanski 2019-06-13 11:37:11 UTC
Created attachment 1580253 [details]
Patched RHEL8 mountstats showing RPC iostat version 1.1 and 9th column of numbers, with NFSv3, 4.0, 4.1, and 4.2 shares mounted

Comment 4 Nathan Scott 2019-06-14 09:45:03 UTC
Thanks Dave, I'll get onto implementing this shortly.

Comment 5 Nathan Scott 2019-06-14 21:20:27 UTC
Dave, I see in your mountstats.py patch in the nfs-utils BZ (1719983) ...

-                print('\t%d ops (%d%%)' % \
-                    (count, ((count * 100) / sends)), end=' ')
+                print('\t%d ops (%d%%) %d completions %d xmits %d errors' % \
+                    (count, ((count * 100) / sends), stats[2], stats[10], stats[9]), end=' ')

This suggests there's another metric patch lurking somewhere for the metric reported above as 'xmits'.  However I don't see this in Tronds tree (and mainline has neither yet).  Is this tenth field going to be added to the kernel, as this userspace patch suggests?

Thanks.

Comment 6 Dave Wysochanski 2019-06-14 23:04:08 UTC
Nathan,

No for now I abandoned the work to try to split out "completions" and "transmissions" and I never submitted those kernel patches, only the error count patches.  Also the one nfs-utils patch that has been merged is for the extra error count so there's only one extra count being added:
http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;h=73491ef272f9131888ef9f45207abbc2055d6aae

I also have a few more nfs-utils patches for other tools that are waiting to be merged.  Those just fixup other tools though for the 'errors' count:
https://bugzilla.redhat.com/show_bug.cgi?id=1719983#c1

Comment 7 Nathan Scott 2019-06-24 21:21:26 UTC
Fixed upstream, will aim for inclusion in 8.2

commit c81f8ee0fd0166b0427fb6aaa9963f4f3c095288
Author: Nathan Scott <nathans>
Date:   Tue Jun 18 08:16:39 2019 +1000

    pmdanfsclient: add per-op, per-mount rpc errors metrics
    
    Resolves Red Hat BZ #1719991

Comment 8 Nathan Scott 2019-10-10 23:48:32 UTC
This commit is part of the PCP rebase for 8.2 and does not require special QE attention - marking it as a duplicate.

*** This bug has been marked as a duplicate of bug 1723598 ***