Bug 1469292

Summary: Metrics for NFS File System resource are not collected if NFSv2 is not exposed by remote NFS server
Product: [JBoss] JBoss Operations Network Reporter: Larry O'Leary <loleary>
Component: Plugin -- OtherAssignee: Ruben Vargas Palma <rvargasp>
Status: CLOSED ERRATA QA Contact: Jeeva Kandasamy <jkandasa>
Severity: high Docs Contact:
Priority: urgent    
Version: JON 3.3.8CC: fbrychta, jkandasa, miburman, rvargasp, spinder
Target Milestone: DR01Keywords: Triaged
Target Release: JON 3.3.10   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-16 03:16:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
verified screenshot none

Description Larry O'Leary 2017-07-10 21:34:04 UTC
Description of problem:
NFS File System resource reports all metrics as NaN if NFS server does not expose NFSv2 mounts.

This is a big problem for modern operating systems such as RHEL 7 which no longer supports NFSv2 and disables it by default. 

The result is that `Free Files` ('fileSystemUsage.freeFiles`), `Free Space` ('fileSystemUsage.free`), `Used` ('fileSystemUsage.used`), `Used Percentage` ('fileSystemUsage.usePercent`) and `Capacity` ('fileSystemUsage.total`) always report a metric value of `NaN`.

Version-Release number of selected component (if applicable):
3.3.8

How reproducible:
Always

Steps to Reproduce:
1.  Configure NFS server to export NFS mounts:

----
yum install -y nfs-utils libnfsidmap
systemctl enable rpcbind
systemctl enable nfs-server
systemctl start rpcbind
systemctl start nfs-server
systemctl start rpc-statd
systemctl start nfs-idmapd
firewall-cmd --permanent --zone public --add-service mountd
firewall-cmd --permanent --zone public --add-service rpc-bind
firewall-cmd --permanent --zone public --add-service nfs
firewall-cmd --reload

# where our exports are mounted and shared from
mkdir -p /export/data
# a simple example directory to make available via NFS
mkdir -p /data
# manual mount
# mount --bind /data /export/data
# mount on boot or mount -a
echo '/data    /export/data   none    bind  0  0' >>/etc/fstab
# create the export entries for our subnet/clients
echo '/export      10.10.0.0/21(rw,fsid=0,insecure,no_subtree_check,async)' >>/etc/exports
echo '/export/data 10.10.0.0/21(rw,nohide,insecure,no_subtree_check,async)' >>/etc/exports
# refresh/reload the NFS exports
exportfs -r

# if not using RHEL 7, explicitly disable NFSv2:
echo "RPCNFSDARGS=\"--no-nfs-version 2\"" >>/etc/sysconfig/nfs
----

2.  Install, configure and start JBoss ON 3.3 system.
3.  Configure agent host to mount NFS mounts exported from NFS server:

----
yum install -y nfs-utils libnfsidmap
systemctl enable rpcbind
systemctl start rpcbind
mkdir -p /mnt/data
# mount $HOSTNAME:/export/data /mnt/data
echo "${HOSTNAME}"':/export/data /mnt/data nfs rw,sync,hard,intr 0 0' >>/etc/fstab
mount -a
----

4.  From JBoss ON system inventory, perform service/detail discovery scan for agent so new NFS mounts can be discovered and imported.
5.  From imported NFS file system resource, enable all metrics with a collection interval of 30 seconds.
6.  View the NFS file system resource's Metrics Monitoring page and Traits Monitoring page.

Actual results:
All Live Values report `--no data available--` and the most recent minimum, maximum, and average report `NaN`.

Expected results:
Measurements Free Files, Free Space, Used, Used Percentage and trait Capacity should reflect a numeric value of >= 0.

Additional info:
This issue is a direct result of the RPC ping that is performed by the Sigar code. A hard-coded version is used for the ping[1].

If version 2 of the RPC program is not available, the ping will return (9) RPC: Program/version mismatch. 

Please note that if the ping used version 3 or version 4 or if NFSv2 was enabled on the remote NFS server (if it is an option), all begins to work fine. Basically, the version used by ping is an exact match. Meaning that whatever version is used by ping must be what is running on the remote server. In legacy RHEL versions NFSv2, v3, and v4 were started by default, As of RHEL 7, only v3 and v4 are started as v2 is no longer supported. It is possible that other supported platforms have various other version restrictions.

[1]: https://github.com/hyperic/sigar/blob/sigar-1.6/bindings/java/src/org/hyperic/sigar/NfsFileSystem.java#L50

Comment 2 Michael Burman 2017-10-31 11:46:14 UTC
Ruben, can you add v4 support to the PR?

https://github.com/rhq-project/rhq/pull/308

Comment 3 Ruben Vargas Palma 2017-10-31 16:06:09 UTC
@Mike

I've already updated the PR with v4 support, also addressed Larry's comment on the order of ping test.

Comment 4 Michael Burman 2017-11-29 12:40:32 UTC
In the master:

commit 34a3d974543fdec9fcae7d33e10e4574100132fd
Author: Ruben Vargas <ruben.vp8510>
Date:   Tue Jul 18 17:00:41 2017 -0500

    Fixes BZ1469292 - Now RPC ping support NFSv3 and NFSv4

Comment 6 Simeon Pinder 2017-12-29 12:12:02 UTC
Moving to ON_QA as available for test with the latest build:

JON 3.3.10 DR01 artifacts are available for test from here:
http://download.eng.bos.redhat.com/brewroot/packages/org.jboss.on-jboss-on-parent/3.3.0.GA/164/maven/org/jboss/on/jon-server-patch/3.3.0.GA/jon-server-patch-3.3.0.GA.zip
 *Note: jon-server-patch-3.3.0.GA.zip maps to DR01 build of
 jon-server-3.3.0.GA-update-10.zip.

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=635136

Comment 7 Jeeva Kandasamy 2018-01-04 15:04:35 UTC
Created attachment 1376996 [details]
verified screenshot

Followed steps from comment#1 as Larry mentioned. Shows NFS file system metrics and traits. The screenshot is attached.

JON Version: 3.3.0.GA Update 10
Build Number: 8c98c29:28bd066

NFS mount details:
------------------
[hudson@jk-jon ~]$ mount -v | grep /mnt/data
jk-nfs.bc.jonqe.lab.eng.bos.redhat.com:/export/data on /mnt/data type nfs (rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.23.165,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.16.23.165)

Comment 8 Jeeva Kandasamy 2018-01-04 15:07:57 UTC
Typo on my comment#7

Followed steps from comment#0(description) as Larry mentioned, to verifying this bug.

Comment 11 errata-xmlrpc 2018-02-16 03:16:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0325