Bug 1264514

Summary: nfsometer fails on NFSv3 and NFSv4 shares
Product: Red Hat Enterprise Linux 7 Reporter: Shahish Nambiar <snambiar>
Component: nfsometerAssignee: Steve Dickson <steved>
Status: CLOSED ERRATA QA Contact: JianHong Yin <jiyin>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.1CC: dros, dwysocha, eguan, sbhat, smayhew, steved, swhiteho
Target Milestone: rcKeywords: TestCaseProvided
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: nfsometer-1.7-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 01:38:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1203710, 1295577, 1313485    

Description Shahish Nambiar 2015-09-18 16:13:02 UTC
Description of problem:
nfsometer fails on NFSv3 and NFSv4 shares.

Below are the details:

On NFSv4 protocol:
=================
# nfsometer -o v4 10.65.233.20:/data 
> Using results directory: /root/nfsometer_results
> Probing 10.65.233.20:/data: v4
Mounting: 10.65.233.20:/data (options: v4)...
Syncing: 10.65.233.20:/data....
Unmounting: 10.65.233.20:/data...

Requested: 10 workloads X 1 options X 1 runs = 10 traces
Results directory already has 1 matching traces
Need to run 9 of 10 requested traces
 filebench_fileserver - needs 1 runs of v4
 filebench_varmail - needs 1 runs of v4
 gitclone - needs 1 runs of v4
 python - needs 1 runs of v4
 dd_100m_1k - needs 1 runs of v4
 filebench_networkfs - needs 1 runs of v4
 kernel - needs 1 runs of v4
 cthon - needs 1 runs of v4
 filebench_webserver - needs 1 runs of v4

> Removing old error result directories from /tmp/nfsometer_trace-*

> Trace 1/9: 1 of 1 for filebench_fileserver: v4

< SETUP WORKLOAD >
Mounting: 10.65.233.20:/data (options: v4)...
removing run directory...done
Syncing: 10.65.233.20:/data....
Unmounting: 10.65.233.20:/data...

< RUN WORKLOAD >
Mounting: 10.65.233.20:/data (options: v4)...
Running command: cd /root/.nfsometer/workload_files/filebench_fileserver && filebench -f nfsometer-fileserver.f
Traceback (most recent call last):
  File "/usr/bin/nfsometer", line 130, in <module>
    main()
  File "/usr/bin/nfsometer", line 123, in main
    mode_fetch_trace(opts, fetch_only=fetch_only)
  File "/usr/bin/nfsometer", line 77, in mode_fetch_trace
    trace.run_traces(collection, opts, fetch_only=fetch_only)
  File "/usr/lib/python2.7/site-packages/nfsometerlib/trace.py", line 862, in run_traces
    workload_obj.run()
  File "/usr/lib/python2.7/site-packages/nfsometerlib/workloads.py", line 199, in run
    raise e
nfsometerlib.cmd.CmdErrorCode: command "( time ( sh /tmp/nfsometer_trace/command.sh > /tmp/nfsometer_trace/test.log 2>&1 ) ) 2> /tmp/nfsometer_trace/test.time" exited with non-zero status: 1
Syncing: 10.65.233.20:/data....
Unmounting: 10.65.233.20:/data...
WARNING: Moving failed rundir to /tmp/nfsometer_trace-error-1442524199 - it will be deleted on next run of this script!
You have new mail in /var/spool/mail/root

# echo $?
1
#
 

On NFSv3 protocol:
=================
# nfsometer -o v3 10.65.233.20:/data 
> Using results directory: /root/nfsometer_results
> Probing 10.65.233.20:/data: v3
Mounting: 10.65.233.20:/data (options: v3)...
Syncing: 10.65.233.20:/data....
Unmounting: 10.65.233.20:/data...

Requested: 10 workloads X 1 options X 1 runs = 10 traces
Need to run 10 of 10 requested traces
 dd_100m_100k - needs 1 runs of v3
 filebench_fileserver - needs 1 runs of v3
 filebench_varmail - needs 1 runs of v3
 gitclone - needs 1 runs of v3
 python - needs 1 runs of v3
 dd_100m_1k - needs 1 runs of v3
 filebench_networkfs - needs 1 runs of v3
 kernel - needs 1 runs of v3
 cthon - needs 1 runs of v3
 filebench_webserver - needs 1 runs of v3

> Removing old error result directories from /tmp/nfsometer_trace-*

> Trace 1/10: 1 of 1 for dd_100m_100k: v3

< SETUP WORKLOAD >
Mounting: 10.65.233.20:/data (options: v3)...
removing run directory...done
Syncing: 10.65.233.20:/data....
Unmounting: 10.65.233.20:/data...

< RUN WORKLOAD >
Mounting: 10.65.233.20:/data (options: v3)...
Running command: dd if=/dev/zero of=./dd_file.100m_100k bs=102400 count=1024
Syncing: 10.65.233.20:/data....
Unmounting: 10.65.233.20:/data...
Results copied to: nfsometer_trace-dd_100m_100k-shahish-openstack.gsslab.pnq.redhat.com-1442524746.27

> Trace 2/10: 1 of 1 for filebench_fileserver: v3

< SETUP WORKLOAD >
Mounting: 10.65.233.20:/data (options: v3)...
removing run directory...done
Syncing: 10.65.233.20:/data....
Unmounting: 10.65.233.20:/data...

< RUN WORKLOAD >
Mounting: 10.65.233.20:/data (options: v3)...
Running command: cd /root/.nfsometer/workload_files/filebench_fileserver && filebench -f nfsometer-fileserver.f
Traceback (most recent call last):
  File "/usr/bin/nfsometer", line 130, in <module>
    main()
  File "/usr/bin/nfsometer", line 123, in main
    mode_fetch_trace(opts, fetch_only=fetch_only)
  File "/usr/bin/nfsometer", line 77, in mode_fetch_trace
    trace.run_traces(collection, opts, fetch_only=fetch_only)
  File "/usr/lib/python2.7/site-packages/nfsometerlib/trace.py", line 862, in run_traces
    workload_obj.run()
  File "/usr/lib/python2.7/site-packages/nfsometerlib/workloads.py", line 199, in run
    raise e
nfsometerlib.cmd.CmdErrorCode: command "( time ( sh /tmp/nfsometer_trace/command.sh > /tmp/nfsometer_trace/test.log 2>&1 ) ) 2> /tmp/nfsometer_trace/test.time" exited with non-zero status: 1
Syncing: 10.65.233.20:/data....
Unmounting: 10.65.233.20:/data...
WARNING: Moving failed rundir to /tmp/nfsometer_trace-error-1442524809 - it will be deleted on next run of this script!
You have new mail in /var/spool/mail/root

# echo $?
1
#



Version-Release number of selected component (if applicable):
nfsometer-1.7-0.el7.noarch


How reproducible:
Always



Actual results:


Expected results:


Additional info:

Comment 4 Steve Dickson 2016-02-12 19:37:05 UTC
Dros... any ideas?

Comment 5 Weston Andros Adamson 2016-02-22 14:17:22 UTC
It's just a guess until I can see what actually happened in one of the error trace directories.

Comment 6 Weston Andros Adamson 2016-02-22 14:23:55 UTC
FWIW, it works for me.

Comment 7 Scott Mayhew 2016-02-22 21:11:30 UTC
(In reply to Weston Andros Adamson from comment #6)
> FWIW, it works for me.

I took a look at the support case that spawned this bug and I'm pretty sure that the customer's original issue was that they were running nfsometer as root against an export that had root squashing enabled.  I've sent some patches that makes nfsometer print out some error messages instead of a backtrace in this scenario (and a handful of other scenarios I've come across).

Comment 8 Steve Dickson 2016-02-22 21:40:10 UTC
(In reply to Scott Mayhew from comment #7)
> (In reply to Weston Andros Adamson from comment #6)
> > FWIW, it works for me.
> 
> I took a look at the support case that spawned this bug and I'm pretty sure
> that the customer's original issue was that they were running nfsometer as
> root against an export that had root squashing enabled.  I've sent some
> patches that makes nfsometer print out some error messages instead of a
> backtrace in this scenario (and a handful of other scenarios I've come
> across).

Thank you!

Comment 9 Steve Whitehouse 2016-02-23 12:04:10 UTC
I've set devel cond nack on this based on the comments in the bug. Please update that if that status is not correct.

Comment 10 Steve Dickson 2016-04-04 19:08:17 UTC
commit 9c5998207051f6f059f3d06721488c5afbc1fa11
Author: Scott Mayhew <smayhew>
Date:   Mon Feb 22 16:02:11 2016 -0500

    trace.py: Add some more error handling to probe_mounts()

commit 13716bdee39dd8d332b982703423d5895e0c0829
Author: Scott Mayhew <smayhew>
Date:   Mon Feb 22 16:02:10 2016 -0500

    cmd.py: redefine CmdErrorCode
    
commit aa55e4dd8b72640ad9e0b7b4898db43bad32e046
Author: Weston Andros Adamson <dros>
Date:   Thu Oct 15 10:08:12 2015 -0400

    trace.py: ignore stderr output from sysctl

Comment 16 errata-xmlrpc 2016-11-04 01:38:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2253.html