Bug 1507776
| Summary: | [Ganesha] : Filebench errors out with an "Input/Output Error" almost immediately when IO resumes post failback. | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Ambarish <asoman> |
| Component: | nfs-ganesha | Assignee: | Soumya Koduri <skoduri> |
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Manisha Saini <msaini> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | rhgs-3.3 | CC: | amukherj, bturner, dang, ffilz, jthottan, kkeithle, mbenjamin, msaini, pasik, rhinduja, rhs-bugs, sanandpa, skoduri, storage-qa-internal |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-05-06 11:56:12 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Ambarish
2017-10-31 06:29:29 UTC
FileBench WML :
set $dir=/gluster-mount/d1
set $nfiles=100000
set $meandirwidth=200
set $filesize=cvar(type=cvar-gamma,parameters=mean:131072;gamma:1.5)
set $nthreads=50
set $iosize=1m
set $meanappendsize=16k
define fileset name=bigfileset,path=$dir,size=$filesize,entries=$nfiles,dirwidth=$meandirwidth,prealloc=80
define process name=filereader,instances=1
{
thread name=filereaderthread,memsize=10m,instances=$nthreads
{
flowop createfile name=createfile1,filesetname=bigfileset,fd=1
flowop writewholefile name=wrtfile1,srcfd=1,fd=1,iosize=$iosize
flowop closefile name=closefile1,fd=1
flowop openfile name=openfile1,filesetname=bigfileset,fd=1
flowop appendfilerand name=appendfilerand1,iosize=$meanappendsize,fd=1
flowop closefile name=closefile2,fd=1
flowop openfile name=openfile2,filesetname=bigfileset,fd=1
flowop readwholefile name=readfile1,fd=1,iosize=$iosize
flowop closefile name=closefile3,fd=1
flowop deletefile name=deletefile1,filesetname=bigfileset
flowop statfile name=statfile1,filesetname=bigfileset
}
}
echo "File-server Version 3.0 personality successfully loaded"
run 72000
~
I dont think this is a regression(Can try if required).I have hit bugs like this before (with Bonnie etc) where I've failed to provide a reproducer to Dev. With Filebench,we have an easy reproducer. From brick logs:
[2017-10-30 09:37:14.084037] W [inodelk.c:399:pl_inodelk_log_cleanup] 0-butcher-server: releasing lock on 00000000-0000-0000-0000-000000000001 held by {client=0x7f41b0013400, pid=-3 lk-owner=60da0318fd7e0000}
[2017-10-30 09:37:14.084071] I [MSGID: 115013] [server-helpers.c:289:do_fd_cleanup] 0-butcher-server: fd cleanup on /
[2017-10-30 09:37:14.084221] I [MSGID: 101055] [client_t.c:440:gf_client_unref] 0-butcher-server: Shutting down connection gqas007.sbu.lab.eng.bos.redhat.com-28664-2017/10/30-09:36:28:32553-butcher-client-23-0-0
[2017-10-30 09:49:00.864932] I [MSGID: 113109] [posix.c:1613:posix_mkdir] 0-butcher-posix: mkdir (00000000-0000-0000-0000-000000000001/bigfileset): failing preop of mkdir (/bricks4/brick/bigfileset) as on-disk xattr value differs from argument value for key trusted.glusterfs.dht [Input/output error]
[2017-10-30 09:49:00.865015] E [MSGID: 115056] [server-rpc-fops.c:516:server_mkdir_cbk] 0-butcher-server: 24: MKDIR /bigfileset (00000000-0000-0000-0000-000000000001/bigfileset) client: gqas013.sbu.lab.eng.bos.redhat.com-31011-2017/10/30-09:35:03:531335-butcher-client-23-0-0 [Input/output error]
[2017-10-30 09:49:35.552548] I [MSGID: 115056] [server-rpc-fops.c:472:server_rmdir_cbk] 0-butcher-server: 192: RMDIR /bigfileset/00000005/00000003/00000012/00000005/00000009/00000017/00000085 (fe369743-f7f0-4d7b-8892-b3a2e21f869a/00000085) ==> (No such file or directory) [No such file or directory]
(In reply to Ambarish from comment #3) > I dont think this is a regression(Can try if required).I have hit bugs like > this before (with Bonnie etc) where I've failed to provide a reproducer to > Dev. > > With Filebench,we have an easy reproducer. As nfs-ganesha sources haven't changed from RHGS 3.3.0 to RHGS 3.3.1, this is definitely not a regression with ganesha build. Most likely the error seem to have generated from gluster stack. Will look into sos reports and tcpdumps once attached. |