This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours

Bug 821081

Summary: F17 alpha PPC64: Running fsracer test on NFSv4 mounted file system generated call trace
Product: [Fedora] Fedora Reporter: IBM Bug Proxy <bugproxy>
Component: kernelAssignee: Jeff Layton <jlayton>
Status: CLOSED UPSTREAM QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: low Docs Contact:
Priority: unspecified    
Version: 17CC: bfields, braden, gansalmon, graeme.russ, itamar, jherbold, jiali, jkachuck, jlayton, jonathan, kernel-maint, lsof, madhu.chinakonda, ovasik, steved, wgomerin
Target Milestone: ---   
Target Release: ---   
Hardware: ppc64   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-03 08:51:39 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Attachments:
Description Flags
dmesg.txt
none
var-log-messages.txt
none
more-details-log.txt
none
racer.tar.gz none

Description IBM Bug Proxy 2012-05-11 16:00:41 EDT
While running fsracer file system test on NFSv4 mounted file system on Fedora 17 alpha PPC64 on JUNO IOCL LPAR, it generated following call trace in /var/log/messages & demsg of NFSv4 Client.

[ 7454.059246] NFS: v4 server 9.47.82.130 does not accept raw uid/gids. Reenabling the idmapper.
[ 7511.449220] NFS: v4 server 9.47.82.130 does not accept raw uid/gids. Reenabling the idmapper.
[ 7530.368791] NFS: v4 server 9.47.82.130 does not accept raw uid/gids. Reenabling the idmapper.
[ 7803.414947] ------------[ cut here ]------------
[ 7803.414958] WARNING: at fs/inode.c:290
[ 7803.414961] Modules linked in: nfs fscache windfarm_smu_sat i2c_core hvcs hvcserver ibmveth windfarm_pid nfsd nfs_acl auth_rpcgss lockd sunrpc ibmvscsic scsi_transport_srp scsi_tgt [last unloaded: scsi_wait_scan]
[ 7803.414993] NIP: c00000000025d104 LR: d000000002474290 CTR: c00000000025d0d0
[ 7803.414999] REGS: c0000000e2903440 TRAP: 0700   Not tainted  (3.3.1-5.fc17.ppc64)
[ 7803.415002] MSR: 8000000000029032 <SF,EE,ME,IR,DR,RI>  CR: 44088022  XER: 20000000
[ 7803.415014] CFAR: d0000000024c84b8
[ 7803.415018] TASK = c0000000d1920000[1555] 'cat' THREAD: c0000000e2900000 CPU: 1
[ 7803.415023] GPR00: d000000002474290 c0000000e29036c0 c0000000012f1d38 c0000000f65834f0 
[ 7803.415030] GPR04: c0000000f65834f0 c0000000f65834f0 0000000000000002 0000000000000000 
[ 7803.415038] GPR08: 0000000000000000 0000000000000000 0000000000000001 c00000000025d0d0 
[ 7803.415046] GPR12: d0000000024c84a0 c00000000ec60800 0000000000000000 0000000000000000 
[ 7803.415054] GPR16: 0000000000000000 c0000000d1920000 0000000000080000 0000000000000000 
[ 7803.415061] GPR20: c0000000d1920a20 0000000000000001 0000000000000001 c0000000f42e1810 
[ 7803.415069] GPR24: c0000000f1800720 0000000000000000 c0000000f64b3fb0 c0000000f6187bc0 
[ 7803.415076] GPR28: c0000000f64b3f00 c0000000f65834f0 c000000001270490 c0000000e29036c0 
[ 7803.415093] NIP [c00000000025d104] .drop_nlink+0x34/0x80
[ 7803.415120] LR [d000000002474290] .nfs_dentry_iput+0x50/0xa0 [nfs]
[ 7803.415123] Call Trace:
[ 7803.415130] [c0000000e29036c0] [c0000000e2903750] 0xc0000000e2903750 (unreliable)
[ 7803.415153] [c0000000e2903750] [d000000002474290] .nfs_dentry_iput+0x50/0xa0 [nfs]
[ 7803.415161] [c0000000e29037e0] [c000000000257760] .d_kill+0x110/0x240
[ 7803.415168] [c0000000e2903890] [c0000000002585ec] .dput+0x19c/0x370
[ 7803.415174] [c0000000e2903940] [c000000000239c58] .fput+0x1e8/0x2f0
[ 7803.415181] [c0000000e2903a00] [c0000000002339cc] .filp_close+0xbc/0x110
[ 7803.415190] [c0000000e2903aa0] [c00000000009fef8] .put_files_struct+0x128/0x1c0
[ 7803.415197] [c0000000e2903b60] [c0000000000a0098] .exit_files+0x68/0x90
[ 7803.415203] [c0000000e2903bf0] [c0000000000a07a0] .do_exit+0x1f0/0x960
[ 7803.415209] [c0000000e2903d00] [c0000000000a1274] .do_group_exit+0x64/0x100
[ 7803.415216] [c0000000e2903da0] [c0000000000a133c] .SyS_exit_group+0x2c/0x30
[ 7803.415223] [c0000000e2903e30] [c0000000000098e4] syscall_exit+0x0/0x40
[ 7803.415227] Instruction dump:
[ 7803.415231] fbe1fff8 fba1ffe8 f8010010 f821ff71 7c3f0b78 7c7d1b78 60000000 60000000 
[ 7803.415240] 813d0048 2f890000 7d400026 554afffe <0b0a0000> 3929ffff 2f890000 913d0048 
[ 7803.415250] ---[ end trace 12a1a13949e190cb ]---


--- server details ---

[root@elm17f130 ~]# uname -a
Linux elm17f130 3.3.4-1.fc17.ppc64 #1 SMP Tue May 1 12:48:10 MST 2012 ppc64 ppc64 ppc64 GNU/Linux

[root@elm17f130 ~]# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,relatime,size=2059072k,nr_inodes=32173,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
/dev/mapper/vg-lv_root on / type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=25,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /media type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/sda2 on /boot type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)

/dev/sda5 on /nfs/nfs1 type ext3 (rw,relatime,user_xattr,barrier=1,nodelalloc,data=ordered)
/dev/sda6 on /nfs/nfs2 type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)
/dev/sda7 on /nfs/nfs3 type xfs (rw,relatime,attr2,noquota)
/dev/sda8 on /nfs/nfs4 type reiserfs (rw,relatime)
/dev/sda9 on /nfs/nfs5 type ext2 (rw,relatime,user_xattr,barrier=1)
/dev/sda5 on /NFS/NFS1 type ext3 (rw,relatime,user_xattr,barrier=1,nodelalloc,data=ordered)
/dev/sda6 on /NFS/NFS2 type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)
/dev/sda8 on /NFS/NFS3 type reiserfs (rw,relatime)
/dev/sda8 on /NFS/NFS4 type reiserfs (rw,relatime)
/dev/sda8 on /NFS/NFS5 type reiserfs (rw,relatime)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
sunrpc on /proc/fs/nfsd type nfsd (rw,relatime)

--- client details ---
[root@elm17f131 ~]# uname -a
Linux elm17f131 3.3.1-5.fc17.ppc64 #1 SMP Wed Apr 11 06:19:45 MST 2012 ppc64 ppc64 ppc64 GNU/Linux

[root@elm17f131 ~]# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
devtmpfs on /dev type devtmpfs (rw,nosuid,relatime,seclabel,size=2002112k,nr_inodes=31283,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,relatime,seclabel)
tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,seclabel,mode=755)
/dev/mapper/vg-lv_root on / type ext4 (rw,relatime,seclabel,user_xattr,barrier=1,data=ordered)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=25,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
tmpfs on /media type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,mode=755)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
securityfs on /sys/kernel/security type securityfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
sunrpc on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/sda2 on /boot type ext4 (rw,relatime,seclabel,user_xattr,barrier=1,data=ordered)
9.47.82.130: on /NFSCLI type nfs4 (rw,relatime,vers=4,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=9.47.82.131,minorversion=0,local_lock=none,addr=9.47.82.130)
9.47.82.130:/NFS1 on /NFSCLI/NFS1 type nfs4 (rw,relatime,vers=4,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=9.47.82.131,minorversion=0,local_lock=none,addr=9.47.82.130)
9.47.82.130:/NFS3 on /NFSCLI/NFS3 type nfs4 (rw,relatime,vers=4,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=9.47.82.131,minorversion=0,local_lock=none,addr=9.47.82.130)


--- steps to reproduce ---
   
setup on NFSv4 server:
-----------------------------------
1) create five file systems say ext2,ext3,ext4,xfs,reiserfs 
2)then follow below steps to setup & configure nfsv4 server 
mkdir /nfs /NFS
mkdir /nfs/nfs1 /nfs/nfs2 /nfs/nfs3 /nfs/nfs4 /nfs/nfs5
mkdir /NFS/NFS1 /NFS/NFS2 /NFS/NFS3 /NFS/NFS4 /NFS/NFS5

chmod 777 /nfs /nfs/*
chmod 777 /NFS /NFS/*
mount /dev/sda5 /nfs/nfs1
mount /dev/sda6 /nfs/nfs2
mount /dev/sda7 /nfs/nfs3
mount /dev/sda8 /nfs/nfs4
mount /dev/sda9 /nfs/nfs5

mount --bind /nfs/nfs1 /NFS/NFS1
mount --bind /nfs/nfs2 /NFS/NFS2
mount --bind /nfs/nfs4 /NFS/NFS3
mount --bind /nfs/nfs4 /NFS/NFS4
mount --bind /nfs/nfs4 /NFS/NFS5

chmod 777 /nfs /nfs/*
chmod 777 /NFS /NFS/*

systemctl start nfs-server.service

echo "/NFS *(rw,fsid=0,insecure,no_subtree_check,sync)" > /etc/exports
echo "/NFS/NFS1 *(rw,nohide,insecure,no_subtree_check,sync)" >> /etc/exports
echo "/NFS/NFS2 *(rw,nohide,insecure,no_subtree_check,sync)" >> /etc/exports
echo "/NFS/NFS3 *(rw,nohide,insecure,no_subtree_check,sync)" >> /etc/exports
echo "/NFS/NFS4 *(rw,nohide,insecure,no_subtree_check,sync)" >> /etc/exports
echo "/NFS/NFS5 *(rw,nohide,insecure,no_subtree_check,sync)" >> /etc/exports

systemctl restart nfs-server.service
exportfs

Setup on NFSv4 Client:
------------------------------------
1) mkdir /NFSVCLI  (create a directoiry where you will mount nfsv4 filesystem from server)
2) chmod 777 /FAVCLI
3) mount <IP of NFSV4 server>:/ /NFSVCLI
4) copy racer.tar.gz & untar to each NFSV4 mounted client directory and start the test as 
   nohup ./racer.sh &

After 1 to 2 hours you may get call trace.

Note: attached files named dmesg.txt & var-log-message.txt captured from NFSV4 client system.

Similar bug has been reported at https://bugzilla.redhat.com/show_bug.cgi?id=813130
Comment 1 IBM Bug Proxy 2012-05-11 16:00:57 EDT
Created attachment 583914 [details]
dmesg.txt
Comment 2 IBM Bug Proxy 2012-05-11 16:01:07 EDT
Created attachment 583915 [details]
var-log-messages.txt
Comment 3 IBM Bug Proxy 2012-05-11 16:01:18 EDT
Created attachment 583916 [details]
more-details-log.txt
Comment 4 Ondrej Vasik 2012-05-11 16:11:23 EDT
Filesystem package has nothing to do with filesystems generally. Let's go with nfs-utils (and Steve Dickson) for further reassignment.
Comment 5 Jian Li 2012-05-13 22:04:41 EDT
hi IBM,
could you provide fsracer test?
Comment 6 IBM Bug Proxy 2012-05-14 02:50:55 EDT
------- Comment From maknayak@in.ibm.com 2012-05-14 06:47 EDT-------
(In reply to comment #12)
> hi IBM,
> could you provide fsracer test?

Hello RedHat,
Attached tar ball of racer test named "racer.tar.gz".

Thanks...
Manas
Comment 7 IBM Bug Proxy 2012-05-14 02:51:10 EDT
Created attachment 584257 [details]
racer.tar.gz


------- Comment (attachment only) From maknayak@in.ibm.com 2012-05-14 06:48 EDT-------
Comment 8 Jian Li 2012-05-14 03:38:18 EDT
(In reply to comment #7)
> Created attachment 584257 [details]
> racer.tar.gz
> 
> 
> ------- Comment (attachment only) From maknayak@in.ibm.com 2012-05-14 06:48
> EDT-------

Thanks very much
lijian
Comment 9 IBM Bug Proxy 2012-05-25 02:01:03 EDT
------- Comment From kamaleshb@in.ibm.com 2012-05-25 05:58 EDT-------
Hi RedHat,

Any updates on the bug ?
Thanks.
Comment 10 J. Bruce Fields 2012-05-25 16:09:11 EDT
Are you seeing any other symptoms?  Or just the warning?
Comment 11 IBM Bug Proxy 2012-05-29 11:40:47 EDT
------- Comment From maknayak@in.ibm.com 2012-05-29 15:36 EDT-------
(In reply to comment #17)
> Are you seeing any other symptoms?  Or just the warning?

I am getting only warnings , did not notice any other issue yet.

Thanks...
Manas
Comment 12 IBM Bug Proxy 2012-06-25 09:51:03 EDT
------- Comment From kamaleshb@in.ibm.com 2012-06-25 13:40 EDT-------
== Comment: #21 -Sanjeev Patro  <sanpatr1@in.ibm.com> -2012-06-21 03:33:52  ==

We tested with F17 PPC64 GA build, this issue is still persists.
Comment 13 Jeff Layton 2012-06-25 10:11:53 EDT
Looks like a sillyrename cleanup raced with an unlink on the server or something. 

The file got removed and the nlink count on the server went to 0 and the client updated the inode attributes. Later, when we went to clean up the sillyrename we tried to decrement the nlink counter but it was already 0 and we got this warning.

The right fix is probably to make the client not use drop_nlink on a delete. Simply mark the attribute cache as needing to be updated afterward. That has implications for how the inode cache is managed though and will need to be considered carefully.
Comment 14 Jeff Layton 2012-10-09 10:04:56 EDT
*** Bug 854050 has been marked as a duplicate of this bug. ***
Comment 15 Jeff Layton 2012-11-14 09:55:09 EST
Resetting the severity to "low". This is not an urgent issue. The warning is harmless, though it might look a little scary...
Comment 16 Jeff Layton 2012-11-28 11:39:50 EST
*** Bug 873510 has been marked as a duplicate of this bug. ***
Comment 17 Josh Boyer 2012-12-03 12:51:26 EST
*** Bug 881933 has been marked as a duplicate of this bug. ***
Comment 18 Josh Boyer 2012-12-13 08:21:49 EST
*** Bug 886775 has been marked as a duplicate of this bug. ***
Comment 19 Jeff Layton 2012-12-17 09:53:50 EST
Looks like we may have a fix queueing up for 3.8:

    https://patchwork.kernel.org/patch/1880501/
    https://patchwork.kernel.org/patch/1880511/

...and somewhat related:

    https://patchwork.kernel.org/patch/1881281/
Comment 20 Josh Boyer 2012-12-17 09:58:01 EST
(In reply to comment #19)
> Looks like we may have a fix queueing up for 3.8:
> 
>     https://patchwork.kernel.org/patch/1880501/
>     https://patchwork.kernel.org/patch/1880511/
> 
> ...and somewhat related:
> 
>     https://patchwork.kernel.org/patch/1881281/

None of them appear to be CC'd to stable.... is that something that should happen?
Comment 21 Jeff Layton 2012-12-17 10:11:22 EST
This warning is scary-looking and somewhat of a nuisance, but it's mostly harmless. 3.8 seems fine to me...
Comment 22 Jeff Layton 2013-01-03 08:51:39 EST
Since this is in 3.8, I'm going to go ahead and close with a resolution of UPSTREAM.