From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3) Gecko/20041020 Description of problem: When running the find command on an NFS mounted filesystem, the error "Value too large for defined data type" is returned. The error does not happen for all NFS mounted filesystems. Suspecting large size or number of files is the issue. Could re-create the error on another large NFS mounted filesystem, but not on a smaller one. lnx-gx270-6$ df -h . Filesystem Size Used Avail Use% Mounted on mink:/export/dist 133G 116G 17G 88% /data/cf/dist lnx-gx270-7$ find . -mtime -1 -print find: .: Value too large for defined data type When I downgrade to findutils-4.1.7-25 from the FC2 distribution, the error no longer occurs. I do receive the same error, "Value too large for defined data type", with findutils-4.1.20-3.i386.rpm from FC3-test3. Version-Release number of selected component (if applicable): findutils-4.1.20-7 How reproducible: Always Steps to Reproduce: 1.NFS mount 133 GB filesystem 2.find . -mtime -1 -print 3. Actual Results: find: .: Value too large for defined data type Expected Results: files modified within the last day should have been listed. Additional info:
Tried with a 135Gb NFS filesystem but couldn't reproduce it. 1. What options is the NFS filesystem mounted with? 2. Please attach 'trace' from 'strace find . -mtime 1 -print 2>trace'. Thanks.
Created attachment 107624 [details] strace find . -mtime 1 -print 2>/tmp/trace strace output attached
[root@lnx-gx270 mnt]# mkdir mink [root@lnx-gx270 mnt]# mount mink:/export/dist/linux /mnt/mink [root@lnx-gx270 mnt]# df | grep mink mink:/export/dist/linux 138732992 123869376 14863616 90% /mnt/mink [root@lnx-gx270 mnt]# mount | grep mink mink:/export/dist/linux on /mnt/mink type nfs (rw,addr=137.38.224.3) [root@lnx-gx270 mnt]# cd /mnt/mink [root@lnx-gx270 mink]# find . -mtime -1 -print find: .: Value too large for defined data type
How about 'ltrace find . -mtime 1 -print 2>/tmp/ltrace'? I haven't tracked down what the problem is yet -- the system calls all seem to have succeeded!
Created attachment 107644 [details] ltrace find . -mtime 1 -print 2>/tmp/ltrace find ltrace output is attached
What's 'ls -l' say in that directory?
$ cd /mnt/mink $ ls -l total 71888 drwxr-xr-x 2 distadm sdss_adm 4096 Sep 20 2002 62_apps lrwxr-xr-x 1 root root 13 Sep 20 2002 62_standard_apps -> standard_apps drwxr-xr-x 2 root root 9 Sep 20 2002 62_updates drwxr-xr-x 2 distadm sdss_adm 4096 Apr 2 2003 72_apps drwxr-xr-x 2 root root 4096 Apr 2 2003 72_rpms drwxrwxr-x 2 distadm sdss_adm 4096 Sep 20 2002 72_rpms_NOT_USED drwxr-xr-x 3 root root 4096 Jan 28 2003 72_standard_apps drwxrwxr-x 2 root cfccn 265 Apr 4 2003 72_updates drwxr-xr-x 2 root root 9 Nov 19 2002 80_standard_apps drwxr-xr-x 3 root root 4096 Nov 19 2002 80_standard_apps.org drwxr-xr-x 2 root root 9 Nov 18 2002 80_updates drwxr-xr-x 2 root root 4096 Oct 28 10:29 90_apps drwxr-xr-x 2 root root 4096 Oct 14 2003 90_standard_apps drwxr-xr-x 2 root root 4096 Sep 22 2003 90_standard_apps.jim drwxr-xr-x 6 root root 4096 May 12 2004 90_updates drwxr-xr-x 2 root root 9 May 30 2003 90_updates.jim lrwxr-xr-x 1 root root 7 Sep 20 2002 apps -> 72_apps drwxr-xr-x 2 root root 34 Sep 20 2002 bin drwxr-xr-x 3 root root 21 Jan 17 2003 boot drwxr-xr-x 3 distadm sdss_adm 26 Sep 20 2002 extras drwxr-xr-x 2 root root 199 Aug 12 16:41 fc2_apps drwxr-xr-x 2 root root 4096 Aug 12 13:16 fc2_standard_apps drwxr-xr-x 2 root root 368 Sep 30 14:42 fc2_updates drwxr-xr-x 2 root root 345 Nov 30 15:42 fc3_apps drwxr-xr-x 2 knox root 4096 Nov 12 15:23 fc3_standard_apps drwxr-xr-x 2 knox root 4096 Oct 19 08:39 fc3_standard_apps.O drwxr-xr-x 3 root root 196 Nov 3 08:01 fc3t3_updates drwxr-xr-x 4 root root 264 Nov 30 15:42 fc3_updates drwxr-xr-x 2 root root 108 Oct 14 08:36 fc3_updates.O drwxr-xr-x 7 knox root 87 Nov 30 09:43 ISO_images drwxrwxrwx 5 distadm sdss_adm 16384 Nov 29 09:04 kickstart-hosts -rw-r--r-- 1 root root 73512960 Oct 8 2003 lnx-xni.tar drwxr-xr-x 2 knox root 4096 Sep 24 2003 My90_standard_apps drwxr-xr-x 4 knox root 4096 Oct 10 2003 My90_updates drwxrwxrwx 2 distadm sdss_adm 205 Sep 20 2002 notes drwxr-xr-x 8 distadm sdss_adm 163 Sep 20 2002 propack drwxrwxrwx 18 distadm sdss_adm 368 Nov 29 12:59 redhat drwxrwxr-x 2 distadm sdss_adm 4096 Jan 15 2004 rpms drwxrwxrwx 4 distadm sdss_adm 4096 Sep 27 14:57 src drwxrwxrwx 2 distadm sdss_adm 4096 Sep 20 2002 standard_apps drwxr-xr-x 8 distadm root 103 Sep 20 2002 standard_dist drwxr-xr-x 5 knox root 381 Sep 27 15:28 test drwxr-xr-x 2 distadm root 4096 Sep 20 2002 test_standard_apps drwxrwxrwt 3 root root 337 Jul 25 2003 tmp drwxr-xr-x 2 root root 96 Sep 27 2002 updated_72_rpms drwxr-xr-x 6 knox root 189 Apr 4 2003 updates_c640 drwxr-xr-x 4 root root 93 Mar 10 2003 updates_gx260 drwxr-xr-x 2 knox root 88 Sep 20 2002 VMWare3.0_1455 drwxrwxrwx 4 distadm root 35 Sep 20 2002 xfs -rw-r--r-- 1 root root 953 Dec 5 2001 xircom_serial.c drwxr-xr-x 2 knox root 33 Sep 20 2002 xni $
Thanks. Also, what sort of device is mink, and what underlying filesystem is that export? What does it have in common with the other NFS export you tried? I haven't been able to reproduce this at all, but it seems to be due to readdir() setting errno to EOVERFLOW at some point. Previous versions of findutils did not check errno after readdir() returned, but readdir() is quite entitled to set it even when otherwise appearing successful.
The filesystem mounted from mink is an xlv on an IRIX 6.5.x server. RAID 5 device. Filesystem Type blocks use avail %use Mounted on/dev/xlv/dist xfs 277465984 238976576 38489408 87 /export/dist mink-675# xfs_growfs -n /export/dist meta-data=/export/dist isize=512 agcount=134, agsize=258840 blks data = bsize=4096 blocks=34684416, imaxpct=25 = sunit=8 swidth=8 blks, unwritten=1 naming =version 1 bsize=4096 log =internal bsize=4096 blocks=1168 realtime =none extsz=65536 blocks=0, rtextents=0 -------------------------- The second filesystem that fails is also an xlv xfs type filesystem mounted from an IRIX 6.5.x server. RAID 5 device. [root@lnx-gx270 ~]# mkdir /mnt/ecad [root@lnx-gx270 ~]# mount ecad.engr:/ecad /mnt/ecad [root@lnx-gx270 ~]# cd /mnt/ecad [root@lnx-gx270 ecad]# df -k . Filesystem 1K-blocks Used Available Use% Mounted on ecad.engr:/ecad 174016352 169716192 4300160 98% /mnt/ecad [root@lnx-gx270 ecad]# find . -mtime -1 -print find: .: Value too large for defined data type # xfs_growfs -n /ecad meta-data=/ecad isize=256 agcount=168, agsize=260293 blks data = bsize=4096 blocks=43505086, imaxpct=25 = sunit=0 swidth=0 blks, unwritten=1 naming =version 1 bsize=4096 log =internal bsize=4096 blocks=1000 realtime =none extsz=65536 blocks=0, rtextents=0 ----------------------- Numerous other xlv filesystems that are the same type xlv/xfs/RAID5 mounted from an IRIX server report no error. A find in the following NFS mounted filesystem works OK. [root@lnx-gx270 ecad]# cd [root@lnx-gx270 ~]# mkdir /mnt/bobcat [root@lnx-gx270 ~]# mount bobcat:/export/sw_ibm /mnt/bobcat [root@lnx-gx270 ~]# cd /mnt/bobcat [root@lnx-gx270 bobcat]# df -k . Filesystem 1K-blocks Used Available Use% Mounted on bobcat:/export/sw_ibm 65528000 57470752 8057248 88% /mnt/bobcat [root@lnx-gx270 bobcat]# find . -mtime -1 -print . ./test bobcat-152# xfs_growfs -n /export/sw_ibm meta-data=/export/sw_ibm isize=512 agcount=63, agsize=262144 blks data = bsize=4096 blocks=16384000, imaxpct=25 = sunit=8 swidth=8 blks, unwritten=1 naming =version 2 bsize=4096 log =internal bsize=4096 blocks=2000 realtime =none extsz=65536 blocks=0, rtextents=0
See also: http://savannah.gnu.org/bugs/?func=detailitem&item_id=8455 Could you please try running this in that directory?: ls -R1 | awk '(length > 200) {print length, $0;}' | sort -n | tail What does it say?
[root@lnx-gx270 mink]# cd /mnt/mink [root@lnx-gx270 mink]# pwd /mnt/mink [root@lnx-gx270 mink]# ls -R1 | awk '(length > 200) {print length, $0;}' | sort -n | tail root@lnx-gx270 mink]# Nothing reported...
Can you try capturing the NFS traffic: 1. tcpdump -v -w nfs.pcap -s 200 host mink 2. find /mnt/mink -mtime -1 -print and attaching the nfs.pcap file? Just press control-C on the tcpdump when it's captured the NFS traffic from running 'find'.
Created attachment 107782 [details] tcpdump -v -w /tmp/nfs.pcap -s 200 host mink tcpdump output file is attached
Okay, that looks like NFSv3, which puts an end to the theory that it was a combination of 2Gb files and NFSv2 causing the problem.
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=180065 This seems to indicate that it is in fact a kernel bug.
I've tried 2.6.9-1.698_FC3 and the problem still exists with findutils-4.1.20-7.
kernel 2.6.9-1.724_FC3 still has this problem as well. Is there a status update on this or a workaround I can implement locally?
Created attachment 110904 [details] Simple test program which trips the problem This simple test program fails when run against an Irix 6.5.8 NFS server, but works against an Irix 6.5.24 server.
Created attachment 110905 [details] strace output when run against Irix 6.5.8 NFS server
Created attachment 110906 [details] strace output when run against Irix 6.5.24 NFS server
I looked into the Irix bug database. This has been identified as an incompatiblity issue between the NFS protocol standard and how glibc expects 64bit system calls to work. If the XFS filesystem on the server was created with XFS version 2, the problem does not occur. A technical service bulletin was issued a couple years ago identifying the symptons of this problem and the workaround. I am working with our administrators to update the server that I was having issues with. I will add to this bug the results of further testing.
One more data point. I tested this with a straight bitkeeper export of 2.6.10 and the same problems were seen.
We're seeing the same with an Irix 6.5.19m server. Does anyone know if this is fixed in 6.5.22 (the latest public maintenance release)? Otherwise, workarounds would be appreciated. Thanks!
Using XFS version 2 logs on our 6.5.8m box did work. There was a discussion about this problem on the linux kernel mailing list. http://marc.theaimsgroup.com/?l=linux-kernel&m=110803989703599&w=2
This is not a glibc bug, but depending on how NFS protocol specifies this either Irix NFS server bug, or Linux NFS client code bug. struct dirent's d_off field has type off_t, which is a signed, not unsigned type. Therefore, when widening this to off64_t, it should be sign-extended, not zero-extended. The glibc side is not going to change to work around bugs in kernel or Irix server, wherever it is.
Moving to XFS 2 on the Irix server fixed this for me too.
An update has been released for Fedora Core 3 (kernel-2.6.12-1.1372_FC3) which may contain a fix for your problem. Please update to this new kernel, and report whether or not it fixes your problem. If you have updated to Fedora Core 4 since this bug was opened, and the problem still occurs with the latest updates for that release, please change the version field of this bug to 'fc4'. Thank you.
FWIW, I'm had the same problem with an RHEL4 client and an IRIX 6.5.25m server. "Converting" (backup, remake, restore) the exported filesystem to XFS 2 fixed the problem.
I have the same problem without any NFS being involved. I'm running FC3 with kernel 2.6.11-1.27_FC3, and Apache httpd-20.52-3.1, using only ext3 file systems (no NFS). One of my web directories contains a DVD .iso file larger than 2GB. Attempting to fetch it from the web server results in the client being told that there is a permission error. The error log says: [Thu Sep 01 11:31:22 2005] [error] [client 67.111.12.152] (75)Value too large for defined data type: access to /~eric/software/fedora/fc4/stentz-dvd-i386/FC4-i386-DVD.iso failed I'm updating the kernel now to 2.6.12-1.1376_FC3, and will report here later today on whether it has fixed the problem.
Now running 2.6.12-1.1376_FC3, and the problem still occurs. Same message to log as before.
I'm just curious if making the daemons in the nfs-utils rpm 64bit applications (i.e. defining the __USE_FILE_OFFSET64 compile flag) will help with this problem. In http://people.redhat.com/steved/bz141167 is an nfs-utils rpm that is compiled with __USE_FILE_OFFSET64 flag defined. If your able, please try this experimental rpms to if it helps.
This is a mass-update to all currently open Fedora Core 3 kernel bugs. Fedora Core 3 support has transitioned to the Fedora Legacy project. Due to the limited resources of this project, typically only updates for new security issues are released. As this bug isn't security related, it has been migrated to a Fedora Core 4 bug. Please upgrade to this newer release, and test if this bug is still present there. This bug has been placed in NEEDINFO_REPORTER state. Due to the large volume of inactive bugs in bugzilla, if this bug is still in this state in two weeks time, it will be closed. Should this bug still be relevant after this period, the reporter can reopen the bug at any time. Any other users on the Cc: list of this bug can request that the bug be reopened by adding a comment to the bug. Thank you.
This is a mass-update to all currently open kernel bugs. A new kernel update has been released (Version: 2.6.15-1.1830_FC4) based upon a new upstream kernel release. Please retest against this new kernel, as a large number of patches go into each upstream release, possibly including changes that may address this problem. This bug has been placed in NEEDINFO_REPORTER state. Due to the large volume of inactive bugs in bugzilla, if this bug is still in this state in two weeks time, it will be closed. Should this bug still be relevant after this period, the reporter can reopen the bug at any time. Any other users on the Cc: list of this bug can request that the bug be reopened by adding a comment to the bug. If this bug is a problem preventing you from installing the release this version is filed against, please see bug 169613. Thank you.
Closing per previous comment.