Bug 1297203 - readdir false-failure with non-Linux
readdir false-failure with non-Linux
Status: ASSIGNED
Product: GlusterFS
Classification: Community
Component: posix (Show other bugs)
mainline
Unspecified FreeBSD
medium Severity medium
: ---
: ---
Assigned To: Pranith Kumar K
: Triaged
Depends On:
Blocks: 1369447 1369448
  Show dependency treegraph
 
Reported: 2016-01-10 09:30 EST by 2510
Modified: 2016-08-23 08:58 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1369447 1369448 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Experimental Patch for glusterfs-3.6.8. (2.52 KB, patch)
2016-01-10 09:30 EST, 2510
no flags Details | Diff

  None (edit)
Description 2510 2016-01-10 09:30:14 EST
Created attachment 1113329 [details]
Experimental Patch for glusterfs-3.6.8.

Description of problem:
With GlusterFS on FreeBSD and VMware ESXi as NFS client, I encountered the following error and accessing storage takes long time.

[2016-01-10 20:20:41.485157] E [posix.c:4902:posix_fill_readdir] 0-gv0-posix: seekdir(0x5) failed on dir=0x806813940: Invalid argument (offset reused from another DIR * structure?)
[2016-01-10 20:20:41.485283] I [server-rpc-fops.c:1882:server_readdirp_cbk] 0-gv0-server: 36451: READDIRP -2 (d3608328-6167-42fe-8ec8-e6cde384e1ab) ==> (Invalid argument)

How reproducible:
Always.

Steps to Reproduce:
1. Build on FreeBSD 10.1-amd64 environment.
2. Create a single-brick volume on UFS or ZFS pool.
3. Connect from VMware ESXi 5.1 to GlusterFS, as NFS storage.

Actual results:
Got the errors and accessing storage from ESXi takes too long.

Expected results:
No errors logged.

Additional info:
Reading GlusterFS posix storage code I found two problems.
One is __posix_fd_ctx_get (in xlators/storage/posix/src/posix-helpers.c) does not set pfd->dir_eof, and another one is posix_fill_readdir does not check whether pfd->dir_eof is set.

dir_eof is added for NetBSD port, with bug 1129939 and review http://review.gluster.org/8926.
Comment 1 Pranith Kumar K 2016-06-15 03:20:25 EDT
hi,
     Is this bug still giving you a problem. I do not have any FreeBSD machines, but if you could help, I will be happy to work with you to fix this issue correctly for you. Sorry was busy with other commitments, so couldn't spend time on this issue.

Pranith
Comment 2 2510 2016-06-15 08:58:30 EDT
Hello,

Yes, still it is a 'problem' for me.


Recently I found more important problem that glusterfs reuses a value ('cookie') returned by telldir for another DIR (opendir'ed for same directory), when accessing through gluster's NFS.
This problem also blocks me from using glusterfs.

I'm working for these bugs on GitHub. Here is a patch fixes problem above, but it is only for FreeBSD+UFS, and it does not fix the root problem.

https://github.com/2510/glusterfs-freebsd/blob/develop/patches/glusterfs-3.6.8.patch3
Comment 3 Pranith Kumar K 2016-06-15 10:16:16 EDT
Will it be possible for you to collaborate with me to work together to find the Root cause and find a fix that works for you as well? Do you think you can share some test machines so that we can work together on those machines to fix this? I mainly develop on Linux. Do you think it is possible to recreate this bug on a VM with freeBSD?

Pranith
Comment 4 2510 2016-06-15 10:31:38 EDT
Yes, it is reproducible on a VM, but we need a test NFS client that sends readdirs.
Please wait for a while. I'll prepare the machine.
Comment 5 Pranith Kumar K 2016-06-15 10:36:09 EDT
Hey,
    I work in India TimeZone. It is 8PM here and I have some personal work now. Do you mind if we catch up on this tomorrow? Please let me know your timezone as well so that we can find a time that works for both of us.

Pranith
Comment 6 2510 2016-06-15 10:59:30 EDT
Hello.

I am in Asia/Tokyo(Japan) timezone (UTC+9), and I am working for this problem personally. (not for corporation/business works)

And, okay, preparing a dev VM will take some more time.
Can you send me SSH public key so I can set up an account for you?

Thanks,
2510
Comment 7 2510 2016-06-16 08:45:58 EDT
A dev VM is ready.
Comment 8 Niklaas Baudet von Gersdorff 2016-07-18 03:06:07 EDT
I run into the same and other problems when using gluster on FreeBSD. At some point accessing the gluster becomes very slow and my systems slow down tremendously. Have a look at this log: http://sprunge.us/MMhM

On the contrary to the issue mentioned above, I disabled NFS because it didn't work either on FreeBSD. I already filed a bug report on FreeBSD's bugzilla: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209752
Comment 9 Pranith Kumar K 2016-07-18 06:31:52 EDT
(In reply to 2510 from comment #7)
> A dev VM is ready.

I can come online, sorry I missed this email. And because of the update on comment-8 I came to know of this again. Please let me know.
Comment 10 2510 2016-07-24 05:04:10 EDT
(In reply to Pranith Kumar K from comment #9)
> I can come online, sorry I missed this email. And because of the update on
> comment-8 I came to know of this again. Please let me know.

Can you tell me your SSH public key?
I'll create your account.
Comment 11 Pranith Kumar K 2016-07-25 21:57:26 EDT
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkduuGBq++zm/JKYVUcfM6YOqzYp2Dj0ag3OvlkFTXyNZ1QVOoEWuH9MAeF/MlHd14nLvFKSdpI+qr+faY+Wtyt/Za09YnizyMBuEo9hIw307EwynOdfAO8N/PKLAvtsNQ7Xk3UHUfHrvVuJr5qZFs1sWNau67/DBxd3bUO/FUl3FZoZqWg3/qsG8ZTCVEPc4N0qY9xiDFxgDh81lmK8t24S8d9RfMrKtpPbSe75HW1CxqM6AGLpQtDscIydGqmRYYcYSn9box4T3erbVxNpcpSlk6K1akMJhbuNoEbDfD7n4t8X/BLj/h3gJIUTlrXnpPj+hluiHDmeBlhu7a7ctd pk@dhcp35-190.lab.eng.blr.redhat.com
Comment 12 2510 2016-07-27 09:21:50 EDT
Okay, I created your account and VM is ready for dev.

------------------------------------------------------------------------

* ssh pk@153.126.154.119 with your key.

* Here's my work. You may copy to your home directory.

/home/2510/glusterfs-3.6.8-orig
  - Original glusterfs-3.6.8, without any patches.
/home/2510/glusterfs-3.6.8-patched
  - with some patches, from /home/2510/glusterfs-freebsd/patches/*

* You can use sudo (for mount/umount, install/start/stop glusterfs)
* To mount or umount NFS directory:

$ sudo mount -t nfs 10.0.0.1:/testvol /mnt/testvol
$ sudo umount /mnt/testvol

* To start/stop glusterfs service:

$ sudo service start glusterfsd
$ sudo killall glusterd glusterfsd

* This VM is built for this ticket.
  I do not use for other purposes.

* Device for a brick is /dev/vtbd0p4, and mounted on /mnt/brick.

------------------------------------------------------------------------

Currently, I suspect that glusterfs NFS reuses cookie returned by seekdir over different DIR.
(It means the ticket my subject, the false-positive, is wrong)

This can be tested by following:

1) Install glusterfs and run.
2) Create a volume (with a brick), then mount gluster volume via nfs.

# mkdir /mnt/testvol
# mount -t nfs 10.0.0.1:/testvol /mnt/testvol

3) Create many files on it.

# sudo sh -c 'for i in `seq 0 1 200`; do touch /mnt/testvol/$i; done'

4) List directory entries.

# ls /mnt/testvol

With original glusterfs, only 51 entries are listed. (entries numbered 48 to 200 are missing)
With patched glusterfs, all entries are listed.
Comment 13 Pranith Kumar K 2016-07-29 14:18:58 EDT
(In reply to 2510 from comment #12)
> Okay, I created your account and VM is ready for dev.
> 
> ------------------------------------------------------------------------
> 
> * ssh pk@153.126.154.119 with your key.
> 
> * Here's my work. You may copy to your home directory.
> 
> /home/2510/glusterfs-3.6.8-orig
>   - Original glusterfs-3.6.8, without any patches.
> /home/2510/glusterfs-3.6.8-patched
>   - with some patches, from /home/2510/glusterfs-freebsd/patches/*
> 
> * You can use sudo (for mount/umount, install/start/stop glusterfs)
> * To mount or umount NFS directory:
> 
> $ sudo mount -t nfs 10.0.0.1:/testvol /mnt/testvol
> $ sudo umount /mnt/testvol
> 
> * To start/stop glusterfs service:
> 
> $ sudo service start glusterfsd
> $ sudo killall glusterd glusterfsd
> 
> * This VM is built for this ticket.
>   I do not use for other purposes.
> 
> * Device for a brick is /dev/vtbd0p4, and mounted on /mnt/brick.
> 
> ------------------------------------------------------------------------
> 
> Currently, I suspect that glusterfs NFS reuses cookie returned by seekdir
> over different DIR.
> (It means the ticket my subject, the false-positive, is wrong)
> 
> This can be tested by following:
> 
> 1) Install glusterfs and run.
> 2) Create a volume (with a brick), then mount gluster volume via nfs.
> 
> # mkdir /mnt/testvol
> # mount -t nfs 10.0.0.1:/testvol /mnt/testvol
> 
> 3) Create many files on it.
> 
> # sudo sh -c 'for i in `seq 0 1 200`; do touch /mnt/testvol/$i; done'
> 
> 4) List directory entries.
> 
> # ls /mnt/testvol
> 
> With original glusterfs, only 51 entries are listed. (entries numbered 48 to
> 200 are missing)
> With patched glusterfs, all entries are listed.

Thanks for this VM. I may have to show this problem to some of our NFS devs too. Please keep the VM until this bug is completely fixed. I will let you know once we are done with finding the Root cause

Note You need to log in before you can comment on or make changes to this bug.