Created attachment 892756 [details]
Test script that catches the bug. (To be run in upstream regression test suite)

Description of problem:
With /var/lib/glusterd residing on an xfs partition, glusterd restart fails to restore existing volumes to memory.

[root@localhost glusterfs]# df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/sda1      xfs        49G   29G   21G  58% /           <---- xfs partition
devtmpfs       devtmpfs  3.8G     0  3.8G   0% /dev
tmpfs          tmpfs     3.8G  5.4M  3.8G   1% /dev/shm
tmpfs          tmpfs     3.8G  932K  3.8G   1% /run
tmpfs          tmpfs     3.8G     0  3.8G   0% /sys/fs/cgroup
tmpfs          tmpfs     3.8G  3.1M  3.8G   1% /tmp
/dev/sda2      ext4       96G   68G   24G  74% /home
/dev/sda3      xfs        49G   15G   35G  30% /export1
/dev/sda6      xfs        88G   33M   88G   1% /bricks

[root@localhost glusterfs]# glusterd
[root@localhost glusterfs]# gluster volume create rep replica 2 kritika:/d/backends/{1,2} force
Multiple bricks of a replicate volume are present on the same server. This setup is not optimal.
Do you still want to continue creating the volume?  (y/n) y
gluster v volume create: rep: success: please start the volume to access data
[root@localhost glusterfs]# gluster v i
 
Volume Name: rep
Type: Replicate
Volume ID: e313f8e9-471c-4e87-a8e9-9f0cfd2a2458
Status: Created
Snap Volume: no
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: kritika:/d/backends/1
Brick2: kritika:/d/backends/2

[root@localhost glusterfs]# killall glusterd
[root@localhost glusterfs]# glusterd
gluster v i
[root@localhost glusterfs]# gluster v i
No volumes present



Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
0. Make sure /var/lib/glusterd is on xfs partition. If not, modify glusterd.vol to set working-directory option to a path that lies on xfs partition.
1. Start glusterd
2. Create a volume.
3. Execute 'gluster volume info'. You will see that the created volume is listed in the output.
3. Kill glusterd
4. Restart glusterd.
5. Execute 'gluster volume info' again. The output seen will be 'No volumes present'.

Actual results:
volume info after restarting glusterd fails with the error 'No volumes present'

Expected results:
glusterd MUST restore the volume metadata from backend.

Additional info:

1. The same test works fine when /var/lib/glusterd resides in ext4 fs.
2. Snippet from gdb:

glusterd_store_retrieve_volumes (this=0x646020, snap=0x0) at glusterd-store.c:2754
2754	        int32_t                ret              = -1;
(gdb) n
2755	        char                   path[PATH_MAX]   = {0,};
(gdb) 
2756	        glusterd_conf_t       *priv             = NULL;
(gdb) 
2757	        DIR                   *dir              = NULL;
(gdb) 
2758	        struct dirent         *entry            = NULL;
(gdb) 
2759	        glusterd_volinfo_t    *volinfo          = NULL;
(gdb) 
2761	        GF_ASSERT (this);
(gdb) 
2762	        priv = this->private;
(gdb) 
2764	        GF_ASSERT (priv);
(gdb) 
2766	        if (snap)
(gdb) 
2770	                snprintf (path, PATH_MAX, "%s/%s", priv->workdir,
(gdb) 
2773	        dir = opendir (path);
(gdb) p path
$1 = "/var/lib/glusterd/vols", '\000' <repeats 4073 times>
(gdb) n
2775	        if (!dir) {
(gdb) p dir
$2 = (DIR *) 0x65ddb0
(gdb) n
2780	        glusterd_for_each_entry (entry, dir);
(gdb) n
2782	        while (entry) {
(gdb) p entry->d_name
$3 = "rep", '\000' <repeats 18 times>, "\220\336e\000\000\000\000\000\377\377\377\377\377\377\377\377\000\000\000\000\000\000\000\000\240\336e", '\000' <repeats 29 times>, "\377\377\377\377", '\000' <repeats 20 times>, "\200&,\367\377\177", '\000' <repeats 148 times>
(gdb) p entry->d_type 
$4 = 0 '\000'
(gdb) n
2783	                if ( entry->d_type != DT_DIR ||
(gdb) n


Note: The value of DT_DIR is 4 but in this case the value returned is 0.

3. Excerpts from man page of readdir(3):

       On Linux, the dirent structure is defined as follows:

           struct dirent {
               ino_t          d_ino;       /* inode number */
               off_t          d_off;       /* not an offset; see NOTES */
               unsigned short d_reclen;    /* length of this record */
---------------------------------------------------------------------------
               unsigned char  d_type;      /* type of file; not supported |
                                              by all file system types */ |
---------------------------------------------------------------------------
               char           d_name[256]; /* filename */
           };