Hide Forgot
Description of problem: ****************************** While doing I/O from cifs client and doing graph switch multiple times, seeing input output error on client and huge warning messages on client logs. [2016-10-19 14:10:07.870224] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a4f5cd8) with refcount(2) found du ring cleanup [2016-10-19 14:10:07.870263] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a543010) with refcount(2) found du ring cleanup [2016-10-19 14:10:07.870301] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a542f54) with refcount(2) found du ring cleanup [2016-10-19 14:10:07.870339] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a542e98) with refcount(2) found du ring cleanup [2016-10-19 14:10:07.870402] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a542ddc) with refcount(2) found du ring cleanup [2016-10-19 14:10:07.870443] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a542d20) with refcount(2) found du ring cleanup [2016-10-19 14:10:07.870481] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a542c64) with refcount(2) found du ring cleanup [2016-10-19 14:10:07.870528] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a542ba8) with refcount(2) found du ring cleanup [2016-10-19 14:10:07.870569] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a542aec) with refcount(2) found du ring cleanup [2016-10-19 14:10:07.870607] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a542a30) with refcount(2) found du ring cleanup [2016-10-19 14:10:07.870670] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a542974) with refcount(2) found du ring cleanup [2016-10-19 14:10:07.870710] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a4f5c1c) with refcount(2) found du ring cleanup [2016-10-19 14:10:07.870748] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a5428b8) with refcount(2) found du ring cleanup [2016-10-19 14:10:07.870787] W [inode.c:1809:inode_table_destroy] (-->/lib64/libgfapi.so.0(glfs_fini+0x40d) [0x7fd4f61f15ed] -->/lib64/libglusterfs.so.0(inode_table_des troy_all+0x51) [0x7fd4f5af85d1] -->/lib64/libglusterfs.so.0(inode_table_destroy+0xd7) [0x7fd4f5af84d7] ) 0-gfapi: Active inode(0x7fd48a5427fc) with refcount(2) found du ring cleanup : Version-Release number of selected component (if applicable): glusterfs-3.8.4-2.26.git0a405a4.el7rhgs.x86_64 How reproducible: With multiple graph switch Steps to Reproduce: 1.As explained in description 2.Command used dd if=/dev/zero of=file2 bs=1G count=1024 3. Actual results: Input output error and huge warning logs. Expected results: there should not be input output errors and huge warning logs generated. Additional info:
I see that the log messages show that there was a fini, was there a disconnect? Could you try graph-switches like quick-read/add-brick etc. ? Is this seen when md-cache is disabled as well?
Tried the test with graph-switches by turning readdir-ahead on and off and quick-read on and off, didn't observed the issue. Will test once again with write-behind and md-cache off and update the bug.
Keeping the needinfo back to Surabhi as we still need the results when md-cache is turned off along with write-behind.
Vivek , Could you please test and update the results as per comment#3
Summary of tests ------------------ 1. With mdcache disabled Command used dd ll /mnt/cifs (in a loop) where /mnt/cifs is the mount point write-behind on & off (in a loop) Create deep directory using python script ll /mnt/cifs (in a loop) where /mnt/cifs is the mount point write-behind on & off (in a loop) 2. With md-cache enabled Command used dd ll /mnt/cifs (in a loop) where /mnt/cifs is the mount point write-behind on & off (in a loop) Create deep directory using python script ll /mnt/cifs (in a loop) where /mnt/cifs is the mount point write-behind on & off (in a loop) 3. Distaf test cases that does a graph switch ON & OFF while dd command is ran. Ran this on a loop Well trying all the above test scenarios the original bug related to IO error was NOT seen. But i was hitting issue with the above scenario where the dd command was getting hang and for which i have reported another bug #1396449. So i suspect that it might happen that because of the hang issue of the dd command the IO error is not discovered and that we can wait for the fix to #1396449 and then we can re-try the above steps to reproduce the IO error.
Is it possible to attach statedump of smbd process when I/O is hung?
As per the triaging we all have the agreement that this BZ has to be fixed in rhgs-3.2.0. Providing qa_ack
Versions: -------- glusterfs-3.8.4-9.el7rhgs.x86_64 samba-client-4.4.6-3.el7rhgs.x86_64 1. With md-cache enabled Command used dd ll /mnt/cifs (in a loop) where /mnt/cifs is the mount point write-behind on & off (in a loop) Command used dd ll /mnt/cifs (in a loop) where /mnt/cifs is the mount point readdir-ahead on & off (in a loop) Create deep directory using python script ll /mnt/cifs (in a loop) where /mnt/cifs is the mount point write-behind on & off (in a loop) 2. Distaf test cases that does a graph switch ON & OFF while dd command is ran. Ran this on a loop No Input Output errors or huge warning messages were encountered.