Description of problem: Upgrading to 3.7.16-1 on Debian breaks Fuse clients to connect to glusterfsd. Version-Release number of selected component (if applicable): 3.7.16-1 How reproducible: easy Steps to Reproduce: 1. Upgrade from 3.7.15-1 to 3.7.16-1 2. Mount your GlusterFS Fuse Mount 3. See them freeze forever on access Actual results: Mounting success but accessing fuse mount freeze. Expected results: Access?! Additional info:
Initially after downgrading back to 3.6.15-1 and restarting glusterfsd the fuse mounts are accessable again.
Jules, what version are the clients? 3.6.16 or 3.6.15? It'll be helpful if you could also provide the client logs, which will be of great help for the initial debugging.
Clients are v3.6.16 Here is the glusterfs client log: [2016-10-24 14:17:19.756677] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.16 (args: /usr/sbin/glusterfs --acl --volfile-server=node2d --volfile-server=node2c --volfile-id=/netshare /storage) [2016-10-24 14:17:19.910945] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-10-24 14:17:20.274477] I [graph.c:269:gf_add_cmdline_options] 0-netshare-md-cache: adding option 'cache-posix-acl' for volume 'netshare-md-cache' with value 'true' Final graph: +------------------------------------------------------------------------------+ 1: volume netshare-client-0 2: type protocol/client 3: option ping-timeout 5 4: option remote-host node2c 5: option remote-subvolume /storage/gfs/netshare 6: option transport-type socket 7: option send-gids true 8: end-volume 9: 10: volume netshare-client-1 11: type protocol/client 12: option ping-timeout 5 13: option remote-host node2d 14: option remote-subvolume /storage/gfs/netshare 15: option transport-type socket 16: option send-gids true 17: end-volume 18: 19: volume netshare-replicate-0 20: type cluster/replicate 21: option background-self-heal-count 20 22: option metadata-self-heal on 23: option data-self-heal on 24: option entry-self-heal on 25: option data-self-heal-window-size 2 26: option self-heal-readdir-size 2KB 27: subvolumes netshare-client-0 netshare-client-1 28: end-volume 29: 30: volume netshare-dht 31: type cluster/distribute 32: option min-free-disk 5% 33: option readdir-optimize on 34: subvolumes netshare-replicate-0 35: end-volume 36: 37: volume netshare-write-behind 38: type performance/write-behind 39: subvolumes netshare-dht 40: end-volume 41: 42: volume netshare-read-ahead 43: type performance/read-ahead 44: subvolumes netshare-write-behind 45: end-volume 46: 47: volume netshare-io-cache 48: type performance/io-cache 49: option cache-size 2GB 50: subvolumes netshare-read-ahead 51: end-volume 52: 53: volume netshare-quick-read 54: type performance/quick-read 55: option cache-size 2GB 56: subvolumes netshare-io-cache 57: end-volume 58: 59: volume netshare-open-behind 60: type performance/open-behind 61: subvolumes netshare-quick-read 62: end-volume 63: 64: volume netshare-md-cache 65: type performance/md-cache 66: option cache-posix-acl true 67: subvolumes netshare-open-behind 68: end-volume 69: 70: volume netshare 71: type debug/io-stats 72: option log-level WARNING 73: option latency-measurement off 74: option count-fop-hits off 75: subvolumes netshare-md-cache 76: end-volume 77: 78: volume posix-acl-autoload 79: type system/posix-acl 80: subvolumes netshare 81: end-volume 82: 83: volume meta-autoload 84: type meta 85: subvolumes posix-acl-autoload 86: end-volume 87: +------------------------------------------------------------------------------+ [2016-10-24 14:17:30.118453] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1355: LOOKUP() /conf/mail/amavis/conf.d/01-debian => -1 (Permission denied) [2016-10-24 14:17:30.118646] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1357: LOOKUP() /conf/mail/amavis/conf.d/01-debian => -1 (Permission denied) [2016-10-24 14:17:30.118811] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1358: LOOKUP() /conf/mail/amavis/conf.d/05-domain_id => -1 (Permission denied) [2016-10-24 14:17:30.118942] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1360: LOOKUP() /conf/mail/amavis/conf.d/05-domain_id => -1 (Permission denied) [2016-10-24 14:17:30.119074] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1361: LOOKUP() /conf/mail/amavis/conf.d/05-node_id => -1 (Permission denied) [2016-10-24 14:17:30.119199] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1363: LOOKUP() /conf/mail/amavis/conf.d/05-node_id => -1 (Permission denied) [2016-10-24 14:17:30.119364] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1364: LOOKUP() /conf/mail/amavis/conf.d/15-av_scanners => -1 (Permission denied) [2016-10-24 14:17:30.119500] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1366: LOOKUP() /conf/mail/amavis/conf.d/15-av_scanners => -1 (Permission denied) [2016-10-24 14:17:30.119632] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1367: LOOKUP() /conf/mail/amavis/conf.d/15-content_filter_mode => -1 (Permission denied) [2016-10-24 14:17:30.119771] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1369: LOOKUP() /conf/mail/amavis/conf.d/15-content_filter_mode => -1 (Permission denied) [2016-10-24 14:17:30.119930] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1370: LOOKUP() /conf/mail/amavis/conf.d/20-debian_defaults => -1 (Permission denied) [2016-10-24 14:17:30.120070] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1372: LOOKUP() /conf/mail/amavis/conf.d/20-debian_defaults => -1 (Permission denied) [2016-10-24 14:17:30.120220] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1373: LOOKUP() /conf/mail/amavis/conf.d/25-amavis_helpers => -1 (Permission denied) [2016-10-24 14:17:30.120359] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1375: LOOKUP() /conf/mail/amavis/conf.d/25-amavis_helpers => -1 (Permission denied) [2016-10-24 14:17:30.120490] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1376: LOOKUP() /conf/mail/amavis/conf.d/30-template_localization => -1 (Permission denied) [2016-10-24 14:17:30.120615] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1378: LOOKUP() /conf/mail/amavis/conf.d/30-template_localization => -1 (Permission denied) [2016-10-24 14:17:30.120785] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1379: LOOKUP() /conf/mail/amavis/conf.d/50-user => -1 (Permission denied) [2016-10-24 14:17:30.120938] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1381: LOOKUP() /conf/mail/amavis/conf.d/50-user => -1 (Permission denied) [2016-10-25 01:14:14.036228] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 129916: OPEN() /conf/mail/amavis/conf.d/01-debian => -1 (Permission denied) [2016-10-25 01:14:14.036409] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 129917: LOOKUP() /conf/mail/amavis/conf.d/01-debian.moved => -1 (Permission denied) [2016-10-25 01:14:14.036528] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 129918: OPEN() /conf/mail/amavis/conf.d/05-domain_id => -1 (Permission denied) [2016-10-25 01:14:14.036601] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 129919: LOOKUP() /conf/mail/amavis/conf.d/05-domain_id.moved => -1 (Permission denied) [2016-10-25 01:14:14.036705] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 129920: OPEN() /conf/mail/amavis/conf.d/05-node_id => -1 (Permission denied) [2016-10-25 01:14:14.036784] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 129921: LOOKUP() /conf/mail/amavis/conf.d/05-node_id.moved => -1 (Permission denied) [2016-10-25 01:14:14.036880] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 129922: OPEN() /conf/mail/amavis/conf.d/15-av_scanners => -1 (Permission denied) [2016-10-25 01:14:14.036943] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 129923: LOOKUP() /conf/mail/amavis/conf.d/15-av_scanners.moved => -1 (Permission denied) [2016-10-25 01:14:14.037045] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 129924: OPEN() /conf/mail/amavis/conf.d/15-content_filter_mode => -1 (Permission denied) [2016-10-25 01:14:14.037111] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 129925: LOOKUP() /conf/mail/amavis/conf.d/15-content_filter_mode.moved => -1 (Permission denied) [2016-10-25 01:14:14.037220] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 129926: OPEN() /conf/mail/amavis/conf.d/20-debian_defaults => -1 (Permission denied) [2016-10-25 01:14:14.037297] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 129927: LOOKUP() /conf/mail/amavis/conf.d/20-debian_defaults.moved => -1 (Permission denied) [2016-10-25 01:14:14.037386] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 129928: OPEN() /conf/mail/amavis/conf.d/25-amavis_helpers => -1 (Permission denied) [2016-10-25 01:14:14.037491] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 129929: LOOKUP() /conf/mail/amavis/conf.d/25-amavis_helpers.moved => -1 (Permission denied) [2016-10-25 01:14:14.037601] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 129930: OPEN() /conf/mail/amavis/conf.d/30-template_localization => -1 (Permission denied) [2016-10-25 01:14:14.037672] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 129931: LOOKUP() /conf/mail/amavis/conf.d/30-template_localization.moved => -1 (Permission denied) [2016-10-25 01:14:14.037830] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 129932: OPEN() /conf/mail/amavis/conf.d/50-user => -1 (Permission denied) [2016-10-25 01:14:14.037900] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 129933: LOOKUP() /conf/mail/amavis/conf.d/50-user.moved => -1 (Permission denied) [2016-10-26 01:39:05.352399] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 402813: OPEN() /conf/mail/amavis/conf.d/01-debian => -1 (Permission denied) [2016-10-26 01:39:05.352555] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 402814: LOOKUP() /conf/mail/amavis/conf.d/01-debian.moved => -1 (Permission denied) [2016-10-26 01:39:05.352693] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 402815: OPEN() /conf/mail/amavis/conf.d/05-domain_id => -1 (Permission denied) [2016-10-26 01:39:05.352792] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 402816: LOOKUP() /conf/mail/amavis/conf.d/05-domain_id.moved => -1 (Permission denied) [2016-10-26 01:39:05.352910] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 402817: OPEN() /conf/mail/amavis/conf.d/05-node_id => -1 (Permission denied) [2016-10-26 01:39:05.352993] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 402818: LOOKUP() /conf/mail/amavis/conf.d/05-node_id.moved => -1 (Permission denied) [2016-10-26 01:39:05.353091] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 402819: OPEN() /conf/mail/amavis/conf.d/15-av_scanners => -1 (Permission denied) [2016-10-26 01:39:05.353178] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 402820: LOOKUP() /conf/mail/amavis/conf.d/15-av_scanners.moved => -1 (Permission denied) [2016-10-26 01:39:05.353293] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 402821: OPEN() /conf/mail/amavis/conf.d/15-content_filter_mode => -1 (Permission denied) [2016-10-26 01:39:05.353373] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 402822: LOOKUP() /conf/mail/amavis/conf.d/15-content_filter_mode.moved => -1 (Permission denied) [2016-10-26 01:39:05.353482] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 402823: OPEN() /conf/mail/amavis/conf.d/20-debian_defaults => -1 (Permission denied) [2016-10-26 01:39:05.353584] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 402824: LOOKUP() /conf/mail/amavis/conf.d/20-debian_defaults.moved => -1 (Permission denied) [2016-10-26 01:39:05.353675] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 402825: OPEN() /conf/mail/amavis/conf.d/25-amavis_helpers => -1 (Permission denied) [2016-10-26 01:39:05.353765] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 402826: LOOKUP() /conf/mail/amavis/conf.d/25-amavis_helpers.moved => -1 (Permission denied) [2016-10-26 01:39:05.353897] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 402827: OPEN() /conf/mail/amavis/conf.d/30-template_localization => -1 (Permission denied) [2016-10-26 01:39:05.353976] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 402828: LOOKUP() /conf/mail/amavis/conf.d/30-template_localization.moved => -1 (Permission denied) [2016-10-26 01:39:05.354072] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 402829: OPEN() /conf/mail/amavis/conf.d/50-user => -1 (Permission denied) [2016-10-26 01:39:05.354149] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 402830: LOOKUP() /conf/mail/amavis/conf.d/50-user.moved => -1 (Permission denied) [2016-10-27 00:57:51.509142] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 666088: OPEN() /conf/mail/amavis/conf.d/01-debian => -1 (Permission denied) [2016-10-27 00:57:51.509286] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 666089: LOOKUP() /conf/mail/amavis/conf.d/01-debian.moved => -1 (Permission denied) [2016-10-27 00:57:51.509364] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 666090: OPEN() /conf/mail/amavis/conf.d/05-domain_id => -1 (Permission denied) [2016-10-27 00:57:51.509434] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 666091: LOOKUP() /conf/mail/amavis/conf.d/05-domain_id.moved => -1 (Permission denied) [2016-10-27 00:57:51.509493] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 666092: OPEN() /conf/mail/amavis/conf.d/05-node_id => -1 (Permission denied) [2016-10-27 00:57:51.509548] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 666093: LOOKUP() /conf/mail/amavis/conf.d/05-node_id.moved => -1 (Permission denied) [2016-10-27 00:57:51.509605] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 666094: OPEN() /conf/mail/amavis/conf.d/15-av_scanners => -1 (Permission denied) [2016-10-27 00:57:51.509659] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 666095: LOOKUP() /conf/mail/amavis/conf.d/15-av_scanners.moved => -1 (Permission denied) [2016-10-27 00:57:51.509727] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 666096: OPEN() /conf/mail/amavis/conf.d/15-content_filter_mode => -1 (Permission denied) [2016-10-27 00:57:51.509782] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 666097: LOOKUP() /conf/mail/amavis/conf.d/15-content_filter_mode.moved => -1 (Permission denied) [2016-10-27 00:57:51.509855] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 666098: OPEN() /conf/mail/amavis/conf.d/20-debian_defaults => -1 (Permission denied) [2016-10-27 00:57:51.509920] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 666099: LOOKUP() /conf/mail/amavis/conf.d/20-debian_defaults.moved => -1 (Permission denied) [2016-10-27 00:57:51.509984] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 666100: OPEN() /conf/mail/amavis/conf.d/25-amavis_helpers => -1 (Permission denied) [2016-10-27 00:57:51.510051] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 666101: LOOKUP() /conf/mail/amavis/conf.d/25-amavis_helpers.moved => -1 (Permission denied) [2016-10-27 00:57:51.510127] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 666102: OPEN() /conf/mail/amavis/conf.d/30-template_localization => -1 (Permission denied) [2016-10-27 00:57:51.510181] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 666103: LOOKUP() /conf/mail/amavis/conf.d/30-template_localization.moved => -1 (Permission denied) [2016-10-27 00:57:51.510235] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 666104: OPEN() /conf/mail/amavis/conf.d/50-user => -1 (Permission denied) [2016-10-27 00:57:51.510288] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 666105: LOOKUP() /conf/mail/amavis/conf.d/50-user.moved => -1 (Permission denied) [2016-10-28 01:43:15.116415] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 942585: OPEN() /conf/mail/amavis/conf.d/01-debian => -1 (Permission denied) [2016-10-28 01:43:15.116593] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 942586: LOOKUP() /conf/mail/amavis/conf.d/01-debian.moved => -1 (Permission denied) [2016-10-28 01:43:15.116717] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 942587: OPEN() /conf/mail/amavis/conf.d/05-domain_id => -1 (Permission denied) [2016-10-28 01:43:15.116789] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 942588: LOOKUP() /conf/mail/amavis/conf.d/05-domain_id.moved => -1 (Permission denied) [2016-10-28 01:43:15.116866] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 942589: OPEN() /conf/mail/amavis/conf.d/05-node_id => -1 (Permission denied) [2016-10-28 01:43:15.116934] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 942590: LOOKUP() /conf/mail/amavis/conf.d/05-node_id.moved => -1 (Permission denied) [2016-10-28 01:43:15.117029] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 942591: OPEN() /conf/mail/amavis/conf.d/15-av_scanners => -1 (Permission denied) [2016-10-28 01:43:15.117121] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 942592: LOOKUP() /conf/mail/amavis/conf.d/15-av_scanners.moved => -1 (Permission denied) [2016-10-28 01:43:15.117199] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 942593: OPEN() /conf/mail/amavis/conf.d/15-content_filter_mode => -1 (Permission denied) [2016-10-28 01:43:15.117267] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 942594: LOOKUP() /conf/mail/amavis/conf.d/15-content_filter_mode.moved => -1 (Permission denied) [2016-10-28 01:43:15.117338] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 942595: OPEN() /conf/mail/amavis/conf.d/20-debian_defaults => -1 (Permission denied) [2016-10-28 01:43:15.117410] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 942596: LOOKUP() /conf/mail/amavis/conf.d/20-debian_defaults.moved => -1 (Permission denied) [2016-10-28 01:43:15.117484] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 942597: OPEN() /conf/mail/amavis/conf.d/25-amavis_helpers => -1 (Permission denied) [2016-10-28 01:43:15.117563] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 942598: LOOKUP() /conf/mail/amavis/conf.d/25-amavis_helpers.moved => -1 (Permission denied) [2016-10-28 01:43:15.117673] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 942599: OPEN() /conf/mail/amavis/conf.d/30-template_localization => -1 (Permission denied) [2016-10-28 01:43:15.117741] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 942600: LOOKUP() /conf/mail/amavis/conf.d/30-template_localization.moved => -1 (Permission denied) [2016-10-28 01:43:15.117812] W [fuse-bridge.c:988:fuse_fd_cbk] 0-glusterfs-fuse: 942601: OPEN() /conf/mail/amavis/conf.d/50-user => -1 (Permission denied) [2016-10-28 01:43:15.117877] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 942602: LOOKUP() /conf/mail/amavis/conf.d/50-user.moved => -1 (Permission denied) [2016-10-28 12:33:44.461823] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7f04c01190a4] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f04c137a725] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x7f04c137a5a7] ) 0-: received signum (15), shutting down
[2016-10-28 12:52:14.528621] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.16 (args: /usr/sbin/glusterfs --acl --volfile-server=node2d --volfile-server=node2c --volfile-id=/netshare /storage) [2016-10-28 12:52:14.956640] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-10-28 12:52:15.848575] I [graph.c:269:gf_add_cmdline_options] 0-netshare-md-cache: adding option 'cache-posix-acl' for volume 'netshare-md-cache' with value 'true' Final graph: +------------------------------------------------------------------------------+ 1: volume netshare-client-0 2: type protocol/client 3: option ping-timeout 5 4: option remote-host node2c 5: option remote-subvolume /storage/gfs/netshare 6: option transport-type socket 7: option send-gids true 8: end-volume 9: 10: volume netshare-client-1 11: type protocol/client 12: option ping-timeout 5 13: option remote-host node2d 14: option remote-subvolume /storage/gfs/netshare 15: option transport-type socket 16: option send-gids true 17: end-volume 18: 19: volume netshare-replicate-0 20: type cluster/replicate 21: option background-self-heal-count 20 22: option metadata-self-heal on 23: option data-self-heal on 24: option entry-self-heal on 25: option data-self-heal-window-size 2 26: option self-heal-readdir-size 2KB 27: subvolumes netshare-client-0 netshare-client-1 28: end-volume 29: 30: volume netshare-dht 31: type cluster/distribute 32: option min-free-disk 5% 33: option readdir-optimize on 34: subvolumes netshare-replicate-0 35: end-volume 36: 37: volume netshare-write-behind 38: type performance/write-behind 39: subvolumes netshare-dht 40: end-volume 41: 42: volume netshare-read-ahead 43: type performance/read-ahead 44: subvolumes netshare-write-behind 45: end-volume 46: 47: volume netshare-io-cache 48: type performance/io-cache 49: option cache-size 2GB 50: subvolumes netshare-read-ahead 51: end-volume 52: 53: volume netshare-quick-read 54: type performance/quick-read 55: option cache-size 2GB 56: subvolumes netshare-io-cache 57: end-volume 58: 59: volume netshare-open-behind 60: type performance/open-behind 61: subvolumes netshare-quick-read 62: end-volume 63: 64: volume netshare-md-cache 65: type performance/md-cache 66: option cache-posix-acl true 67: subvolumes netshare-open-behind 68: end-volume 69: 70: volume netshare 71: type debug/io-stats 72: option log-level WARNING 73: option latency-measurement off 74: option count-fop-hits off 75: subvolumes netshare-md-cache 76: end-volume 77: 78: volume posix-acl-autoload 79: type system/posix-acl 80: subvolumes netshare 81: end-volume 82: 83: volume meta-autoload 84: type meta 85: subvolumes posix-acl-autoload 86: end-volume 87: +------------------------------------------------------------------------------+ [2016-10-28 12:53:03.346269] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1866: LOOKUP() /conf/mail/amavis/conf.d/01-debian => -1 (Permission denied) [2016-10-28 12:53:03.346560] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1867: LOOKUP() /conf/mail/amavis/conf.d/01-debian => -1 (Permission denied) [2016-10-28 12:53:03.346967] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1869: LOOKUP() /conf/mail/amavis/conf.d/05-domain_id => -1 (Permission denied) [2016-10-28 12:53:03.347103] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1871: LOOKUP() /conf/mail/amavis/conf.d/05-domain_id => -1 (Permission denied) [2016-10-28 12:53:03.347258] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1872: LOOKUP() /conf/mail/amavis/conf.d/05-node_id => -1 (Permission denied) [2016-10-28 12:53:03.347377] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1873: LOOKUP() /conf/mail/amavis/conf.d/05-node_id => -1 (Permission denied) [2016-10-28 12:53:03.347516] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1875: LOOKUP() /conf/mail/amavis/conf.d/15-av_scanners => -1 (Permission denied) [2016-10-28 12:53:03.347609] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1876: LOOKUP() /conf/mail/amavis/conf.d/15-av_scanners => -1 (Permission denied) [2016-10-28 12:53:03.347723] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1878: LOOKUP() /conf/mail/amavis/conf.d/15-content_filter_mode => -1 (Permission denied) [2016-10-28 12:53:03.347809] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1879: LOOKUP() /conf/mail/amavis/conf.d/15-content_filter_mode => -1 (Permission denied) [2016-10-28 12:53:03.347910] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1881: LOOKUP() /conf/mail/amavis/conf.d/20-debian_defaults => -1 (Permission denied) [2016-10-28 12:53:03.348037] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1883: LOOKUP() /conf/mail/amavis/conf.d/20-debian_defaults => -1 (Permission denied) [2016-10-28 12:53:03.348161] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1884: LOOKUP() /conf/mail/amavis/conf.d/25-amavis_helpers => -1 (Permission denied) [2016-10-28 12:53:03.348287] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1886: LOOKUP() /conf/mail/amavis/conf.d/25-amavis_helpers => -1 (Permission denied) [2016-10-28 12:53:03.348397] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1887: LOOKUP() /conf/mail/amavis/conf.d/30-template_localization => -1 (Permission denied) [2016-10-28 12:53:03.348500] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1889: LOOKUP() /conf/mail/amavis/conf.d/30-template_localization => -1 (Permission denied) [2016-10-28 12:53:03.348626] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1890: LOOKUP() /conf/mail/amavis/conf.d/50-user => -1 (Permission denied) [2016-10-28 12:53:03.348743] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 1892: LOOKUP() /conf/mail/amavis/conf.d/50-user => -1 (Permission denied) [2016-10-28 13:12:37.048137] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-netshare-client-0: server 10.1.20.1:49159 has not responded in the last 5 seconds, disconnecting. [2016-10-28 13:12:37.048571] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-10-28 13:12:32.020037 (xid=0x2110) [2016-10-28 13:12:37.048601] W [MSGID: 114031] [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-netshare-client-0: remote operation failed. Path: /conf (b28cb9a1-1cfd-4296-9595-874cc6cc8056) [Transport endpoint is not connected] [2016-10-28 13:12:37.049956] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2016-10-28 13:12:32.020050 (xid=0x2111) [2016-10-28 13:12:37.049972] W [rpc-clnt-ping.c:208:rpc_clnt_ping_cbk] 0-netshare-client-0: socket disconnected [2016-10-28 13:12:37.050417] W [rpc-clnt.c:1653:rpc_clnt_submit] 0-netshare-client-0: failed to submit rpc-request (XID: 0x2112 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (netshare-client-0) [2016-10-28 13:12:37.050447] W [MSGID: 114031] [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-netshare-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2016-10-28 13:13:43.159065] E [socket.c:2292:socket_connect_finish] 0-netshare-client-0: connection to 10.1.20.1:24007 failed (No route to host) [2016-10-28 13:12:37.051508] W [MSGID: 114031] [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-netshare-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2016-10-28 13:15:18.079711] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-netshare-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2016-10-28 13:17:20.952559] W [MSGID: 108015] [afr-self-heal-entry.c:69:afr_selfheal_entry_delete] 0-netshare-replicate-0: expunging file c5972d7b-17de-48ab-9d5c-b4a4a27c6ae6/dovecot-uidlist (485c47c7-1ebb-4721-b51f-35c28e375b18) on netshare-client-0 [2016-10-28 13:17:24.078183] W [MSGID: 108015] [afr-self-heal-entry.c:69:afr_selfheal_entry_delete] 0-netshare-replicate-0: expunging file a0e3d62a-6478-47e7-85f8-cc587d3a2b7a/dovecot-uidlist (98155ec4-3cfe-4208-badc-83c566956946) on netshare-client-0 [2016-10-28 13:17:28.015808] W [MSGID: 114028] [client-lk.c:369:delete_granted_locks_owner] 0-netshare-client-0: fdctx not valid [Invalid argument] [2016-10-28 13:18:27.379573] W [MSGID: 108015] [afr-self-heal-entry.c:69:afr_selfheal_entry_delete] 0-netshare-replicate-0: expunging file b453d72c-077c-4d47-94d5-baa593fdcf71/1477660254.M841368P2673.mail-01,S=2006,W=2057:2,a (d7d750cc-e354-42cf-866b-ada8aa0895be) on netshare-client-0 [2016-10-28 13:23:01.302461] W [fuse-resolve.c:66:fuse_resolve_entry_cbk] 0-fuse: 2e556bde-2555-455f-86f2-29616fb6ef0d/dovecot-uidlist: failed to resolve (Stale file handle) [2016-10-28 13:33:02.154725] W [fuse-resolve.c:66:fuse_resolve_entry_cbk] 0-fuse: 9d7096c6-31e4-4234-9412-858a7a0af57c/dovecot.index: failed to resolve (Stale file handle) [2016-10-28 13:33:02.311722] W [fuse-resolve.c:66:fuse_resolve_entry_cbk] 0-fuse: 9d7096c6-31e4-4234-9412-858a7a0af57c/dovecot-keywords: failed to resolve (Stale file handle) [2016-10-29 01:24:09.113425] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155047: LOOKUP() /conf/mail/amavis/conf.d/01-debian => -1 (Permission denied) [2016-10-29 01:24:09.113604] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155048: LOOKUP() /conf/mail/amavis/conf.d/01-debian => -1 (Permission denied) [2016-10-29 01:24:09.113775] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155050: LOOKUP() /conf/mail/amavis/conf.d/05-domain_id => -1 (Permission denied) [2016-10-29 01:24:09.113853] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155051: LOOKUP() /conf/mail/amavis/conf.d/05-domain_id => -1 (Permission denied) [2016-10-29 01:24:09.113936] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155053: LOOKUP() /conf/mail/amavis/conf.d/05-node_id => -1 (Permission denied) [2016-10-29 01:24:09.114011] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155054: LOOKUP() /conf/mail/amavis/conf.d/05-node_id => -1 (Permission denied) [2016-10-29 01:24:09.114101] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155056: LOOKUP() /conf/mail/amavis/conf.d/15-av_scanners => -1 (Permission denied) [2016-10-29 01:24:09.114185] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155057: LOOKUP() /conf/mail/amavis/conf.d/15-av_scanners => -1 (Permission denied) [2016-10-29 01:24:09.114265] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155059: LOOKUP() /conf/mail/amavis/conf.d/15-content_filter_mode => -1 (Permission denied) [2016-10-29 01:24:09.114337] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155060: LOOKUP() /conf/mail/amavis/conf.d/15-content_filter_mode => -1 (Permission denied) [2016-10-29 01:24:09.114410] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155062: LOOKUP() /conf/mail/amavis/conf.d/20-debian_defaults => -1 (Permission denied) [2016-10-29 01:24:09.114508] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155063: LOOKUP() /conf/mail/amavis/conf.d/20-debian_defaults => -1 (Permission denied) [2016-10-29 01:24:09.114593] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155065: LOOKUP() /conf/mail/amavis/conf.d/25-amavis_helpers => -1 (Permission denied) [2016-10-29 01:24:09.114669] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155066: LOOKUP() /conf/mail/amavis/conf.d/25-amavis_helpers => -1 (Permission denied) [2016-10-29 01:24:09.114743] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155068: LOOKUP() /conf/mail/amavis/conf.d/30-template_localization => -1 (Permission denied) [2016-10-29 01:24:09.114818] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155069: LOOKUP() /conf/mail/amavis/conf.d/30-template_localization => -1 (Permission denied) [2016-10-29 01:24:09.114917] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155071: LOOKUP() /conf/mail/amavis/conf.d/50-user => -1 (Permission denied) [2016-10-29 01:24:09.115025] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 155072: LOOKUP() /conf/mail/amavis/conf.d/50-user => -1 (Permission denied) [2016-10-29 08:42:00.979281] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-netshare-client-0: server 10.1.20.1:49159 has not responded in the last 5 seconds, disconnecting. [2016-10-29 08:42:00.979826] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2016-10-29 05:07:10.893092 (xid=0x21d80) [2016-10-29 08:42:00.979857] E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-netshare-client-0: remote operation failed [Transport endpoint is not connected] [2016-10-29 08:42:00.980057] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2016-10-29 05:07:20.941329 (xid=0x21da5) [2016-10-29 08:42:00.980278] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2016-10-29 05:07:30.970826 (xid=0x21daa) [2016-10-29 08:42:00.980471] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2016-10-29 05:07:41.008092 (xid=0x21daf) [2016-10-29 08:42:00.980617] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2016-10-29 05:07:51.117968 (xid=0x21db4) [2016-10-29 08:42:00.980752] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2016-10-29 05:08:01.218324 (xid=0x21db9) [2016-10-29 08:42:00.980884] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2016-10-29 05:08:11.271745 (xid=0x21dbe) [2016-10-29 08:42:00.981031] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2016-10-29 05:08:21.326249 (xid=0x21dc3) [2016-10-29 08:42:00.981157] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2016-10-29 05:08:31.370418 (xid=0x21e08) [2016-10-29 08:42:00.981299] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2016-10-29 05:08:41.433237 (xid=0x21e0d) [2016-10-29 08:42:00.981422] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2016-10-29 05:08:51.446256 (xid=0x21e26) [2016-10-29 08:42:00.981545] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2016-10-29 05:08:54.476734 (xid=0x21e2a) [2016-10-29 08:42:00.981666] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f5863707ab3] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f58634d128f] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f58634d13ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x89)[0x7f58634d29e9] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x160)[0x7f58634d3330] ))))) 0-netshare-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2016-10-29 08:41:55.978587 (xid=0x23bda) [2016-10-29 08:42:00.981679] W [rpc-clnt-ping.c:208:rpc_clnt_ping_cbk] 0-netshare-client-0: socket disconnected The message "E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-netshare-client-0: remote operation failed [Transport endpoint is not connected]" repeated 11 times between [2016-10-29 08:42:00.979857] and [2016-10-29 08:42:00.981558] [2016-10-29 08:42:02.601745] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7f58629890a4] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x564f0cace725] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x564f0cace5a7] ) 0-: received signum (15), shutting down
Thanks for the logs. But, in the future it will be better if you can attach log files to the bug by using the 'Add an attachment' link above. The logs show a lot of permission denied errors, when attempting to access files. This is most likely because the glusterfs brick processes on the server are having permission errors. I'm not sure why this is happening. I'll look around, in the meantime could you also share the brick logs from the servers. They should be under /var/log/glusterfs/bricks/.
Created attachment 1216497 [details] node1 Bricklog
Created attachment 1216498 [details] Node2 Bricklog
The brick log files don't have logs for the same time period as when the mount happened, so they haven't been as helpful. The client logs you provided, don't have any logs about portmap, changing port or client-handshake. One of the logs even has a no route to host error, "[2016-10-28 13:13:43.159065] E [socket.c:2292:socket_connect_finish] 0-netshare-client-0: connection to 10.1.20.1:24007 failed (No route to host)" along with a portmap error. This leads me to believe that there was some sort of an issue with the network at that point in time. Could it be a firewall issue? Because of this the client wasn't connected to the bricks, leading to the failures you saw. Brick logs from around '2016-10-28 13:13:43.159065' can help check if the bricks are to blame.
Jules, Thanks for sharing the logs. (Jules shared the logs privately with me). I did go through the logs, but I cannot find any specific issue with Gluster. As I mentioned in the previous comment, I see a lot of no route to host errors in the client logs. Along with these, I don't see any logs in the logs in the clients about changing ports, which should be present after a successful mount. To make it clearer the following happens on clients when you mount. 1. First the volfile is fetched from the address you give in the mount command. The client connects to GlusterD's port (24007) at the address and gets the volfile for the volume. 2. The client then builds and loads the graph, and dumps the volfile in the logs. 3. After this the graph is activated. This is when the client connects to the bricks. This connection happens in 2 steps. 3.a. For each brick the client first connects to GlusterD (24007) on the bricks host and gets the port for the brick. This is known as the portmap query. 3.b. With the ports obtained, the client reconnects to the actual bricks with the new ports. This gives a 'changing port to N (from M)' log entry. 4. The client mount point is now ready to serve requests. In the logs, I don't see successful occurances of 3.a and 3.b. There are multiple error logs of 'no route to host' for addresses 10.1.20.1:24007 and 10.1.20.2:24007 from the client translators. This means that portmap couldn't succeed, which the presence of 'failed to get port number' logs also support. With this in mind, I can arrive at 2 possible causes, 1. There was an issue with the network at this point in time. 2. GlusterD on the servers had an issue. 1 can be mostly discarded because the clients were able to fetch volfiles from the servers. Although possible, it is highly improbable that the network had and issue between the time of fetching volfiles and doing portmap queries. And it happening again, and again also doesn't help this cause. This leaves glusterd. GlusterD can sometimes enter into deadlocks, preventing it from accepting new connections or serving new requests. If glusterd is deadlocked, further mount attempts would also not succeed (unless glusterd was restarted in between). So to investigate this further, could you share the glusterd logs from the servers. These are also in /var/log/glusterfs named as 'etc*-glusterd.log'. If this cannot help figure out what went wrong, then this might have been something very special that happened once.
Exactly the same issue happens if i try to upgrade a two node gluster from version glusterfs 3.9.1 to glusterfs 3.10.0 With following set: auth.allow: 127.0.0.1,10.1.20.*,10.1.30.* [2017-02-28 18:40:21.086240] W [MSGID: 114043] [client-handshake.c:1105:client_setvolume_cbk] 0-netshare-client-1: failed to set the volume [Permission denied] [2017-02-28 18:40:21.086276] W [MSGID: 114007] [client-handshake.c:1134:client_setvolume_cbk] 0-netshare-client-1: failed to get 'process-uuid' from reply dict [Invalid argument] [2017-02-28 18:40:21.086315] E [MSGID: 114044] [client-handshake.c:1140:client_setvolume_cbk] 0-netshare-client-1: SETVOLUME on remote-host failed [Permission denied] [2017-02-28 18:40:21.086333] E [fuse-bridge.c:5322:notify] 0-fuse: Server authenication failed. Shutting down. [2017-02-28 18:40:21.111209] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064) [0x7f4d63021064] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55d72e5c5db5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d72e5c5c27] ) 0-: received signum (15), shutting down [2017-02-28 18:40:41.710274] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.0 (args: /usr/sbin/glusterfs --acl --volfile-max-fetch-attempts=5 --volfile-server=nodex2 --volfile-server=nodex1 --volfile-id=/netshare /netshare) [2017-02-28 18:40:41.714256] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
This bug is getting closed because GlusteFS-3.7 has reached its end-of-life. Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS. If this bug still exists in newer GlusterFS releases, please reopen this bug against the newer release.
as said it happens in 3.9 also
I added a tag of rpc-ping-timeout since I saw a ping timeout expiry. However the ping-timeout used here is 5 seconds which is not the default. Apart from that there is not much of evidence to mark this as a bug in rpc/transport layers. I also see some authentication problems as indicated by logs in setvolume_cbk
Jules, sorry for delayed response on this one. GlusterFS has moved multiple versions since then. At least from 3.8.x series to 4.1.x series upgrade issues were not seen. We never tested 3.6.x (or 3.7.x) to 4.1.x upgrade. We will be closing the issue as EOL, and recommend you to upgrade (with a planned migration downtime if possible) to higher versions.