Description of problem: [2019-03-07 13:51:35.683031] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'max_count' is not sent on wire [Invalid argument] [2019-03-07 13:51:35.683056] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'quota-list-count' is not sent on wire [Invalid argument] [2019-03-07 13:51:35.683057] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'gfid' is not sent on wire [Invalid argument] The message "W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'default-soft-limit' is not sent on wire [Invalid argument]" repeated 2 times between [2019-03-07 13:51:35.682913] and [2019-03-07 13:51:35.683120] [2019-03-07 13:51:35.683121] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'volume-uuid' is not sent on wire [Invalid argument] [2019-03-07 13:51:35.683152] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'max_count' is not sent on wire [Invalid argument] [2019-03-07 13:51:35.683153] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'quota-list-count' is not sent on wire [Invalid argument] Version-Release number of selected component (if applicable): 4.1.7 How reproducible: Everytime Steps to Reproduce: 1.Turn on quotas for volume 2. Tail quotad.log log Actual results: Log is filled with constantly repeating messages as above. System drive fills and Glusterd daemon fails Expected results: Logs are not filled with these messages Additional info:
We experience the same problem: less /var/log/quotad.log … [2019-04-14 04:26:40.249481] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'volume-uuid' is not sent on wire [Invalid argument] [2019-04-14 04:26:40.249513] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'trusted.glusterfs.quota.size' is not sent on wire [Invalid argument] The message "W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'volume-uuid' is not sent on wire [Invalid argument]" repeated 80088 times between [2019-04-14 04:26:40.249481] and [2019-04-14 04:28:40.120911] The message "W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'trusted.glusterfs.quota.size' is not sent on wire [Invalid argument]" repeated 80088 times between [2019-04-14 04:26:40.249513] and [2019-04-14 04:28:40.120912] [2019-04-14 04:28:40.122513] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'volume-uuid' is not sent on wire [Invalid argument] [2019-04-14 04:28:40.122545] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'trusted.glusterfs.quota.size' is not sent on wire [Invalid argument] The message "W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'volume-uuid' is not sent on wire [Invalid argument]" repeated 37540 times between [2019-04-14 04:28:40.122513] and [2019-04-14 04:30:39.718206] The message "W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'trusted.glusterfs.quota.size' is not sent on wire [Invalid argument]" repeated 37540 times between [2019-04-14 04:28:40.122545] and [2019-04-14 04:30:39.718207] [2019-04-14 04:30:42.000322] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'volume-uuid' is not sent on wire [Invalid argument] [2019-04-14 04:30:42.000350] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'trusted.glusterfs.quota.size' is not sent on wire [Invalid argument] The message "W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'volume-uuid' is not sent on wire [Invalid argument]" repeated 49409 times between [2019-04-14 04:30:42.000322] and [2019-04-14 04:32:40.046852] The message "W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'trusted.glusterfs.quota.size' is not sent on wire [Invalid argument]" repeated 49409 times between [2019-04-14 04:30:42.000350] and [2019-04-14 04:32:40.046853] [2019-04-14 04:32:40.345351] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'volume-uuid' is not sent on wire [Invalid argument] [2019-04-14 04:32:40.345382] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'trusted.glusterfs.quota.size' is not sent on wire [Invalid argument] The message "W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'volume-uuid' is not sent on wire [Invalid argument]" repeated 29429 times between [2019-04-14 04:32:40.345351] and [2019-04-14 04:34:40.098453] The message "W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'trusted.glusterfs.quota.size' is not sent on wire [Invalid argument]" repeated 29429 times between [2019-04-14 04:32:40.345382] and [2019-04-14 04:34:40.098454] … And more in every second. We updated from Gluster 3.12.14 to 4.1.7 at March 21st and that's the first appearance of these log mesages: zless quotad.4.log … [2019-03-21 15:28:45.195091] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'volume-uuid' is not sent on wire [Invalid argument] [2019-03-21 15:28:45.195100] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'volume-uuid' is not sent on wire [Invalid argument] [2019-03-21 15:28:45.195103] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'trusted.glusterfs.quota.size' is not sent on wire [Invalid argument] [2019-03-21 15:28:45.195155] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'trusted.glusterfs.quota.size' is not sent on wire [Invalid argument] [2019-03-21 15:28:45.195268] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'volume-uuid' is not sent on wire [Invalid argument] [2019-03-21 15:28:45.195305] W [MSGID: 101016] [glusterfs3.h:743:dict_to_xdr] 0-dict: key 'volume-uuid' is not sent on wire [Invalid argument] … Now some typical information about our system. # cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 9 (stretch)" NAME="Debian GNU/Linux" VERSION_ID="9" VERSION="9 (stretch)" ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" # gluster --version glusterfs 4.1.7 # gluster volume info $VOLUME Volume Name: $VOLUME Type: Distributed-Disperse Volume ID: 0a64f278-f432-4793-8188-346557a4e146 Status: Started Snapshot Count: 0 Number of Bricks: 15 x (4 + 2) = 90 Transport-type: tcp Bricks: Brick1: gluster01.:/srv/glusterfs/bricks/$VOLUME0100/data Brick2: gluster01.:/srv/glusterfs/bricks/$VOLUME0101/data Brick3: gluster02.:/srv/glusterfs/bricks/$VOLUME0200/data Brick4: gluster02.:/srv/glusterfs/bricks/$VOLUME0201/data Brick5: gluster05.:/srv/glusterfs/bricks/$VOLUME0500/data Brick6: gluster05.:/srv/glusterfs/bricks/$VOLUME0501/data Brick7: gluster01.:/srv/glusterfs/bricks/$VOLUME0102/data Brick8: gluster01.:/srv/glusterfs/bricks/$VOLUME0103/data Brick9: gluster02.:/srv/glusterfs/bricks/$VOLUME0202_new/data Brick10: gluster02.:/srv/glusterfs/bricks/$VOLUME0203/data Brick11: gluster05.:/srv/glusterfs/bricks/$VOLUME0502/data Brick12: gluster05.:/srv/glusterfs/bricks/$VOLUME0503/data Brick13: gluster01.:/srv/glusterfs/bricks/$VOLUME0104/data Brick14: gluster01.:/srv/glusterfs/bricks/$VOLUME0105/data Brick15: gluster02.:/srv/glusterfs/bricks/$VOLUME0204/data Brick16: gluster02.:/srv/glusterfs/bricks/$VOLUME0205/data Brick17: gluster05.:/srv/glusterfs/bricks/$VOLUME0504/data Brick18: gluster05.:/srv/glusterfs/bricks/$VOLUME0505/data Brick19: gluster01.:/srv/glusterfs/bricks/$VOLUME0106/data Brick20: gluster01.:/srv/glusterfs/bricks/$VOLUME0107/data Brick21: gluster02.:/srv/glusterfs/bricks/$VOLUME0206/data Brick22: gluster02.:/srv/glusterfs/bricks/$VOLUME0207/data Brick23: gluster05.:/srv/glusterfs/bricks/$VOLUME0506/data Brick24: gluster05.:/srv/glusterfs/bricks/$VOLUME0507/data Brick25: gluster01.:/srv/glusterfs/bricks/$VOLUME0108/data Brick26: gluster01.:/srv/glusterfs/bricks/$VOLUME0109/data Brick27: gluster02.:/srv/glusterfs/bricks/$VOLUME0208/data Brick28: gluster02.:/srv/glusterfs/bricks/$VOLUME0209/data Brick29: gluster05.:/srv/glusterfs/bricks/$VOLUME0508/data Brick30: gluster05.:/srv/glusterfs/bricks/$VOLUME0509/data Brick31: gluster01.:/srv/glusterfs/bricks/$VOLUME0110/data Brick32: gluster01.:/srv/glusterfs/bricks/$VOLUME0111/data Brick33: gluster02.:/srv/glusterfs/bricks/$VOLUME0210/data Brick34: gluster02.:/srv/glusterfs/bricks/$VOLUME0211/data Brick35: gluster05.:/srv/glusterfs/bricks/$VOLUME0510/data Brick36: gluster05.:/srv/glusterfs/bricks/$VOLUME0511/data Brick37: gluster06.:/srv/glusterfs/bricks/$VOLUME0600/data Brick38: gluster06.:/srv/glusterfs/bricks/$VOLUME0601/data Brick39: gluster07.:/srv/glusterfs/bricks/$VOLUME0700/data Brick40: gluster07.:/srv/glusterfs/bricks/$VOLUME0701/data Brick41: gluster08.:/srv/glusterfs/bricks/$VOLUME0800/data Brick42: gluster08.:/srv/glusterfs/bricks/$VOLUME0801/data Brick43: gluster06.:/srv/glusterfs/bricks/$VOLUME0602/data Brick44: gluster06.:/srv/glusterfs/bricks/$VOLUME0603/data Brick45: gluster07.:/srv/glusterfs/bricks/$VOLUME0702/data Brick46: gluster07.:/srv/glusterfs/bricks/$VOLUME0703/data Brick47: gluster08.:/srv/glusterfs/bricks/$VOLUME0802/data Brick48: gluster08.:/srv/glusterfs/bricks/$VOLUME0803/data Brick49: gluster06.:/srv/glusterfs/bricks/$VOLUME0604/data Brick50: gluster06.:/srv/glusterfs/bricks/$VOLUME0605/data Brick51: gluster07.:/srv/glusterfs/bricks/$VOLUME0704/data Brick52: gluster07.:/srv/glusterfs/bricks/$VOLUME0705/data Brick53: gluster08.:/srv/glusterfs/bricks/$VOLUME0804/data Brick54: gluster08.:/srv/glusterfs/bricks/$VOLUME0805/data Brick55: gluster06.:/srv/glusterfs/bricks/$VOLUME0606/data Brick56: gluster06.:/srv/glusterfs/bricks/$VOLUME0607/data Brick57: gluster07.:/srv/glusterfs/bricks/$VOLUME0706/data Brick58: gluster07.:/srv/glusterfs/bricks/$VOLUME0707/data Brick59: gluster08.:/srv/glusterfs/bricks/$VOLUME0806/data Brick60: gluster08.:/srv/glusterfs/bricks/$VOLUME0807/data Brick61: gluster06.:/srv/glusterfs/bricks/$VOLUME0608/data Brick62: gluster06.:/srv/glusterfs/bricks/$VOLUME0609/data Brick63: gluster07.:/srv/glusterfs/bricks/$VOLUME0708/data Brick64: gluster07.:/srv/glusterfs/bricks/$VOLUME0709/data Brick65: gluster08.:/srv/glusterfs/bricks/$VOLUME0808/data Brick66: gluster08.:/srv/glusterfs/bricks/$VOLUME0809/data Brick67: gluster06.:/srv/glusterfs/bricks/$VOLUME0610/data Brick68: gluster06.:/srv/glusterfs/bricks/$VOLUME0611/data Brick69: gluster07.:/srv/glusterfs/bricks/$VOLUME0710/data Brick70: gluster07.:/srv/glusterfs/bricks/$VOLUME0711/data Brick71: gluster08.:/srv/glusterfs/bricks/$VOLUME0810/data Brick72: gluster08.:/srv/glusterfs/bricks/$VOLUME0811/data Brick73: gluster01.:/srv/glusterfs/bricks/$VOLUME0112/data Brick74: gluster01.:/srv/glusterfs/bricks/$VOLUME0113/data Brick75: gluster02.:/srv/glusterfs/bricks/$VOLUME0212/data Brick76: gluster02.:/srv/glusterfs/bricks/$VOLUME0213/data Brick77: gluster05.:/srv/glusterfs/bricks/$VOLUME0512/data Brick78: gluster05.:/srv/glusterfs/bricks/$VOLUME0513/data Brick79: gluster01.:/srv/glusterfs/bricks/$VOLUME0114/data Brick80: gluster01.:/srv/glusterfs/bricks/$VOLUME0115/data Brick81: gluster02.:/srv/glusterfs/bricks/$VOLUME0214/data Brick82: gluster02.:/srv/glusterfs/bricks/$VOLUME0215/data Brick83: gluster05.:/srv/glusterfs/bricks/$VOLUME0514/data Brick84: gluster05.:/srv/glusterfs/bricks/$VOLUME0515/data Brick85: gluster01.:/srv/glusterfs/bricks/$VOLUME0116/data Brick86: gluster01.:/srv/glusterfs/bricks/$VOLUME0117/data Brick87: gluster02.:/srv/glusterfs/bricks/$VOLUME0216/data Brick88: gluster02.:/srv/glusterfs/bricks/$VOLUME0217/data Brick89: gluster05.:/srv/glusterfs/bricks/$VOLUME0516/data Brick90: gluster05.:/srv/glusterfs/bricks/$VOLUME0517/data Options Reconfigured: performance.md-cache-timeout: 5 performance.rda-cache-limit: 100MB performance.parallel-readdir: off performance.cache-refresh-timeout: 1 features.quota-deem-statfs: on features.inode-quota: on features.quota: on auth.allow: $IPSPACE transport.address-family: inet nfs.disable: on
Hi Ryan, We have marked these logs as DEBUG level from GlusterFS-6.0. Please upgrade to get the fixes.
Hi Amar, Thanks! Do you know when/if these will be backported into the 4.1 branch? Best, Ryan
https://review.gluster.org/#/q/I4157a7ec7d5ec9c2948b2bbc1e4cb8317f28d6b8 is the patch I am talking about. We had not ported the changes to 4.1 branch then. Considering we are getting good feedback about 6.x release (both performance, and stability), I would request you to consider an upgrade if that is possible. Because 4.1 would go out of active support in a month after glusterfs-7.0 release.