Hide Forgot
Description of problem: ----------------------- While trying to get all the volume options on the gluster volume, observed errors in glusterd log file ( /var/log/glusterfs/glusterd.log ) which explained "Failed to get", for few volume option(keys) Version-Release number of selected component (if applicable): ------------------------------------------------------------- RHEL 7.2 RHGS 3.2.0 ( interim build - glusterfs-3.8.4-1.el7rhgs ) How reproducible: ----------------- Always Steps to Reproduce: ------------------- 1. Get all the volume options of the volume # gluster volume get <vol-name> all 2. Observe glusterd log file ( /var/log/glusterfs/glusterd.log ) Actual results: --------------- There are errors in glusterd.log regarding failure to get few volume options Expected results: ----------------- All volume options should be available
Following is the snip of the error message in glusterd.log, while executing 'gluster volume get <vol-name> all' <snip> 2016-09-27 14:00:59.653983] E [MSGID: 106332] [glusterd-utils.c:10907:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nufa key [2016-09-27 14:00:59.655034] E [MSGID: 106332] [glusterd-utils.c:10907:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for switch key [2016-09-27 14:00:59.658876] E [MSGID: 106332] [glusterd-utils.c:10907:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for strict-readdir key [2016-09-27 14:00:59.661641] E [MSGID: 106332] [glusterd-utils.c:10907:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for readdir-failover key [2016-09-27 14:00:59.669480] E [MSGID: 106332] [glusterd-utils.c:10907:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for transport.socket.ssl-enabled key [2016-09-27 14:00:59.669891] E [MSGID: 106332] [glusterd-utils.c:10907:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for transport.socket.own-thread key [2016-09-27 14:00:59.684201] E [MSGID: 106332] [glusterd-utils.c:10907:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nfs.transport-type key [2016-09-27 14:00:59.689305] E [MSGID: 106332] [glusterd-utils.c:10907:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nfs.log-level key [2016-09-27 14:00:59.696334] E [MSGID: 106332] [glusterd-utils.c:10907:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for xattr-user-namespace-mode key [2016-09-27 14:00:59.697801] E [MSGID: 106332] [glusterd-utils.c:10907:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for config key [2016-09-27 14:00:59.697857] E [MSGID: 106332] [glusterd-utils.c:10907:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for config key [2016-09-27 14:00:59.698902] E [MSGID: 106332] [glusterd-utils.c:10907:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for op-version key </snip>
While trying to get one of the specific value [root@ ~]# gluster volume get repvol nufa Option Value ------ ----- volume get option failed. Check the cli/glusterd log file for more details The following is the snip from cli.log <snip> [2016-09-27 14:03:14.680650] I [cli.c:730:main] 0-cli: Started running gluster with version 3.8.4 [2016-09-27 14:03:14.687156] I [cli-cmd-volume.c:2267:cli_check_gsync_present] 0-: geo-replication not installed [2016-09-27 14:03:14.687650] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-09-27 14:03:14.687801] I [socket.c:2386:socket_event_handler] 0-transport: disconnecting now [2016-09-27 14:03:14.690586] E [cli-rpc-ops.c:11420:gf_cli_get_vol_opt_cbk] 0-cli: Failed to retrieve key value for value1 fromthe dictionary [2016-09-27 14:03:14.690849] I [input.c:31:cli_batch] 0-: Exiting with: -2 <snip> The value of the option key here is referred as 'value1'. It should have been printed as 'nufa' That should also be fixed to print the actual value (key) used with 'gluster volume get'
This doesn't look like a regression and the same logs are seen in with rhgs-3.1.3 bits.
While trying to retrieve option values, the following 3 steps are essentially followed: 1. Firstly it is checked if the key (which is the option name in this case) is set as a global option in glusterd (stored in 'opts' member of struct glusterd_conf_t). If found, the value is returned. Else, 2. It is checked if a value for the key is set in the dictionary that stores the options for the respective volinfo. 3. Lastly, the default value set for the option in the respective translator is tried to be retrieved. In this case what happens is that all of the above 3 steps fail. These options are not set in the respective volume options entry tables or in the options table for the respective translators. Hence, trying to retrieve the values for these options fails, leading to the error logs. However, once we set the value for one of these options explicitly, we can successfully retrieve its values using the volume get functionality. For example: # gluster volume get tv1 nfs.transport-type Option Value ------ ----- volume get option failed. Check the cli/glusterd log file for more details # gluster volume set tv1 nfs.transport-type tcp volume set: success # gluster volume get tv1 nfs.transport-type Option Value ------ ----- nfs.transport-type tcp nfs.transport-type is one of the options for which this behaviour has been observed. When run on tv1 without the option being set, we get the error. Once set, the option's value can be obtained in the 2nd step from the steps listed above and no error is thrown. While this example deals with one of the options, this RCA holds true for 'volume get <volname> all' as well. All options for which these error logs are observed do not have an entry in the respective translators. This needs to be fixed at the level of each translator, so that these options are available in the options table for the respective translator. For now to address this issue at the glusterd level, the log types for these errors could be changed to warnings, instead of errors.