Bug 1500398
Summary: | [GSS] gluster volume status all clients --xml gives blank result (no error) | |||
---|---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Cal Calhoun <ccalhoun> | |
Component: | cli | Assignee: | Sanju <srakonde> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Rahul Hinduja <rhinduja> | |
Severity: | medium | Docs Contact: | ||
Priority: | medium | |||
Version: | rhgs-3.2 | CC: | amukherj, ccalhoun, nbalacha, rhs-bugs, srakonde, storage-qa-internal | |
Target Milestone: | --- | |||
Target Release: | --- | |||
Hardware: | x86_64 | |||
OS: | Linux | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | If docs needed, set a value | ||
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1502691 (view as bug list) | Environment: | ||
Last Closed: | 2018-03-31 11:43:10 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1502691 |
Description
Cal Calhoun
2017-10-10 13:50:06 UTC
Results from requested command run: [root@mdb-gsn-01 alex]# gluster volume status all clients --xml [root@mdb-gsn-01 alex]# echo $? 2 From the customer: # cat /var/log/glusterfs/cli.log [2017-11-06 12:45:08.566178] I [cli.c:762:main] 0-cli: Started running gluster with version 3.8.4 [2017-11-06 12:45:08.566219] D [cli.c:664:cli_rpc_init] 0-cli: Connecting to glusterd using default socket [2017-11-06 12:45:08.566254] D [rpc-clnt.c:1063:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2017-11-06 12:45:08.566259] D [rpc-clnt.c:1077:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2017-11-06 12:45:08.566267] D [rpc-transport.c:279:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.8.4/rpc-transport/socket.so [2017-11-06 12:45:08.573709] D [socket.c:4102:socket_init] 0-glusterfs: disabling nodelay [2017-11-06 12:45:08.573723] D [socket.c:4154:socket_init] 0-glusterfs: Configued transport.tcp-user-timeout=0 [2017-11-06 12:45:08.573728] D [socket.c:4239:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2017-11-06 12:45:08.573732] D [socket.c:4242:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2017-11-06 12:45:08.573735] D [socket.c:4259:socket_init] 0-glusterfs: using system polling thread [2017-11-06 12:45:08.573743] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-11-06 12:45:08.573749] T [socket.c:3122:socket_connect] 0-glusterfs: connecting 0x562130dc0ab0, state=0 gen=0 sock=-1 [2017-11-06 12:45:08.573754] T [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/glusterd.socket [2017-11-06 12:45:08.574488] T [name.c:125:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-11-06 12:45:08.575179] D [rpc-clnt.c:1063:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2017-11-06 12:45:08.575189] D [rpc-clnt.c:1077:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2017-11-06 12:45:08.575194] D [rpc-transport.c:279:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.8.4/rpc-transport/socket.so [2017-11-06 12:45:08.575217] D [socket.c:4154:socket_init] 0-glusterfs: Configued transport.tcp-user-timeout=0 [2017-11-06 12:45:08.575222] D [socket.c:4239:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2017-11-06 12:45:08.575225] D [socket.c:4242:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2017-11-06 12:45:08.575229] D [socket.c:4259:socket_init] 0-glusterfs: using system polling thread [2017-11-06 12:45:08.575233] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-11-06 12:45:08.575236] T [socket.c:3122:socket_connect] 0-glusterfs: connecting 0x562130df7fc0, state=0 gen=0 sock=-1 [2017-11-06 12:45:08.575241] T [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/gluster/quotad.socket [2017-11-06 12:45:08.575257] T [name.c:125:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-11-06 12:45:08.575333] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575351] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575356] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575360] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575365] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575393] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575398] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575404] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575410] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575417] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575422] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575428] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575432] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575436] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575446] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575450] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575455] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.575460] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740100] D [cli-cmd-volume.c:2346:cli_check_gsync_present] 0-cli: Returning 0 [2017-11-06 12:45:08.740144] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740155] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740180] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740185] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740198] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740207] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740221] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740230] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740235] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740243] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740248] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740254] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740266] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740272] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740280] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740286] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740290] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740294] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740298] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740307] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740313] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740318] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740322] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740327] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740332] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740336] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740340] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740344] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740349] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740354] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740357] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740361] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740365] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740378] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740385] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740390] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740395] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740401] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740407] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740412] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740423] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740429] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740434] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740445] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740450] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740454] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740461] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-11-06 12:45:08.740587] I [MSGID: 101190] [event-epoll.c:602:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-11-06 12:45:08.740628] T [cli.c:268:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-11-06 12:45:08.740640] T [cli-quotad-client.c:89:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-11-06 12:45:08.740646] I [socket.c:2465:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-11-06 12:45:08.741171] T [cli-quotad-client.c:95:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT [2017-11-06 12:45:08.741491] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f99e2777242] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f99e1c8eccb] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5f)[0x7f99e1c8b0ff] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7f99e1c8bbe0] (--> /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f99e1c879e3] ))))) 0-: /var/run/gluster/quotad.socket: ping timer event already removed [2017-11-06 12:45:08.743100] T [rpc-clnt.c:1511:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: [2017-11-06 12:45:08.743124] T [rpc-clnt.c:1368:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 132, payload: 68, rpc hdr: 64 [2017-11-06 12:45:08.743274] T [socket.c:3114:socket_connect] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f99e2777242] (--> /usr/lib64/glusterfs/3.8.4/rpc-transport/socket.so(+0xa664)[0x7f99d8f4a664] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x3a9)[0x7f99e1c8c009] (--> gluster(cli_submit_request+0x16f)[0x56212f52a6df] (--> gluster(cli_cmd_submit+0x93)[0x56212f52c703] ))))) 0-glusterfs: connect () called on transport already connected [2017-11-06 12:45:08.743295] T [rpc-clnt.c:1714:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 27) to rpc-transport (glusterfs) [2017-11-06 12:45:08.743304] D [rpc-clnt-ping.c:276:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning [2017-11-06 12:45:08.787233] T [rpc-clnt.c:691:rpc_clnt_reply_init] 0-glusterfs: received rpc message (RPC XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 27) from rpc-transport (glusterfs) [2017-11-06 12:45:08.787272] D [cli-rpc-ops.c:8194:gf_cli_status_cbk] 0-cli: Received response to status cmd [2017-11-06 12:45:08.787349] D [cli-cmd.c:390:cli_cmd_submit] 0-cli: Returning 0 [2017-11-06 12:45:08.787404] D [cli-rpc-ops.c:8486:gf_cli_status_volume] 0-cli: Returning: 0 [2017-11-06 12:45:08.787441] T [rpc-clnt.c:1511:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: [2017-11-06 12:45:08.787461] T [rpc-clnt.c:1368:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 152, payload: 88, rpc hdr: 64 [2017-11-06 12:45:08.787648] T [socket.c:3114:socket_connect] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f99e2777242] (--> /usr/lib64/glusterfs/3.8.4/rpc-transport/socket.so(+0xa664)[0x7f99d8f4a664] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x3a9)[0x7f99e1c8c009] (--> gluster(cli_submit_request+0x16f)[0x56212f52a6df] (--> gluster(cli_cmd_submit+0x93)[0x56212f52c703] ))))) 0-glusterfs: connect () called on transport already connected [2017-11-06 12:45:08.787665] T [rpc-clnt.c:1714:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2 Program: Gluster CLI, ProgVers: 2, Proc: 27) to rpc-transport (glusterfs) [2017-11-06 12:45:08.787671] D [rpc-clnt-ping.c:276:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning [2017-11-06 12:45:08.881670] T [rpc-clnt.c:691:rpc_clnt_reply_init] 0-glusterfs: received rpc message (RPC XID: 0x2 Program: Gluster CLI, ProgVers: 2, Proc: 27) from rpc-transport (glusterfs) [2017-11-06 12:45:08.881710] D [cli-rpc-ops.c:8194:gf_cli_status_cbk] 0-cli: Received response to status cmd [2017-11-06 12:45:08.893410] D [cli-cmd.c:390:cli_cmd_submit] 0-cli: Returning 0 [2017-11-06 12:45:08.893427] D [cli-rpc-ops.c:8486:gf_cli_status_volume] 0-cli: Returning: 0 [2017-11-06 12:45:08.893442] T [rpc-clnt.c:1511:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: [2017-11-06 12:45:08.893448] T [rpc-clnt.c:1368:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 172, payload: 108, rpc hdr: 64 [2017-11-06 12:45:08.893597] T [socket.c:3114:socket_connect] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f99e2777242] (--> /usr/lib64/glusterfs/3.8.4/rpc-transport/socket.so(+0xa664)[0x7f99d8f4a664] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x3a9)[0x7f99e1c8c009] (--> gluster(cli_submit_request+0x16f)[0x56212f52a6df] (--> gluster(cli_cmd_submit+0x93)[0x56212f52c703] ))))) 0-glusterfs: connect () called on transport already connected [2017-11-06 12:45:08.893615] T [rpc-clnt.c:1714:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3 Program: Gluster CLI, ProgVers: 2, Proc: 27) to rpc-transport (glusterfs) [2017-11-06 12:45:08.893620] D [rpc-clnt-ping.c:276:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning [2017-11-06 12:45:08.902897] T [rpc-clnt.c:691:rpc_clnt_reply_init] 0-glusterfs: received rpc message (RPC XID: 0x3 Program: Gluster CLI, ProgVers: 2, Proc: 27) from rpc-transport (glusterfs) [2017-11-06 12:45:08.903080] D [cli-rpc-ops.c:8194:gf_cli_status_cbk] 0-cli: Received response to status cmd [2017-11-06 12:45:08.903402] D [cli-cmd.c:390:cli_cmd_submit] 0-cli: Returning 0 [2017-11-06 12:45:08.903419] D [cli-rpc-ops.c:8486:gf_cli_status_volume] 0-cli: Returning: 0 [2017-11-06 12:45:08.903429] I [input.c:31:cli_batch] 0-: Exiting with: 0 @Atin, I apologise for not being thorough. Node mdb-gsn-01 is on 3.8.4-44 but I missed the fact that node mdb-gsn-02 is still on 3.8.4-18. mdb-gsn-01 glusterfs-server-3.8.4-44.el7rhgs.x86_64 mdb-gsn-02 glusterfs-server-3.8.4-18.4.el7rhgs.x86_64 As a standard, I do check all nodes but I must have missed it in this case. I will be more diligent in the future. Thank you for pointing out my error. -Cal |