Bug 1443051
Summary: | Brick Multiplexing: Unable to activate Snapshot | ||
---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Nag Pavan Chilakam <nchilaka> |
Component: | core | Assignee: | Mohit Agrawal <moagrawa> |
Status: | CLOSED ERRATA | QA Contact: | Anil Shah <ashah> |
Severity: | urgent | Docs Contact: | |
Priority: | unspecified | ||
Version: | rhgs-3.3 | CC: | amukherj, rcyriac, rhinduja, rhs-bugs, storage-qa-internal |
Target Milestone: | --- | Keywords: | Regression |
Target Release: | RHGS 3.3.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | brick-multiplexing | ||
Fixed In Version: | glusterfs-3.8.4-25 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2017-09-21 04:37:54 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1417151 |
Description
Nag Pavan Chilakam
2017-04-18 11:51:35 UTC
glusterd log The message "I [MSGID: 106488] [glusterd-handler.c:1539:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 61 times between [2017-04-18 11:53:27.468745] and [2017-04-18 11:53:43.855660] [2017-04-18 11:55:17.939480] I [MSGID: 106487] [glusterd-handler.c:1476:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-18 12:04:04.292755] I [MSGID: 106487] [glusterd-handler.c:1476:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-18 12:09:53.291071] E [MSGID: 106024] [glusterd-snapshot.c:5689:glusterd_snapshot_activate_deactivate_prevalidate] 0-management: Snapshot rep2_20_snap1_GMT-2017.04.18-12.08.32 is already deactivated. [2017-04-18 12:09:53.291183] W [MSGID: 106042] [glusterd-snapshot.c:8794:glusterd_snapshot_prevalidate] 0-management: Snapshot deactivate validation failed [2017-04-18 12:09:53.291212] W [MSGID: 106122] [glusterd-mgmt.c:167:gd_mgmt_v3_pre_validate_fn] 0-management: Snapshot Prevalidate Failed [2017-04-18 12:09:53.291240] E [MSGID: 106122] [glusterd-mgmt.c:916:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Snapshot on local node [2017-04-18 12:09:53.291260] E [MSGID: 106122] [glusterd-mgmt.c:2272:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Pre Validation Failed [2017-04-18 12:10:02.840547] E [MSGID: 106005] [glusterd-utils.c:5396:glusterd_brick_start] 0-management: Unable to start brick 10.70.35.45:/run/gluster/snaps/094dceaf071f4425936152795ab2f888/brick1/rep2_20 [2017-04-18 12:10:02.841015] E [MSGID: 106041] [glusterd-snapshot.c:6201:glusterd_snapshot_activate_commit] 0-management: Failed to activate snap volume 094dceaf071f4425936152795ab2f888 of the snap rep2_20_snap1_GMT-2017.04.18-12.08.32 [2017-04-18 12:10:02.841121] W [MSGID: 106041] [glusterd-snapshot.c:8551:glusterd_snapshot] 0-management: Failed to activate snapshot [2017-04-18 12:10:02.841166] W [MSGID: 106123] [glusterd-mgmt.c:283:gd_mgmt_v3_commit_fn] 0-management: Snapshot Commit Failed [2017-04-18 12:10:02.841192] E [MSGID: 106123] [glusterd-mgmt.c:1456:glusterd_mgmt_v3_commit] 0-management: Commit failed for operation Snapshot on local node [2017-04-18 12:10:02.841211] E [MSGID: 106123] [glusterd-mgmt.c:2333:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Commit Op Failed [2017-04-18 12:10:40.401605] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-04-18 12:10:40.401756] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-04-18 12:10:40.404707] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-04-18 12:10:40.404797] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-04-18 12:10:40.460927] I [run.c:191:runner_log] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c82) [0x7ff647f88c82] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4735) [0x7ff647f88735] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7ff6533fc105] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=rep2_15 -o features.uss=enable --gd-workdir=/var/lib/glusterd [2017-04-18 12:10:40.533349] I [run.c:191:runner_log] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c82) [0x7ff647f88c82] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4735) [0x7ff647f88735] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7ff6533fc105] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=rep2_15 -o features.uss=enable --gd-workdir=/var/lib/glusterd [2017-04-18 12:10:44.278068] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick snapd-rep2_15 on port 49158 [2017-04-18 12:10:49.237155] W [MSGID: 101095] [xlator.c:147:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.8.4/xlator/features/ganesha.so: cannot open shared object file: No such file or directory [2017-04-18 12:16:46.094428] E [MSGID: 106005] [glusterd-utils.c:5396:glusterd_brick_start] 0-management: Unable to start brick 10.70.35.45:/run/gluster/snaps/094dceaf071f4425936152795ab2f888/brick1/rep2_20 [2017-04-18 12:16:46.094808] E [MSGID: 106041] [glusterd-snapshot.c:6201:glusterd_snapshot_activate_commit] 0-management: Failed to activate snap volume 094dceaf071f4425936152795ab2f888 of the snap rep2_20_snap1_GMT-2017.04.18-12.08.32 [2017-04-18 12:16:46.094886] W [MSGID: 106041] [glusterd-snapshot.c:8551:glusterd_snapshot] 0-management: Failed to activate snapshot [2017-04-18 12:16:46.094951] W [MSGID: 106123] [glusterd-mgmt.c:283:gd_mgmt_v3_commit_fn] 0-management: Snapshot Commit Failed [2017-04-18 12:16:46.094979] E [MSGID: 106123] [glusterd-mgmt.c:1456:glusterd_mgmt_v3_commit] 0-management: Commit failed for operation Snapshot on local node [2017-04-18 12:16:46.095037] E [MSGID: 106123] [glusterd-mgmt.c:2333:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Commit Op Failed (END) RCA: With multiplexing enabled we are creating socket file in different location namely in volume configuration directory, For example : /var/lib/glusterd/snaps/Scheduled-job1-vol0_GMT-2017.04.22-09.20.01/eeadecc528164cb7880e0f747c6cc710/run/daemon-gluster-arch-srv1.lab.eng.centos.com.socket So the issue is that unix socket has a imposed limit of 108 character in path. Most of the time with a decent snap name and hostname this limit will crosss, that is why the snap brick start is failing. In previous case we were creating socket path in /var/run/gluster/2950de2ce6ccd8c926136c9696496797.socket upstream patch : https://review.gluster.org/#/c/17101/ Upstream patches : https://review.gluster.org/#/q/topic:bug-1444596 Downstream patches: https://code.engineering.redhat.com/gerrit/#/c/105595/ https://code.engineering.redhat.com/gerrit/#/c/105596/ Able to activate snapshot when brick multiplexing is enabled Bug verified on build glusterfs-3.8.4-25.el7rhgs.x86_64 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2774 |