Bug 1293669 - [rhgs docker image] Bind-mounting of gluster bricks does not seems to be successful always
[rhgs docker image] Bind-mounting of gluster bricks does not seems to be succ...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: rhgs-server-container (Show other bugs)
3.1
Unspecified Unspecified
medium Severity high
: ---
: RHGS 3.1.2
Assigned To: Mohamed Ashiq
Prasanth
: TestBlocker, ZStream
Depends On: 1294459
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-22 11:07 EST by Prasanth
Modified: 2016-12-19 12:31 EST (History)
7 users (show)

See Also:
Fixed In Version: rhgs-server-docker-3.1.2-6
Doc Type: Bug Fix
Doc Text:
Cause: systemd issue. https://bugzilla.redhat.com/show_bug.cgi?id=1285863 Consequence: Bind mount from Atomic host to RHGS container did not seem to be successful all the time. Fix: systemd-sysv-219-19.el7_2.4.x86_64 systemd-219-19.el7_2.4.x86_64 systemd-libs-219-19.el7_2.4.x86_64 Result: Bind Mount happens successfully in the container from Atomic Host successfully always
Story Points: ---
Clone Of:
: 1294459 (view as bug list)
Environment:
Last Closed: 2016-12-19 12:31:05 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
dmesg log (74.96 KB, text/plain)
2015-12-22 11:07 EST, Prasanth
no flags Details
Image working details (4.06 KB, text/plain)
2016-01-11 23:45 EST, Mohamed Ashiq
no flags Details

  None (edit)
Description Prasanth 2015-12-22 11:07:20 EST
Created attachment 1108670 [details]
dmesg log

Description of problem:

Bind-mounting of gluster bricks does not seems to be successful always


Version-Release number of selected component (if applicable):
#####
-bash-4.2# cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.2

-bash-4.2# sudo docker images
REPOSITORY                                                               TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7   latest              7dfbd648bde2        18 hours ago        252.7 MB
#####

How reproducible: Very often


Steps to Reproduce:
####
* Prepare the setup for Snapshot as per https://docs.google.com/document/d/1o4_funQmy91dmIG_qAuxowDwfk1jrRIyrGx2wzXNNmU/edit?ts=560bc1a5&pli=1 including the provided work-arounds
* sudo docker pull rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7
* sudo docker run --privileged=True --net=host -d --name snapnode2 -v /etc/glusterfs/:/etc/glusterfs/ -v /var/lib/glusterd/:/var/lib/glusterd/ -v /var/log/glusterfs/:/var/log/glusterfs/ -v /mnt/brick1:/b1 -v /mnt/brick2:/b2 -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /dev:/dev -ti rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7 /sbin/init
* sudo docker ps
* sudo docker exec -ti 9bf966067e6d /bin/bash
* df -h

Actual results: In some of the nodes, it's seen that the brick paths are not getting bind-mounted inside the rhgs container properly


Expected results: Bind-mount should happen successfully


Additional info:

Console logs are pasted below for your reference:

###############
-bash-4.2# sudo docker pull rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7
Using default tag: latest
7dfbd648bde2: Pulling image (latest) from rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7, endpoint: https://rcm-img-docker01.build.eng.bos.redhat.co7dfbd648bde2: Download complete 
6c3a84d798dc: Download complete 
Status: Downloaded newer image for rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7:latest

-bash-4.2#  df -h
Filesystem                           Size  Used Avail Use% Mounted on
/dev/mapper/rhelah_dhcp37--114-root  3.0G  940M  2.1G  31% /
devtmpfs                             3.9G     0  3.9G   0% /dev
tmpfs                                3.9G     0  3.9G   0% /dev/shm
tmpfs                                3.9G  376K  3.9G   1% /run                                                                                                             
tmpfs                                3.9G     0  3.9G   0% /sys/fs/cgroup                                                                                                   
/dev/sda1                            297M   87M  211M  30% /boot                                                                                                            
tmpfs                                783M     0  783M   0% /run/user/0                                                                                                      
/dev/mapper/RHGS_VG1-vol1             10G   33M   10G   1% /var/mnt/brick1                                                                                                  
/dev/mapper/RHGS_VG2-vol2             10G   33M   10G   1% /var/mnt/brick2

-bash-4.2# sudo docker run --privileged=True --net=host -d --name snapnode2 -v /etc/glusterfs/:/etc/glusterfs/ -v /var/lib/glusterd/:/var/lib/glusterd/ -v /var/log/glusterfs/:/var/log/glusterfs/ -v /mnt/brick1:/b1 -v /mnt/brick2:/b2 -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /dev:/dev -ti rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7 /sbin/init
9bf966067e6d476c6be36c60a3112b8995e2f27d9d6321cc720e1858d1fa1a41

 
-bash-4.2# sudo docker ps
CONTAINER ID        IMAGE                                                                    COMMAND             CREATED             STATUS              PORTS               NAMES
9bf966067e6d        rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7   "/sbin/init"        3 minutes ago       Up 3 minutes                            snapnode2

 
-bash-4.2# sudo docker exec -ti 9bf966067e6d /bin/bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@dhcp37-114 /]# 
[root@dhcp37-114 /]# 
[root@dhcp37-114 /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/dm-15      100G  290M  100G   1% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           3.9G  8.3M  3.9G   1% /run
###############
Comment 2 Prasanth 2015-12-23 06:56:48 EST
>> *) In “docker run” ( same as the command mentioned in Page 27 , however we need to add extra switch “-v /dev:/dev” ) command we also have to export “/dev/” tree as a volume to get snapshot working inside the RHGS containers.

If I don't do the above work-around, I could see that the bind-mounting of bricks is happening successfully everytime. However, if I try to create a snapshot without the above work-around, it fails: 

######
[root@dhcp37-150 /]# gluster snapshot create snap1 dock-vol1 no-timestamp
snapshot create: failed: Commit failed on localhost. Please check log file for details.
Snapshot command failed
######


From the logs:
############
[2015-12-23 02:21:51.452399] I [MSGID: 106057] [glusterd-snapshot.c:6218:glusterd_do_snap_cleanup] 0-management: Snapshot (snap1) does not exist [Invalid argument]
[2015-12-23 05:02:17.425202] I [MSGID: 106057] [glusterd-snapshot.c:6218:glusterd_do_snap_cleanup] 0-management: Snapshot (snap1) does not exist [Invalid argument]
[2015-12-23 05:02:23.621146] E [MSGID: 106058] [glusterd-snapshot.c:5050:glusterd_update_fs_label] 0-management: Failed to change filesystem label of /run/gluster/snaps/70ce8446fac8462199ea46fe002231b5/brick2/brick1 brick to 9152e4b2895a
[2015-12-23 05:02:23.621185] E [MSGID: 106058] [glusterd-snapshot.c:5110:glusterd_take_brick_snapshot] 0-management: Failed to update file-system label for /run/gluster/snaps/70ce8446fac8462199ea46fe002231b5/brick2/brick1 brick
[2015-12-23 05:02:23.639089] E [MSGID: 106098] [glusterd-snapshot-utils.c:2710:glusterd_mount_lvm_snapshot] 0-management: mounting the snapshot logical device /dev/RHGS_VG1/70ce8446fac8462199ea46fe002231b5_0 failed (error: Bad file descriptor)
[2015-12-23 05:02:23.639137] E [MSGID: 106059] [glusterd-snapshot.c:4767:glusterd_snap_brick_create] 0-management: Failed to mount lvm snapshot.
[2015-12-23 05:02:23.639146] W [MSGID: 106055] [glusterd-snapshot.c:4798:glusterd_snap_brick_create] 0-management: unmounting the snap brick mount /run/gluster/snaps/70ce8446fac8462199ea46fe002231b5/brick2
[2015-12-23 05:02:23.647179] E [MSGID: 106095] [glusterd-snapshot-utils.c:3362:glusterd_umount] 0-management: umounting /run/gluster/snaps/70ce8446fac8462199ea46fe002231b5/brick2 failed (Bad file descriptor) [Bad file descriptor]
[2015-12-23 05:02:23.647225] E [MSGID: 106050] [glusterd-snapshot.c:5124:glusterd_take_brick_snapshot] 0-management: not able to create the brick for the snap snap1, volume 70ce8446fac8462199ea46fe002231b5
[2015-12-23 05:02:23.647235] E [MSGID: 106030] [glusterd-snapshot.c:6322:glusterd_take_brick_snapshot_task] 0-management: Failed to take backend snapshot for brick 10.70.37.114:/run/gluster/snaps/70ce8446fac8462199ea46fe002231b5/brick2/brick1 volume(70ce8446fac8462199ea46fe002231b5)
[2015-12-23 05:02:23.647293] E [MSGID: 106030] [glusterd-snapshot.c:6464:glusterd_schedule_brick_snapshot] 0-management: Failed to create snapshot
[2015-12-23 05:02:23.647306] E [MSGID: 106030] [glusterd-snapshot.c:6790:glusterd_snapshot_create_commit] 0-management: Failed to take backend snapshot snap1
[2015-12-23 05:02:23.648241] E [MSGID: 106030] [glusterd-snapshot.c:8132:glusterd_snapshot] 0-management: Failed to create snapshot
[2015-12-23 05:02:23.648259] W [MSGID: 106123] [glusterd-mgmt.c:272:gd_mgmt_v3_commit_fn] 0-management: Snapshot Commit Failed
[2015-12-23 05:02:23.648270] E [MSGID: 106123] [glusterd-mgmt.c:1414:glusterd_mgmt_v3_commit] 0-management: Commit failed for operation Snapshot on local node
[2015-12-23 05:02:23.648278] E [MSGID: 106123] [glusterd-mgmt.c:2285:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Commit Op Failed
[2015-12-23 05:02:24.201886] I [MSGID: 106057] [glusterd-snapshot.c:6218:glusterd_do_snap_cleanup] 0-management: Snapshot (snap1) does not exist [Invalid argument]
############
Comment 3 Humble Chirammal 2015-12-28 01:39:01 EST
(In reply to Prasanth from comment #2)

We need to export /dev ( -v /dev:/dev)  to the container, because at time of snapshot creation, it creates an LV which need the "/dev/" access. so, lets move aside the bz comment#2. 

Coming back to the bugzilla report, this looks like an issue pop up in an inconsistent way in *some* systems. Is that correct?  Whats the frequency of this issue?  Because, we tried to reproduce this issue in couple of systems without luck. 

@Ashiq can you please share the system details here ?

Also, as an isolation step, would it be possible for you to run couple of containers (systemd based) in your setup (by exporting a thin volume as mount points and by exporting /dev tree) and find the result ? it doesnt looks like an issue ( if there is one) with RHGS containers, rather a race condition comes in some combination of docker bits.
Comment 5 Humble Chirammal 2015-12-28 02:00:59 EST
@Prasanth, apart from the isolation steps mentioned in comment#3, can you please reproduce this issue by enabling debug option of docker ? Meanwhile we will try to reproduce this issue in few more systems.
Comment 6 Prasanth 2015-12-29 01:43:40 EST
(In reply to Humble Chirammal from comment #3)
> (In reply to Prasanth from comment #2)
> 
> We need to export /dev ( -v /dev:/dev)  to the container, because at time of
> snapshot creation, it creates an LV which need the "/dev/" access. so, lets
> move aside the bz comment#2. 

The above isolation was just to figure out and confirm that the race condition which we are hitting very often has something related to /dev exporting. But since we really need to bind-mount /dev for Snapshot to work, we need to figure out a way to make it work somehow or find a temporary work-around.


> 
> Coming back to the bugzilla report, this looks like an issue pop up in an
> inconsistent way in *some* systems. Is that correct?  Whats the frequency of
> this issue?  Because, we tried to reproduce this issue in couple of systems
> without luck. 

You are correct. I've been seeing this issue in atleast 2-3 machines always in all the 4 node setups that I've created so far. It was *never* successful in bind-mounting all the bricks in *all* the 4 nodes on any of the setup's. But I wouldn't say it's 100% reproducible as it works in some machines either in the first attempt or after multiple retries. 

> 
> @Ashiq can you please share the system details here ?

In Ashiq's setup, I could see only 1 brick available. Please try to have more than one brick per node and try to setup a 4 node cluster as the probability of hitting this bug is more in that case and moreover that is when i start seeing this issue!

> 
> Also, as an isolation step, would it be possible for you to run couple of
> containers (systemd based) in your setup (by exporting a thin volume as
> mount points and by exporting /dev tree) and find the result ? it doesnt
> looks like an issue ( if there is one) with RHGS containers, rather a race
> condition comes in some combination of docker bits.

I can try this as well and update the BZ with the results.
Comment 14 Mohamed Ashiq 2016-01-11 23:45 EST
Created attachment 1113850 [details]
Image working details
Comment 19 Prasanth 2016-01-21 08:32:23 EST
(In reply to Mohamed Ashiq from comment #17)
> The fix is available in
> 
> docker pull docker-registry.usersys.redhat.com/gluster/rhgs-3.1.2:5

Bind-mounting of gluster bricks seems to be working fine with the latest image. However, I'm trying the same in multiple setups to re-confirm the same before I move this BZ to VERIFIED.
Comment 20 Prasanth 2016-01-27 01:31:34 EST
Verified as fixed in rhgs-server-docker-3.1.2-6

Bind-mounting of gluster bricks seems to be working always as expected in all the setups that I tried. 

##########
-bash-4.2# df -h
Filesystem                           Size  Used Avail Use% Mounted on
/dev/mapper/rhelah_dhcp42--208-root  3.0G  940M  2.1G  31% /
devtmpfs                             3.9G     0  3.9G   0% /dev
tmpfs                                3.9G     0  3.9G   0% /dev/shm
tmpfs                                3.9G  420K  3.9G   1% /run
tmpfs                                3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                            297M   87M  211M  30% /boot
tmpfs                                783M     0  783M   0% /run/user/0
/dev/mapper/RHGS_VG1-vol1             10G   33M   10G   1% /var/mount/brick1
/dev/mapper/RHGS_VG2-vol2             10G   33M   10G   1% /var/mount/brick2
/dev/mapper/RHGS_VG3-vol3             10G   33M   10G   1% /var/mount/brick3
/dev/mapper/RHGS_VG4-vol4             10G   33M   10G   1% /var/mount/brick4

-bash-4.2# sudo docker -D run -d --privileged=true --net=host --name snapnode1 -v /etc/glusterfs/:/etc/glusterfs/ -v /var/lib/glusterd/:/var/lib/glusterd/ -v /var/log/glusterfs/:/var/log/glusterfs/ -v /var/mount/brick1:/b1 -v /var/mount/brick2:/b2 -v /var/mount/brick3:/b3 -v /var/mount/brick4:/b4 -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /dev:/dev rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7:3.1.2-6 
2ff93a40d54c018b48e3d745ef81285b3d115e472d4b20d398122bb986ef969e
DEBU[0000] End of CmdRun(), Waiting for hijack to finish.

-bash-4.2# docker ps
CONTAINER ID        IMAGE                                                                            COMMAND             CREATED             STATUS              PORTS               NAMES
2ff93a40d54c        rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7:3.1.2-6   "/usr/sbin/init"    4 seconds ago       Up 3 seconds                            snapnode1
-bash-4.2# df -h
Filesystem                           Size  Used Avail Use% Mounted on
/dev/mapper/rhelah_dhcp42--208-root  3.0G  940M  2.1G  31% /
devtmpfs                             3.9G     0  3.9G   0% /dev
tmpfs                                3.9G     0  3.9G   0% /dev/shm
tmpfs                                3.9G  432K  3.9G   1% /run
tmpfs                                3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                            297M   87M  211M  30% /boot
tmpfs                                783M     0  783M   0% /run/user/0

-bash-4.2# docker exec -ti 2ff93a40d54c /bin/bash
 
[root@dhcp42-208 /]# df -h
Filesystem                           Size  Used Avail Use% Mounted on
/dev/dm-25                           100G  293M  100G   1% /
/dev/mapper/RHGS_VG4-vol4             10G   33M   10G   1% /b4
/dev/mapper/RHGS_VG2-vol2             10G   33M   10G   1% /b2
devtmpfs                             3.9G     0  3.9G   0% /dev
tmpfs                                3.9G     0  3.9G   0% /dev/shm
/dev/mapper/RHGS_VG1-vol1             10G   33M   10G   1% /b1
/dev/mapper/RHGS_VG3-vol3             10G   33M   10G   1% /b3
/dev/mapper/rhelah_dhcp42--208-root  3.0G  979M  2.1G  32% /etc/hosts
tmpfs                                3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs                                3.9G  8.4M  3.9G   1% /run

[root@dhcp42-208 /]# mount |grep RHGS
/dev/mapper/RHGS_VG4-vol4 on /b4 type xfs (rw,relatime,seclabel,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota)
/dev/mapper/RHGS_VG2-vol2 on /b2 type xfs (rw,relatime,seclabel,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota)
/dev/mapper/RHGS_VG1-vol1 on /b1 type xfs (rw,relatime,seclabel,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota)
/dev/mapper/RHGS_VG3-vol3 on /b3 type xfs (rw,relatime,seclabel,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota)

[root@dhcp42-208 /]# cat /etc/redhat-storage-release 
Red Hat Gluster Storage Server 3.1 Update 2 ( Container)

[root@dhcp42-208 /]# rpm -qa |grep glusterfs
glusterfs-libs-3.7.5-16.el7rhgs.x86_64
glusterfs-client-xlators-3.7.5-16.el7rhgs.x86_64
glusterfs-fuse-3.7.5-16.el7rhgs.x86_64
glusterfs-geo-replication-3.7.5-16.el7rhgs.x86_64
glusterfs-3.7.5-16.el7rhgs.x86_64
glusterfs-api-3.7.5-16.el7rhgs.x86_64
glusterfs-cli-3.7.5-16.el7rhgs.x86_64
glusterfs-server-3.7.5-16.el7rhgs.x86_64
#########

Note You need to log in before you can comment on or make changes to this bug.