Bug 1293667 - [rhgs docker image] Stopping/killing a running rhgs container hangs for ever
Summary: [rhgs docker image] Stopping/killing a running rhgs container hangs for ever
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: rhgs-server-container
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: RHGS 3.1.2
Assignee: Mohamed Ashiq
QA Contact: Prasanth
URL:
Whiteboard:
Depends On: 1294459
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-12-22 15:51 UTC by Prasanth
Modified: 2016-12-19 17:43 UTC (History)
7 users (show)

Fixed In Version: RHELAH 7.2.2, rhgs-server-docker-3.1.2-7
Doc Type: Bug Fix
Doc Text:
Cause: systemd issue. https://bugzilla.redhat.com/show_bug.cgi?id=1285863 Consequence: docker stop in Atomic host hangs. Fix: systemd-sysv-219-19.el7_2.4.x86_64 systemd-219-19.el7_2.4.x86_64 systemd-libs-219-19.el7_2.4.x86_64 Result: docker stop runs successfully in Atomic Host.
Clone Of:
Environment:
Last Closed: 2016-12-19 17:43:39 UTC
Embargoed:


Attachments (Terms of Use)
dmesg log (74.96 KB, text/plain)
2015-12-22 15:54 UTC, Prasanth
no flags Details

Description Prasanth 2015-12-22 15:51:38 UTC
Description of problem:

Stopping or killing a running rhgs container image seems to hang for ever.

Version-Release number of selected component (if applicable):

#####
-bash-4.2# cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.2

-bash-4.2# sudo docker images
REPOSITORY                                                               TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7   latest              7dfbd648bde2        18 hours ago        252.7 MB
#####

How reproducible: Mostly


Steps to Reproduce:
####
* Prepare the setup for Snapshot as per https://docs.google.com/document/d/1o4_funQmy91dmIG_qAuxowDwfk1jrRIyrGx2wzXNNmU/edit?ts=560bc1a5&pli=1 including the provided work-arounds
* sudo docker pull rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7
* sudo docker run --privileged=True --net=host -d --name snapnode2 -v /etc/glusterfs/:/etc/glusterfs/ -v /var/lib/glusterd/:/var/lib/glusterd/ -v /var/log/glusterfs/:/var/log/glusterfs/ -v /mnt/brick1:/b1 -v /mnt/brick2:/b2 -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /dev:/dev -ti rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7 /sbin/init
* sudo docker ps
* sudo docker exec -ti 9bf966067e6d /bin/bash
* exit
* sudo docker stop 9bf966067e6d
####

Actual results: Try to stop or kill a running rhgs container seems to be hanging always.


Expected results: Stop/kill should be successful


Additional info:

From Atomic Host:

############
-bash-4.2#  df -h
Filesystem                           Size  Used Avail Use% Mounted on
/dev/mapper/rhelah_dhcp37--114-root  3.0G  940M  2.1G  31% /
devtmpfs                             3.9G     0  3.9G   0% /dev
tmpfs                                3.9G     0  3.9G   0% /dev/shm
tmpfs                                3.9G  376K  3.9G   1% /run                                                                                                             
tmpfs                                3.9G     0  3.9G   0% /sys/fs/cgroup                                                                                                   
/dev/sda1                            297M   87M  211M  30% /boot                                                                                                            
tmpfs                                783M     0  783M   0% /run/user/0                                                                                                      
/dev/mapper/RHGS_VG1-vol1             10G   33M   10G   1% /var/mnt/brick1                                                                                                  
/dev/mapper/RHGS_VG2-vol2             10G   33M   10G   1% /var/mnt/brick2


-bash-4.2# lvs
  LV          VG                Attr       LSize  Pool       Origin Data%  Meta%  Move Log Cpy%Sync Convert
  thin_pool0  RHGS_VG1          twi-aotz-- 19.00g                   0.06   0.57                            
  vol1        RHGS_VG1          Vwi-a-tz-- 10.00g thin_pool0        0.11                                   
  thin_pool0  RHGS_VG2          twi-aotz-- 19.00g                   0.06   0.57                            
  vol2        RHGS_VG2          Vwi-a-tz-- 10.00g thin_pool0        0.11                                   
  docker-pool rhelah_dhcp37-114 twi-aot---  4.07g                   11.65  0.73                            
  root        rhelah_dhcp37-114 -wi-ao----  3.00g                                                          
  swap        rhelah_dhcp37-114 -wi-ao----  1.50g  

############

Comment 1 Prasanth 2015-12-22 15:54:46 UTC
Created attachment 1108668 [details]
dmesg log

Comment 3 Mohamed Ashiq 2015-12-24 17:23:03 UTC
I am trying to reproduce the issue 

-bash-4.2# cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.2

-bash-4.2# lvs
  LV          VG                Attr       LSize  Pool     Origin Data%  Meta%  Move Log Cpy%Sync Convert
  docker-pool rhelah_dhcp42-124 twi-a-t---  5.87g                 0.99   0.16                            
  root        rhelah_dhcp42-124 -wi-ao----  3.00g                                                        
  swap        rhelah_dhcp42-124 -wi-ao----  2.00g                                                        
  lvol0       vg1               -wi------- 20.00m                                                        
  thinpool    vg1               twi-aotz--  5.00g                 0.00   0.73                            
  thinvol     vg1               Vwi-a-tz-- 10.00g thinpool        0.00                 

-bash-4.2# df -h
Filesystem                           Size  Used Avail Use% Mounted on
/dev/mapper/rhelah_dhcp42--124-root  3.0G  939M  2.1G  31% /
devtmpfs                             902M     0  902M   0% /dev
tmpfs                                920M     0  920M   0% /dev/shm
tmpfs                                920M  440K  920M   1% /run
tmpfs                                920M     0  920M   0% /sys/fs/cgroup
/dev/vda1                            297M   87M  211M  30% /boot
tmpfs                                184M     0  184M   0% /run/user/0
/dev/mapper/vg1-thinvol               10G   33M   10G   1% /var/mnt/brick

-bash-4.2# rpm -qa | grep docker
docker-1.8.2-8.el7.x86_64
docker-selinux-1.8.2-8.el7.x86_64
python-docker-py-1.4.0-118.el7.noarch

-bash-4.2# docker images
REPOSITORY                                                               TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7   latest              7dfbd648bde2        2 days ago          252.7 MB

-bash-4.2# docker -D run -d --privileged=true --net=host --name node -v /etc/glusterfs:/etc/glusterfs -v /var/log/glusterfs:/var/log/glusterfs -v /var/lib/glusterd:/var/lib/glusterd -v /var/mnt/brick:/b1 -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /dev:/dev rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7
044e2c7e7c9a3859c5b5d2f88d2da1ad6edf4e25d4cd9d3c000441321fddc28d
DEBU[0000] End of CmdRun(), Waiting for hijack to finish. 

-bash-4.2# docker exec -it $(docker ps -laq) /bin/bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory

[root@dhcp42-124 /]# df -h
Filesystem                           Size  Used Avail Use% Mounted on
/dev/dm-10                           100G  289M  100G   1% /
/dev/mapper/vg1-thinvol               10G   33M   10G   1% /b1
devtmpfs                             902M     0  902M   0% /dev
tmpfs                                920M     0  920M   0% /dev/shm
/dev/mapper/rhelah_dhcp42--124-root  3.0G  940M  2.1G  31% /etc/hosts
tmpfs                                920M     0  920M   0% /sys/fs/cgroup
tmpfs                                920M  8.4M  912M   1% /run


Mounted the vol
ashiq #  ps -aux | grep /ash
root      9987  0.0  0.0 614904  9592 ?        Ssl  22:38   0:00 /usr/local/sbin/glusterfs --volfile-server=10.70.42.124 --volfile-id=/vol /ash

Created snapshot
[root@dhcp42-124 /]# gluster snapshot create snap1 vol
snapshot create: success: Snap snap1_GMT-2015.12.24-10.17.08 created successfully

[root@dhcp42-124 /]# exit
exit

-bash-4.2# docker ps
CONTAINER ID        IMAGE                                                                    COMMAND             CREATED             STATUS              PORTS               NAMES
044e2c7e7c9a        rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7   "/usr/sbin/init"    13 minutes ago      Up 13 minutes                           node

-bash-4.2# docker stop 044e2c7e7c9a
044e2c7e7c9a

It worked, Will check again with other setups too.

How often did you see this happening?

This is the test system 10.70.42.124

Will keep on trying this, if it fails.

Please let me know if I am missing something.

Comment 4 Humble Chirammal 2015-12-28 07:03:06 UTC
This seems a behavior caused due to https://bugzilla.redhat.com/show_bug.cgi?id=1293669 . That said, going through the logs, I can see there are lots of messages which says that the "unmounting" of filesystems fails and it repeat in a loop manner. iic, the docker run command was successful on the bind mount eventhough you were missing some mount points inside the container. It could be that docker is trying to unmount the filesystem which are not yet mounted inside the container. 

To isolate it,

Is this issue experienced only in containers which failed to bind mount certain paths ? Or Are you experiencing this issue in all the containers even after successful mount of all filesystems which are specified in the command line ?
Are there any other container tried to start with --net=host option in the same system ?

Comment 5 Prasanth 2015-12-31 13:41:30 UTC
(In reply to Humble Chirammal from comment #4)
> This seems a behavior caused due to
> https://bugzilla.redhat.com/show_bug.cgi?id=1293669 . That said, going
> through the logs, I can see there are lots of messages which says that the
> "unmounting" of filesystems fails and it repeat in a loop manner. iic, the
> docker run command was successful on the bind mount eventhough you were
> missing some mount points inside the container. It could be that docker is
> trying to unmount the filesystem which are not yet mounted inside the
> container. 
> 
> To isolate it,
> 
> Is this issue experienced only in containers which failed to bind mount
> certain paths ? Or Are you experiencing this issue in all the containers
> even after successful mount of all filesystems which are specified in the
> command line ?

No, this issue is not seen in other non-rhgs containers where the bind-mount is happening successfully. AFAIK, this issue is only seen with the rhgs containers where it fails to mount/un-mount the filesystem and docker stop hangs

> Are there any other container tried to start with --net=host option in the
> same system ?

No

Comment 6 Humble Chirammal 2016-01-05 13:01:44 UTC
(In reply to Prasanth from comment #5)
> (In reply to Humble Chirammal from comment #4)
> > This seems a behavior caused due to
> > https://bugzilla.redhat.com/show_bug.cgi?id=1293669 . That said, going
> > through the logs, I can see there are lots of messages which says that the
> > "unmounting" of filesystems fails and it repeat in a loop manner. iic, the
> > docker run command was successful on the bind mount eventhough you were
> > missing some mount points inside the container. It could be that docker is
> > trying to unmount the filesystem which are not yet mounted inside the
> > container. 
> > 
> > To isolate it,
> > 
> > Is this issue experienced only in containers which failed to bind mount
> > certain paths ? Or Are you experiencing this issue in all the containers
> > even after successful mount of all filesystems which are specified in the
> > command line ?
> 
> No, this issue is not seen in other non-rhgs containers where the bind-mount
> is happening successfully. AFAIK, this issue is only seen with the rhgs
> containers where it fails to mount/un-mount the filesystem and docker stop
> hangs
> 

Thanks for the confirmation. This looks like a side effect of bz#1294459. I have marked this bug as a dependency to the same bz.

Comment 10 Prasanth 2016-01-21 13:13:43 UTC
Stopping a container running the latest image (rhgs-server-rhel7:3.1.2-6) still seems to hang for ever in one of the host. See below: 

##########
-bash-4.2# docker ps
CONTAINER ID        IMAGE                                                                    COMMAND             CREATED             STATUS              PORTS               NAMES
0eb52bae4664        rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7   "/usr/sbin/init"    25 seconds ago      Up 22 seconds                           snapnode1
 
-bash-4.2# docker stop 0eb52bae4664
^C-bash-4.2# 
-bash-4.2# 
-bash-4.2# reboot 
Connection to 10.70.37.81 closed by remote host.
Connection to 10.70.37.81 closed.



-- Logs begin at Tue 2016-01-19 08:55:38 IST, end at Thu 2016-01-21 11:43:30 IST. --
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99869]: umount: /var: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99869]: (In some cases useful info about processes that use
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99869]: the device is found by lsof(8) or fuser(1))
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Unit usr.mount is bound to inactive unit dev-mapper-rhelah_dhcp37\x2d\x2d81\x2droot.device. Stopping, too.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99870]: umount: /usr: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99870]: (In some cases useful info about processes that use
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99870]: the device is found by lsof(8) or fuser(1))
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99872]: umount: /usr: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99872]: (In some cases useful info about processes that use
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99872]: the device is found by lsof(8) or fuser(1))
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99873]: umount: /var: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99873]: (In some cases useful info about processes that use
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99873]: the device is found by lsof(8) or fuser(1))
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99871]: umount: /sysroot: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99871]: (In some cases useful info about processes that use
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99871]: the device is found by lsof(8) or fuser(1))
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Failed to forward Released message: No buffer space available
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Failed to forward Released message: No buffer space available
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Failed to forward Released message: No buffer space available
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99880]: umount: /usr: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99883]: umount: /var: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99883]: (In some cases useful info about processes that use
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99883]: the device is found by lsof(8) or fuser(1))
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99882]: umount: /sysroot: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99882]: (In some cases useful info about processes that use
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99882]: the device is found by lsof(8) or fuser(1))
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99884]: umount: /usr: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99884]: (In some cases useful info about processes that use
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99884]: the device is found by lsof(8) or fuser(1))
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99885]: umount: /var: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99886]: umount: /sysroot: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99890]: umount: /var: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99888]: umount: /usr: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99893]: umount: /var: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99894]: umount: /usr: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99894]: (In some cases useful info about processes that use
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99894]: the device is found by lsof(8) or fuser(1))
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99896]: umount: /usr: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99897]: umount: /var: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99903]: umount: /sysroot: target is busy.
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Failed to forward Released message: No buffer space available
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Failed to forward Released message: No buffer space available
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Failed to forward Released message: No buffer space available
Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99908]: umount: /sysroot: target is busy
##########

Do let me know if you need the setup or any further details from my side.

Comment 11 Prasanth 2016-01-21 13:21:48 UTC
Additional Info: 

AFAIK, this is happening on the nodes in which you are hitting Bug 1294776. If one ore all of the LV's for bricks are getting unmounted from the atomic host automatically on starting a RHGS container, stopping seems to hang forever. Otherwise, it works fine as expected.

Hope this helps!

Comment 12 Mohamed Ashiq 2016-01-21 14:08:46 UTC
(In reply to Prasanth from comment #10)
> Stopping a container running the latest image (rhgs-server-rhel7:3.1.2-6)
> still seems to hang for ever in one of the host. See below: 
> 
> ##########
> -bash-4.2# docker ps
> CONTAINER ID        IMAGE                                                   
> COMMAND             CREATED             STATUS              PORTS           
> NAMES
> 0eb52bae4664       
> rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7  
> "/usr/sbin/init"    25 seconds ago      Up 22 seconds                       
> snapnode1
>  
> -bash-4.2# docker stop 0eb52bae4664
> ^C-bash-4.2# 
> -bash-4.2# 
> -bash-4.2# reboot 
> Connection to 10.70.37.81 closed by remote host.
> Connection to 10.70.37.81 closed.
> 
> 
> 
> -- Logs begin at Tue 2016-01-19 08:55:38 IST, end at Thu 2016-01-21 11:43:30
> IST. --
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99869]: umount:
> /var: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99869]: (In some
> cases useful info about processes that use
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99869]: the device
> is found by lsof(8) or fuser(1))
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Unit usr.mount
> is bound to inactive unit dev-mapper-rhelah_dhcp37\x2d\x2d81\x2droot.device.
> Stopping, too.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99870]: umount:
> /usr: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99870]: (In some
> cases useful info about processes that use
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99870]: the device
> is found by lsof(8) or fuser(1))
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99872]: umount:
> /usr: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99872]: (In some
> cases useful info about processes that use
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99872]: the device
> is found by lsof(8) or fuser(1))
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99873]: umount:
> /var: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99873]: (In some
> cases useful info about processes that use
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99873]: the device
> is found by lsof(8) or fuser(1))
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99871]: umount:
> /sysroot: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99871]: (In some
> cases useful info about processes that use
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99871]: the device
> is found by lsof(8) or fuser(1))
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Failed to
> forward Released message: No buffer space available
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Failed to
> forward Released message: No buffer space available
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Failed to
> forward Released message: No buffer space available
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99880]: umount:
> /usr: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99883]: umount:
> /var: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99883]: (In some
> cases useful info about processes that use
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99883]: the device
> is found by lsof(8) or fuser(1))
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99882]: umount:
> /sysroot: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99882]: (In some
> cases useful info about processes that use
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99882]: the device
> is found by lsof(8) or fuser(1))
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99884]: umount:
> /usr: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99884]: (In some
> cases useful info about processes that use
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99884]: the device
> is found by lsof(8) or fuser(1))
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99885]: umount:
> /var: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99886]: umount:
> /sysroot: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99890]: umount:
> /var: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99888]: umount:
> /usr: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99893]: umount:
> /var: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99894]: umount:
> /usr: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99894]: (In some
> cases useful info about processes that use
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99894]: the device
> is found by lsof(8) or fuser(1))
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99896]: umount:
> /usr: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99897]: umount:
> /var: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99903]: umount:
> /sysroot: target is busy.
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Failed to
> forward Released message: No buffer space available
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Failed to
> forward Released message: No buffer space available
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com systemd[1]: Failed to
> forward Released message: No buffer space available
> Jan 19 08:55:38 dhcp37-81.lab.eng.blr.redhat.com umount[99908]: umount:
> /sysroot: target is busy
> ##########
> 
> Do let me know if you need the setup or any further details from my side.

The logs are from Jan 19th, however it looks like  the mentioned container started (   25 seconds ago  ) on Thu 2016-01-21 . Are you facing the issue with this container ? because we would like to confirm that, the logs provided in c#10 belongs to the failed attempt or not?

Comment 13 Prasanth 2016-01-22 07:37:59 UTC
(In reply to Mohamed Ashiq from comment #12)
> (In reply to Prasanth from comment #10)

> The logs are from Jan 19th, however it looks like  the mentioned container
> started (   25 seconds ago  ) on Thu 2016-01-21 . Are you facing the issue
> with this container ? because we would like to confirm that, the logs
> provided in c#10 belongs to the failed attempt or not?

If i remember it correctly, the container was started on 2016-01-19 itself. Can you help me understand how you came to a conclusion that the container was started on Thu 2016-01-21? 

To clarify, the test results are from 2016-01-19 itself from the same container mentioned in my comment and the logs provided in c#10 do belongs to the failed attempt.

I can also try once again in my setup and attach the latest test results to avoid any further confusion. Hope this helps!

Comment 15 Prasanth 2016-02-16 07:53:10 UTC
Verified as fixed in RHELAH 7.2.2, rhgs-server-docker-3.1.2-7

Stopping a rhgs container now works as expected.

#####
-bash-4.2# cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.2


-bash-4.2# rpm-ostree status
  TIMESTAMP (UTC)         VERSION   ID             OSNAME               REFSPEC                                                        
* 2016-02-12 16:43:35     7.2.2     a903629278     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard     
  2015-12-03 19:40:36     7.2.1     aaf67b91fa     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard 


-bash-4.2# docker ps
CONTAINER ID        IMAGE                                                                            COMMAND             CREATED             STATUS              PORTS               NAMES
1463f8a547a7        rcm-img-docker01.build.eng.bos.redhat.com:5001/rhgs3/rhgs-server-rhel7:3.1.2-7   "/usr/sbin/init"    17 hours ago        Up 17 hours                             node2
-bash-4.2# 
-bash-4.2# 
-bash-4.2# docker stop 1463f8a547a7
1463f8a547a7
-bash-4.2# docker rm 1463f8a547a7
1463f8a547a7
#####


Note You need to log in before you can comment on or make changes to this bug.