Bug 1406483
| Summary: | [ESXi][open-vm-tools][RHEL7.4]When running docker containers, the open-vm-tools SyncDriver fails to quiesce filesystems and returns an error | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Stephen Beal <sbeal> | ||||||
| Component: | open-vm-tools | Assignee: | Richard W.M. Jones <rjones> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Yaju Cao <yacao> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | urgent | ||||||||
| Version: | 7.3 | CC: | ailan, bbreard, boyang, brdwyer, cavery, dphillip, gene.siepka, jbryant, jherrman, jjarvis, jmagrini, jreznik, jsavanyo, ldu, leiwang, mkalinin, ravindrakumar, rjones, rmcswain, sbeal, vmware-gos-qa, yacao | ||||||
| Target Milestone: | rc | Keywords: | ZStream | ||||||
| Target Release: | --- | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | open-vm-tools-10.0.5-4.el7 | Doc Type: | Bug Fix | ||||||
| Doc Text: |
Prior to this update, taking a snapshot of a Red Hat Enterprise Linux 7 guest on the VMWare ESXi hypervisor failed if docker images were running on the guest. With this update, the vmtoolsd service properly quiesces the guest file system in the described scenario, which makes it possible for the snapshots to be taken.
|
Story Points: | --- | ||||||
| Clone Of: | |||||||||
| : | 1419062 1421792 (view as bug list) | Environment: | |||||||
| Last Closed: | 2017-08-01 18:04:22 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 1419062, 1421792 | ||||||||
| Attachments: |
|
||||||||
Per customer: "Yes, 100% reproducible. With running containers, the VM snapshot fails. Stop the containers, the VM snapshot succeeds. open-vm-tools itself isn't running in a container? This is open-vm-tools running on the host, and containers happen to be running also on the same host? (In reply to Stephen Beal from comment #0) > [Dec 15 10:23:53.645] [ debug] [vmsvc] SyncDriver: opening path 'net'. > [Dec 15 10:23:53.645] [ debug] [vmsvc] SyncDriver: failed to open 'net': 2 > (No such file or directory) What is this mount point 'net' for? This typically happens for stale mount points. If there is a valid mount point, we should be able to open() it. It would be very helpful to get output of 'mount' command from the problematic VM. Could we get the output of the 'mount' command running in the host when a container is running? "[root@ltstpnasv22 ~]# mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=931040k,nr_inodes=232760,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/mapper/vg_os-lv_root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/mapper/vg_os-lv_usr on /usr type xfs (rw,relatime,seclabel,attr2,inode64,noquota) selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=300,minproto=5,maxproto=5,direct) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel) nfsd on /proc/fs/nfsd type nfsd (rw,relatime) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel) /dev/sda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/mapper/vg_os-lv_opt on /opt type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/mapper/vg_os-lv_opt_openv on /opt/openv type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/mapper/vg_os-lv_home on /home type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/mapper/vg_os-lv_tmp on /tmp type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/mapper/vg_app0-lv_opt_splunk on /opt/splunk type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/mapper/vg_os-lv_var on /var type xfs (rw,relatime,seclabel,attr2,inode64,noquota) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) /dev/mapper/vg_os-lv_var_opt on /var/opt type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/mapper/vg_os-lv_var_log on /var/log type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/mapper/vg_os-lv_var_log_audit on /var/log/audit type xfs (rw,relatime,seclabel,attr2,inode64,noquota) linuxhome.staples.com:/nethome on /nethome type nfs (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,soft,proto=tcp,timeo=1000,retrans=2,sec=sys,mountaddr=10.4.15.46,mountvers=3,mountport=1234,mountproto=tcp,local_lock=none,addr=10.4.15.46) unixhome.staples.com:/unixhome on /home/net1 type nfs (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,soft,proto=tcp,timeo=1000,retrans=2,sec=sys,mountaddr=10.4.15.48,mountvers=3,mountport=1234,mountproto=tcp,local_lock=none,addr=10.4.15.48) /dev/mapper/vg_os-lv_var on /var/lib/docker/100000.100000/devicemapper type xfs (rw,relatime,seclabel,attr2,inode64,noquota) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=188416k,mode=700) /dev/mapper/docker-253:10-4316585-b48569d5609021d414a3aa91308b683f4c7eb14725bd9c9abe1f83ffa20fdd62 on /var/lib/docker/100000.100000/devicemapper/mnt/b48569d5609021d414a3aa91308b683f4c7eb14725bd9c9abe1f83ffa20fdd62 type xfs (rw,relatime,seclabel,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota) shm on /var/lib/docker/100000.100000/containers/bbf85ac246c75c175eb71772d9a9b06fbd88a46b8260d77dfd4eba6cc1f14353/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,size=65536k) proc on /run/docker/netns/68b23eba68a3 type proc (rw,nosuid,nodev,noexec,relatime) /dev/mapper/docker-253:10-4316585-bc984e26f0ab96e88f2b923a727235fbbce6106f5bf68e34e8066fa4cf40004c on /var/lib/docker/100000.100000/devicemapper/mnt/bc984e26f0ab96e88f2b923a727235fbbce6106f5bf68e34e8066fa4cf40004c type xfs (rw,relatime,seclabel,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota) shm on /var/lib/docker/100000.100000/containers/e3b0fb32747919010bc825356043bf5eeb01b1412b49302e3beb64c2ceec824d/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,size=65536k) proc on /run/docker/netns/649937cc3362 type proc (rw,nosuid,nodev,noexec,relatime) [root@ltstpnasv22 ~]# I believe this is all fixed in open-vm-tools 10.1+ https://github.com/vmware/open-vm-tools/blob/master/ReleaseNotes.md I believe this should also alleviate issues with the VMHGFS kernel module for 7.4. (In reply to BDwyerTech from comment #10) > I believe this is all fixed in open-vm-tools 10.1+ > > https://github.com/vmware/open-vm-tools/blob/master/ReleaseNotes.md > > I believe this should also alleviate issues with the VMHGFS kernel module > for 7.4. Failure to open a mount path is not fixed in 10.1 yet. (In reply to Ravindra Kumar from comment #7) > It would be very helpful to get output of 'mount' command from the > problematic VM. Please see it in comment#9 (In reply to Stephen Beal from comment #9) > "[root@ltstpnasv22 ~]# mount > sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel) > proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) > ... > cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup > (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls) > ... > linuxhome.staples.com:/nethome on /nethome type nfs > (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,soft,proto=tcp, > timeo=1000,retrans=2,sec=sys,mountaddr=10.4.15.46,mountvers=3,mountport=1234, > mountproto=tcp,local_lock=none,addr=10.4.15.46) > unixhome.staples.com:/unixhome on /home/net1 type nfs > (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,soft,proto=tcp, > timeo=1000,retrans=2,sec=sys,mountaddr=10.4.15.48,mountvers=3,mountport=1234, > mountproto=tcp,local_lock=none,addr=10.4.15.48) > ... > proc on /run/docker/netns/68b23eba68a3 type proc > (rw,nosuid,nodev,noexec,relatime) > /dev/mapper/docker-253:10-4316585- > ... > proc on /run/docker/netns/649937cc3362 type proc > (rw,nosuid,nodev,noexec,relatime) > [root@ltstpnasv22 ~]# I don't see any 'net' in the list. Also, this is a bit weird path as I'd expect all paths to be absolute. I don't see an answer to Richard's questions in comment #3. Could we please get answers to those too? If open-vm-tools is running inside a container, that is not a supported configuration yet. Hi Ravindra, I apologize on behalf of RH for the update not being visible to answer your question in #3 no it is not being ran in a container. This was confirmed by the customer directly. I have spoken with my docker specialist and he is assuming that this may be because of the mount namespaces docker creates for containers - specifically that OVT does not know how to handle them. This is purely conjectural as he has not looked at the OVT code and is going entirely off of what the customer has said with regards to reproducing the error. Piling on here since I just discovered this bug.. Having same issue. RHEL 7.2 Guest on VMware. RHEL Guest running docker containers. [root@vcld16rdabpws10 ~]# docker version Client: Version: 1.12.5 API version: 1.24 Go version: go1.6.4 Git commit: 7392c3b Built: Fri Dec 16 02:24:06 2016 OS/Arch: linux/amd64 Server: Version: 1.12.5 API version: 1.24 Go version: go1.6.4 Git commit: 7392c3b Built: Fri Dec 16 02:24:06 2016 OS/Arch: linux/amd64 [root@vcld16rdabpws10 ~]# [root@vcld16rdabpws10 ~]# vmware-toolbox-cmd -v 10.0.9.55972 (build-3917699) VMware ESX hosts are 6.0.0 Output from Mount command below: [root@vcld16rdabpws10 ~]# mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,size=2959256k,nr_inodes=739814,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu) cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/mapper/vgOS-root on / type xfs (rw,relatime,attr2,inode64,noquota) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=300,minproto=5,maxproto=5,direct) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime) nfsd on /proc/fs/nfsd type nfsd (rw,relatime) /dev/mapper/vgOS-tmp on /tmp type xfs (rw,relatime,attr2,inode64,noquota) /dev/sda1 on /boot type xfs (rw,relatime,attr2,inode64,noquota) /dev/mapper/vgAPPS-apps on /apps type xfs (rw,relatime,attr2,inode64,noquota) /dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro) /dev/mapper/vgOS-home on /home type xfs (rw,relatime,attr2,inode64,noquota) /dev/mapper/vgOS-opt on /opt type xfs (rw,relatime,attr2,inode64,noquota) /dev/mapper/vgOS-var on /var type xfs (rw,relatime,attr2,inode64,noquota) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) /dev/mapper/vgOS-var on /var/lib/docker/devicemapper type xfs (rw,relatime,attr2,inode64,noquota) /dev/mapper/docker-253:4-16782674-9bf253461a5d2430b83b097b0b14351ceb57c344bf9621f4a0dbaf2c3914f8de on /var/lib/docker/devicemapper/mnt/9bf253461a5d2430b83b097b0b14351ceb57c344bf9621f4a0dbaf2c3914f8de type xfs (rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota) shm on /var/lib/docker/containers/7254067d95b8ec0dc51ae567a76a33054104edc581dc00225f0371eecca32963/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k) proc on /run/docker/netns/fcbcbab093a6 type proc (rw,nosuid,nodev,noexec,relatime) /dev/mapper/docker-253:4-16782674-f1a721e6f4edbfec8ecf1b922cdf5baca54c24c8bb751eac98eaa9bec41691f7 on /var/lib/docker/devicemapper/mnt/f1a721e6f4edbfec8ecf1b922cdf5baca54c24c8bb751eac98eaa9bec41691f7 type xfs (rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota) shm on /var/lib/docker/containers/0f5279893df63a1ae995ce25edadfd2c021df7b66d439d042911696e55663d75/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k) proc on /run/docker/netns/014bc315e959 type proc (rw,nosuid,nodev,noexec,relatime) tmpfs on /run/user/3743680 type tmpfs (rw,nosuid,nodev,relatime,size=593884k,mode=700,uid=3743680,gid=20) Updating what we have done for this issue over last 2 weeks. We could reproduce it by running a container on docker 1.12. We used 'docker run -d -P training/webapp python app.py' which created a mount point '/run/docker/netns/f919fe29adb6' that was not a directory. This mount point shows up as net:[LARGE_NUMBER] in /etc/mtab and /proc/mounts: # grep net /etc/mtab cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0 proc net:[4026532713] proc rw,nosuid,nodev,noexec,relatime 0 0 # grep net /proc/mounts cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0 proc net:[4026532713] proc rw,nosuid,nodev,noexec,relatime 0 0 # grep net /proc/self/mounts cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0 proc net:[4026532713] proc rw,nosuid,nodev,noexec,relatime 0 0 'mount' command that looks at /proc/self/mountinfo shows the same mount point differently. # mount | grep net cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls) proc on /run/docker/netns/f919fe29adb6 type proc (rw,nosuid,nodev,noexec,relatime) # grep net /proc/self/mountinfo 30 24 0:25 / /sys/fs/cgroup/net_cls rw,nosuid,nodev,noexec,relatime shared:13 - cgroup cgroup rw,net_cls 265 23 0:3 / /run/docker/netns/f919fe29adb6 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw # fsfreeze -f /run/docker/netns/f919fe29adb6 fsfreeze: /run/docker/netns/f919fe29adb6: is not a directory With modified fsfreeze to allow non-directories: # ./fsfreeze -f /run/docker/netns/f919fe29adb6 fsfreeze: /run/docker/netns/f919fe29adb6: freeze failed: Operation not supported # stat /run/docker/netns/f919fe29adb6 File: â/run/docker/netns/f919fe29adb6â Size: 0 Blocks: 0 IO Block: 1024 regular empty file Device: 3h/3d Inode: 4026532713 Links: 1 Access: (0444/-r--r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:proc_t:s0 Access: 2017-01-26 19:28:10.703557060 -0500 Modify: 2017-01-26 19:28:10.703557060 -0500 Change: 2017-01-26 19:28:10.703557060 -0500 Birth: - # file /run/docker/netns/f919fe29adb6 /run/docker/netns/f919fe29adb6: empty # ls -l /run/docker/netns/f919fe29adb6 -r--r--r--. 1 root root 0 Jan 26 19:28 /run/docker/netns/f919fe29adb6 We tried different types of containers even for a simple one like hello-world container, docker creates such special mount point. Running a test program that uses getmntent() C-APIs just like vmtoolsd: # ~/mountlist | grep net fsname=cgroup, dir=/sys/fs/cgroup/net_cls, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,net_cls, freq=0, passno=0 fsname=proc, dir=net:[4026532713], type=proc, opts=rw,nosuid,nodev,noexec,relatime, freq=0, passno=0 <<<<=== This is how vmtoolsd sees it. # cat /etc/redhat-release Red Hat Enterprise Linux Client release 7.2 (Maipo) # uname -a Linux promb-2s-dhcp95-204.eng.vmware.com 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux # docker version Client: Version: 1.12.6 API version: 1.24 Go version: go1.6.4 Git commit: 78d1802 Built: Tue Jan 10 20:20:01 2017 OS/Arch: linux/amd64 Server: Version: 1.12.6 API version: 1.24 Go version: go1.6.4 Git commit: 78d1802 Built: Tue Jan 10 20:20:01 2017 OS/Arch: linux/amd64 [root@promb-2s-dhcp95-204 util-linux]# vmtoolsd -v VMware Tools daemon, version 10.1.0.57774 (build-4449150) We are going to put a workaround in vmtoolsd to tolerate such mount points, but this needs to be looked into from Linux perspective as well. Basically, why mtab/mounts entry is not matching with /proc/self/mountinfo? Richard, could we please include folks who work on 'mount' and /etc/mtab entries to answer the inconsistency between mtab and /proc/self/mountinfo? Take a look at my comment in the other bug: https://bugzilla.redhat.com/show_bug.cgi?id=1166465#c60 Created attachment 1246011 [details] 10.0.5 based patch for special mount points Rich, attached is the patch we have implemented for working around the docker mount point issue. This is built for 10.0.5 code and can be applied on top of the last patch we provided for bug 1398802. Package containing the patch: http://oirase.annexia.org/tmp/bz1406483 Thanks for that Richard! It's going to help speed along my testing! Verify this bug with open-vm-tools-10.1.5-3.el7.x86_64, this issue is fixed. Reproduce/Verify steps: 1. Install docker 2. Start a network container, e.g. $ docker run -d -P --name web training/webapp python app.py 3. Take snapshot, choose 'Quiesce guest file system' Test results: At step 3, open-vm-tools-10.0.5-2.el7 failed, open-vm-tools-10.1.5-3.el7.x86_64 can complete successfully Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1906 |
Created attachment 1233931 [details] sosreport Description of problem: [Dec 15 10:23:53.626] [ debug] [vmsvc] SyncDriver: opening path '/opt/openv'. [Dec 15 10:23:53.626] [ debug] [vmsvc] SyncDriver: freezing path '/opt/openv'. [Dec 15 10:23:53.634] [ debug] [vmsvc] SyncDriver: successfully froze '/opt/openv'. [Dec 15 10:23:53.634] [ debug] [vmsvc] SyncDriver: opening path '/opt/splunk'. [Dec 15 10:23:53.634] [ debug] [vmsvc] SyncDriver: freezing path '/opt/splunk'. [Dec 15 10:23:53.644] [ debug] [vmsvc] SyncDriver: successfully froze '/opt/splunk'. [Dec 15 10:23:53.644] [ debug] [vmsvc] SyncDriver: opening path '/tmp'. [Dec 15 10:23:53.644] [ debug] [vmsvc] SyncDriver: freezing path '/tmp'. [Dec 15 10:23:53.644] [ debug] [vmsvc] SyncDriver: freeze on '/tmp' returned: 16 (Device or resource busy) [Dec 15 10:23:53.644] [ debug] [vmsvc] SyncDriver: opening path '/var/tmp'. [Dec 15 10:23:53.644] [ debug] [vmsvc] SyncDriver: freezing path '/var/tmp'. [Dec 15 10:23:53.644] [ debug] [vmsvc] SyncDriver: freeze on '/var/tmp' returned: 16 (Device or resource busy) [Dec 15 10:23:53.644] [ debug] [vmsvc] SyncDriver: opening path '/var/lib/docker/100000.100000/containers/b26df88b6051915c51cad65dc0a17a2236bd230b19dbfb4cc7dec783b1469606/shm'. [Dec 15 10:23:53.644] [ debug] [vmsvc] SyncDriver: freezing path '/var/lib/docker/100000.100000/containers/b26df88b6051915c51cad65dc0a17a2236bd230b19dbfb4cc7dec783b1469606/shm'. [Dec 15 10:23:53.645] [ debug] [vmsvc] SyncDriver: freeze on '/var/lib/docker/100000.100000/containers/b26df88b6051915c51cad65dc0a17a2236bd230b19dbfb4cc7dec783b1469606/shm' returned: 95 (Operation not supported) [Dec 15 10:23:53.645] [ debug] [vmsvc] SyncDriver: opening path 'net'. [Dec 15 10:23:53.645] [ debug] [vmsvc] SyncDriver: failed to open 'net': 2 (No such file or directory) [Dec 15 10:23:54.069] [ debug] [vmsvc] RpcIn: received 5 bytes, content:"ping\00" [Dec 15 10:23:54.069] [ debug] [vmsvc] RpcIn: sending 3 bytes [Dec 15 10:23:55.017] [ debug] [vmsvc] RpcChannel: Sending: 35 bytes [Dec 15 10:23:55.017] [ debug] [vmsvc] RpcChannel: Recved 0 bytes [Dec 15 10:23:55.521] [ info] [guestinfo] New value for stats-interval is 20s. [Dec 15 10:23:55.521] [ info] [guestinfo] New value for poll-interval is 30s. Version-Release number of selected component (if applicable): open-vm-tools-10.0.5-2.el7.x86_64 RHEL 7.3 (3.10.0-514.el7.x86_64) How reproducible: Additional info: This only occurs when there are docker images running.