Per customer:

"Yes, 100% reproducible.

With running containers, the VM snapshot fails.

Stop the containers, the VM snapshot succeeds.

open-vm-tools itself isn't running in a container?  This is open-vm-tools
running on the host, and containers happen to be running also on the
same host?

(In reply to Stephen Beal from comment #0)
> [Dec 15 10:23:53.645] [   debug] [vmsvc] SyncDriver: opening path 'net'.
> [Dec 15 10:23:53.645] [   debug] [vmsvc] SyncDriver: failed to open 'net': 2
> (No such file or directory)

What is this mount point 'net' for? This typically happens for stale mount points. If there is a valid mount point, we should be able to open() it.

It would be very helpful to get output of 'mount' command from the problematic VM.

Could we get the output of the 'mount' command running in the host
when a container is running?

"[root@ltstpnasv22 ~]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=931040k,nr_inodes=232760,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/vg_os-lv_root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg_os-lv_usr on /usr type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
/dev/sda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg_os-lv_opt on /opt type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg_os-lv_opt_openv on /opt/openv type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg_os-lv_home on /home type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg_os-lv_tmp on /tmp type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg_app0-lv_opt_splunk on /opt/splunk type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg_os-lv_var on /var type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/mapper/vg_os-lv_var_opt on /var/opt type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg_os-lv_var_log on /var/log type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg_os-lv_var_log_audit on /var/log/audit type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
linuxhome.staples.com:/nethome on /nethome type nfs (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,soft,proto=tcp,timeo=1000,retrans=2,sec=sys,mountaddr=10.4.15.46,mountvers=3,mountport=1234,mountproto=tcp,local_lock=none,addr=10.4.15.46)
unixhome.staples.com:/unixhome on /home/net1 type nfs (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,soft,proto=tcp,timeo=1000,retrans=2,sec=sys,mountaddr=10.4.15.48,mountvers=3,mountport=1234,mountproto=tcp,local_lock=none,addr=10.4.15.48)
/dev/mapper/vg_os-lv_var on /var/lib/docker/100000.100000/devicemapper type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=188416k,mode=700)
/dev/mapper/docker-253:10-4316585-b48569d5609021d414a3aa91308b683f4c7eb14725bd9c9abe1f83ffa20fdd62 on /var/lib/docker/100000.100000/devicemapper/mnt/b48569d5609021d414a3aa91308b683f4c7eb14725bd9c9abe1f83ffa20fdd62 type xfs (rw,relatime,seclabel,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota)
shm on /var/lib/docker/100000.100000/containers/bbf85ac246c75c175eb71772d9a9b06fbd88a46b8260d77dfd4eba6cc1f14353/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,size=65536k)
proc on /run/docker/netns/68b23eba68a3 type proc (rw,nosuid,nodev,noexec,relatime)
/dev/mapper/docker-253:10-4316585-bc984e26f0ab96e88f2b923a727235fbbce6106f5bf68e34e8066fa4cf40004c on /var/lib/docker/100000.100000/devicemapper/mnt/bc984e26f0ab96e88f2b923a727235fbbce6106f5bf68e34e8066fa4cf40004c type xfs (rw,relatime,seclabel,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota)
shm on /var/lib/docker/100000.100000/containers/e3b0fb32747919010bc825356043bf5eeb01b1412b49302e3beb64c2ceec824d/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,size=65536k)
proc on /run/docker/netns/649937cc3362 type proc (rw,nosuid,nodev,noexec,relatime)
[root@ltstpnasv22 ~]#

I believe this is all fixed in open-vm-tools 10.1+ 

https://github.com/vmware/open-vm-tools/blob/master/ReleaseNotes.md

I believe this should also alleviate issues with the VMHGFS kernel module for 7.4.

(In reply to BDwyerTech from comment #10)
> I believe this is all fixed in open-vm-tools 10.1+ 
> 
> https://github.com/vmware/open-vm-tools/blob/master/ReleaseNotes.md
> 
> I believe this should also alleviate issues with the VMHGFS kernel module
> for 7.4.

Failure to open a mount path is not fixed in 10.1 yet.

(In reply to Ravindra Kumar from comment #7)
> It would be very helpful to get output of 'mount' command from the
> problematic VM.

Please see it in comment#9

(In reply to Stephen Beal from comment #9)
> "[root@ltstpnasv22 ~]# mount
> sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
> proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
> ...
> cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup
> (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
> ...
> linuxhome.staples.com:/nethome on /nethome type nfs
> (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,soft,proto=tcp,
> timeo=1000,retrans=2,sec=sys,mountaddr=10.4.15.46,mountvers=3,mountport=1234,
> mountproto=tcp,local_lock=none,addr=10.4.15.46)
> unixhome.staples.com:/unixhome on /home/net1 type nfs
> (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,soft,proto=tcp,
> timeo=1000,retrans=2,sec=sys,mountaddr=10.4.15.48,mountvers=3,mountport=1234,
> mountproto=tcp,local_lock=none,addr=10.4.15.48)
> ...
> proc on /run/docker/netns/68b23eba68a3 type proc
> (rw,nosuid,nodev,noexec,relatime)
> /dev/mapper/docker-253:10-4316585-
> ...
> proc on /run/docker/netns/649937cc3362 type proc
> (rw,nosuid,nodev,noexec,relatime)
> [root@ltstpnasv22 ~]#

I don't see any 'net' in the list. Also, this is a bit weird path as I'd expect all paths to be absolute.

I don't see an answer to Richard's questions in comment #3. Could we please get answers to those too? If open-vm-tools is running inside a container, that is not a supported configuration yet.

Hi Ravindra,

I apologize on behalf of RH for the update not being visible to answer your question in #3 no it is not being ran in a container. This was confirmed by the customer directly. I have spoken with my docker specialist and he is assuming that this may be because of the mount namespaces docker creates for containers - specifically that OVT does not know how to handle them. This is purely conjectural as he has not looked at the OVT code and is going entirely off of what the customer has said with regards to reproducing the error.

Piling on here since I just discovered this bug.. Having same issue. 

RHEL 7.2 Guest on VMware. RHEL Guest running docker containers.
 
[root@vcld16rdabpws10 ~]# docker version
Client:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:24:06 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:24:06 2016
 OS/Arch:      linux/amd64
[root@vcld16rdabpws10 ~]#

[root@vcld16rdabpws10 ~]# vmware-toolbox-cmd -v
10.0.9.55972 (build-3917699)

VMware ESX hosts are 6.0.0 

Output from Mount command below:

[root@vcld16rdabpws10 ~]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=2959256k,nr_inodes=739814,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/vgOS-root on / type xfs (rw,relatime,attr2,inode64,noquota)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/mapper/vgOS-tmp on /tmp type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sda1 on /boot type xfs (rw,relatime,attr2,inode64,noquota)
/dev/mapper/vgAPPS-apps on /apps type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
/dev/mapper/vgOS-home on /home type xfs (rw,relatime,attr2,inode64,noquota)
/dev/mapper/vgOS-opt on /opt type xfs (rw,relatime,attr2,inode64,noquota)
/dev/mapper/vgOS-var on /var type xfs (rw,relatime,attr2,inode64,noquota)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/mapper/vgOS-var on /var/lib/docker/devicemapper type xfs (rw,relatime,attr2,inode64,noquota)
/dev/mapper/docker-253:4-16782674-9bf253461a5d2430b83b097b0b14351ceb57c344bf9621f4a0dbaf2c3914f8de on /var/lib/docker/devicemapper/mnt/9bf253461a5d2430b83b097b0b14351ceb57c344bf9621f4a0dbaf2c3914f8de type xfs (rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota)
shm on /var/lib/docker/containers/7254067d95b8ec0dc51ae567a76a33054104edc581dc00225f0371eecca32963/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
proc on /run/docker/netns/fcbcbab093a6 type proc (rw,nosuid,nodev,noexec,relatime)
/dev/mapper/docker-253:4-16782674-f1a721e6f4edbfec8ecf1b922cdf5baca54c24c8bb751eac98eaa9bec41691f7 on /var/lib/docker/devicemapper/mnt/f1a721e6f4edbfec8ecf1b922cdf5baca54c24c8bb751eac98eaa9bec41691f7 type xfs (rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota)
shm on /var/lib/docker/containers/0f5279893df63a1ae995ce25edadfd2c021df7b66d439d042911696e55663d75/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
proc on /run/docker/netns/014bc315e959 type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/3743680 type tmpfs (rw,nosuid,nodev,relatime,size=593884k,mode=700,uid=3743680,gid=20)

Updating what we have done for this issue over last 2 weeks. We could reproduce it by running a container on docker 1.12. We used 'docker run -d -P training/webapp python app.py' which created a mount point '/run/docker/netns/f919fe29adb6' that was not a directory. This mount point shows up as net:[LARGE_NUMBER] in /etc/mtab and /proc/mounts:

# grep net /etc/mtab
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
proc net:[4026532713] proc rw,nosuid,nodev,noexec,relatime 0 0
# grep net /proc/mounts
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
proc net:[4026532713] proc rw,nosuid,nodev,noexec,relatime 0 0
# grep net /proc/self/mounts
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
proc net:[4026532713] proc rw,nosuid,nodev,noexec,relatime 0 0

'mount' command that looks at /proc/self/mountinfo shows the same mount point differently.

# mount | grep net
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
proc on /run/docker/netns/f919fe29adb6 type proc (rw,nosuid,nodev,noexec,relatime)
# grep net /proc/self/mountinfo
30 24 0:25 / /sys/fs/cgroup/net_cls rw,nosuid,nodev,noexec,relatime shared:13 - cgroup cgroup rw,net_cls
265 23 0:3 / /run/docker/netns/f919fe29adb6 rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw

# fsfreeze -f /run/docker/netns/f919fe29adb6
fsfreeze: /run/docker/netns/f919fe29adb6: is not a directory

With modified fsfreeze to allow non-directories:

# ./fsfreeze -f /run/docker/netns/f919fe29adb6
fsfreeze: /run/docker/netns/f919fe29adb6: freeze failed: Operation not supported

# stat /run/docker/netns/f919fe29adb6
  File: â/run/docker/netns/f919fe29adb6â
  Size: 0               Blocks: 0          IO Block: 1024   regular empty file
Device: 3h/3d   Inode: 4026532713  Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:proc_t:s0
Access: 2017-01-26 19:28:10.703557060 -0500
Modify: 2017-01-26 19:28:10.703557060 -0500
Change: 2017-01-26 19:28:10.703557060 -0500
 Birth: -
# file /run/docker/netns/f919fe29adb6
/run/docker/netns/f919fe29adb6: empty
# ls -l /run/docker/netns/f919fe29adb6
-r--r--r--. 1 root root 0 Jan 26 19:28 /run/docker/netns/f919fe29adb6

We tried different types of containers even for a simple one like hello-world container, docker creates such special mount point.

Running a test program that uses getmntent() C-APIs just like vmtoolsd:

# ~/mountlist | grep net
fsname=cgroup, dir=/sys/fs/cgroup/net_cls, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,net_cls, freq=0, passno=0
fsname=proc, dir=net:[4026532713], type=proc, opts=rw,nosuid,nodev,noexec,relatime, freq=0, passno=0  <<<<=== This is how vmtoolsd sees it.

# cat /etc/redhat-release
Red Hat Enterprise Linux Client release 7.2 (Maipo)
# uname -a
Linux promb-2s-dhcp95-204.eng.vmware.com 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
# docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:20:01 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:20:01 2017
 OS/Arch:      linux/amd64
[root@promb-2s-dhcp95-204 util-linux]# vmtoolsd -v
VMware Tools daemon, version 10.1.0.57774 (build-4449150)

We are going to put a workaround in vmtoolsd to tolerate such mount points, but this needs to be looked into from Linux perspective as well. Basically, why mtab/mounts entry is not matching with /proc/self/mountinfo?

Richard, could we please include folks who work on 'mount' and /etc/mtab entries to answer the inconsistency between mtab and /proc/self/mountinfo?

Take a look at my comment in the other bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1166465#c60

Created attachment 1246011 [details]
10.0.5 based patch for special mount points

Rich, attached is the patch we have implemented for working around the docker mount point issue. This is built for 10.0.5 code and can be applied on top of the last patch we provided for bug 1398802.

Package containing the patch:

http://oirase.annexia.org/tmp/bz1406483

Thanks for that Richard! It's going to help speed along my testing!

Verify this bug with open-vm-tools-10.1.5-3.el7.x86_64, this issue is fixed.

Reproduce/Verify steps:
1. Install docker
2. Start a network container, e.g.
$ docker run -d -P --name web training/webapp python app.py
3. Take snapshot, choose 'Quiesce guest file system'

Test results:
At step 3, open-vm-tools-10.0.5-2.el7 failed, open-vm-tools-10.1.5-3.el7.x86_64 can complete successfully

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1906