Bug 1901882

Hi Ravindra,

Could you please take a look at this. Is this the same issue as BZ 1898069 and BZ 1880404?

Thanks,

Cathy

Cathy, I don't seem to have access to the comments in this bug. I can see only your comment #3.

(In reply to Ravindra Kumar from comment #4)
> Cathy, I don't seem to have access to the comments in this bug. I can see
> only your comment #3.

I added vmware group, you should have access now.

Hi Ravindra,
Any update for this issue? does this bug same issue as  BZ 1880404?

Thanks,
ldu

I suspect that the customer script is doing all the freezing in this case because vmtoolsd runs "/usr/sbin/pre-freeze-script" before quiescing any file systems. Can we get a copy of the customer bash script? Also, we need to know why are they invoking "freeze" commands from the "pre-freeze-script" as that can interfere with the freeze steps performed by vmtoolsd?

I assume this is reproducible, so can we also redirect vmtoolsd logs to vmware.log and collect that from the VM's directory? For redirecting vmtoolsd logs to vmware.log in VM's directory following settings need to be added to /etc/vmware-tools/tools.conf inside the guest:

# Start
[logging]
vmsvc.level=debug
vmsvc.handler=vmx
# End

(In reply to Grzegorz Halat from comment #11)
> 
> $ cat /usr/sbin/pre-freeze-script
> /opt/SYMCquiesce/bin/freeze
> 
> /opt/SYMCquiesce is probably some backup software from Veritas.
> 
> > I assume this is reproducible, so can we also redirect vmtoolsd logs to vmware.log and collect that from the VM's directory?
> 
> Do you still need this? I did not ask them yet to reproduce with debug logs.

Thanks Grzegorz, I do not believe the debug logs are needed here.  In this particular instance, they would not have provided any useful information.  An explanation follows.

Reviewing the entire bugzilla comments, we see in comment 2 that the vmtoolsd process is stuck.

> Now, let's check vmtoolsd processes running in the system:
>
> crash> ps | grep vmtools
>    1042      1  12  ffff9a4f8f930000  UN   0.0  351704   5884  vmtoolsd
>   10300   1042   9  ffff9a38fabf9080  IN   0.0  351704   2304  vmtoolsd

> Userspace stack traces of vmtoolsd processes:
> core files directory(galvatron): /cores/retrace/tasks/649245749/results
> files: core.10300.vmtoolsd, core.1042.vmtoolsd
>
> PID 1042 - it was stuck in writing a message to log file and it hanged here, but this probably doesn't matter:
> (gdb) bt
> #0  0x00007fda97a4f6fd in write () at ../sysdeps/unix/syscall-template.S:81
> #1  0x00007fda9716529c in g_io_unix_write (channel=0x558b2d6f1e00, buf=0x558b2d776480 "[2020-11-20T01:19:12.295Z] [ warning] [guestinfo] GuestInfoGetDiskDevice: Missing disk device name; VMDK mapping unavailable for \"/var/lib/kubelet/plugins/ceph.rook.io/rook-ceph-system/mounts/pvc-3514"..., count=252, bytes_written=<optimized out>, err=0x0)
>     at giounix.c:251
>
>    ...
>
> PID 10300 - it executed a shell script and was waiting for comletion of a process with PID 10301:
> (gdb) bt
> #0  0x00007fda97a501d9 in __libc_waitpid (pid=pid@entry=10301, stat_loc=stat_loc@entry=0x7ffe9441b654, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:40
> #1  0x00007fda982e7e4b in ProcMgrWaitForProcCompletion (pid=10301, validExitCode=0x0, validExitCode@entry=0x7ffe9441b697 "", exitCode=exitCode@entry=0x7ffe9441b69c) at procMgrPosix.c:1592
> #2  0x00007fda982e931b in ProcMgr_ExecAsync (cmd=cmd@entry=0x558b2d7110b0 "\"/usr/sbin/pre-freeze-script\" freeze", userArgs=userArgs@entry=0x0) at procMgrPosix.c:1737
> #3  0x00007fda8c76543f in VmBackupRunNextScript (op=op@entry=0x558b2d7124b0) at scriptOps.c:170
> #4  0x00007fda8c76583a in VmBackup_NewScriptOp (type=type@entry=VMBACKUP_SCRIPT_FREEZE, state=0x558b2d714c00) at scriptOps.c:546
> #5  0x00007fda8c7663e0 in VmBackupStartScripts (type=VMBACKUP_SCRIPT_FREEZE) at stateMachine.c:409

The vmtoolsd process [PID 1042] is attempting to write a warning message to the vmsvc log file while collecting the disk device mapping information for the RHEL Linux VM.  Just prior to that, vmtoolsd has kicked off the "/usr/sbin/pre-freeze-script" [PID 10300].

And we see from comment 11 that the pre-freeze-script is executing the Veritas NetBackup SYMCquiesce freeze command.

I believe that what is happening, and I have discussed this with Ravindra, is that the /var/log filesystem has been frozen by the SYMCquiesce/bin/freeze just before the attempt to write the warning to "/var/log/vmware-vmsvc.log".  Ergo the hang.

Had  the "tools.conf" been updated with
    "vmsvc.handler=vmx"
to send the vmsvc log messages the the VMX log (vmware.log) on the ESXi host, the hang would not have been reproduced.

The warning message that is being logged

[ warning] [guestinfo] GuestInfoGetDiskDevice: Missing disk device name; VMDK mapping unavailable for \"/var/lib/kubelet/plugins/ceph.rook.io/rook-ceph-system/mounts/pvc-3514"

is an example of the disk device mapping "log spew" reported against open-vm-tools 11.0.x.  This problem is being addressed in open-vm-tools 11.0.5 for RHEL 7.9 in https://bugzilla.redhat.com/show_bug.cgi?id=1911853.

Cathy, what is the status of bug 1911853?

I believe fixing the vmsvc log spew in 11.0.5 open-vm-tools for RHEL 7.0 will alleviate this customer's problem.

(In reply to John Wolfe from comment #13)
> The warning message that is being logged
> 
> [ warning] [guestinfo] GuestInfoGetDiskDevice: Missing disk device name;
> VMDK mapping unavailable for
> \"/var/lib/kubelet/plugins/ceph.rook.io/rook-ceph-system/mounts/pvc-3514"
> 
> is an example of the disk device mapping "log spew" reported against
> open-vm-tools 11.0.x.  This problem is being addressed in open-vm-tools
> 11.0.5 for RHEL 7.9 in https://bugzilla.redhat.com/show_bug.cgi?id=1911853.
> 
> Cathy, what is the status of bug 1911853?
> 
> I believe fixing the vmsvc log spew in 11.0.5 open-vm-tools for RHEL 7.0
> will alleviate this customer's problem.

It's done and waiting for approval for 7.9.z which it should get.

(In reply to John Wolfe from comment #12)

> Had  the "tools.conf" been updated with
>     "vmsvc.handler=vmx"
> to send the vmsvc log messages the the VMX log (vmware.log) on the ESXi
> host, the hang would not have been reproduced.

OK, I will ask the customer to update the config and check if it solves the issue.

(In reply to Grzegorz Halat from comment #15)
> (In reply to John Wolfe from comment #12)
> 
> > Had  the "tools.conf" been updated with
> >     "vmsvc.handler=vmx"
> > to send the vmsvc log messages the the VMX log (vmware.log) on the ESXi
> > host, the hang would not have been reproduced.
> 
> OK, I will ask the customer to update the config and check if it solves the
> issue.

That is the work-around until the updated 11.0.5 open-vm-tools is available.

One thing I would like to confirm is verifying why customer has such configuration with NetBackup that is leading to this failure. As per #c11:

$ cat /usr/sbin/pre-freeze-script
/opt/SYMCquiesce/bin/freeze

Can we ask customer or Symantec to find out who creates /usr/sbin/pre-freeze-script in the guest? Is it customer or NetBackup software? And, what does the invoked command /opt/SYMCquiesce/bin/freeze do? Does it freeze all file systems as its name suggests? If yes, then it is a known issue and workaround provided by John will help. We also need to understand from customer and/or Symantec the reason to have /usr/sbin/pre-freeze-script perform freeze operation as vmtoolsd is supposed to do that anyway. We don't expect pre-freeze-script to freeze file systems, generally it is not needed. It is also worth getting a copy of /etc/vmware-tools/tools.conf from the failing VM, so that we can see if customer or NetBackup software has also disabled vmtoolsd syncDriver.

While searching for 'SYMCquiesce' on Internet, I came across this document for NetBackup 8.1 - https://www.veritas.com/support/en_US/doc/103228346-127350715-0/v124981393-127350715, which says:
=============================
Support has ended for the SYMCquiesce utility for Linux virtual machines
Support for the SYMCquiesce utility for Linux virtual machines has been discontinued starting in this NetBackup release. Newer operating systems provide native support for a similar functionality. Please contact your operating system vendor and VMware for additional information.
=============================

I would strongly suggest the customer to check with Symantec if SYMCquiesce needs to be even there on the system. If there is no reason for it to be there, they should uninstall it and make sure that "enableSyncDriver=false" is NOT set in /etc/vmware-tools/tools.conf.

Another Veritas link on this topic for reference - https://vox.veritas.com/t5/NetBackup/SYMCquiesce/td-p/844376.

Closing this BZ as a duplicate of bug 1911853 (which will avoid the issue described in this BZ).

*** This bug has been marked as a duplicate of bug 1911853 ***