Description of problem: vmtoolsd hook 1 core of cpu 100% systemctl status vmtoolsd shows passive (failed at boot) and disable vmtoolsd does not change anything CPU hooking happends after login So it must be the version af vmtoolsd which is started at login Is HW depend...a F22 with Virtualbox 5 installed on a USB HD with F22(64bit) as guest in Virtualbox. Booted on AMD Phenom II X4 gives the problem Booted on a Intel I7 gives not the problem On a F19 host (AMD phenom HW) Vbox Guest can be either F22 or F20, gives also the error Happends only if Virtualbox guest settings has paravirtualization enabled with KVM, not with legacy systemd-detect-virt says KVM Version-Release number of selected component (if applicable): Open-vm-tools 9.10.2-1.F22 How reproducible: See above Actual results: 1 cpu core is 100% load Expected results: No significant cpu load Additional info: Remove open-vm-tools packaged removes the problem Knud
Im seeing this also
The mistake is properly that open-vm-tools is getting installed at all. Kbuntu 15.10 64 bit does not install open-vm-tools at all as a guest in virtualbox. But if you instal open-vm-tools in Kbuntu it behaves as expected systemctl status open-vm-tools says: ....condition failed.... ...=vmware was not met open-vm-tools is not started In Fedora there must be a bug... Knud
(In reply to Knudch from comment #2) > The mistake is properly that open-vm-tools is getting installed at all. > Kbuntu 15.10 64 bit does not install open-vm-tools at all as a guest in > virtualbox. > But if you instal open-vm-tools in Kbuntu it behaves as expected > systemctl status open-vm-tools says: > ....condition failed.... > ...=vmware was not met > > open-vm-tools is not started > > In Fedora there must be a bug... > > Knud F23 same problem
I will just tell you - to set expectations - that approximately no one cares about this semi-proprietary hypervisor & Fedora. I think you should use KVM. Despite that, what does 'systemd-detect-virt' say inside this guest?
When paravirtualization is enabled with KVM systemd-detect-virt says KVM and the problem is present. When paravirtualization is enabled with legacy then systemd-detect-virt says legacy and the problem does not exist. Im ok using legacy, just me-tooing a bug :) Thanks
systemd-detect-virt says: oracle That's to bad if nobody cares about one of the best supported and from usability best virtualization environment. And other distro's do without this "bug"
F22 Systemd-detect-virt says: KVM F23 says: oracle Actually I don't think it matters what it reports because just the vmware-check prg hooks/hangs at 100% 1 core load, at install of the package open-vm-tools
(In reply to Knudch from comment #2) > The mistake is properly that open-vm-tools is getting installed at all. > Kbuntu 15.10 64 bit does not install open-vm-tools at all as a guest in > virtualbox. Yes, we need to avoid open-vm-tools installation in Fedora on non-VMware VMs. I will see if we can make that change in Fedora 24. (In reply to Knudch from comment #7) > F22 Systemd-detect-virt says: KVM > F23 says: oracle Not sure why is that, Richard would know better. > Actually I don't think it matters what it reports because just the > vmware-check prg hooks/hangs at 100% 1 core load, at install of the package > open-vm-tools I agree. vmtoolsd uses same code as vmware-checkvm to detect VMware platform. If that code hangs/fails, vmtoolsd running as the logged in user would be hit by the same behavior. vmtoolsd running as root user is guarded by systemd virtualization check which uses systemd-detect-virt (AFAIK) and therefore it would not run unless systemd-detect-virt reports VMware, only vmtoolsd running as the logged in user would be impacted.
Is the answer here for vmtoold running as the logged in user to run 'systemd-detect-virt' first? If there is a script that runs vmtoolsd, that should be easy enough: if [ "$(systemd-detect-virt)" = "VMware" ]; then vmtoolsd & fi
It happends in several situations 1. At the installation time of open-vm-tools fx.during a net-install, from that momment that vmware-check is started the installations slows down, checking with TOP shows 100% cpu on 1 core from vmware-check 2. In a installed system it happend after user login, vmtoolsd suck 100% 1 core
I dont know if it is possibel.... But make a check with systemd-detect-virt before starting install of open-vm-tools or making the package optional Ideal....fix the bug
(In reply to Richard W.M. Jones from comment #9) > Is the answer here for vmtoold running as the logged in user > to run 'systemd-detect-virt' first? If there is a script that > runs vmtoolsd, that should be easy enough: > > if [ "$(systemd-detect-virt)" = "VMware" ]; then > vmtoolsd & > fi It's called from /etc/xdg/autostart/vmware-user.desktop which will require another level of redirection/wrapper to have additional call to systemd-detect-virt. (In reply to Knudch from comment #10) > It happends in several situations > > 1. At the installation time of open-vm-tools fx.during a net-install, from > that momment that vmware-check is started the installations slows down, > checking with TOP shows 100% cpu on 1 core from vmware-check We can work around this by guarding the call behind a systemd-detect-virt check. > 2. In a installed system it happend after user login, vmtoolsd suck 100% 1 > core vmtoolsd in this case is launched by /etc/xdg/autostart/vmware-user.desktop. We will need to write another wrapper to call into systemd-detect-virt. (In reply to Knudch from comment #11) > I dont know if it is possibel.... > > But make a check with systemd-detect-virt before starting install of > open-vm-tools > or making the package optional > > Ideal....fix the bug I think simplest is to uninstall the package because this is not intended for non-VMware cases anyway. Like I said ealier, I will see if I can push the change to not install the package on non-VMware.
As Knudch stated, when doing a net-install of Fedora 23 in Virtualbox, the installer hangs (indefinitely) when trying to install open-vm-tools, preventing the installation from completing. I am having this problem using the net-install image downloaded yesterday. Is there a workaround to avoid installing open-vm-tools until this is fixed?
Same problem in F24 prevent install in Virtualbox
Live-Workstation (23 and 24 beta 1.6) completely unusable because of this in a VirtualBox
--- WORKING HACKY FIX --- I was able to `uninstall open-vm-tools` by using the following hack. Whenever I tried to uninstall open-vm-tools, I could notice a new process `vmware-checkvm` that never used to complete execution, as a result uninstall of open-vm-tools used to fail. So here's what I did and it worked for me - I replaced '/usr/bin/vmware-checkvm' with an empty bash script! ---- START: hack ---- sudo su mv /usr/bin/vmware-checkvm /usr/bin/vmware-checkvm.CRAP echo '#!/bin/bash' > /usr/bin/vmware-checkvm chmod +x /usr/bin/vmware-checkvm rpm -e open-vm-tools ---- END: hack ---- After this, the package `open-vm-tools` was successfully removed from my VM. I tried it on Fedora 23.
Thank you Amit, much appreciated - your hack worked. BOO, RedHat!
Actually this should not run at all, as there is a conditional check to run only on vmware [1], and the condition is validated by "virt-what", so vmtools should never run. This is what is intended and what happens on my kvm/xen hosts. Can you run virt-what in the VirtualBox guest and tell me the output? [1] http://pkgs.fedoraproject.org/cgit/rpms/open-vm-tools.git/tree/vmtoolsd.service#n4
(In reply to Simone Caronni from comment #18) > Actually this should not run at all, as there is a conditional check to run > only on vmware [1], and the condition is validated by "virt-what", so > vmtools should never run. This is what is intended and what happens on my > kvm/xen hosts. > > Can you run virt-what in the VirtualBox guest and tell me the output? > > [1] > http://pkgs.fedoraproject.org/cgit/rpms/open-vm-tools.git/tree/vmtoolsd. > service#n4 In F23 and F25 VirtualBox kvm
You're right that it shouldn't be running at all, but actually the test is done by 'systemd-detect-virt'. It would be useful to see the output of that program instead of 'virt-what'.
F23 x64 systemd-detect-virt => oracle in a Vbox
open-vm-tools-10.0.5-3.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2016-f2b2eb3aca
open-vm-tools-10.0.5-3.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-3037b0acb9
open-vm-tools-10.0.5-3.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-ed90ccf507
I have added a workaround in the installer, so that installer is not stuck during install/uninstall. vmtoolsd using 100% CPU still needs to be fixed upstream. Could somebody please provide a coredump of vmtoolsd process when it is using 100% CPU?
I have tried the update open-vm-tools-10.0.5-3.fc23 (64bit) in a FC23 KDE live Vbox session. steps: verify that vmtoolsd hooks cpu on boot=> login => yes killed and removed open-vm-tools-10.0.0-7 installed open-vm-tools-10.0.5-3 => no CPU hooked enabled and started vmtoolsd via systemctl => no CPU hooked systemctl status vmtoolsd says =>start condition faild...= vmware was not met So far a succes Just remember that the bug behavior are related to AMD CPU....it happends on no Intel HW I have
(In reply to Knudch from comment #26) > I have tried the update open-vm-tools-10.0.5-3.fc23 (64bit) in a FC23 KDE > live Vbox session. > > steps: > verify that vmtoolsd hooks cpu on boot=> login => yes > > killed and removed open-vm-tools-10.0.0-7 > > installed open-vm-tools-10.0.5-3 => no CPU hooked > > enabled and started vmtoolsd via systemctl => no CPU hooked > > systemctl status vmtoolsd says =>start condition faild...= vmware was not met > > So far a succes > > Just remember that the bug behavior are related to AMD CPU....it happends on > no Intel HW I have It is important to log out and in after the update to see if it works ! There will also be no CPU hook with the original open-vm-tools packgde after vmtoolsd are stop and start also not if packgde is removed and added again, first after LOG out/in CPU is hooked But with the updated packgde it is ok...no CPU hooked also after LOG out => in
(In reply to Knudch from comment #27) > But with the updated packgde it is ok...no CPU hooked also after LOG out => > in Thanks Knudch. This is interesting. It looks like you may not need this fix at all in that case. Could you please verify the same test with 10.0.5-2 package? There are two aspects of this issue 'vmware-checkvm' called by RPM and 'vmtoolsd' started after login. This fix applies to only RPM, there is no change in 'vmtoolsd' from 10.0.5-2. 'vmtoolsd' issue needs to be fixed in upstream first. So, expectation is that RPM should not get stuck in vmware-checkvm, but vmtoolsd may still take 100% CPU. There are two instances of vmtoolsd usually, one running as system (what you have tried) and one running as logged in user. To get vmtoolsd started as logged in user, you need to "reboot and login" or "logout and login" after installing the new package. We will consider the new package good as long as RPM issue is solved.
open-vm-tools-10.0.5-3.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-f2b2eb3aca
open-vm-tools-10.0.5-3.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-ed90ccf507
open-vm-tools-10.0.5-3.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-3037b0acb9
(In reply to Ravindra Kumar from comment #28) > (In reply to Knudch from comment #27) > > But with the updated packgde it is ok...no CPU hooked also after LOG out => > > in > > Thanks Knudch. This is interesting. It looks like you may not need this fix > at all in that case. Could you please verify the same test with 10.0.5-2 > package? > > There are two aspects of this issue 'vmware-checkvm' called by RPM and > 'vmtoolsd' started after login. This fix applies to only RPM, there is no > change in 'vmtoolsd' from 10.0.5-2. 'vmtoolsd' issue needs to be fixed in > upstream first. So, expectation is that RPM should not get stuck in > vmware-checkvm, but vmtoolsd may still take 100% CPU. There are two > instances of vmtoolsd usually, one running as system (what you have tried) > and one running as logged in user. To get vmtoolsd started as logged in > user, you need to "reboot and login" or "logout and login" after installing > the new package. We will consider the new package good as long as RPM issue > is solved. I am aware that there are 2 situations, one during a install and later in the actual installation(after login). But I guess they are caused by the same problem in the open-vm-tools package Basic problem is that the code in vmware-checkvm is faulty in AMD environment It leads to a CPU hook during install because the check if open-vm-tools shall be installed is done by vmware-checkvm If the install is finished the it generates a CPU hook problem a soon as vmtoolsd is tried started using same code/vmware-checkvm I will check the 10.0.5-2 package
It seems that open-vm-tools are getting installed even in a Intel based HW Virtualbox. Maybe 2 different bugs ? 1. The check to install open-vm-tools is general failing in a Virtualbox environment. 2. In a AMD based HW it has a 2nd consequence => hooking up 1 core 1. Is checked with F23 KDE installed from KDE live but I think it is the same with most releases
(In reply to Knudch from comment #33) > It seems that open-vm-tools are getting installed even in a Intel based HW > Virtualbox. > > Maybe 2 different bugs ? Just as a reference, hese are all the tools that run by default on RHEL7Fedora systems, conditionally checking the virtualization type in systemd. Being conditonally run only if the virtualization layer is present, they are installed by default if you don't select "Minimal Installation", with a "Minimal Installation" they are not installed: # cat /usr/lib/systemd/system-preset/90-default.preset | egrep -i "kvm|spice|hyper|xen|vmt|qemu" enable spice-vdagentd.service enable qemu-guest-agent.service enable vmtoolsd.service enable xendomains.service enable xenstored.service enable xenconsoled.service enable hypervkvpd.service enable hypervvssd.service They are installed to provide the best "out of the box experience" when installing the system as a guest. They can of course be removed after the system is installed. Also the base drivers for those platforms are in the kernel-core package, so you don't need the full set of modules.
(In reply to Simone Caronni from comment #34) > (In reply to Knudch from comment #33) > > It seems that open-vm-tools are getting installed even in a Intel based HW > > Virtualbox. > > > > Maybe 2 different bugs ? > > Just as a reference, hese are all the tools that run by default on > RHEL7Fedora systems, conditionally checking the virtualization type in > systemd. > > Being conditonally run only if the virtualization layer is present, they are > installed by default if you don't select "Minimal Installation", with a > "Minimal Installation" they are not installed: > > # cat /usr/lib/systemd/system-preset/90-default.preset | egrep -i > "kvm|spice|hyper|xen|vmt|qemu" > enable spice-vdagentd.service > enable qemu-guest-agent.service > enable vmtoolsd.service > enable xendomains.service > enable xenstored.service > enable xenconsoled.service > enable hypervkvpd.service > enable hypervvssd.service > > They are installed to provide the best "out of the box experience" when > installing the system as a guest. They can of course be removed after the > system is installed. > > Also the base drivers for those platforms are in the kernel-core package, so > you don't need the full set of modules. Ok... Then my #1 case is not a bug We are back to that either vmware-checkvm or/and vmtoolsd are buggy on AMD based HW Virtualbox
open-vm-tools-10.0.5-3 seems to be a safe solution 1. dnf install 10.0.5-2 => during install htop show vmware-checkvm hook's 1 core 100%, executing for a long time(or infinite) 2. dnf install 10.0.5-3 => during install htop show briefly systemd-detect-virt and no cpu load, finish fast. vmware-checkvm from console hook's one core 100% I think it works as expected
(In reply to Knudch from comment #36) > 2. dnf install 10.0.5-3 => during install htop show briefly > systemd-detect-virt and no cpu load, finish fast. > vmware-checkvm from console hook's one core 100% > > I think it works as expected Thanks Knudch for testing this. Could you please help with following? 1. Does 'vmtoolsd -n vmusr' still consume 100% CPU after use logs in? 2. Coredump of vmware-checkvm and/or 'vmtoolsd -n vmusr' when it is taking 100% CPU. Also, please share the details of your environment that we can use to reproduce this issue ourselves? I believe we should be able to repro it by running Fedora in a VirtualBox VM on Intel HW. Please confirm.
My environment: AMD Phennom II quad core, 32G ram Vbox 5.x with paravirt set to KVM, 2 core to the guest Host OS fedora19, has been checked with F22 and F23 To reproduce problem: In a console start either vmtoolsd or vmware-checkvm result => vmtoolsd/vmware-checkvm hook's 1 core start vmtoolsd -n vmusr does not change anything On 3 other Intel HW problems does not happend(both vmtoolsd and vmware-checkvm) On 1 other AMD HW the problem has also been seen Coredump: Have never tried it before....advise ?
(In reply to Knudch from comment #38) Thanks Knudch. > start vmtoolsd -n vmusr does not change anything 'vmtoolsd -n vmusr' starts automatically when you log in to the desktop console. > On 3 other Intel HW problems does not happend(both vmtoolsd and > vmware-checkvm) > > On 1 other AMD HW the problem has also been seen That sounds like AMD specific issue? > Coredump: > Have never tried it before....advise ? Run 'vmware-checkvm' by hand and check its pid when it is stuck. Run 'gcore' from another terminal. See 'man gcore' or use 'gcore -o <coredump_filename> <pid>' as an example. 'gcore' comes with 'gdb' package.
(In reply to Ravindra Kumar from comment #39) > > start vmtoolsd -n vmusr does not change anything > > 'vmtoolsd -n vmusr' starts automatically when you log in to the desktop > console. I mean just run 'top' after you login and see how much CPU 'vmtoolsd' process is burning.
open-vm-tools-10.0.5-3.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.
open-vm-tools-10.0.5-3.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
gcore -o vmware-checkvm 1594 => vmware-checkvm.gdb.1594 vmware-checkvm started from console For completness of info...the session with vmware-checkvm can be stopped with CTRL-C Attached the coredump file
Created attachment 1171550 [details] coredump gcore -o vmware-checkvm 1594
tested on WIN 10 system with AMD FX 6350 6 core + 16G ram Same result vmware-checkvm hook 1 core 100%
!!!! Virtualbox 5.1.0 BETA3 does not have the problem vmware-checkvm returns correctly "not running in a virtual machine"
open-vm-tools-10.0.5-3.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.
(In reply to Knudch from comment #46) Thanks for all the testing, Knudch. > Virtualbox 5.1.0 BETA3 does not have the problem > > vmware-checkvm returns correctly "not running in a virtual machine" This is great news, probably, we can assume this was a problem on VirtualBox side that has been fixed on their side?
Difficult to say....can also be a different behavior(not wrong) from VirtualBox which makes vmware-checkvm to hang up A question could be if vmware-checkvm/vmtoolsd is robust enough ? Anyway thanks for your efforts
I will make a question in the Virtualbox BETA forum... Do they have fixed this by purpose or coincident ?
(In reply to Knudch from comment #49) > A question could be if vmware-checkvm/vmtoolsd is robust enough ? I think it was until this issue was found :) (In reply to Knudch from comment #50) > I will make a question in the Virtualbox BETA forum... > Do they have fixed this by purpose or coincident ? Yep, that will be helpful. Thanks!
Nobody has replied on my posting in Virtual Box Beta forum.... I still think that vmtoolsd should be more robust... But difficult to say when we do not know what the VirtualBox people has change in the 5.1 version ? But I consider this Bug as closed because VirtualBox 5.1 is released thanks for everybody's effort Knud
Thanks Knud with further follow-up. I think we are fine for now based on your comment #46. We will keep looking for ways to make vmtoolsd more robust.
This bug appears to have been reported against 'rawhide' during the Fedora 25 development cycle. Changing version to '25'.
This message is a reminder that Fedora 25 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 25. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '25'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 25 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 25 changed to end-of-life (EOL) status on 2017-12-12. Fedora 25 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.