Description of problem: Running two KVM virtual machines and ksmd takes lot of CPU. Version-Release number of selected component (if applicable): kernel-2.6.31.5-127.fc12.x86_64 How reproducible: Do not know currently problem exits, VMs are in use. Steps to Reproduce: 1. Start two kvm VMs. 2. Monitor CPU Actual results: High CPU usage. About 10-25% Expected results: Well ksmd might less aggressively be searching for pages ??. Additional info: 1) PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6546 qemu 20 0 1396m 1.0g 1468 S 180.1 33.9 2325:48 qemu-kvm 6645 qemu 20 0 1392m 977m 1468 S 132.7 32.4 1402:57 qemu-kvm 46 root 25 5 0 0 0 S 12.9 0.0 261:03.42 ksmd 2) Host is upgraded from FC11 to FC 12 with yum. 3) I also started ksmtuned [root@porto-1-fedora trunk]# service ksmtuned status ksmtuned (pid 1844) is running... put no help after 15 minutes of running. 4) Server has Intel(R) Xeon(R) CPU E5405 @ 2.00GHz and both quest have to VCPU allocated.
I experience the same problem. With just one VM running (Also F12, kernel-2.6.31.5-127.fc12.x86_64)
If U are running just one VM (since there are no pages to merge in one VM) or want to disable ksm, do the following. 1) Stop ksmtuned service ksmtuned stop 2) Disable ksm... echo 0 >/sys/kernel/mm/ksm/run I decided to run without ksm (because of high CPU), although it saved some memory with two VMs. Yes , I know this is bugzilla and not a wiki, but decided to write here anyway a quick solution.
I today updated to: kernel-2.6.31.6-145.fc12.x86_64 and at least for me the problem seems to be corrected: And yes I have ksm on again... [root@porto-1-fedora qemu]# cat /sys/kernel/mm/ksm/run 1 Since I reported this bug, I am also closing it with "works for me", because this is a low-key issue. If somebody still experiences the problem with newer kernels, he can open this again.
..oops sorry too much coffee/debugging today, problem still exist. I just had putty open to wrong server ;).
I have the same problem here, fedora 12kvm, two guest running fedora 12 and I had around 25-30% of the cpu used by the ksm process, untill I stop my VMs. KSM is working but use a lot of cpu.. I am available if someone want to test a solution.
I have the same issue here. was wondering why I showed fairly high cpu usage. ksmd shows 40-60% cpu usage in top. I googled and realized what ksm is. It sounds very useful, but it would be nice if it did it's job on less cpu. To the poster with the ksmd shutdown advice - this might not be a wiki, but that advice is useful all the same to those of us only running one VM and thus not needing ksmd. Thanks for the info! Linux localhost 2.6.31.6-162.fc12.x86_64 #1 SMP Fri Dec 4 00:06:26 EST 2009 x86_64 x86_64 x86_64 GNU/Linux Potentially Relevant Packages: libvirt-0.7.1-15.fc12.x86_64 qemu-kvm-0.11.0-12.fc12.x86_64 virt-manager-0.8.0-7.fc12.noarch virt-mem-0.3.1-9.fc12.x86_64
I am also seeing about 30-40% CPU usage running one guest.
Same for me, 20-60% CPU with one KVM guest. I shut it down according to Ari Tilli's instructions and indeed it stops eating CPU now. However I do like the concept of ksm, so I would certainly re-enable when it can do its work with less CPU. By the way, I'm running 2.6.31.6-166.fc12.i686.PAE
I am experiencing the same problem with kernel 2.6.31.6-166.fc12.x86_64. its around 45-60 % of 1 CPU
As I see it's possibly to fine tune ksmd by /etc/ksmtuned.conf which can help to make it less aggressively in CPU eating. But I don't find detailed description of options in it for now :(
23641 root 6.3 19.4 67:43.59 qemu-kvm 32 root 1.3 0.0 103:20.31 ksmd /etc/ksmtuned.conf # How long ksmtuned should sleep between tuning adjustments # KSM_MONITOR_INTERVAL=60 KSM_MONITOR_INTERVAL=480 # Millisecond sleep between ksm scans for 16Gb server. # Smaller servers sleep more, bigger sleep less. # KSM_SLEEP_MSEC=10 KSM_SLEEP_MSEC=50 # KSM_NPAGES_BOOST=300 # KSM_NPAGES_DECAY=-50 # KSM_NPAGES_MIN=64 # KSM_NPAGES_MAX=1250 KSM_NPAGES_MAX=256 # KSM_THRES_COEF=20 # KSM_THRES_CONST=2048 KSM_THRES_CONST=1024 ============================== with default /etc/ksmtuned.conf ksmd cpu usage was 30-45 % with one kvm
Seeing the same issue with: libvirt-0.7.1-15.fc12.x86_64 qemu-kvm-0.11.0-12.fc12.x86_64 kernel-2.6.31.9-174.fc12.x86_64 top - 16:36:45 up 20:16, 3 users, load average: 0.20, 0.51, 1.43 Tasks: 303 total, 1 running, 302 sleeping, 0 stopped, 0 zombie Cpu(s): 1.0%us, 6.9%sy, 0.0%ni, 92.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 6176288k total, 6134448k used, 41840k free, 698548k buffers Swap: 2120560k total, 323144k used, 1797416k free, 48876k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 74 root 25 5 0 0 0 S 45.4 0.0 446:21.47 ksmd 20104 qemu 20 0 3089m 2.2g 1240 S 12.6 38.0 70:03.19 qemu-kvm 19591 qemu 20 0 5033m 2.5g 1256 S 8.0 42.2 121:46.73 qemu-kvm The virtual machines run different distributions (host fedora12, with rawhide and rhel 5.4 guests).
(In reply to comment #12) > Seeing the same issue with: > libvirt-0.7.1-15.fc12.x86_64 > qemu-kvm-0.11.0-12.fc12.x86_64 > kernel-2.6.31.9-174.fc12.x86_64 > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 74 root 25 5 0 0 0 S 45.4 0.0 446:21.47 ksmd > 20104 qemu 20 0 3089m 2.2g 1240 S 12.6 38.0 70:03.19 qemu-kvm Try to change variables in /etc/ksmtuned.conf as in my above. For me this decrease CPU usage by ksmd from 30-45 % to 1.5-2%. I don't know what all variables I change exactly mean, but CPU usage became much lower.
HI all, I have the same problem with ksmd in 2.6.31.12-174.2.3.fc12_x86_64. I have changed the variables in ksmtuned.conf with no effect. Actually, I noticed that stopping ksmtuned does not have any effect at all.The CPU usage is ~13% - host is freebsd 6.4 i386. Just wondering, what is the expected CPU usage with one guest running? top - 15:37:59 up 2 days, 1:25, 3 users, load average: 0.00, 0.00, 0.00 Tasks: 191 total, 2 running, 189 sleeping, 0 stopped, 0 zombie Cpu(s): 1.4%us, 10.1%sy, 0.0%ni, 88.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1798004k total, 1683440k used, 114564k free, 86844k buffers Swap: 4000144k total, 53264k used, 3946880k free, 433016k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32 root 25 5 0 0 0 S 13.9 0.0 281:39.91 ksmd 1372 qemu 20 0 879m 522m 3120 R 8.0 29.8 333:44.04 qemu-kvm
Changing the KSM_MONITOR_INTERVAL had no effect, I'm getting 50% usage on a Core2 system with 8G. I'm running a single 5G CentOS5.4 VM. I changed the interval to 500 and restarted both ksm_tuned and ksmd, it had zero effect so for now I've diabled ksm. I'm running this kernel, Linux hornet 2.6.31.12-174.2.22.fc12.x86_64 #1 x86_64 x86_64 x86_64 The VM is using the following kernel Linux tomcat 2.6.18-164.11.1.el5 #1 x86_64 x86_64 x86_64
I believe KSM Tuned just controls KSM.. the monitor interval probably just changes how frequently it checks the status of the system (probably memory usage) to determine if it should run KSM
Problem still here with latest FC13 64 bit kernel 2.6.33.5-124.fc13.x86_64 on an Core i5 with one VM running, ksmd was running 45% cpu consistently. Turned ksmd off as per that marvellous post from Ari.
For me on FC13, ksmd kept coming back even after following Ari's steps for disabling ksmtuned and ksmd. Instead, I used the following steps to permanently disable both services: # service ksm stop # service ksmtuned stop # chkconfig ksm off # chkconfig ksmtuned off
On my 20GB 4-core F13 box, ksmd was taking 30-60% of a cpu with just one kvm running, giving a 8-15% overhead for this feature only. There appear to be few diagnostics to justify this cost, so off it goes.
I have a single Windows VM hovering between 32% and 58% CPU while KVM is varying at about half that of the itself CPU usage. I noticed the fans in my machine were getting louder, checked `top` and eventually found this report.
(In reply to comment #20) > I have a single Windows VM hovering between 32% and 58% CPU while KVM is > varying at about half that of the itself CPU usage. I noticed the fans in my > machine were getting louder, checked `top` and eventually found this report. This was on Fedora 13 (2.6.33.6-147.fc13.x86_64). Seems like the Windows guest was doing auto updates. However, that still seems high for `ksmd` considering I only had one Windows guest.
Found the same problem. 30% of CPU use even when the single VM I have is idle. My system is Fedora 13 with the following packages: kernel-2.6.33.6-147.2.4.fc13.x86_64 libvirt-0.8.2-1.fc13.x86_64 qemu-kvm-0.12.3-8.fc13.x86_64 virt-mem-0.3.1-9.fc12.x86_64 BTW, isn't it a bit strange that this last package has "fc12" in its name? This is a fresh installation with just default repositories.
(In reply to comment #22) > Found the same problem. 30% of CPU use even when the single VM I have is idle. I'm seeing the same thing... fully up to date F13 system, with the following packages: kernel-2.6.34.6-47.fc13.x86_64 libvirt-0.8.2-1.fc13.x86_64 virt-mem-0.3.1-9.fc12.x86_64 qemu-kvm-0.12.5-1.fc13.x86_64 Mouse and keyboard operations get sluggish in the VM, and running 'top' on the host shows that ksmd is taking between 29 and 33 percent of the CPU.
I had a similar problem with just one 512MB qemu-kvm instance on a 4GB system. After some bugzilla research, I see two reasons why ksmtuned wrongly keeps ksm scanning: * ksmtuned underestimates free memory due to misinterpreting /proc/meminfo and ignoring cached memory (bug 626658) * ksmtuned (arguably) overestimates committed memory by counting vsz rather than rsz (this is fixed, perhaps accidentally, by bug 609016) I applied these fixes to my local /usr/sbin/ksmtuned and now ksmtuned doesn't start ksm scanning until 3 or 4 identical 512MB vms are started, and ksm quickly frees enough pages that it is stopped again after two tuning intervals. These fixes look like they are in qemu 0.13 and in F14. I suppose there's still a theoretical bug when enough heterogenous vms are started to reduce free memory below the ksmtuned threshold but ksm can't free any more pages and keeps fruitlessly scanning pages at full speed. (Or if memory pressure comes from another program on the host?) But maybe the system would start swapping and equilibrium would be restored? But I'm not an expert and so this paragraph is all wild speculation.
This message is a reminder that Fedora 12 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 12. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '12'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 12's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 12 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Version bumped to 13 (Comment #23)
Might as well bump it to f14. I'm still suffering from this on up to date f14. I tried to re-enable ksmd and ksm tune services for trying them out again, but it's no better: 73 root 25 5 0 0 0 R 56.0 0 0.0 2336:51 ksmd 32417 qemu 20 0 2569m 659m 812 S 13.9 6 8.2 2659:30 qemu-kvm 28604 qemu 20 0 4641m 3.1g 852 S 13.2 7 38.9 2546:54 qemu-kvm I'm running two rhel5 guests, and ksmd just doesn't calm down. The guests are idling. The status has been such for many days. $ rpm -qa *kvm* etherboot-zroms-kvm-5.4.4-18.fc13.noarch qemu-kvm-0.13.0-1.fc14.x86_64 etherboot-roms-kvm-5.4.4-18.fc13.noarch qemu-kvm-tools-0.13.0-1.fc14.x86_64
adjusting version as per comment #27
FYI, if this helpful, this exists in RHEL 6.1 as well Host+guests here are all RHEL 6.1. All environments are idling, plenty of memory on the host. ksmd has gone as high as 40%. Hopefully you find this information useful, otherwise I apologize in advance if this is the wrong forum for this. Linux prometheus 2.6.32-220.2.1.el6.x86_64 #1 SMP PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 39 root 25 5 0 0 0 R 21.9 0.0 221:52.42 ksmd 1287 qemu 20 0 1352m 309m 1100 R 6.6 3.9 59:11.85 qemu-kvm 1359 qemu 20 0 1353m 428m 1116 R 6.6 5.4 59:00.32 qemu-kvm 1426 qemu 20 0 1353m 326m 1120 R 6.3 4.1 58:31.47 qemu-kvm 1221 qemu 20 0 1354m 314m 1100 R 5.6 4.0 58:10.90 qemu-kvm
CentOS 6.3 - 2.6.32-220.4.2.el6.x86_64 load average: 13.55, 14.78, 14.50 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 180 root 25 5 0 0 0 S 67.7 0.0 82092:14 ksmd 3206 qemu 20 0 3846m 1.7g 884 S 21.5 7.4 167503:49 qemu-kvm 3438 qemu 20 0 3284m 1.8g 668 S 21.5 7.7 24006:15 qemu-kvm 3172 qemu 20 0 3324m 2.0g 668 S 17.5 8.5 34500:04 qemu-kvm 3041 qemu 20 0 3284m 2.8g 640 S 8.3 11.8 19700:13 qemu-kvm 3108 qemu 20 0 3835m 2.9g 744 S 7.6 12.4 16641:31 qemu-kvm 27686 qemu 20 0 3843m 2.9g 744 S 7.6 12.5 37708:11 qemu-kvm 3504 qemu 20 0 3284m 2.9g 640 S 6.9 12.2 18620:15 qemu-kvm 3715 qemu 20 0 3284m 2.9g 636 S 6.3 12.4 18173:29 qemu-kvm
F14 is end of life, so closing this. If anyone is still having issues on F16+ then reassign. RHEL issues should go through Red Hat support.
just had this on F17 x86_64 with the default /etc/ksmtuned.conf reopening qemu-kvm-1.0-18.fc17.x86_64 libvirt-0.9.11.4-3.fc17.x86_64 under 3.5.0-2.fc17.x86_64 kernel Running a single RHEL6 VM on this "Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz" with 2GB on the host and 1GB for the guest my system got extremely sluggish. I needed the host urgently. So upon seeing ksmd using a lot of CPU, when load hit about 7, I stopped it (sysctl stop). Please do tell if you need real testing. I have adjusted the ksmtuned values as per Comment #11 and it will be activated again on next reboot.
This problem still exists on FC18. Can somebody please update the Version(s) field of this ticket?
Just ran into this on Fedora 18 Beta. Seems like it's more of a documentation / operational issue than a 'bug' - ksmd is doing what it's supposed to do, right?
Well. see my comment #24. If the system is under memory pressure (less free memory than the ksmtuned threshold) then ksmd will scan pages at full speed indefinitely. While it does make sense to scan aggressively in these conditions, it doesn't make as much sense to continue scanning at full pace once the system reaches a steady state and there are no more pages suitable for merging. This isn't a purely theoretical concern - I'm fairly sure I've seen this on the same system (though now 8GB) mentioned in my earlier comment, but haven't taken the time to verify my hypothesis.
(In reply to comment #35) > Well. see my comment #24. If the system is under memory pressure (less free > memory than the ksmtuned threshold) then ksmd will scan pages at full speed > indefinitely. While it does make sense to scan aggressively in these > conditions, it doesn't make as much sense to continue scanning at full pace > once the system reaches a steady state and there are no more pages suitable > for merging. > > This isn't a purely theoretical concern - I'm fairly sure I've seen this on > the same system (though now 8GB) mentioned in my earlier comment, but > haven't taken the time to verify my hypothesis. When I saw it, I had three 1536 MB RAM guests running in an 8 GB host. I don't think I can reproduce it with just two of them.
I'm seeing this too on RHEL6, with about 16 VMs, mostly centos6. Restarting all VMs and libvirtd seemed to have worked around it for now. I've stopped ksm/ksmtuned to see if this will happen again or not. The machine is a 16GB server and less much less then 16GB handed out to VMs
This message is a reminder that Fedora 17 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 17. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '17'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 17's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 17 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior to Fedora 17's end of life. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Per comment 33, could somebody please bump the version field on this ticket to 18?
I am also seeing this in F18. Just fire up Virtual Manager and started to install Debian Jessie as a new virtual machine. I have also other 8 VMs but they are switch off now. top shows the following: KiB Mem: 5982540 total, 5784736 used, 197804 free, 5460 buffers KiB Swap: 4385708 total, 2655932 used, 1729776 free, 866328 cached 29311 qemu 20 0 3891m 1.8g 1836 S 15.9 32.3 5:10.11 /usr/bin/qemu-kvm 12074 xxx 20 0 2305m 753m 24m S 1.0 12.9 494:24.88 /usr/lib64/firefo 2871 xxx 20 0 2638m 500m 6912 S 0.3 8.6 80:49.83 /usr/bin/java -Xm 1697 xxx 20 0 1644m 459m 8656 S 0.0 7.9 52:11.59 korgac Hope that it helps. Regards
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 18 kernel bugs. Fedora 18 has now been rebased to 3.11.4-101.fc18. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 19, and are still experiencing this issue, please change the version to Fedora 19. If you experience different issues, please open a new bug report for those.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. It has been over a month since we asked you to test the 3.11 kernel updates and let us know if your issue has been resolved or is still a problem. When this happened, the bug was set to needinfo. Because the needinfo is still set, we assume either this is no longer a problem, or you cannot provide additional information to help us resolve the issue. As a result we are closing with insufficient data. If this is still a problem, we apologize, feel free to reopen the bug and provide more information so that we can work towards a resolution If you experience different issues, please open a new bug report for those.
I generally run my VMs on Sl6.4 (RHEL 6.4) servers however to see if there is an issue on Fedora 19 I fired up two SL6.4 VMs on my F19 workstation (iCore7 with 32G). qemu-kvm is eating about 5% of the CPU with two idle SL6.4 VMs running (both with the same kernel). I don't see how to disable ksmtuned on F19, it's not listed as a service on F19 the way it is on SL6.4 so I can't tell if it's the culprit or not.
Seeing the exact same issue on CentOS 7 x86_64. ksmd is using 40% CPU constantly with only one VM running, increasing the load on the system significantly. System specs: CentOS Linux release 7.0.1406 (Core) Linux zaphod 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan 29 18:05:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux ipxe-roms-qemu-20130517-5.gitc4bce43.el7.noarch qemu-img-1.5.3-60.el7_0.11.x86_64 qemu-kvm-1.5.3-60.el7_0.11.x86_64 libvirt-daemon-driver-qemu-1.1.1-29.el7_0.7.x86_64 qemu-kvm-common-1.5.3-60.el7_0.11.x86_64 Please reopen and specify what information is missing for the closing reason: ( INSUFFICIENT_DATA)
Please open bugs for CentOS in their bug tracker. The Fedora kernel team doesn't work on CentOS.