Red Hat Bugzilla – Bug 523127
[RHEL-6 Xen]: Hotplugging vcpus to a 64-bit domU doesn't work
Last modified: 2011-01-05 05:10:58 EST
+++ This bug was initially created as a clone of Bug #499611 +++
Description of problem:
Following the test case here:
I'm trying to plug and unplug virtual CPUs from an F-11 x86_64 Xen guest. I boot up the guest with 8 vcpus, and all 8 are shown in /proc/cpuinfo. Then, on the dom0, I issue:
# xm vcpu-set f11 2
Which unplugs 6 of the vcpus. Now only 2 cpus are shown in the guest. However, if I now do:
# xm vcpu-set f11 8
Nothing happens. What should happen is that we should hotplug 6 more vcpus into the guest, and now the guest should have the original 8 vcpus.
--- Additional comment from firstname.lastname@example.org on 2009-05-08 11:21:53 EDT ---
Another fairly important thing to get working in upstream and Fedora.
--- Additional comment from email@example.com on 2009-05-11 13:36:24 EDT ---
It needs some usermode support to work. It will generate uevents, and a script is needed to do the actual onlining. You can see that the cpus will appear and disappear in /sys/devices/system/cpu/present.
This is a behaviour change from older Xen kernels which did this internally, but seemed like the better approach.
--- Additional comment from firstname.lastname@example.org on 2009-05-11 13:49:01 EDT ---
So I suppose the question here would be where should such a script live? With the pv_ops domU kernel, there is no difference in a PV domU install and a bare metal.
--- Additional comment from email@example.com on 2009-05-11 13:57:48 EDT ---
Do you mean "which package" or "which filesystem"?
The idea is that this should be the same as a native system with hotplug physical cpus; you insert the CPU module, then a hotplug script actually onlines them. There's nothing Xen-specific about the script or what it does.
(Though I'm not certain that real cpu hotplug works like this.)
--- Additional comment from firstname.lastname@example.org on 2009-05-15 12:19:47 EDT ---
Right, so it appears that real hotplug can work like this just fine as well. The real question: is there any reason we should not make udev online cpus on an add? The arguments I have seen against this are power consumption related. Udev certainly gets events on the CPU add or removal, and making it online a CPU on add would be trivial. It has certainly changed from a mechanism question to a policy question.
--- Additional comment from email@example.com on 2009-05-15 12:31:06 EDT ---
The uevents / sysfs notication/online Jeremy mentions in comment #2, sounds like exactly the same thing Glauba came up against when trying todo KVM CPU hotplug. I can't remember whether he ended up using this or not, but it would make sense for us to have same CPU onlining process for KVM and Xen guests if practical.
--- Additional comment from firstname.lastname@example.org on 2009-05-15 12:43:42 EDT ---
kvm would work the same way. Essentially it is a udev rule. After thinking on it a bit, I believe the rule is necessary. I find it hard to believe that people who really care about power management would add a CPU to a running system without a desire to online that CPU. Most power management would be achieved by taking existing CPUs online or offline without attaching or removing them. The udev rule would only online a CPU when it is attached, meaning CPUs that you specifically offline but leave present would not change state.
--- Additional comment from email@example.com on 2009-05-20 12:28:44 EDT ---
Created an attachment (id=344840)
CPU online patch
Attached a patch which automatically onlines CPUs as they are added. Tested with several add and removes on F-11.
--- Additional comment from firstname.lastname@example.org on 2009-06-09 11:19:08 EDT ---
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.
More information and reason for this action is here:
--- Additional comment from email@example.com on 2009-06-30 07:12:08 EDT ---
Why does the kernel not bring the CPUs online? Aren't real CPUs brought online immediately?
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release. Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release. This request is not yet committed for
Somewhat related to this, there was recently a bug with bare-metal hotplug:
So, we might be able to ask the people who worked on that what the expected behavior is; is the kernel responsible for doing the hotplug in the end, or is it userland?
See this comment:
--- Additional comment from firstname.lastname@example.org on 2009-05-11 13:36:24 EDT ---
It needs some usermode support to work. It will generate uevents, and a script
is needed to do the actual onlining. You can see that the cpus will appear and
disappear in /sys/devices/system/cpu/present.
This is a behaviour change from older Xen kernels which did this internally,
but seemed like the better approach.
So it seems we really need the udev rule now for CPU hotplugging. The rule also looks dead easy and simple, so granting Devel ACK for this.
Thanks & regards, Phil
There were patches posted recently for pv_ops kernel to support hotplug:
I'm not quite sure if they were accepted or not, but worth looking into. We now have at least 2 solutions for this; we just need to figure out which one we want to use for RHEL-6.
please test udev-147-1.fc13
(In reply to comment #6)
> There were patches posted recently for pv_ops kernel to support hotplug:
> I'm not quite sure if they were accepted or not, but worth looking into. We
> now have at least 2 solutions for this; we just need to figure out which one we
> want to use for RHEL-6.
This pertains to physical (host) cpu hotplug, not domain vcpu hotplug, which is quite distinct. I don't think its relevant to this bug.
(In reply to comment #8)
> (In reply to comment #6)
> > There were patches posted recently for pv_ops kernel to support hotplug:
> > http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00657.html
> > http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00658.html
> > I'm not quite sure if they were accepted or not, but worth looking into. We
> > now have at least 2 solutions for this; we just need to figure out which one we
> > want to use for RHEL-6.
> This pertains to physical (host) cpu hotplug, not domain vcpu hotplug, which is
> quite distinct. I don't think its relevant to this bug.
Yeah, you are right. I mis-read the upstream posting. Sorry about that!
udev-145-14.fc12 has been pushed to the Fedora 12 stable repository. If problems still persist, please make note of it in this bug report.
Fixed in 'udev-147-1.fc13'. 'udev-147-2.1.el6' included in compose 'RHEL6.0-20091216.nightly'.
Moving to ON_QA.
I tested on latest rhel6 64b PV xen domu with latest rhel5 dom0
'xm vcpu-set' unfortunately has no affect on /sys/devices/system/cpu/present. We're probably missing something from upstream in the kernel, or possibly something broke in the kernel, because I'm pretty sure it used to work (so a kernel issue). However, I can online/offline the guest cpus with echo 0|1 > /sys/devices/system/cpu/cpu?/online fine.
Also confirmed that vcpu hot plug is working on rhel6 pv guests. It didn't work for me before (comment 15), because the xen userspace has regressed. I was using -115 rather than -105. I'm opening a bug to xen userspace for that issue, but I concur that this bug is verified.
Note the expected behavior is a default maxvcpus == vcpus originally defined. It's possible to add more if you set the maxvcpus param in your config to something higher, such as 16.
Okay, so this bug seems to be repaired.
test of changing numbers of cpus PASS
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.