This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 523127 - [RHEL-6 Xen]: Hotplugging vcpus to a 64-bit domU doesn't work
[RHEL-6 Xen]: Hotplugging vcpus to a 64-bit domU doesn't work
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: udev (Show other bugs)
6.0
All Linux
high Severity medium
: rc
: ---
Assigned To: Harald Hoyer
Jan Ščotka
:
Depends On: 499611
Blocks: 633349 630947
  Show dependency treegraph
 
Reported: 2009-09-14 04:24 EDT by Chris Lalancette
Modified: 2011-01-05 05:10 EST (History)
17 users (show)

See Also:
Fixed In Version: udev-147-1.fc13
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 499611
: 630947 (view as bug list)
Environment:
Last Closed: 2010-11-10 16:48:56 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Chris Lalancette 2009-09-14 04:24:06 EDT
+++ This bug was initially created as a clone of Bug #499611 +++

Description of problem:
Following the test case here:

https://fedoraproject.org/wiki/QA:Testcase_Virtualization_XenDomU_CPU_hotplug

I'm trying to plug and unplug virtual CPUs from an F-11 x86_64 Xen guest.  I boot up the guest with 8 vcpus, and all 8 are shown in /proc/cpuinfo.  Then, on the dom0, I issue:

# xm vcpu-set f11 2

Which unplugs 6 of the vcpus.  Now only 2 cpus are shown in the guest.  However, if I now do:

# xm vcpu-set f11 8

Nothing happens.  What should happen is that we should hotplug 6 more vcpus into the guest, and now the guest should have the original 8 vcpus.

--- Additional comment from clalance@redhat.com on 2009-05-08 11:21:53 EDT ---

Justin,
     Another fairly important thing to get working in upstream and Fedora.

Chris Lalancette

--- Additional comment from jeremy@goop.org on 2009-05-11 13:36:24 EDT ---

It needs some usermode support to work.  It will generate uevents, and a script is needed to do the actual onlining.  You can see that the cpus will appear and disappear in /sys/devices/system/cpu/present.

This is a behaviour change from older Xen kernels which did this internally, but seemed like the better approach.

--- Additional comment from jforbes@redhat.com on 2009-05-11 13:49:01 EDT ---

So I suppose the question here would be where should such a script live? With the pv_ops domU kernel, there is no difference in a PV domU install and a bare metal.

--- Additional comment from jeremy@goop.org on 2009-05-11 13:57:48 EDT ---

Do you mean "which package" or "which filesystem"?

The idea is that this should be the same as a native system with hotplug physical cpus; you insert the CPU module, then a hotplug script actually onlines them.  There's nothing Xen-specific about the script or what it does.

(Though I'm not certain that real cpu hotplug works like this.)

--- Additional comment from jforbes@redhat.com on 2009-05-15 12:19:47 EDT ---

Right, so it appears that real hotplug can work like this just fine as well.  The real question: is there any reason we should not make udev online cpus on an add?  The arguments I have seen against this are power consumption related.  Udev certainly gets events on the CPU add or removal, and making it online a CPU on add would be trivial.  It has certainly changed from a mechanism question to a policy question.

--- Additional comment from berrange@redhat.com on 2009-05-15 12:31:06 EDT ---

The uevents / sysfs  notication/online Jeremy mentions in comment #2, sounds like exactly the same thing Glauba came up against when trying todo KVM CPU hotplug. I can't remember whether he ended up using this or not, but it would make sense for us to have same CPU onlining process for KVM and Xen guests if practical.

--- Additional comment from jforbes@redhat.com on 2009-05-15 12:43:42 EDT ---

kvm would work the same way.  Essentially it is a udev rule.  After thinking on it a bit, I believe the rule is necessary.  I find it hard to believe that people who really care about power management would add a CPU to a running system without a desire to online that CPU.  Most power management would be achieved by taking existing CPUs online or offline without attaching or removing them.  The udev rule would only online a CPU when it is attached, meaning CPUs that you specifically offline but leave present would not change state.

--- Additional comment from jforbes@redhat.com on 2009-05-20 12:28:44 EDT ---

Created an attachment (id=344840)
CPU online patch

Attached a patch which automatically onlines CPUs as they are added.  Tested with several add and removes on F-11.

--- Additional comment from fedora-triage-list@redhat.com on 2009-06-09 11:19:08 EDT ---


This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

--- Additional comment from harald@redhat.com on 2009-06-30 07:12:08 EDT ---

Why does the kernel not bring the CPUs online? Aren't real CPUs brought online immediately?
Comment 2 RHEL Product and Program Management 2009-09-14 04:50:24 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 3 Chris Lalancette 2009-09-24 05:21:01 EDT
Somewhat related to this, there was recently a bug with bare-metal hotplug:

https://bugzilla.redhat.com/show_bug.cgi?id=523505

So, we might be able to ask the people who worked on that what the expected behavior is; is the kernel responsible for doing the hotplug in the end, or is it userland?

Chris Lalancette
Comment 4 Phil Knirsch 2009-10-02 10:16:36 EDT
See this comment:

--- Additional comment from jeremy@goop.org on 2009-05-11 13:36:24 EDT ---

It needs some usermode support to work.  It will generate uevents, and a script
is needed to do the actual onlining.  You can see that the cpus will appear and
disappear in /sys/devices/system/cpu/present.

This is a behaviour change from older Xen kernels which did this internally,
but seemed like the better approach.


So it seems we really need the udev rule now for CPU hotplugging. The rule also looks dead easy and simple, so granting Devel ACK for this.

Thanks & regards, Phil
Comment 6 Chris Lalancette 2009-11-16 04:45:11 EST
There were patches posted recently for pv_ops kernel to support hotplug:

http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00657.html
http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00658.html

I'm not quite sure if they were accepted or not, but worth looking into.  We now have at least 2 solutions for this; we just need to figure out which one we want to use for RHEL-6.

Chris Lalancette
Comment 7 Harald Hoyer 2009-11-16 05:42:42 EST
please test udev-147-1.fc13

https://koji.fedoraproject.org/koji/buildinfo?buildID=140950
Comment 8 Jeremy Fitzhardinge 2009-11-16 13:35:37 EST
(In reply to comment #6)
> There were patches posted recently for pv_ops kernel to support hotplug:
> 
> http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00657.html
> http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00658.html
> 
> I'm not quite sure if they were accepted or not, but worth looking into.  We
> now have at least 2 solutions for this; we just need to figure out which one we
> want to use for RHEL-6.

This pertains to physical (host) cpu hotplug, not domain vcpu hotplug, which is quite distinct.  I don't think its relevant to this bug.
Comment 9 Chris Lalancette 2009-11-17 04:36:58 EST
(In reply to comment #8)
> (In reply to comment #6)
> > There were patches posted recently for pv_ops kernel to support hotplug:
> > 
> > http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00657.html
> > http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00658.html
> > 
> > I'm not quite sure if they were accepted or not, but worth looking into.  We
> > now have at least 2 solutions for this; we just need to figure out which one we
> > want to use for RHEL-6.
> 
> This pertains to physical (host) cpu hotplug, not domain vcpu hotplug, which is
> quite distinct.  I don't think its relevant to this bug.  

Yeah, you are right.  I mis-read the upstream posting.  Sorry about that!

Chris Lalancette
Comment 10 Fedora Update System 2009-11-27 16:47:30 EST
udev-145-14.fc12 has been pushed to the Fedora 12 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 11 releng-rhel@redhat.com 2009-12-16 16:53:03 EST
Fixed in 'udev-147-1.fc13'. 'udev-147-2.1.el6' included in compose 'RHEL6.0-20091216.nightly'.
Moving to ON_QA.
Comment 15 Andrew Jones 2010-09-03 06:09:52 EDT
I tested on latest rhel6 64b PV xen domu with latest rhel5 dom0

rhel6 packages:
kernel-2.6.32-70.el6.x86_64
udev-147-2.29.el6.x86_64

'xm vcpu-set' unfortunately has no affect on /sys/devices/system/cpu/present. We're probably missing something from upstream in the kernel, or possibly something broke in the kernel, because I'm pretty sure it used to work (so a kernel issue). However, I can online/offline the guest cpus with echo 0|1 > /sys/devices/system/cpu/cpu?/online fine.
Comment 23 Andrew Jones 2010-09-07 09:06:48 EDT
Also confirmed that vcpu hot plug is working on rhel6 pv guests. It didn't work for me before (comment 15), because the xen userspace has regressed. I was using -115 rather than -105. I'm opening a bug to xen userspace for that issue, but I concur that this bug is verified.

Note the expected behavior is a default maxvcpus == vcpus originally defined. It's possible to add more if you set the maxvcpus param in your config to something higher, such as 16.
Comment 24 Jan Ščotka 2010-09-07 09:17:02 EDT
Okay, so this bug seems to be repaired.
test of changing numbers of cpus PASS
Comment 25 releng-rhel@redhat.com 2010-11-10 16:48:56 EST
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.