RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 523127 - [RHEL-6 Xen]: Hotplugging vcpus to a 64-bit domU doesn't work
Summary: [RHEL-6 Xen]: Hotplugging vcpus to a 64-bit domU doesn't work
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: udev
Version: 6.0
Hardware: All
OS: Linux
high
medium
Target Milestone: rc
: ---
Assignee: Harald Hoyer
QA Contact: Jan Ščotka
URL:
Whiteboard:
Depends On: 499611
Blocks: 630947 633349
TreeView+ depends on / blocked
 
Reported: 2009-09-14 08:24 UTC by Chris Lalancette
Modified: 2011-01-05 10:10 UTC (History)
17 users (show)

Fixed In Version: udev-147-1.fc13
Doc Type: Bug Fix
Doc Text:
Clone Of: 499611
: 630947 (view as bug list)
Environment:
Last Closed: 2010-11-10 21:48:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Chris Lalancette 2009-09-14 08:24:06 UTC
+++ This bug was initially created as a clone of Bug #499611 +++

Description of problem:
Following the test case here:

https://fedoraproject.org/wiki/QA:Testcase_Virtualization_XenDomU_CPU_hotplug

I'm trying to plug and unplug virtual CPUs from an F-11 x86_64 Xen guest.  I boot up the guest with 8 vcpus, and all 8 are shown in /proc/cpuinfo.  Then, on the dom0, I issue:

# xm vcpu-set f11 2

Which unplugs 6 of the vcpus.  Now only 2 cpus are shown in the guest.  However, if I now do:

# xm vcpu-set f11 8

Nothing happens.  What should happen is that we should hotplug 6 more vcpus into the guest, and now the guest should have the original 8 vcpus.

--- Additional comment from clalance on 2009-05-08 11:21:53 EDT ---

Justin,
     Another fairly important thing to get working in upstream and Fedora.

Chris Lalancette

--- Additional comment from jeremy on 2009-05-11 13:36:24 EDT ---

It needs some usermode support to work.  It will generate uevents, and a script is needed to do the actual onlining.  You can see that the cpus will appear and disappear in /sys/devices/system/cpu/present.

This is a behaviour change from older Xen kernels which did this internally, but seemed like the better approach.

--- Additional comment from jforbes on 2009-05-11 13:49:01 EDT ---

So I suppose the question here would be where should such a script live? With the pv_ops domU kernel, there is no difference in a PV domU install and a bare metal.

--- Additional comment from jeremy on 2009-05-11 13:57:48 EDT ---

Do you mean "which package" or "which filesystem"?

The idea is that this should be the same as a native system with hotplug physical cpus; you insert the CPU module, then a hotplug script actually onlines them.  There's nothing Xen-specific about the script or what it does.

(Though I'm not certain that real cpu hotplug works like this.)

--- Additional comment from jforbes on 2009-05-15 12:19:47 EDT ---

Right, so it appears that real hotplug can work like this just fine as well.  The real question: is there any reason we should not make udev online cpus on an add?  The arguments I have seen against this are power consumption related.  Udev certainly gets events on the CPU add or removal, and making it online a CPU on add would be trivial.  It has certainly changed from a mechanism question to a policy question.

--- Additional comment from berrange on 2009-05-15 12:31:06 EDT ---

The uevents / sysfs  notication/online Jeremy mentions in comment #2, sounds like exactly the same thing Glauba came up against when trying todo KVM CPU hotplug. I can't remember whether he ended up using this or not, but it would make sense for us to have same CPU onlining process for KVM and Xen guests if practical.

--- Additional comment from jforbes on 2009-05-15 12:43:42 EDT ---

kvm would work the same way.  Essentially it is a udev rule.  After thinking on it a bit, I believe the rule is necessary.  I find it hard to believe that people who really care about power management would add a CPU to a running system without a desire to online that CPU.  Most power management would be achieved by taking existing CPUs online or offline without attaching or removing them.  The udev rule would only online a CPU when it is attached, meaning CPUs that you specifically offline but leave present would not change state.

--- Additional comment from jforbes on 2009-05-20 12:28:44 EDT ---

Created an attachment (id=344840)
CPU online patch

Attached a patch which automatically onlines CPUs as they are added.  Tested with several add and removes on F-11.

--- Additional comment from fedora-triage-list on 2009-06-09 11:19:08 EDT ---


This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

--- Additional comment from harald on 2009-06-30 07:12:08 EDT ---

Why does the kernel not bring the CPUs online? Aren't real CPUs brought online immediately?

Comment 2 RHEL Program Management 2009-09-14 08:50:24 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

Comment 3 Chris Lalancette 2009-09-24 09:21:01 UTC
Somewhat related to this, there was recently a bug with bare-metal hotplug:

https://bugzilla.redhat.com/show_bug.cgi?id=523505

So, we might be able to ask the people who worked on that what the expected behavior is; is the kernel responsible for doing the hotplug in the end, or is it userland?

Chris Lalancette

Comment 4 Phil Knirsch 2009-10-02 14:16:36 UTC
See this comment:

--- Additional comment from jeremy on 2009-05-11 13:36:24 EDT ---

It needs some usermode support to work.  It will generate uevents, and a script
is needed to do the actual onlining.  You can see that the cpus will appear and
disappear in /sys/devices/system/cpu/present.

This is a behaviour change from older Xen kernels which did this internally,
but seemed like the better approach.


So it seems we really need the udev rule now for CPU hotplugging. The rule also looks dead easy and simple, so granting Devel ACK for this.

Thanks & regards, Phil

Comment 6 Chris Lalancette 2009-11-16 09:45:11 UTC
There were patches posted recently for pv_ops kernel to support hotplug:

http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00657.html
http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00658.html

I'm not quite sure if they were accepted or not, but worth looking into.  We now have at least 2 solutions for this; we just need to figure out which one we want to use for RHEL-6.

Chris Lalancette

Comment 7 Harald Hoyer 2009-11-16 10:42:42 UTC
please test udev-147-1.fc13

https://koji.fedoraproject.org/koji/buildinfo?buildID=140950

Comment 8 Jeremy Fitzhardinge 2009-11-16 18:35:37 UTC
(In reply to comment #6)
> There were patches posted recently for pv_ops kernel to support hotplug:
> 
> http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00657.html
> http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00658.html
> 
> I'm not quite sure if they were accepted or not, but worth looking into.  We
> now have at least 2 solutions for this; we just need to figure out which one we
> want to use for RHEL-6.

This pertains to physical (host) cpu hotplug, not domain vcpu hotplug, which is quite distinct.  I don't think its relevant to this bug.

Comment 9 Chris Lalancette 2009-11-17 09:36:58 UTC
(In reply to comment #8)
> (In reply to comment #6)
> > There were patches posted recently for pv_ops kernel to support hotplug:
> > 
> > http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00657.html
> > http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00658.html
> > 
> > I'm not quite sure if they were accepted or not, but worth looking into.  We
> > now have at least 2 solutions for this; we just need to figure out which one we
> > want to use for RHEL-6.
> 
> This pertains to physical (host) cpu hotplug, not domain vcpu hotplug, which is
> quite distinct.  I don't think its relevant to this bug.  

Yeah, you are right.  I mis-read the upstream posting.  Sorry about that!

Chris Lalancette

Comment 10 Fedora Update System 2009-11-27 21:47:30 UTC
udev-145-14.fc12 has been pushed to the Fedora 12 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 11 releng-rhel@redhat.com 2009-12-16 21:53:03 UTC
Fixed in 'udev-147-1.fc13'. 'udev-147-2.1.el6' included in compose 'RHEL6.0-20091216.nightly'.
Moving to ON_QA.

Comment 15 Andrew Jones 2010-09-03 10:09:52 UTC
I tested on latest rhel6 64b PV xen domu with latest rhel5 dom0

rhel6 packages:
kernel-2.6.32-70.el6.x86_64
udev-147-2.29.el6.x86_64

'xm vcpu-set' unfortunately has no affect on /sys/devices/system/cpu/present. We're probably missing something from upstream in the kernel, or possibly something broke in the kernel, because I'm pretty sure it used to work (so a kernel issue). However, I can online/offline the guest cpus with echo 0|1 > /sys/devices/system/cpu/cpu?/online fine.

Comment 23 Andrew Jones 2010-09-07 13:06:48 UTC
Also confirmed that vcpu hot plug is working on rhel6 pv guests. It didn't work for me before (comment 15), because the xen userspace has regressed. I was using -115 rather than -105. I'm opening a bug to xen userspace for that issue, but I concur that this bug is verified.

Note the expected behavior is a default maxvcpus == vcpus originally defined. It's possible to add more if you set the maxvcpus param in your config to something higher, such as 16.

Comment 24 Jan Ščotka 2010-09-07 13:17:02 UTC
Okay, so this bug seems to be repaired.
test of changing numbers of cpus PASS

Comment 25 releng-rhel@redhat.com 2010-11-10 21:48:56 UTC
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.