Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1048080

Summary:

Qemu-kvm NUMA emulation failed

Product:

Red Hat Enterprise Linux 7

Reporter:

Shaolong Hu <shu>

Component:

qemu-kvm

Assignee:

Eduardo Habkost <ehabkost>

Status:

CLOSED CURRENTRELEASE

QA Contact:

Virtualization Bugs <virt-bugs>

Severity:

high

Docs Contact:

Priority:

high

Version:

7.0

CC:

acathrow, Axel.Thimm, gbeshers, hhuang, huding, juzhang, michen, mst, nzimmer, qiguo, scrandall, shu, sluo, virt-maint, ypu

Target Milestone:

Keywords:

Regression, TestBlocker

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

qemu-kvm-1.5.3-47.el7

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2014-06-13 11:42:07 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

722241, 989628

Attachments:

Description	Flags
dmesg when boot with 4 nodes and 4 cpus and 4G mem	none

Description Shaolong Hu 2014-01-03 05:44:50 UTC

Description of problem:
--------------------------
NUMA emulation failed on qemu-kvm-1.5.3-30.el7.x86_64, it works with qemu-kvm-1.5.3-21.el7.x86_64 and our test record suggests it passed with qemu-kvm-1.5.3-19.el7.x86_64, so this is possibly a regression between 21 and 30.


Version-Release number of selected component (if applicable):
--------------------------------------------------------------
qemu-kvm-1.5.3-30.el7.x86_64
guest: 3.10.0-64.el7.x86_64


How reproducible:
-------------------
100%


Steps to Reproduce:
----------------------
1.boot guest with:

-numa node,mem=1024,cpus=0,nodeid=0 -numa node,mem=1024,cpus=1,nodeid=1 -numa node,mem=1024,cpus=2,nodeid=2 -numa node,mem=1024,cpus=3,nodeid=3

2. in qemu monitor:

(qemu) info numa
4 nodes
node 0 cpus: 0
node 0 size: 1024 MB
node 1 cpus: 1
node 1 size: 1024 MB
node 2 cpus: 2
node 2 size: 1024 MB
node 3 cpus: 3
node 3 size: 1024 MB

3. in guest "numactl --hardware" shows only one node, old version (qemu-kvm-1.5.3-21.el7.x86_64) with same steps and guest image works correctly.

Comment 3 Qian Guo 2014-01-06 07:57:59 UTC

I did not hit this issue with rhel6.5 guest, following is my infos, and hope it helps:

components:
host:
# uname -r
3.10.0-64.el7.x86_64
# rpm -q qemu-kvm
qemu-kvm-1.5.3-31.el7.x86_64
# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 12 13 14 15 16 17
node 0 size: 16340 MB
node 0 free: 10794 MB
node 1 cpus: 6 7 8 9 10 11 18 19 20 21 22 23
node 1 size: 16384 MB
node 1 free: 15023 MB
node distances:
node   0   1 
  0:  10  21 
  1:  21  10

Guest:
# uname -r
2.6.32-438.el6.x86_64

Steps:
1.Boot guest with cli:
# /usr/libexec/qemu-kvm -cpu SandyBridge -m 2G -smp 2,sockets=1,cores=2,threads=1 -M pc -enable-kvm  -name rhel651 -rtc base=localtime,clock=host,driftfix=slew -drive file=/mnt/rhel6u5x64cp1.raw,if=none,format=raw,werror=stop,rerror=stop,media=disk,id=drive-scsi0-disk0 -device virtio-scsi-pci,id=scsi0,addr=0x4 -device scsi-hd,scsi-id=0,lun=0,bus=scsi0.0,drive=drive-scsi0-disk0,id=virtio-disk0 -nodefaults -nodefconfig -monitor stdio   -netdev tap,id=hostnet0,script=/etc/qemu-ifup,vhost=on -device virtio-net-pci,netdev=hostnet0,mac=54:43:1a:10:0b:02,id=vnet0 -spice port=5901,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0  -numa node,cpus=0,nodeid=0,mem=1024 -numa node,cpus=1,nodeid=1,mem=1024

2.Check inside guest:

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0
node 0 size: 1023 MB
node 0 free: 717 MB
node 1 cpus: 1
node 1 size: 1023 MB
node 1 free: 787 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 


the guest has 2 nodes.

Comment 4 Qian Guo 2014-01-06 09:02:07 UTC

Tested this scenario for several times, found that only boot guest with 
-m 2G -smp 2,sockets=1,cores=2,threads=1
-numa node,cpus=0,nodeid=0,mem=1024 -numa node,cpus=1,nodeid=1,mem=1024

that the numa works in guest.

I tried that when change any of above values, it can reproduced (numa won't work).

So the bug is reproduced by reporter.

Comment 5 Eduardo Habkost 2014-01-06 19:26:32 UTC

(In reply to Qian Guo from comment #4)
> Tested this scenario for several times, found that only boot guest with 
> -m 2G -smp 2,sockets=1,cores=2,threads=1
> -numa node,cpus=0,nodeid=0,mem=1024 -numa node,cpus=1,nodeid=1,mem=1024
> 
> that the numa works in guest.
> 
> I tried that when change any of above values, it can reproduced (numa won't
> work).
> 

Please show the full command-line of at least one case where it doesn't work for you, because I can't reproduce it.

Note that you can't change only one of the values on the commannd-line: you must make the "-m" argument match the sum of numa node sizes, or the guest will ignore the NUMA tables from the system.

Comment 6 Qian Guo 2014-01-07 03:14:43 UTC

(In reply to Eduardo Habkost from comment #5)
> (In reply to Qian Guo from comment #4)
> > Tested this scenario for several times, found that only boot guest with 
> > -m 2G -smp 2,sockets=1,cores=2,threads=1
> > -numa node,cpus=0,nodeid=0,mem=1024 -numa node,cpus=1,nodeid=1,mem=1024
> > 
> > that the numa works in guest.
> > 
> > I tried that when change any of above values, it can reproduced (numa won't
> > work).
> > 
> 
> Please show the full command-line of at least one case where it doesn't work
> for you, because I can't reproduce it.
> 
> Note that you can't change only one of the values on the commannd-line: you
> must make the "-m" argument match the sum of numa node sizes, or the guest
> will ignore the NUMA tables from the system.
For comment #4, it means that only 2 nodes work. I tried with more than 2 nodes, can not work, following is the cli that not work:

# /usr/libexec/qemu-kvm -cpu SandyBridge -m 8G -smp 8 -M pc -enable-kvm  -name rhel651 -rtc base=localtime,clock=host,driftfix=slew -drive file=/home/rhel6u5/rhel6u5x64cp1.raw,if=none,format=raw,werror=stop,rerror=stop,media=disk,id=drive-scsi0-disk0 -device virtio-scsi-pci,id=scsi0,addr=0x4 -device scsi-hd,scsi-id=0,lun=0,bus=scsi0.0,drive=drive-scsi0-disk0,id=virtio-disk0 -nodefaults -nodefconfig -monitor stdio   -netdev tap,id=hostnet0,script=/etc/qemu-ifup,vhost=on -device virtio-net-pci,netdev=hostnet0,mac=54:43:1a:10:0b:02,id=vnet0 -spice port=5901,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -numa node -numa node -numa node -numa node -numa node -numa node -numa node

# /usr/libexec/qemu-kvm -cpu SandyBridge -m 4G -smp 4,sockets=1,cores=4,threads=1 -M pc -enable-kvm  -name rhel651 -rtc base=localtime,clock=host,driftfix=slew -drive file=/home/rhel6u5/rhel6u5x64cp1.raw,if=none,format=raw,werror=stop,rerror=stop,media=disk,id=drive-scsi0-disk0 -device virtio-scsi-pci,id=scsi0,addr=0x4 -device scsi-hd,scsi-id=0,lun=0,bus=scsi0.0,drive=drive-scsi0-disk0,id=virtio-disk0 -nodefaults -nodefconfig -monitor stdio   -netdev tap,id=hostnet0,script=/etc/qemu-ifup,vhost=on -device virtio-net-pci,netdev=hostnet0,mac=54:43:1a:10:0b:02,id=vnet0 -spice port=5901,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -numa node,mem=1024,cpus=0,nodeid=0 -numa node,mem=1024,cpus=1,nodeid=1 -numa node,mem=1024,cpus=2,nodeid=2 -numa node,mem=1024,cpus=3,nodeid=3


# /usr/libexec/qemu-kvm -cpu SandyBridge -m 4G -smp 4,sockets=1,cores=4,threads=1 -M pc -enable-kvm  -name rhel651 -rtc base=localtime,clock=host,driftfix=slew -drive file=/home/rhel6u5/rhel6u5x64cp1.raw,if=none,format=raw,werror=stop,rerror=stop,media=disk,id=drive-scsi0-disk0 -device virtio-scsi-pci,id=scsi0,addr=0x4 -device scsi-hd,scsi-id=0,lun=0,bus=scsi0.0,drive=drive-scsi0-disk0,id=virtio-disk0 -nodefaults -nodefconfig -monitor stdio   -netdev tap,id=hostnet0,script=/etc/qemu-ifup,vhost=on -device virtio-net-pci,netdev=hostnet0,mac=54:43:1a:10:0b:02,id=vnet0 -spice port=5901,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -numa node -numa node -numa node -numa node

Comment 7 Qian Guo 2014-01-07 03:32:23 UTC

(In reply to Qian Guo from comment #6)
> (In reply to Eduardo Habkost from comment #5)
> > (In reply to Qian Guo from comment #4)
> > > Tested this scenario for several times, found that only boot guest with 
> > > -m 2G -smp 2,sockets=1,cores=2,threads=1
> > > -numa node,cpus=0,nodeid=0,mem=1024 -numa node,cpus=1,nodeid=1,mem=1024
> > > 
> > > that the numa works in guest.
> > > 
> > > I tried that when change any of above values, it can reproduced (numa won't
> > > work).
> > > 
> > 
> > Please show the full command-line of at least one case where it doesn't work
> > for you, because I can't reproduce it.
> > 
> > Note that you can't change only one of the values on the commannd-line: you
> > must make the "-m" argument match the sum of numa node sizes, or the guest
> > will ignore the NUMA tables from the system.
> For comment #4, it means that only 2 nodes work. I tried with more than 2
> nodes, can not work, following is the cli that not work:
> 

> # /usr/libexec/qemu-kvm -cpu SandyBridge -m 4G -smp
> 4,sockets=1,cores=4,threads=1 -M pc -enable-kvm  -name rhel651 -rtc
> base=localtime,clock=host,driftfix=slew -drive
> file=/home/rhel6u5/rhel6u5x64cp1.raw,if=none,format=raw,werror=stop,
> rerror=stop,media=disk,id=drive-scsi0-disk0 -device
> virtio-scsi-pci,id=scsi0,addr=0x4 -device
> scsi-hd,scsi-id=0,lun=0,bus=scsi0.0,drive=drive-scsi0-disk0,id=virtio-disk0
> -nodefaults -nodefconfig -monitor stdio   -netdev
> tap,id=hostnet0,script=/etc/qemu-ifup,vhost=on -device
> virtio-net-pci,netdev=hostnet0,mac=54:43:1a:10:0b:02,id=vnet0 -spice
> port=5901,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864
> -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -numa
> node,mem=1024,cpus=0,nodeid=0 -numa node,mem=1024,cpus=1,nodeid=1 -numa
> node,mem=1024,cpus=2,nodeid=2 -numa node,mem=1024,cpus=3,nodeid=3
> 
> 
For this cli, in the rhel6.5 guest, the dmesg shows following messages:
...
Setting APIC routing to flat.
SRAT: PXM 0 -> APIC 0 -> Node 0
SRAT: PXM 1 -> APIC 1 -> Node 1
SRAT: PXM 2 -> APIC 2 -> Node 2
SRAT: PXM 3 -> APIC 3 -> Node 3
SRAT: Node 0 PXM 0 0-a0000
SRAT: Node 0 PXM 0 100000-40000000
SRAT: Node 1 PXM 1 40000000-80000000
SRAT: Node 2 PXM 2 80000000-c0000000
SRAT: Node 3 PXM 3 c0000000-100000000
SRAT: PXMs only cover 3583MB of your 4095MB e820 RAM. Not used.
SRAT: SRAT not used.
No NUMA configuration found
Faking a node at 0000000000000000-0000000120000000
...

I will attach the whole dmesg file in this bug.

Comment 8 Qian Guo 2014-01-07 03:33:34 UTC

Created attachment 846430 [details]
dmesg when boot with 4 nodes and 4 cpus and 4G mem

Comment 9 Eduardo Habkost 2014-01-07 20:34:05 UTC

Regression was introduced by the patches for bug 1034876. It is reproducible on latest qemu.git master upstream, too.

Comment 10 Eduardo Habkost 2014-01-14 14:01:02 UTC

Fix submitted upstream at: http://article.gmane.org/gmane.comp.emulators.qemu/250095

Comment 11 Eduardo Habkost 2014-01-17 19:48:05 UTC

*** Bug 1054416 has been marked as a duplicate of this bug. ***

Comment 13 Miroslav Rezanina 2014-02-12 12:00:51 UTC

Fix included in qemu-kvm-1.5.3-47.el7

Comment 15 Sibiao Luo 2014-02-18 05:37:33 UTC

###### Reproduce this issue with the same steps as comment #0 on qemu-kvm-1.5.3-30.el7.x86_64.
host info:
# uname -r && rpm -q qemu-kvm
3.10.0-88.el7.x86_64
qemu-kvm-1.5.3-30.el7.x86_64
guest info:
# uname -r
3.10.0-88.el7.x86_64

Steps and Results:
# /usr/libexec/qemu-kvm -M pc -cpu host -m 4096 -smp 4,sockets=1,cores=4,threads=1 -enable-kvm -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo -uuid 990ea161-6b67-47b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -drive file=/home/RHEL_7.0_20140212.1_Server_x86_64.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,discard=on -device virtio-scsi-pci,id=scsi1,bus=pci.0,addr=0x4 -device scsi-hd,drive=drive-system-disk,id=system-disk,bootindex=1 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:01:02:B6:40:21,bus=pci.0,addr=0x5 -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :1 -spice disable-ticketing,port=5931 -monitor stdio -numa node,mem=1024,cpus=0,nodeid=0 -numa node,mem=1024,cpus=1,nodeid=1 -numa node,mem=1024,cpus=2,nodeid=2 -numa node,mem=1024,cpus=3,nodeid=3
(qemu) info numa
4 nodes
node 0 cpus: 0
node 0 size: 1024 MB
node 1 cpus: 1
node 1 size: 1024 MB
node 2 cpus: 2
node 2 size: 1024 MB
node 3 cpus: 3
node 3 size: 1024 MB
(qemu) 
# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3
node 0 size: 4095 MB
node 0 free: 3281 MB
node distances:
node   0 
  0:  10

###### Verified this issue with the qemu-kvm-1.5.3-47.el7.x86_64 version.
host info:
# uname -r && rpm -q qemu-kvm
3.10.0-88.el7.x86_64
qemu-kvm-1.5.3-47.el7.x86_64
guest info:
# uname -r
3.10.0-88.el7.x86_64

Steps and Results:

(qemu) info numa
4 nodes
node 0 cpus: 0
node 0 size: 1024 MB
node 1 cpus: 1
node 1 size: 1024 MB
node 2 cpus: 2
node 2 size: 1024 MB
node 3 cpus: 3
node 3 size: 1024 MB
(qemu)

guest ] # numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0
node 0 size: 1023 MB
node 0 free: 684 MB
node 1 cpus: 1
node 1 size: 1024 MB
node 1 free: 881 MB
node 2 cpus: 2
node 2 size: 1023 MB
node 2 free: 828 MB
node 3 cpus: 3
node 3 size: 1024 MB
node 3 free: 867 MB
node distances:
node   0   1   2   3 
  0:  10  20  20  20 
  1:  20  10  20  20 
  2:  20  20  10  20 
  3:  20  20  20  10

Base on above, this issue has been fixed correctly, move to VERIFIED status.

Best Regards,
sluo

Comment 16 Axel Thimm 2014-02-26 18:15:43 UTC

I see a similar issue with a 6.5 host running 6.5 guests:
/usr/libexec/qemu-kvm [...] -smp 24,sockets=2,cores=6,threads=2 -numa node,nodeid=0,cpus=0-5,12-17,mem=3072 -numa node,nodeid=1,cpus=6-11,18-23,mem=3072 [...]
and lscpu/numactl -H on the guest gives:
node 0 cpus: 0 1 2 3 4 5 12 13 14 15 16 17 18 19 20 21 22 23
node 0 size: 3071 MB
node 0 free: 2575 MB
node 1 cpus: 6 7 8 9 10 11
node 1 size: 3071 MB
node 1 free: 2965 MB

Is this the same bug? If so, is there a fix underway for RHEL6.5 as well? Should I open a separate report?

Comment 17 Eduardo Habkost 2014-02-26 18:38:47 UTC

(In reply to Axel Thimm from comment #16)
> I see a similar issue with a 6.5 host running 6.5 guests:
> /usr/libexec/qemu-kvm [...] -smp 24,sockets=2,cores=6,threads=2 -numa
> node,nodeid=0,cpus=0-5,12-17,mem=3072 -numa
> node,nodeid=1,cpus=6-11,18-23,mem=3072 [...]

The above command-line is invalid. Multiple CPU ranges can't be specified this way.

Comment 18 Axel Thimm 2014-02-27 10:25:05 UTC

Thank you Eduardo, then my issue is off topic wrt this bug.
Still, if anyone has been googled to this report: It looks like the correct syntax would be something like

cpus=A-B,cpus=C-D

which is something libvirt cannot map in RHEL6.5. Since most hyperthreaded hardware numbers nodes in this particular way of non-contiguous sets, this is something you stumble over often, if you map topology one to one. I'll probably remap the cpus with vcpupin, so that the resulting set is contiguous.

Comment 19 Ludek Smid 2014-06-13 11:42:07 UTC

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.