Bug 1472167 - Guest OS does not show any NUMA nodes
Guest OS does not show any NUMA nodes
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
7.4
x86_64 Linux
unspecified Severity high
: rc
: ---
Assigned To: Eduardo Habkost
Yumei Huang
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-18 04:24 EDT by Artyom
Modified: 2017-08-26 16:30 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-26 16:30:18 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Artyom 2017-07-18 04:24:33 EDT
Description of problem:
I run a virtual machine in the RHEVM with next QEMU parameters:
/usr/libexec/qemu-kvm 
-name guest=golden_env_mixed_virtio_0,debug-threads=on 
-S 
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-389-golden_env_mixed_vir/master-key.aes 
-machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off 
-cpu Conroe 
-m size=1048576k,slots=16,maxmem=4194304k 
-realtime mlock=off 
-smp 2,maxcpus=16,sockets=16,cores=1,threads=1 
-numa node,nodeid=0,cpus=0,mem=512 
-numa node,nodeid=1,cpus=1,mem=512 
-uuid fcbe94d1-7cea-4170-852e-71627f3b1d30 
-smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7.4-18.el7,serial=55157E90-C28C-11E0-983A-5CF3FC78F458,uuid=fcbe94d1-7cea-4170-852e-71627f3b1d30 
....
but when I start the VM and run numactl command it shows me only one NUMA node
# numactl -H
available: 1 nodes (0)
node 0 cpus: 0
node 0 size: 1023 MB
node 0 free: 674 MB
node distances:
node   0 
  0:  10 


Version-Release number of selected component (if applicable):
VM packages:
# uname -r
3.10.0-691.el7.x86_64
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 (Maipo)
# rpm -qa | grep numa
numactl-libs-2.0.9-6.el7_2.x86_64
numactl-2.0.9-6.el7_2.x86_64

Host packages:
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 (Maipo)
# uname -r
3.10.0-691.el7.x86_64
# rpm -qa | grep qemu
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7.noarch
qemu-kvm-ev-2.6.0-28.el7.10.1.x86_64
qemu-img-ev-2.6.0-28.el7.10.1.x86_64
qemu-kvm-common-ev-2.6.0-28.el7.10.1.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Create VM with above parameters
2. Start the VM
3. Check VM NUMA nodes

Actual results:
numactl -H command shows only one NUMA node

Expected results:
numactl -H command must show two NUMA nodes

Additional info:
I checked VM with the same QEMU parameters, but with different kernel and all works as expected
Working configuration:
# uname -r
3.10.0-514.10.2.el7.x86_64
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.3 (Maipo)
# rpm -qa | grep numa
numactl-libs-2.0.9-6.el7_2.x86_64
numactl-2.0.9-6.el7_2.x86_64
============================================================================
Also I checked dmesg of problematic VM and I can see:
[    0.000000] found SMP MP-table at [mem 0x000f71a0-0x000f71af] mapped at [ffff8800000f71a0]
[    0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576
[    0.000000] BRK [0x01fe9000, 0x01fe9fff] PGTABLE
[    0.000000] BRK [0x01fea000, 0x01feafff] PGTABLE
[    0.000000] BRK [0x01feb000, 0x01febfff] PGTABLE
[    0.000000] BRK [0x01fec000, 0x01fecfff] PGTABLE
[    0.000000] RAMDISK: [mem 0x35848000-0x36c1bfff]
[    0.000000] Early table checksum verification disabled
[    0.000000] ACPI BIOS Error (bug): A valid RSDP was not found (20130517/tbxfroot-243)
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000003fffbfff]
[    0.000000] NODE_DATA(0) allocated [mem 0x3ffd5000-0x3fffbfff]
[    0.000000] crashkernel=auto resulted in zero bytes of reserved memory.
[    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[    0.000000] kvm-clock: cpu 0, msr 0:3ff85001, primary cpu clock
[    0.000000] kvm-clock: using sched offset of 7024018732 cycles
============================================================================
Both kernels have the same NUMA configuration:
# cat config-3.10.0-691.el7.x86_64 | grep -i numa
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANTS_PROT_NUMA_PROT_NONE=y
CONFIG_ARCH_USES_NUMA_PROT_NONE=y
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
CONFIG_NUMA_BALANCING=y
# CONFIG_X86_NUMACHIP is not set
CONFIG_NUMA=y
CONFIG_AMD_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
# CONFIG_NUMA_EMU is not set
CONFIG_USE_PERCPU_NUMA_NODE_ID=y
CONFIG_ACPI_NUMA=y

# cat /boot/config-3.10.0-514.10.2.el7.x86_64 | grep -i numa
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANTS_PROT_NUMA_PROT_NONE=y
CONFIG_ARCH_USES_NUMA_PROT_NONE=y
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
CONFIG_NUMA_BALANCING=y
# CONFIG_X86_NUMACHIP is not set
CONFIG_NUMA=y
CONFIG_AMD_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
# CONFIG_NUMA_EMU is not set
CONFIG_USE_PERCPU_NUMA_NODE_ID=y
CONFIG_ACPI_NUMA=y

I am pretty sure it does not numactl bug, but I believe you will know to do propogation, because I just do not sure what component is problematic.
Comment 2 Yumei Huang 2017-07-31 04:02:00 EDT
Hi Artyom, 
I tried and failed to reproduce this bug. Could you please help check my steps? Thanks!

Test details:
host kernel: 3.10.0-691.el7.x86_64
guest kernel: 3.10.0-691.el7.x86_64
numactl-2.0.9-6.el7_2.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.2

CMDline:
# /usr/libexec/qemu-kvm   rhel73-bak.qcow2  \

-m size=1048576k,slots=16,maxmem=4194304k  \

-smp 2,maxcpus=16,sockets=16,cores=1,threads=1 \

-numa node,nodeid=0,cpus=0,mem=512  \

-numa node,nodeid=1,cpus=1,mem=512  \

-netdev tap,id=tap0 -device virtio-net-pci,id=net0,netdev=tap0 \

-monitor stdio -vnc :1

Check numa in guest:
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0
node 0 size: 511 MB
node 0 free: 185 MB
node 1 cpus: 1
node 1 size: 511 MB
node 1 free: 38 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10
Comment 3 Artyom 2017-07-31 04:34:32 EDT
It looks fine, maybe the single difference that I have QEMU packages with ev suffix:
# rpm -qa | grep qemu
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7.noarch
qemu-kvm-ev-2.6.0-28.el7.10.1.x86_64
qemu-img-ev-2.6.0-28.el7.10.1.x86_64
qemu-kvm-common-ev-2.6.0-28.el7.10.1.x86_64

I do not sure if it really relevant.

I sent you my environment details via email, so maybe you will recognize some additional difference.

Sorry not much help from me:(
Comment 4 Eduardo Habkost 2017-07-31 10:00:43 EDT
(In reply to Artyom from comment #3)
> It looks fine, maybe the single difference that I have QEMU packages with ev
> suffix:
> # rpm -qa | grep qemu
> libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
> ipxe-roms-qemu-20170123-1.git4e85b27.el7.noarch
> qemu-kvm-ev-2.6.0-28.el7.10.1.x86_64
> qemu-img-ev-2.6.0-28.el7.10.1.x86_64
> qemu-kvm-common-ev-2.6.0-28.el7.10.1.x86_64
> 
> I do not sure if it really relevant.

It is, as the ACPI tables are generated by QEMU and the guest is reporting broken ACPI tables.  Are you able reproduce using the equivalent qemu-kvm-rhev packages?
Comment 5 Artyom 2017-07-31 10:10:05 EDT
I believe qemu-kvm-ev coming as part of vdsm dependencies, so it can be problematic for me to change from ev to rhev packages, also when I tried to install it with ev packages already installed I got warning message
Package 10:qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64 is obsoleted by 10:qemu-kvm-ev-2.6.0-28.el7.10.1.x86_64 which is already installed, so if it obsoleted why do we need to check it?
Comment 6 Eduardo Habkost 2017-07-31 10:22:00 EDT
(In reply to Artyom from comment #5)
> I believe qemu-kvm-ev coming as part of vdsm dependencies, so it can be
> problematic for me to change from ev to rhev packages, also when I tried to
> install it with ev packages already installed I got warning message
> Package 10:qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64 is obsoleted by
> 10:qemu-kvm-ev-2.6.0-28.el7.10.1.x86_64 which is already installed, so if it
> obsoleted why do we need to check it?

Because this is a RHEL bug report, and qemu-kvm-ev is a CentOS package.  :)
Comment 7 Artyom 2017-07-31 11:17:53 EDT
Oh thanks for the explanation, it always a lot of problems with packages when we work upstream.
Ok I downloaded rhev packages:
# rpm -qa | grep qemu
qemu-kvm-common-rhev-2.6.0-28.el7_3.10.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7.noarch
qemu-kvm-tools-rhev-2.6.0-28.el7_3.10.x86_64
qemu-img-rhev-2.6.0-28.el7_3.10.x86_64
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.10.x86_64

but still, I can see only one NUMA node and I can see ACPI message
[    0.000000] Early table checksum verification disabled
[    0.000000] ACPI BIOS Error (bug): A valid RSDP was not found (20130517/tbxfroot-243)
Comment 8 Eduardo Habkost 2017-08-18 19:00:56 EDT
I'm not being able to reproduce it, either.  Can I get access to the system where the problem is happening?
Comment 9 Artyom 2017-08-26 16:30:18 EDT
I can not reproduce it on the oVirt the last build, so I will close the bug and reopen it in the case if I will see the problem again.

Note You need to log in before you can comment on or make changes to this bug.