Bug 1472376
Summary: | qemu-kvm: warning: CPU(s) not present in any NUMA nodes | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [oVirt] ovirt-engine | Reporter: | Marko Vrgotic <m.vrgotic> | ||||||||
Component: | BLL.Virt | Assignee: | Michal Skrivanek <michal.skrivanek> | ||||||||
Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | meital avital <mavital> | ||||||||
Severity: | low | Docs Contact: | |||||||||
Priority: | unspecified | ||||||||||
Version: | 4.1.3.5 | CC: | bugs, m.vrgotic, tjelinek | ||||||||
Target Milestone: | --- | Flags: | m.vrgotic:
needinfo-
m.vrgotic: needinfo- |
||||||||
Target Release: | --- | ||||||||||
Hardware: | x86_64 | ||||||||||
OS: | Linux | ||||||||||
Whiteboard: | |||||||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
Doc Text: | Story Points: | --- | |||||||||
Clone Of: | Environment: |
Testing
|
|||||||||
Last Closed: | 2017-09-06 06:53:28 UTC | Type: | Bug | ||||||||
Regression: | --- | Mount Type: | --- | ||||||||
Documentation: | --- | CRM: | |||||||||
Verified Versions: | Category: | --- | |||||||||
oVirt Team: | Virt | RHEL 7.3 requirements from Atomic Host: | |||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||
Embargoed: | |||||||||||
Attachments: |
|
Description
Marko Vrgotic
2017-07-18 15:10:16 UTC
(In reply to Marko Vrgotic from comment #0) > Created attachment 1300492 [details] > Screen shot of the VM states, from oVirt admin portal > > Description of problem: > > To begin with general overview: > I am new to oVirt, so apologies up front if wrong component and/or team is > selected. > > Environment: Testing & Learning > current HW: Dell PE R210 with Virtualization enabled and following specs: > 1. Intel® Xeon® Processor X3430 : processor family Lynnfield (officially not > supported by oVirt) > 2. 16 GB of RAM > 3: currently one server is acting as ovirt-engine, one as ovirt-node and one > as ovirt-nfs server > > oVirt accepted selecting Nehalem as CPU family for the Cluster, but actuall > CPU family of the X3430 is Lynnfield. I am mentioning this since I am not > sure if this is direct connection to following problem. this should be ok > > > Problem description: > Failing to spawn new VM. The boot process start, but while tailing the log > of the VM I see following "WARNING" message: > > "2017-07-18T14:50:50.785490Z qemu-kvm: warning: CPU(s) not present in any > NUMA nodes: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 > 26 27 28 29 30 31 > 2017-07-18T14:50:50.785651Z qemu-kvm: warning: All CPU(s) up to maxcpus > should be described in NUMA config" this should not be the cause > > > Version-Release number of selected component (if applicable): > > OS: CentOS 7.3 > Kernel: 3.10.0-327.el7.x86_64 > oVirt version: 4.1.3.5 > > > How reproducible: > each time I try to start VM, regardless of selecting single/multiple CPU > sockets with single/multiple cores > > > > Actual results: > > The VM is reported UP, but CPU goes 100% and machine is stuck - no console > output. > > > Expected results: > > I would like to verify if this issue is caused by missmatch in CPU family or > potentially a bug or a setting I have missed. > > Additional info: > > full log: > "2017-07-18 14:50:50.689+0000: starting up libvirt version: 2.0.0, package: > 10.el7_3.9 (CentOS BuildSystem <http://bugs.centos.org>, > 2017-05-25-20:52:28, c1bm.rdu2.centos.org), qemu version: 2.6.0 > (qemu-kvm-ev-2.6.0-28.el7.10.1), hostname: ovirt-node-1.awesome.avinity.tv > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin > QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name > guest=test-jk-2,debug-threads=on -S -object > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-10-test-jk- > 2/master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off -cpu Nehalem > -m size=4194304k,slots=16,maxmem=16777216k -realtime mlock=off -smp > 2,maxcpus=32,sockets=16,cores=2,threads=1 -numa > node,nodeid=0,cpus=0-1,mem=4096 -uuid 068a0cb3-4450-4681-a5a2-4904b92c2b46 > -smbios 'type=1,manufacturer=oVirt,product=oVirt > Node,version=7-3.1611.el7.centos,serial=4C4C4544-0051-5310-8048-B7C04F56344A, > uuid=068a0cb3-4450-4681-a5a2-4904b92c2b46' -no-user-config -nodefaults > -device sga -chardev > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-10-test-jk-2/monitor. > sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc > base=2017-07-18T14:50:50,driftfix=slew -global > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot > menu=on,splash-time=10000,strict=on -device > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3 -device > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive > if=none,id=drive-ide0-1-0,readonly=on -device > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive > file=/rhev/data-center/2331bba5-4ee6-4782-9714-24578abbd60f/31ca115f-7d01- > 458a-a1aa-69096df9f336/images/05d68028-202e-4e48-9a37-2f938ffcc1ae/e7517ebf- > 1546-4c12-ae5e-669b319ab69b,format=raw,if=none,id=drive-virtio-disk0, > serial=05d68028-202e-4e48-9a37-2f938ffcc1ae,cache=none,werror=stop, > rerror=stop,aio=threads -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0, > id=virtio-disk0,bootindex=1 -netdev > tap,fd=31,id=hostnet0,vhost=on,vhostfd=33 -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:54,bus=pci.0, > addr=0x7,bootindex=2 -netdev tap,fd=34,id=hostnet1,vhost=on,vhostfd=35 > -device > virtio-net-pci,netdev=hostnet1,id=net1,mac=00:1a:4a:16:01:55,bus=pci.0, > addr=0x8,bootindex=3 -chardev > socket,id=charserial0,path=/var/run/ovirt-vmconsole-console/068a0cb3-4450- > 4681-a5a2-4904b92c2b46.sock,server,nowait -device > isa-serial,chardev=charserial0,id=serial0 -chardev > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/068a0cb3-4450- > 4681-a5a2-4904b92c2b46.com.redhat.rhevm.vdsm,server,nowait -device > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0, > name=com.redhat.rhevm.vdsm -chardev > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/068a0cb3-4450- > 4681-a5a2-4904b92c2b46.org.qemu.guest_agent.0,server,nowait -device > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1, > name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 > -vnc 172.16.107.244:1,password -k en-us -device > qxl-vga,id=video0,ram_size=67108864,vram_size=33554432,vram64_size_mb=0, > vgamem_mb=16,bus=pci.0,addr=0x2 -object > rng-random,id=objrng0,filename=/dev/urandom -device > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -msg timestamp=on > 2017-07-18T14:50:50.785490Z qemu-kvm: warning: CPU(s) not present in any > NUMA nodes: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 > 26 27 28 29 30 31 > 2017-07-18T14:50:50.785651Z qemu-kvm: warning: All CPU(s) up to maxcpus > should be described in NUMA config" - What is in the VM itself? Any operating system? If yes and try it without OS, will it behave? - if you try to connect to serial console of it, anything interesting there? (http://www.ovirt.org/develop/release-management/features/engine/serial-console/) - If you try to run it with as few devices as possible, does it start? (e.g. no disks, no nics, no sound card etc?) If yes, than it would be good to narrow down which device causes this. Thank you note the numa mapping is bug 1437559 indeed not relevant to your issue Dear Tomas, I will execute as suggested. As soon as I have the first results, I will update the ticket. -- Marko Ok, so far I have failed to boot the VM: disabling sound-card, is allowed disabling nic allowed disabling disk - it requires at least one disk so its not allowed In any of the cases above, same result, failed to boot. VM state is reported as UP. Using .VV to connect, I do connect but only black screen. Using virsh console, I receive different output, I think its based on OS version selected (Tired with Linux and Red Hat Enterprise Linux 7 x64 and with Red Hat Enterprise Linux 6): =========================== virsh # console 14 Connected to domain vmid2 Escape character is ^] error: internal error: cannot find character device <null> virsh # console 13 Connected to domain vmid1 Escape character is ^] error: internal error: character device console0 is not using a PTY =========================== Further actions: Disabled migration Disabled any spice connector Enabled only VirtIO console In both cases, this is the VM devices list: ============================================== unix {} unix {} qxl {vgamem=16384, heads=1, vram=8192, ram=65536} spice {} cdrom {path=} ide {} console {enableSocket=true, consoleType=serial} virtio-scsi {} spicevmc {} virtio-serial {} disk 1 {} usb {index=0, model=piix3-uhci} ============================================== Still same observed in the logs of the VMs: ============================================== 2017-07-19 09:18:13.367+0000: starting up libvirt version: 2.0.0, package: 10.el7_3.9 (CentOS BuildSystem <http://bugs.centos.org>, 2017-05-25-20:52:28, c1bm.rdu2.centos.org), qemu version: 2.6.0 (qemu-kvm-ev-2.6.0-28.el7.10.1), hostname: ovirt-node-1.awesome.avinity.tv LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name guest=vmid1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-13-vmid1/master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off -cpu Nehalem -m size=4194304k,slots=16,maxmem=16777216k -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -numa node,nodeid=0,cpus=0-1,mem=4096 -uuid 28d565e0-c866-4f7c-8cee-54d6bbf58535 -smbios 'type=1,manufacturer=oVirt,product=oVirt Node,version=7-3.1611.el7.centos,serial=4C4C4544-0051-5310-8048-B7C04F56344A,uuid=28d565e0-c866-4f7c-8cee-54d6bbf58535' -no-user-config -nodefaults -device sga -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-13-vmid1/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2017-07-19T09:18:13,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x3 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/2331bba5-4ee6-4782-9714-24578abbd60f/31ca115f-7d01-458a-a1aa-69096df9f336/images/87dd1159-5270-4eb9-8312-5b9be1cded60/de511eb1-3d27-4b36-b7eb-fcf5483147e6,format=raw,if=none,id=drive-virtio-disk0,serial=87dd1159-5270-4eb9-8312-5b9be1cded60,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev socket,id=charserial0,path=/var/run/ovirt-vmconsole-console/28d565e0-c866-4f7c-8cee-54d6bbf58535.sock,server,nowait -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/28d565e0-c866-4f7c-8cee-54d6bbf58535.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/28d565e0-c866-4f7c-8cee-54d6bbf58535.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice tls-port=5900,addr=172.16.107.244,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,disable-copy-paste,disable-agent-file-xfer,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=33554432,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg timestamp=on 2017-07-19T09:18:13.509968Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 2017-07-19T09:18:13.510153Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config main_channel_link: add main channel client main_channel_handle_parsed: net test: latency 5.996000 ms, bitrate 149806159 bps (142.866286 Mbps) inputs_connect: inputs channel client create red_dispatcher_set_cursor_peer: =================================== Not sure if following is at all helpful, but here is also the qemu-kvm ps output: ======================= /usr/libexec/qemu-kvm -name guest=vmid2,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-14-vmid2/master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off -cpu Nehalem -m size=4194304k,slots=16,maxmem=16777216k -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -numa node,nodeid=0,cpus=0-1,mem=4096 -uuid 7cc39f50-f54e-41e2-a352-e6213445adeb -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-3.1611.el7.centos,serial=4C4C4544-0051-5310-8048-B7C04F56344A,uuid=7cc39f50-f54e-41e2-a352-e6213445adeb -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-14-vmid2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2017-07-19T09:34:05,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/2331bba5-4ee6-4782-9714-24578abbd60f/31ca115f-7d01-458a-a1aa-69096df9f336/images/f8e4888f-078c-4d97-931f-fd58f5dddaf0/127ebfbe-850a-473a-8bc7-5d1b2aac75cc,format=raw,if=none,id=drive-scsi0-0-0-0,serial=f8e4888f-078c-4d97-931f-fd58f5dddaf0,cache=none,werror=stop,rerror=stop,aio=threads -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/7cc39f50-f54e-41e2-a352-e6213445adeb.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/7cc39f50-f54e-41e2-a352-e6213445adeb.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice tls-port=5901,addr=172.16.107.244,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=8388608,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg timestamp=on ======================== Do you think I should strip it down more? Also, how to do it, as I have disabled/removed anything that was allowed to be enabled/disabled. Any other suggestions or log files I could check? Still unable to connect to console. I very much appreciate your help here, as I am preparing here for setting up an oVirt environment for my company Operations department. NUMA seem not to be supported by DellR210, as it is disabled in the oVirt and here is not option to enable it. I will check if it is support&possible to enable it and how. Created attachment 1301007 [details]
remoteviewer console.vv connection output
Created attachment 1301020 [details]
list of currently enabled devices
Its still directly visible how to disable spice, virtio-scsi, but even with those ones removed, I am still reaching the same problem.
Any ideas? Dear Team, I do not yet know how the bugzilla interface works. I did sent some updates and require help or additional guidance to further troubleshoot the issue. Please assist. If I am not requesting help or flagging correct team, please assist. Kind regards, Marko (In reply to Marko Vrgotic from comment #4) > Ok, so far I have failed to boot the VM: > > disabling sound-card, is allowed > disabling nic allowed > disabling disk - it requires at least one disk so its not allowed you can boot without a disk - you just need to set the boot order to boot first from a network and than it will try to PXE boot. Or, you can boot from a CD. > > In any of the cases above, same result, failed to boot. > VM state is reported as UP. > Using .VV to connect, I do connect but only black screen. > Using virsh console, I receive different output, I think its based on OS > version selected (Tired with Linux and Red Hat Enterprise Linux 7 x64 and > with Red Hat Enterprise Linux 6): > =========================== > > > Do you think I should strip it down more? Also, how to do it, as I have > disabled/removed anything that was allowed to be enabled/disabled. > > Any other suggestions or log files I could check? you could: - in run once dialog you can start it in pause mode and enable boot menu to see if you can reach BIOS (e.g. in run once pick "start in pause mode" and "enable boot menu", than when it is paused, open SPICE/VNC console and un-pause the VM by clicking the play button) - if you think it is caused by CPU mismatch, you can try to set a custom one for this particular VM in edit VM -> system -> advanced options -> custom cpu type. It is a free text with type ahead but you can pretty much put anything there. - you could try to use the serial console to connect to the VM to see what is happening in there: http://www.ovirt.org/develop/release-management/features/engine/serial-console/ > > Still unable to connect to console. > > I very much appreciate your help here, as I am preparing here for setting up > an oVirt environment for my company Operations department. My sincere apologies. I was not able to follow up on the investigation, due to urgency of other matters. Can somebody advise me, if there is an option to set the ticket too in case the reporter (in this case Me) is not able to follow up or delaying the update? (In reply to Marko Vrgotic from comment #11) > My sincere apologies. > I was not able to follow up on the investigation, due to urgency of other > matters. > > Can somebody advise me, if there is an option to set the ticket too in case > the reporter (in this case Me) is not able to follow up or delaying the > update? keep the needinfo flag :) Do you think you will have time to followup in the near future? I will not be able to do it this week, but I am 100% sure that next week I will proceed with it. great, thank you, so keeping the needinfo until than to mark it that we are waiting for this information. Dear Tomas, Thank you for the patience and understanding. Please, give me few more days. I am still spending more time trying to "sell" my ovirt project to management, than actually doing fun stuff. I will try to reproduce the issue this week and come back with updates. Again, thank you for the patience. keep if on "needinfo" for a little while. Kind regards, Marko I will close this bug now so it will disappear from my radar, but please feel free to reopen when you will have some more data to move forward. Since version 4.2 and deploying on R610s - I have not seen this issue at all. Please close the ticket. |