Bug 1173218

Can you provide the /var/log/libvirt/qemu/$GUEST.log file showing the full error, and also the /proc/meminfo file fromthe host

The original logs have been wiped through snapshot restoration unfortunately since the time I removed the hugepages setting from vm configuration.

I just did a quick test on F21, I could reproduce it with these versions:

  libvirt-daemon-kvm-1.2.9.1.1.fc21.x86_64
  qemu-system-86-2.1.2-6.fc21.x86_64

With hugepages configured fo 512, and with this snippet:

<memoryBacking>
  <hugepages>
    <page size='2048' unit='KiB' nodeset='0'>
  </hugepages>
</memoryBacking>


$ cat /proc/meminfo 
MemTotal:       12182016 kB
MemFree:         8632352 kB
MemAvailable:    9700128 kB
Buffers:          154424 kB
Cached:          1284588 kB
SwapCached:            0 kB
Active:          1510252 kB
Inactive:         640432 kB
Active(anon):     718272 kB
Inactive(anon):   259792 kB
Active(file):     791980 kB
Inactive(file):   380640 kB
Unevictable:        6376 kB
Mlocked:            6376 kB
SwapTotal:       3905532 kB
SwapFree:        3905532 kB
Dirty:              1320 kB
Writeback:             0 kB
AnonPages:        718044 kB
Mapped:           253216 kB
Shmem:            261696 kB
Slab:             180780 kB
SReclaimable:     128196 kB
SUnreclaim:        52584 kB
KernelStack:        7232 kB
PageTables:        24836 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     9472252 kB
Committed_AS:    3075288 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      394832 kB
VmallocChunk:   34359278212 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:     512
HugePages_Free:      512
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      122964 kB
DirectMap2M:     2899968 kB
DirectMap1G:     9437184 kB




From /var/log/libvirt/qemu/f21-vm.log:

. . .
2014-12-12 13:35:57.389+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name f21-vm -S -machine pc-i440fx-2.1,accel=kvm,usb=off -m 8000 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 3dc5a06d-ab64-46b5-9379-f5cb7c8971b5 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/f21-vm.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x3.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x3 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x3.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x3.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/lib/libvirt/images/f21vm.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:14:91:01,bus=pci.0,addr=0x2 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/f21-vm.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
2014-12-12 13:35:57.391+0000: 18134: debug : virFileClose:99 : Closed fd 22
2014-12-12 13:35:57.391+0000: 18134: debug : virFileClose:99 : Closed fd 29
2014-12-12 13:35:57.391+0000: 18135: debug : virExec:692 : Run hook 0x7ff79d5f3850 0x7ff7a61070d0
2014-12-12 13:35:57.392+0000: 18135: debug : qemuProcessHook:2904 : Obtaining domain lock
2014-12-12 13:35:57.392+0000: 18135: debug : virSecuritySELinuxSetSecuritySocketLabel:2156 : Setting VM f21-vm socket context system_u:system_r:svirt_t:s0:c372,c838
2014-12-12 13:35:57.392+0000: 18134: debug : virFileClose:99 : Closed fd 3
2014-12-12 13:35:57.392+0000: 18135: debug : virDomainLockProcessStart:178 : plugin=0x7ff7940ee6f0 dom=0x7ff794191c50 paused=1 fd=0x7ff7a6106cd4
2014-12-12 13:35:57.392+0000: 18135: debug : virDomainLockManagerNew:133 : plugin=0x7ff7940ee6f0 dom=0x7ff794191c50 withResources=1
2014-12-12 13:35:57.392+0000: 18135: debug : virLockManagerPluginGetDriver:281 : plugin=0x7ff7940ee6f0
2014-12-12 13:35:57.392+0000: 18135: debug : virLockManagerNew:305 : driver=0x7ff7b735c500 type=0 nparams=5 params=0x7ff7a6106bc0 flags=0
2014-12-12 13:35:57.392+0000: 18135: debug : virLockManagerLogParams:98 :   key=uuid type=uuid value=3dc5a06d-ab64-46b5-9379-f5cb7c8971b5
2014-12-12 13:35:57.392+0000: 18135: debug : virLockManagerLogParams:91 :   key=name type=string value=f21-vm
2014-12-12 13:35:57.392+0000: 18135: debug : virLockManagerLogParams:79 :   key=id type=uint value=6
2014-12-12 13:35:57.392+0000: 18135: debug : virLockManagerLogParams:79 :   key=pid type=uint value=18135
2014-12-12 13:35:57.392+0000: 18135: debug : virLockManagerLogParams:94 :   key=uri type=cstring value=qemu:///system
2014-12-12 13:35:57.392+0000: 18135: debug : virDomainLockManagerNew:145 : Adding leases
2014-12-12 13:35:57.392+0000: 18135: debug : virDomainLockManagerNew:150 : Adding disks
2014-12-12 13:35:57.392+0000: 18135: debug : virDomainLockManagerAddImage:90 : Add disk /var/lib/libvirt/images/f21vm.qcow2
2014-12-12 13:35:57.392+0000: 18135: debug : virLockManagerAddResource:332 : lock=0x7ff78c011210 type=0 name=/var/lib/libvirt/images/f21vm.qcow2 nparams=0 params=(nil) flags=0
2014-12-12 13:35:57.392+0000: 18135: debug : virLockManagerAcquire:350 : lock=0x7ff78c011210 state='<null>' flags=3 action=0 fd=0x7ff7a6106cd4
2014-12-12 13:35:57.392+0000: 18135: debug : virLockManagerFree:387 : lock=0x7ff78c011210
2014-12-12 13:35:57.392+0000: 18135: debug : virObjectUnref:259 : OBJECT_UNREF: obj=0x7ff794155540
2014-12-12 13:35:57.392+0000: 18135: debug : qemuProcessHook:2931 : Hook complete ret=0
2014-12-12 13:35:57.392+0000: 18135: debug : virExec:694 : Done hook 0
2014-12-12 13:35:57.392+0000: 18135: debug : virExec:701 : Setting child security label to system_u:system_r:svirt_t:s0:c372,c838
2014-12-12 13:35:57.392+0000: 18135: debug : virExec:731 : Setting child uid:gid to 107:107 with caps 0
2014-12-12 13:35:57.392+0000: 18135: debug : virCommandHandshakeChild:432 : Notifying parent for handshake start on 26
2014-12-12 13:35:57.392+0000: 18135: debug : virCommandHandshakeChild:440 : Waiting on parent for handshake complete on 27
2014-12-12 13:35:57.465+0000: 18135: debug : virFileClose:99 : Closed fd 26
2014-12-12 13:35:57.465+0000: 18135: debug : virFileClose:99 : Closed fd 27
2014-12-12 13:35:57.465+0000: 18135: debug : virCommandHandshakeChild:460 : Handshake with parent is done
char device redirected to /dev/pts/2 (label charserial0)
2014-12-12 13:35:57.675+0000: shutting down

From libvirtd debug logs:

[. . .]
2014-12-13 10:18:16.758+0000: 1199: error : qemuMonitorIORead:554 : Unable to read from monitor: Connection reset by peer
2014-12-13 10:18:16.758+0000: 1199: debug : virFileClose:99 : Closed fd 25
2014-12-13 10:18:16.758+0000: 1199: debug : qemuMonitorIO:717 : Error on monitor Unable to read from monitor: Connection reset by peer
2014-12-13 10:18:16.758+0000: 1199: debug : virEventPollUpdateHandle:152 : EVENT_POLL_UPDATE_HANDLE: watch=13 events=12
2014-12-13 10:18:16.758+0000: 1199: debug : virEventPollInterruptLocked:725 : Skip interrupt, 1 139667026139264
2014-12-13 10:18:16.758+0000: 1199: debug : qemuMonitorIO:748 : Triggering error callback
2014-12-13 10:18:16.758+0000: 1199: debug : qemuProcessHandleMonitorError:364 : Received error on 0x7f06a01e9240 'f21-vm'
2014-12-13 10:18:16.758+0000: 1224: debug : qemuMonitorSend:984 : Send command resulted in error Unable to read from monitor: Connection reset by peer
2014-12-13 10:18:16.758+0000: 1199: debug : virObjectNew:202 : OBJECT_NEW: obj=0x7f06c547fa80 classname=virDomainEvent
[. . .]
2014-12-13 10:18:16.960+0000: 1199: error : virDBusCall:1537 : error from service: TerminateMachine: No machine 'qemu-f21-vm' known
2014-12-13 10:18:16.960+0000: 1199: debug : qemuRemoveCgroup:1222 : Failed to terminate cgroup for f21-vm
[. . .]

Hmm, testing with a simple QEMU CLI seems to work here.

Test: Boot a 2 GB guest with 'nr_hugepages' as 1024


`meminfo` *before* booting the guest 
------------------------------------

$ grep -i Huge /proc/meminfo 
AnonHugePages:         0 kB
HugePages_Total:    1054
HugePages_Free:     1054
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB


Boot a 2GB memory guest with huge pages
---------------------------------------

#!/bin/bash
set -x

qemu-system-x86_64 \
  -m 2048 \
  -mem-path /dev/hugepages/libvirt/qemu \
  -nographic \
  -nodefconfig \
  -nodefaults \
  -machine accel=kvm \
  -cpu host \
  -device virtio-scsi-pci,id=scsi \
  -serial stdio \
  -drive file=/var/lib/libvirt/images/f21vm.qcow2,format=qcow2,cache=writeback

From the boot messages
----------------------

. . .
[    0.017006] Initializing cgroup subsys hugetlb
.
.
.
[    0.339517] HugeTLB registered 2 MB page size, pre-allocated 0 pages
.
.
.
         Mounting Huge Pages File System...
.
.
.
[  OK  ] Mounted Huge Pages File System
.
.
.
[  OK  ] Mounted Huge Pages File System.
. . .


`meminfo` *after* booting the guest 
------------------------------------

grep -i Huge /proc/meminfo 
AnonHugePages:         0 kB
HugePages_Total:    1054
HugePages_Free:      851
HugePages_Rsvd:      821
HugePages_Surp:        0
Hugepagesize:       2048 kB

(In reply to Kashyap Chamarthy from comment #3)
> I just did a quick test on F21, I could reproduce it with these versions:
> 
>   libvirt-daemon-kvm-1.2.9.1.1.fc21.x86_64
>   qemu-system-86-2.1.2-6.fc21.x86_64
> 
> With hugepages configured fo 512, and with this snippet:
> 
> <memoryBacking>
>   <hugepages>
>     <page size='2048' unit='KiB' nodeset='0'>
>   </hugepages>
> </memoryBacking>
> 
> 
> $ cat /proc/meminfo 
> MemTotal:       12182016 kB
> MemFree:         8632352 kB
> MemAvailable:    9700128 kB
> Buffers:          154424 kB
> Cached:          1284588 kB
> SwapCached:            0 kB
> Active:          1510252 kB
> Inactive:         640432 kB
> Active(anon):     718272 kB
> Inactive(anon):   259792 kB
> Active(file):     791980 kB
> Inactive(file):   380640 kB
> Unevictable:        6376 kB
> Mlocked:            6376 kB
> SwapTotal:       3905532 kB
> SwapFree:        3905532 kB
> Dirty:              1320 kB
> Writeback:             0 kB
> AnonPages:        718044 kB
> Mapped:           253216 kB
> Shmem:            261696 kB
> Slab:             180780 kB
> SReclaimable:     128196 kB
> SUnreclaim:        52584 kB
> KernelStack:        7232 kB
> PageTables:        24836 kB
> NFS_Unstable:          0 kB
> Bounce:                0 kB
> WritebackTmp:          0 kB
> CommitLimit:     9472252 kB
> Committed_AS:    3075288 kB
> VmallocTotal:   34359738367 kB
> VmallocUsed:      394832 kB
> VmallocChunk:   34359278212 kB
> HardwareCorrupted:     0 kB
> AnonHugePages:         0 kB
> HugePages_Total:     512
> HugePages_Free:      512

So you have 512 2M pages, which is 1G...

> HugePages_Rsvd:        0
> HugePages_Surp:        0
> Hugepagesize:       2048 kB
> DirectMap4k:      122964 kB
> DirectMap2M:     2899968 kB
> DirectMap1G:     9437184 kB
> 
> 
> 
> 
> From /var/log/libvirt/qemu/f21-vm.log:
> 
> . . .
> 2014-12-12 13:35:57.389+0000: starting up
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
> QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name f21-vm -S -machine
> pc-i440fx-2.1,accel=kvm,usb=off -m 8000 -mem-prealloc

But here you say to allocate ~8G and preallocate all the guest memory. And since the preallocation fails, qemu dies and libvirt refuses to start the domain. So this is practically misconfiguration. However, I admit that the qemu error message is rather misleading than helpful. I'm switching component to qemu so that they can fix it.

You're right, Michal. It was a mis-configuration indeed. Correct test with libvirt below.

Tested with current libvirt git)

    $ git log | head -2
    commit 8c7320055c1341615b4cf7265b88e5574a5dedc4
    Author: Michal Privoznik <mprivozn>

    $ git describe
    v1.2.11-rc2-8-g8c73200


Test: 2 MB (2048 kB) page size with 1024 total huge pages.


(1) Check hugepages info from the proc filesystem:

$ grep -i Huge /proc/meminfo 
AnonHugePages:         0 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB


(2) Start the libvirt guest and check if hugepages configuration is correctly reflected in the guest XML configuration:

$ virsh start f21-vm

$ virsh dumpxml f21-vm | grep hugepages -B3 -A2 
  <memory unit='KiB'>2000896</memory>
  <currentMemory unit='KiB'>2000000</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>


(3) QEMU invocation of the libvirt guest:

$ cat /var/log/libvirt/qemu/f21-vm.log
2014-12-13 15:46:00.201+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name f21-vm -S -machine pc-i440fx-2.1,accel=kvm,usb=off -m 1954 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 3dc5a06d-ab64-46b5-9379-f5cb7c8971b5 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/f21-vm.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x3.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x3 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x3.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x3.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/lib/libvirt/images/f21vm.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:14:91:01,bus=pci.0,addr=0x2 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/f21-vm.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
. . .


(4) Check hugepages info again from the proc filesystem:

grep -i Huge /proc/meminfo 
AnonHugePages:         0 kB
HugePages_Total:    1024
HugePages_Free:       47
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Snaper, from my test in comment #7, huge pages seems to work on Fedora 21 if  configured correctly. Can you please test it in your environment and please update here if you're still able to reproduce the issue.

Kashyap I replicated your configuration but I'm still seeing the error. It could be I need different parameters for it to work.

This time I have the information that was requested by Dan. The qemu log is a excerpt showing the relevant events when the vm wouldn't start up. I minimized timestamp information for privacy.


Error:
Error starting domain: Unable to read from monitor: Connection reset by peer

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1388, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 999, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: Unable to read from monitor: Connection reset by peer
 


meminfo:
MemTotal:        3476096 kB
MemFree:          676948 kB
MemAvailable:    1464064 kB
Buffers:           48332 kB
Cached:           896656 kB
SwapCached:         2888 kB
Active:          1232920 kB
Inactive:        1338560 kB
Active(anon):     816200 kB
Inactive(anon):   832428 kB
Active(file):     416720 kB
Inactive(file):   506132 kB
Unevictable:          32 kB
Mlocked:              32 kB
SwapTotal:       3538940 kB
SwapFree:        3529344 kB
Dirty:               168 kB
Writeback:             0 kB
AnonPages:       1625188 kB
Mapped:           257696 kB
Shmem:             22140 kB
Slab:             114124 kB
SReclaimable:      66396 kB
SUnreclaim:        47728 kB
KernelStack:        4720 kB
PageTables:        35368 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     5276988 kB
Committed_AS:    3849328 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      110912 kB
VmallocChunk:   34359599964 kB
HardwareCorrupted:     0 kB
AnonHugePages:    854016 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      210684 kB
DirectMap2M:     3403776 kB
DirectMap1G:           0 kB


guest.log:
2014-12: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -name Whonix-Gateway -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu qemu64,-kvmclock,+kvm_pv_eoi -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 05a76e6b-343a-4cbd-8cab-ac7fd0978d3f -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Whonix-Gateway.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/Whonix-Gateway.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=24,id=hostnet0,vhost=on,
vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a6:f0:73,bus=pci.0,addr=0x3 -netdev tap,fd=26,id=hostnet1,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:6c:53:e1,bus=pci.0,addr=0x4 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,disable-copy-paste,disable-agent-file-xfer,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x5 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -object rng-random,id=rng0,filename=/dev/random -device virtio-rng-pci,rng=rng0,bus=pci.0,addr=0x9 -msg timestamp=on
qemu: terminating on signal 15 from pid 1100
2014-12: shutting down
2014-12: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -name Whonix-Gateway -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu qemu64,-kvmclock,+kvm_pv_eoi -m 512 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 05a76e6b-343a-4cbd-8cab-ac7fd0978d3f -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Whonix-Gateway.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/Whonix-Gateway.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,
bootindex=1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a6:f0:73,bus=pci.0,addr=0x3 -netdev tap,fd=27,id=hostnet1,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:6c:53:e1,bus=pci.0,addr=0x4 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,disable-copy-paste,disable-agent-file-xfer,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x5 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -object rng-random,id=rng0,filename=/dev/random -device virtio-rng-pci,rng=rng0,bus=pci.0,addr=0x9 -msg timestamp=on
2014-12: shutting down
2014-12: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -name Whonix-Gateway -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu qemu64,-kvmclock,+kvm_pv_eoi -m 512 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 05a76e6b-343a-4cbd-8cab-ac7fd0978d3f -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Whonix-Gateway.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/Whonix-Gateway.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,
bootindex=1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a6:f0:73,bus=pci.0,addr=0x3 -netdev tap,fd=27,id=hostnet1,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:6c:53:e1,bus=pci.0,addr=0x4 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,disable-copy-paste,disable-agent-file-xfer,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x5 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -object rng-random,id=rng0,filename=/dev/random -device virtio-rng-pci,rng=rng0,bus=pci.0,addr=0x9 -msg timestamp=on
2014-12: shutting down

(In reply to snaper from comment #9)
> Kashyap I replicated your configuration but I'm still seeing the error. It
> could be I need different parameters for it to work.

Seems like you didn't exactly replicate my configuration -- you missed to configure Huge pages on the host, refer below.

[. . .]

> Error starting domain: Unable to read from monitor: Connection reset by peer

[. . .]

> HardwareCorrupted:     0 kB
> AnonHugePages:    854016 kB
> HugePages_Total:       0

This tells us that, hugepages is *not* configured on the host. You'd need to activate the nr_hugepages setting in the proc filesystem:

  $ echo 1054 > /proc/sys/vm/nr_hugepages

  NOTE: To make the above value persistent, you'd need to set:

      $ echo "vm.nr_hugepages=1054" >> /etc/sysctl.conf

Then, `grep` for the HugePages_Total:

  $ grep -i HugePages_Total /proc/meminfo 
  HugePages_Total:    1054


Then, boot a libvirt virtual machine with 2 GB memory with appropriate XML setting as noted in comment #7. Please try this test and report results here.

Thank you. After applying the hugepages settings you provided, on the host, the vm boots normally with no errors and it works.

Were hugepages ever turned on by default on Fedora 20 or on Linux in general? or do they need to be done manually?

If no, is libvirt now failing the vm start because it makes hugepage availability a hard requirement while in the past it didn't?

(In reply to snaper from comment #11)
> Thank you. After applying the hugepages settings you provided, on the host,
> the vm boots normally with no errors and it works.
> 
> Were hugepages ever turned on by default on Fedora 20 or on Linux in
> general? or do they need to be done manually?
> 
> If no, is libvirt now failing the vm start because it makes hugepage
> availability a hard requirement while in the past it didn't?

No, libvirt hasn't changed. I mean, there's been a work in that area, but no behaviour change, or change of defaults. The reason why huge pages pool is not allocated by default at host startup is that linux kernel behaves differently to them. I mean, standard malloc() will get you regular system pages, while allocating a huge page boils down to a different syscall. Moreover, if you move some pages to the huge page pool, you'll effectively cut off you RAM - those pages can't be used by malloc(). That's why the pool isn't allocated by default.
So, you need to allocate the pool yourself (either as shown by Kashyap, or via 'virsh allocpages').

And yes, if you configure your guest to use huge pages, but you don't have any in your host, domain startup process will fail. Like if anything else configured was missing, e.g. a hostdev.

Just a little update, I've tried out the current qemu.git and I'm getting this error when trying to reproduce:

virsh # start dummy
error: Failed to start domain dummy
error: internal error: process exited while connecting to monitor: 2014-12-16T09:04:26.232577Z qemu-system-x86_64: unable to map backing store for hugepages: Cannot allocate memory

virsh # freepages --all --pagesize 2M
Node 0:
2048KiB: 0

Node 1:
2048KiB: 0

Node 2:
2048KiB: 0

Node 3:
2048KiB: 0


Digging into qemu code I've found this commit:

commit 7f56e740a68c9f4ccebf7ad7590e82fbb30ffc87
Author:     Paolo Bonzini <pbonzini>
AuthorDate: Wed May 14 17:43:20 2014 +0800
Commit:     Michael S. Tsirkin <mst>
CommitDate: Thu Jun 19 18:44:20 2014 +0300

    memory: add error propagation to file-based RAM allocation
    
    Right now, -mem-path will fall back to RAM-based allocation in some
    cases.  This should never happen with "-object memory-file", prepare
    the code by adding correct error propagation.
    
    Signed-off-by: Paolo Bonzini <pbonzini>
    Signed-off-by: Hu Tao <hutao.com>
    Acked-by: Michael S. Tsirkin <mst>
    Signed-off-by: Michael S. Tsirkin <mst>
    
    MST: drop \n at end of error messages


In addition, the commit message explains why it previously worked for you and now it does not. QEMU silently ignored huge pages setting and allocated regular system pages if no huge pages were present.

Thanks everyone who helped. 

I have another ticket for kvm shared folder errors that appeared with Fedora 21. I would appreciate if someone took a look. Its harder to figure out what is wrong with it because error messages are vague.

https://bugzilla.redhat.com/show_bug.cgi?id=1173224

I'm closing this one as the error message was fixed.