Bug 1245864

(In reply to zhangpei from comment #0)
> Description of problem:
> In the guest, hotplug memory >= 13 times, then reboot, the qemu process
> dead.    
> 
> 
> 
> Version-Release number of selected component (if applicable):
> Host:
> Kernel:3.10.0-296.el7.x86_64
> qemu-kvm-rhev:qemu-kvm-rhev-2.3.0-12.el7.x86_64
> 
> Guest:
> kernel:3.10.0-295.el7.x86_64
> systemd:systemd-219-8.el7.x86_64
> 
> 
> 
> How reproducible:
> 100%
> 
> 
> 
> Steps to Reproduce:
> 1. start  a guest with qemu
> #</usr/libexec/qemu-kvm command> -m 1G,slots=256,maxmem=40G -numa node
> -monitor unix:/tmp/monitor4,server,nowait
> 
> 
> 2.hotplug 10M memory 13 times manually (also tried 15times, 20 times, 255
> times with scripts, the same results, all failed)
> (qemu)object_add memory-backend-ram,id=memX,size=10M     (X from 1 to 13)
> (qemu)device_add pc-dimm,id=dimmX,memdev=memX         
> 
> the script
> #!/bin/bash
> i=0 
> while [ $i -lt 256 ] 
> do 
> echo $i
> sleep 3 
> echo "object_add memory-backend-ram,id=mem$i,size=10M"|nc -U /tmp/monitor3 
> sleep 2
> echo "device_add pc-dimm,id=dimm$i,memdev=mem$i,slot=$i"|nc -U /tmp/monitor3 
> i=$(($i+1)) 
> done 
> 
> 
> 
> 3.reboot the guest
> in the guest
> #reboot
> 
> 
> 
> Actual results:
> 
> Host:
> [root@dhcp-12-153 ~]# /usr/libexec/qemu-kvm -name rhel7.2-4 -machine 
> pc-i440fx-rhel7.2.0,accel=kvm,usb=off -cpu SandyBridge -m
> 1G,slots=256,maxmem=40G -numa node -smp 4,sockets=2,cores=2,threads=1 -uuid
> 82b1a01e-5f6c-4f5f-8d28-3854a74e9b5b -drive
> file=/home/rhel7.2-4.qcow2,if=none,id=drive-virtio-blk-0,format=qcow2
> -device virtio-blk-pci,drive=drive-virtio-blk-0,bootindex=1,id=virtio-blk01
> -netdev tap,id=hostnet0 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=12:54:00:6c:89:9d -device
> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.
> 0 -spice
> port=5914,addr=0.0.0.0,disable-ticketing,image-compression=off,seamless-
> migration=on  -monitor stdio -serial unix:/tmp/monitor4,server,nowait
> QEMU 2.3.0 monitor - type 'help' for more information
> (qemu) main_channel_link: add main channel client
> main_channel_handle_parsed: net test: latency 0.732000 ms, bitrate
> 22021505376 bps (21001.344086 Mbps)
> inputs_connect: inputs channel client create
> red_dispatcher_set_cursor_peer: 
> 
> (qemu) object_add memory-backend-ram,id=mem1,size=10M
> (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
> (qemu) 
> (qemu) object_add memory-backend-ram,id=mem2,size=10M
> (qemu) device_add pc-dimm,id=dimm2,memdev=mem2
> (qemu) 
> (qemu) object_add memory-backend-ram,id=mem3,size=10M
> (qemu) device_add pc-dimm,id=dimm3,memdev=mem3
> (qemu) 
> (qemu) object_add memory-backend-ram,id=mem4,size=10M
> (qemu) device_add pc-dimm,id=dimm4,memdev=mem4
> (qemu) 
> (qemu) object_add memory-backend-ram,id=mem5,size=10M
> (qemu) device_add pc-dimm,id=dimm5,memdev=mem5
> (qemu) 
> (qemu) object_add memory-backend-ram,id=mem6,size=10M
> (qemu) device_add pc-dimm,id=dimm6,memdev=mem6
> (qemu) 
> (qemu) object_add memory-backend-ram,id=mem7,size=10M
> (qemu) device_add pc-dimm,id=dimm7,memdev=mem7
> (qemu) 
> (qemu) object_add memory-backend-ram,id=mem8,size=10M
> (qemu) device_add pc-dimm,id=dimm8,memdev=mem8
> (qemu) 
> (qemu) object_add memory-backend-ram,id=mem9,size=10M
> (qemu) device_add pc-dimm,id=dimm9,memdev=mem9
> (qemu) 
> (qemu) object_add memory-backend-ram,id=mem10,size=10M
> (qemu) device_add pc-dimm,id=dimm10,memdev=mem10
> (qemu) 
> (qemu) object_add memory-backend-ram,id=mem11,size=10M
> (qemu) device_add pc-dimm,id=dimm11,memdev=mem11
> (qemu) 
> (qemu) object_add memory-backend-ram,id=mem12,size=10M
> (qemu) device_add pc-dimm,id=dimm12,memdev=mem12
> (qemu) 
> (qemu) object_add memory-backend-ram,id=mem13,size=10M
> (qemu) device_add pc-dimm,id=dimm13,memdev=mem13
> (qemu) inputs_detach_tablet: 
> qemu-kvm: virtio: error trying to map MMIO memory
can you try with windows guest + virtio, pls?

> [root@dhcp-12-153 ~]# ps aux|grep qemu
> root     13531  0.0  0.0 112648   964 pts/0    S+   09:54   0:00 grep
> --color=auto qemu
> [root@dhcp-12-153 ~]#
> 
> 
> Guest:
> [root@dhcp-12-153 ~]# nc -U /tmp/monitor4
> [    0.484842] dracut-cmdline[108]: dracut-7.2 (Maipo) dracut-033-284.el7
> [    0.485920] dracut-cmdline[108]: Using kernel command line parameters:
> [    1.819767] intel_rapl: no valid rapl domains found in package 0
> 
> 
> Red Hat Enterprise Linux Server 7.2 (Maipo)
> Kernel 3.10.0-295.el7.x86_64 on an x86_64
> 
> dhcp-9-183 login: [   38.249631] acpi PNP0C80:01:
> acpi_memory_enable_device() error
> [   38.250762] acpi PNP0C80:03: acpi_memory_enable_device() error
> [   38.251742] acpi PNP0C80:05: acpi_memory_enable_device() error
> [   38.252822] acpi PNP0C80:07: acpi_memory_enable_device() error
> [   38.253858] acpi PNP0C80:09: acpi_memory_enable_device() error
> [   38.254901] acpi PNP0C80:0b: acpi_memory_enable_device() error
these errors are  normal since linux doesn't support small dimms,
you need to hot plug at least 128Mb dims for linux to be able handle them
SO there is not point to test 10Mb sizes with linux guest. Only windows can
handle that fine.

> Rebooting.
> [   91.185112] Restarting system.
> [    0.240133] acpi PNP0C80:01: acpi_memory_enable_device() error
> [    0.241304] acpi PNP0C80:03: acpi_memory_enable_device() error
> [    0.243142] acpi PNP0C80:05: acpi_memory_enable_device() error
> [    0.244114] acpi PNP0C80:07: acpi_memory_enable_device() error
> [    0.245317] acpi PNP0C80:09: acpi_memory_enable_device() error
> [    0.246506] acpi PNP0C80:0b: acpi_memory_enable_device() error
> [    0.454249] dracut-cmdline[116]: dracut-7.2 (Maipo) dracut-033-284.el7
> [    0.478756] dracut-cmdline[116]: Using kernel command line parameters:
> [    1.834722] intel_rapl: no valid rapl domains found in package 0
> [    1.849611] nommu_map_sg: overflow 107c31000+16 of device mask ffffffff
these affects baremetal as well,
use only virtue without IDE driver* or increase startup RAM to 4Gb
if you search bugzilla for nommu_map_sg, you'll find relevant BZ that's unlikely to be fixed.

  *  -nodefaults + only virtio drives also works for me.

> [    1.861058] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [    1.862574] ata2.00: cmd a0/01:00:00:10:00/00:00:00:00:00/a0 tag 0 dma
> 16400 in
> [    1.862574]          Get configuration 46 00 00 00 00 00 00 00 10 00res
> 50/00:03:00:08:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [    1.867058] ata2.00: status: { DRDY }
> 
> 
> 
> Expected results:
> The guest works well.
10Mb will work only with Windows.

> 
> 
> Additional info:

(In reply to Igor Mammedov from comment #2)
> (In reply to zhangpei from comment #0)
> > Description of problem:
> > In the guest, hotplug memory >= 13 times, then reboot, the qemu process
> > dead.    
> > 
> > 
> > 
> > Version-Release number of selected component (if applicable):
> > Host:
> > Kernel:3.10.0-296.el7.x86_64
> > qemu-kvm-rhev:qemu-kvm-rhev-2.3.0-12.el7.x86_64
> > 
> > Guest:
> > kernel:3.10.0-295.el7.x86_64
> > systemd:systemd-219-8.el7.x86_64
> > 
> > 
> > 
> > How reproducible:
> > 100%
> > 
> > 
> > 
> > Steps to Reproduce:
> > 1. start  a guest with qemu
> > #</usr/libexec/qemu-kvm command> -m 1G,slots=256,maxmem=40G -numa node
> > -monitor unix:/tmp/monitor4,server,nowait
> > 
> > 
> > 2.hotplug 10M memory 13 times manually (also tried 15times, 20 times, 255
> > times with scripts, the same results, all failed)
> > (qemu)object_add memory-backend-ram,id=memX,size=10M     (X from 1 to 13)
> > (qemu)device_add pc-dimm,id=dimmX,memdev=memX         
> > 
> > the script
> > #!/bin/bash
> > i=0 
> > while [ $i -lt 256 ] 
> > do 
> > echo $i
> > sleep 3 
> > echo "object_add memory-backend-ram,id=mem$i,size=10M"|nc -U /tmp/monitor3 
> > sleep 2
> > echo "device_add pc-dimm,id=dimm$i,memdev=mem$i,slot=$i"|nc -U /tmp/monitor3 
> > i=$(($i+1)) 
> > done 
> > 
> > 
> > 
> > 3.reboot the guest
> > in the guest
> > #reboot
> > 
> > 
> > 
> > Actual results:
> > 
> > Host:
> > [root@dhcp-12-153 ~]# /usr/libexec/qemu-kvm -name rhel7.2-4 -machine 
> > pc-i440fx-rhel7.2.0,accel=kvm,usb=off -cpu SandyBridge -m
> > 1G,slots=256,maxmem=40G -numa node -smp 4,sockets=2,cores=2,threads=1 -uuid
> > 82b1a01e-5f6c-4f5f-8d28-3854a74e9b5b -drive
> > file=/home/rhel7.2-4.qcow2,if=none,id=drive-virtio-blk-0,format=qcow2
> > -device virtio-blk-pci,drive=drive-virtio-blk-0,bootindex=1,id=virtio-blk01
> > -netdev tap,id=hostnet0 -device
> > virtio-net-pci,netdev=hostnet0,id=net0,mac=12:54:00:6c:89:9d -device
> > qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.
> > 0 -spice
> > port=5914,addr=0.0.0.0,disable-ticketing,image-compression=off,seamless-
> > migration=on  -monitor stdio -serial unix:/tmp/monitor4,server,nowait
> > QEMU 2.3.0 monitor - type 'help' for more information
> > (qemu) main_channel_link: add main channel client
> > main_channel_handle_parsed: net test: latency 0.732000 ms, bitrate
> > 22021505376 bps (21001.344086 Mbps)
> > inputs_connect: inputs channel client create
> > red_dispatcher_set_cursor_peer: 
> > 
> > (qemu) object_add memory-backend-ram,id=mem1,size=10M
> > (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
> > (qemu) 
> > (qemu) object_add memory-backend-ram,id=mem2,size=10M
> > (qemu) device_add pc-dimm,id=dimm2,memdev=mem2
> > (qemu) 
> > (qemu) object_add memory-backend-ram,id=mem3,size=10M
> > (qemu) device_add pc-dimm,id=dimm3,memdev=mem3
> > (qemu) 
> > (qemu) object_add memory-backend-ram,id=mem4,size=10M
> > (qemu) device_add pc-dimm,id=dimm4,memdev=mem4
> > (qemu) 
> > (qemu) object_add memory-backend-ram,id=mem5,size=10M
> > (qemu) device_add pc-dimm,id=dimm5,memdev=mem5
> > (qemu) 
> > (qemu) object_add memory-backend-ram,id=mem6,size=10M
> > (qemu) device_add pc-dimm,id=dimm6,memdev=mem6
> > (qemu) 
> > (qemu) object_add memory-backend-ram,id=mem7,size=10M
> > (qemu) device_add pc-dimm,id=dimm7,memdev=mem7
> > (qemu) 
> > (qemu) object_add memory-backend-ram,id=mem8,size=10M
> > (qemu) device_add pc-dimm,id=dimm8,memdev=mem8
> > (qemu) 
> > (qemu) object_add memory-backend-ram,id=mem9,size=10M
> > (qemu) device_add pc-dimm,id=dimm9,memdev=mem9
> > (qemu) 
> > (qemu) object_add memory-backend-ram,id=mem10,size=10M
> > (qemu) device_add pc-dimm,id=dimm10,memdev=mem10
> > (qemu) 
> > (qemu) object_add memory-backend-ram,id=mem11,size=10M
> > (qemu) device_add pc-dimm,id=dimm11,memdev=mem11
> > (qemu) 
> > (qemu) object_add memory-backend-ram,id=mem12,size=10M
> > (qemu) device_add pc-dimm,id=dimm12,memdev=mem12
> > (qemu) 
> > (qemu) object_add memory-backend-ram,id=mem13,size=10M
> > (qemu) device_add pc-dimm,id=dimm13,memdev=mem13
> > (qemu) inputs_detach_tablet: 
> > qemu-kvm: virtio: error trying to map MMIO memory
> can you try with windows guest + virtio, pls?
> 

I tested windows guest (win7) with virtio, and the guest can reboot well.
BTW, I tested all scenarios with virtio drivers.

> > [root@dhcp-12-153 ~]# ps aux|grep qemu
> > root     13531  0.0  0.0 112648   964 pts/0    S+   09:54   0:00 grep
> > --color=auto qemu
> > [root@dhcp-12-153 ~]#
> > 
> > 
> > Guest:
> > [root@dhcp-12-153 ~]# nc -U /tmp/monitor4
> > [    0.484842] dracut-cmdline[108]: dracut-7.2 (Maipo) dracut-033-284.el7
> > [    0.485920] dracut-cmdline[108]: Using kernel command line parameters:
> > [    1.819767] intel_rapl: no valid rapl domains found in package 0
> > 
> > 
> > Red Hat Enterprise Linux Server 7.2 (Maipo)
> > Kernel 3.10.0-295.el7.x86_64 on an x86_64
> > 
> > dhcp-9-183 login: [   38.249631] acpi PNP0C80:01:
> > acpi_memory_enable_device() error
> > [   38.250762] acpi PNP0C80:03: acpi_memory_enable_device() error
> > [   38.251742] acpi PNP0C80:05: acpi_memory_enable_device() error
> > [   38.252822] acpi PNP0C80:07: acpi_memory_enable_device() error
> > [   38.253858] acpi PNP0C80:09: acpi_memory_enable_device() error
> > [   38.254901] acpi PNP0C80:0b: acpi_memory_enable_device() error
> these errors are  normal since linux doesn't support small dimms,
> you need to hot plug at least 128Mb dims for linux to be able handle them

I tested linux guest(rhel 7.2) with 128Mb dimms, and the guest can reboot well.

> SO there is not point to test 10Mb sizes with linux guest. Only windows can
> handle that fine.

when testing hotplug/unhotplug memory feature afterwards, do we only test >=128Mb dimm in linux guest? and any size dimm in windows guest? Could you please tell me ? Thanks.

> 
> > Rebooting.
> > [   91.185112] Restarting system.
> > [    0.240133] acpi PNP0C80:01: acpi_memory_enable_device() error
> > [    0.241304] acpi PNP0C80:03: acpi_memory_enable_device() error
> > [    0.243142] acpi PNP0C80:05: acpi_memory_enable_device() error
> > [    0.244114] acpi PNP0C80:07: acpi_memory_enable_device() error
> > [    0.245317] acpi PNP0C80:09: acpi_memory_enable_device() error
> > [    0.246506] acpi PNP0C80:0b: acpi_memory_enable_device() error
> > [    0.454249] dracut-cmdline[116]: dracut-7.2 (Maipo) dracut-033-284.el7
> > [    0.478756] dracut-cmdline[116]: Using kernel command line parameters:
> > [    1.834722] intel_rapl: no valid rapl domains found in package 0
> > [    1.849611] nommu_map_sg: overflow 107c31000+16 of device mask ffffffff
> these affects baremetal as well,
> use only virtue without IDE driver* or increase startup RAM to 4Gb

I tested 4Gb RAM, and the  nommu_map_sg error disappeared.


> if you search bugzilla for nommu_map_sg, you'll find relevant BZ that's
> unlikely to be fixed.

I found the bug you mentioned. 
Bug 987871 - nommu_map_sg: overflow 107c15000+4096 of device mask ffffffff


>   *  -nodefaults + only virtio drives also works for me.

With -nodefaults and without -nodefaults are the same result:

both startup with 4G RAM:
10Mb dimms without -nodefaults :fail
10Mb dimms with -nodefaults :fail

128Mb dimms without -nodefaults :work
128Mb dimms with -nodefaults :work

> > [    1.861058] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> > [    1.862574] ata2.00: cmd a0/01:00:00:10:00/00:00:00:00:00/a0 tag 0 dma
> > 16400 in
> > [    1.862574]          Get configuration 46 00 00 00 00 00 00 00 10 00res
> > 50/00:03:00:08:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> > [    1.867058] ata2.00: status: { DRDY }
> > 
> > 
> > 
> > Expected results:
> > The guest works well.
> 10Mb will work only with Windows.
> > 
> > 
> > Additional info:

Debugged and reported issue upstream.
http://lists.nongnu.org/archive/html/qemu-devel/2015-08/msg00522.html

Problem is in virtio limitation on QEMU side that it can't handle case when guest allocated buffer crosses boundary between 2 memory regions.

From IRC discussion it's could be possible to remove limitation in virtio but it would be a rather complex re-factoring and I don't have enough virtio expertise to do it. So I's prefer to leave it to virtio maintainers.

patches at 
http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-November/msg00230.html

Reproduce:
- host
qemu-kvm-rhev-2.3.0-21.el7
kernel-3.10.0-418.el7.x86_64

- guest
kernel-3.10.0-327.el7.x86_64
systemd-219-19.el7.x86_64

Steps:
1. boot guest 
/usr/libexec/qemu-kvm -name rhel72-1 -m 4G,slots=100,maxmem=20G -smp 4  \
-drive file=/home/guest/RHEL-Server-7.2-64-virtio.raw,if=none,id=drive-data-disk,format=raw,cache=none -device virtio-blk-pci,drive=drive-data-disk,id=data-disk  \
-netdev tap,id=hostnet1,vhost=on -device e1000,netdev=hostnet1,id=net2,mac=10:1a:4a:42:0a:0a,bus=pci.0,addr=0xa \
-usb -device usb-tablet,id=input0 -vga qxl -vnc :1  \
-monitor unix:/tmp/monitor3,server,nowait  -monitor stdio -serial unix:/tmp/monitor4,server,nowait

2. hotplug 10M memory 25 times via script:
#!/bin/bash
i=0 
while [ $i -lt 25 ] 
do 
echo $i
sleep 3 
echo "object_add memory-backend-ram,id=mem$i,size=10M"|nc -U /tmp/monitor3 
sleep 2
echo "device_add pc-dimm,id=dimm$i,memdev=mem$i,slot=$i"|nc -U /tmp/monitor3 
i=$(($i+1)) 
done 

3. reboot guest
# reboot

After step 3, qemu quits and prints "(qemu) qemu-kvm: virtio: error trying to map MMIO memory" .
So the bug is reproduced. 


Verify:
- host
qemu-kvm-rhev-2.6.0-4.el7
kernel-3.10.0-418.el7.x86_64

- guest
kernel-3.10.0-327.el7.x86_64
systemd-219-19.el7.x86_64

With same steps as above, qemu quits and prints "(qemu) qemu-kvm: Looped descriptor". 

Hi Igor, 
would you please help check if anything wrong with my steps or this bug is not fixed well with qemu-kvm-rhev-2.6.0-4.el7 ? 

Thanks.

[...]
> 
> Verify:
> - host
> qemu-kvm-rhev-2.6.0-4.el7
> kernel-3.10.0-418.el7.x86_64
> 
> - guest
> kernel-3.10.0-327.el7.x86_64
> systemd-219-19.el7.x86_64
> 
> With same steps as above, qemu quits and prints "(qemu) qemu-kvm: Looped
> descriptor". 
> 
> Hi Igor, 
> would you please help check if anything wrong with my steps or this bug is
> not fixed well with qemu-kvm-rhev-2.6.0-4.el7 ? 
To me it looks like a new unrelated issue.
Probably it's virtio regression, open a new BZ for it and CC virtio guys.

(In reply to Igor Mammedov from comment #14)
> [...]
> > 
> > Verify:
> > - host
> > qemu-kvm-rhev-2.6.0-4.el7
> > kernel-3.10.0-418.el7.x86_64
> > 
> > - guest
> > kernel-3.10.0-327.el7.x86_64
> > systemd-219-19.el7.x86_64
> > 
> > With same steps as above, qemu quits and prints "(qemu) qemu-kvm: Looped
> > descriptor". 
> > 
> > Hi Igor, 
> > would you please help check if anything wrong with my steps or this bug is
> > not fixed well with qemu-kvm-rhev-2.6.0-4.el7 ? 
> To me it looks like a new unrelated issue.
> Probably it's virtio regression, open a new BZ for it and CC virtio guys.

QE have filed bz[1] to track this issue. Would this bug be considered as verified ? 

[1] Bug 1341411 - QEMU quit and print "Looped descriptor" when reboot guest after hotplug 10M memory >=20 times

(In reply to Yumei Huang from comment #15)
> (In reply to Igor Mammedov from comment #14)
> > [...]
> > > 
> > > Verify:
> > > - host
> > > qemu-kvm-rhev-2.6.0-4.el7
> > > kernel-3.10.0-418.el7.x86_64
> > > 
> > > - guest
> > > kernel-3.10.0-327.el7.x86_64
> > > systemd-219-19.el7.x86_64
> > > 
> > > With same steps as above, qemu quits and prints "(qemu) qemu-kvm: Looped
> > > descriptor". 
> > > 
> > > Hi Igor, 
> > > would you please help check if anything wrong with my steps or this bug is
> > > not fixed well with qemu-kvm-rhev-2.6.0-4.el7 ? 
> > To me it looks like a new unrelated issue.
> > Probably it's virtio regression, open a new BZ for it and CC virtio guys.
> 
> QE have filed bz[1] to track this issue. Would this bug be considered as
> verified ? 
I'd think so.
[...]

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html