Bug 1020722

Summary: qemu-kvm quit on "create_userspace_phys_mem: File exists. assigned_dev_iomem_map: Error: create new mapping failed" while repeatedly hot plugging/unplugging intel dual port 82576 PFs to win2012 guest
Product: Red Hat Enterprise Linux 6 Reporter: Chao Yang <chayang>
Component: qemu-kvmAssignee: Alex Williamson <alex.williamson>
Status: CLOSED WONTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.5CC: acathrow, alex.williamson, bsarathy, chayang, juzhang, michen, mkenneth, qzhang, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-12 12:48:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chao Yang 2013-10-18 08:29:19 UTC
Description of problem:
This failure happened when trying to execute the loop for 42 times.

Version-Release number of selected component (if applicable):
2.6.32-423.el6.x86_64
qemu-kvm-0.12.1.2-2.412.el6.x86_64

How reproducible:
2/2

Steps to Reproduce:
1. start a win2012 guest by:
/usr/libexec/qemu-kvm -name win8_64_amd-1 -M rhel6.5.0 -cpu host -enable-kvm -m 2048 -realtime mlock=off -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/home/en_windows_server_2012_x64_dvd_915478.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=win2012.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -net none -spice port=5900,disable-ticketing -k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -device virtio-balloon-pci,id=balloon0,bus=pci.0 -monitor unix:/opt/monitor,server,nowait -boot menu=on -fda /usr/share/virtio-win/virtio-win-1.6.6_amd64.vfd

2. repeatedly hot plug and unplug intel dual port 82576 PFs:
#!/bin/bash

for i in $(seq 1 500) ; do
	echo "----------------------------------------"
	echo
	echo "##############the $i loop###############"
	echo "device_add pci-assign,host=08:00.0,id=PF" | nc -U /opt/monitor
	echo "info pci" | nc -U /opt/monitor
	sleep 6
	echo "device_del PF" | nc -U /opt/monitor
	echo "info pci" | nc -U /opt/monitor
	sleep 6
	echo "device_add pci-assign,host=08:00.1,id=PF-1" | nc -U /opt/monitor
	echo "info pci" | nc -U /opt/monitor
	sleep 6
	echo "device_del PF-1" | nc -U /opt/monitor
	echo "info pci" | nc -U /opt/monitor
	sleep 6
done

3.

Actual results:
It failed to perform after 41 iterations for first attempt. 101 for second attempt.

Expected results:


Additional info:

Comment 2 Alex Williamson 2014-04-16 19:24:52 UTC
Please re-test and provide logs of script and error message.  Please also install debug package for qemu-kvm and connect to process prior to test and provide backtrace on error.

Hotplug tests like this should generally actively test whether the guest released the device rather than relying on a sleep.

Comment 3 Chao Yang 2014-04-17 07:37:33 UTC
(In reply to Alex Williamson from comment #2)
> Please re-test and provide logs of script and error message.  Please also
> install debug package for qemu-kvm and connect to process prior to test and
> provide backtrace on error.
> 

I am queueing to take this system, will update once I get it.

> Hotplug tests like this should generally actively test whether the guest
> released the device rather than relying on a sleep.

If I don't add any sleep, it will lead to 'duplicated ID'.

Comment 4 Alex Williamson 2014-04-17 12:39:57 UTC
(In reply to Chao Yang from comment #3)
> (In reply to Alex Williamson from comment #2)
> > Please re-test and provide logs of script and error message.  Please also
> > install debug package for qemu-kvm and connect to process prior to test and
> > provide backtrace on error.
> > 
> 
> I am queueing to take this system, will update once I get it.
> 
> > Hotplug tests like this should generally actively test whether the guest
> > released the device rather than relying on a sleep.
> 
> If I don't add any sleep, it will lead to 'duplicated ID'.

By actively test, I mean check whether the device has been removed at the end of the sleep rather than assuming it worked (ie. parse the output of 'info pci' rather than simply dump it).

Comment 5 Chao Yang 2014-06-10 06:39:00 UTC
Reproduced again with host kernel-2.6.32-477.el6.x86_64
qemu-kvm-0.12.1.2-2.427.el6.x86_64 while repeatedly hot plugging/unplugging intel dual port 82576 PFs to/from windows 2012 guest at 18th times.

Breakpoint 2, assigned_dev_iomem_map (pci_dev=0x7fffe51757f0, region_num=0, e_phys=4269670400, e_size=131072, type=0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/device-assignment.c:269
269	{
(gdb) bt
#0  assigned_dev_iomem_map (pci_dev=0x7fffe51757f0, region_num=0, e_phys=4269670400, e_size=131072, type=0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/device-assignment.c:269
#1  0x00007ffff7dcab0b in pci_update_mappings (d=0x7fffe51757f0) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/pci.c:1096
#2  0x00007ffff7f668a7 in assigned_dev_pci_write_config (d=0x7fffe51757f0, address=4, val=1030, len=2)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/device-assignment.c:484
#3  0x00007ffff7de15ad in kvm_handle_io (env=0x7ffff8866f20) at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:148
#4  kvm_run (env=0x7ffff8866f20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1061
#5  0x00007ffff7de16a9 in kvm_cpu_exec (env=0x7ffff8866f20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1756
#6  0x00007ffff7de259d in kvm_main_loop_cpu (_env=0x7ffff8866f20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2018
#7  ap_main_loop (_env=0x7ffff8866f20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2074
#8  0x00007ffff76f29d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffff4914b7d in clone () from /lib64/libc.so.6
(gdb) c
Continuing.
create_userspace_phys_mem: File exists
assigned_dev_iomem_map: Error: create new mapping failed

Comment 6 Chao Yang 2014-06-10 07:00:44 UTC
If you need additional information, please needinfo me. I have reserved this system for 3 days.

Comment 7 Alex Williamson 2014-06-10 18:14:24 UTC
Can this be reproduced with libvirt using virsh nodedev operations?

Comment 8 Chao Yang 2014-06-12 05:12:42 UTC
(In reply to Alex Williamson from comment #7)
> Can this be reproduced with libvirt using virsh nodedev operations?

No, using virsh attach-device/detach-device failed to reproduce this issue.

#!/bin/bash


for i in $(seq 1 500); do
	
	echo " ---------- The $i iterations ------------"

	virsh attach-device test pci.xml
	sleep 3
	virsh detach-device test pci.xml


	while virsh qemu-monitor-command test --hmp "info pci" | grep "8086:10c9"; do
		sleep 3
	done	

	virsh attach-device test pci-1.xml
	sleep 3
	virsh detach-device test pci-1.xml

	while virsh qemu-monitor-command test --hmp "info pci" | grep "8086:10c9"; do
		sleep 3
	done
done

# cat pci.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
	<source>
       		<address domain='0x0000' bus='0x23' slot='0x00' function='0x0'/>
        </source>
</hostdev>