Hi,
  Can you check that the source has really hot unplugged the guest.
After you do your:
    device_del "ba"

do an;
    info pci

  if the device is still there then migration will still expect it.
What was the guest booted into when you tired this - was it already booted into the guest OS?

(In reply to Dr. David Alan Gilbert from comment #2)
> Hi,
>   Can you check that the source has really hot unplugged the guest.
> After you do your:
>     device_del "ba"
> 
> do an;
>     info pci
> 
>   if the device is still there then migration will still expect it.
> What was the guest booted into when you tired this - was it already booted
> into the guest OS?

Operate as requested, results as bellow:
(qemu) device_del ba
(qemu) 
(qemu) 
(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc060 [0xc06f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 11.
      BAR0: I/O at 0xc020 [0xc03f].
      BAR1: 32 bit memory at 0xfc052000 [0xfc052fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net0"
  Bus  0, device  11, function 0:
    VGA controller: PCI device 1b36:0100
      IRQ 11.
      BAR0: 32 bit memory at 0xf4000000 [0xf7ffffff].
      BAR1: 32 bit memory at 0xf8000000 [0xfbffffff].
      BAR2: 32 bit memory at 0xfc050000 [0xfc051fff].
      BAR3: I/O at 0xc040 [0xc05f].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id "video0"
(qemu) info balloon 
No balloon device has been activated
(qemu) 

 When i do migration, guest os is already entered.

(In reply to aihua liang from comment #3)
> (In reply to Dr. David Alan Gilbert from comment #2)
> > Hi,
> >   Can you check that the source has really hot unplugged the guest.
> > After you do your:
> >     device_del "ba"
> > 
> > do an;
> >     info pci
> > 
> >   if the device is still there then migration will still expect it.
> > What was the guest booted into when you tired this - was it already booted
> > into the guest OS?
> 
> Operate as requested, results as bellow:
> (qemu) device_del ba
> (qemu) 
> (qemu) 
> (qemu) info pci
>   Bus  0, device   0, function 0:
>     Host bridge: PCI device 8086:1237
>       id ""
>   Bus  0, device   1, function 0:
>     ISA bridge: PCI device 8086:7000
>       id ""
>   Bus  0, device   1, function 1:
>     IDE controller: PCI device 8086:7010
>       BAR4: I/O at 0xc060 [0xc06f].
>       id ""
>   Bus  0, device   1, function 3:
>     Bridge: PCI device 8086:7113
>       IRQ 9.
>       id ""
>   Bus  0, device   3, function 0:
>     Ethernet controller: PCI device 1af4:1000
>       IRQ 11.
>       BAR0: I/O at 0xc020 [0xc03f].
>       BAR1: 32 bit memory at 0xfc052000 [0xfc052fff].
>       BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
>       id "net0"
>   Bus  0, device  11, function 0:
>     VGA controller: PCI device 1b36:0100
>       IRQ 11.
>       BAR0: 32 bit memory at 0xf4000000 [0xf7ffffff].
>       BAR1: 32 bit memory at 0xf8000000 [0xfbffffff].
>       BAR2: 32 bit memory at 0xfc050000 [0xfc051fff].
>       BAR3: I/O at 0xc040 [0xc05f].
>       BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
>       id "video0"
> (qemu) info balloon 
> No balloon device has been activated
> (qemu) 
> 
>  When i do migration, guest os is already entered.
   
   BTW,qemu cmds given about balloon:
    Wrong cmdline： -device virtio-balloon-pci,id=balloon0,bus=pci.0,id=ba \
    Correct cmdline: -device virtio-balloon-pci,bus=pci.0,id=ba \
   
   **Both script in src and dst need to be updated to the correct one.**

Interesting;  I've tried repeating it on the current 2.6.0-9 package and it seems OK from the test I did; I was using a slightly simpler line:


/usr/libexec/qemu-kvm -nographic -M pc,accel=kvm -m 4G -device virtio-balloon-pci,bus=pci.0,id=ba  /home/vms/f20.qcow2

can you retest it with that version pelase and also just check that the device_del is being done on the source side;  if it still fails, are you using a script to drive this and if so I'd appreciate a copy of the script.

(In reply to Dr. David Alan Gilbert from comment #5)
> Interesting;  I've tried repeating it on the current 2.6.0-9 package and it
> seems OK from the test I did; I was using a slightly simpler line:
> 
> 
> /usr/libexec/qemu-kvm -nographic -M pc,accel=kvm -m 4G -device
> virtio-balloon-pci,bus=pci.0,id=ba  /home/vms/f20.qcow2
> 
> can you retest it with that version pelase and also just check that the
> device_del is being done on the source side;  if it still fails, are you
> using a script to drive this and if so I'd appreciate a copy of the script.

Sorry for seeing the mail today，i‘ll try again and give a response later.

(In reply to aihua liang from comment #6)
> (In reply to Dr. David Alan Gilbert from comment #5)
> > Interesting;  I've tried repeating it on the current 2.6.0-9 package and it
> > seems OK from the test I did; I was using a slightly simpler line:
> > 
> > 
> > /usr/libexec/qemu-kvm -nographic -M pc,accel=kvm -m 4G -device
> > virtio-balloon-pci,bus=pci.0,id=ba  /home/vms/f20.qcow2
> > 
> > can you retest it with that version pelase and also just check that the
> > device_del is being done on the source side;  if it still fails, are you
> > using a script to drive this and if so I'd appreciate a copy of the script.
> 
> Sorry for seeing the mail today，i‘ll try again and give a response later.

I have tested in qemu-kvm-rhev 2.6.0-8, this problem not exist when i do hot unplug virtio balloon device.

But when i hot plug a memory balloon device on source, then i do live migration, the same error info still exist in dst.
*****souce operation************
 (qemu)device_add virtio-balloon-pci,id=balloon0 
 (qemu)migrate -d tcp:$dest_ip:port
 (qemu)info status           ----->paused(postmigrate)

*****error message in dst******
  qemu-kvm:Unknown savevm section or instance "0000:00:02.0/virtio-balloon 0"
  qemu-kvm load of migration failed:invalid argument.

(In reply to aihua liang from comment #7)
> (In reply to aihua liang from comment #6)
> > (In reply to Dr. David Alan Gilbert from comment #5)
> > > Interesting;  I've tried repeating it on the current 2.6.0-9 package and it
> > > seems OK from the test I did; I was using a slightly simpler line:
> > > 
> > > 
> > > /usr/libexec/qemu-kvm -nographic -M pc,accel=kvm -m 4G -device
> > > virtio-balloon-pci,bus=pci.0,id=ba  /home/vms/f20.qcow2
> > > 
> > > can you retest it with that version pelase and also just check that the
> > > device_del is being done on the source side;  if it still fails, are you
> > > using a script to drive this and if so I'd appreciate a copy of the script.
> > 
> > Sorry for seeing the mail today，i‘ll try again and give a response later.
> 
> I have tested in qemu-kvm-rhev 2.6.0-8, this problem not exist when i do hot
> unplug virtio balloon device.
> 
> But when i hot plug a memory balloon device on source, then i do live
> migration, the same error info still exist in dst.
> *****souce operation************
>  (qemu)device_add virtio-balloon-pci,id=balloon0 
>  (qemu)migrate -d tcp:$dest_ip:port
>  (qemu)info status           ----->paused(postmigrate)
> 
> *****error message in dst******
>   qemu-kvm:Unknown savevm section or instance "0000:00:02.0/virtio-balloon 0"
>   qemu-kvm load of migration failed:invalid argument.

Make sure you use an addr= field on both the device_add and on the qemu command line on the destination to ensure that they end up in the same PCI slot.
If it still fails please show me the source/destination qemu command lines and matching device_add.

 No relationship with "addr".
    When i start dest guest without virtio balloon device, migration of hot plug virtio balloon device will fail.
1)Qemu-cmds:
  src-guest:
/usr/libexec/qemu-kvm \
-M pc \
-name rhel7.3-4 \
-machine pc,accel=kvm,usb=off,vmport=off \
-cpu host \
-m 4G \
-smp 8,sockets=8,cores=1,threads=1 \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
-mon chardev=qmp_id_catch_monitor,id=monitor,mode=readline -no-hpet \
-drive file=/home/73test/img/se_test.qcow2,if=none,id=drive-system-disk0,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop \
-device ide-drive,drive=drive-system-disk0,id=d0,bus=ide.0,unit=0 \
-vga qxl \
-enable-kvm \
-monitor stdio \
-vnc 0:20 \

  dest-guest:
/usr/libexec/qemu-kvm \
-M pc \
-name rhel7.3-4 \
-machine pc,accel=kvm,usb=off,vmport=off \
-cpu host \
-m 4G \
-smp 8,sockets=8,cores=1,threads=1 \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
-mon chardev=qmp_id_catch_monitor,id=monitor,mode=readline -no-hpet \
-drive file=/home/balloon/img/se_test.qcow2,if=none,id=drive-system-disk0,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop \
-device ide-drive,drive=drive-system-disk0,id=d0,bus=ide.0,unit=0 \
-vga qxl \
-enable-kvm \
-monitor stdio \
-vnc 0:20 \
-incoming tcp:0:4000 \

2)Hmp cmds on src:
  (qemu)device_add virtio-balloon-pci,id=ba   ---->discovery virtio balloon device via cmd "lspci" in guest.
  (qemu)migrate -d tcp:$dest_ip:port
  (qemu)info status      --->status:paused

3) Result: 
    Migration fail with error message：
     qemu-kvm:Unknown savevm section or instance "0000:00:02.0/virtio-balloon 0"
     qemu-kvm load of migration failed:invalid argument.


 When i start dest guest with virtio balloon device, migration of hot plug virtio balloon device will success.
1)Qemu-cmds:
  src-guest:
/usr/libexec/qemu-kvm \
-M pc \
-name rhel7.3-4 \
-machine pc,accel=kvm,usb=off,vmport=off \
-cpu host \
-m 4G \
-smp 8,sockets=8,cores=1,threads=1 \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
-mon chardev=qmp_id_catch_monitor,id=monitor,mode=readline -no-hpet \
-drive file=/home/73test/img/se_test.qcow2,if=none,id=drive-system-disk0,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop \
-device ide-drive,drive=drive-system-disk0,id=d0,bus=ide.0,unit=0 \
-vga qxl \
-enable-kvm \
-monitor stdio \
-vnc 0:20 \

  dest-guest:
/usr/libexec/qemu-kvm \
-M pc \
-name rhel7.3-4 \
-machine pc,accel=kvm,usb=off,vmport=off \
-cpu host \
-m 4G \
-smp 8,sockets=8,cores=1,threads=1 \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
-mon chardev=qmp_id_catch_monitor,id=monitor,mode=readline -no-hpet \
-drive file=/home/balloon/img/se_test.qcow2,if=none,id=drive-system-disk0,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop \
-device ide-drive,drive=drive-system-disk0,id=d0,bus=ide.0,unit=0 \
-vga qxl \
-enable-kvm \
-monitor stdio \
-vnc 0:20 \
-incoming tcp:0:4000 \
-virtio balloon 
 
2)Hmp cmds on src:
  (qemu)device_add virtio-balloon-pci,id=ba   ---->discovery virtio balloon device via cmd "lspci" in guest.
  (qemu)migrate -d tcp:$dest_ip:port
  (qemu)info status      --->status:paused
3) Result: 
    Migration success.

Hot plug migration fail when starting dest guest without balloon device 
But hot unplug migration success when starting dest guest with balloon device
Any foundational difference between these two operations?

(In reply to aihua liang from comment #9)
>  No relationship with "addr".
>     When i start dest guest without virtio balloon device, migration of hot
> plug virtio balloon device will fail.
> 1)Qemu-cmds:
>   src-guest:
> /usr/libexec/qemu-kvm \
> -M pc \
> -name rhel7.3-4 \
> -machine pc,accel=kvm,usb=off,vmport=off \
> -cpu host \
> -m 4G \
> -smp 8,sockets=8,cores=1,threads=1 \
> -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
> -mon chardev=qmp_id_catch_monitor,id=monitor,mode=readline -no-hpet \
> -drive
> file=/home/73test/img/se_test.qcow2,if=none,id=drive-system-disk0,
> format=qcow2,cache=none,aio=native,werror=stop,rerror=stop \
> -device ide-drive,drive=drive-system-disk0,id=d0,bus=ide.0,unit=0 \
> -vga qxl \
> -enable-kvm \
> -monitor stdio \
> -vnc 0:20 \
> 
>   dest-guest:
> /usr/libexec/qemu-kvm \
> -M pc \
> -name rhel7.3-4 \
> -machine pc,accel=kvm,usb=off,vmport=off \
> -cpu host \
> -m 4G \
> -smp 8,sockets=8,cores=1,threads=1 \
> -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
> -mon chardev=qmp_id_catch_monitor,id=monitor,mode=readline -no-hpet \
> -drive
> file=/home/balloon/img/se_test.qcow2,if=none,id=drive-system-disk0,
> format=qcow2,cache=none,aio=native,werror=stop,rerror=stop \
> -device ide-drive,drive=drive-system-disk0,id=d0,bus=ide.0,unit=0 \
> -vga qxl \
> -enable-kvm \
> -monitor stdio \
> -vnc 0:20 \
> -incoming tcp:0:4000 \
> 
> 2)Hmp cmds on src:
>   (qemu)device_add virtio-balloon-pci,id=ba   ---->discovery virtio balloon
> device via cmd "lspci" in guest.
>   (qemu)migrate -d tcp:$dest_ip:port
>   (qemu)info status      --->status:paused
> 
> 3) Result: 
>     Migration fail with error message：
>      qemu-kvm:Unknown savevm section or instance
> "0000:00:02.0/virtio-balloon 0"
>      qemu-kvm load of migration failed:invalid argument.
> 
> 
>  When i start dest guest with virtio balloon device, migration of hot plug
> virtio balloon device will success.
> 1)Qemu-cmds:
>   src-guest:
> /usr/libexec/qemu-kvm \
> -M pc \
> -name rhel7.3-4 \
> -machine pc,accel=kvm,usb=off,vmport=off \
> -cpu host \
> -m 4G \
> -smp 8,sockets=8,cores=1,threads=1 \
> -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
> -mon chardev=qmp_id_catch_monitor,id=monitor,mode=readline -no-hpet \
> -drive
> file=/home/73test/img/se_test.qcow2,if=none,id=drive-system-disk0,
> format=qcow2,cache=none,aio=native,werror=stop,rerror=stop \
> -device ide-drive,drive=drive-system-disk0,id=d0,bus=ide.0,unit=0 \
> -vga qxl \
> -enable-kvm \
> -monitor stdio \
> -vnc 0:20 \
> 
>   dest-guest:
> /usr/libexec/qemu-kvm \
> -M pc \
> -name rhel7.3-4 \
> -machine pc,accel=kvm,usb=off,vmport=off \
> -cpu host \
> -m 4G \
> -smp 8,sockets=8,cores=1,threads=1 \
> -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
> -mon chardev=qmp_id_catch_monitor,id=monitor,mode=readline -no-hpet \
> -drive
> file=/home/balloon/img/se_test.qcow2,if=none,id=drive-system-disk0,
> format=qcow2,cache=none,aio=native,werror=stop,rerror=stop \
> -device ide-drive,drive=drive-system-disk0,id=d0,bus=ide.0,unit=0 \
> -vga qxl \
> -enable-kvm \
> -monitor stdio \
> -vnc 0:20 \
> -incoming tcp:0:4000 \
> -virtio balloon 
>  
> 2)Hmp cmds on src:
>   (qemu)device_add virtio-balloon-pci,id=ba   ---->discovery virtio balloon
> device via cmd "lspci" in guest.
>   (qemu)migrate -d tcp:$dest_ip:port
>   (qemu)info status      --->status:paused
> 3) Result: 
>     Migration success.
> 
> Hot plug migration fail when starting dest guest without balloon device 
> But hot unplug migration success when starting dest guest with balloon device
> Any foundational difference between these two operations?


This is the correct behaviour.
Whenever you hotplug/hotunplug a device you should match it on the commandline of the destination.
In the 1st case you are hotplugging on the source but haven't added it on the destination qemu; when the source sends the data for the hotplugged device the destination correctly gives the error stating the device that's been received isn't found.

The opposite case; where a device is unplugged on the source but still present on the destination is also an incorrect test; however we don't have anything to detect it.

Remember: The destination hardware must always match the source; if you hot plug something on the source add it to the destination command line.