Looking at the logs it doesn't look like there's anything left.

Inspecting the QOM-tree I found nothing (although I didn't know where precisely to look for, qom-tree didn't work for me for some reason).

Is there anything more libvirt could do, actually?  Look at what we do on the monitor when I attach and then detach the device:

# We plug the backend:
msg={"execute":"object-add","arguments":{"qom-type":"memory-backend-file","id":"shmmem-shmem0","props":{"mem-path":"/dev/shm/my_shmem0","size":4194304,"share":true}},"id":"libvirt-13"}
reply={"return": {}, "id": "libvirt-13"}

# Then the device itself:
msg={"execute":"device_add","arguments":{"driver":"ivshmem-plain","id":"shmem0","memdev":"shmmem-shmem0","bus":"pci.0","addr":"0x9"},"id":"libvirt-14"}
reply={"return": {}, "id": "libvirt-14"}

# And then check if it is plugged (it is, shmem0 is there):
msg={"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-15"}
reply={"return": [{"name": "rng0", "type": "child<virtio-rng-pci>"}, {"name": "sound0-codec0", "type": "child<hda-duplex>"}, {"name": "virtio-serial0", "type": "child<virtio-serial-pci>"}, {"name": "video0", "type": "child<qxl-vga>"}, {"name": "serial0", "type": "child<isa-serial>"}, {"name": "sound0", "type": "child<intel-hda>"}, {"name": "balloon0", "type": "child<virtio-balloon-pci>"}, {"name": "channel1", "type": "child<virtserialport>"}, {"name": "channel0", "type": "child<virtserialport>"}, {"name": "net0", "type": "child<virtio-net-pci>"}, {"name": "input0", "type": "child<usb-tablet>"}, {"name": "shmem0", "type": "child<ivshmem-plain>"}, {"name": "redir1", "type": "child<usb-redir>"}, {"name": "redir0", "type": "child<usb-redir>"}, {"name": "usb", "type": "child<ich9-usb-ehci1>"}, {"name": "type", "type": "string"}, {"name": "ide0-0-0", "type": "child<ide-cd>"}], "id": "libvirt-15"}

# When unplugging we remove the device:
msg={"execute":"device_del","arguments":{"id":"shmem0"},"id":"libvirt-16"}
reply={"return": {}, "id": "libvirt-16"}

# Then wait for the event:
event={"timestamp": {"seconds": 1535560504, "microseconds": 629824}, "event": "DEVICE_DELETED", "data": {"device": "shmem0", "path": "/machine/peripheral/shmem0"}}

# Remove the backend
msg={"execute":"object-del","arguments":{"id":"shmmem-shmem0"},"id":"libvirt-17"}
reply={"return": {}, "id": "libvirt-17"}

# And check that it was removed:
msg={"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-18"}
reply={"return": [{"name": "rng0", "type": "child<virtio-rng-pci>"}, {"name": "sound0-codec0", "type": "child<hda-duplex>"}, {"name": "virtio-serial0", "type": "child<virtio-serial-pci>"}, {"name": "video0", "type": "child<qxl-vga>"}, {"name": "serial0", "type": "child<isa-serial>"}, {"name": "sound0", "type": "child<intel-hda>"}, {"name": "balloon0", "type": "child<virtio-balloon-pci>"}, {"name": "channel1", "type": "child<virtserialport>"}, {"name": "channel0", "type": "child<virtserialport>"}, {"name": "net0", "type": "child<virtio-net-pci>"}, {"name": "input0", "type": "child<usb-tablet>"}, {"name": "redir1", "type": "child<usb-redir>"}, {"name": "redir0", "type": "child<usb-redir>"}, {"name": "usb", "type": "child<ich9-usb-ehci1>"}, {"name": "type", "type": "string"}, {"name": "ide0-0-0", "type": "child<ide-cd>"}], "id": "libvirt-18"}

Dave and Juan,

As Martin commented above, libvirt seems to be cleaning up the 
ivshmem device properly after the device unplug.

However as described by the intial problem report, migration 
is still failing:

  7.Do migration again:
  # virsh migrate iommu1 qemu+ssh://10.66.4.101/system --live --verbose --unsafe
  error: internal error: unable to execute QEMU command 'migrate': Migration 
  is disabled when using feature 'peer mode' in device 'ivshmem'

Would it be possible for you to provide some information on the 
"dirty state" that is causing migration code to generate this error 
message and thus failing the migration operation.

Thanks.

it should work (TM).
I can see hw/misc/ivshmem.c registers a 'migration blocker' in it's ivshmem_common_realize and in it's ivshmem_exit it does:

    if (s->migration_blocker) {
        migrate_del_blocker(s->migration_blocker);
        error_free(s->migration_blocker);
    }

so all peachy in theory.

so I guess the question is whether ivshmem_exit is being called

> so I guess the question is whether ivshmem_exit is being called

Indeed.  It isn't.  Upstream commit 2aece63c8a9 (v2.7.0) screwed it up.

We got that bug in the rebase to v2.9.0 for RHEL 7.4's qemu-kvm-rhev.
I'm pretty sure RHEL-7.3's qemu-kvm-rhev worked, i.e. this is a regression.
Not a recent one, though.

Fixed upstream in commit b266f1d1123396f9f5df865508f7555ab0c9582a.

Fix included in qemu-kvm-rhev-2.12.0-19.el7

Reproduce:
qemu-kvm-rhev-2.12.0-17.el7

1. Boot guest on src host, do hotplug ivshmem and unplug it

# /usr/libexec/qemu-kvm -m 1024,slots=4,maxmem=32G -smp 16,cores=8,threads=1,sockets=2 -numa node -vnc :0 -monitor stdio /home/kvm_autotest_root/images/rhel76-64-virtio-scsi.qcow2 

(qemu) object_add memory-backend-file,id=mem1,size=512M,mem-path=/mnt/kvm_hugepage,share
(qemu) device_add ivshmem-plain,id=ivshmem1,memdev=mem1

(qemu) device_del ivshmem1   
(qemu) object_del mem1 


2. Boot guest on dst host with "-incoming tcp:0:1234"


3. Migrate guest from src host to dst host

(qemu) migrate -d tcp:xxxx:1234
Migration is disabled when using feature 'peer mode' in device 'ivshmem'

Migration failed with above error message. 


Verify:
qemu-kvm-rhev-2.12.0-20.el7

Same steps as above, migration works well. 

(qemu) migrate -d tcp:xxx:1234
(qemu) 
(qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off return-path: off pause-before-switchover: off x-multifd: off dirty-bitmaps: off late-block-activate: off 
Migration status: active
total time: 2994 milliseconds
expected downtime: 300 milliseconds
setup: 22 milliseconds
transferred ram: 98361 kbytes
throughput: 268.58 mbps
remaining ram: 945040 kbytes
total ram: 1065800 kbytes
duplicate: 5660 pages
skipped: 0 pages
normal: 24530 pages
normal bytes: 98120 kbytes
dirty sync count: 1
page size: 4 kbytes

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2553