Bug 1425273

Also checked q35 machine type with block device, didn't reproduce the issue

1.Boot guest with block device
2.unplug block device through hmp
(qemu) device_del virtio-disk1
3.do the local migration
4.migrated successfully

[1]/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-m 4G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/seabios.log,id=seabios \
-vga qxl \
-spice port=5931,disable-ticketing \
-qmp tcp:0:4445,server,nowait \
-device ioh3420,id=root.0,slot=1 \
-drive file=/home/test/block1.qcow2,if=none,id=drive1,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,id=virtio-disk1,drive=drive1,bus=root.0 \
-device e1000e,netdev=dev1,mac=9a:6a:6b:6c:6d:6a,id=net1 \
-netdev tap,id=dev1,vhost=on \
-device ioh3420,id=root.1,slot=2 \
-drive file=/home/test/q35-seabios.qcow2,if=none,id=drive0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,id=virtio-disk0,drive=drive0,bus=root.1,bootindex=0 \
-monitor stdio \

Thanks
Jing Zhao

with the version of qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64

migration failed after unplug virtio-net-pci
migration failed after plug virtio-net-pci

with the version of qemu-kvm-rhev-debuginfo-2.8.0-5.el7.x86_64

migration failed after unplug virtio-net-pci deivce
migration failed after plug virtio-net-pci device

Hi,

Can you please provide more information:
When you removed the e1000e device, did you have another NIC so the migration process can use?
When you hot-plugged the e1000e device, was it the only NIC in the system?
Can you provide the exact steps the failed migration for virtio-net-pci device?
Did you check PC, Q35 or both for virtio-net-pci?

Thanks,
Marcel

(In reply to Marcel Apfelbaum from comment #5)
> Hi,
> 
> Can you please provide more information:
> When you removed the e1000e device, did you have another NIC so the
> migration process can use?

  I only use one NIC,but when i tried to use two NIC ,the results is still failed
> When you hot-plugged the e1000e device, was it the only NIC in the system?

  Yes,it was the only NIC in the system,if it has a NIC before hot plugged in the system,the results is successful
> Can you provide the exact steps the failed migration for virtio-net-pci
> device?

1 Boot guest with virtio-net-pci device [1]
2 Hot plug virtio-net-pci device and then migrate
(qemu) info network 
hub 0
 \ hub0port1: user.0: index=0,type=user,net=10.0.2.0,restrict=off
 \ hub0port0: e1000.0: index=0,type=nic,model=e1000,macaddr=52:54:00:12:34:56
(qemu) netdev_add tap,vhost=on,id=dev1
(qemu) device_add virtio-net-pci,netdev=dev1,id=net1,mac=9a:6a:6b:6c:6d:6a,bus=root.0
(qemu) info network 
hub 0
 \ hub0port1: user.0: index=0,type=user,net=10.0.2.0,restrict=off
 \ hub0port0: e1000.0: index=0,type=nic,model=e1000,macaddr=52:54:00:12:34:56
net1: index=0,type=nic,model=virtio-net-pci,macaddr=9a:6a:6b:6c:6d:6a
 \ dev1: index=0,type=tap,ifname=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
(qemu) migrate -d tcp:10.66.4.211:5800
(qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off 
Migration status: failed
total time: 0 milliseconds

dest status:
(qemu) info network 
net1: index=0,type=nic,model=virtio-net-pci,macaddr=9a:6a:6b:6c:6d:6a
 \ dev1: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
(qemu) qemu-kvm: Unknown ramblock "0000:00:02.0/e1000.rom", cannot accept migration
qemu-kvm: error while loading state for instance 0x0 of device 'ram'
qemu-kvm: load of migration failed: Invalid argument
red_channel_client_disconnect: rcc=0x7f3d2efae000 (channel=0x7f3d2da5c600 type=2 id=0)
red_channel_client_disconnect: rcc=0x7f3d2dcd0000 (channel=0x7f3d2da4e580 type=4 id=0)

[1]
src command:
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-m 4G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/seabios.log,id=seabios \
-vga qxl \
-spice port=5931,disable-ticketing \
-qmp tcp:0:4445,server,nowait \
-device ioh3420,id=root.0,slot=1 \
-device ioh3420,id=root.1,slot=2 \
-drive file=/home/demo/1.img-seabios,if=none,id=drive0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,id=virtio-disk0,drive=drive0,bus=root.1,bootindex=0 \
-monitor stdio \

dest command:
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-m 4G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/seabios1.log,id=seabios \
-vga qxl \
-spice port=5932,disable-ticketing \
-qmp tcp:0:4446,server,nowait \
-device ioh3420,id=root.0,slot=1 \
-device virtio-net-pci,netdev=dev1,mac=9a:6a:6b:6c:6d:6a,id=net1,bus=root.0 \
-netdev tap,id=dev1,vhost=on \
-device ioh3420,id=root.1,slot=2 \
-drive file=/home/demo/1.img-seabios,if=none,id=drive0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,id=virtio-disk0,drive=drive0,bus=root.1,bootindex=0 \
-monitor stdio \
-incoming tcp:0:5800 \

> Did you check PC, Q35 or both for virtio-net-pci?

  Yes.for PC:hot plugged/unplugged e1000e device or virtio-net-pci device,and no matter how many NIC in the system,the results is failed.
      for Q35:hot plugged e1000e device or virtio-net-pci device and it is not the only NIC in the system,the results is successful.hot unplugged a virtio-net-pci device and it has two NIC in the system,the results is still successful.
> Thanks,
> Marcel

Thanks,
jinchen

Hi,Marcel

  Sorry,due to my mistakes,for the PC,hot plugged/unplugged e1000 device rather than e1000e device,but the results is right,hot plugged/unplugged is failed.

Thanks,
jinchen

(In reply to jinchen from comment #7)
> Hi,Marcel
> 
>   Sorry,due to my mistakes,for the PC,hot plugged/unplugged e1000 device
> rather than e1000e device,but the results is right,hot plugged/unplugged is
> failed.
> 
> Thanks,
> jinchen

Thank you for the reply. I am having trouble understanding the results.
Please try to fill the following table:

machine - operation | virtio-nic | e1000   | e1000e  |  virtio block
-------------------------------------------------------------------------
PC - hotplug        |   ok/fail  | ok/fail |  -----  |     ok/fail  
-------------------------------------------------------------------------
PC - hot-unplug     |   ok/fail  | ok/fail |  ----   |     ok/fail 
-------------------------------------------------------------------------
Q35 - hotplug       |   ok/fail  |  -----  | ok/fail |     ok/fail  
-------------------------------------------------------------------------
Q35 - hot-unplug    |   ok/fail  |  ----   | ok/fail |     ok/fail 

Thanks,
Marcel

Hi David,

Can you please have a look to the migration command line
and see if the migration parameters are correct with respect
to hot-plug/hot-unplug operations? (e.g what the destination side command line should be if we hot-plug/hot-unplug a device before migration starts.)

Thanks,
Marcel

Jinchen:
  When you do hotplug, you must always specify the bus and address for *all* PCI/PCIe devices both on the commandline and when you hot-add them.
  If you don't specify it, then when you run the destination with a different command line with the unplugged device missing, other devices will change their auto-allocated slot numbers and the migration will be confused.

Please confirm if you can reproduce the bug when specifying all addresses and busses.

Marcel: There is a bug here - it looks like the RAMBlock associated with the e1000e isn't being deleted.

see the error: (qemu) qemu-kvm: Unknown ramblock "", cannot accept migration

that empty ramblock name is odd.
I added some debug to dump the list of RAMBlock names at the start of migrate and then did:

./x86_64-softmmu/qemu-system-x86_64 -nographic  -device e1000e,id=foo -m 1G -M pc,accel=kvm my.img

booted Linux
device_del foo

now the e1000e has gone from the 'info pci' but the RAMBlock is still there if I print out the list of RAMBlock's when I start the migrate.

According to comment 10,with address for *all* PCI/PCIe devices

machine - operation | virtio-nic | e1000   | e1000e  |  virtio block
-------------------------------------------------------------------------
PC - hotplug        |   fail     | fail    |  -----  |     fail  
-------------------------------------------------------------------------
PC - hot-unplug     |   fail     | fail    |  ----   |     fail 
-------------------------------------------------------------------------
Q35 - hotplug       |   fail     |  -----  | fail    |     fail  
-------------------------------------------------------------------------
Q35 - hot-unplug    |   fail     |  ----   | fail    |     fail

(In reply to Dr. David Alan Gilbert from comment #11)
> Marcel: There is a bug here - it looks like the RAMBlock associated with the
> e1000e isn't being deleted.
> 
> see the error: (qemu) qemu-kvm: Unknown ramblock "", cannot accept migration
> 
> that empty ramblock name is odd.

Empty ramblock name is set by qemu_ram_unset_idstr():

pci_qdev_unrealize()
-> pci_del_option_rom()
   -> vmstate_unregister_ram()
      -> qemu_ram_unset_idstr()
         -> memset(block->idstr, 0, sizeof(block->idstr));

pci_dev_unrealize() is called on the "device_del", so according to the code, an empty ROM name is what is expected on unpluging a PCI card. I think migration code should not send RAMblock with empty name.

(In reply to Laurent Vivier from comment #13)
> (In reply to Dr. David Alan Gilbert from comment #11)
> > Marcel: There is a bug here - it looks like the RAMBlock associated with the
> > e1000e isn't being deleted.
> > 
> > see the error: (qemu) qemu-kvm: Unknown ramblock "", cannot accept migration
> > 
> > that empty ramblock name is odd.
> 
> Empty ramblock name is set by qemu_ram_unset_idstr():
> 
> pci_qdev_unrealize()
> -> pci_del_option_rom()
>    -> vmstate_unregister_ram()
>       -> qemu_ram_unset_idstr()
>          -> memset(block->idstr, 0, sizeof(block->idstr));
> 
> pci_dev_unrealize() is called on the "device_del", so according to the code,
> an empty ROM name is what is expected on unpluging a PCI card. I think
> migration code should not send RAMblock with empty name.

The question though is why the RAMBlock isn't deleted rather than just having it's name unset.

There's probably quite a few places we'd have to skip a RAMBlock we wanted to avoid.

Dave

This appears in:

commit b0e56e0b63f350691b52d3e75e89bb64143fbeff
Author: Hu Tao <hutao.com>
Date:   Wed Apr 2 15:13:27 2014 +0800

    unset RAMBlock idstr when unregister MemoryRegion
    
    Signed-off-by: Hu Tao <hutao.com>
    Signed-off-by: Paolo Bonzini <pbonzini>

diff --git a/savevm.c b/savevm.c
index da8aa24..7b2c410 100644
--- a/savevm.c
+++ b/savevm.c
@@ -1209,7 +1209,7 @@ void vmstate_register_ram(MemoryRegion *mr, DeviceState *dev)
 
 void vmstate_unregister_ram(MemoryRegion *mr, DeviceState *dev)
 {
-    /* Nothing do to while the implementation is in RAMBlock */
+    qemu_ram_unset_idstr(memory_region_get_ram_addr(mr) & TARGET_PAGE_MASK);
 }
 
From
https://lists.nongnu.org/archive/html/qemu-devel/2014-04/msg00282.html

"When hotplug an memdev that was previously plugged and unplugged,
RAMBlock idstr is not cleared and triggers an assert error in
qemu_ram_set_idstr(). This series fixes it."

Unplugging the card calls pci_qdev_unrealize(), which unregister the PCI device memory (with do_pci_unregister_device()).

Then qemu_ram_free() is normally called by memory_region_finalize(). But memory_region_finalize() is not called because obj->ref is not 1 (checked in object_unref()).

Paolo has fixed the problem with:

http://patchwork.ozlabs.org/patch/736979/

diff --git a/hw/net/e1000e.c b/hw/net/e1000e.c
index b0f429b..6e23493 100644
--- a/hw/net/e1000e.c
+++ b/hw/net/e1000e.c
@@ -306,7 +306,7 @@  e1000e_init_msix(E1000EState *s)
 static void
 e1000e_cleanup_msix(E1000EState *s)
 {
-    if (msix_enabled(PCI_DEVICE(s))) {
+    if (msix_present(PCI_DEVICE(s))) {
         e1000e_unuse_msix_vectors(s, E1000E_MSIX_VEC_NUM);
         msix_uninit(PCI_DEVICE(s), &s->msix, &s->msix);
     }

 (In reply to jinchen from comment #12)
> According to comment 10,with address for *all* PCI/PCIe devices
> 
> machine - operation | virtio-nic | e1000   | e1000e  |  virtio block
> -------------------------------------------------------------------------
> PC - hotplug        |   fail     | fail    |  -----  |     fail  
> -------------------------------------------------------------------------
> PC - hot-unplug     |   fail     | fail    |  ----   |     fail 
> -------------------------------------------------------------------------
> Q35 - hotplug       |   fail     |  -----  | fail    |     fail  
> -------------------------------------------------------------------------
> Q35 - hot-unplug    |   fail     |  ----   | fail    |     fail

Can you please tests again with brew:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=12746756

Thanks,
Marcel

Hi Marcel

 Also failed with qemu-kvm-rhev-2.8.0-6.el7.test.x86_64 that you provided.

Thanks
Jing

(In reply to jingzhao from comment #19)
> Hi Marcel
> 
>  Also failed with qemu-kvm-rhev-2.8.0-6.el7.test.x86_64 that you provided.
> 
> Thanks
> Jing

With which error?
What exact command line were you using this time?

(In reply to Dr. David Alan Gilbert from comment #20)
> (In reply to jingzhao from comment #19)
> > Hi Marcel
> > 
> >  Also failed with qemu-kvm-rhev-2.8.0-6.el7.test.x86_64 that you provided.
> > 
> > Thanks
> > Jing
> 
> With which error?
> What exact command line were you using this time?

The same error with above
(qemu) qemu-kvm: Unknown ramblock "", cannot accept migration
qemu-kvm: error while loading state for instance 0x0 of device 'ram'
qemu-kvm: load of migration failed: Invalid argument

the qemu command line used:
[1] src host:
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-m 4G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/seabios.log,id=seabios \
-vga qxl \
-spice port=5931,disable-ticketing \
-qmp tcp:0:4445,server,nowait \
-device ioh3420,id=root.0,slot=1 \
-device e1000e,netdev=dev1,mac=9a:6a:6b:6c:6d:6a,id=net1,bus=root.0 \
-netdev tap,id=dev1,vhost=on \
-device ioh3420,id=root.1,slot=2 \
-drive file=/mnt/test/q35-seabios.qcow2,if=none,id=drive0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,id=virtio-disk0,drive=drive0,bus=root.1,bootindex=0 \
-monitor stdio \
-vnc :1 \

[2] delete net in src host 
(qemu) netdev_del dev1  
(qemu) device_del net1 

[3] In src host:
(qemu) migrate -d tcp:10.66.4.211:5800
(qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off 
Migration status: failed

[4]dest host:
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-m 4G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/seabios.log,id=seabios \
-vga qxl \
-vnc :0 \
-spice port=5931,disable-ticketing \
-qmp tcp:0:4445,server,nowait \
-device ioh3420,id=root.0,slot=1 \
-device ioh3420,id=root.1,slot=2 \
-drive file=/mnt/test/q35-seabios.qcow2,if=none,id=drive0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,id=virtio-disk0,drive=drive0,bus=root.1,bootindex=0 \
-monitor stdio \
-vnc :1 \
-incoming tcp:0:5800 \


Thanks
Jing

I can confirm it still fails with Marcel's rpm, but it does seem to work if I take Paolo's patch and apply it to upstream qemu.
Marcel what was in that build?

(In reply to Dr. David Alan Gilbert from comment #22)
> I can confirm it still fails with Marcel's rpm, but it does seem to work if
> I take Paolo's patch and apply it to upstream qemu.
> Marcel what was in that build?

Well, it was supposed to be the latest qemu-kvm-rhev with Paolo's patch applied, now I am not sure anymore...

Thanks,
Marcel

Set to post as Paolo's patch has landed in v2.9.0-rc0.

1.Reproduce the bz on qemu-kvm-rhev-2.8.0-6.el7.x86_64

2.Also failed on qemu-kvm-rhev-2.9.0-1.el7.x86_64 & kernel-3.10.0-657.el7.x86_64

Following is the detailed info:

1) boot guest with qemu command line [1]

2) unplug e1000e device in the src
(qemu) device_del net1  
(qemu) netdev_del dev1 

3) do the migration in the src
(qemu) migrate -d tcp:10.66.6.246:5800
(qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off 
Migration status: active
total time: 2002 milliseconds
expected downtime: 300 milliseconds
setup: 13 milliseconds
transferred ram: 68846 kbytes
throughput: 268.58 mbps
remaining ram: 3439744 kbytes
total ram: 4325840 kbytes
duplicate: 204795 pages
skipped: 0 pages
normal: 16729 pages
normal bytes: 66916 kbytes
dirty sync count: 1
(qemu) mig
migrate                 migrate_cancel          migrate_incoming        
migrate_set_cache_size  migrate_set_capability  migrate_set_downtime    
migrate_set_parameter   migrate_set_speed       migrate_start_postcopy  
(qemu) migrate_se
migrate_set_cache_size  migrate_set_capability  migrate_set_downtime    
migrate_set_parameter   migrate_set_speed       
(qemu) migrate_set_downtime 1  
(qemu) migrate_set_speed 1G
(qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off 
Migration status: completed
total time: 9218 milliseconds
downtime: 189 milliseconds
setup: 13 milliseconds
transferred ram: 1331923 kbytes
throughput: 1183.84 mbps
remaining ram: 0 kbytes
total ram: 4325840 kbytes
duplicate: 761749 pages
skipped: 0 pages
normal: 330661 pages
normal bytes: 1322644 kbytes
dirty sync count: 3

4) check the status in dest:
(qemu) red_dispatcher_loadvm_commands: 
qemu-kvm: Unknown savevm section or instance '0000:00:02.0/pcie-root-port' 0
qemu-kvm: load of migration failed: Invalid argument

[1]
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-m 4G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/seabios.log,id=seabios \
-vga qxl \
-vnc :0 \
-qmp tcp:0:4445,server,nowait \
-device pcie-root-port,id=root.0,slot=1 \
-device e1000e,netdev=dev1,mac=9a:6a:6b:6c:6d:6a,id=net1,bus=root.0 \
-netdev tap,id=dev1,vhost=on \
-device pcie-root-port,id=root.1,slot=2 \
-drive file=/home/test/rhel/rhel74.qcow2,if=none,id=drive0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,id=virtio-disk0,drive=drive0,bus=root.1,bootindex=0 \
-monitor stdio \


According to above info, change the bz status to assign

(In reply to jingzhao from comment #27)
> 1.Reproduce the bz on qemu-kvm-rhev-2.8.0-6.el7.x86_64
> 
> 2.Also failed on qemu-kvm-rhev-2.9.0-1.el7.x86_64 &
> kernel-3.10.0-657.el7.x86_64
> 
> Following is the detailed info:
> 
> 1) boot guest with qemu command line [1]
...
> 4) check the status in dest:
> (qemu) red_dispatcher_loadvm_commands: 
> qemu-kvm: Unknown savevm section or instance '0000:00:02.0/pcie-root-port' 0
> qemu-kvm: load of migration failed: Invalid argument

You don't test with the same command line, it looks like another bug, on pcie, not e1000.

Hi Jing,
  Please test it with the ioh3420 as per the original bug and make sure the original bug is fixed.
  Please test again with the pcie-root-port and file a separate bug if that fails - I can't reproduce it here.

Actually, I *can* reproduce this - I'll file a separate bz for it

(In reply to Dr. David Alan Gilbert from comment #31)
> Actually, I *can* reproduce this - I'll file a separate bz for it

I had filed a new bz for tracking the new bz (bz 1446080). 

Thanks
Jing

(In reply to Dr. David Alan Gilbert from comment #30)
> Hi Jing,
>   Please test it with the ioh3420 as per the original bug and make sure the
> original bug is fixed.
>   Please test again with the pcie-root-port and file a separate bug if that
> fails - I can't reproduce it here.

Hi David

Also failed with ioh3420 device, the failed info is same with the pcie-root-port device. following is the failed info

(qemu) red_dispatcher_loadvm_commands: 
qemu-kvm: Unknown savevm section or instance '0000:00:02.0/ioh-3240-express-root-port' 0
qemu-kvm: load of migration failed: Invalid argument

The different failed info, can we conside the bz fixed? 


Thanks
Jing

(In reply to jingzhao from comment #33)
> (In reply to Dr. David Alan Gilbert from comment #30)
> > Hi Jing,
> >   Please test it with the ioh3420 as per the original bug and make sure the
> > original bug is fixed.
> >   Please test again with the pcie-root-port and file a separate bug if that
> > fails - I can't reproduce it here.
> 
> Hi David
> 
> Also failed with ioh3420 device, the failed info is same with the
> pcie-root-port device. following is the failed info
> 
> (qemu) red_dispatcher_loadvm_commands: 
> qemu-kvm: Unknown savevm section or instance
> '0000:00:02.0/ioh-3240-express-root-port' 0
> qemu-kvm: load of migration failed: Invalid argument
> 
> The different failed info, can we conside the bz fixed? 
> 
> 
> Thanks
> Jing

Jing
  You must specify 'addr=' on all PCI and PCIe devices to ensure they keep the same address on source/destination, e.g. for:

-device pcie-root-port,id=root.0,slot=1 \

you need

-device pcie-root-port,id=root.0,slot=1,addr=4 

(I picked 4, but you need to find a free number).

Thanks David's help

According to comment 34 and migrated successfully when pcie with addr parameter, so according to comment 27, comment 34 can be verified the bz on  qemu-kvm-rhev-2.9.0-1.el7.x86_64.

Thanks
Jing

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392