Bug 1463168

Summary: can't update the cdrom after ejecting cdrom in the guest and then doing save/restore operation
Product: Red Hat Enterprise Linux 7 Reporter: lijuan men <lmen>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: lijuan men <lmen>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.4CC: dyuan, jsuchane, lmen, lmiksik, pkrempa, rbalakri, xuzhang
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-3.9.0-11.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 10:50:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
dest_libvirtd.log none

Description lijuan men 2017-06-20 09:52:27 UTC
Description of problem:
can't update the cdrom after ejecting cdrom in the guest and then doing save/restore operation

Version-Release number of selected component (if applicable):
libvirt-3.2.0-10.el7.x86_64
qemu-kvm-rhev-2.9.0-12.el7.x86_64

How reproducible:
100%

Steps to Reproduce:

1.start a guest with cdrom:
 <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/rhel7.3.iso'/>
      <target dev='sda' bus='scsi'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
[root@lmen1 ~]# virsh start test
Domain test started


2.eject the iso file in the guest
in the guest,run the command:
#eject /dev/sr0

3.save and restore the guest

[root@lmen1 ~]# virsh save test /tmp/save
Domain test saved to /tmp/save

[root@lmen1 ~]# virsh restore /tmp/save
Domain restored from /tmp/save

[root@lmen1 ~]# virsh dumpxml test
...
 <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <readonly/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
...

4.use change-media to update iso file
[root@lmen1 ~]# virsh domblklist test
Target     Source
------------------------------------------------
vda        /var/lib/libvirt/images/uefi.qcow2
sda        -

[root@lmen1 ~]# virsh change-media test sda /var/lib/libvirt/images/rhel7.3.iso --update
error: Failed to complete action update on media
error: internal error: timed out waiting for disk tray status update


Actual results:
failed to update

Expected results:
update successfully

Additional info:
If I migrate the guest after  ejecting cdrom in the guest,the cdrom can not be updated too.

Comment 2 Peter Krempa 2017-09-27 09:40:31 UTC
Fixed upstream:

commit 4b480d10768c7a288c10e2e39f6a2bf275a0eb69 
Author: Peter Krempa <pkrempa>
Date:   Mon Sep 25 16:16:08 2017 +0200

    qemu: process: Refresh data from qemu monitor after migration
    
    Some values we read from the qemu monitor may be changed with the actual
    state by the incoming migration. This means that we should refresh
    certain things only after the migration has finished.
    
    This is mostly visible in the cdrom tray state, which is by default
    closed but may be opened by the guest OS. This would be refreshed before
    qemu transferred the actual state and thus libvirt would think that the
    tray is closed.
    
    Note that this patch moves only a few obvious query commands. Others may
    be moved later after individual assessment.

Comment 5 Peter Krempa 2018-01-31 16:44:08 UTC
Could you please add debug logs from the destination side?

Comment 6 lijuan men 2018-02-01 05:46:35 UTC
Created attachment 1389344 [details]
dest_libvirtd.log

Comment 7 lijuan men 2018-02-01 05:48:49 UTC
(In reply to Peter Krempa from comment #5)
> Could you please add debug logs from the destination side?

upload the log
libvirt version is libvirt-3.9.0-10.el7.x86_64

Comment 8 Peter Krempa 2018-02-01 18:39:23 UTC
(In reply to lijuan men from comment #4)
> version:
> libvirt-3.9.0-6.el7.x86_64
> qemu-kvm-rhev-2.10.0-14.el7.x86_64
> 
> 
> the steps in description part can be verified,the issue is fixed
> 
> but for the "Additional info" part:
> If I migrate the guest after  ejecting cdrom in the guest,the cdrom can not
> be updated too.
> 
> I can still reproduce it:
> 
> prepare the env:
> 1)prepare a nfs server and 2 hosts,and mount the nfs server to the hosts
> #mount -t nfs server-IP:/dir /var/lib/libvirt/images/
> 
> steps:
> 1.start a guest with xml:
>   <disk type='file' device='cdrom'>
>       <driver name='qemu' type='raw'/>
>       <source file='/var/lib/libvirt/images/a.iso'/>
>       <target dev='sda' bus='scsi'/>
>       <readonly/>
>     </disk>
> [root@lmen1 ~]# virsh start test
> Domain test started
> 
> 
> 2.eject the iso file in the guest
> in the guest,run the command:
> #eject /dev/sr0
> 
> 3.migrate the guest to another host:
> [root@lmen1 ~]# virsh migrate test --live --unsafe
> qemu+ssh://HOSTNAME/system
> 
> 4.use change-media to update iso file in the destination host:
> 
> [root@intel-e52650-16-1 ~]# virsh domblklist test
> Target     Source
> ------------------------------------------------
> vda        /var/lib/libvirt/images/RHEL.qcow2
> sda        -
> 
> [root@intel-e52650-16-1 ~]#  virsh change-media test sda
> /var/lib/libvirt/images/a.iso 
> error: Failed to complete action update on media
> error: internal error: timed out waiting for disk tray status update

So the code I've added by the patch mentioned above was supposed to fix also the migration case but I failed to notice that the code path I've selected actually does not wait for the migration to finish at the point where we reload the tray state, so for remote migration the code will still not work.

I've sent a patch fixing this issue upstream:

https://www.redhat.com/archives/libvir-list/2018-February/msg00052.html

Comment 11 Peter Krempa 2018-02-02 09:56:57 UTC
I've pushed the patch mentioned in commit 8 upstream:

commit 93db7eea1b86408e02852a8c886d473f4f5007e3
Author: Peter Krempa <pkrempa>
Date:   Thu Feb 1 15:02:17 2018 +0100

    qemu: migration: Refresh device information after transferring state
    
    In my first approach in 4b480d10768c I overlooked the comment in
    qemuMigrationRunIncoming stating that during actual migration the
    qemuMigrationRunIncoming does not wait until the migration is complete
    but rather offloads that to the Finish phase of migration.
    
    This means that during actual migration qemuProcessRefreshState was
    called prior to qemu actually transferring the full state and thus the
    queries did not get the correct information. The approach worked only
    for restore, where we wait for the migration to finish during qemu
    startup.
    
    Fix the issue by calling qemuProcessRefreshState both from
    qemuProcessStart if there's no incomming migration and from
    qemuMigrationFinish so that the code actually works as expected.

Comment 15 lijuan men 2018-02-08 09:49:41 UTC
> Does the file exist on the destination? If yes, can it be accessed if you
> close the tray from the guest and try to mount it?
> 
> If yes, then it's partially wrong. If the media is not accessible it's okay.
> 

thanks,Peter

I tried it,the media is not accessible

So,based on the above comments,mark the bug as verified

Comment 19 errata-xmlrpc 2018-04-10 10:50:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0704