Bug 1529454

Can reproduce it on libvirt but can't reproduce on qemu when using local nfs server.(nfs server is on the same host with vms.)

Reproduce by qemu：
  kernel version:3.10.0-824.el7.x86_64
  qemu version:qemu-kvm-rhev-2.10.0-13.el7.x86_64

Test steps:
 1.Create an img with format luks
    qemu-img create -f luks --object secret,id=sec0,data=`printf %s "redhat"|base64`,format=base64 -o key-secret=sec0 luks.img 1G

 2.Create a local nfs server, then mount it.

 3.Start vm1 with qemu cmds:
    /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga cirrus  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20171102-094326-DoIK24Xo,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel75-64-virtio-scsi.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bus=pci.0 \
    -object secret,id=sec0,data=`printf %s "redhat" |base64`,format=base64 \
    -drive file=/mnt/luks.img,key-secret=sec0,driver=luks,id=drive_data1,if=none \
    -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \
    -device virtio-net-pci,mac=9a:12:13:14:15:16,id=id9HuGl4,vectors=4,netdev=idpS7PTn,bus=pci.0  \
    -netdev tap,id=idpS7PTn,vhost=on \
    -m 2690  \
    -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
    -cpu 'Opteron_G2',+kvm_pv_unhalt \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=c \
    -enable-kvm \
    -monitor stdio \
 4. Start vm2 with cmds:
    /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga cirrus  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20171102-094326-DoIK24XP,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/rhel75-64-virtio-scsi.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bus=pci.0 \
    -device virtio-net-pci,mac=9a:12:13:14:15:17,id=id9HuGl4,vectors=4,netdev=idpS7PTn,bus=pci.0  \
    -netdev tap,id=idpS7PTn,vhost=on \
    -m 2048  \
    -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
    -cpu 'Opteron_G2',+kvm_pv_unhalt \
    -vnc :1  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=c \
    -enable-kvm \
    -monitor stdio \
 5. Hotplug luks.img to vm2.
   {'execute':'__com.redhat_drive_add', 'arguments': {'file':'/home/luks.img','format':'qcow2','id':'drive_data1','cache':'none','aio':'native'}}
   result: Hotplug fail.

 6. Hotplug luks.img from vm1.
   {'execute':'device_del','arguments':{'id':'data1'}}

Actual Result:
  After step4, device hot unplug successfully, but guest reboot with bug 1530577.

Additional info:
 When nfs server is not on the same host with vms, don't hit the bug by libvirt.

Here are some more clues when I tried to reproduce this issue with virsh cmd, pls check if it helps.


1. only happens in nfs mount dir
2. only happens when luks image with mode=600, if changed to mode=644, everything will be ok
3. when attach the device to the second vm with expected failure, the owner/group of that image changed to root, as follow:
## ll /mnt/luks.img -Z
-rw-------. qemu qemu system_u:object_r:nfs_t:s0       /mnt/luks.img

## virsh attach-device avocado-vt-vm2 disk.xml
error: Failed to attach device from disk.xml
error: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-scsi0-0-0-1' could not be initialized

## ll /mnt/luks.img -Z
-rw-------. root root system_u:object_r:nfs_t:s0       /mnt/luks.img

=====
and following qemu log generated when crash happens.

Unexpected error in raw_apply_lock_bytes() at block/file-posix.c:642:          
Failed to lock byte 100                                                        
2017-12-27 05:54:01.262+0000: shutting down, reason=crashed

(In reply to aihua liang from comment #2)
> Can reproduce it on libvirt but can't reproduce on qemu when using local nfs
> server.(nfs server is on the same host with vms.)
> 
> Reproduce by qemu：
>   kernel version:3.10.0-824.el7.x86_64
>   qemu version:qemu-kvm-rhev-2.10.0-13.el7.x86_64
> 
> Test steps:
>  1.Create an img with format luks
>     qemu-img create -f luks --object secret,id=sec0,data=`printf %s
> "redhat"|base64`,format=base64 -o key-secret=sec0 luks.img 1G
> 
>  2.Create a local nfs server, then mount it.
> 
>  3.Start vm1 with qemu cmds:
>     /usr/libexec/qemu-kvm \
>     -name 'avocado-vt-vm1'  \
>     -sandbox off  \
>     -machine pc  \
>     -nodefaults  \
>     -vga cirrus  \
>     -chardev
> socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20171102-
> 094326-DoIK24Xo,server,nowait \
>     -mon chardev=qmp_id_qmpmonitor1,mode=control  \
>     -drive
> id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,
> file=/home/kvm_autotest_root/images/rhel75-64-virtio-scsi.qcow2 \
>     -device virtio-blk-pci,id=image1,drive=drive_image1,bus=pci.0 \
>     -object secret,id=sec0,data=`printf %s "redhat" |base64`,format=base64 \
>     -drive
> file=/mnt/luks.img,key-secret=sec0,driver=luks,id=drive_data1,if=none \
>     -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \
>     -device
> virtio-net-pci,mac=9a:12:13:14:15:16,id=id9HuGl4,vectors=4,netdev=idpS7PTn,
> bus=pci.0  \
>     -netdev tap,id=idpS7PTn,vhost=on \
>     -m 2690  \
>     -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
>     -cpu 'Opteron_G2',+kvm_pv_unhalt \
>     -vnc :0  \
>     -rtc base=utc,clock=host,driftfix=slew  \
>     -boot menu=off,strict=off,order=cdn,once=c \
>     -enable-kvm \
>     -monitor stdio \
>  4. Start vm2 with cmds:
>     /usr/libexec/qemu-kvm \
>     -name 'avocado-vt-vm1'  \
>     -sandbox off  \
>     -machine pc  \
>     -nodefaults  \
>     -vga cirrus  \
>     -chardev
> socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20171102-
> 094326-DoIK24XP,server,nowait \
>     -mon chardev=qmp_id_qmpmonitor1,mode=control  \
>     -drive
> id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,
> file=/home/rhel75-64-virtio-scsi.qcow2 \
>     -device virtio-blk-pci,id=image1,drive=drive_image1,bus=pci.0 \
>     -device
> virtio-net-pci,mac=9a:12:13:14:15:17,id=id9HuGl4,vectors=4,netdev=idpS7PTn,
> bus=pci.0  \
>     -netdev tap,id=idpS7PTn,vhost=on \
>     -m 2048  \
>     -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
>     -cpu 'Opteron_G2',+kvm_pv_unhalt \
>     -vnc :1  \
>     -rtc base=utc,clock=host,driftfix=slew  \
>     -boot menu=off,strict=off,order=cdn,once=c \
>     -enable-kvm \
>     -monitor stdio \
>  5. Hotplug luks.img to vm2.
>    {'execute':'__com.redhat_drive_add', 'arguments':
> {'file':'/home/luks.img','format':'qcow2','id':'drive_data1','cache':'none',
> 'aio':'native'}}
>    result: Hotplug fail.
> 
>  6. Hotplug luks.img from vm1.
>    {'execute':'device_del','arguments':{'id':'data1'}}
> 
> Actual Result:
>   After step4, device hot unplug successfully, but guest reboot with bug
> 1530577.
> 
> Additional info:
>  When nfs server is not on the same host with vms, don't hit the bug by
> libvirt.


Yes,according to yisun's description in comment 3，the unpluged img's permission is also one key point.

Retest it with remote nfs server with img mode=644, can also reproduce the bug by libvirt, but can't still reproduce it by qemu.

(In reply to yisun from comment #3)
> Here are some more clues when I tried to reproduce this issue with virsh
> cmd, pls check if it helps.
> 
> 
> 1. only happens in nfs mount dir
> 2. only happens when luks image with mode=600, if changed to mode=644,
> everything will be ok
...

> and following qemu log generated when crash happens.
> 
> Unexpected error in raw_apply_lock_bytes() at block/file-posix.c:642:       
> 
> Failed to lock byte 100                                                     

There may be both a qemu problem (for crashing instead of a graceful error on permissions errors) and a libvirt problem (for not setting correct permissions when NFS is involved on a LUKS block device).  We'll need to do more investigations on where the root cause(s) are; leaving this bug assigned to qemu for now.

Created attachment 1565150 [details]
core dump log

Tested with latest rhel7.7, not hit this issue. qemu not crash, just prompts warning message.

(With the same steps that mentioned in Comment 10)


Details:

Versions:
Host:
kernel-3.10.0-1053.el7.x86_64
qemu-kvm-rhev-2.12.0-32.el7

Guest:
win2019-64 with virtio-win-prewhql-0.1-163.iso



After step 5,  qemu not aborted.


(qemu) info block
drive-ua-3518b268-45bf-4172-858c-ff6e896e6d66 (#block135): /home/kvm_autotest_root/images/win2019-64-virtio-scsi.qcow2 (qcow2)
    Attached to:      ua-3518b268-45bf-4172-858c-ff6e896e6d66
    Cache mode:       writeback, direct

drive-ua-24adc352-fe0d-491e-86ca-58f379892847 (#block300): /home/nfs_cdrom_test/test.iso (raw, read-only)
    Attached to:      ua-24adc352-fe0d-491e-86ca-58f379892847
    Removable device: not locked, tray closed
    Cache mode:       writeback
  
(qemu) eject drive-ua-24adc352-fe0d-491e-86ca-58f379892847 
warning: Failed to unlock byte 100
warning: Failed to unlock byte 201
   
(qemu) info block
drive-ua-3518b268-45bf-4172-858c-ff6e896e6d66 (#block135): /home/kvm_autotest_root/images/win2019-64-virtio-scsi.qcow2 (qcow2)
    Attached to:      ua-3518b268-45bf-4172-858c-ff6e896e6d66
    Cache mode:       writeback, direct

drive-ua-24adc352-fe0d-491e-86ca-58f379892847: [not inserted]
    Attached to:      ua-24adc352-fe0d-491e-86ca-58f379892847
    Removable device: not locked, tray open

(qemu) info status 
VM status: running
     
(qemu) system_powerdown



I found this bug is reported for RHEL7 in the beginning. And it was moved to RHEL8 by Ademar. QE thought it won't be fixed on RHEL7.

Ademar Reis 2018-12-14 16:09:26 UTC
Component: qemu-kvm-rhev → qemu-kvm
Version: 7.5 → ---
Product: Red Hat Enterprise Linux 7 → Red Hat Enterprise Linux Advanced Virtualization
Flags: rhel-7.7.0?

(In reply to Xueqiang Wei from comment #24)
> Tested with latest rhel7.7, not hit this issue. qemu not crash, just prompts
> warning message.
>
> I found this bug is reported for RHEL7 in the beginning. And it was moved to
> RHEL8 by Ademar. QE thought it won't be fixed on RHEL7.
> 
> Ademar Reis 2018-12-14 16:09:26 UTC
> Component: qemu-kvm-rhev → qemu-kvm
> Version: 7.5 → ---
> Product: Red Hat Enterprise Linux 7 → Red Hat Enterprise Linux Advanced
> Virtualization
> Flags: rhel-7.7.0?

Hi Ademar,

Could you please confirm the status of this bug ?
Do we need a RHEL.7 clone ?

Thanks.

Answering for Ademar on this...

True - this bug was moved to RHEL AV for capacity reasons; however, as you see above starting at comment 13 there was an escalation and interest for a customer that has a RHEL7 environment. I assumed that by answering the needinfo on Eric in comment 12 that the bug would have got a retest. At the time I wasn't even thinking RHEL7/RHELAV -  it was more it looks like it's related to other changes.

Anyway - I really don't think there's a need to clone this to RHEL7. At this point, it would be a clone of bz 1603104. I suppose if you felt compelled to log this work somehow you could clone and then duplicate.  Similarly for RHEL8 there is already bz 1694148 and RHEL AV bz 1652572. I probably will end up moving this to RHEL7 and closing it as a duplicate of 1603104 since that where the customer case ended up. This course of action would be helpful I think if there was some sort of z-stream request.

In the long run, I hope the customer will be happy the core issue is resolved.

Thanks to Xueqiang Wei for doing the validation so quickly!

Suffice to say the history on this is really confusing.

It seems between comment 6 and 24 the focus of the original problem was altered to a subset based on receiving the same error noted in comment 5 for a different code path. It would seem that the error shown in comment 5 matched what was shown in comment 6 and by comment 10 we had a reproducer. By comment 24 a determination is made that the issue noted/raised starting in comment 6 is resolved. That still leaves the original issue and those noted in comment 2 and comment 3 not fully understood.

In comment 27, Max notes a similar problem using a different path with perhaps a bit of contradiction to the original problem at least with respect to:

>> Additional info:
>> if we don't use nfs server,just test the above scenario using local img,can not reproduce this issue
>>

So going back to the original issue, we still need to determine if those are a problem.

Furthermore, comment 3 shows a somewhat interesting way to utilize the default directory storage pool as the mount point for some NFS server. The storage pool XML isn't shown, but I'll assume it starts with "<pool type='dir'>" instead of "<pool type='netfs'>".  Why is this important?  Well there's quite a bit of code in libvirt that handles NFS storage pools a bit differently - especially as it relates to file protections and security labels which perhaps had an impact. Beyond that I recall note that 'qemu-img' and 'virsh vol-create-as' do have some differences when it comes to permissions/protections on files. There's also this thing in NFS known as rootsquash which could come into play, but perhaps not so much in this case.

Whether the storage pool type is a red herring or not hasn't been fully investigated yet. There's just not enough information. Knowing whether there was any sort of <permissions> data applied from the storage pool XML or whether the defaults were taken I'm not sure. Would using virsh vol-create instead of qemu-img do anything different? Would using an NFS pool vs. a directory pool have a difference.  Quite a few variables in play.

I do think what's noted in comment 2:

>> Additional info:
>>  When nfs server is not on the same host with vms, don't hit the bug by libvirt.
>>

Would seem to indicate usage of a "real" NFS storage pool is different than when using a directory storage pool that's mounted as an NFS client.  There *is* a difference within libvirt.

I think it would be a better to go back to the original problem and retest using the most recent rhel-7.7 bits. Try to ignore anything after comment 5 because I believe that's a different, but related problem.

When it comes to the storage pool type concerns - those may be different, let's try to separate them - please be specific. If we need to create more bzs based on different ways to configure, then so be it. I don't want to assume anything. Hopefully I've provided enough data to help set up the tests, but if not, just ask.

(In reply to John Ferlan from comment #28)
> Suffice to say the history on this is really confusing.
> 
> It seems between comment 6 and 24 the focus of the original problem was
> altered to a subset based on receiving the same error noted in comment 5 for
> a different code path. It would seem that the error shown in comment 5
> matched what was shown in comment 6 and by comment 10 we had a reproducer.
> By comment 24 a determination is made that the issue noted/raised starting
> in comment 6 is resolved. That still leaves the original issue and those
> noted in comment 2 and comment 3 not fully understood.
> 
> In comment 27, Max notes a similar problem using a different path with
> perhaps a bit of contradiction to the original problem at least with respect
> to:
> 
> >> Additional info:
> >> if we don't use nfs server,just test the above scenario using local img,can not reproduce this issue
> >>
> 
> So going back to the original issue, we still need to determine if those are
> a problem.
> 
> Furthermore, comment 3 shows a somewhat interesting way to utilize the
> default directory storage pool as the mount point for some NFS server. The
> storage pool XML isn't shown, but I'll assume it starts with "<pool
> type='dir'>" instead of "<pool type='netfs'>".  Why is this important?  Well
> there's quite a bit of code in libvirt that handles NFS storage pools a bit
> differently - especially as it relates to file protections and security
> labels which perhaps had an impact. Beyond that I recall note that
> 'qemu-img' and 'virsh vol-create-as' do have some differences when it comes
> to permissions/protections on files. There's also this thing in NFS known as
> rootsquash which could come into play, but perhaps not so much in this case.
> 
> Whether the storage pool type is a red herring or not hasn't been fully
> investigated yet. There's just not enough information. Knowing whether there
> was any sort of <permissions> data applied from the storage pool XML or
> whether the defaults were taken I'm not sure. Would using virsh vol-create
> instead of qemu-img do anything different? Would using an NFS pool vs. a
> directory pool have a difference.  Quite a few variables in play.
> 
> I do think what's noted in comment 2:
> 
> >> Additional info:
> >>  When nfs server is not on the same host with vms, don't hit the bug by libvirt.
> >>
> 
> Would seem to indicate usage of a "real" NFS storage pool is different than
> when using a directory storage pool that's mounted as an NFS client.  There
> *is* a difference within libvirt.
> 
> I think it would be a better to go back to the original problem and retest
> using the most recent rhel-7.7 bits. Try to ignore anything after comment 5
> because I believe that's a different, but related problem.
> 
> When it comes to the storage pool type concerns - those may be different,
> let's try to separate them - please be specific. If we need to create more
> bzs based on different ways to configure, then so be it. I don't want to
> assume anything. Hopefully I've provided enough data to help set up the
> tests, but if not, just ask.

I retest this bug with libvirt-4.5.0-22.virtcov.el7.x86_64 qemu-kvm-rhev-2.12.0-32.el7.x86_64 but not reproduced:
1. Prepare nfs mount and luks images
2. Start a vm with luks image:
➜  ~ virsh domblklist avocado-vt-vm1 
Target     Source
------------------------------------------------
vda        /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2
fda        /tmp/a
sdb        /var/lib/libvirt/images/nfs/luks.img

3. Live attach to another vm and then detach it
➜  ~ virsh attach-device PC luks-disk.xml
error: Failed to attach device from luks-disk.xml
error: internal error: unable to execute QEMU command 'device_add': Failed to get "write" lock

➜  ~ virsh detach-disk avocado-vt-vm1 sdb
Disk detached successfully

I believe the customer problem noted from comment 6 to comment 24 was more relevant to bz 1603104. I have added a comment in that bz to trace back to here.

The fix for bz#1603104 will be in rhel-7.7.0 which has not shipped - the ask for that probably should be done through the other bz. The last email on this that I got is the customer was informed that a fix would be in 7.7 and there is hope they will not ask for a 7.6.z backport.

It's also my belief now given the results above that this bz would be duplicated to bz#1603104. But yes, I'm also hesitating slightly...

From what I understand bz#1722090 is a bit different (it doesn't involve NFS like the customer case did).

I'm keeping this on my short term radar.

*** Bug 1728461 has been marked as a duplicate of this bug. ***

Minor update to C31, the to duplicate bug to is bz#1709791 since that's for RHEL AV

*** This bug has been marked as a duplicate of bug 1709791 ***