Bug 1232106

Summary: Disk hotplug/hot-unplug didn't works well in power pc with ppc64le
Product: Red Hat Enterprise Linux 7 Reporter: zhenfeng wang <zhwang>
Component: libvirtAssignee: Andrea Bolognani <abologna>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.2CC: abologna, dyuan, dzheng, ebenahar, gsun, hannsj_uhl, mzhan, rbalakri
Target Milestone: rc   
Target Release: ---   
Hardware: ppc64le   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-13 09:06:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1201513    

Description zhenfeng wang 2015-06-16 05:13:37 UTC
Description of problem:
Found 2 issues during hotplug/hot-unplug disk
1.The guest os could detect the persistent disk, couldn't detact the hotplug disk
2.the fdisk command didn't work after hot-unplug the disk


Version-Release number:
kernel-3.10.0-244.ael7b.ppc64le
libvirt-1.2.16-1.el7.ppc64le
qemu-kvm-rhev-2.3.0-2.el7.ppc64le


How reproducible:
100%

Steps to Reproduce:
1.Start a ppc64LEguest on the ppc64LE host
#virsh start virt-tests-vm1

2.Create a image
#qemu-img create /var/lib/libvirt/images/test.img 100M

3.Attach the img to the guest, could see the image in the guest's xml,
# virsh attach-disk virt-tests-vm1 /var/lib/libvirt/images/test.img sdb
Disk attached successfully

#virsh dumpxml virt-tests-vm1
--
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/test.img'/>
      <backingStore/>
      <target dev='sdb' bus='scsi'/>
      <alias name='scsi0-0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>

4.Check the image inside the guest, however couldn't see the new attached disk
guest#fdisk -l

5.Destroy the guest and add the disk in the guest's xml
#virsh dumpxml virt-tests-vm1
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/test.img'/>
      <backingStore/>
      <target dev='sdb' bus='scsi'/>
      <alias name='scsi0-0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>

6.Start the guest, after the guest start completely, could see the new attached disk
with the fdisk -l command

# fdisk -l

Disk /dev/sdb: 104 MB, 104857600 bytes, 204800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

7.Detach the disk, then login the guest, found the fdisk command couldn't
list the disk info of the system anymore

# virsh detach-disk virt-tests-vm1 sdb
Disk detached successfully

guest# fdisk -l
fdisk: cannot open /dev/sdb: Input/output error


Actual results:
1.The guest os could detect the persistent disk, couldn't detact the hotplug disk
2.the fdisk command didn't work after hot-unplug the disk

Expected results:
should work well

Comment 2 Dan Zheng 2015-07-28 08:39:43 UTC
I also tested with attach-disk virtio.

Steps:
1. Start the guest
2. Run aatach-disk
 # virsh attach-disk dzhengvm1 /var/lib/virt_test/images/test3.img vdd --live --config --iothread 3 --targetbus virtio --driver qemu --subdriver raw --type disk
Disk attached successfully
3. Check the dumpxml and the disk is already inserted.
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' iothread='3'/>
      <source file='/var/lib/virt_test/images/test3.img'/>
      <target dev='vdd' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
4. Check within guest, but can not find the disk via fdisk. 
5. If I configured the same disk in XML and start the guest, then the disk can be listed in guest via fdisk.


Could you please help confirm if they are due to same root cause? If not , I would like file a new bug.

Comment 3 Andrea Bolognani 2015-08-11 09:54:22 UTC
In the first case, you're attaching a disk to the SCSI bus. It's
not enough to simply attach it for the guest to use it, you need
to ask the guest to scan the SCSI bus using something like

  for h in /sys/class/scsi_host/host*; do
    echo '- - -' >"$h/scan"
  done

Likewise, you need to ask the guest to release the disk using
something like

  echo 1 >/sys/class/block/sdb/device/delete

before detaching it using virsh commands. None of this is
specific to ppc64: you get the same behaviour on x86.

In the second case, though, it should work just fine, and indeed
it does in my test environment.

I'm using the following packages:

  libvirt-daemon-1.2.17-4.el7.ppc64le
  qemu-kvm-rhev-2.3.0-16.el7.ppc64le
  kernel-3.10.0-302.el7.ppc64le

Can you please confirm the issue is still present after updating
the system? And if that's the case, can you please provide the
full guest XML configuration?

Comment 4 Dan Zheng 2015-08-13 02:14:25 UTC
(In reply to Andrea Bolognani from comment #3)
> In the first case, you're attaching a disk to the SCSI bus. It's
> not enough to simply attach it for the guest to use it, you need
> to ask the guest to scan the SCSI bus using something like
> 
>   for h in /sys/class/scsi_host/host*; do
>     echo '- - -' >"$h/scan"
>   done
> 
> Likewise, you need to ask the guest to release the disk using
> something like
> 
>   echo 1 >/sys/class/block/sdb/device/delete
> 
> before detaching it using virsh commands. None of this is
> specific to ppc64: you get the same behaviour on x86.
> 
> In the second case, though, it should work just fine, and indeed
> it does in my test environment.
> 
> I'm using the following packages:
> 
>   libvirt-daemon-1.2.17-4.el7.ppc64le
>   qemu-kvm-rhev-2.3.0-16.el7.ppc64le
>   kernel-3.10.0-302.el7.ppc64le
> 
> Can you please confirm the issue is still present after updating
> the system? And if that's the case, can you please provide the
> full guest XML configuration?

Hi Andrea,
Using same packages as yours, it also works for me now. Thanks.

Comment 5 Andrea Bolognani 2015-08-13 09:06:32 UTC
Closing the bug then. Thanks.

Comment 6 Dan Zheng 2015-08-14 01:34:44 UTC
(In reply to Andrea Bolognani from comment #5)
> Closing the bug then. Thanks.

Andrea,
Sorry, my comment is not accurate. I did test only for the second case, and did not test the first (scsi) case. So we need future test.
Could you please change the status back? Thank you.

Comment 7 zhenfeng wang 2015-08-14 03:14:59 UTC
Hi Andrea
Retry the comment0's issue with the latest libvirt qemu kernel packet, However, could still hit the issue while hotplug a disk with scsi bus, but couldn't hit the issue with the virtio bus, so i think we could change the bug status back to track the original comment0's issue, how do you think about it?

pkginfo
qemu-kvm-rhev-2.3.0-16.el7.ppc64le
kernel-3.10.0-302.el7.ppc64le
libvirt-1.2.17-5.el7.ppc64le

Comment 8 Andrea Bolognani 2015-08-14 06:31:51 UTC
That's not a bug, though, it's just how scsi works.

You get the same behaviour if you create an x86 guest with something like

    <controller type='scsi' index='0' model='lsilogic'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>

so that virtio is not involved: the user is required to esplicitly ask the
guest to scan for new disks after attach and stop using existing disks before
detach, using the commands in Comment 3 or equivalent.

Have you tried following the proper attach / detach procedure for scsi disks?

Comment 9 zhenfeng wang 2015-08-14 08:16:44 UTC
Hi Andrea
it works for me if follow the comment3's attach/detach procedure for scsi disks. BTW, i found a similar qemu bug 1241886 for this issue, maybe this issue could be implemented after that bug fixed.

Comment 10 Andrea Bolognani 2015-08-14 08:43:27 UTC
But there's nothing to implement :)

If you take a look at the Red Hat Enterprise Linux 7 Storage
Administration Guide, chapter 26.6 (Adding a Storage Device
or Path) and 26.4 (Removing a Storage Device) you will see
similar commands being documented.

When using virtio you don't need to worry about it, but when
using scsi some user intervention is required.

The QEMU bug refers to pci hotplug, which is a different
topic altogether.

Comment 11 David Gibson 2015-12-07 01:11:05 UTC
*** Bug 1285355 has been marked as a duplicate of this bug. ***