Bug 1285355

Summary: [ppc64le] Guest can't detect sPAPR-VSCSI hotplugged disk
Product: Red Hat Enterprise Linux 7 Reporter: Elad <ebenahar>
Component: qemu-kvm-rhevAssignee: David Gibson <dgibson>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.2CC: alitke, amureini, bazulay, bugs, dyuan, dzheng, ebenahar, ecohen, gklein, gsun, hannsj_uhl, istein, juzhang, knoel, lsurette, michal.skrivanek, michen, mzhan, qzhang, tnisan, virt-maint, ycui, yeylon, ylavi, zhwang
Target Milestone: rc   
Target Release: ---   
Hardware: ppc64le   
OS: Unspecified   
Whiteboard: storage
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-07 01:11:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1201513    
Attachments:
Description Flags
logs from engine and host
none
logs from engine and host, includes libvirtd.log none

Description Elad 2015-11-25 12:57:00 UTC
Created attachment 1098797 [details]
logs from engine and host

Description of problem:
On PPC64le, a hotplugged sPAPR-VSCS thin disk is not visible by the guest after the hotplug operation. Reproduced on both block and file storage.
sPAPR-VSCS preallocated disk hotplug works as expected.

Version-Release number of selected component (if applicable):

Engine:
rhevm-3.6.0.3-0.1.el6.noarch

Host:
Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux ibm-p8-rhevm-03.rhts.eng.bos.redhat.com 3.10.0-327.2.1.el7.ppc64le #1 SMP Fri Nov 13 14:35:26 EST 2015 ppc64le ppc64le ppc64le GNU/Linux
vdsm-python-4.17.10.1-0.el7ev.noarch
vdsm-yajsonrpc-4.17.10.1-0.el7ev.noarch
vdsm-4.17.10.1-0.el7ev.noarch
vdsm-jsonrpc-4.17.10.1-0.el7ev.noarch
vdsm-cli-4.17.10.1-0.el7ev.noarch
vdsm-xmlrpc-4.17.10.1-0.el7ev.noarch
vdsm-infra-4.17.10.1-0.el7ev.noarch
libvirt-daemon-driver-storage-1.2.17-13.el7_2.2.ppc64le
libvirt-docs-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-driver-secret-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-driver-lxc-1.2.17-13.el7_2.2.ppc64le
libvirt-devel-1.2.17-13.el7_2.2.ppc64le
libvirt-lock-sanlock-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-driver-network-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-config-nwfilter-1.2.17-13.el7_2.2.ppc64le
libvirt-1.2.17-13.el7_2.2.ppc64le
libvirt-login-shell-1.2.17-13.el7_2.2.ppc64le
libvirt-python-1.2.17-2.el7.ppc64le
libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-driver-interface-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-config-network-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-lxc-1.2.17-13.el7_2.2.ppc64le
libvirt-debuginfo-1.2.17-13.el7_2.2.ppc64le
libvirt-client-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-driver-qemu-1.2.17-13.el7_2.2.ppc64le
libvirt-daemon-kvm-1.2.17-13.el7_2.2.ppc64le
qemu-kvm-tools-rhev-2.3.0-31.el7_2.3.ppc64le
qemu-kvm-common-rhev-2.3.0-31.el7_2.3.ppc64le
ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
qemu-kvm-rhev-2.3.0-31.el7_2.3.ppc64le
qemu-kvm-rhev-debuginfo-2.3.0-31.el7_2.3.ppc64le
qemu-img-rhev-2.3.0-31.el7_2.3.ppc64le


Guest:
Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux dhcp167-130.klab.eng.bos.redhat.com 3.10.0-327.2.1.el7.ppc64le #1 SMP Fri Nov 13 14:35:26 EST 2015 ppc64le ppc64le ppc64le GNU/Linux


How reproducible:
Always

Steps to Reproduce:
On PPC64LE env.
1. Create a VM with RHEL OS 
2. Hotplug a sPAPR-VSCSI thin disk (storage type doesn't matter)
3. On the guest, check lsblk

Actual results:
The disk is not seen by the guest.


Before disk hotplug, on the guest:

[root@dhcp167-130 dev]# date && lsblk
Wed 25 Nov 07:42:31 EST 2015
NAME                       MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                         11:0    1 1024M  0 rom  
vda                        252:0    0   10G  0 disk 
├─vda1                     252:1    0    4M  0 part 
├─vda2                     252:2    0  500M  0 part /boot
└─vda3                     252:3    0  9.5G  0 part 
  ├─rhel_dhcp167--130-root 253:0    0  8.5G  0 lvm  /
  └─rhel_dhcp167--130-swap 253:1    0    1G  0 lvm  [SWAP]


==============================================================================
Hotplug In engine.log:

2015-11-25 14:44:33,397 INFO  [org.ovirt.engine.core.bll.HotPlugDiskToVmCommand] (org.ovirt.thread.pool-7-thread-19) [33fe6e34] Lock Acquired to object 'EngineLock:{exclusiveLocks='[b831c495-c3e0-4986-abde-746d9f0573cf=<DISK, ACTION_TYPE_FAILED_DISKS_LOCKED$diskAliases golden_env_mixed_virtio_0_Disk1>]', sharedLocks='[2e5e8165-aadc-4819-8f4b-5f950aab4936=<VM, ACTION_TYPE_FAILED_VM_IS_LOCKED>]'}'
2015-11-25 14:44:33,413 INFO  [org.ovirt.engine.core.bll.HotPlugDiskToVmCommand] (org.ovirt.thread.pool-7-thread-19) [33fe6e34] Running command: HotPlugDiskToVmCommand internal: true. Entities affected :  ID: 2e5e8165-aadc-4819-8f4b-5f950aab4936 Type: VMAction group CONFIGURE_VM_STORAGE with role type USER
2015-11-25 14:44:33,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-7-thread-19) [33fe6e34] START, HotPlugDiskVDSCommand(HostName = host_mixed_1, HotPlugDiskVDSParameters:{runAsync='true', hostId='296817a6-8cc4-42bf-bcbb-287ea1e640c4', vmId='2e5e8165-aadc-4819-8f4b-5f950aab4936', diskId='b831c495-c3e0-4986-abde-746d9f0573cf', addressMap='[unit=3, bus=0, target=0, controller=0, type=drive]'}), log id: 535d1ddd
2015-11-25 14:44:36,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-7-thread-19) [33fe6e34] FINISH, HotPlugDiskVDSCommand, log id: 535d1ddd
2015-11-25 14:44:36,368 INFO  [org.ovirt.engine.core.bll.HotPlugDiskToVmCommand] (org.ovirt.thread.pool-7-thread-19) [33fe6e34] Lock freed to object 'EngineLock:{exclusiveLocks='[b831c495-c3e0-4986-abde-746d9f0573cf=<DISK, ACTION_TYPE_FAILED_DISKS_LOCKED$diskAliases golden_env_mixed_virtio_0_Disk1>]', sharedLocks='[2e5e8165-aadc-4819-8f4b-5f950aab4936=<VM, ACTION_TYPE_FAILED_VM_IS_LOCKED>]'}'

==============================================================================
Hotplug call in vdsm.log:

Thread-41416::DEBUG::2015-11-25 07:44:30,741::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.hotplugDisk' in bridge with {u'params': {u'vmId': u'2e5e8165-aadc-4819-8f4b-5f950aab4936', u'drive': {u'iface': u'scsi', u'format': u'cow', u'optional': u'false', u'poolID': u'41c0d682-de18-45cf-8f70-81471be76a99', u'volumeID': u'452e0080-f5c7-43ca-984b-eebe82ad4f09', u'imageID': u'b831c495-c3e0-4986-abde-746d9f0573cf', u'specParams': {}, u'readonly': u'false', u'domainID': u'2f9b37ca-b015-4c24-a89d-daddd5c7169e', u'deviceId': u'b831c495-c3e0-4986-abde-746d9f0573cf', u'address': {u'bus': u'0', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'3'}, u'device': u'disk', u'shared': u'false', u'propagateErrors': u'off', u'type': u'disk'}}, u'vmID': u'2e5e8165-aadc-4819-8f4b-5f950aab4936'}


Thread-41416::INFO::2015-11-25 07:44:33,410::vm::2577::virt.vm::(hotplugDisk) vmId=`2e5e8165-aadc-4819-8f4b-5f950aab4936`::Hotplug disk xml: <disk device="disk" snapshot="no" type="block">
        <address bus="0" controller="0" target="0" type="drive" unit="3"/>
        <source dev="/rhev/data-center/41c0d682-de18-45cf-8f70-81471be76a99/2f9b37ca-b015-4c24-a89d-daddd5c7169e/images/b831c495-c3e0-4986-abde-746d9f0573cf/452e0080-f5c7-43ca-984b-eebe82ad4f09"/>
        <target bus="scsi" dev="sdd"/>
        <serial>b831c495-c3e0-4986-abde-746d9f0573cf</serial>
        <driver cache="none" error_policy="stop" io="native" name="qemu" type="qcow2"/>
</disk>


Thread-41416::DEBUG::2015-11-25 07:44:33,452::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'VM.hotplugDisk' in bridge with {u'acpiEnable': u'true', u'emulatedMachine': u'pseries-rhel7.2.0', u'table
tEnable': u'true', 'pid': '21191', u'memGuaranteedSize': 512, u'spiceSslCipherSuite': u'DEFAULT', 'displaySecurePort': '-1', u'timeOffset': u'0', u'cpuType': u'POWER8', u'smp': u'1', 'pauseCode': 'NOERR', u'smartc
ardEnable': u'false', u'custom': {u'device_d752f9f4-a715-4e73-a37d-ce01ffa3dfc3device_88af3c97-657a-4c61-a6b2-41fa41b909f3device_7fa97eef-41b0-42dd-9a92-07df8b81c69c': u"VmDevice:{id='VmDeviceId:{deviceId='7fa97ee
f-41b0-42dd-9a92-07df8b81c69c', vmId='2e5e8165-aadc-4819-8f4b-5f950aab4936'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}", u'device_d752f9f4-a715-4e73-a37d-ce01ffa3dfc3device_88af3c97-657a-4c61-a6b2-41fa41b909f3': u"VmDevice:{id='VmDeviceId:{deviceId='88af3c97-657a-4c61-a6b2-41fa41b909f3', vmId='2e5e8165-aadc-4819-8f4b-5f950aab4936'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}", u'device_d752f9f4-a715-4e73-a37d-ce01ffa3dfc3': u"VmDevice:{id='VmDeviceId:{deviceId='d752f9f4-a715-4e73-a37d-ce01ffa3dfc3', vmId='2e5e8165-aadc-4819-8f4b-5f950aab4936'}', device='scsi', type='CONTROLLER', bootOrder='0', specParams='[]', address='{type=spapr-vio, reg=0x2000}', managed='false', plugged='true', readOnly='false', deviceAlias='scsi0', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}"}, u'vmType': u'kvm', u'memSize': 1024, u'smpCoresPerSocket': u'1', u'vmName': u'golden_env_mixed_virtio_0', u'nice': u'0', 'status': 'Up', u'bootMenuEnable': u'false', u'vmId': u'2e5e8165-aadc-4819-8f4b-5f950aab4936', 'displayIp': '10.16.69.249', 'displayPort': u'5900', 'guestDiskMapping': {u'0QEMU_QEMU_CD-ROM': {u'name': u'/dev/sr0'}, u'118a0eac-b5c2-43ce-9': {u'name': u'/dev/vdb'}, u'a96f8b40-4e38-4ac6-a': {u'name': u'/dev/vda'}}, u'spiceSecureChannels': u'smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard', 'nicModel': 'rtl8139,pv', u'keyboardLayout': u'en-us', u'kvmEnable': u'true', u'transparentHugePages': u'true', u'displayNetwork': u'ovirtmgmt', u'devices': [{'target': 1048576, u'specParams': {u'model': u'virtio'}, 'alias': u'balloon0', 'deviceType': u'balloon', u'deviceId': u'9dc46d73-4e71-4f90-9f93-b2ccfd2df212', u'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'memballoon', u'type': u'balloon'}, {u'index': u'0', 'alias': u'scsi0', 'deviceType': u'controller', u'address': {u'type': u'spapr-vio', u'reg': u'0x2000'}, u'device': u'scsi', u'type': u'controller'}, {u'specParams': {}, 'alias': u'virtio-serial0', 'deviceType': u'controller', u'deviceId': u'5d14ea14-eef2-4676-af1f-183f4e9222c2', u'address': {u'slot': u'0x03', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'virtio-serial', u'type': u'controller'}, {u'specParams': {u'vram': u'32768', u'heads': u'1'}, 'alias': u'video0', 'deviceType': u'video', u'deviceId': u'4bc4f999-f260-41ca-9cab-53abf09222cd', u'address': {u'slot': u'0x07', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'vga', u'type': u'video'}, {u'specParams': {'displayNetwork': u'ovirtmgmt', 'displayIp': '10.16.69.249'}, 'deviceType': u'graphics', u'deviceId': u'269c582e-45f3-426c-96c1-499bf138d6fa', u'device': u'vnc', u'type': u'graphics', 'port': u'5900'}, {u'nicModel': u'pv', u'macAddr': u'00:1a:4a:16:01:51', u'linkActive': True, u'network': u'ovirtmgmt', u'specParams': {u'inbound': {}, u'outbound': {}}, u'filter': u'vdsm-no-mac-spoofing', 'alias': u'net0', 'deviceType': u'interface', u'deviceId': u'61cda16a-a8da-40b4-bf48-3477988b055e', u'address': {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'bridge', u'type': u'interface', 'name': u'vnet0'}, {u'index': u'0', u'iface': u'scsi', 'name': u'sda', 'alias': u'scsi0-0-0-0', u'specParams': {u'path': u''}, u'readonly': 'True', 'deviceType': u'disk', u'deviceId': u'659de35b-c05f-49c7-bb72-1df41d0030bf', u'address': {u'bus': u'0', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, u'device': u'cdrom', u'shared': u'false', u'path': '', u'type': u'disk'}, {u'poolID': u'41c0d682-de18-45cf-8f70-81471be76a99', 'volumeInfo': {'domainID': u'47bd4adb-7bd4-425d-98e3-b105f34a76a0', 'volType': 'path', 'leaseOffset': 0, 'volumeID': u'e48cca8c-4cf6-429c-ab6c-56eff92497ec', 'leasePath': u'/rhev/data-center/mnt/10.16.29.93:_ppc__ge__1__nfs__1/47bd4adb-7bd4-425d-98e3-b105f34a76a0/images/a96f8b40-4e38-4ac6-a51f-dc26c4f2cc54/e48cca8c-4cf6-429c-ab6c-56eff92497ec.lease', 'imageID': u'a96f8b40-4e38-4ac6-a51f-dc26c4f2cc54', 'path': u'/rhev/data-center/mnt/10.16.29.93:_ppc__ge__1__nfs__1/47bd4adb-7bd4-425d-98e3-b105f34a76a0/images/a96f8b40-4e38-4ac6-a51f-dc26c4f2cc54/e48cca8c-4cf6-429c-ab6c-56eff92497ec'}, u'index': 0, u'iface': u'virtio', 'apparentsize': '10737418240', u'specParams': {}, u'imageID': u'a96f8b40-4e38-4ac6-a51f-dc26c4f2cc54', u'readonly': 'False', 'deviceType': u'disk', u'shared': u'false', 'truesize': '2491695104', u'type': u'disk', u'domainID': u'47bd4adb-7bd4-425d-98e3-b105f34a76a0', 'reqsize': '0', u'format': u'raw', u'deviceId': u'a96f8b40-4e38-4ac6-a51f-dc26c4f2cc54', u'address': {u'slot': u'0x05', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'device': u'disk', 'path': u'/rhev/data-center/41c0d682-de18-45cf-8f70-81471be76a99/47bd4adb-7bd4-425d-98e3-b105f34a76a0/images/a96f8b40-4e38-4ac6-a51f-dc26c4f2cc54/e48cca8c-4cf6-429c-ab6c-56eff92497ec', u'propagateErrors': u'off', u'optional': u'false', 'name': u'vda', u'bootOrder': u'1', u'volumeID': u'e48cca8c-4cf6-429c-ab6c-56eff92497ec', 'alias': u'virtio-disk0', 'volumeChain': [{'domainID': u'47bd4adb-7bd4-425d-98e3-b105f34a76a0', 'volType': 'path', 'leaseOffset': 0, 'volumeID': u'e48cca8c-4cf6-429c-ab6c-56eff92497ec', 'leasePath': u'/rhev/data-center/mnt/10.16.29.93:_ppc__ge__1__nfs__1/47bd4adb-7bd4-425d-98e3-b105f34a76a0/images/a96f8b40-4e38-4ac6-a51f-dc26c4f2cc54/e48cca8c-4cf6-429c-ab6c-56eff92497ec.lease', 'imageID': u'a96f8b40-4e38-4ac6-a51f-dc26c4f2cc54', 'path': u'/rhev/data-center/mnt/10.16.29.93:_ppc__ge__1__nfs__1/47bd4adb-7bd4-425d-98e3-b105f34a76a0/images/a96f8b40-4e38-4ac6-a51f-dc26c4f2cc54/e48cca8c-4cf6-429c-ab6c-56eff92497ec'}]}, {'device': u'usb', 'alias': u'usb', 'deviceType': 'controller', 'type': 'controller', 'address': {u'slot': u'0x02', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}}, {'device': u'unix', 'alias': u'channel0', 'deviceType': u'channel', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}, {'device': u'unix', 'alias': u'channel1', 'deviceType': u'channel', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}, {u'address': {u'bus': u'0', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'3'}, 'volumeInfo': {'domainID': u'2f9b37ca-b015-4c24-a89d-daddd5c7169e', 'volType': 'path', 'leaseOffset': 111149056, 'volumeID': '452e0080-f5c7-43ca-984b-eebe82ad4f09', 'leasePath': '/dev/2f9b37ca-b015-4c24-a89d-daddd5c7169e/leases', 'imageID': u'b831c495-c3e0-4986-abde-746d9f0573cf', 'path': u'/rhev/data-center/mnt/blockSD/2f9b37ca-b015-4c24-a89d-daddd5c7169e/images/b831c495-c3e0-4986-abde-746d9f0573cf/452e0080-f5c7-43ca-984b-eebe82ad4f09'}, 'index': '3', u'iface': u'scsi', 'apparentsize': '1073741824', u'specParams': {}, u'imageID': u'b831c495-c3e0-4986-abde-746d9f0573cf', u'readonly': 'False', 'deviceType': u'disk', u'shared': u'false', 'truesize': '1073741824', u'type': u'disk', u'domainID': u'2f9b37ca-b015-4c24-a89d-daddd5c7169e', 'reqsize': '0', u'format': u'cow', u'deviceId': u'b831c495-c3e0-4986-abde-746d9f0573cf', u'poolID': u'41c0d682-de18-45cf-8f70-81471be76a99', u'device': u'disk', 'path': u'/rhev/data-center/41c0d682-de18-45cf-8f70-81471be76a99/2f9b37ca-b015-4c24-a89d-daddd5c7169e/images/b831c495-c3e0-4986-abde-746d9f0573cf/452e0080-f5c7-43ca-984b-eebe82ad4f09', u'propagateErrors': u'off', u'optional': u'false', 'name': u'sdd', u'volumeID': u'452e0080-f5c7-43ca-984b-eebe82ad4f09', 'alias': u'scsi0-0-0-3', 'volumeChain': [{'domainID': u'2f9b37ca-b015-4c24-a89d-daddd5c7169e', 'volType': 'path', 'leaseOffset': 111149056, 'volumeID': '452e0080-f5c7-43ca-984b-eebe82ad4f09', 'leasePath': '/dev/2f9b37ca-b015-4c24-a89d-daddd5c7169e/leases', 'imageID': u'b831c495-c3e0-4986-abde-746d9f0573cf', 'path': u'/rhev/data-center/mnt/blockSD/2f9b37ca-b015-4c24-a89d-daddd5c7169e/images/b831c495-c3e0-4986-abde-746d9f0573cf/452e0080-f5c7-43ca-984b-eebe82ad4f09'}]}], 'clientIp': '', 'statusTime': '43051670420', u'display': 'vnc'}

==============================================================================
After disk hotplug, on the guest:

[root@dhcp167-130 dev]# date && lsblk
Wed 25 Nov 07:48:09 EST 2015
NAME                       MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                         11:0    1 1024M  0 rom  
vda                        252:0    0   10G  0 disk 
├─vda1                     252:1    0    4M  0 part 
├─vda2                     252:2    0  500M  0 part /boot
└─vda3                     252:3    0  9.5G  0 part 
  ├─rhel_dhcp167--130-root 253:0    0  8.5G  0 lvm  /
  └─rhel_dhcp167--130-swap 253:1    0    1G  0 lvm  [SWAP]

The disk is not seen by the guest

Expected results:
Disk hotplug should succeed.

Additional info: logs from engine and host

Comment 1 Michal Skrivanek 2015-11-25 13:33:04 UTC
push it down to qemu?

Comment 2 Elad 2015-11-25 14:28:15 UTC
The same for hot-unplug

Comment 3 Amit Aviram 2015-11-25 15:42:56 UTC
In VDSM side the logs show that the hot plugging is successful- namely VDSM got the disk plug request with all its attributes and sent them to libvirt. no error occured, and the request returned success.. probably it got lost somewhere later in the libvirt-qemu chain..

Comment 4 Tal Nisan 2015-11-25 15:51:32 UTC
Elad, please include QEMU & Libvirt logs.

Michal, we'll wait for the relevant logs first and then decide how to proceed.

Comment 5 Elad 2015-11-25 16:01:34 UTC
QEMU log is attached (see golden_env_mixed_virtio_0.log)
Libvirt log doesn't exist on RHEL7.2

Comment 6 Elad 2015-11-26 07:46:32 UTC
Correction: 
The issue reproduced for sPAPR-VSCSI preallocated disk also.

Comment 8 Elad 2015-11-26 12:54:46 UTC
Created attachment 1099277 [details]
logs from engine and host, includes libvirtd.log

(In reply to Yaniv Kaul from comment #7)
> - Did Virt QE confirm it's working from their side?
Didn't check with them, I'll send them a request for their tests results.
> - What do you mean 'libvirt log doesn't exist' ? So how do we get its logs?
Sorry, my confusion. It exists, attached.

From libvirtd.log (the log is 5 hours early):


2015-11-26 12:45:13.061+0000: 11709: debug : virDomainAttachDevice:8339 : dom=0x3fff84002310, (VM: name=vm-1, uuid=6bc01080-7231-47f6-8639-c358425ea602), xml=<disk device="disk" snapshot="no" type="block">
        <address bus="0" controller="0" target="0" type="drive" unit="7"/>
        <source dev="/rhev/data-center/2a0ea7f8-bafe-4b0b-a5a1-4ce1a33271a6/42d4f247-1173-4275-9588-39789a850ddf/images/bd67d409-b6b6-4320-bea7-0a9e40f36944/237eab8b-16a5-4f8d-b5ff-643b3f52eb19"/>
        <target bus="scsi" dev="sdf"/>
        <serial>bd67d409-b6b6-4320-bea7-0a9e40f36944</serial>
        <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw"/>
</disk>


> - On the guest, it's better to run 'lsscsi' and/or 'lspci' - to also ensure
> this is not an issue of mis-identifying the device as a disk.

After hotplug action:

[root@dhcp167-130 ~]# date && lsscsi
Thu 26 Nov 07:52:44 EST 2015
[0:0:0:0]    cd/dvd  QEMU     QEMU CD-ROM      2.3.  /dev/sr0

Comment 11 zhenfeng wang 2015-11-27 05:24:38 UTC
Hi Elad
We met a similar issue with this bug previous that the iscsi disk won't detect by guest after we hotplug it, dev suggest scan it inside the guest in bug 1232106, hope it's helpfull for you.

pkginfo
libvirt-1.2.17-13.el7.ppc64le

steps
1.Prepare a running guest
# virsh list
 Id    Name                           State
----------------------------------------------------
 5     rhel72                        running

2.Attach an iscsi disk to the guest
#virsh attach-disk rhel72 /var/lib/libvirt/images/test.img sdb

3.Check the guest's xml
#virsh dumpxml rhel72
--
<disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/test.img'/>
      <backingStore/>
      <target dev='sdb' bus='scsi'/>
      <alias name='scsi0-0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='scsi' index='0'>
      <alias name='scsi0'/>
      <address type='spapr-vio' reg='0x1000'/>
    </controller>

4.Login guest, couldn't detact the scsi disk

5.scan it scsi bus manually inside guest, then we could find it 
guest#for h in /sys/class/scsi_host/host*; do
    echo '- - -' >"$h/scan"
  done

guest# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda             8:0    0  100M  0 disk 
vda           252:0    0    8G  0 disk 
├─vda1        252:1    0    4M  0 part 
├─vda2        252:2    0  500M  0 part /boot
└─vda3        252:3    0  7.5G  0 part 
  ├─rhel-root 253:0    0  6.7G  0 lvm  /
  └─rhel-swap 253:1    0  820M  0 lvm  [SWAP]

Comment 12 Allon Mureinik 2015-11-30 12:26:02 UTC
Elad, can you please check zhenfeng wang's proposed procedure? This looks as though it may not be bug.

Comment 13 Elad 2015-11-30 12:48:00 UTC
After scanning the scsi bus in the guest, the sPAPR-VSCSI hotplugged disk appears.

[root@dhcp167-130 ~]# lsscsi 
[0:0:0:0]    cd/dvd  QEMU     QEMU CD-ROM      2.3.  /dev/sr0 
[0:0:0:1]    cd/dvd  QEMU     QEMU CD-ROM      2.3.  /dev/sr1
[root@dhcp167-130 ~]# echo '- - -' > /sys/class/scsi_host/host0/scan
[root@dhcp167-130 ~]# lsscsi
[0:0:0:0]    cd/dvd  QEMU     QEMU CD-ROM      2.3.  /dev/sr0 
[0:0:0:1]    cd/dvd  QEMU     QEMU CD-ROM      2.3.  /dev/sr1 
[0:0:0:3]    disk    QEMU     QEMU HARDDISK    2.3.  /dev/sda 

Allon, in any other disk interface, the hotplugged disk appears without the need to scan the scsi bus. I guess this is more of a RHEL issue.

Checked with RHEL7.2 installed on the guest, kernel-3.10.0-327.el7.ppc64le. On the host: kernel-3.10.0-327.2.1.el7.ppc64le, libvirt-daemon-1.2.17-13.el7_2.2.ppc64le, vdsm-4.17.11-0.el7ev.noarch

Comment 14 Yaniv Kaul 2015-12-03 14:22:11 UTC
(In reply to Elad from comment #13)
> After scanning the scsi bus in the guest, the sPAPR-VSCSI hotplugged disk
> appears.
> 
> [root@dhcp167-130 ~]# lsscsi 
> [0:0:0:0]    cd/dvd  QEMU     QEMU CD-ROM      2.3.  /dev/sr0 
> [0:0:0:1]    cd/dvd  QEMU     QEMU CD-ROM      2.3.  /dev/sr1
> [root@dhcp167-130 ~]# echo '- - -' > /sys/class/scsi_host/host0/scan
> [root@dhcp167-130 ~]# lsscsi
> [0:0:0:0]    cd/dvd  QEMU     QEMU CD-ROM      2.3.  /dev/sr0 
> [0:0:0:1]    cd/dvd  QEMU     QEMU CD-ROM      2.3.  /dev/sr1 
> [0:0:0:3]    disk    QEMU     QEMU HARDDISK    2.3.  /dev/sda 
> 
> Allon, in any other disk interface, the hotplugged disk appears without the
> need to scan the scsi bus. I guess this is more of a RHEL issue.

Please move to RHEL, then.

> 
> Checked with RHEL7.2 installed on the guest, kernel-3.10.0-327.el7.ppc64le.
> On the host: kernel-3.10.0-327.2.1.el7.ppc64le,
> libvirt-daemon-1.2.17-13.el7_2.2.ppc64le, vdsm-4.17.11-0.el7ev.noarch

Comment 15 Allon Mureinik 2015-12-03 16:08:22 UTC
Pushing down to qemu to start the investigation - should possibly (probably?) be moved to the right team within the platform group.

Comment 16 David Gibson 2015-12-04 05:46:58 UTC
Right, there is currently no notification implemented to tell a guest an spapr-vscsi disk has been plugged so it could rescan the scsi bus.

I'm not sure if this is possible with the existing protocols, or would require protocol extensions.  If the latter, then this is not going to happen - if you want hotplug you should use virtio instead.

But if the existing SRP protocol can support hotplug, we should implement it - I'll need to look into this.

Comment 17 Michal Skrivanek 2015-12-04 08:30:48 UTC
In light of comment #13 and https://bugzilla.redhat.com/show_bug.cgi?id=1232106#c8
I'm removing the blocker flag and decreasing the severity. I would think we can even close it the same way as bug 1232106, but leaving it up to David's/Karen's consideration

Comment 18 David Gibson 2015-12-07 01:11:05 UTC
Actually, as far as I can tell, this is a duplicate of bug 1232106, and as such, not a bug.

SCSI disks simply aren't detected without an explicit rescan from the guest side.

Unless there's some different in behaviour from *SCSI* disks on x86, or from virtio-scsi, I don't see that there's any bug here.

*** This bug has been marked as a duplicate of bug 1232106 ***