Bug 1464908 - [RFE] Add SCSI-3 PR support to qemu (similar to mpathpersist)
Summary: [RFE] Add SCSI-3 PR support to qemu (similar to mpathpersist)
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.4
Hardware: Unspecified
OS: Linux
urgent
high
Target Milestone: rc
: 7.5
Assignee: Paolo Bonzini
QA Contact: Xueqiang Wei
URL:
Whiteboard:
Keywords: FutureFeature
Depends On:
Blocks: 1420851 1519019 1519021 RHEV_SCSI_reserve_Win_DirectLUN RHEV_SCSI_reserve_Win_SharedDisk 1457437 1469590 1470007 1484075 1558125
TreeView+ depends on / blocked
 
Reported: 2017-06-26 08:41 UTC by Martin Tessun
Modified: 2018-04-11 00:28 UTC (History)
16 users (show)

(edit)
Clone Of:
: 1470007 1484075 1519019 (view as bug list)
(edit)
Last Closed: 2018-04-11 00:26:27 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:1104 None None None 2018-04-11 00:28 UTC

Description Martin Tessun 2017-06-26 08:41:06 UTC
For clusters in a VM that are using SCSI-3 persistent reservation, a feature is needed in qemu that works similar to "mpathpersist".

The reason mpathpersist can't be used is because the reservation key is not know to the underlying system, and this key is typically configured in multipath.conf.

In fact, mpathpersist does have an option to specify a reservation key independent of what is in multipath.conf, which could be ideally used by qemu, injecting the "sniffed" key from the VM directly to mpathpersist.

Anyways QEMU somehow needs to reuse or reimplement libmpathpersist.so,
and optionally use that code to forward PR IN and PR OUT commands to the
paths to ensure correct reservations across all paths.

Comment 12 Miya Chen 2017-11-15 01:47:32 UTC
Karen, QE could test it, please provide a test build as early as possible and let us know if there are any requirements about the test matrix, or we will just need to test one linux guest and one windows guest, thanks.

Comment 13 Paolo Bonzini 2017-11-15 15:15:21 UTC
It's enough to test any Linux guest for the low-level test scenario of comment 4.

Comment 18 Miroslav Rezanina 2017-12-05 12:58:10 UTC
Fix included in qemu-kvm-rhev-2.10.0-11.el7

Comment 21 Laszlo Ersek 2017-12-07 12:42:39 UTC
Please call "SCSI-3" "SCSI-3" in such contexts, not "S3"; S3 is an overloaded term and easy to mix up with ACPI S3. Thank you all.

Comment 22 Xueqiang Wei 2017-12-14 06:15:04 UTC
According to Comment 4, tested on qemu-kvm-rhev-2.10.0-11.el7, the result is as expected.


Details:

Host:
Kernel-3.10.0-800.el7.x86_64
qemu-kvm-rhev-2.10.0-11.el7

Guest:
Kernel-3.10.0-800.el7.x86_64


1. Define an iSCSI LUN on another host and connect to it, and then  configure multipath on host.

# multipath -ll
mpatha (360a9800050334c33424a78564837386e) dm-3 NETAPP  ,LUN             
size=40G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=2 status=active
  |- 7:0:0:0 sdb 8:16 active ready running
  `- 6:0:0:0 sdc 8:32 active ready running

2. Make sure service "qemu-pr-helper" is started.
# systemctl start qemu-pr-helper
# systemctl status qemu-pr-helper

3. start guest with /dev/sdb
/usr/libexec/qemu-kvm \
    -name 'vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga cirrus  \
    -device pvpanic,ioport=0x505,id=idMsD9iM  \
    -chardev socket,id=serial_id_serial0,path=./serial-serial0-20171115-015618-AbdCcNqE,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20171115-015618-AbdCcNqE,path=./seabios-20171115-015618-AbdCcNqE,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20171115-015618-AbdCcNqE,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -device virtio-net-pci,mac=9a:e1:e2:e3:e4:e5,id=idHfzzeN,vectors=4,netdev=idWPegY6,bus=pci.0,addr=0x5  \
    -netdev tap,id=idWPegY6,vhost=on \
    -m 2048  \
    -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
    -cpu host \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=c \
    -enable-kvm \
    -monitor stdio \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3,iothread=iothread0 \
    -drive file=/home/kvm_autotest_root/images/rhel75-64-virtio-scsi.qcow2,if=none,id=drive-scsi-disk0,format=qcow2,cache=none \
    -device scsi-hd,bus=scsi0.0,drive=drive-scsi-disk0,id=scsi-disk0,bootindex=0,share-rw=on \
    -device virtio-scsi-pci,id=scsi1,bus=pci.0,addr=0x4 \
    -object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock \
    -drive file=/dev/sdb,format=raw,aio=native,if=none,cache=none,media=disk,werror=stop,rerror=stop,id=drive_data1,file.pr-manager=helper0 \
    -device scsi-block,drive=drive_data1,id=data1,bus=scsi1.0,channel=0,scsi-id=0,lun=1 \
    -qmp tcp:0:6666,server,nowait \

4. The script, executed with "sh test-persist /dev/sdb" (where sdb is the disk under test)

cat test-persist.sh
#! /bin/sh
sg_persist --no-inquiry -v --out --register-ignore --param-sark 123aaa "$@"
sg_persist --no-inquiry --in -k "$@"
sg_persist --no-inquiry -v --out --reserve --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --release --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --register --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -k "$@"

# sh test-persist.sh /dev/sdb 
    Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00 
PR out: command (Register and ignore existing key) successful
  PR generation=0x1, 1 registered reservation key follows:
    0x123aaa
    Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00 
PR out: command (Reserve) successful
  PR generation=0x1, Reservation follows:
    Key=0x123aaa
    scope: LU_SCOPE,  type: Write Exclusive, registrants only
    Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00 
PR out: command (Release) successful
  PR generation=0x1, there is NO reservation held
    Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00 
PR out: command (Register) successful
  PR generation=0x2, there are NO registered reservation keys


5. shutdown guest, change "/dev/sdb" to "/dev/mapper/mpatha" and then boot guest.

6. executed with "sh test-persist /dev/sdb" again, same when /dev/sdb is replaced by a multipath device (/dev/mapper/mpatha)

# sh test-persist.sh /dev/sdb
    Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00 
PR out: command (Register and ignore existing key) successful
  PR generation=0x6, 2 registered reservation keys follow:
    0x123aaa
    0x123aaa
    Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00 
PR out: command (Reserve) successful
  PR generation=0x6, Reservation follows:
    Key=0x123aaa
    scope: LU_SCOPE,  type: Write Exclusive, registrants only
    Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00 
PR out: command (Release) successful
  PR generation=0x6, there is NO reservation held
    Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00 
PR out: command (Register) successful
  PR generation=0x8, there are NO registered reservation keys

Comment 23 Xueqiang Wei 2017-12-18 08:04:46 UTC
Tested on FC host, the result is also as expected.


Details:

Host:
kernel-3.10.0-823.el7.x86_64
qemu-kvm-rhev-2.10.0-12.el7

Guest:
kernel-3.10.0-748.el7.x86_64


1. multipath information on host:

# multipath -ll
mpathc (36005076300810b3e0000000000000027) dm-1 IBM     ,2145            
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 4:0:1:2 sdg 8:96 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 4:0:0:2 sdd 8:48 active ready running
mpathb (36005076300810b3e0000000000000026) dm-0 IBM     ,2145            
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 4:0:0:1 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 4:0:1:1 sdf 8:80 active ready running
mpatha (36005076300810b3e0000000000000025) dm-2 IBM     ,2145            
size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 4:0:1:0 sde 8:64 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 4:0:0:0 sdb 8:16 active ready running

2. Make sure service "qemu-pr-helper" is started.
# systemctl start qemu-pr-helper
# systemctl status qemu-pr-helper

3. start guest with /dev/sdb
/usr/libexec/qemu-kvm \
    -name 'vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga cirrus  \
    -device pvpanic,ioport=0x505,id=idMsD9iM  \
    -chardev socket,id=serial_id_serial0,path=./serial-serial0-20171115-015618-AbdCcNqE,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20171115-015618-AbdCcNqE,path=./seabios-20171115-015618-AbdCcNqE,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20171115-015618-AbdCcNqE,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -device virtio-net-pci,mac=9a:e1:e2:e3:e4:e5,id=idHfzzeN,vectors=4,netdev=idWPegY6,bus=pci.0,addr=0x5  \
    -netdev tap,id=idWPegY6,vhost=on \
    -m 2048  \
    -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
    -cpu host \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=c \
    -enable-kvm \
    -monitor stdio \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3,iothread=iothread0 \
    -drive file=/home/kvm_autotest_root/images/rhel75-64-virtio-scsi.qcow2,if=none,id=drive-scsi-disk0,format=qcow2,cache=none \
    -device scsi-hd,bus=scsi0.0,drive=drive-scsi-disk0,id=scsi-disk0,bootindex=0,share-rw=on \
    -device virtio-scsi-pci,id=scsi1,bus=pci.0,addr=0x4 \
    -object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock \
    -drive file=/dev/sdb,format=raw,aio=native,if=none,cache=none,media=disk,werror=stop,rerror=stop,id=drive_data1,file.pr-manager=helper0 \
    -device scsi-block,drive=drive_data1,id=data1,bus=scsi1.0,channel=0,scsi-id=0,lun=1 \
    -qmp tcp:0:6666,server,nowait \

4. The script, executed with "sh test-persist /dev/sdb" (where sdb is the disk under test)

cat test-persist.sh
#! /bin/sh
sg_persist --no-inquiry -v --out --register-ignore --param-sark 123aaa "$@"
sg_persist --no-inquiry --in -k "$@"
sg_persist --no-inquiry -v --out --reserve --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --release --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --register --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -k "$@"

# sh test-persist.sh /dev/sdb
   Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00
PR out: command (Register and ignore existing key) successful
  PR generation=0x1, 1 registered reservation key follows:
    0x123aaa
    Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00
PR out: command (Reserve) successful
  PR generation=0x1, Reservation follows:
    Key=0x123aaa
    scope: LU_SCOPE,  type: Write Exclusive, registrants only
    Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00
PR out: command (Release) successful
  PR generation=0x1, there is NO reservation held
    Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00
PR out: command (Register) successful
  PR generation=0x2, there are NO registered reservation keys 


5. shutdown guest, change "/dev/sdb" to "/dev/mapper/mpatha" and then boot guest.

6. executed with "sh test-persist /dev/sdb" again, same when /dev/sdb is replaced by a multipath device (/dev/mapper/mpatha)

# sh test-persist.sh /dev/sdb
   Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00
PR out: command (Register and ignore existing key) successful
  PR generation=0x6, 2 registered reservation keys follow:
    0x123aaa
    0x123aaa
    Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00
PR out: command (Reserve) successful
  PR generation=0x6, Reservation follows:
    Key=0x123aaa
    scope: LU_SCOPE,  type: Write Exclusive, registrants only
    Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00
PR out: command (Release) successful
  PR generation=0x6, there is NO reservation held
    Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00
PR out: command (Register) successful
  PR generation=0x8, there are NO registered reservation keys



Hi Paolo,

The service "qemu-pr-helper" is not started after install qemu-kvm-rhev, I have to start it manually. 
Is it necessary to start it automatically?

Comment 26 Paolo Bonzini 2017-12-20 11:42:46 UTC
> Is it necessary to start it automatically?

No, libvirt will take care of starting one daemon per QEMU process.

Comment 27 CongLi 2017-12-21 01:05:39 UTC
Thanks Paolo.

Based on comment 22, comment 23, comment 26, set this bug to 'VERIFIED'.

Thanks.

Comment 28 Peixiu Hou 2017-12-25 09:22:34 UTC
Hi Paolo,

I tried to test pr-manager-helper with libiscsi disk, command as follows:
=================================================================================
-object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock -device virtio-scsi-pci,id=scsi-hotadd -drive file=iscsi://10.66.8.241/iqn.2016-06.local.server:sas/0,if=none,media=disk,format=raw,rerror=stop,werror=stop,readonly=off,aio=threads,cache=none,cache.direct=on,id=drive-hotadd,serial=sas-test,file.pr-manager=helper0 -device scsi-block,drive=drive-hotadd,bus=scsi-hotadd.0
=================================================================================
But it will report "Block protocol 'iscsi' doesn't support the option 'pr-manager'" warning, detail as follows:
=================================================================================
-drive file=iscsi://10.66.8.241/iqn.2016-06.local.server:sas/0,if=none,media=disk,format=raw,rerror=stop,werror=stop,readonly=off,aio=threads,cache=none,cache.direct=on,id=drive-hotadd,serial=sas-test,file.pr-manager=helper0: Block protocol 'iscsi' doesn't support the option 'pr-manager'
=================================================================================
Could you help to confirm if it is normal?

Thanks a lot~
Peixiu

Comment 30 Paolo Bonzini 2018-01-04 18:22:17 UTC
Yes, libiscsi doesn't support multipath and therefore doesn't need pr-manager yet.  Support will be added if libiscsi (or QEMU in general) ever starts supporting multipath natively, without dm-mpath in the kernel.

Comment 32 errata-xmlrpc 2018-04-11 00:26:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1104


Note You need to log in before you can comment on or make changes to this bug.