Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1464908

Summary: [RFE] Add SCSI-3 PR support to qemu (similar to mpathpersist)
Product: Red Hat Enterprise Linux 7 Reporter: Martin Tessun <mtessun>
Component: qemu-kvm-rhevAssignee: Paolo Bonzini <pbonzini>
Status: CLOSED ERRATA QA Contact: Xueqiang Wei <xuwei>
Severity: high Docs Contact:
Priority: urgent    
Version: 7.4CC: aliang, boruvka.michal, chayang, coli, cww, hhuang, juzhang, kchamart, knoel, lersek, lijin, michen, pbonzini, phou, virt-maint, xuwei
Target Milestone: rcKeywords: FutureFeature
Target Release: 7.5   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.10.0-11.el7 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 1470007 1484075 1519019 (view as bug list) Environment:
Last Closed: 2018-04-11 00:26:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1111783, 1111784, 1420851, 1457437, 1469590, 1470007, 1484075, 1519019, 1519021, 1558125    

Description Martin Tessun 2017-06-26 08:41:06 UTC 
For clusters in a VM that are using SCSI-3 persistent reservation, a feature is needed in qemu that works similar to "mpathpersist".

The reason mpathpersist can't be used is because the reservation key is not know to the underlying system, and this key is typically configured in multipath.conf.

In fact, mpathpersist does have an option to specify a reservation key independent of what is in multipath.conf, which could be ideally used by qemu, injecting the "sniffed" key from the VM directly to mpathpersist.

Anyways QEMU somehow needs to reuse or reimplement libmpathpersist.so,
and optionally use that code to forward PR IN and PR OUT commands to the
paths to ensure correct reservations across all paths.
Comment 3 Paolo Bonzini 2017-08-22 14:21:56 UTC 
RFC patches posted:

http://patchwork.ozlabs.org/patch/804418/
http://patchwork.ozlabs.org/patch/804421/
http://patchwork.ozlabs.org/patch/804417/
http://patchwork.ozlabs.org/patch/804419/
http://patchwork.ozlabs.org/patch/804422/
http://patchwork.ozlabs.org/patch/804428/
http://patchwork.ozlabs.org/patch/804425/
http://patchwork.ozlabs.org/patch/804423/
http://patchwork.ozlabs.org/patch/804420/
http://patchwork.ozlabs.org/patch/804432/

This will need a libvirt patch too (should be easy) and selinux policy.
Comment 12 Miya Chen 2017-11-15 01:47:32 UTC 
Karen, QE could test it, please provide a test build as early as possible and let us know if there are any requirements about the test matrix, or we will just need to test one linux guest and one windows guest, thanks.
Comment 13 Paolo Bonzini 2017-11-15 15:15:21 UTC 
It's enough to test any Linux guest for the low-level test scenario of comment 4.
Comment 18 Miroslav Rezanina 2017-12-05 12:58:10 UTC 
Fix included in qemu-kvm-rhev-2.10.0-11.el7
Comment 21 Laszlo Ersek 2017-12-07 12:42:39 UTC 
Please call "SCSI-3" "SCSI-3" in such contexts, not "S3"; S3 is an overloaded term and easy to mix up with ACPI S3. Thank you all.
Comment 22 Xueqiang Wei 2017-12-14 06:15:04 UTC 
According to Comment 4, tested on qemu-kvm-rhev-2.10.0-11.el7, the result is as expected.


Details:

Host:
Kernel-3.10.0-800.el7.x86_64
qemu-kvm-rhev-2.10.0-11.el7

Guest:
Kernel-3.10.0-800.el7.x86_64


1. Define an iSCSI LUN on another host and connect to it, and then  configure multipath on host.

# multipath -ll
mpatha (360a9800050334c33424a78564837386e) dm-3 NETAPP  ,LUN             
size=40G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=2 status=active
  |- 7:0:0:0 sdb 8:16 active ready running
  `- 6:0:0:0 sdc 8:32 active ready running

2. Make sure service "qemu-pr-helper" is started.
# systemctl start qemu-pr-helper
# systemctl status qemu-pr-helper

3. start guest with /dev/sdb
/usr/libexec/qemu-kvm \
    -name 'vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga cirrus  \
    -device pvpanic,ioport=0x505,id=idMsD9iM  \
    -chardev socket,id=serial_id_serial0,path=./serial-serial0-20171115-015618-AbdCcNqE,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20171115-015618-AbdCcNqE,path=./seabios-20171115-015618-AbdCcNqE,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20171115-015618-AbdCcNqE,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -device virtio-net-pci,mac=9a:e1:e2:e3:e4:e5,id=idHfzzeN,vectors=4,netdev=idWPegY6,bus=pci.0,addr=0x5  \
    -netdev tap,id=idWPegY6,vhost=on \
    -m 2048  \
    -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
    -cpu host \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=c \
    -enable-kvm \
    -monitor stdio \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3,iothread=iothread0 \
    -drive file=/home/kvm_autotest_root/images/rhel75-64-virtio-scsi.qcow2,if=none,id=drive-scsi-disk0,format=qcow2,cache=none \
    -device scsi-hd,bus=scsi0.0,drive=drive-scsi-disk0,id=scsi-disk0,bootindex=0,share-rw=on \
    -device virtio-scsi-pci,id=scsi1,bus=pci.0,addr=0x4 \
    -object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock \
    -drive file=/dev/sdb,format=raw,aio=native,if=none,cache=none,media=disk,werror=stop,rerror=stop,id=drive_data1,file.pr-manager=helper0 \
    -device scsi-block,drive=drive_data1,id=data1,bus=scsi1.0,channel=0,scsi-id=0,lun=1 \
    -qmp tcp:0:6666,server,nowait \

4. The script, executed with "sh test-persist /dev/sdb" (where sdb is the disk under test)

cat test-persist.sh
#! /bin/sh
sg_persist --no-inquiry -v --out --register-ignore --param-sark 123aaa "$@"
sg_persist --no-inquiry --in -k "$@"
sg_persist --no-inquiry -v --out --reserve --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --release --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --register --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -k "$@"

# sh test-persist.sh /dev/sdb 
    Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00 
PR out: command (Register and ignore existing key) successful
  PR generation=0x1, 1 registered reservation key follows:
    0x123aaa
    Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00 
PR out: command (Reserve) successful
  PR generation=0x1, Reservation follows:
    Key=0x123aaa
    scope: LU_SCOPE,  type: Write Exclusive, registrants only
    Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00 
PR out: command (Release) successful
  PR generation=0x1, there is NO reservation held
    Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00 
PR out: command (Register) successful
  PR generation=0x2, there are NO registered reservation keys


5. shutdown guest, change "/dev/sdb" to "/dev/mapper/mpatha" and then boot guest.

6. executed with "sh test-persist /dev/sdb" again, same when /dev/sdb is replaced by a multipath device (/dev/mapper/mpatha)

# sh test-persist.sh /dev/sdb
    Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00 
PR out: command (Register and ignore existing key) successful
  PR generation=0x6, 2 registered reservation keys follow:
    0x123aaa
    0x123aaa
    Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00 
PR out: command (Reserve) successful
  PR generation=0x6, Reservation follows:
    Key=0x123aaa
    scope: LU_SCOPE,  type: Write Exclusive, registrants only
    Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00 
PR out: command (Release) successful
  PR generation=0x6, there is NO reservation held
    Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00 
PR out: command (Register) successful
  PR generation=0x8, there are NO registered reservation keys
Comment 23 Xueqiang Wei 2017-12-18 08:04:46 UTC 
Tested on FC host, the result is also as expected.


Details:

Host:
kernel-3.10.0-823.el7.x86_64
qemu-kvm-rhev-2.10.0-12.el7

Guest:
kernel-3.10.0-748.el7.x86_64


1. multipath information on host:

# multipath -ll
mpathc (36005076300810b3e0000000000000027) dm-1 IBM     ,2145            
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 4:0:1:2 sdg 8:96 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 4:0:0:2 sdd 8:48 active ready running
mpathb (36005076300810b3e0000000000000026) dm-0 IBM     ,2145            
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 4:0:0:1 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 4:0:1:1 sdf 8:80 active ready running
mpatha (36005076300810b3e0000000000000025) dm-2 IBM     ,2145            
size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 4:0:1:0 sde 8:64 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 4:0:0:0 sdb 8:16 active ready running

2. Make sure service "qemu-pr-helper" is started.
# systemctl start qemu-pr-helper
# systemctl status qemu-pr-helper

3. start guest with /dev/sdb
/usr/libexec/qemu-kvm \
    -name 'vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga cirrus  \
    -device pvpanic,ioport=0x505,id=idMsD9iM  \
    -chardev socket,id=serial_id_serial0,path=./serial-serial0-20171115-015618-AbdCcNqE,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20171115-015618-AbdCcNqE,path=./seabios-20171115-015618-AbdCcNqE,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20171115-015618-AbdCcNqE,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -device virtio-net-pci,mac=9a:e1:e2:e3:e4:e5,id=idHfzzeN,vectors=4,netdev=idWPegY6,bus=pci.0,addr=0x5  \
    -netdev tap,id=idWPegY6,vhost=on \
    -m 2048  \
    -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
    -cpu host \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=c \
    -enable-kvm \
    -monitor stdio \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3,iothread=iothread0 \
    -drive file=/home/kvm_autotest_root/images/rhel75-64-virtio-scsi.qcow2,if=none,id=drive-scsi-disk0,format=qcow2,cache=none \
    -device scsi-hd,bus=scsi0.0,drive=drive-scsi-disk0,id=scsi-disk0,bootindex=0,share-rw=on \
    -device virtio-scsi-pci,id=scsi1,bus=pci.0,addr=0x4 \
    -object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock \
    -drive file=/dev/sdb,format=raw,aio=native,if=none,cache=none,media=disk,werror=stop,rerror=stop,id=drive_data1,file.pr-manager=helper0 \
    -device scsi-block,drive=drive_data1,id=data1,bus=scsi1.0,channel=0,scsi-id=0,lun=1 \
    -qmp tcp:0:6666,server,nowait \

4. The script, executed with "sh test-persist /dev/sdb" (where sdb is the disk under test)

cat test-persist.sh
#! /bin/sh
sg_persist --no-inquiry -v --out --register-ignore --param-sark 123aaa "$@"
sg_persist --no-inquiry --in -k "$@"
sg_persist --no-inquiry -v --out --reserve --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --release --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --register --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -k "$@"

# sh test-persist.sh /dev/sdb
   Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00
PR out: command (Register and ignore existing key) successful
  PR generation=0x1, 1 registered reservation key follows:
    0x123aaa
    Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00
PR out: command (Reserve) successful
  PR generation=0x1, Reservation follows:
    Key=0x123aaa
    scope: LU_SCOPE,  type: Write Exclusive, registrants only
    Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00
PR out: command (Release) successful
  PR generation=0x1, there is NO reservation held
    Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00
PR out: command (Register) successful
  PR generation=0x2, there are NO registered reservation keys 


5. shutdown guest, change "/dev/sdb" to "/dev/mapper/mpatha" and then boot guest.

6. executed with "sh test-persist /dev/sdb" again, same when /dev/sdb is replaced by a multipath device (/dev/mapper/mpatha)

# sh test-persist.sh /dev/sdb
   Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00
PR out: command (Register and ignore existing key) successful
  PR generation=0x6, 2 registered reservation keys follow:
    0x123aaa
    0x123aaa
    Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00
PR out: command (Reserve) successful
  PR generation=0x6, Reservation follows:
    Key=0x123aaa
    scope: LU_SCOPE,  type: Write Exclusive, registrants only
    Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00
PR out: command (Release) successful
  PR generation=0x6, there is NO reservation held
    Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00
PR out: command (Register) successful
  PR generation=0x8, there are NO registered reservation keys



Hi Paolo,

The service "qemu-pr-helper" is not started after install qemu-kvm-rhev, I have to start it manually. 
Is it necessary to start it automatically?
Comment 26 Paolo Bonzini 2017-12-20 11:42:46 UTC 
> Is it necessary to start it automatically?

No, libvirt will take care of starting one daemon per QEMU process.
Comment 27 CongLi 2017-12-21 01:05:39 UTC 
Thanks Paolo.

Based on comment 22, comment 23, comment 26, set this bug to 'VERIFIED'.

Thanks.
Comment 28 Peixiu Hou 2017-12-25 09:22:34 UTC 
Hi Paolo,

I tried to test pr-manager-helper with libiscsi disk, command as follows:
=================================================================================
-object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock -device virtio-scsi-pci,id=scsi-hotadd -drive file=iscsi://10.66.8.241/iqn.2016-06.local.server:sas/0,if=none,media=disk,format=raw,rerror=stop,werror=stop,readonly=off,aio=threads,cache=none,cache.direct=on,id=drive-hotadd,serial=sas-test,file.pr-manager=helper0 -device scsi-block,drive=drive-hotadd,bus=scsi-hotadd.0
=================================================================================
But it will report "Block protocol 'iscsi' doesn't support the option 'pr-manager'" warning, detail as follows:
=================================================================================
-drive file=iscsi://10.66.8.241/iqn.2016-06.local.server:sas/0,if=none,media=disk,format=raw,rerror=stop,werror=stop,readonly=off,aio=threads,cache=none,cache.direct=on,id=drive-hotadd,serial=sas-test,file.pr-manager=helper0: Block protocol 'iscsi' doesn't support the option 'pr-manager'
=================================================================================
Could you help to confirm if it is normal?

Thanks a lot~
Peixiu
Comment 30 Paolo Bonzini 2018-01-04 18:22:17 UTC 
Yes, libiscsi doesn't support multipath and therefore doesn't need pr-manager yet.  Support will be added if libiscsi (or QEMU in general) ever starts supporting multipath natively, without dm-mpath in the kernel.
Comment 32 errata-xmlrpc 2018-04-11 00:26:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1104