Hide Forgot
For clusters in a VM that are using SCSI-3 persistent reservation, a feature is needed in qemu that works similar to "mpathpersist". The reason mpathpersist can't be used is because the reservation key is not know to the underlying system, and this key is typically configured in multipath.conf. In fact, mpathpersist does have an option to specify a reservation key independent of what is in multipath.conf, which could be ideally used by qemu, injecting the "sniffed" key from the VM directly to mpathpersist. Anyways QEMU somehow needs to reuse or reimplement libmpathpersist.so, and optionally use that code to forward PR IN and PR OUT commands to the paths to ensure correct reservations across all paths.
RFC patches posted: http://patchwork.ozlabs.org/patch/804418/ http://patchwork.ozlabs.org/patch/804421/ http://patchwork.ozlabs.org/patch/804417/ http://patchwork.ozlabs.org/patch/804419/ http://patchwork.ozlabs.org/patch/804422/ http://patchwork.ozlabs.org/patch/804428/ http://patchwork.ozlabs.org/patch/804425/ http://patchwork.ozlabs.org/patch/804423/ http://patchwork.ozlabs.org/patch/804420/ http://patchwork.ozlabs.org/patch/804432/ This will need a libvirt patch too (should be easy) and selinux policy.
Karen, QE could test it, please provide a test build as early as possible and let us know if there are any requirements about the test matrix, or we will just need to test one linux guest and one windows guest, thanks.
It's enough to test any Linux guest for the low-level test scenario of comment 4.
Fix included in qemu-kvm-rhev-2.10.0-11.el7
Please call "SCSI-3" "SCSI-3" in such contexts, not "S3"; S3 is an overloaded term and easy to mix up with ACPI S3. Thank you all.
According to Comment 4, tested on qemu-kvm-rhev-2.10.0-11.el7, the result is as expected. Details: Host: Kernel-3.10.0-800.el7.x86_64 qemu-kvm-rhev-2.10.0-11.el7 Guest: Kernel-3.10.0-800.el7.x86_64 1. Define an iSCSI LUN on another host and connect to it, and then configure multipath on host. # multipath -ll mpatha (360a9800050334c33424a78564837386e) dm-3 NETAPP ,LUN size=40G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=2 status=active |- 7:0:0:0 sdb 8:16 active ready running `- 6:0:0:0 sdc 8:32 active ready running 2. Make sure service "qemu-pr-helper" is started. # systemctl start qemu-pr-helper # systemctl status qemu-pr-helper 3. start guest with /dev/sdb /usr/libexec/qemu-kvm \ -name 'vm1' \ -sandbox off \ -machine pc \ -nodefaults \ -vga cirrus \ -device pvpanic,ioport=0x505,id=idMsD9iM \ -chardev socket,id=serial_id_serial0,path=./serial-serial0-20171115-015618-AbdCcNqE,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20171115-015618-AbdCcNqE,path=./seabios-20171115-015618-AbdCcNqE,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20171115-015618-AbdCcNqE,iobase=0x402 \ -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \ -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \ -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \ -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \ -device virtio-net-pci,mac=9a:e1:e2:e3:e4:e5,id=idHfzzeN,vectors=4,netdev=idWPegY6,bus=pci.0,addr=0x5 \ -netdev tap,id=idWPegY6,vhost=on \ -m 2048 \ -smp 2,maxcpus=2,cores=1,threads=1,sockets=2 \ -cpu host \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,strict=off,order=cdn,once=c \ -enable-kvm \ -monitor stdio \ -object iothread,id=iothread0 \ -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3,iothread=iothread0 \ -drive file=/home/kvm_autotest_root/images/rhel75-64-virtio-scsi.qcow2,if=none,id=drive-scsi-disk0,format=qcow2,cache=none \ -device scsi-hd,bus=scsi0.0,drive=drive-scsi-disk0,id=scsi-disk0,bootindex=0,share-rw=on \ -device virtio-scsi-pci,id=scsi1,bus=pci.0,addr=0x4 \ -object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock \ -drive file=/dev/sdb,format=raw,aio=native,if=none,cache=none,media=disk,werror=stop,rerror=stop,id=drive_data1,file.pr-manager=helper0 \ -device scsi-block,drive=drive_data1,id=data1,bus=scsi1.0,channel=0,scsi-id=0,lun=1 \ -qmp tcp:0:6666,server,nowait \ 4. The script, executed with "sh test-persist /dev/sdb" (where sdb is the disk under test) cat test-persist.sh #! /bin/sh sg_persist --no-inquiry -v --out --register-ignore --param-sark 123aaa "$@" sg_persist --no-inquiry --in -k "$@" sg_persist --no-inquiry -v --out --reserve --param-rk 123aaa --prout-type 5 "$@" sg_persist --no-inquiry --in -r "$@" sg_persist --no-inquiry -v --out --release --param-rk 123aaa --prout-type 5 "$@" sg_persist --no-inquiry --in -r "$@" sg_persist --no-inquiry -v --out --register --param-rk 123aaa --prout-type 5 "$@" sg_persist --no-inquiry --in -k "$@" # sh test-persist.sh /dev/sdb Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00 PR out: command (Register and ignore existing key) successful PR generation=0x1, 1 registered reservation key follows: 0x123aaa Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00 PR out: command (Reserve) successful PR generation=0x1, Reservation follows: Key=0x123aaa scope: LU_SCOPE, type: Write Exclusive, registrants only Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00 PR out: command (Release) successful PR generation=0x1, there is NO reservation held Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00 PR out: command (Register) successful PR generation=0x2, there are NO registered reservation keys 5. shutdown guest, change "/dev/sdb" to "/dev/mapper/mpatha" and then boot guest. 6. executed with "sh test-persist /dev/sdb" again, same when /dev/sdb is replaced by a multipath device (/dev/mapper/mpatha) # sh test-persist.sh /dev/sdb Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00 PR out: command (Register and ignore existing key) successful PR generation=0x6, 2 registered reservation keys follow: 0x123aaa 0x123aaa Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00 PR out: command (Reserve) successful PR generation=0x6, Reservation follows: Key=0x123aaa scope: LU_SCOPE, type: Write Exclusive, registrants only Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00 PR out: command (Release) successful PR generation=0x6, there is NO reservation held Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00 PR out: command (Register) successful PR generation=0x8, there are NO registered reservation keys
Tested on FC host, the result is also as expected. Details: Host: kernel-3.10.0-823.el7.x86_64 qemu-kvm-rhev-2.10.0-12.el7 Guest: kernel-3.10.0-748.el7.x86_64 1. multipath information on host: # multipath -ll mpathc (36005076300810b3e0000000000000027) dm-1 IBM ,2145 size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 4:0:1:2 sdg 8:96 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 4:0:0:2 sdd 8:48 active ready running mpathb (36005076300810b3e0000000000000026) dm-0 IBM ,2145 size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 4:0:0:1 sdc 8:32 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 4:0:1:1 sdf 8:80 active ready running mpatha (36005076300810b3e0000000000000025) dm-2 IBM ,2145 size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 4:0:1:0 sde 8:64 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 4:0:0:0 sdb 8:16 active ready running 2. Make sure service "qemu-pr-helper" is started. # systemctl start qemu-pr-helper # systemctl status qemu-pr-helper 3. start guest with /dev/sdb /usr/libexec/qemu-kvm \ -name 'vm1' \ -sandbox off \ -machine pc \ -nodefaults \ -vga cirrus \ -device pvpanic,ioport=0x505,id=idMsD9iM \ -chardev socket,id=serial_id_serial0,path=./serial-serial0-20171115-015618-AbdCcNqE,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20171115-015618-AbdCcNqE,path=./seabios-20171115-015618-AbdCcNqE,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20171115-015618-AbdCcNqE,iobase=0x402 \ -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \ -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \ -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \ -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \ -device virtio-net-pci,mac=9a:e1:e2:e3:e4:e5,id=idHfzzeN,vectors=4,netdev=idWPegY6,bus=pci.0,addr=0x5 \ -netdev tap,id=idWPegY6,vhost=on \ -m 2048 \ -smp 2,maxcpus=2,cores=1,threads=1,sockets=2 \ -cpu host \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,strict=off,order=cdn,once=c \ -enable-kvm \ -monitor stdio \ -object iothread,id=iothread0 \ -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3,iothread=iothread0 \ -drive file=/home/kvm_autotest_root/images/rhel75-64-virtio-scsi.qcow2,if=none,id=drive-scsi-disk0,format=qcow2,cache=none \ -device scsi-hd,bus=scsi0.0,drive=drive-scsi-disk0,id=scsi-disk0,bootindex=0,share-rw=on \ -device virtio-scsi-pci,id=scsi1,bus=pci.0,addr=0x4 \ -object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock \ -drive file=/dev/sdb,format=raw,aio=native,if=none,cache=none,media=disk,werror=stop,rerror=stop,id=drive_data1,file.pr-manager=helper0 \ -device scsi-block,drive=drive_data1,id=data1,bus=scsi1.0,channel=0,scsi-id=0,lun=1 \ -qmp tcp:0:6666,server,nowait \ 4. The script, executed with "sh test-persist /dev/sdb" (where sdb is the disk under test) cat test-persist.sh #! /bin/sh sg_persist --no-inquiry -v --out --register-ignore --param-sark 123aaa "$@" sg_persist --no-inquiry --in -k "$@" sg_persist --no-inquiry -v --out --reserve --param-rk 123aaa --prout-type 5 "$@" sg_persist --no-inquiry --in -r "$@" sg_persist --no-inquiry -v --out --release --param-rk 123aaa --prout-type 5 "$@" sg_persist --no-inquiry --in -r "$@" sg_persist --no-inquiry -v --out --register --param-rk 123aaa --prout-type 5 "$@" sg_persist --no-inquiry --in -k "$@" # sh test-persist.sh /dev/sdb Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00 PR out: command (Register and ignore existing key) successful PR generation=0x1, 1 registered reservation key follows: 0x123aaa Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00 PR out: command (Reserve) successful PR generation=0x1, Reservation follows: Key=0x123aaa scope: LU_SCOPE, type: Write Exclusive, registrants only Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00 PR out: command (Release) successful PR generation=0x1, there is NO reservation held Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00 PR out: command (Register) successful PR generation=0x2, there are NO registered reservation keys 5. shutdown guest, change "/dev/sdb" to "/dev/mapper/mpatha" and then boot guest. 6. executed with "sh test-persist /dev/sdb" again, same when /dev/sdb is replaced by a multipath device (/dev/mapper/mpatha) # sh test-persist.sh /dev/sdb Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00 PR out: command (Register and ignore existing key) successful PR generation=0x6, 2 registered reservation keys follow: 0x123aaa 0x123aaa Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00 PR out: command (Reserve) successful PR generation=0x6, Reservation follows: Key=0x123aaa scope: LU_SCOPE, type: Write Exclusive, registrants only Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00 PR out: command (Release) successful PR generation=0x6, there is NO reservation held Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00 PR out: command (Register) successful PR generation=0x8, there are NO registered reservation keys Hi Paolo, The service "qemu-pr-helper" is not started after install qemu-kvm-rhev, I have to start it manually. Is it necessary to start it automatically?
> Is it necessary to start it automatically? No, libvirt will take care of starting one daemon per QEMU process.
Thanks Paolo. Based on comment 22, comment 23, comment 26, set this bug to 'VERIFIED'. Thanks.
Hi Paolo, I tried to test pr-manager-helper with libiscsi disk, command as follows: ================================================================================= -object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock -device virtio-scsi-pci,id=scsi-hotadd -drive file=iscsi://10.66.8.241/iqn.2016-06.local.server:sas/0,if=none,media=disk,format=raw,rerror=stop,werror=stop,readonly=off,aio=threads,cache=none,cache.direct=on,id=drive-hotadd,serial=sas-test,file.pr-manager=helper0 -device scsi-block,drive=drive-hotadd,bus=scsi-hotadd.0 ================================================================================= But it will report "Block protocol 'iscsi' doesn't support the option 'pr-manager'" warning, detail as follows: ================================================================================= -drive file=iscsi://10.66.8.241/iqn.2016-06.local.server:sas/0,if=none,media=disk,format=raw,rerror=stop,werror=stop,readonly=off,aio=threads,cache=none,cache.direct=on,id=drive-hotadd,serial=sas-test,file.pr-manager=helper0: Block protocol 'iscsi' doesn't support the option 'pr-manager' ================================================================================= Could you help to confirm if it is normal? Thanks a lot~ Peixiu
Yes, libiscsi doesn't support multipath and therefore doesn't need pr-manager yet. Support will be added if libiscsi (or QEMU in general) ever starts supporting multipath natively, without dm-mpath in the kernel.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:1104