Bug 1714160 - Guest with 'reservations' for a disk start failed
Summary: Guest with 'reservations' for a disk start failed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.7
Hardware: x86_64
OS: Linux
high
unspecified
Target Milestone: rc
: ---
Assignee: Markus Armbruster
QA Contact: Xueqiang Wei
URL:
Whiteboard:
Depends On:
Blocks: 1714891
TreeView+ depends on / blocked
 
Reported: 2019-05-27 09:26 UTC by yanqzhan@redhat.com
Modified: 2019-08-22 09:22 UTC (History)
13 users (show)

Fixed In Version: qemu-kvm-rhev-2.12.0-32.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1714891 (view as bug list)
Environment:
Last Closed: 2019-08-22 09:20:54 UTC
Target Upstream Version:


Attachments (Terms of Use)
libvirtd_qemu_logs (63.18 KB, application/x-7z-compressed)
2019-05-27 09:31 UTC, yanqzhan@redhat.com
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:2553 None None None 2019-08-22 09:22:05 UTC

Description yanqzhan@redhat.com 2019-05-27 09:26:32 UTC
Description of problem:
When try to start a guest with 'reservations' for disk, failed with error "No persistent reservation manager with id 'pr-helper0'".

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.12.0-29.el7.x86_64
libvirt-4.5.0-18.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Edit a guest with  'reservations' for a disk
   <disk type='block' device='lun'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/sdb'>
        <reservations managed='yes'/>
      </source>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='3' unit='0'/>
    </disk>

2. Try to start the guest:
# virsh start rhel7.6
error: Failed to start domain rhel7.6
error: internal error: qemu unexpectedly closed the monitor: 2019-05-27T07:51:31.476989Z qemu-kvm: -drive file=/dev/sdb,file.pr-manager=pr-helper0,format=raw,if=none,id=drive-scsi0-0-3-0: No persistent reservation manager with id 'pr-helper0'

3.manually start qemu-pr-helper.service
# systemctl start qemu-pr-helper.service

4.Try to start guest again:
still fails

5. Try to start guest with following reservation, also fails.
        <reservations managed='no'>
          <source type='unix' path='/var/run/qemu-pr-helper.sock' mode='client'/>
        </reservations>


Actual results:
As step 2,4,5.

Expected results:
Guest should start successfully without error.

Additional info:
1. works well for qemu-kvm-rhev-2.12.0-28.el7.x86_64
2. libvirtd and qemu logs pls refer to attachement.

Comment 2 yanqzhan@redhat.com 2019-05-27 09:31:31 UTC
Created attachment 1573868 [details]
libvirtd_qemu_logs

Comment 4 Michal Privoznik 2019-05-28 12:05:09 UTC
This was reported here:

http://lists.nongnu.org/archive/html/qemu-devel/2019-05/msg03612.html

Comment 5 Xueqiang Wei 2019-05-29 05:53:30 UTC
tested with qemu cmd lines, also hit this issue.


Versions:
kernel-3.10.0-1046.el7.x86_64
qemu-kvm-rhev-2.12.0-29.el7


1. start service
   # systemctl start qemu-pr-helper
   # systemctl status qemu-pr-helper
● qemu-pr-helper.service - Persistent Reservation Daemon for QEMU
   Loaded: loaded (/usr/lib/systemd/system/qemu-pr-helper.service; static; vendor preset: disabled)
   Active: active (running) since Wed 2019-05-29 01:41:27 EDT; 7min ago
 Main PID: 10570 (qemu-pr-helper)
    Tasks: 1
   CGroup: /system.slice/qemu-pr-helper.service
           └─10570 /usr/bin/qemu-pr-helper

2. boot guest with below cmd lines
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1 \
    -device pcie-root-port,id=pcie_root_port_0,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_1,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_2,slot=4,chassis=4,addr=0x4,bus=pcie.0  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_yvw268de/monitor-qmpmonitor1-20181017-004217-U4Tik3JV,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_yvw268de/monitor-catch_monitor-20181017-004217-U4Tik3JV,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idaVJ26s  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_yvw268de/serial-serial0-20181017-004217-U4Tik3JV,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20181017-004217-U4Tik3JV,path=/var/tmp/avocado_yvw268de/seabios-20181017-004217-U4Tik3JV,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20181017-004217-U4Tik3JV,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-5,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-6,addr=0x0,iothread=iothread0 \
    -drive file=/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.qcow2,if=none,format=qcow2,cache=none,werror=stop,rerror=stop,id=drive-system \
    -device scsi-hd,drive=drive-system,id=image1,bootindex=0 \
    -device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0 \
    -device virtio-scsi-pci,id=scsi1,bus=pcie.0-root-port-8,addr=0x0 \
    -object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock \
    -drive file=/dev/sdc,if=none,format=raw,id=drive-data,file.pr-manager=helper0 \
    -device scsi-block,drive=drive-data,id=data-disk1,bus=scsi1.0 \
    -device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:82:83:84:85:86,id=idWBc2X6,vectors=4,netdev=idX17Mug,bus=pcie.0-root-port-7,addr=0x0  \
    -netdev tap,id=idX17Mug,vhost=on \
    -m 4G  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Skylake-Client',hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=d,menu=off,strict=off  \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:4444,server,nowait \



after step 2:

# sh mpath_persistent.sh 
qemu-kvm: -drive file=/dev/sdc,if=none,format=raw,id=drive-data,file.pr-manager=helper0: No persistent reservation manager with id 'helper0'

Comment 6 Markus Armbruster 2019-06-04 15:16:49 UTC
Simplified reproducer:
$ qemu-kvm -object pr-manager-helper,id=pr-helper0,path=/tmp/pr-helper0.sock-drive -drive file=/dev/mapper/crypt,file.pr-manager=pr-helper0,format=raw,if=none

Fails with "No persistent reservation manager with id 'pr-helper0".

Comment 7 Markus Armbruster 2019-06-04 15:22:02 UTC
Proposed upstream fix:
https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg00481.html

It's as simple as patches get.  I'd rather apply it now than to z-stream later.

Comment 11 Miroslav Rezanina 2019-06-11 16:34:35 UTC
Fix included in qemu-kvm-rhev-2.12.0-32.el7

Comment 13 Xueqiang Wei 2019-06-12 07:43:47 UTC
tested with qemu-kvm-rhev-2.12.0-32.el7, not hit this issue. So set status to VERIFIED.

Versions:
kernel-3.10.0-1055.el7.x86_64
qemu-kvm-rhev-2.12.0-32.el7


1. start service
   # systemctl start qemu-pr-helper
   # systemctl status qemu-pr-helper
● qemu-pr-helper.service - Persistent Reservation Daemon for QEMU
   Loaded: loaded (/usr/lib/systemd/system/qemu-pr-helper.service; static; vendor preset: disabled)
   Active: active (running) since Wed 2019-06-12 02:17:47 EDT; 1s ago
 Main PID: 587 (qemu-pr-helper)
    Tasks: 1
   CGroup: /system.slice/qemu-pr-helper.service
           └─587 /usr/bin/qemu-pr-helper

2. boot guest with below cmd lines (pass-through sdc)
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1 \
    -device pcie-root-port,id=pcie_root_port_0,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_1,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_2,slot=4,chassis=4,addr=0x4,bus=pcie.0  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_yvw268de/monitor-qmpmonitor1-20181017-004217-U4Tik3JV,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_yvw268de/monitor-catch_monitor-20181017-004217-U4Tik3JV,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idaVJ26s  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_yvw268de/serial-serial0-20181017-004217-U4Tik3JV,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20181017-004217-U4Tik3JV,path=/var/tmp/avocado_yvw268de/seabios-20181017-004217-U4Tik3JV,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20181017-004217-U4Tik3JV,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-5,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-6,addr=0x0,iothread=iothread0 \
    -drive file=/home/kvm_autotest_root/images/rhel77-64-virtio-scsi.qcow2,if=none,format=qcow2,cache=none,werror=stop,rerror=stop,id=drive-system \
    -device scsi-hd,drive=drive-system,id=image1,bootindex=0 \
    -device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0 \
    -device virtio-scsi-pci,id=scsi1,bus=pcie.0-root-port-8,addr=0x0 \
    -object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock \
    -drive file=/dev/sdc,if=none,format=raw,id=drive-data,file.pr-manager=helper0 \
    -device scsi-block,drive=drive-data,id=data-disk1,bus=scsi1.0 \
    -device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:82:83:84:85:86,id=idWBc2X6,vectors=4,netdev=idX17Mug,bus=pcie.0-root-port-7,addr=0x0  \
    -netdev tap,id=idX17Mug,vhost=on \
    -m 4G  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Westmere',hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=d,menu=off,strict=off  \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:4444,server,nowait \

3. test persistent reservation in guest.

cat test-persistent.sh

#! /bin/sh
sg_persist --no-inquiry -v --out --register-ignore --param-sark 123aaa "$@"
sg_persist --no-inquiry --in -k "$@"
sg_persist --no-inquiry -v --out --reserve --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --release --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --register --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -k "$@"

(1) 
# sh test-persisten.sh /dev/sdb 
    Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00 
PR out: command (Register and ignore existing key) successful
  PR generation=0x1, 1 registered reservation key follows:
    0x123aaa
    Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00 
PR out: command (Reserve) successful
  PR generation=0x1, Reservation follows:
    Key=0x123aaa
    scope: LU_SCOPE,  type: Write Exclusive, registrants only
    Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00 
PR out: command (Release) successful
  PR generation=0x1, there is NO reservation held
    Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00 
PR out: command (Register) successful
  PR generation=0x1, there are NO registered reservation keys

(2) restart guest with multipath (pass-through /dev/mapper/mpatha)

# sh test-persisten.sh /dev/sdb 
    Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00 
PR out: command (Register and ignore existing key) successful
  PR generation=0x2, 2 registered reservation keys follow:
    0x123aaa
    0x123aaa
    Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00 
PR out: command (Reserve) successful
  PR generation=0x2, Reservation follows:
    Key=0x123aaa
    scope: LU_SCOPE,  type: Write Exclusive, registrants only
    Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00 
PR out: command (Release) successful
  PR generation=0x2, there is NO reservation held
    Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00 
PR out: command (Register) successful
  PR generation=0x2, there are NO registered reservation keys


after step 2, guest boot up normally.
after step 3, persistent reservation works well.

Comment 15 errata-xmlrpc 2019-08-22 09:20:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2553


Note You need to log in before you can comment on or make changes to this bug.