RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1714160 - Guest with 'reservations' for a disk start failed
Summary: Guest with 'reservations' for a disk start failed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.7
Hardware: x86_64
OS: Linux
high
unspecified
Target Milestone: rc
: ---
Assignee: Markus Armbruster
QA Contact: Xueqiang Wei
URL:
Whiteboard:
Depends On:
Blocks: 1714891
TreeView+ depends on / blocked
 
Reported: 2019-05-27 09:26 UTC by Yanqiu Zhang
Modified: 2019-08-22 09:22 UTC (History)
13 users (show)

Fixed In Version: qemu-kvm-rhev-2.12.0-32.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1714891 (view as bug list)
Environment:
Last Closed: 2019-08-22 09:20:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
libvirtd_qemu_logs (63.18 KB, application/x-7z-compressed)
2019-05-27 09:31 UTC, Yanqiu Zhang
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:2553 0 None None None 2019-08-22 09:22:05 UTC

Description Yanqiu Zhang 2019-05-27 09:26:32 UTC
Description of problem:
When try to start a guest with 'reservations' for disk, failed with error "No persistent reservation manager with id 'pr-helper0'".

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.12.0-29.el7.x86_64
libvirt-4.5.0-18.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Edit a guest with  'reservations' for a disk
   <disk type='block' device='lun'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/sdb'>
        <reservations managed='yes'/>
      </source>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='3' unit='0'/>
    </disk>

2. Try to start the guest:
# virsh start rhel7.6
error: Failed to start domain rhel7.6
error: internal error: qemu unexpectedly closed the monitor: 2019-05-27T07:51:31.476989Z qemu-kvm: -drive file=/dev/sdb,file.pr-manager=pr-helper0,format=raw,if=none,id=drive-scsi0-0-3-0: No persistent reservation manager with id 'pr-helper0'

3.manually start qemu-pr-helper.service
# systemctl start qemu-pr-helper.service

4.Try to start guest again:
still fails

5. Try to start guest with following reservation, also fails.
        <reservations managed='no'>
          <source type='unix' path='/var/run/qemu-pr-helper.sock' mode='client'/>
        </reservations>


Actual results:
As step 2,4,5.

Expected results:
Guest should start successfully without error.

Additional info:
1. works well for qemu-kvm-rhev-2.12.0-28.el7.x86_64
2. libvirtd and qemu logs pls refer to attachement.

Comment 2 Yanqiu Zhang 2019-05-27 09:31:31 UTC
Created attachment 1573868 [details]
libvirtd_qemu_logs

Comment 4 Michal Privoznik 2019-05-28 12:05:09 UTC
This was reported here:

http://lists.nongnu.org/archive/html/qemu-devel/2019-05/msg03612.html

Comment 5 Xueqiang Wei 2019-05-29 05:53:30 UTC
tested with qemu cmd lines, also hit this issue.


Versions:
kernel-3.10.0-1046.el7.x86_64
qemu-kvm-rhev-2.12.0-29.el7


1. start service
   # systemctl start qemu-pr-helper
   # systemctl status qemu-pr-helper
● qemu-pr-helper.service - Persistent Reservation Daemon for QEMU
   Loaded: loaded (/usr/lib/systemd/system/qemu-pr-helper.service; static; vendor preset: disabled)
   Active: active (running) since Wed 2019-05-29 01:41:27 EDT; 7min ago
 Main PID: 10570 (qemu-pr-helper)
    Tasks: 1
   CGroup: /system.slice/qemu-pr-helper.service
           └─10570 /usr/bin/qemu-pr-helper

2. boot guest with below cmd lines
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1 \
    -device pcie-root-port,id=pcie_root_port_0,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_1,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_2,slot=4,chassis=4,addr=0x4,bus=pcie.0  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_yvw268de/monitor-qmpmonitor1-20181017-004217-U4Tik3JV,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_yvw268de/monitor-catch_monitor-20181017-004217-U4Tik3JV,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idaVJ26s  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_yvw268de/serial-serial0-20181017-004217-U4Tik3JV,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20181017-004217-U4Tik3JV,path=/var/tmp/avocado_yvw268de/seabios-20181017-004217-U4Tik3JV,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20181017-004217-U4Tik3JV,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-5,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-6,addr=0x0,iothread=iothread0 \
    -drive file=/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.qcow2,if=none,format=qcow2,cache=none,werror=stop,rerror=stop,id=drive-system \
    -device scsi-hd,drive=drive-system,id=image1,bootindex=0 \
    -device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0 \
    -device virtio-scsi-pci,id=scsi1,bus=pcie.0-root-port-8,addr=0x0 \
    -object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock \
    -drive file=/dev/sdc,if=none,format=raw,id=drive-data,file.pr-manager=helper0 \
    -device scsi-block,drive=drive-data,id=data-disk1,bus=scsi1.0 \
    -device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:82:83:84:85:86,id=idWBc2X6,vectors=4,netdev=idX17Mug,bus=pcie.0-root-port-7,addr=0x0  \
    -netdev tap,id=idX17Mug,vhost=on \
    -m 4G  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Skylake-Client',hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=d,menu=off,strict=off  \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:4444,server,nowait \



after step 2:

# sh mpath_persistent.sh 
qemu-kvm: -drive file=/dev/sdc,if=none,format=raw,id=drive-data,file.pr-manager=helper0: No persistent reservation manager with id 'helper0'

Comment 6 Markus Armbruster 2019-06-04 15:16:49 UTC
Simplified reproducer:
$ qemu-kvm -object pr-manager-helper,id=pr-helper0,path=/tmp/pr-helper0.sock-drive -drive file=/dev/mapper/crypt,file.pr-manager=pr-helper0,format=raw,if=none

Fails with "No persistent reservation manager with id 'pr-helper0".

Comment 7 Markus Armbruster 2019-06-04 15:22:02 UTC
Proposed upstream fix:
https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg00481.html

It's as simple as patches get.  I'd rather apply it now than to z-stream later.

Comment 11 Miroslav Rezanina 2019-06-11 16:34:35 UTC
Fix included in qemu-kvm-rhev-2.12.0-32.el7

Comment 13 Xueqiang Wei 2019-06-12 07:43:47 UTC
tested with qemu-kvm-rhev-2.12.0-32.el7, not hit this issue. So set status to VERIFIED.

Versions:
kernel-3.10.0-1055.el7.x86_64
qemu-kvm-rhev-2.12.0-32.el7


1. start service
   # systemctl start qemu-pr-helper
   # systemctl status qemu-pr-helper
● qemu-pr-helper.service - Persistent Reservation Daemon for QEMU
   Loaded: loaded (/usr/lib/systemd/system/qemu-pr-helper.service; static; vendor preset: disabled)
   Active: active (running) since Wed 2019-06-12 02:17:47 EDT; 1s ago
 Main PID: 587 (qemu-pr-helper)
    Tasks: 1
   CGroup: /system.slice/qemu-pr-helper.service
           └─587 /usr/bin/qemu-pr-helper

2. boot guest with below cmd lines (pass-through sdc)
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1 \
    -device pcie-root-port,id=pcie_root_port_0,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_1,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device pcie-root-port,id=pcie_root_port_2,slot=4,chassis=4,addr=0x4,bus=pcie.0  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_yvw268de/monitor-qmpmonitor1-20181017-004217-U4Tik3JV,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_yvw268de/monitor-catch_monitor-20181017-004217-U4Tik3JV,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idaVJ26s  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_yvw268de/serial-serial0-20181017-004217-U4Tik3JV,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20181017-004217-U4Tik3JV,path=/var/tmp/avocado_yvw268de/seabios-20181017-004217-U4Tik3JV,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20181017-004217-U4Tik3JV,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-5,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-6,addr=0x0,iothread=iothread0 \
    -drive file=/home/kvm_autotest_root/images/rhel77-64-virtio-scsi.qcow2,if=none,format=qcow2,cache=none,werror=stop,rerror=stop,id=drive-system \
    -device scsi-hd,drive=drive-system,id=image1,bootindex=0 \
    -device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0 \
    -device virtio-scsi-pci,id=scsi1,bus=pcie.0-root-port-8,addr=0x0 \
    -object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock \
    -drive file=/dev/sdc,if=none,format=raw,id=drive-data,file.pr-manager=helper0 \
    -device scsi-block,drive=drive-data,id=data-disk1,bus=scsi1.0 \
    -device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:82:83:84:85:86,id=idWBc2X6,vectors=4,netdev=idX17Mug,bus=pcie.0-root-port-7,addr=0x0  \
    -netdev tap,id=idX17Mug,vhost=on \
    -m 4G  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Westmere',hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=d,menu=off,strict=off  \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:4444,server,nowait \

3. test persistent reservation in guest.

cat test-persistent.sh

#! /bin/sh
sg_persist --no-inquiry -v --out --register-ignore --param-sark 123aaa "$@"
sg_persist --no-inquiry --in -k "$@"
sg_persist --no-inquiry -v --out --reserve --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --release --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --register --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -k "$@"

(1) 
# sh test-persisten.sh /dev/sdb 
    Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00 
PR out: command (Register and ignore existing key) successful
  PR generation=0x1, 1 registered reservation key follows:
    0x123aaa
    Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00 
PR out: command (Reserve) successful
  PR generation=0x1, Reservation follows:
    Key=0x123aaa
    scope: LU_SCOPE,  type: Write Exclusive, registrants only
    Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00 
PR out: command (Release) successful
  PR generation=0x1, there is NO reservation held
    Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00 
PR out: command (Register) successful
  PR generation=0x1, there are NO registered reservation keys

(2) restart guest with multipath (pass-through /dev/mapper/mpatha)

# sh test-persisten.sh /dev/sdb 
    Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00 
PR out: command (Register and ignore existing key) successful
  PR generation=0x2, 2 registered reservation keys follow:
    0x123aaa
    0x123aaa
    Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00 
PR out: command (Reserve) successful
  PR generation=0x2, Reservation follows:
    Key=0x123aaa
    scope: LU_SCOPE,  type: Write Exclusive, registrants only
    Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00 
PR out: command (Release) successful
  PR generation=0x2, there is NO reservation held
    Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00 
PR out: command (Register) successful
  PR generation=0x2, there are NO registered reservation keys


after step 2, guest boot up normally.
after step 3, persistent reservation works well.

Comment 15 errata-xmlrpc 2019-08-22 09:20:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2553


Note You need to log in before you can comment on or make changes to this bug.