Bug 1714160
| Summary: | Guest with 'reservations' for a disk start failed | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Yanqiu Zhang <yanqzhan> | ||||
| Component: | qemu-kvm-rhev | Assignee: | Markus Armbruster <armbru> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Xueqiang Wei <xuwei> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.7 | CC: | coli, dyuan, hhan, jinzhao, jomurphy, juzhang, lmen, mtessun, ngu, qzhang, virt-maint, yanqzhan, yisun | ||||
| Target Milestone: | rc | Keywords: | Regression | ||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | qemu-kvm-rhev-2.12.0-32.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1714891 (view as bug list) | Environment: | |||||
| Last Closed: | 2019-08-22 09:20:54 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1714891 | ||||||
| Attachments: |
|
||||||
Created attachment 1573868 [details]
libvirtd_qemu_logs
This was reported here: http://lists.nongnu.org/archive/html/qemu-devel/2019-05/msg03612.html tested with qemu cmd lines, also hit this issue.
Versions:
kernel-3.10.0-1046.el7.x86_64
qemu-kvm-rhev-2.12.0-29.el7
1. start service
# systemctl start qemu-pr-helper
# systemctl status qemu-pr-helper
● qemu-pr-helper.service - Persistent Reservation Daemon for QEMU
Loaded: loaded (/usr/lib/systemd/system/qemu-pr-helper.service; static; vendor preset: disabled)
Active: active (running) since Wed 2019-05-29 01:41:27 EDT; 7min ago
Main PID: 10570 (qemu-pr-helper)
Tasks: 1
CGroup: /system.slice/qemu-pr-helper.service
└─10570 /usr/bin/qemu-pr-helper
2. boot guest with below cmd lines
/usr/libexec/qemu-kvm \
-S \
-name 'avocado-vt-vm1' \
-sandbox off \
-machine q35 \
-nodefaults \
-device VGA,bus=pcie.0,addr=0x1 \
-device pcie-root-port,id=pcie_root_port_0,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
-device pcie-root-port,id=pcie_root_port_1,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
-device pcie-root-port,id=pcie_root_port_2,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_yvw268de/monitor-qmpmonitor1-20181017-004217-U4Tik3JV,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_yvw268de/monitor-catch_monitor-20181017-004217-U4Tik3JV,server,nowait \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device pvpanic,ioport=0x505,id=idaVJ26s \
-chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_yvw268de/serial-serial0-20181017-004217-U4Tik3JV,server,nowait \
-device isa-serial,chardev=serial_id_serial0 \
-chardev socket,id=seabioslog_id_20181017-004217-U4Tik3JV,path=/var/tmp/avocado_yvw268de/seabios-20181017-004217-U4Tik3JV,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20181017-004217-U4Tik3JV,iobase=0x402 \
-device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
-device qemu-xhci,id=usb1,bus=pcie.0-root-port-5,addr=0x0 \
-device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \
-object iothread,id=iothread0 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-6,addr=0x0,iothread=iothread0 \
-drive file=/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.qcow2,if=none,format=qcow2,cache=none,werror=stop,rerror=stop,id=drive-system \
-device scsi-hd,drive=drive-system,id=image1,bootindex=0 \
-device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0 \
-device virtio-scsi-pci,id=scsi1,bus=pcie.0-root-port-8,addr=0x0 \
-object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock \
-drive file=/dev/sdc,if=none,format=raw,id=drive-data,file.pr-manager=helper0 \
-device scsi-block,drive=drive-data,id=data-disk1,bus=scsi1.0 \
-device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 \
-device virtio-net-pci,mac=9a:82:83:84:85:86,id=idWBc2X6,vectors=4,netdev=idX17Mug,bus=pcie.0-root-port-7,addr=0x0 \
-netdev tap,id=idX17Mug,vhost=on \
-m 4G \
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
-cpu 'Skylake-Client',hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+kvm_pv_unhalt \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-vnc :0 \
-rtc base=utc,clock=host,driftfix=slew \
-boot order=cdn,once=d,menu=off,strict=off \
-enable-kvm \
-monitor stdio \
-qmp tcp:0:4444,server,nowait \
after step 2:
# sh mpath_persistent.sh
qemu-kvm: -drive file=/dev/sdc,if=none,format=raw,id=drive-data,file.pr-manager=helper0: No persistent reservation manager with id 'helper0'
Simplified reproducer: $ qemu-kvm -object pr-manager-helper,id=pr-helper0,path=/tmp/pr-helper0.sock-drive -drive file=/dev/mapper/crypt,file.pr-manager=pr-helper0,format=raw,if=none Fails with "No persistent reservation manager with id 'pr-helper0". Proposed upstream fix: https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg00481.html It's as simple as patches get. I'd rather apply it now than to z-stream later. Fix included in qemu-kvm-rhev-2.12.0-32.el7 tested with qemu-kvm-rhev-2.12.0-32.el7, not hit this issue. So set status to VERIFIED.
Versions:
kernel-3.10.0-1055.el7.x86_64
qemu-kvm-rhev-2.12.0-32.el7
1. start service
# systemctl start qemu-pr-helper
# systemctl status qemu-pr-helper
● qemu-pr-helper.service - Persistent Reservation Daemon for QEMU
Loaded: loaded (/usr/lib/systemd/system/qemu-pr-helper.service; static; vendor preset: disabled)
Active: active (running) since Wed 2019-06-12 02:17:47 EDT; 1s ago
Main PID: 587 (qemu-pr-helper)
Tasks: 1
CGroup: /system.slice/qemu-pr-helper.service
└─587 /usr/bin/qemu-pr-helper
2. boot guest with below cmd lines (pass-through sdc)
/usr/libexec/qemu-kvm \
-S \
-name 'avocado-vt-vm1' \
-sandbox off \
-machine q35 \
-nodefaults \
-device VGA,bus=pcie.0,addr=0x1 \
-device pcie-root-port,id=pcie_root_port_0,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
-device pcie-root-port,id=pcie_root_port_1,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
-device pcie-root-port,id=pcie_root_port_2,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_yvw268de/monitor-qmpmonitor1-20181017-004217-U4Tik3JV,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_yvw268de/monitor-catch_monitor-20181017-004217-U4Tik3JV,server,nowait \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device pvpanic,ioport=0x505,id=idaVJ26s \
-chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_yvw268de/serial-serial0-20181017-004217-U4Tik3JV,server,nowait \
-device isa-serial,chardev=serial_id_serial0 \
-chardev socket,id=seabioslog_id_20181017-004217-U4Tik3JV,path=/var/tmp/avocado_yvw268de/seabios-20181017-004217-U4Tik3JV,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20181017-004217-U4Tik3JV,iobase=0x402 \
-device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
-device qemu-xhci,id=usb1,bus=pcie.0-root-port-5,addr=0x0 \
-device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \
-object iothread,id=iothread0 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-6,addr=0x0,iothread=iothread0 \
-drive file=/home/kvm_autotest_root/images/rhel77-64-virtio-scsi.qcow2,if=none,format=qcow2,cache=none,werror=stop,rerror=stop,id=drive-system \
-device scsi-hd,drive=drive-system,id=image1,bootindex=0 \
-device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0 \
-device virtio-scsi-pci,id=scsi1,bus=pcie.0-root-port-8,addr=0x0 \
-object pr-manager-helper,id=helper0,path=/var/run/qemu-pr-helper.sock \
-drive file=/dev/sdc,if=none,format=raw,id=drive-data,file.pr-manager=helper0 \
-device scsi-block,drive=drive-data,id=data-disk1,bus=scsi1.0 \
-device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 \
-device virtio-net-pci,mac=9a:82:83:84:85:86,id=idWBc2X6,vectors=4,netdev=idX17Mug,bus=pcie.0-root-port-7,addr=0x0 \
-netdev tap,id=idX17Mug,vhost=on \
-m 4G \
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
-cpu 'Westmere',hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+kvm_pv_unhalt \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-vnc :0 \
-rtc base=utc,clock=host,driftfix=slew \
-boot order=cdn,once=d,menu=off,strict=off \
-enable-kvm \
-monitor stdio \
-qmp tcp:0:4444,server,nowait \
3. test persistent reservation in guest.
cat test-persistent.sh
#! /bin/sh
sg_persist --no-inquiry -v --out --register-ignore --param-sark 123aaa "$@"
sg_persist --no-inquiry --in -k "$@"
sg_persist --no-inquiry -v --out --reserve --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --release --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -r "$@"
sg_persist --no-inquiry -v --out --register --param-rk 123aaa --prout-type 5 "$@"
sg_persist --no-inquiry --in -k "$@"
(1)
# sh test-persisten.sh /dev/sdb
Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00
PR out: command (Register and ignore existing key) successful
PR generation=0x1, 1 registered reservation key follows:
0x123aaa
Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00
PR out: command (Reserve) successful
PR generation=0x1, Reservation follows:
Key=0x123aaa
scope: LU_SCOPE, type: Write Exclusive, registrants only
Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00
PR out: command (Release) successful
PR generation=0x1, there is NO reservation held
Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00
PR out: command (Register) successful
PR generation=0x1, there are NO registered reservation keys
(2) restart guest with multipath (pass-through /dev/mapper/mpatha)
# sh test-persisten.sh /dev/sdb
Persistent Reservation Out cmd: 5f 06 00 00 00 00 00 00 18 00
PR out: command (Register and ignore existing key) successful
PR generation=0x2, 2 registered reservation keys follow:
0x123aaa
0x123aaa
Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00
PR out: command (Reserve) successful
PR generation=0x2, Reservation follows:
Key=0x123aaa
scope: LU_SCOPE, type: Write Exclusive, registrants only
Persistent Reservation Out cmd: 5f 02 05 00 00 00 00 00 18 00
PR out: command (Release) successful
PR generation=0x2, there is NO reservation held
Persistent Reservation Out cmd: 5f 00 05 00 00 00 00 00 18 00
PR out: command (Register) successful
PR generation=0x2, there are NO registered reservation keys
after step 2, guest boot up normally.
after step 3, persistent reservation works well.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:2553 |
Description of problem: When try to start a guest with 'reservations' for disk, failed with error "No persistent reservation manager with id 'pr-helper0'". Version-Release number of selected component (if applicable): qemu-kvm-rhev-2.12.0-29.el7.x86_64 libvirt-4.5.0-18.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. Edit a guest with 'reservations' for a disk <disk type='block' device='lun'> <driver name='qemu' type='raw'/> <source dev='/dev/sdb'> <reservations managed='yes'/> </source> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' target='3' unit='0'/> </disk> 2. Try to start the guest: # virsh start rhel7.6 error: Failed to start domain rhel7.6 error: internal error: qemu unexpectedly closed the monitor: 2019-05-27T07:51:31.476989Z qemu-kvm: -drive file=/dev/sdb,file.pr-manager=pr-helper0,format=raw,if=none,id=drive-scsi0-0-3-0: No persistent reservation manager with id 'pr-helper0' 3.manually start qemu-pr-helper.service # systemctl start qemu-pr-helper.service 4.Try to start guest again: still fails 5. Try to start guest with following reservation, also fails. <reservations managed='no'> <source type='unix' path='/var/run/qemu-pr-helper.sock' mode='client'/> </reservations> Actual results: As step 2,4,5. Expected results: Guest should start successfully without error. Additional info: 1. works well for qemu-kvm-rhev-2.12.0-28.el7.x86_64 2. libvirtd and qemu logs pls refer to attachement.