RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1506144 - Failed to get "write" lock when continue inmigrated guest
Summary: Failed to get "write" lock when continue inmigrated guest
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.5-Alt
Hardware: All
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Fam Zheng
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-25 09:05 UTC by Xujun Ma
Modified: 2017-11-06 15:28 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-06 15:28:27 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Xujun Ma 2017-10-25 09:05:19 UTC
Description of problem:
Failed to get "write" lock when continue inmigrated guest

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.10.0-3.el7.ppc64le

How reproducible:
100%

Steps to Reproduce:
1.Boot up a guest with command:
/usr/libexec/qemu-kvm \
 -smp 16,sockets=1,cores=2,threads=8 \
 -S \
 -m 4096 \
 -nodefaults \
 -monitor stdio \
 -device nec-usb-xhci,id=xhci \
 -device usb-tablet,id=tablet \
 -device usb-kbd,id=kbd \
 -device virtio-scsi-pci,bus=pci.0 \
 -device scsi-hd,id=scsi-hd0,drive=scsi-hd0-dr0,bootindex=0 \
 -drive file=rhel74-ppc64le-virtio-scsi.qcow2,if=none,id=scsi-hd0-dr0,format=qcow2,cache=none 
2.Boot up destination guest with command:
/usr/libexec/qemu-kvm \
 -smp 16,sockets=1,cores=2,threads=8 \
 -S \
 -m 4096 \
 -nodefaults \
 -monitor stdio \
 -device nec-usb-xhci,id=xhci \
 -device usb-tablet,id=tablet \
 -device usb-kbd,id=kbd \
 -device virtio-scsi-pci,bus=pci.0 \
 -device scsi-hd,id=scsi-hd0,drive=scsi-hd0-dr0,bootindex=0 \
 -drive file=rhel74-ppc64le-virtio-scsi.qcow2,if=none,id=scsi-hd0-dr0,format=qcow2,cache=none 
 -incoming tcp:0:7777
3.run "c" in hmp of destination guest
(hmp)c

Actual results:
(qemu) c
Failed to get "write" lock
Is another process using the image?

Expected results:
guest continue without any error.

Additional info:
the same issue on x86 and power8 .
only this qemu build,no this problem with last build(qemu-kvm-rhev-2.10.0-2.el7.ppc64le)

Comment 2 Yanan Fu 2017-10-25 11:52:11 UTC
This should be a regression bz on qemu-kvm-rhev-20.10.0-3.el7. (For x86 part).

I guess, this issue may be related with bz 1378241 .


Another failed scenario is:

1. boot one vm with drive_image1

2. Do "driver_mirror" to "target1"
{'execute': 'drive-mirror', 'arguments': {'mode': 'absolute-paths', 'format': 'qcow2', 'device': u'drive_image1', 'speed': 0, 'sync': 'full', 'target': '/home/kvm_autotest_root/images/target1.qcow2'}, 'id': 'GhHUY3Lw'}

3. After "query-block-jobs" enter to ready status (busy = False, and ready = True), stop vm and re-open to the new target:

{'execute': 'block-job-complete', 'arguments': {'device': u'drive_image1'}, 'id': 'bxwVHyx7'}

4. Compare drive_image1 and the new target image will failed:
# qemu-img compare rhel75-64-virtio-scsi.qcow2 target1.qcow2'
qemu-img: Could not open '/home/kvm_autotest_root/images/target1.qcow2': Failed to get shared "write" lock  Is another process using the image?


Expect result:
Image compare can execute normally, and they are equal.


Another info:
qemu command line:
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga qxl \
    -device pci-bridge,id=pci_bridge,bus=pci.0,addr=0x3,chassis_nr=1 \
    -device intel-hda,bus=pci.0,addr=0x4 \
    -device hda-duplex  \
    -chardev socket,id=qmp_id_qmp1,path=/var/tmp/avocado_KUbU0i/monitor-qmp1-20171022-094818-GvZDPgyy,server,nowait \
    -mon chardev=qmp_id_qmp1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_KUbU0i/monitor-catch_monitor-20171022-094818-GvZDPgyy,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idh5Nsk5  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_KUbU0i/serial-serial0-20171022-094818-GvZDPgyy,server,nowait \
    -device isa-serial,chardev=serial_id_serial0 \
    -device virtio-serial-pci,id=virtio_serial_pci0,bus=pci.0,addr=0x5 \
    -chardev socket,path=/var/tmp/avocado_KUbU0i/virtio_port-vs-20171022-094818-GvZDPgyy,nowait,id=idOUa0w2,server \
    -device virtserialport,id=id100IC2,name=vs,bus=virtio_serial_pci0.0,chardev=idOUa0w2 \
    -object rng-random,filename=/dev/random,id=passthrough-nutMLa0t \
    -device virtio-rng-pci,id=virtio-rng-pci-VDXxUIom,rng=passthrough-nutMLa0t,bus=pci.0,addr=0x6  \
    -chardev socket,id=seabioslog_id_20171022-094818-GvZDPgyy,path=/var/tmp/avocado_KUbU0i/seabios-20171022-094818-GvZDPgyy,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20171022-094818-GvZDPgyy,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -device nec-usb-xhci,id=usb2,bus=pci.0,addr=0x7 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x8 \
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel75-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:17:18:19:1a:1b,id=idEM75bG,vectors=4,netdev=idUM4m0x,bus=pci.0,addr=0x9  \
    -netdev tap,id=idUM4m0x,vhost=on,vhostfd=26,fd=25 \
    -m 4096  \
    -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
    -cpu 'Nehalem',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb2.0,port=1  \
    -spice port=3000,password=123456,addr=0,tls-port=3200,x509-dir=/tmp/spice_x509d,tls-channel=main,tls-channel=inputs,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off  \
    -no-hpet \
    -enable-kvm  \
    -watchdog i6300esb \
    -watchdog-action reset \
    -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa \

Comment 3 Yanan Fu 2017-10-25 12:41:21 UTC
Sorry for paste the second scenario.  Checked the bz 1378241

Comment 4 Yanan Fu 2017-10-25 12:48:35 UTC
Sorry for paste the second scenario.  Checked the bz 1378241 carefully, it is normal, may be .

Hi Fam:
From bz 1378241, i got, qemu-img info/check a running guest should be reject. So, what is the expect result when the guest is "stop" ?

Comment 5 Fam Zheng 2017-10-25 13:27:13 UTC
Stop doesn't release the lock, so it's the same.

One can use -U option to igonre the lock (incompatible to "check -r" though).

Comment 6 Ping Li 2017-10-25 14:10:10 UTC
According to the result below, key point of this bug is that whether the migration is finished. If it is not finished, i think we can close the bug as NOTABUG.

packages tested:
kernel-3.10.0-745.el7.x86_64
qemu-kvm-rhev-2.10.0-3.el7

test steps:
1. Boot a guest at the source as below and check the status.
/usr/libexec/qemu-kvm \
    -S \
    ...
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/tests/diskfile/rhel75.qcow2 \
    ...

(qemu) info status
VM status: paused (prelaunch)

2. Boot a guest at the destination as below and check the status.
/usr/libexec/qemu-kvm \
    -S \
    ...
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/tests/diskfile/rhel75.qcow2 \
    ...
    -incoming tcp:0:7777

(qemu) info status
VM status: paused (inmigrate)

3. Start to do migration at the source
(qemu) c
(qemu) info status
VM status: running
(qemu) migrate -d tcp:127.0.0.1:7777
(qemu) info migrate
globals: store-global-state=1, only_migratable=0, send-configuration=1, send-section-footer=1
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off return-path: off 
Migration status: active
total time: 4868 milliseconds
expected downtime: 300 milliseconds
setup: 57 milliseconds
transferred ram: 145036 kbytes
throughput: 268.57 mbps
remaining ram: 157656 kbytes
total ram: 8405832 kbytes
duplicate: 2030308 pages
skipped: 0 pages
normal: 31736 pages
normal bytes: 126944 kbytes
dirty sync count: 1
page size: 4 kbytes

4. Run continue at the destination before the migration is finished.
(qemu) info status
VM status: paused (inmigrate)
(qemu) c
Failed to get "write" lock
Is another process using the image?

5. Run command continue at the destination after the migration is finished.
(qemu) info status
VM status: paused
(qemu) c

Comment 7 Yanan Fu 2017-10-26 03:17:01 UTC
I think this bug is bz. If boot a vm at destination with "- S", no matter execute "cont" or not, the status always be "VM status: paused (inmigrate)" before migrate is finished. So, I think no "write" lock in this situation is more reasonable. thanks!

Comment 8 Fam Zheng 2017-10-26 08:15:12 UTC
Yanan, what do you mean by "no write lock"? The lock is acquired at the src QEMU independently from the existence of dst QEMU.

From pingl's and my tests, migrating to -S dest QEMU works well, as long as you run the "cont" command at the right point.

Is the error that one cannot "cont" until migration is done a problem for libvirt?

Comment 9 Xujun Ma 2017-10-27 06:45:36 UTC
Hi all
I just copy the steps of failed automation case,but it's not relatetd to migration.It's easy to reproduce it,just boot up a guest with a block device,then use "qemu-img info" to check the using qcow2 file .it will raise the same error:
qemu-img: Could not open 'ALT-Server-7.4-ppc64le-virtio-scsi.qcow2': Failed to get shared "write" lock
Is another process using the image?

Comment 10 Fam Zheng 2017-10-27 06:52:26 UTC
Yes, this is expected since bz 1378241.

Comment 11 Yanan Fu 2017-10-27 12:08:35 UTC
(In reply to Fam Zheng from comment #8)
> Yanan, what do you mean by "no write lock"? The lock is acquired at the src
> QEMU independently from the existence of dst QEMU.
> 

Sorry, i misunderstood, i thought the lock is for dst QEMU before. As the status of dst QEMU always be "VM status: paused (inmigrate)" before and after "cont", so i thought it is a bz.

> From pingl's and my tests, migrating to -S dest QEMU works well, as long as
> you run the "cont" command at the right point.
> 
> Is the error that one cannot "cont" until migration is done a problem for
> libvirt?
It is ok when test with with libvirt. I check the QEMU command when boot guest and do migrate with libvirt, it use "-S" for both src and dst QEMU. But VM is running when boot vm in src, and when i see the vm is dst, it is running too.

IMO, libvirt send "cont" to QEMU in background. For dst, "cont" be sent after migrate finished. So it can avoid this problem.

Thanks!

Comment 12 Xujun Ma 2017-10-31 08:20:57 UTC
(In reply to Fam Zheng from comment #10)
> Yes, this is expected since bz 1378241.

ok,but I don't know if it's normal here.we have to modify automation script if it's normal.

Comment 13 Fam Zheng 2017-11-06 15:28:27 UTC
What's described in comment 6 is the expected behavior, I don't see a bug here.


Note You need to log in before you can comment on or make changes to this bug.