Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1588419

Summary: virtlogd did not release the qemu log file on source host after migration in some specific condition
Product: Red Hat Enterprise Linux 9 Reporter: yafu <yafu>
Component: libvirtAssignee: Jiri Denemark <jdenemar>
libvirt sub component: General QA Contact: yafu <yafu>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: low    
Priority: unspecified CC: chhu, dyuan, fjin, gveitmic, jdenemar, jsuchane, klaas, lmen, mkalinin, mtessun, shipatil, virt-maint, xuzhang, yanqzhan
Version: unspecifiedKeywords: Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-06-15 10:51:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1897025    
Attachments:
Description Flags
bz1588419_comment8 none

Description yafu 2018-06-07 09:54:29 UTC 
Description of problem:
Do migration both on the source host and target host with the same guest in:
1.On source host, do p2p migration;
2.On target host, do non-p2p migration while the migration did not complete on the source host;
The virtlogd did not release the qemu log file on source host after migration.


Version-Release number of selected component (if applicable):
libvirt-4.3.0-1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Do p2p migration from source to target:
#virsh migrate ovmf qemu+ssh://10.73.130.49/system --live --verbose --p2p

2.Open another terminal and do non-p2p migration on the target host:
#virsh migrate ovmf guest+ssh://10.66.5.76/system --live --verbose
error: internal error: unexpected async job 2

3.After the migration completed in step1, check the open fd of qemu log file on the source host:
# lsof /var/log/libvirt/qemu/ovmf.log
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
virtlogd 3952 root   14w   REG  253,0   239117 67300588 /var/log/libvirt/qemu/ovmf.log

4.Migrate back the guest on the target host:
# virsh migrate ovmf qemu+ssh://10.66.5.76/system --live --verbose --unsafe 
error: Cannot open log file: '/var/log/libvirt/qemu/ovmf.log': Device or resource busy

Actual resutls:
See step 3&4.

Expected results:
virtlogd should release the log file on source host after migration everytime.

Additional info:
Comment 6 Jiri Denemark 2019-09-05 14:26:10 UTC 
I couldn't reproduce this issue. If you can still reproduce this, could you
please provide some more data:

    - domain XML (attaching it to all libvirt bugs is a good idea)
    - check whether the QEMU process for the ovmf domain is still running on
      the source host when virtlogd has the log file open
    - libvirtd debug logs from both hosts (they should always be attached to
      libvirt bugs, esp. the migration ones)
Comment 7 Jiri Denemark 2019-09-05 15:13:06 UTC 
The following upstream commit fixes similar errors coming from qemuDomainObjTaint.
But unfortunately the cause of the error which would result in failure to start
a domain is still unclear.

commit f709377301b919a9fcbfc366e33057f7848bee28
Refs: v5.7.0-9-gf709377301
Author:     Jiri Denemark <jdenemar>
AuthorDate: Thu Sep 5 15:35:35 2019 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Thu Sep 5 17:09:34 2019 +0200

    qemu: Fix qemuDomainObjTaint with virtlogd

    When virtlogd is used to capture QEMU's stdout, qemuDomainObjTaint would
    always fail to write the message to the log file when QEMU is already
    running (i.e., outside qemuProcessLaunch). This can happen during device
    hotplug or by sending a custom QEMU guest agent command:

       	warning : qemuDomainObjTaint:8757 : Domain id=9 name='blaf'
            uuid=9cfa4e37-2930-405b-bcb4-faac1829dad8 is tainted:
            custom-ga-command
       	error : virLogHandlerDomainOpenLogFile:388 : Cannot open log file:
            '/var/log/libvirt/qemu/blaf.log': Device or resource busy
       	error : virNetClientProgramDispatchError:172 : Cannot open log file:
            '/var/log/libvirt/qemu/blaf.log': Device or resource busy

    The fix is easy, we just need to use the right API for appending a
    message to QEMU log file instead of creating a new log context.

    Signed-off-by: Jiri Denemark <jdenemar>
    Reviewed-by: Ján Tomko <jtomko>
Comment 8 Yanqiu Zhang 2019-09-17 11:49:13 UTC 
Hi,

1. Issue can be reproduced on latest rhel-av8.0.1:
libvirt-daemon-5.0.0-12.module+el8.0.1+3755+6782b0ed.x86_64
qemu-kvm-3.1.0-30.module+el8.0.1+3755+6782b0ed.x86_64

Target:
[root@yalzhang ~]# virsh migrate avocado-vt-vm1 --live qemu+ssh://lenovo-8.0***/system --verbose --unsafe
root***'s password:
error: Cannot open log file: '/var/log/libvirt/qemu/avocado-vt-vm1.log': Device or resource busy

Source: (QEMU process exists)
root*** ~]# ps aux|grep qemu
qemu     12140  1.7  0.1 1684436 40480 ?       Sl   06:53   0:00 /usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-4-avocado-vt-vm1/master-key.aes -machine pc-q35-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Penryn,vme=on,ds=off,acpi=off,ss=on,ht=off,tm=off,pbe=off,dtes64=off,ds_cpl=off,vmx=off,est=off,tm2=off,xtpr=off,pdcm=off,xsave=on,x2apic=on,hypervisor=on -m 1024 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 9accd915-ecc0-49e6-8dd9-d711eba5cd2c -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=30,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 -device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 -drive file=/var/lib/libvirt/migrate/yanqzhan/RHEL-8.0.1-updates-20190904.5-x86_64.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=32,id=hostnet0,vhost=on,vhostfd=34 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:27:ab:a5,bus=pci.1,addr=0x0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=35,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pcie.0,addr=0x1 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.6,addr=0x0 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

[root*** ~]# lsof /var/log/libvirt/qemu/avocado-vt-vm1.log
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/0/gvfs
    | Output information may be incomplete.
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
virtlogd 16729 root   17w   REG  253,0    32218 109427800 /var/log/libvirt/qemu/avocado-vt-vm1.log

[root*** ~]# virsh domstate avocado-vt-vm1 --reason
paused (starting up)

Addistional:
After the failed try(Device or resource busy), qemu process on src host disappears, and log file is released, a next migration try can succeed.


2. On latest rhel-av8.1.0, also encounters this issue, but with different error:
libvirt-daemon-5.6.0-5.module+el8.1.0+4229+2e4e348c.x86_64
qemu-kvm-4.1.0-10.module+el8.1.0+4234+33aa4f57.x86_64

Target:
[root*** ~]#  virsh migrate avocado-vt-vm1 --live qemu+ssh://dell-***/system --verbose
error: internal error: child reported (status=125): Requested operation is not valid: Setting different SELinux label on /var/lib/libvirt/qemu/domain-4-avocado-vt-vm1 which is already in use

Source + Additional:
same situation as on rhel-av8.0.1

Logs pls refer to attachments.
Comment 9 Yanqiu Zhang 2019-09-17 11:52:17 UTC 
Created attachment 1615829 [details]
bz1588419_comment8
Comment 15 John Ferlan 2021-09-09 18:29:53 UTC 
Bulk update: Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.
Comment 24 Jiri Denemark 2022-06-15 10:51:29 UTC 
The bug cannot be reproduced with libvirt versions released in RHEL 8.6 and 9.0 and there are no reports of this issue either, so chances are it was already fixed.