RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 892273 - libvirt monitor socket didn't show up sometimes
Summary: libvirt monitor socket didn't show up sometimes
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: ---
Assignee: Martin Kletzander
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 895901
Blocks: 1051364
TreeView+ depends on / blocked
 
Reported: 2013-01-06 09:09 UTC by Mohua Li
Modified: 2014-06-18 00:43 UTC (History)
12 users (show)

Fixed In Version: libvirt-1.1.1-19.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 895901 1051364 (view as bug list)
Environment:
Last Closed: 2014-06-13 10:45:58 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Mohua Li 2013-01-06 09:09:36 UTC 
Description of problem:

sometimes libvirt monitor socket didn't show up which will make libguestfs fail to startup, just met this issue from time to time, don't know how to reproduce, 
possible reproducer is to start more than 20 appliance 

12/22 13:18:32 INFO |libguestfs:0069| ---------------- Test output ----------------
12/22 13:18:34 INFO |guestfs_su:0686| GS:libguestfs: trace: add_drive_ro "/usr/local/staf/test/RHEV/libguestfs-autotest/autotest/client/tests/libguestfs/images/Augeas.ext2.raw"
12/22 13:18:34 INFO |guestfs_su:0686| GS:libguestfs: trace: add_drive_ro = 0
12/22 13:18:34 INFO |guestfs_su:0686| GS:libguestfs: trace: launch
12/22 13:18:34 INFO |guestfs_su:0686| GS:libguestfs: trace: get_tmpdir
12/22 13:18:34 INFO |guestfs_su:0686| GS:libguestfs: trace: get_tmpdir = "/tmp"
12/22 13:18:36 INFO |guestfs_su:0686| GS:libguestfs: trace: get_cachedir
12/22 13:18:36 INFO |guestfs_su:0686| GS:libguestfs: trace: get_cachedir = "/var/tmp"
12/22 13:18:36 INFO |guestfs_su:0686| GS:libguestfs: trace: get_cachedir
12/22 13:18:36 INFO |guestfs_su:0686| GS:libguestfs: trace: get_cachedir = "/var/tmp"
12/22 13:19:14 INFO |guestfs_su:0686| GS:libguestfs: trace: get_cachedir
12/22 13:19:14 INFO |guestfs_su:0686| GS:libguestfs: trace: get_cachedir = "/var/tmp"
12/22 13:19:18 INFO |guestfs_su:0686| GS:libguestfs: error: could not create appliance through libvirt: monitor socket did not show up.: No such file or directory [code=38 domain=10]
12/22 13:19:18 INFO |guestfs_su:0686| GS:libguestfs: trace: launch = -1 (error)
12/22 13:19:18 INFO |guestfs_su:0686| GS:libguestfs: trace: close
12/22 13:19:18 INFO |guestfs_su:0686| GS:(Process terminated with status 1)
12/22 13:19:18 INFO |libguestfs:0072| ---------------- End of test output ----------------
12/22 13:19:18 ERROR|libguestfs:0092| Test failed: Test 'aug_get' ended with 'libguestfs: error: could not create appliance through libvirt: monitor socket did not show up.: No such file or directory [code=38 domain=10]' 
12/22 13:19:18 DEBUG|libguestfs:0093| Postprocessing on error...
12/22 13:19:18 ERROR|      test:0415| Exception escaping from test:
Traceback (most recent call last):




Version-Release number of selected component (if applicable):
libguestfs-1.20.1-4.el7.x86_64

How reproducible:
sometimes

Steps to Reproduce:
1, startup more than 20 guestfish appliance 
  
Actual results:


Expected results:


Additional info:
Comment 1 Mohua Li 2013-01-11 02:51:37 UTC 
i have tried several test run, and now met this problem quite often, like below,


01/10 21:06:30 INFO |guestfs_su:0686| GS:lvm partition...
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: add_drive "/usr/local/staf/test/RHEV/libguestfs-autotest/autotest/client/tests/libguestfs/images/fs_mount.btrfs.qcow2"
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: add_drive = 0
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: launch
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: get_tmpdir
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: get_tmpdir = "/tmp"
01/10 21:06:31 INFO |guestfs_su:0686| GS:libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
01/10 21:06:31 INFO |guestfs_su:0686| GS:*stdin*:1: libguestfs: error: could not connect to libvirt (URI = NULL): Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused [code=38 domain=7]
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: launch = -1 (error)
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: close
Comment 2 Richard W.M. Jones 2013-01-16 09:02:51 UTC 
I'm pretty sure this has got to be a libvirt bug.  After all,
we send libvirt the same XML every time!  Or perhaps qemu, if
for some reason there is a race/delay between libvirt and qemu
creating the monitor.
Comment 3 Mohua Li 2013-01-22 03:28:55 UTC 
the libvirt version, libvirt-1.0.1-1.el7.x86_64
Comment 4 Huang Wenlong 2013-01-22 08:14:28 UTC 
Hi,moli 

I can not reproduce this bug with our ENV, so if it can be fixed ,please help us to verify it ,thanks very much .


Wenlong
Comment 5 Dave Allan 2013-01-22 15:08:36 UTC 
This problem is thought to be quite rare, so I don't think we can assume it's been fixed yet.
Comment 6 Martin Kletzander 2013-02-11 12:58:25 UTC 
If libvirt's socket is not showing up, we need some libvirt logs to determine where the problem might reside.  Could you attach full debug logs from the time libvirt started without the socket?  The best thing would be trying to reproduce such start with cleared logs, so there are no other unnecessary starts logged in there.  Thank you in advance.
Comment 7 Mohua Li 2013-03-09 09:14:51 UTC 
(In reply to comment #6)
> If libvirt's socket is not showing up, we need some libvirt logs to
> determine where the problem might reside.  Could you attach full debug logs
> from the time libvirt started without the socket?  The best thing would be
> trying to reproduce such start with cleared logs, so there are no other
> unnecessary starts logged in there.  Thank you in advance.

sorry for miss this thread, i will provide the log once i could reproduced,
Comment 10 Martin Kletzander 2013-05-30 10:10:36 UTC 
Closing as INSUFFICIENT_DATA for now, feel free to reopen if there is any new info (logs etc.).
Comment 11 Richard W.M. Jones 2014-01-04 08:52:24 UTC 
This occurs in Fedora 20, and therefore will happen in RHEL 7
too.  More information added to the upstream bug (bug 895901).
Comment 12 bfan 2014-01-07 03:12:20 UTC 
I can't reproduce this bug with steps of reporter, only hit it by the steps of bug 1048818. 

Which way is better to handle this bug, leave it here or close it and reopen again if we find the reproducer 

What's your suggestion? Rich
Comment 13 Richard W.M. Jones 2014-01-07 09:22:02 UTC 
This is a real bug, we should leave it open.

I don't really know if this bug is the same as bug 1048818,
or different.
Comment 14 Martin Kletzander 2014-01-13 08:55:56 UTC 
This is really a matter of setting a qemu monitor socket timeout.  There is a discussion in the upstream thread [1] about making the timeout configurable, but even if that's not accepted, the default might be probably changed.

Martin

[1] https://www.redhat.com/archives/libvir-list/2014-January/msg00408.html
Comment 15 Martin Kletzander 2014-01-17 07:47:09 UTC 
Hopefully fixed upstream by v1.2.1-11-gfe89b68:

commit fe89b687a02d1a8e1dce695a67b4f9d2c254d7b9
Author: Martin Kletzander <mkletzan>
Date:   Thu Jan 9 07:57:59 2014 +0100

    qemu: Change the default unix monitor timeout
Comment 19 yanbing du 2014-01-23 11:22:46 UTC 
Reproduce this bug on libvirt.x86_64 0:1.1.1-18.el7.
1. Write a wrapper for qemu to waits several seconds (>3):
# cat /tmp/qemu-wrapper.py 
#!/usr/bin/env python

import os
import sys
import time

TIMEOUT = 5
QEMU_PATH = "/usr/libexec/qemu-kvm"

time.sleep(TIMEOUT)

sys.argv[0] = QEMU_PATH
os.execv(QEMU_PATH, sys.argv)

#ll -Z /tmp/qemu-wrapper.py 
-rwxr-xr-x. qemu qemu system_u:object_r:qemu_exec_t:s0 /tmp/qemu-wrapper.py

2. Use this wrapper as an emulator for a guest
# virsh dumpxml test|grep emulator
    <emulator>/tmp/qemu-wrapper.py</emulator>
3. Start the guest by virsh
#virsh start test
error: Failed to start domain test
error: monitor socket did not show up: No such file or directory

#tail /var/log/libvirt/libvirtd.log
2014-01-23 07:23:17.949+0000: 3842: error : qemuMonitorOpenUnix:311 : monitor socket did not show up: No such file or directory

After update libvirt to  libvirt.x86_64 0:1.1.1-19.el7
Step 3, guest can start successfully.
# virsh start test
Domain test started

The bug fix patch change the default unix monitor timeout from 3 seconds to 30 seconds, if change the TIMEOUT in the qemu wrapper to more than 30 seconds, it will fail as expected.
Move this bug to VERIFIED.
Comment 21 Ludek Smid 2014-06-13 10:45:58 UTC 
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.
Note You need to log in before you can comment on or make changes to this bug.