This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 892273 - libvirt monitor socket didn't show up sometimes
libvirt monitor socket didn't show up sometimes
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.0
Unspecified Unspecified
urgent Severity urgent
: rc
: ---
Assigned To: Martin Kletzander
Virtualization Bugs
: Reopened
Depends On: 895901
Blocks: 1051364
  Show dependency treegraph
 
Reported: 2013-01-06 04:09 EST by Mohua Li
Modified: 2014-06-17 20:43 EDT (History)
12 users (show)

See Also:
Fixed In Version: libvirt-1.1.1-19.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 895901 1051364 (view as bug list)
Environment:
Last Closed: 2014-06-13 06:45:58 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Mohua Li 2013-01-06 04:09:36 EST
Description of problem:

sometimes libvirt monitor socket didn't show up which will make libguestfs fail to startup, just met this issue from time to time, don't know how to reproduce, 
possible reproducer is to start more than 20 appliance 

12/22 13:18:32 INFO |libguestfs:0069| ---------------- Test output ----------------
12/22 13:18:34 INFO |guestfs_su:0686| GS:libguestfs: trace: add_drive_ro "/usr/local/staf/test/RHEV/libguestfs-autotest/autotest/client/tests/libguestfs/images/Augeas.ext2.raw"
12/22 13:18:34 INFO |guestfs_su:0686| GS:libguestfs: trace: add_drive_ro = 0
12/22 13:18:34 INFO |guestfs_su:0686| GS:libguestfs: trace: launch
12/22 13:18:34 INFO |guestfs_su:0686| GS:libguestfs: trace: get_tmpdir
12/22 13:18:34 INFO |guestfs_su:0686| GS:libguestfs: trace: get_tmpdir = "/tmp"
12/22 13:18:36 INFO |guestfs_su:0686| GS:libguestfs: trace: get_cachedir
12/22 13:18:36 INFO |guestfs_su:0686| GS:libguestfs: trace: get_cachedir = "/var/tmp"
12/22 13:18:36 INFO |guestfs_su:0686| GS:libguestfs: trace: get_cachedir
12/22 13:18:36 INFO |guestfs_su:0686| GS:libguestfs: trace: get_cachedir = "/var/tmp"
12/22 13:19:14 INFO |guestfs_su:0686| GS:libguestfs: trace: get_cachedir
12/22 13:19:14 INFO |guestfs_su:0686| GS:libguestfs: trace: get_cachedir = "/var/tmp"
12/22 13:19:18 INFO |guestfs_su:0686| GS:libguestfs: error: could not create appliance through libvirt: monitor socket did not show up.: No such file or directory [code=38 domain=10]
12/22 13:19:18 INFO |guestfs_su:0686| GS:libguestfs: trace: launch = -1 (error)
12/22 13:19:18 INFO |guestfs_su:0686| GS:libguestfs: trace: close
12/22 13:19:18 INFO |guestfs_su:0686| GS:(Process terminated with status 1)
12/22 13:19:18 INFO |libguestfs:0072| ---------------- End of test output ----------------
12/22 13:19:18 ERROR|libguestfs:0092| Test failed: Test 'aug_get' ended with 'libguestfs: error: could not create appliance through libvirt: monitor socket did not show up.: No such file or directory [code=38 domain=10]' 
12/22 13:19:18 DEBUG|libguestfs:0093| Postprocessing on error...
12/22 13:19:18 ERROR|      test:0415| Exception escaping from test:
Traceback (most recent call last):




Version-Release number of selected component (if applicable):
libguestfs-1.20.1-4.el7.x86_64

How reproducible:
sometimes

Steps to Reproduce:
1, startup more than 20 guestfish appliance 
  
Actual results:


Expected results:


Additional info:
Comment 1 Mohua Li 2013-01-10 21:51:37 EST
i have tried several test run, and now met this problem quite often, like below,


01/10 21:06:30 INFO |guestfs_su:0686| GS:lvm partition...
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: add_drive "/usr/local/staf/test/RHEV/libguestfs-autotest/autotest/client/tests/libguestfs/images/fs_mount.btrfs.qcow2"
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: add_drive = 0
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: launch
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: get_tmpdir
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: get_tmpdir = "/tmp"
01/10 21:06:31 INFO |guestfs_su:0686| GS:libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
01/10 21:06:31 INFO |guestfs_su:0686| GS:*stdin*:1: libguestfs: error: could not connect to libvirt (URI = NULL): Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused [code=38 domain=7]
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: launch = -1 (error)
01/10 21:06:31 INFO |guestfs_su:0686| GS:libguestfs: trace: close
Comment 2 Richard W.M. Jones 2013-01-16 04:02:51 EST
I'm pretty sure this has got to be a libvirt bug.  After all,
we send libvirt the same XML every time!  Or perhaps qemu, if
for some reason there is a race/delay between libvirt and qemu
creating the monitor.
Comment 3 Mohua Li 2013-01-21 22:28:55 EST
the libvirt version, libvirt-1.0.1-1.el7.x86_64
Comment 4 Huang Wenlong 2013-01-22 03:14:28 EST
Hi,moli 

I can not reproduce this bug with our ENV, so if it can be fixed ,please help us to verify it ,thanks very much .


Wenlong
Comment 5 Dave Allan 2013-01-22 10:08:36 EST
This problem is thought to be quite rare, so I don't think we can assume it's been fixed yet.
Comment 6 Martin Kletzander 2013-02-11 07:58:25 EST
If libvirt's socket is not showing up, we need some libvirt logs to determine where the problem might reside.  Could you attach full debug logs from the time libvirt started without the socket?  The best thing would be trying to reproduce such start with cleared logs, so there are no other unnecessary starts logged in there.  Thank you in advance.
Comment 7 Mohua Li 2013-03-09 04:14:51 EST
(In reply to comment #6)
> If libvirt's socket is not showing up, we need some libvirt logs to
> determine where the problem might reside.  Could you attach full debug logs
> from the time libvirt started without the socket?  The best thing would be
> trying to reproduce such start with cleared logs, so there are no other
> unnecessary starts logged in there.  Thank you in advance.

sorry for miss this thread, i will provide the log once i could reproduced,
Comment 10 Martin Kletzander 2013-05-30 06:10:36 EDT
Closing as INSUFFICIENT_DATA for now, feel free to reopen if there is any new info (logs etc.).
Comment 11 Richard W.M. Jones 2014-01-04 03:52:24 EST
This occurs in Fedora 20, and therefore will happen in RHEL 7
too.  More information added to the upstream bug (bug 895901).
Comment 12 bfan 2014-01-06 22:12:20 EST
I can't reproduce this bug with steps of reporter, only hit it by the steps of bug 1048818. 

Which way is better to handle this bug, leave it here or close it and reopen again if we find the reproducer 

What's your suggestion? Rich
Comment 13 Richard W.M. Jones 2014-01-07 04:22:02 EST
This is a real bug, we should leave it open.

I don't really know if this bug is the same as bug 1048818,
or different.
Comment 14 Martin Kletzander 2014-01-13 03:55:56 EST
This is really a matter of setting a qemu monitor socket timeout.  There is a discussion in the upstream thread [1] about making the timeout configurable, but even if that's not accepted, the default might be probably changed.

Martin

[1] https://www.redhat.com/archives/libvir-list/2014-January/msg00408.html
Comment 15 Martin Kletzander 2014-01-17 02:47:09 EST
Hopefully fixed upstream by v1.2.1-11-gfe89b68:

commit fe89b687a02d1a8e1dce695a67b4f9d2c254d7b9
Author: Martin Kletzander <mkletzan@redhat.com>
Date:   Thu Jan 9 07:57:59 2014 +0100

    qemu: Change the default unix monitor timeout
Comment 19 yanbing du 2014-01-23 06:22:46 EST
Reproduce this bug on libvirt.x86_64 0:1.1.1-18.el7.
1. Write a wrapper for qemu to waits several seconds (>3):
# cat /tmp/qemu-wrapper.py 
#!/usr/bin/env python

import os
import sys
import time

TIMEOUT = 5
QEMU_PATH = "/usr/libexec/qemu-kvm"

time.sleep(TIMEOUT)

sys.argv[0] = QEMU_PATH
os.execv(QEMU_PATH, sys.argv)

#ll -Z /tmp/qemu-wrapper.py 
-rwxr-xr-x. qemu qemu system_u:object_r:qemu_exec_t:s0 /tmp/qemu-wrapper.py

2. Use this wrapper as an emulator for a guest
# virsh dumpxml test|grep emulator
    <emulator>/tmp/qemu-wrapper.py</emulator>
3. Start the guest by virsh
#virsh start test
error: Failed to start domain test
error: monitor socket did not show up: No such file or directory

#tail /var/log/libvirt/libvirtd.log
2014-01-23 07:23:17.949+0000: 3842: error : qemuMonitorOpenUnix:311 : monitor socket did not show up: No such file or directory

After update libvirt to  libvirt.x86_64 0:1.1.1-19.el7
Step 3, guest can start successfully.
# virsh start test
Domain test started

The bug fix patch change the default unix monitor timeout from 3 seconds to 30 seconds, if change the TIMEOUT in the qemu wrapper to more than 30 seconds, it will fail as expected.
Move this bug to VERIFIED.
Comment 21 Ludek Smid 2014-06-13 06:45:58 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.