Bug 1344016 - Original error from libvirt: Failed to recv file descriptor: Transport endpoint is not connected [code=38 int1=107]
Summary: Original error from libvirt: Failed to recv file descriptor: Transport endpoi...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: libvirt
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1344509 1344819 1346630 (view as bug list)
Depends On:
Blocks: TRACKER-bugs-affecting-libguestfs
TreeView+ depends on / blocked
 
Reported: 2016-06-08 14:09 UTC by Richard W.M. Jones
Modified: 2016-12-04 08:59 UTC (History)
18 users (show)

Fixed In Version: libvirt-1.3.5-2.fc25.x86_64
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-06-14 11:27:54 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
build.log (104.32 KB, text/plain)
2016-06-08 14:09 UTC, Richard W.M. Jones
no flags Details

Description Richard W.M. Jones 2016-06-08 14:09:38 UTC
Created attachment 1166033 [details]
build.log

Description of problem:

This has regressed, again.

libguestfs: error: could not create appliance through libvirt.
Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct
Original error from libvirt: Failed to recv file descriptor: Transport endpoint is not connected [code=38 int1=107]

See full log attached.

Version-Release number of selected component (if applicable):

libvirt-client 1.3.5-1.fc25

How reproducible:

?

Steps to Reproduce:
1. Run any libguestfs program, libgestfs-test-tool, etc.

Additional info:

See the build log attached.

Comment 1 Richard W.M. Jones 2016-06-08 14:11:35 UTC
Works OK in libvirt 1.3.3.1-2.fc24.

Comment 2 Cole Robinson 2016-06-08 15:10:39 UTC
this isn't the same issue despite the error sounds familiar... in my test libvirt is opening the URI fine, but something is failing when starting the VM. maybe virtlogd related?

Comment 3 Cole Robinson 2016-06-09 20:40:36 UTC
*** Bug 1344509 has been marked as a duplicate of this bug. ***

Comment 4 ShawnO 2016-06-10 13:07:32 UTC
Just wanted to confirm that it's in the 1.3.5-1.fc25 version.  Lucky my dnf directory still had the previous version.

[root@vhost-dev2 packages]# rpm -Uvh --oldpackage libvirt*1.3.4-2.fc25.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:libvirt-client-1.3.4-2.fc25      ################################# [  3%]
   2:libvirt-daemon-1.3.4-2.fc25      warning: /etc/libvirt/libvirtd.conf created as /etc/libvirt/libvirtd.conf.rpmnew
################################# [  6%]
   3:libvirt-daemon-driver-network-1.3################################# [  9%]
   4:libvirt-daemon-driver-nwfilter-1.################################# [ 12%]
   5:libvirt-daemon-config-nwfilter-1.################################# [ 15%]
   6:libvirt-daemon-config-network-1.3################################# [ 18%]
   7:libvirt-daemon-driver-lxc-1.3.4-2################################# [ 21%]
   8:libvirt-daemon-driver-qemu-1.3.4-################################# [ 24%]
   9:libvirt-daemon-driver-interface-1################################# [ 26%]
  10:libvirt-daemon-driver-libxl-1.3.4################################# [ 29%]
  11:libvirt-daemon-driver-nodedev-1.3################################# [ 32%]
  12:libvirt-daemon-driver-secret-1.3.################################# [ 35%]
  13:libvirt-daemon-driver-storage-1.3################################# [ 38%]
  14:libvirt-daemon-driver-uml-1.3.4-2################################# [ 41%]
  15:libvirt-daemon-driver-vbox-1.3.4-################################# [ 44%]
  16:libvirt-daemon-driver-xen-1.3.4-2################################# [ 47%]
  17:libvirt-1.3.4-2.fc25             ################################# [ 50%]
Cleaning up / removing...
  18:libvirt-1.3.5-1.fc25             ################################# [ 53%]
  19:libvirt-daemon-config-network-1.3################################# [ 56%]
  20:libvirt-daemon-config-nwfilter-1.################################# [ 59%]
  21:libvirt-daemon-driver-lxc-1.3.5-1################################# [ 62%]
  22:libvirt-daemon-driver-qemu-1.3.5-################################# [ 65%]
  23:libvirt-daemon-driver-network-1.3################################# [ 68%]
  24:libvirt-daemon-driver-nwfilter-1.################################# [ 71%]
  25:libvirt-daemon-driver-interface-1################################# [ 74%]
  26:libvirt-daemon-driver-libxl-1.3.5################################# [ 76%]
  27:libvirt-daemon-driver-nodedev-1.3################################# [ 79%]
  28:libvirt-daemon-driver-secret-1.3.################################# [ 82%]
  29:libvirt-daemon-driver-storage-1.3################################# [ 85%]
  30:libvirt-daemon-driver-uml-1.3.5-1################################# [ 88%]
  31:libvirt-daemon-driver-vbox-1.3.5-################################# [ 91%]
  32:libvirt-daemon-driver-xen-1.3.5-1################################# [ 94%]
  33:libvirt-daemon-1.3.5-1.fc25      ################################# [ 97%]
  34:libvirt-client-1.3.5-1.fc25      ################################# [100%]
[root@vhost-dev2 packages]#

virsh # list
 Id    Name                           State
----------------------------------------------------
 1     jumpbox                        running
 2     management                     running
 3     Samba-Server2                  running
 4     dev-cloud                      running
 5     Samba-Server1                  running
 6     vRouter2                       running

virsh # quit

everything is working again now.

Comment 5 Cole Robinson 2016-06-11 15:24:20 UTC

*** This bug has been marked as a duplicate of bug 1344819 ***

Comment 6 Cole Robinson 2016-06-11 15:25:20 UTC
*** Bug 1344819 has been marked as a duplicate of this bug. ***

Comment 7 Richard W.M. Jones 2016-06-13 11:12:01 UTC
Looking at this a bit more closely, I wonder if it is related to a bug
we found in libguestfs which occurs only with kernel 4.7.0+

Something changed in the way that SCM_RIGHTS works, so that previously
working file descriptor passing over Unix domain sockets started to
fail (but with a different error - I was seeing -EINVAL).

I'm afraid I didn't investigate the problem too closely, as I took
the opportunity to rewrite the code in question which fixed it too:

https://github.com/libguestfs/libguestfs/commit/69e5410e6264f58082883beb5254726a1113a4a5

Comment 8 Peter Gervase 2016-06-13 12:44:27 UTC
(In reply to ShawnO from comment #4)
> Just wanted to confirm that it's in the 1.3.5-1.fc25 version.  Lucky my dnf
> directory still had the previous version.
> 
...
> everything is working again now.

I didn't have the rpms still on my system and trying to use dnf history didn't work either. I went to 
http://mirror.us.leaseweb.net/fedora/linux/releases/test/24_Alpha/Everything/x86_64/os/Packages/l/
and downloaded a number of libvirt rpms. I then was able to install them and get things working:
# rpm -Uvh --oldpackage /data/downloads/libvirt-*
warning: /data/downloads/libvirt-1.3.2-1.fc24.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 81b46521: NOKEY
Preparing...                          ################################# [100%]
        package libvirt-glib-0.2.3-2.fc24.x86_64 is already installed
# mv libvirt-glib-0.2.3-2.fc24.x86_64.rpm /
# rpm -Uvh --oldpackage /data/downloads/libvirt-*
warning: /data/downloads/libvirt-1.3.2-1.fc24.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 81b46521: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:libvirt-client-1.3.2-1.fc24      ################################# [  3%]
   2:libvirt-daemon-1.3.2-1.fc24      ################################# [  5%]
   3:libvirt-daemon-driver-network-1.3################################# [  8%]
   4:libvirt-daemon-driver-nwfilter-1.################################# [ 11%]
   5:libvirt-daemon-driver-qemu-1.3.2-################################# [ 13%]
   6:libvirt-daemon-driver-interface-1################################# [ 16%]
   7:libvirt-daemon-driver-nodedev-1.3################################# [ 18%]
   8:libvirt-daemon-driver-secret-1.3.################################# [ 21%]
   9:libvirt-daemon-driver-storage-1.3################################# [ 24%]
  10:libvirt-daemon-config-nwfilter-1.################################# [ 26%]
  11:libvirt-daemon-config-network-1.3################################# [ 29%]
  12:libvirt-daemon-driver-lxc-1.3.2-1################################# [ 32%]
  13:libvirt-daemon-driver-libxl-1.3.2################################# [ 34%]
  14:libvirt-daemon-driver-uml-1.3.2-1################################# [ 37%]
  15:libvirt-daemon-driver-vbox-1.3.2-################################# [ 39%]
  16:libvirt-daemon-driver-xen-1.3.2-1################################# [ 42%]
  17:libvirt-1.3.2-1.fc24             ################################# [ 45%]
  18:libvirt-daemon-kvm-1.3.2-1.fc24  ################################# [ 47%]
  19:libvirt-python-1.3.2-2.fc24      ################################# [ 50%]
Cleaning up / removing...
  20:libvirt-1.3.5-1.fc25             ################################# [ 53%]
  21:libvirt-daemon-kvm-1.3.5-1.fc25  ################################# [ 55%]
  22:libvirt-daemon-config-network-1.3################################# [ 58%]
  23:libvirt-daemon-config-nwfilter-1.################################# [ 61%]
  24:libvirt-python-1.3.5-1.fc25      ################################# [ 63%]
  25:libvirt-daemon-driver-qemu-1.3.5-################################# [ 66%]
  26:libvirt-daemon-driver-lxc-1.3.5-1################################# [ 68%]
  27:libvirt-daemon-driver-network-1.3################################# [ 71%]
  28:libvirt-daemon-driver-nwfilter-1.################################# [ 74%]
  29:libvirt-daemon-driver-interface-1################################# [ 76%]
  30:libvirt-daemon-driver-nodedev-1.3################################# [ 79%]
  31:libvirt-daemon-driver-secret-1.3.################################# [ 82%]
  32:libvirt-daemon-driver-storage-1.3################################# [ 84%]
  33:libvirt-daemon-driver-libxl-1.3.5################################# [ 87%]
  34:libvirt-daemon-driver-uml-1.3.5-1################################# [ 89%]
  35:libvirt-daemon-driver-vbox-1.3.5-################################# [ 92%]
  36:libvirt-daemon-driver-xen-1.3.5-1################################# [ 95%]
  37:libvirt-daemon-1.3.5-1.fc25      ################################# [ 97%]
  38:libvirt-client-1.3.5-1.fc25      ################################# [100%]
# systemctl restart libvirtd
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     capsule                        shut off
 -     proxy                          shut off
 -     sat6                           shut off
 -     stlouis                        shut off

# virsh start proxy
Domain proxy started

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 1     proxy                          running
 -     capsule                        shut off
 -     sat6                           shut off
 -     stlouis                        shut off

# 


So just in case you don't have the dnf cache, there's another possible workaround.

Comment 9 Cole Robinson 2016-06-13 12:51:05 UTC
Simpler workaround is grab the RPMs from koji download-build --arch x86_64 libvirt-1.3.4-2.fc25

I'm guessing this is related to the systemd dep patches, some combo of systemd 230, and/or this libvirt commit:

commit c0bc172383c2c955394589e5808457935ae06f1d
Author: Daniel P. Berrange <berrange>
Date:   Mon Jun 6 15:03:27 2016 +0100

    systemd: directly notify systemd instead of using sd_notify

I'm stuck working on other stuff at the moment so I don't have time to check... testing is probably difficult because that patch is required to even make things build with rawhide systemd, but there were these alternate patches that were never applied:

http://www.redhat.com/archives/libvir-list/2016-May/msg02011.html

Comment 10 Cole Robinson 2016-06-13 13:07:27 UTC
(In reply to Richard W.M. Jones from comment #7)
> Looking at this a bit more closely, I wonder if it is related to a bug
> we found in libguestfs which occurs only with kernel 4.7.0+
> 
> Something changed in the way that SCM_RIGHTS works, so that previously
> working file descriptor passing over Unix domain sockets started to
> fail (but with a different error - I was seeing -EINVAL).
> 
> I'm afraid I didn't investigate the problem too closely, as I took
> the opportunity to rewrite the code in question which fixed it too:
> 
> https://github.com/libguestfs/libguestfs/commit/
> 69e5410e6264f58082883beb5254726a1113a4a5

Just saw this on fedora devel, and it lists both libvirt and libguestfs, so maybe you're on to something:

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/VUOTESHSWFRCYPXIVG6BSMAUITS7QCK2/

Comment 11 Richard W.M. Jones 2016-06-13 13:54:43 UTC
I recompiled libvirt against glibc 2.23.90-22.fc25 and it
appears to have fixed things.

http://koji.fedoraproject.org/koji/taskinfo?taskID=14472719

If it passes the libguestfs test suite and works for a few other
people, we can call this bug fixed.  Possibly either mark it as a
dupe of bug 1344830 or blocking that bug.

Comment 12 Pavel Grunt 2016-06-14 11:26:37 UTC
thanks!

I can confirm that libvirt-1.3.5-2.fc25.x86_64 fixes the problem (starting vms using virsh) for me

Comment 13 Jiri Denemark 2016-06-15 06:53:56 UTC
*** Bug 1346630 has been marked as a duplicate of this bug. ***

Comment 14 Robin Powell 2016-12-04 07:38:54 UTC
So what are users on F24 supposed to do in the meantime?  I can't start any systems unless I turn selinux off due to this issue.

Comment 15 Richard W.M. Jones 2016-12-04 08:59:50 UTC
I don't think this bug had anything to do with SELinux, so
you're probably seeing a different problem.  File a new bug
with full debug information as outlined here:

http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs


Note You need to log in before you can comment on or make changes to this bug.