Bug 1756437

Summary: libvirt: XML-RPC error : Cannot recv data: Connection reset by peer while using virt-copy-in command
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Archit Modi <amodi>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED NOTABUG QA Contact: chhu
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.1CC: jsuchane, ptoscano, rjones, xuzhang, yafu, yanqzhan, yoguo
Target Milestone: rc   
Target Release: 8.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-15 14:36:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1758964    

Description Archit Modi 2019-09-27 16:00:46 UTC
Description of problem:
While deploying RHOSP-16,the task Push repository to overcloud image fails while using libvirt libguestfs_backend as shown in the actual results

How reproducible: always


Steps to Reproduce:
1. Deploy RHOS-16 
2. Deployment fails at Push repository to overcloud image stage

Actual results:
TASK [Push repository to overcloud image] **************************************
task path: /home/rhos-ci/jenkins/workspace/DFG-compute-nova-16_director-rhel-virthost-1cont_2comp-ipv4-geneve-live-migration-phase3/infrared/plugins/tripleo-undercloud/tasks/images/repos.yml:66
Friday 27 September 2019  14:48:29 +0000 (0:00:00.079)       0:04:05.932 ****** 
fatal: [undercloud-0]: FAILED! => {
    "changed": true, 
    "cmd": [
        "virt-copy-in", 
        "-a", 
        "overcloud-full.qcow2", 
        "/tmp/oc_repos/yum.repos.d", 
        "/etc/"
    ], 
    "delta": "0:00:00.111275", 
    "end": "2019-09-27 14:48:30.686480", 
    "rc": 1, 
    "start": "2019-09-27 14:48:30.575205"
}

STDERR:

libvirt: XML-RPC error : Cannot recv data: Connection reset by peer
libguestfs: error: could not connect to libvirt (URI = qemu:///session): Cannot recv data: Connection reset by peer [code=38 int1=104]


MSG:

non-zero return code


Expected results:
cmd passes successfully


Additional info:
Since libvirt should be the preferred backend, opening this bz to fix it. As a solution, we have used "direct"

Comment 2 Richard W.M. Jones 2019-09-27 16:49:40 UTC
Please supply the full log when you run ‘virt-copy-in -v -x [rest of args]’

Comment 3 Archit Modi 2019-09-30 22:28:24 UTC
(In reply to Richard W.M. Jones from comment #2)
> Please supply the full log when you run ‘virt-copy-in -v -x [rest of args]’

[stack@undercloud-0 ~]$ virt-copy-in -a overcloud-full.qcow2 /tmp/oc_repos/yum.repos.d /etc/
libvirt: XML-RPC error : Cannot recv data: Connection reset by peer
libguestfs: error: could not connect to libvirt (URI = qemu:///session): Cannot recv data: Connection reset by peer [code=38 int1=104]

Comment 4 Archit Modi 2019-09-30 22:29:40 UTC
Additional info:
[stack@undercloud-0 ~]$ libguestfs-test-tool
     ************************************************************
     *                    IMPORTANT NOTICE
     *
     * When reporting bugs, include the COMPLETE, UNEDITED
     * output below in your bug report.
     *
     ************************************************************
PATH=/home/stack/.local/bin:/home/stack/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
XDG_RUNTIME_DIR=/run/user/1001
SELinux: Enforcing
guestfs_get_append: (null)
guestfs_get_autosync: 1
guestfs_get_backend: libvirt
guestfs_get_backend_settings: []
guestfs_get_cachedir: /var/tmp
guestfs_get_hv: /usr/libexec/qemu-kvm
guestfs_get_memsize: 500
guestfs_get_network: 0
guestfs_get_path: /usr/lib64/guestfs
guestfs_get_pgroup: 0
guestfs_get_program: libguestfs-test-tool
guestfs_get_recovery_proc: 1
guestfs_get_smp: 1
guestfs_get_sockdir: /run/user/1001
guestfs_get_tmpdir: /tmp
guestfs_get_trace: 0
guestfs_get_verbose: 1
host_cpu: x86_64
Launching appliance, timeout set to 600 seconds.
libguestfs: launch: program=libguestfs-test-tool
libguestfs: launch: version=1.38.4rhel=8,release=11.1.module+el8.0.0+4084+cceb9f44,libvirt
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=libvirt
libguestfs: launch: tmpdir=/tmp/libguestfsnW71E7
libguestfs: launch: umask=0002
libguestfs: launch: euid=1001
libguestfs: libvirt version = 4005000 (4.5.0)
libguestfs: guest random name = guestfs-e2wzlvjew06q19pa
libguestfs: connect to libvirt
libguestfs: opening libvirt handle: URI = qemu:///session, auth = default+wrapper, flags = 0
libvirt: XML-RPC error : Cannot recv data: Connection reset by peer
libguestfs: error: could not connect to libvirt (URI = qemu:///session): Cannot recv data: Connection reset by peer [code=38 int1=104]
libguestfs: closing guestfs handle 0x5559e3aec670 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfsnW71E7

Comment 5 Richard W.M. Jones 2019-10-01 07:52:12 UTC
There's something really broken with your copy of libvirt.  It looks like
even this command would fail:

virsh -c qemu:///session list

Comment 6 Jaroslav Suchanek 2019-10-05 13:28:03 UTC
Can you try the command from comment 5? Can you check that you can run libvirt in session mode in your environment?

Comment 7 Archit Modi 2019-10-09 14:56:12 UTC
[stack@undercloud-0 ~]$ virsh -c qemu:///session list
error: failed to connect to the hypervisor
error: Cannot recv data: Connection reset by peer

Comment 8 Richard W.M. Jones 2019-10-09 15:22:50 UTC
It seems to be a problem with permissions in the home directory.

Try starting libvirtd by hand as the non-root stack user:

[stack@undercloud-0 ~]$ libvirtd 
2019-10-09 15:21:18.505+0000: 346395: info : libvirt version: 4.5.0, package: 24.module+el8.1.0+3205+41ff0a42 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2019-05-15-13:28:11, )
2019-10-09 15:21:18.505+0000: 346395: info : hostname: undercloud-0.redhat.local
2019-10-09 15:21:18.505+0000: 346395: error : secretStateInitialize:480 : cannot create config directory '/home/stack/.config/libvirt/secrets': Permission denied
2019-10-09 15:21:18.505+0000: 346395: error : virStateInitialize:667 : Initialisation of secret state driver failed: cannot create config directory '/home/stack/.config/libvirt/secrets': Permission denied
2019-10-09 15:21:18.505+0000: 346395: error : daemonRunStateInit:806 : Driver state initialisation failed

/home/stack/.config exists, but /home/stack/.config/libvirt does not exist.

Also SELinux is enforcing, so I guess it could be an SELinux issue.

Comment 9 Richard W.M. Jones 2019-10-09 15:28:32 UTC
Also:

$ touch .config/hello
touch: cannot touch '.config/hello': Permission denied

There are no SELinux AVCs, and the same happens with SELinux set to Permissive, so
I think we can conclude it's not SELinux.  danpb mentioned that it might be
something to do with container permissions.

Comment 10 Jaroslav Suchanek 2019-10-15 14:36:00 UTC
I am not really sure, how can we help here. Please reopen or move to relevant component. Thanks.