Description of problem: While deploying RHOSP-16,the task Push repository to overcloud image fails while using libvirt libguestfs_backend as shown in the actual results How reproducible: always Steps to Reproduce: 1. Deploy RHOS-16 2. Deployment fails at Push repository to overcloud image stage Actual results: TASK [Push repository to overcloud image] ************************************** task path: /home/rhos-ci/jenkins/workspace/DFG-compute-nova-16_director-rhel-virthost-1cont_2comp-ipv4-geneve-live-migration-phase3/infrared/plugins/tripleo-undercloud/tasks/images/repos.yml:66 Friday 27 September 2019 14:48:29 +0000 (0:00:00.079) 0:04:05.932 ****** fatal: [undercloud-0]: FAILED! => { "changed": true, "cmd": [ "virt-copy-in", "-a", "overcloud-full.qcow2", "/tmp/oc_repos/yum.repos.d", "/etc/" ], "delta": "0:00:00.111275", "end": "2019-09-27 14:48:30.686480", "rc": 1, "start": "2019-09-27 14:48:30.575205" } STDERR: libvirt: XML-RPC error : Cannot recv data: Connection reset by peer libguestfs: error: could not connect to libvirt (URI = qemu:///session): Cannot recv data: Connection reset by peer [code=38 int1=104] MSG: non-zero return code Expected results: cmd passes successfully Additional info: Since libvirt should be the preferred backend, opening this bz to fix it. As a solution, we have used "direct"
Please supply the full log when you run ‘virt-copy-in -v -x [rest of args]’
(In reply to Richard W.M. Jones from comment #2) > Please supply the full log when you run ‘virt-copy-in -v -x [rest of args]’ [stack@undercloud-0 ~]$ virt-copy-in -a overcloud-full.qcow2 /tmp/oc_repos/yum.repos.d /etc/ libvirt: XML-RPC error : Cannot recv data: Connection reset by peer libguestfs: error: could not connect to libvirt (URI = qemu:///session): Cannot recv data: Connection reset by peer [code=38 int1=104]
Additional info: [stack@undercloud-0 ~]$ libguestfs-test-tool ************************************************************ * IMPORTANT NOTICE * * When reporting bugs, include the COMPLETE, UNEDITED * output below in your bug report. * ************************************************************ PATH=/home/stack/.local/bin:/home/stack/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin XDG_RUNTIME_DIR=/run/user/1001 SELinux: Enforcing guestfs_get_append: (null) guestfs_get_autosync: 1 guestfs_get_backend: libvirt guestfs_get_backend_settings: [] guestfs_get_cachedir: /var/tmp guestfs_get_hv: /usr/libexec/qemu-kvm guestfs_get_memsize: 500 guestfs_get_network: 0 guestfs_get_path: /usr/lib64/guestfs guestfs_get_pgroup: 0 guestfs_get_program: libguestfs-test-tool guestfs_get_recovery_proc: 1 guestfs_get_smp: 1 guestfs_get_sockdir: /run/user/1001 guestfs_get_tmpdir: /tmp guestfs_get_trace: 0 guestfs_get_verbose: 1 host_cpu: x86_64 Launching appliance, timeout set to 600 seconds. libguestfs: launch: program=libguestfs-test-tool libguestfs: launch: version=1.38.4rhel=8,release=11.1.module+el8.0.0+4084+cceb9f44,libvirt libguestfs: launch: backend registered: unix libguestfs: launch: backend registered: uml libguestfs: launch: backend registered: libvirt libguestfs: launch: backend registered: direct libguestfs: launch: backend=libvirt libguestfs: launch: tmpdir=/tmp/libguestfsnW71E7 libguestfs: launch: umask=0002 libguestfs: launch: euid=1001 libguestfs: libvirt version = 4005000 (4.5.0) libguestfs: guest random name = guestfs-e2wzlvjew06q19pa libguestfs: connect to libvirt libguestfs: opening libvirt handle: URI = qemu:///session, auth = default+wrapper, flags = 0 libvirt: XML-RPC error : Cannot recv data: Connection reset by peer libguestfs: error: could not connect to libvirt (URI = qemu:///session): Cannot recv data: Connection reset by peer [code=38 int1=104] libguestfs: closing guestfs handle 0x5559e3aec670 (state 0) libguestfs: command: run: rm libguestfs: command: run: \ -rf /tmp/libguestfsnW71E7
There's something really broken with your copy of libvirt. It looks like even this command would fail: virsh -c qemu:///session list
Can you try the command from comment 5? Can you check that you can run libvirt in session mode in your environment?
[stack@undercloud-0 ~]$ virsh -c qemu:///session list error: failed to connect to the hypervisor error: Cannot recv data: Connection reset by peer
It seems to be a problem with permissions in the home directory. Try starting libvirtd by hand as the non-root stack user: [stack@undercloud-0 ~]$ libvirtd 2019-10-09 15:21:18.505+0000: 346395: info : libvirt version: 4.5.0, package: 24.module+el8.1.0+3205+41ff0a42 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2019-05-15-13:28:11, ) 2019-10-09 15:21:18.505+0000: 346395: info : hostname: undercloud-0.redhat.local 2019-10-09 15:21:18.505+0000: 346395: error : secretStateInitialize:480 : cannot create config directory '/home/stack/.config/libvirt/secrets': Permission denied 2019-10-09 15:21:18.505+0000: 346395: error : virStateInitialize:667 : Initialisation of secret state driver failed: cannot create config directory '/home/stack/.config/libvirt/secrets': Permission denied 2019-10-09 15:21:18.505+0000: 346395: error : daemonRunStateInit:806 : Driver state initialisation failed /home/stack/.config exists, but /home/stack/.config/libvirt does not exist. Also SELinux is enforcing, so I guess it could be an SELinux issue.
Also: $ touch .config/hello touch: cannot touch '.config/hello': Permission denied There are no SELinux AVCs, and the same happens with SELinux set to Permissive, so I think we can conclude it's not SELinux. danpb mentioned that it might be something to do with container permissions.
I am not really sure, how can we help here. Please reopen or move to relevant component. Thanks.