Bug 1351057

Summary: lxc: when undefine a vm first, cannot destroy it successfully.
Product: Red Hat Enterprise Linux 7 Reporter: yisun
Component: libvirtAssignee: Martin Kletzander <mkletzan>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.3CC: dyuan, rbalakri, xuzhang, yanyang, ydu, yisun
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-2.0.0-4.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-03 18:47:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description yisun 2016-06-29 06:34:01 UTC
Description of problem:
lxc: when undefine a vm first, cannot destroy it successfully. 

Version-Release number of selected component (if applicable):
libvirt-1.3.5-1.el7.x86_64
qemu-kvm-rhev-2.6.0-5.el7.x86_64
virt-install-1.4.0-1.el7.noarch
kenel-3.10.0-382.el7.x86_64

How reproducible:
100%

Please note, this is a regression issue and not reproducible in libvirt 1.3.4-1
Steps to Reproduce:
1. # virt-install --connect lxc:/// --name avocado-vt-vm1 --ram 500 --noautoconsole
WARNING No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.

Starting install...
Creating domain... | 0 B 00:00:00
Domain creation completed.


2. # virsh -c lxc:/// undefine avocado-vt-vm1
Domain avocado-vt-vm1 has been undefined

3. # virsh -c lxc:/// destroy avocado-vt-vm1
error: Disconnected from lxc:/// due to I/O error
error: Failed to destroy domain avocado-vt-vm1
error: End of file while reading data: Input/output error

Actual results:
after undefine vm, destroy failed. 

Expected results:
Destroy vm should be successful in last step. 

Additional info:
During the test, we met another issue but not always reproducible. It's also a IO error issue when try to virt-install a lxc vm. We met this issue with a autoscript, and machines are not reserved, so when this bug have assignee, we'll provide a machine to debug. Steps as follow: 

# virt-install --connect lxc:/// --name avocado-vt-vm1 --ram 500 --noautoconsole; virsh -c lxc:/// list --all
WARNING  No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.

Starting install...
ERROR    End of file while reading data: Input/output error
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect lxc:/// start avocado-vt-vm1
otherwise, please restart your installation.
error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused

system log:
Jun 28 06:27:37 hp-dl320eg8-08 journal: internal error: guest failed to start: Failure in libvirt_lxc startup: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Jun 28 06:27:37 hp-dl320eg8-08 kernel: libvirtd[17119]: segfault at a0 ip 00007f366ab2bcd5 sp 00007f3677eb39b0 error 4 in libvirt_driver_lxc.so[7f366ab13000+43000]
Jun 28 06:27:37 hp-dl320eg8-08 systemd: libvirtd.service: main process exited, code=killed, status=11/SEGV
Jun 28 06:27:37 hp-dl320eg8-08 systemd: Unit libvirtd.service entered failed state.
Jun 28 06:27:37 hp-dl320eg8-08 systemd: libvirtd.service failed.
Jun 28 06:27:37 hp-dl320eg8-08 systemd: libvirtd.service holdoff time over, scheduling restart.
Jun 28 06:27:37 hp-dl320eg8-08 systemd: Starting Virtualization daemon...
Jun 28 06:27:37 hp-dl320eg8-08 systemd: Started Virtualization daemon.

Comment 2 Jiri Denemark 2016-06-29 07:09:44 UTC
*** Bug 1351055 has been marked as a duplicate of this bug. ***

Comment 3 Martin Kletzander 2016-07-20 13:00:31 UTC
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2016-July/msg00807.html

Comment 4 Martin Kletzander 2016-07-24 07:10:23 UTC
Fixed upstream with v2.0.0-231-ge4200bbb9d9b:

commit e4200bbb9d9bbc58a51ca55c6073146126447afb
Author: Martin Kletzander <mkletzan>
Date:   Wed Jul 20 09:23:03 2016 +0200

    lxc: Don't crash by forgetting to ref transient domains

Comment 5 Martin Kletzander 2016-07-24 09:44:04 UTC
Moving to POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2016-July/msg00815.html

Comment 7 Michal Privoznik 2016-08-04 08:53:04 UTC
*** Bug 1363773 has been marked as a duplicate of this bug. ***

Comment 8 yisun 2016-08-05 07:23:07 UTC
Verified on libvirt-2.0.0-4.el7.x86_64: PASSED

# virsh -c lxc:// list --all
 Id    Name                           State
----------------------------------------------------
 32810 avocado-vt-vm1                 running

# getenforce 
Enforcing

# virsh -c lxc:// undefine avocado-vt-vm1
Domain avocado-vt-vm1 has been undefined

# virsh -c lxc:// destroy avocado-vt-vm1
Domain avocado-vt-vm1 destroyed

# virsh -c lxc:// list --all
 Id    Name                           State
----------------------------------------------------



(in another terminal, no "libvirtd.service: main process exited")
# tailf /var/log/messages 

Aug  5 03:06:09 hp-dl320eg8-10 systemd-logind: New session 32 of user root.
Aug  5 03:06:09 hp-dl320eg8-10 systemd: Started Session 32 of user root.
Aug  5 03:06:09 hp-dl320eg8-10 systemd: Starting Session 32 of user root.
Aug  5 03:09:14 hp-dl320eg8-10 dbus-daemon: dbus[699]: avc:  received setenforce notice (enforcing=1)
Aug  5 03:09:14 hp-dl320eg8-10 dbus[699]: avc:  received setenforce notice (enforcing=1)
Aug  5 03:09:14 hp-dl320eg8-10 dbus[699]: [system] Reloaded configuration
Aug  5 03:09:14 hp-dl320eg8-10 dbus-daemon: dbus[699]: [system] Reloaded configuration
Aug  5 03:09:37 hp-dl320eg8-10 journal: End of file while reading data: Input/output error
Aug  5 03:09:37 hp-dl320eg8-10 systemd-machined: Machine lxc-32810-avocado-vt-vm1 terminated.
Aug  5 03:09:37 hp-dl320eg8-10 kernel: virbr0: port 1(vnet0) entered disabled state
Aug  5 03:09:37 hp-dl320eg8-10 kernel: device vnet0 left promiscuous mode
Aug  5 03:09:37 hp-dl320eg8-10 kernel: virbr0: port 1(vnet0) entered disabled state
Aug  5 03:09:37 hp-dl320eg8-10 NetworkManager[701]: <info>  [1470380977.7891] device (vnet0): driver 'veth' does not support carrier detection.
^C

Comment 10 errata-xmlrpc 2016-11-03 18:47:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html