Bug 1044769

Summary: Deleted container fails to completely uninstall correctly
Product: [Fedora] Fedora Reporter: Matthew Harmsen <mharmsen>
Component: libvirt-sandboxAssignee: Daniel Berrangé <berrange>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: alee, berrange, dwalsh, nkinder, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-06-29 13:37:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Matthew Harmsen 2013-12-18 23:41:19 UTC
After uninstalling a container, if a container with the same name is re-installed, it will fail to start since the removal of the original container leaves remnants of itself behind preventing operation of the new container.

Example (names and times may have been changed in construction of this example):

(1) Start with a running container called 'pki-ca' on the machine called 'vm=100.example.com':

# systemctl status -l pki-ca_sandbox.service
pki-ca_sandbox.service - Secure Sandbox Container pki-ca
   Loaded: loaded (/etc/systemd/system/pki-ca_sandbox.service; enabled)
   Active: active (running) since Tue 2013-12-10 08:55:54 EST; 7s ago
  Process: 732 ExecStop=/usr/bin/virsh -c lxc:/// destroy pki-ca (code=exited, status=1/FAILURE)
 Main PID: 944 (virt-sandbox-se)
   CGroup: /system.slice/pki-ca_sandbox.service
           `-944 /usr/libexec/virt-sandbox-service-util -c lxc:/// -s pki-ca

Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [ INFO ] Update UTMP about System Reboot/Shutdown is not active.
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [DEPEND] Dependency failed for Update UTMP about System Runlevel Changes.
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [  OK  ] Reached target System Initialization.
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [  OK  ] Listening on D-Bus System Message Bus Socket.
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [  OK  ] Reached target Sockets.
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [  OK  ] Reached target Timers.
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [  OK  ] Reached target Basic System.
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: Starting Apache Tomcat Web Application Container...
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [  OK  ] Started Apache Tomcat Web Application Container.
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [  OK  ] Reached target Multi-User System.

# ls -ld /var/run/libvirt/lxc/pki-ca*
drwxr-xr-x. 2 root root   40 Dec 10 08:55 /var/run/libvirt/lxc/pki-ca.dev
drwxr-xr-x. 2 root root   40 Dec 10 08:55 /var/run/libvirt/lxc/pki-ca.devpts
drwxr-xr-x. 2 root root   40 Dec 10 08:55 /var/run/libvirt/lxc/pki-ca.fuse
-rw-------. 1 root root    3 Dec 10 08:55 /var/run/libvirt/lxc/pki-ca.pid
srwx------. 1 root root    0 Dec 10 08:55 /var/run/libvirt/lxc/pki-ca.sock
-rw-------. 1 root root 5056 Dec 10 08:55 /var/run/libvirt/lxc/pki-ca.xml

# ls /var/lib/libvirt/filesystems/pki-ca
etc  home  root  usr  var



(2) Next, shutdown the 'pki-ca' container and delete it from 'vm-100.example.com':

# systemctl stop pki-ca_sandbox.service

# virt-sandbox-service delete pki-ca



(3) Verify that the 'pki-ca' container has been successfully removed:

# ls /var/lib/libvirt/filesystems/pki-ca
ls: cannot access /var/lib/libvirt/filesystems/pki-ca: No such file or directory



(4) Unfortunately, the following 'pki-ca' container remnants remain:

# ls -ld /var/run/libvirt/lxc/pki-ca*
total 0
drwxr-xr-x. 2 root root 40 Dec 10 08:55 /var/run/libvirt/lxc/pki-ca.dev
drwxr-xr-x. 2 root root 40 Dec 10 08:55 /var/run/libvirt/lxc/pki-ca.devpts
drwxr-xr-x. 2 root root 40 Dec 10 08:55 /var/run/libvirt/lxc/pki-ca.fuse
srwx------. 1 root root  0 Dec 10 08:55 /var/run/libvirt/lxc/pki-ca.sock

# systemctl status -l pki-ca_sandbox.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [  OK  ] Reached target Sockets.
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [  OK  ] Reached target Timers.
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [  OK  ] Reached target Basic System.
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: Starting Apache Tomcat Web Application Container...
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [  OK  ] Started Apache Tomcat Web Application Container.
Dec 10 08:55:55 vm-100.example.com virt-sandbox-service-util[944]: [  OK  ] Reached target Multi-User System.
Dec 17 07:44:26 vm-100.example.com systemd[1]: Stopping Secure Sandbox Container pki-ca...
Dec 17 07:44:27 vm-100.example.com virsh[31411]: Domain pki-ca destroyed
Dec 17 07:44:27 vm-100.example.com systemd[1]: Stopped Secure Sandbox Container pki-ca.
Dec 17 07:44:56 vm-100.example.com systemd[1]: Stopped Secure Sandbox Container pki-ca.



(5) Reinstall 'pki-ca' on 'vm-100.example.com' and attempt to restart the 'pki-ca' container (note the failure):

# virt-sandbox-service create --security="static,label=system_u:system_r:svirt_lxc_net_t:s0:c1,c2"  -N dhcp,source=lan -C -u tomcat.service pki-ca
Created sandbox container dir /var/lib/libvirt/filesystems/pki-ca
Created unit file /etc/systemd/system/pki-ca_sandbox.service
Created sandbox config /etc/libvirt-sandbox/services/pki-ca/config/sandbox.cfg

# systemctl start pki-ca_sandbox.service

# systemctl enable pki-ca_sandbox.service
ln -s '/etc/systemd/system/pki-ca_sandbox.service' '/etc/systemd/system/multi-user.target.wants/pki-ca_sandbox.service'

# systemctl status -l pki-ca_sandbox.service
pki-ca_sandbox.service - Secure Sandbox Container pki-ca
   Loaded: loaded (/etc/systemd/system/pki-ca_sandbox.service; enabled)
   Active: failed (Result: exit-code) since Tue 2013-12-17 08:59:39 EST; 39s ago
 Main PID: 32050 (code=exited, status=1/FAILURE)

Dec 17 08:59:39 vm-100.example.com systemd[1]: Started Secure Sandbox Container pki-ca.
Dec 17 08:59:39 vm-100.example.com systemd[1]: pki-ca_sandbox.service: main process exited, code=exited, status=1/FAILURE
Dec 17 08:59:39 vm-100.example.com virsh[32104]: error: Failed to destroy domain pki-ca
Dec 17 08:59:39 vm-100.example.com virsh[32104]: error: Requested operation is not valid: Domain is not running
Dec 17 08:59:39 vm-100.example.com systemd[1]: pki-ca_sandbox.service: control process exited, code=exited status=1
Dec 17 08:59:39 vm-100.example.com systemd[1]: Unit pki-ca_sandbox.service entered failed state.



(6) The current workaround involves rebooting the 'vm-100.example.com' and restarting the 'pki-ca' container:

# reboot

# systemctl start pki-ca_sandbox.service



(7) Verify that the 'pki-ca' container has been successfully reinstalled:

# systemctl status -l pki-ca_sandbox.service
pki-ca_sandbox.service - Secure Sandbox Container pki-ca
   Loaded: loaded (/etc/systemd/system/pki-ca_sandbox.service; enabled)
   Active: active (running) since Tue 2013-12-17 09:28:04 EST; 7s ago
  Process: 732 ExecStop=/usr/bin/virsh -c lxc:/// destroy pki-ca (code=exited, status=1/FAILURE)
 Main PID: 941 (virt-sandbox-se)
   CGroup: /system.slice/pki-ca_sandbox.service
           `-941 /usr/libexec/virt-sandbox-service-util -c lxc:/// -s pki-ca

Dec 17 09:28:05 vm-100.example.com virt-sandbox-service-util[941]: [ INFO ] Update UTMP about System Reboot/Shutdown is not active.
Dec 17 09:28:05 vm-100.example.com virt-sandbox-service-util[941]: [DEPEND] Dependency failed for Update UTMP about System Runlevel Changes.
Dec 17 09:28:05 vm-100.example.com virt-sandbox-service-util[941]: [  OK  ] Reached target System Initialization.
Dec 17 09:28:05 vm-100.example.com virt-sandbox-service-util[941]: [  OK  ] Listening on D-Bus System Message Bus Socket.
Dec 17 09:28:05 vm-100.example.com virt-sandbox-service-util[941]: [  OK  ] Reached target Sockets.
Dec 17 09:28:05 vm-100.example.com virt-sandbox-service-util[941]: [  OK  ] Reached target Timers.
Dec 17 09:28:05 vm-100.example.com virt-sandbox-service-util[941]: [  OK  ] Reached target Basic System.
Dec 17 09:28:05 vm-100.example.com virt-sandbox-service-util[941]: Starting Apache Tomcat Web Application Container...
Dec 17 09:28:05 vm-100.example.com virt-sandbox-service-util[941]: [  OK  ] Started Apache Tomcat Web Application Container.
Dec 17 09:28:05 vm-100.example.com virt-sandbox-service-util[941]: [  OK  ] Reached target Multi-User System.

# ls -ld /var/run/libvirt/lxc/pki-ca*
drwxr-xr-x. 2 root root   40 Dec 17 09:28 /var/run/libvirt/lxc/pki-ca.dev
drwxr-xr-x. 2 root root   40 Dec 17 09:28 /var/run/libvirt/lxc/pki-ca.devpts
drwxr-xr-x. 2 root root   40 Dec 17 09:28 /var/run/libvirt/lxc/pki-ca.fuse
-rw-------. 1 root root    3 Dec 17 09:28 /var/run/libvirt/lxc/pki-ca.pid
srwx------. 1 root root    0 Dec 17 09:28 /var/run/libvirt/lxc/pki-ca.sock
-rw-------. 1 root root 5056 Dec 17 09:28 /var/run/libvirt/lxc/pki-ca.xml

# ls /var/lib/libvirt/filesystems/pki-ca
etc  home  root  usr  var

Comment 1 Daniel Berrangé 2013-12-19 13:55:25 UTC
Can you tell me what RPM versions of libvirt-sandbox and systemd you have installed ? Also please provide the /etc/systemd/system/pki-ca_sandbox.service file contents

Comment 2 Matthew Harmsen 2013-12-19 23:10:54 UTC
(In reply to Daniel Berrange from comment #1)
> Can you tell me what RPM versions of libvirt-sandbox and systemd you have
> installed ?

# rpm -q libvirt-sandbox systemd
libvirt-sandbox-0.5.1-1.fc20.x86_64
systemd-208-9.fc20.x86_64

> Also please provide the /etc/systemd/system/pki-ca_sandbox.service file contents

# cat /etc/systemd/system/pki-ca_sandbox.service 

[Unit]
Description=Secure Sandbox Container pki-ca
Requires=libvirtd.service
After=libvirtd.service
ReloadPropagatedFrom=tomcat.service

[Service]
Type=simple
ExecStart=/usr/libexec/virt-sandbox-service-util -c lxc:/// -s pki-ca
ExecReload=/usr/bin/virt-sandbox-service -c lxc:/// reload -u tomcat.service pki-ca
ExecStop=/usr/bin/virsh -c lxc:/// destroy pki-ca

[Install]
WantedBy=multi-user.target

Comment 3 Fedora End Of Life 2015-05-29 10:02:56 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 4 Fedora End Of Life 2015-06-29 13:37:14 UTC
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.