Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1574887 - ansible instalation prerequisite.yml fails with FAILED! => {"changed": false, "msg": "rmtree failed: [Errno 16] Device or resource busy: '/var/lib/docker'"}
Summary: ansible instalation prerequisite.yml fails with FAILED! => {"changed": false...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.9.z
Assignee: Scott Dodson
QA Contact: Wenkai Shi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-04 09:24 UTC by Priyanka Kanthale
Modified: 2018-08-29 14:43 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
If CRI-O was enabled and /var/lib/docker was a mount point rather than a directory, the installation would fail with the error "Device or resource busy: '/var/lib/docker'". The installation playbooks have been fixed to account for /var/lib/docker being a mountpoint and this error should no longer occur.
Clone Of:
Environment:
Last Closed: 2018-08-29 14:42:31 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:2549 0 None None None 2018-08-29 14:43:15 UTC

Description Priyanka Kanthale 2018-05-04 09:24:31 UTC
Description of problem:

When running prerequisite.yml the installation fails with:

fatal: [xyz]: FAILED! => {"changed": false, "msg": "rmtree failed: [Errno 16] Device or resource busy: '/var/lib/docker'"}

On system /var/lib/docker is a mount point per:

 https://docs.openshift.com/container-platform/3.9/install_config/install/host_preparation.html

#  df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/docker--vg-dockerlv   50G   33M   50G   1% /var/lib/docker

The playbook ignores this situation and it does not un-mount it before trying to delete

https://github.com/openshift/openshift-ansible/blob/33b127157d3339caa690e6b238a81f7fe83f1383/roles/container_runtime/tasks/common/setup_docker_symlink.yml
  - name: Remove the old docker location
      file:
        state: absent
        path: "{{ docker_default_storage_path }}"

Customers workaround: 

added an unmount action in the playbook and get passed this problem

 - name: "xyz unmount /var/lib/docker"
      shell: umount "{{ docker_default_storage_path }}"
      become: true
   - name: Remove the old docker location
      file:
        state: absent
        path: "{{ docker_default_storage_path }}"


Actual results:
Playbook for prerequisite.yml fails. 

Expected results:
It should not fail.

Comment 1 Daein Park 2018-08-01 10:42:43 UTC
If the docker storage is configured with overlay2 to install with CRI-O, you can encounter this issue due to the storage driver differences of unmounted status as follows,

* devicemapper was mounted to /var/lib/docker/containers and /var/lib/docker/devicemapper as top level mount.
  And the docker service stopped with unmount of all the top level mount points.
  
* overlay2 was mounted to /var/lib/docker as top level mount, /var/lib/docker/containers and /var/lib/docker/overlay2 mounted as submounts.
  And the docker service stop result in unmount of all the submount points without unmounting top level mount point.

So the following task is required after stopping docker service, ensure the docker storage mount point is unmounted.

~~~
    - name: Umount the top level mount point
      mount:
        path: "{{ docker_default_storage_path }}"
        state: unmounted
~~~

I've opened a PR here: https://github.com/openshift/openshift-ansible/pull/9382

Comment 2 Scott Dodson 2018-08-02 12:59:17 UTC
Pr has been merged and backported to 3.9.

Comment 4 Wenkai Shi 2018-08-21 06:55:14 UTC
Verified with version openshift-ansible-3.9.41-1.git.0.4c55974.el7.noarch, during docker storage is configured with overlay2 to install with CRI-O, the playbook is ensure the unmount to top level mount point.

ansible-playbook -i hosts -v /usr/share/asnible/openshift-ansible/playbooks/prerequisites.yml
...
TASK [container_runtime : ensure the unmount of top level mount point] *********
Tuesday 21 August 2018  14:42:00 +0800 (0:00:00.746)       0:02:14.289 ******** 
ok: [master.example.com] => {"changed": false, "dump": "0", "failed": false, "fstab": "/etc/fstab", "name": "/var/lib/docker", "opts": "defaults", "passno": "0"}
ok: [node.example.com] => {"changed": false, "dump": "0", "failed": false, "fstab": "/etc/fstab", "name": "/var/lib/docker", "opts": "defaults", "passno": "0"}
...

Comment 6 errata-xmlrpc 2018-08-29 14:42:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2549


Note You need to log in before you can comment on or make changes to this bug.