Bug 1574887
| Summary: | ansible instalation prerequisite.yml fails with FAILED! => {"changed": false, "msg": "rmtree failed: [Errno 16] Device or resource busy: '/var/lib/docker'"} | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Priyanka Kanthale <pkanthal> |
| Component: | Installer | Assignee: | Scott Dodson <sdodson> |
| Status: | CLOSED ERRATA | QA Contact: | Wenkai Shi <weshi> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.9.0 | CC: | aos-bugs, dapark, jialiu, jokerman, mmccomas, wmeng |
| Target Milestone: | --- | ||
| Target Release: | 3.9.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
If CRI-O was enabled and /var/lib/docker was a mount point rather than a directory, the installation would fail with the error "Device or resource busy: '/var/lib/docker'". The installation playbooks have been fixed to account for /var/lib/docker being a mountpoint and this error should no longer occur.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-08-29 14:42:31 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
If the docker storage is configured with overlay2 to install with CRI-O, you can encounter this issue due to the storage driver differences of unmounted status as follows,
* devicemapper was mounted to /var/lib/docker/containers and /var/lib/docker/devicemapper as top level mount.
And the docker service stopped with unmount of all the top level mount points.
* overlay2 was mounted to /var/lib/docker as top level mount, /var/lib/docker/containers and /var/lib/docker/overlay2 mounted as submounts.
And the docker service stop result in unmount of all the submount points without unmounting top level mount point.
So the following task is required after stopping docker service, ensure the docker storage mount point is unmounted.
~~~
- name: Umount the top level mount point
mount:
path: "{{ docker_default_storage_path }}"
state: unmounted
~~~
I've opened a PR here: https://github.com/openshift/openshift-ansible/pull/9382
Pr has been merged and backported to 3.9. Verified with version openshift-ansible-3.9.41-1.git.0.4c55974.el7.noarch, during docker storage is configured with overlay2 to install with CRI-O, the playbook is ensure the unmount to top level mount point.
ansible-playbook -i hosts -v /usr/share/asnible/openshift-ansible/playbooks/prerequisites.yml
...
TASK [container_runtime : ensure the unmount of top level mount point] *********
Tuesday 21 August 2018 14:42:00 +0800 (0:00:00.746) 0:02:14.289 ********
ok: [master.example.com] => {"changed": false, "dump": "0", "failed": false, "fstab": "/etc/fstab", "name": "/var/lib/docker", "opts": "defaults", "passno": "0"}
ok: [node.example.com] => {"changed": false, "dump": "0", "failed": false, "fstab": "/etc/fstab", "name": "/var/lib/docker", "opts": "defaults", "passno": "0"}
...
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2549 |
Description of problem: When running prerequisite.yml the installation fails with: fatal: [xyz]: FAILED! => {"changed": false, "msg": "rmtree failed: [Errno 16] Device or resource busy: '/var/lib/docker'"} On system /var/lib/docker is a mount point per: https://docs.openshift.com/container-platform/3.9/install_config/install/host_preparation.html # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/docker--vg-dockerlv 50G 33M 50G 1% /var/lib/docker The playbook ignores this situation and it does not un-mount it before trying to delete https://github.com/openshift/openshift-ansible/blob/33b127157d3339caa690e6b238a81f7fe83f1383/roles/container_runtime/tasks/common/setup_docker_symlink.yml - name: Remove the old docker location file: state: absent path: "{{ docker_default_storage_path }}" Customers workaround: added an unmount action in the playbook and get passed this problem - name: "xyz unmount /var/lib/docker" shell: umount "{{ docker_default_storage_path }}" become: true - name: Remove the old docker location file: state: absent path: "{{ docker_default_storage_path }}" Actual results: Playbook for prerequisite.yml fails. Expected results: It should not fail.