Description of problem:
======================
purge-cluster fails on OSD nodes and  ceph rpm, log files and files from /var/run/ceph are not removed


Version-Release number of selected component (if applicable):
==============================================================
ceph-ansible-1.0.5-23.el7scon.noarch
ceph-installer-1.0.12-3.el7scon.noarch
ceph-common-10.2.2-9.el7cp.x86_64

How reproducible:
==================
always


Steps to Reproduce:
===================
1. follow document and create cluster having one mon and 3 OSD node(each node has 3 OSD) - give all values as argument
2. make sure cluster health is OK
3. purge-cluster ans make sure you gave same argument

 
[root@magna044 ceph-ansible]# ansible-playbook purge-cluster.yml -vv -i  /etc/ansible/hosts  --extra-vars '{"ceph_stable": true, "ceph_origin": "distro", "ceph_stable_rh_storage": true, "monitor_interface": "eno1", "journal_collocation": true, "devices": ["/dev/sdb", "/dev/sdc", "/dev/sdd"], "journal_size": 100, "public_network": "x.x.x.x","cephx": true, "fetch_directory": "~/abc", "radosgw_civetweb_port": "8080", "calamari":true}'

Actual results:
==============

TASK: [check for anything running ceph] *************************************** 
<magna078> REMOTE_MODULE command ps awux | grep -v grep | grep -q -- ceph- #USE_SHELL
<magna084> REMOTE_MODULE command ps awux | grep -v grep | grep -q -- ceph- #USE_SHELL
<magna085> REMOTE_MODULE command ps awux | grep -v grep | grep -q -- ceph- #USE_SHELL
<magna090> REMOTE_MODULE command ps awux | grep -v grep | grep -q -- ceph- #USE_SHELL
failed: [magna090] => {"changed": true, "cmd": "ps awux | grep -v grep | grep -q -- ceph-", "delta": "0:00:00.017119", "end": "2016-07-11 02:23:16.142153", "failed": true, "failed_when_result": true, "rc": 0, "start": "2016-07-11 02:23:16.125034", "stdout_lines": [], "warnings": []}
stderr: grep: write error
changed: [magna078] => {"changed": true, "cmd": "ps awux | grep -v grep | grep -q -- ceph-", "delta": "0:00:00.016761", "end": "2016-07-11 02:23:16.142792", "failed": false, "failed_when_result": false, "rc": 1, "start": "2016-07-11 02:23:16.126031", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}
failed: [magna084] => {"changed": true, "cmd": "ps awux | grep -v grep | grep -q -- ceph-", "delta": "0:00:00.017559", "end": "2016-07-11 02:23:16.145490", "failed": true, "failed_when_result": true, "rc": 0, "start": "2016-07-11 02:23:16.127931", "stdout_lines": [], "warnings": []}
stderr: grep: write error
failed: [magna085] => {"changed": true, "cmd": "ps awux | grep -v grep | grep -q -- ceph-", "delta": "0:00:00.017466", "end": "2016-07-11 02:23:16.147364", "failed": true, "failed_when_result": true, "rc": 0, "start": "2016-07-11 02:23:16.129898", "stdout_lines": [], "warnings": []}
stderr: grep: write error


...
...
...


PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/root/purge-cluster.retry

localhost                  : ok=1    changed=0    unreachable=0    failed=0   
magna078                   : ok=20   changed=13   unreachable=0    failed=0   
magna084                   : ok=7    changed=4    unreachable=0    failed=1   
magna085                   : ok=7    changed=4    unreachable=0    failed=1   
magna090                   : ok=7    changed=4    unreachable=0    failed=1   




----->


[root@magna044 ceph-ansible]# rpm -qa | grep ceph
ceph-ansible-1.0.5-23.el7scon.noarch
mod_fastcgi-2.4.7-1.ceph.el7.x86_64
iozone-3.424-2_ceph.el7.x86_64
ceph-installer-1.0.12-3.el7scon.noarch


[root@magna084 ubuntu]# rpm -qa  |grep ceph
ceph-common-10.2.2-9.el7cp.x86_64
ceph-osd-10.2.2-9.el7cp.x86_64
libcephfs1-10.2.2-9.el7cp.x86_64
ceph-base-10.2.2-9.el7cp.x86_64
ceph-selinux-10.2.2-9.el7cp.x86_64
mod_fastcgi-2.4.7-1.ceph.el7.x86_64
iozone-3.424-2_ceph.el7.x86_64
python-cephfs-10.2.2-9.el7cp.x86_64


=============

[root@magna084 ubuntu]# ls -l /var/lib/ceph
total 20
drwxr-x---. 2 ceph ceph 4096 Jun 27 22:03 bootstrap-mds
drwxr-xr-x. 2 ceph ceph 4096 Jul 11 02:19 bootstrap-osd
drwxr-x---. 2 ceph ceph 4096 Jun 27 22:03 bootstrap-rgw
drwxr-x---. 5 ceph ceph 4096 Jul 11 02:20 osd
drwxr-x---. 2 ceph ceph 4096 Jul 11 02:20 tmp
[root@magna084 ubuntu]# ls -l /var/run/ceph
total 0
[root@magna084 ubuntu]# ls -l /var/log/ceph
total 596
-rw-r--r--. 1 root root   1278 Jul 11 02:20 ceph-osd.0.log
-rw-r--r--. 1 ceph ceph 195226 Jul 11 02:23 ceph-osd.1.log
-rw-r--r--. 1 ceph ceph 178303 Jul 11 02:23 ceph-osd.4.log
-rw-r--r--. 1 ceph ceph 208059 Jul 11 02:23 ceph-osd.8.log
-rw-r--r--. 1 root root  10755 Jul 11 02:20 ceph-osd.admin.log
-rw-r--r--. 1 root root      0 Jul 11 02:20 qemu-guest-4877.log
-rw-r--r--. 1 root root      0 Jul 11 02:20 qemu-guest-4906.log
-rw-r--r--. 1 root root      0 Jul 11 02:20 qemu-guest-5058.log
-rw-r--r--. 1 root root      0 Jul 11 02:20 qemu-guest-5520.log
-rw-r--r--. 1 root root      0 Jul 11 02:20 qemu-guest-5549.log
-rw-r--r--. 1 root root      0 Jul 11 02:20 qemu-guest-5717.log
-rw-r--r--. 1 root root      0 Jul 11 02:20 qemu-guest-5769.log
-rw-r--r--. 1 root root      0 Jul 11 02:20 qemu-guest-6266.log
-rw-r--r--. 1 root root      0 Jul 11 02:20 qemu-guest-6295.log
-rw-r--r--. 1 root root      0 Jul 11 02:20 qemu-guest-6428.log
-rw-r--r--. 1 root root    453 Jul 11 02:22 qemu-guest-7512.log
-rw-r--r--. 1 root root    453 Jul 11 02:22 qemu-guest-7532.log


Additional info: