Description of problem:
purge cluster fails on ceph 2.0 ubuntu cluster on oth mon and osd's

Version-Release number of selected component (if applicable):
Ceph: 10.2.0-4redhat1xenial
Ceph-ansible: 1.0.5-7

How reproducible:
Always

Steps to Reproduce:
1. purge an existing cluster with purge-cluster.yml
2. it fails on mon's and osd's


Actual results:
Purge cluster fails

Expected results:
It should succeed

Additional info:

existing cluster:
root@magna052:~# ceph osd tree
ID WEIGHT  TYPE NAME         UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1 5.39694 root default                                        
-2 2.69847     host magna058                                   
 0 0.89949         osd.0          up  1.00000          1.00000 
 3 0.89949         osd.3          up  1.00000          1.00000 
 5 0.89949         osd.5          up  1.00000          1.00000 
-3 2.69847     host magna077                                   
 1 0.89949         osd.1          up  1.00000          1.00000 
 2 0.89949         osd.2          up  1.00000          1.00000 
 4 0.89949         osd.4          up  1.00000          1.00000 


TASK: [check for anything running ceph] *************************************** 
failed: [magna052] => {"changed": true, "cmd": "ps awux | grep -v grep | grep -q -- ceph-", "delta": "0:00:00.022120", "end": "2016-05-06 06:53:07.232169", "failed": true, "failed_when_result": true, "rc": 0, "start": "2016-05-06 06:53:07.210049", "stdout_lines": [], "warnings": []}
changed: [magna046]
failed: [magna009] => {"changed": true, "cmd": "ps awux | grep -v grep | grep -q -- ceph-", "delta": "0:00:00.021764", "end": "2016-05-06 06:53:07.244531", "failed": true, "failed_when_result": true, "rc": 0, "start": "2016-05-06 06:53:07.222767", "stdout_lines": [], "warnings": []}


TASK: [remove Upstart and apt logs and cache] ********************************* 
failed: [magna046] => {"changed": true, "cmd": "find /var -name '*ceph*' -delete", "delta": "0:00:00.127150", "end": "2016-05-06 06:53:46.685084", "rc": 1, "start": "2016-05-06 06:53:46.557934", "warnings": []}
stderr: find: cannot delete ‘/var/lib/ceph’: Directory not empty

FATAL: all hosts have already failed -- aborting


after running purge:
root@magna052:~# dpkg -l | grep ceph
ii  ceph-base                            10.2.0-4redhat1xenial                    amd64        common ceph daemon libraries and management tools
ii  ceph-common                          10.2.0-4redhat1xenial                    amd64        common utilities to mount and interact with a ceph storage cluster
ii  ceph-fuse                            10.2.0-4redhat1xenial                    amd64        FUSE-based client for the Ceph distributed file system
ii  ceph-mds                             10.2.0-4redhat1xenial                    amd64        metadata server for the ceph distributed file system
ii  ceph-mon                             10.2.0-4redhat1xenial                    amd64        monitor server for the ceph storage system
ii  libcephfs1                           10.2.0-4redhat1xenial                    amd64        Ceph distributed file system client library
ii  python-cephfs                        10.2.0-4redhat1xenial                    amd64        Python libraries for the Ceph libcephfs library
root@magna052:~# 


root@magna052:~# systemctl status ceph*
* ceph-mon.target - ceph target allowing to start/stop all ceph-mon@.service instances at once
   Loaded: loaded (/lib/systemd/system/ceph-mon.target; enabled; vendor preset: enabled)
   Active: active since Thu 2016-05-05 05:10:22 UTC; 1 day 1h ago

May 05 05:10:22 magna052 systemd[1]: Reached target ceph target allowing to start/stop all ceph-mon@.service instances at once.

* ceph-mon - Ceph cluster monitor daemon
   Loaded: loaded (/lib/systemd/system/ceph-mon@.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2016-05-05 05:10:58 UTC; 1 day 1h ago
 Main PID: 16558 (ceph-mon)
    Tasks: 25
   CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon
           `-16558 /usr/bin/ceph-mon -f --cluster ceph --id magna052 --setuser ceph --setgroup ceph

May 05 05:10:58 magna052 systemd[1]: Started Ceph cluster monitor daemon.
May 05 05:10:59 magna052 ceph-mon[16558]: 2016-05-05 05:10:59.167556 7f9c7696c580 -1 WARNING: 'mon addr' config option 10.8.128.52:0/0 does not match monmap file
May 05 05:10:59 magna052 ceph-mon[16558]:          continuing with monmap configuration
May 05 05:10:59 magna052 ceph-mon[16558]: starting mon.magna052 rank 0 at 10.8.128.52:6789/0 mon_data /var/lib/ceph/mon/ceph-magna052 fsid 762e1fa8-8ed9-49d3-b377-04e4d86b23e4

* ceph.target - ceph target allowing to start/stop all ceph*@.service instances at once
   Loaded: loaded (/lib/systemd/system/ceph.target; enabled; vendor preset: enabled)
   Active: active since Thu 2016-05-05 05:10:18 UTC; 1 day 1h ago

May 05 05:10:18 magna052 systemd[1]: Reached target ceph target allowing to start/stop all ceph*@.service instances at once.

* ceph-mds.target - ceph target allowing to start/stop all ceph-mds@.service instances at once
   Loaded: loaded (/lib/systemd/system/ceph-mds.target; enabled; vendor preset: enabled)
   Active: active since Thu 2016-05-05 05:10:20 UTC; 1 day 1h ago



ubuntu@magna058:~$ systemctl status ceph*
* ceph-mds.target - ceph target allowing to start/stop all ceph-mds@.service instances at once
   Loaded: loaded (/lib/systemd/system/ceph-mds.target; enabled; vendor preset: enabled)
   Active: active since Thu 2016-05-05 05:12:00 UTC; 1 day 1h ago

* ceph.target - ceph target allowing to start/stop all ceph*@.service instances at once
   Loaded: loaded (/lib/systemd/system/ceph.target; enabled; vendor preset: enabled)
   Active: active since Thu 2016-05-05 05:11:58 UTC; 1 day 1h ago

* ceph-osd.target - ceph target allowing to start/stop all ceph-osd@.service instances at once
   Loaded: loaded (/lib/systemd/system/ceph-osd.target; enabled; vendor preset: enabled)
   Active: active since Thu 2016-05-05 05:12:01 UTC; 1 day 1h ago

* ceph-osd - Ceph object storage daemon
   Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2016-05-05 05:12:48 UTC; 1 day 1h ago
 Main PID: 17322 (ceph-osd)
   CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd
           `-17322 /usr/bin/ceph-osd -f --cluster ceph --id 3 --setuser ceph --setgroup ceph

* ceph-osd - Ceph object storage daemon
   Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2016-05-05 05:13:00 UTC; 1 day 1h ago
 Main PID: 17977 (ceph-osd)
   CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd
           `-17977 /usr/bin/ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph

* ceph-disk - Ceph disk activation: /dev/sdb2
   Loaded: loaded (/lib/systemd/system/ceph-disk@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2016-05-05 05:12:25 UTC; 1 day 1h ago
 Main PID: 16238 (code=exited, status=1/FAILURE)

* ceph-osd - Ceph object storage daemon
   Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2016-05-05 05:12:33 UTC; 1 day 1h ago
 Main PID: 16587 (ceph-osd)
   CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd
           `-16587 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph

* ceph-disk - Ceph disk activation: /dev/sdc2
   Loaded: loaded (/lib/systemd/system/ceph-disk@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2016-05-05 05:12:43 UTC; 1 day 1h ago
 Main PID: 16950 (code=exited, status=1/FAILURE)