moving to Ceph as this seems completely unrelated to OpenStack.

I don't really understand.  Did you also restart the OSD node?  Can you attach an OSD log?  I need more information here.  I think the clock skew is a red herring.

I am running a negative test in a virtual environment, basically I pull the plug on all the machines running my entire environment, which includes Ceph, wait for all the services to start back up again, and verify each one of them is healthy and recovered after the simulated power outage. So far, this is the only warning/error I got out of this particular test. 
I'm no Ceph expert but I don't think OSD is used by Openstack. However, before the test the cluster was showing OK, and after... well you saw the output. 

If you need a specific log, please let me know where I can find it. Or ping me on irc 'dyasny'on #rhos-mgt and I'll let you into the system to take a look directly

The simplest thing would be for you to reproduce it and then leave the vms in that state for me to look at.

[stack@instack ~]$ ssh heat-admin@192.0.2.8                                                                                                                                                           |
Last login: Wed May 18 21:18:16 2016 from 192.0.2.1
[heat-admin@overcloud-cephstorage-2 ~]$ sudo systemctl status 'ceph*'                                                                                                                                 |
● ceph.service - LSB: Start Ceph distributed file system daemons at boot time
   Loaded: loaded (/etc/rc.d/init.d/ceph)
   Active: failed (Result: exit-code) since Wed 2016-05-18 20:16:22 UTC; 1 day 23h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1146 ExecStart=/etc/rc.d/init.d/ceph start (code=exited, status=1/FAILURE)

May 18 20:16:09 overcloud-cephstorage-2.localdomain ceph[1146]: 2016-05-18 20:16:09.523755 7f7a28451700  0 -- :/1002337 >> 192.168.110.13:6789/0 pipe(0x7f7a18008280 sd=3 :0 s=1 pgs=0 c...79b0).fault
May 18 20:16:12 overcloud-cephstorage-2.localdomain ceph[1146]: 2016-05-18 20:16:12.524921 7f7a28350700  0 -- :/1002337 >> 192.168.110.17:6789/0 pipe(0x7f7a18000c00 sd=3 :0 s=1 pgs=0 c...1120).fault
May 18 20:16:15 overcloud-cephstorage-2.localdomain ceph[1146]: 2016-05-18 20:16:15.525163 7f7a28451700  0 -- :/1002337 >> 192.168.110.11:6789/0 pipe(0x7f7a18008280 sd=3 :0 s=1 pgs=0 c...c520).fault
May 18 20:16:18 overcloud-cephstorage-2.localdomain ceph[1146]: 2016-05-18 20:16:18.525428 7f7a28350700  0 -- :/1002337 >> 192.168.110.17:6789/0 pipe(0x7f7a18000c00 sd=4 :0 s=1 pgs=0 c...13a0).fault
May 18 20:16:21 overcloud-cephstorage-2.localdomain ceph[1146]: 2016-05-18 20:16:21.526002 7f7a28451700  0 -- :/1002337 >> 192.168.110.11:6789/0 pipe(0x7f7a18008280 sd=3 :0 s=1 pgs=0 c...c520).fault
May 18 20:16:21 overcloud-cephstorage-2.localdomain ceph[1146]: failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.1 --keyring=/var/lib/ceph/osd/ceph-1/keyring osd cru...ot=default'
May 18 20:16:22 overcloud-cephstorage-2.localdomain systemd[1]: ceph.service: control process exited, code=exited status=1
May 18 20:16:22 overcloud-cephstorage-2.localdomain systemd[1]: Failed to start LSB: Start Ceph distributed file system daemons at boot time.
May 18 20:16:22 overcloud-cephstorage-2.localdomain systemd[1]: Unit ceph.service entered failed state.
May 18 20:16:22 overcloud-cephstorage-2.localdomain systemd[1]: ceph.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

sudo systemctl start 'ceph*'

works

I guess the systemd script has a 30s timeout and the OSD script couldn't get at the mons in that time (which makes sense, the mons also got restarted and probably weren't up yet).  I'm not really sure what the right behavior here is.

Marking as an RFE and assigning to branto.

access to host provided for debugging

Hi all,

we override the systemd defaults in RHCEPH 2 so this should no longer be an issue there -- we do restart the ceph daemons on failure albeit in a limited fashion (3 times). We did not do anything like that in 1.3 since we did not support systemd just yet -- we used SysV init scripts in 1.3.

@Dan: Can you please retest with OSP 10?

In OSP10, we are currently seeing https://bugzilla.redhat.com/show_bug.cgi?id=1374465 which stops MONs from starting, so OSDs also never do.

I don't see this issue with OSP 11 and ceph-10.2.7-16. Moving to verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1497