Description of problem: [RFE] Boot time Ceph OSD's activation adding partprobe in place of udev rules[1]. This will help avoid race conditions between Ceph udev rules and ceph-disk. We have tested adding partprobe in rc.local as following and it has helped multiple customers. --- Below are the steps we have been using to implement the workaround. 1. Edit /etc/rc.local and append the command: partprobe 2. Set /etc/rc.d/rc.local as executable: # chmod 755 /etc/rc.d/rc.local 3. Enable the rc.local service: # systemctl enable rc-local.service 4. And then, reboot your system. --- If we do not want to choose rc.local we can use some other preprocessing scripts for running partprobe. [1] /usr/lib/udev/rules.d/95-ceph-osd.rules Version-Release number of selected component (if applicable): Red Hat Ceph Storage 2.3
Please provide an explanation why this helps. Running partprobe cannot replace udev events because it relies on them. When partprobe runs against a device, it will trigger udev events that will start the OSD. At boot time it will race against the udev events generated by the kernel when the machine comes up and the disks appear. It is my understanding that it helped in some cases but I lack information to understand why. If the machines in question boot without the kernel firing any udev event, that will help indeed. Is that the case ?
(In reply to Loic Dachary from comment #3) > Please provide an explanation why this helps. Running partprobe cannot > replace udev events because it relies on them. When partprobe runs against a > device, it will trigger udev events that will start the OSD. Thanks for the explanation and correcting me. I misunderstood it because I thought partprobe bypasses ceph udev rule and directly trigger activation. I have changed the bug title to fix my misunderstanding. > At boot time it will race against the udev events generated by the kernel > when the machine comes up and the disks appear. I think here you meant on boot time ceph-disk activate triggers the udev envent for OSD activate and it race against the udev events generated by the kernel. or we already use partprobe in ceph-disk activate to activate the devices in boot time? and udev events generated by partprobe for OSD disk activation are racing against the udev events generated by kernel. > > It is my understanding that it helped in some cases but I lack information > to understand why. If the machines in question boot without the kernel > firing any udev event, that will help indeed. Is that the case ? We have suggested this to 4 customers and it has helped them and I can see only one thing why it is helping is adding it in rc.local because when rc.local gets executed all the resources are available in the system and all the kernel udev events are already completed so no race against kernel udev events.
> I think here you meant on boot time ceph-disk activate triggers the udev envent > for OSD activate and it race against the udev events generated by the kernel. > > or > > we already use partprobe in ceph-disk activate to activate the devices in boot time? and udev events generated by partprobe for OSD disk activation are racing against the udev events generated by kernel. partprobe is not used at all during the boot phase. What it does is essentially telling the kernel to forget everything it knows about a device and then inspect the device to fire udev events as if the device was just plugged in. This is quite useful when the device is modified and that's in these situations only that ceph-disk calls partprobe. At boot time the kernel is supposed to send udev events for every disks it has and these udev events will then call ceph-disk activate to run the OSD daemon. Since you had good experience using partprobe at boot time and saw no race of any kind, it either means you got lucky (but since you did that on 4 customers it does not sound like luck). Or it means that for some reasons the udev events are *not* fired when the kernel boots (that sounds more likely). Can you figure that out by generating a full sos report right before rc.local tries to run partprobe ? Note: I don't think the context in which rc.local provides any guarantees that each and every udev event was fired and acted upon when it completes. And even though, the udev event running ceph-disk activate actually delegates the bulk of the action to a systemd service working asynchronously with a global lock.
The workaround here doesn't appear to have resolved the issue for us unfortunately. Is there any indication as to what the upstream issue is?
@Stephen did you try the upstream fix for https://bugzilla.redhat.com/show_bug.cgi?id=1472409 ? It will be in the next release but you can test it by manually changing the timeout in from 300 to 10000 in the ceph-disk@.service file (see https://github.com/ceph/ceph/pull/17133/files)
@loic - we found that it ends up bleeding into the default "down / out" interval. Is there a recommended amount by which we should also increase the "mon osd down out interval", or do you perhaps have any insight into why it takes so long? I assume it directly relates to either the size or number of disks and perhaps is a factor of both?
> we found that it ends up bleeding into the default "down / out" interval. This is an interesting idea. Could you explain in detail what you mean by that ? > any insight into why it takes so long? I don't know for what reason the initialization of a single OSD would take more than a few seconds on a given platform. It would be interesting to get detailed information about what happens on a machine where it is the case. I.e. how much time is spent doing what.
(In reply to Loic Dachary from comment #9) > > we found that it ends up bleeding into the default "down / out" interval. > > This is an interesting idea. Could you explain in detail what you mean by > that ? We have the following in our ceph.conf: mon_osd_auto_mark_new_in = false mon_osd_auto_mark_auto_out_in = false mon_osd_down_out_interval = 600 The default for mon_osd_down_out_interval is 300 so we've modified this from 5 minutes to 10 minutes. We ask Ceph to not auto mark in, disks that were auto marked out to control rebalancing to some extent with mon_osd_auto_mark_auto_out_in = false When OSDs take longer than 10 minutes to come up (If we set CEPH_DISK_TIMEOUT = 10000, ~3 hours) they will be auto marked out and won't come back in necessarily. Even if we didn't have mon_osd_auto_mark_auto_out_in = false, you run the risk of: 1) Some number of OSDs being auto marked out if they do not come up in 10 minutes 2) Rebalancing / recovery begins for those marked auto out 3) Now OSDs come up and get marked auto in 4) Rebalancing / recovery begins again Even if there isn't a large amount of rebalancing, it's possible OSDs can flap under the load of figuring out what should be in / out and what has / has not been rebalanced. In a cluster with 10TB OSDs for instance, this can be fairly painful. The strange thing is this happens with or without data in the cluster and under zero or high load it seems. We see this more so with larger numbers of OSDs in a single host. We also observe one OSD seems to come up every 20 seconds, with 60 OSDs this can be upwards of 20 minutes. > > any insight into why it takes so long? > > I don't know for what reason the initialization of a single OSD would take > more than a few seconds on a given platform. It would be interesting to get > detailed information about what happens on a machine where it is the case. > I.e. how much time is spent doing what. I'm happy to provide any details I can.
*** This bug has been marked as a duplicate of bug 1458007 ***