Description of problem: Even when the osd process is not up, the status from the service is active/loaded. [ubuntu@magna034 ~]$ sudo systemctl status ceph.target ● ceph.target - ceph target allowing to start/stop all ceph*@.service instances at once Loaded: loaded (/usr/lib/systemd/system/ceph.target; enabled; vendor preset: enabled) Active: active since Thu 2016-06-16 06:10:34 UTC; 11h ago [ubuntu@magna034 ~]$ ps -eaf | grep ceph ubuntu 11626 11574 0 17:16 pts/0 00:00:00 grep --color=auto ceph [ubuntu@magna034 ~]$ sudo systemctl list-units | grep ceph sys-devices-pci0000:00-0000:00:1f.2-ata2-host1-target1:0:0-1:0:0:0-block-sdb-sdb1.device loaded active plugged Hitachi_HUA722010CLA330 ceph\x20data sys-devices-pci0000:00-0000:00:1f.2-ata2-host1-target1:0:0-1:0:0:0-block-sdb-sdb2.device loaded active plugged Hitachi_HUA722010CLA330 ceph\x20journal sys-devices-pci0000:00-0000:00:1f.2-ata3-host2-target2:0:0-2:0:0:0-block-sdc-sdc1.device loaded active plugged Hitachi_HUA722010CLA330 ceph\x20data sys-devices-pci0000:00-0000:00:1f.2-ata3-host2-target2:0:0-2:0:0:0-block-sdc-sdc2.device loaded active plugged Hitachi_HUA722010CLA330 ceph\x20journal sys-devices-pci0000:00-0000:00:1f.2-ata4-host3-target3:0:0-3:0:0:0-block-sdd-sdd1.device loaded active plugged Hitachi_HUA722010CLA330 ceph\x20data sys-devices-pci0000:00-0000:00:1f.2-ata4-host3-target3:0:0-3:0:0:0-block-sdd-sdd2.device loaded active plugged Hitachi_HUA722010CLA330 ceph\x20journal var-lib-ceph-osd-ceph\x2d0.mount loaded active mounted /var/lib/ceph/osd/ceph-0 var-lib-ceph-osd-ceph\x2d1.mount loaded active mounted /var/lib/ceph/osd/ceph-1 var-lib-ceph-osd-ceph\x2d2.mount loaded active mounted /var/lib/ceph/osd/ceph-2 system-ceph\x2ddisk.slice loaded active active system-ceph\x2ddisk.slice system-ceph\x2dosd.slice loaded active active system-ceph\x2dosd.slice ceph.target loaded active active ceph target allowing to start/stop all ceph*@.service instances at once Version-Release number of selected component (if applicable): How reproducible: 1/1 Steps to Reproduce: 1) Upgraded one the node from 1.3.2 to 2.0 latest 2) rebooted the node, the osd's were up for some time and then died silently 3) regardless of the osd issue the systemctl service files should be reporting the accurate state of the osd process. Please let me know what logs you want me attached for this bug.
Boris, any idea why ceph.target wouldn't display the correct state there?
We had permission issue for /var/log/ceph, not sure that actually created this issue, but also the other issue(dont want to open another bz just now) is we are missing quite a few service files (eg ceph@target, ceph-osd@target) after the upgrade please feel free to use magna034 [ubuntu@magna034 ~]$ sudo systemctl list-units | grep ceph sys-devices-pci0000:00-0000:00:1f.2-ata2-host1-target1:0:0-1:0:0:0-block-sdb-sdb1.device loaded active plugged Hitachi_HUA722010CLA330 ceph\x20data sys-devices-pci0000:00-0000:00:1f.2-ata2-host1-target1:0:0-1:0:0:0-block-sdb-sdb2.device loaded active plugged Hitachi_HUA722010CLA330 ceph\x20journal sys-devices-pci0000:00-0000:00:1f.2-ata3-host2-target2:0:0-2:0:0:0-block-sdc-sdc1.device loaded active plugged Hitachi_HUA722010CLA330 ceph\x20data sys-devices-pci0000:00-0000:00:1f.2-ata3-host2-target2:0:0-2:0:0:0-block-sdc-sdc2.device loaded active plugged Hitachi_HUA722010CLA330 ceph\x20journal sys-devices-pci0000:00-0000:00:1f.2-ata4-host3-target3:0:0-3:0:0:0-block-sdd-sdd1.device loaded active plugged Hitachi_HUA722010CLA330 ceph\x20data sys-devices-pci0000:00-0000:00:1f.2-ata4-host3-target3:0:0-3:0:0:0-block-sdd-sdd2.device loaded active plugged Hitachi_HUA722010CLA330 ceph\x20journal var-lib-ceph-osd-ceph\x2d0.mount loaded active mounted /var/lib/ceph/osd/ceph-0 var-lib-ceph-osd-ceph\x2d1.mount loaded active mounted /var/lib/ceph/osd/ceph-1 var-lib-ceph-osd-ceph\x2d2.mount loaded active mounted /var/lib/ceph/osd/ceph-2 ceph-osd loaded active running Ceph object storage daemon ceph-osd loaded active running Ceph object storage daemon ceph-osd loaded active running Ceph object storage daemon system-ceph\x2ddisk.slice loaded active active system-ceph\x2ddisk.slice system-ceph\x2dosd.slice loaded active active system-ceph\x2dosd.slice
The output of * sudo systemctl status ceph\*.service ceph\*.target should be more useful, here. Anyway, this looks like the ceph-osd@{0,1,2}.service services were not enabled after upgrade. If these are not enabled (and so they are not even attempted to be started), then ceph.target is correctly in the loaded and active state. Does it help if you manually enable and start the in-the-middle targets (ceph-osd.target, ceph-mon.target, ...)? What is the output of the status command that I suggested when you are hitting this? Does this happen after clean install of 1.3.2 and upgrade to 2.0?
I think this could be the issue(services not enabled), I will try this soon and update.
On one node i tried to enabale the ceph.target but even after reboot it didn't work, on other node i could get the ceph.target only after reboot but its not working as shown below, feel free to login to that system [ubuntu@magna036 ~]$ sudo systemctl status ceph.target ● ceph.target - ceph target allowing to start/stop all ceph*@.service instances at once Loaded: loaded (/usr/lib/systemd/system/ceph.target; enabled; vendor preset: enabled) Active: active since Mon 2016-06-20 18:42:36 UTC; 1min 3s ago [ubuntu@magna036 ~]$ sudo systemctl stop ceph.target [ubuntu@magna036 ~]$ sudo systemctl status ceph.target ● ceph.target - ceph target allowing to start/stop all ceph*@.service instances at once Loaded: loaded (/usr/lib/systemd/system/ceph.target; enabled; vendor preset: enabled) Active: inactive (dead) since Mon 2016-06-20 18:43:45 UTC; 1s ago Jun 20 18:43:45 magna036 systemd[1]: Stopped target ceph target allowing to start/stop all ceph*@.service instances at once. Jun 20 18:43:45 magna036 systemd[1]: Stopping ceph target allowing to start/stop all ceph*@.service instances at once. [ubuntu@magna036 ~]$ ps -eaf | grep ceph ceph 2729 1 0 18:42 ? 00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 3 --setuser ceph --setgroup ceph ceph 2935 1 0 18:42 ? 00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph ceph 3185 1 0 18:43 ? 00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph ubuntu 3563 3492 0 18:43 pts/0 00:00:00 grep --color=auto ceph
Also on the other note, I think we should start recommending reboot instead of restart of ceph daemons, I see the old init files are interfering too much with newer systemd files during restart.
OK, I think I was able to hit ~your issue (well, at least parts of it, I did an update from latest upstream hammer to latest upstream jewel) so some notes: The old daemons continue to run because they keep running with the old defaults but this might be ok. The most critical issue is that we do not change permissions on upgrade (that is why the new daemons are not starting). I'm thinking we should write up a document on how to upgrade from 1.3.2 to 2.0. There are several things that users will probably have to do that might not be all that straight-forward. A scratch write-up would probably look like this: 1.) stop (not disable) the old daemons 2.) upgrade the packages 3.) fix the permissions for the osd/mon/etc files so that ceph user/group could access them (stg like 'chown -R ceph:ceph /var/lib/ceph/' for starters) 4.) do a full SELinux re-label ('touch /.autorelabel') (probably optional but won't hurt) 5.) reboot btw: AFAIK, the targets do not have to necessarily enter a failed state if one of the processes that it depends on fail but I would need to spend more time on that. In any way, I'd consider that as a blocker for 2.0.
Ups, let me fix the last line: I'd NOT consider that as a blocker for 2.0.
(In reply to Boris Ranto from comment #8) > The old daemons continue to run because they keep running with the old > defaults but this might be ok. QE noticed that the old radosgw PID still holds onto port 80, preventing the new systemd-based process from listening on that port (bug 1347323). So we'll want to be sure the instructions tell users to stop the RHCS 1.3 services on a system before upgrading to RHCS 2.0.
What I meant there is that the behaviour is somehow intentional as we do try to minimize the amount of restarts on upgrades. For the upgrade, we should definitely recommend to stop the services first and I have included it in the scratch steps in my previous comment.
Boris, I think we should change this BZ to track the issue you described above: that the .targets do not enter a failed state if one of the processes that the .target depends on fails. QE agreed that is a cosmetic thing that can be re-targeted to 2.1. What do you think?
Ken, this is not a cosmetic thing, I believe we need to change the upgrade procedure a bit, unresolved it will leave system in a bad state during and after upgrade.
@Ken: I don't think it is anything new, it most probably worked like that since we introduced the systemd support. If it is only visible when doing 1.3.2 -> 2.0 upgrade, we can "fix" this by documenting the upgrade procedure. @Vasu: Same here, the most critical thing here is that the files don't automatically get chown'd to ceph user/group. Anyway, it is usually considered ok to require some additional steps when doing a major version upgrade (which 1.3.2 -> 2.0 definitely is). All of the problems should disappear if we provide some reasonable steps on how to upgrade from 1.3.2 to 2.0. I've already suggested the basic steps in Comment 8, I didn't test them just yet, though.
Boris, I am going to try those steps( the first 3 steps exist), steps 4 and 5 are new and I will try that on new setup, but if that doesn't solve ceph.target we should be willing to fix that in 2.0
I think .targets are like runlevels - you can't check the dependent services based on "systemctl status <foo>.target"
@Ken: Yep, you are somehow right. Targets are a somewhat more general version of runlevels -- actually, "runlevels" in systemd are implemented as targets. @Vasu: The first three (especially the chown part) could maybe suffice (as I wrote in the comment, the relabelling part is optional, I suggest the reboot to get a bit of a systemd cleanup, unit files reload, etc). Did you manage to run into any of these issues even when you followed the steps directly? The chown that I suggested might not be sufficient, we might need to chown some files in /var/log/ceph and similar as well, I guess. Anyway, if we do that chown step right then I believe we should be fine and the daemons should be starting automatically.
After some IRC discussion with Vasu, I've come to realize that we should add one more step to the docs and this could probably supersede the 5th 'reboot' step: Enable all sub-targets * sudo systemctl ceph-\*.target and start the services back * sudo systemctl start ceph.target
Fixing the first command... * sudo systemctl enable ceph-\*.target && sudo systemctl start ceph-\*.target
The above steps work reliably so we should make changes to current docs 1) stop the old daemons 2) upgrade the packages 3) fix the permissions for /var/lib/ceph and /var/log/ceph to ceph:ceph 4) For RHEL: do a full SELinux re-label ('touch /.autorelabel') 5) reboot had to rerun fix permission here as well a) fix the permissions for /var/lib/ceph and /var/log/ceph to ceph:ceph b) make sure firewalld rules have open ports for osd daemons 6) Enable all sub-targets * sudo systemctl ceph-\*.target (mon, osd, radosgw) * sudo systemctl ceph.target
@Vasu: We should probably swap 5 and 6, if you enable the targets before reboot, they should start automatically -- that is unless there really is a reproducible permission error after reboot which seems really rather weird and we should probably re-examine the steps if there really is. Also, 5 b) should probably happen before the reboot itself.
Verified
*** Bug 1347448 has been marked as a duplicate of this bug. ***