Hide Forgot
See: http://lists.freedesktop.org/archives/systemd-devel/2015-February/028482.html This is a pretty recent change; I'd been putting off debugging because only gnome-contnuous was broken, but now that Fedora 22 Atomic Host is, filing here. I'll try to get a more precise regression range or bisect. In the meantime, any hints for debugging appreciated.
I'm strongly suspecting this is http://cgit.freedesktop.org/systemd/systemd/commit/?id=06e97888883e2cc12eb6514e80c7f0014295f59b Testing now. If that's the case, there seem to be two questions here: - Why would the device be going inactive? - Regardless, we probably don't want these mounts to be bound to the root device, they should last until shutdown
So the answer here is that systemd-remount-rootfs.service is somehow causing the device to become inactive when remounting / read-write. If I boot with "rw" things work fine. I'll follow up on the list, but it'd be good if we could carry a revert of that patch until it's debugged. Any objections to that?
Pushed temporarily: http://pkgs.fedoraproject.org/cgit/systemd.git/commit/?h=f22&id=9bbe0e92dc59d5a42258c729b105a7d9901eb35e
I suspect this is also behind https://bugzilla.redhat.com/show_bug.cgi?id=1195899 . I'm going to test that theory out now.
systemd-219-5.fc22 has been submitted as an update for Fedora 22. https://admin.fedoraproject.org/updates/systemd-219-5.fc22
Package systemd-219-5.fc22: * should fix your issue, * was pushed to the Fedora 22 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing systemd-219-5.fc22' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2015-2655/systemd-219-5.fc22 then log in and leave karma (feedback).
systemd-219-6.fc22 has been submitted as an update for Fedora 22. https://admin.fedoraproject.org/updates/systemd-219-6.fc22
systemd-219-5.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.
systemd-219-6.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.
Note lots of discussion upstream continues on this: http://lists.freedesktop.org/archives/systemd-devel/2015-May/032030.html
Reaffecting Fedora 23.
I re-ported the patches from walters and applied them to 222: https://maxamillion.fedorapeople.org/systemd-222-5.fc23.src.rpm In initial testing this appears to fix the issue.
I went ahead an put it in F23 DistGit and got a build in koji http://koji.fedoraproject.org/koji/taskinfo?taskID=11169150 If I can get a +1 I'll send an update to bodhi.
We have an atomic host image with this systemd. It seems to boot fine, thus not observing the problem reported in this bug. The image is at: https://dustymabe.fedorapeople.org/Fedora-Cloud-Atomic-23-specialsauce.qcow2.xz
Is there an upstream issue filed on github, or a mailing list discussion with an outcome? Can we reproduce the issue without atomic?
One of the mail threads referenced early on has someone that says they can reproduce (because of recreating a similar situation) on non-atomic: http://lists.freedesktop.org/archives/systemd-devel/2015-February/028810.html I'm going to flag Colin as needinfo on this though because he has the most context.
AFAIK it's already fixed in upstream master.
Nevermind comment #17 - I tried reverting the patch in: https://git.gnome.org/browse/gnome-continuous/commit/manifest.json?id=a3a88b427367ca0f5054f1847efcc1a3e15cb1b7 But it did break, and Vadim reverted: https://git.gnome.org/browse/gnome-continuous/commit/manifest.json?id=a6789105bf6a691b77ef631dee31a35ed5a7d52c (I find gnome-continuous convenient to test the combination of systemd and ostree because it constantly builds, ships, and tests git master of both)
(In reply to Colin Walters from comment #18) > Nevermind comment #17 - I tried reverting the patch in: > > https://git.gnome.org/browse/gnome-continuous/commit/manifest. > json?id=a3a88b427367ca0f5054f1847efcc1a3e15cb1b7 > > But it did break, and Vadim reverted: > > https://git.gnome.org/browse/gnome-continuous/commit/manifest. > json?id=a6789105bf6a691b77ef631dee31a35ed5a7d52c > > (I find gnome-continuous convenient to test the combination of systemd and > ostree because it constantly builds, ships, and tests git master of both) So it's not fixed in upstream master?
Harald, Here is one outcome of a mailing list discussion: http://lists.freedesktop.org/archives/systemd-devel/2015-February/028871.html That corresponds to this git commit: https://github.com/systemd/systemd/commit/628c89cc68ab96fce2de7ebba5933725d147aecc However, since this commit is in v220, v221, and v222 and F23 is using v222 I would say that this "fix" does not fix the problem we are observing. The only thing that has worked (I think) is the patches linked in comment #3. The systemd from Fedora 22 is still carrying 0014-unit-When-stopping-due-to-BindsTo-log-which-unit-cau.patch. The systemd from F23 has no such patch and thus we see the issue. Thoughts?
My suspicion is that commit fixes the bug for loop devices, but doesn't help ostree, which does a bind mount in the initramfs. It might work to change ostree to generate a systemd unit file for the mount, rather than just calling mount?
systemd-222-6.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-1e06faabb7
systemd-222-6.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with $ su -c 'dnf --enablerepo=updates-testing update systemd' You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-1e06faabb7
systemd-222-6.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
So we got this fixed for F23 by carrying a patch. What do we need to do in order to get this fixed upstream so we don't have to carry patches in the future?
(In reply to Dusty Mabe from comment #25) > So we got this fixed for F23 by carrying a patch. What do we need to do in > order to get this fixed upstream so we don't have to carry patches in the > future? File an Issue on the upstream github tracker. https://github.com/systemd/systemd/issues
https://github.com/systemd/systemd/issues/1556