Bug 1195761

Summary: ostree-prepare-root broken with 219
Product: [Fedora] Fedora Reporter: Colin Walters <walters>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 23CC: admiller, awilliam, dustymabe, harald, johannbg, jsynacek, lnykryn, msekleta, s, systemd-maint, vpavlin, walters, zbyszek
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: systemd-219-6.fc22 systemd-222-6.fc23 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-03 17:40:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1194590    

Description Colin Walters 2015-02-24 14:09:34 UTC
See:

http://lists.freedesktop.org/archives/systemd-devel/2015-February/028482.html

This is a pretty recent change; I'd been putting off debugging because only gnome-contnuous was broken, but now that Fedora 22 Atomic Host is, filing here.

I'll try to get a more precise regression range or bisect.  In the meantime, any hints for debugging appreciated.

Comment 1 Colin Walters 2015-02-24 15:16:35 UTC
I'm strongly suspecting this is http://cgit.freedesktop.org/systemd/systemd/commit/?id=06e97888883e2cc12eb6514e80c7f0014295f59b

Testing now.  If that's the case, there seem to be two questions here:
 - Why would the device be going inactive?
 - Regardless, we probably don't want these mounts to be bound to the root device, they should last until shutdown

Comment 2 Colin Walters 2015-02-24 20:03:44 UTC
So the answer here is that systemd-remount-rootfs.service is somehow causing the device to become inactive when remounting / read-write.  If I boot with "rw" things work fine.

I'll follow up on the list, but it'd be good if we could carry a revert of that patch until it's debugged.  Any objections to that?

Comment 4 Adam Williamson 2015-02-25 22:19:47 UTC
I suspect this is also behind https://bugzilla.redhat.com/show_bug.cgi?id=1195899 . I'm going to test that theory out now.

Comment 5 Fedora Update System 2015-02-25 23:35:06 UTC
systemd-219-5.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/systemd-219-5.fc22

Comment 6 Fedora Update System 2015-02-26 17:42:08 UTC
Package systemd-219-5.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-219-5.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-2655/systemd-219-5.fc22
then log in and leave karma (feedback).

Comment 7 Fedora Update System 2015-03-04 01:55:32 UTC
systemd-219-6.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/systemd-219-6.fc22

Comment 8 Fedora Update System 2015-03-05 01:13:42 UTC
systemd-219-5.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 9 Fedora Update System 2015-03-09 08:27:43 UTC
systemd-219-6.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 10 Colin Walters 2015-06-02 15:44:01 UTC
Note lots of discussion upstream continues on this: http://lists.freedesktop.org/archives/systemd-devel/2015-May/032030.html

Comment 11 Colin Walters 2015-09-18 17:04:49 UTC
Reaffecting Fedora 23.

Comment 12 Adam Miller 2015-09-21 20:02:21 UTC
I re-ported the patches from walters and applied them to 222:

https://maxamillion.fedorapeople.org/systemd-222-5.fc23.src.rpm

In initial testing this appears to fix the issue.

Comment 13 Adam Miller 2015-09-21 20:17:04 UTC
I went ahead an put it in F23 DistGit and got a build in koji 

http://koji.fedoraproject.org/koji/taskinfo?taskID=11169150

If I can get a +1 I'll send an update to bodhi.

Comment 14 Dusty Mabe 2015-09-22 13:05:26 UTC
We have an atomic host image with this systemd. It seems to boot fine, thus not observing the problem reported in this bug. The image is at:

https://dustymabe.fedorapeople.org/Fedora-Cloud-Atomic-23-specialsauce.qcow2.xz

Comment 15 Harald Hoyer 2015-09-22 20:00:25 UTC
Is there an upstream issue filed on github, or a mailing list discussion with an outcome?
Can we reproduce the issue without atomic?

Comment 16 Dusty Mabe 2015-09-22 20:16:34 UTC
One of the mail threads referenced early on has someone that says they can reproduce (because of recreating a similar situation) on non-atomic:

http://lists.freedesktop.org/archives/systemd-devel/2015-February/028810.html

I'm going to flag Colin as needinfo on this though because he has the most context.

Comment 17 Colin Walters 2015-09-24 16:58:10 UTC
AFAIK it's already fixed in upstream master.

Comment 18 Colin Walters 2015-09-24 17:01:31 UTC
Nevermind comment #17 - I tried reverting the patch in:

https://git.gnome.org/browse/gnome-continuous/commit/manifest.json?id=a3a88b427367ca0f5054f1847efcc1a3e15cb1b7

But it did break, and Vadim reverted:

https://git.gnome.org/browse/gnome-continuous/commit/manifest.json?id=a6789105bf6a691b77ef631dee31a35ed5a7d52c

(I find gnome-continuous convenient to test the combination of systemd and ostree because it constantly builds, ships, and tests git master of both)

Comment 19 Dusty Mabe 2015-09-24 17:03:16 UTC
(In reply to Colin Walters from comment #18)
> Nevermind comment #17 - I tried reverting the patch in:
> 
> https://git.gnome.org/browse/gnome-continuous/commit/manifest.
> json?id=a3a88b427367ca0f5054f1847efcc1a3e15cb1b7
> 
> But it did break, and Vadim reverted:
> 
> https://git.gnome.org/browse/gnome-continuous/commit/manifest.
> json?id=a6789105bf6a691b77ef631dee31a35ed5a7d52c
> 
> (I find gnome-continuous convenient to test the combination of systemd and
> ostree because it constantly builds, ships, and tests git master of both)

So it's not fixed in upstream master?

Comment 20 Dusty Mabe 2015-09-24 19:01:04 UTC
Harald,

Here is one outcome of a mailing list discussion:

http://lists.freedesktop.org/archives/systemd-devel/2015-February/028871.html

That corresponds to this git commit:

https://github.com/systemd/systemd/commit/628c89cc68ab96fce2de7ebba5933725d147aecc


However, since this commit is in v220, v221, and v222 and F23 is using v222 I would say that this "fix" does not fix the problem we are observing. The only thing that has worked (I think) is the patches linked in comment #3. The systemd from Fedora 22 is still carrying 0014-unit-When-stopping-due-to-BindsTo-log-which-unit-cau.patch. The systemd from F23 has no such patch and thus we see the issue.

Thoughts?

Comment 21 Colin Walters 2015-09-24 19:09:45 UTC
My suspicion is that commit fixes the bug for loop devices, but doesn't help ostree, which does a bind mount in the initramfs.

It might work to change ostree to generate a systemd unit file for the mount, rather than just calling mount?

Comment 22 Fedora Update System 2015-09-25 14:10:21 UTC
systemd-222-6.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-1e06faabb7

Comment 23 Fedora Update System 2015-09-27 00:55:08 UTC
systemd-222-6.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update systemd'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-1e06faabb7

Comment 24 Fedora Update System 2015-10-03 17:39:23 UTC
systemd-222-6.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 25 Dusty Mabe 2015-10-08 13:16:10 UTC
So we got this fixed for F23 by carrying a patch. What do we need to do in order to get this fixed upstream so we don't have to carry patches in the future?

Comment 26 Harald Hoyer 2015-10-13 09:48:52 UTC
(In reply to Dusty Mabe from comment #25)
> So we got this fixed for F23 by carrying a patch. What do we need to do in
> order to get this fixed upstream so we don't have to carry patches in the
> future?

File an Issue on the upstream github tracker.

https://github.com/systemd/systemd/issues

Comment 27 Colin Walters 2015-10-13 13:24:24 UTC
https://github.com/systemd/systemd/issues/1556