Bug 1253527

(In reply to Colin Walters from comment #0)
> > rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem
> 
> I'm seeing this trying to use anaconda in Imagefactory to generate an Atomic
> Host cloud image.

so /run is mounted readonly??

(In reply to Harald Hoyer from comment #2)
> (In reply to Colin Walters from comment #0)
> > > rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem
> > 
> > I'm seeing this trying to use anaconda in Imagefactory to generate an Atomic
> > Host cloud image.
> 
> so /run is mounted readonly??

Presumably, but I haven't been able to get in at the right point to debug this yet.

This is still broken for me; I suspect it has something to do with:

* systemd tries to unmount everything and mounts the remaining read-only

(from dracut-shutdown-service.8.asc).

It might be systemd not understanding some of the ostree setup, such as the /sysroot mount.

One thing honestly that would make all of this simpler is to go to "read-only for all filesystem at shutdown", not "unmounted".  XFS for example has `xfs_freeze`.  Not sure offhand if ext4 or btrfs etc. have equivalents.

It looks like there's an existing rd.break=pre-shutdown, so I'll try that.

Created attachment 1074586 [details]
Screenshot

In rd.break=pre-shutdown

hmm, is this a side effect of the "read-only root support" ?

AFAIK, /run is not remounted ro by systemd. Otherwise, you would see these kind of things on every system.

Can you please screenshot the contents of 

# cat /proc/self/mountinfo

BTW anyone else can grab RHELAH-Continuous ISO installers here:

http://aos-ostree.rhev-ci-vms.eng.rdu2.redhat.com/rhel-atomic/7.2-continuous/images/installer/

The way I've been debugging it is to edit the kernel command line via TAB at the boot prompt to add rd.break=pre-shutdown, or just Alt-F2 in Anaconda before rebooting to see the state there.

I think I figured this out.  For some reason, tmpfs mounts all share state.  So if any tmpfs mount is remounted ro, *all* are.  And anaconda+ostree sets up mounts in /mnt/sysimage/ostree/deploy/$x/dev etc.

Cleaning these up before shutdown should fix it.  That'd be a change to rpmostreepayload, should be low risk.

Better long term would be changing Anaconda to use a mount namespace (containers) for %post.

Do you have a patch?

Created attachment 1074977 [details]
untested patch

While I don't see any obvious problems with the patch in comment #10, what I am curious about is:  Why is it suddenly our problem that the system can't reboot with something mounted RO?  Why isn't that the rest of the system's problem?  What happens if someone manually sets this same situation up post-install?

(In reply to Chris Lumens from comment #12)
> While I don't see any obvious problems with the patch in comment #10, what I
> am curious about is:  Why is it suddenly our problem that the system can't
> reboot with something mounted RO?  Why isn't that the rest of the system's
> problem?  What happens if someone manually sets this same situation up
> post-install?

This bug seems to be specific to the combination of anaconda + ostree, it doesn't happen on the installed system.  I think it has something to do with the way we're leaving mounts alive in the target root with rpmostreepayload, which is a payload-specific issue in Anaconda.

But to answer your question, were we're talking about the initramfs which is just an in-memory filesystem, there's no real reason to have it remounted read-only, same with /run - it's transient state.  Even "stateless" systems have /tmp mounted writable, everything would break if it wasn't.

Okay, on second glance I do see one problem:  We should be unmounting in the reverse order that we mount things.  That's just in case you mount /a and then /a/b.  Aside from that, if this patch works for you, I have no problems merging it in.

Upstream PR: https://github.com/rhinstaller/anaconda/pull/370

I tried a few things here but I couldn't really do this without touching generic Anaconda code, sorry.  However I think it's *very* low risk as we just have a new method with a default `pass`.

Should we clone this bug for Fedora to get a saner fix upstream?

We don't strictly need a bug for anaconda-master if we can talk about it on a mailing list, or in a github PR, or wherever.  But there's no harm in making a new bug either.

(In reply to Colin Walters from comment #0)
> > rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem
> 
> I'm seeing this trying to use anaconda in Imagefactory to generate an Atomic
> Host cloud image.

I'm also seeing this after installing from installation.iso and trying to shut down the system.

(In reply to Alexander Todorov from comment #19)
> (In reply to Colin Walters from comment #0)
> > > rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem
> > 
> > I'm seeing this trying to use anaconda in Imagefactory to generate an Atomic
> > Host cloud image.
> 
> I'm also seeing this after installing from installation.iso and trying to
> shut down the system.

With which installer?  Isn't this fixed now?

During comment #19 we had an installer.iso with an older anaconda. Moving to VERIFIED now.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2284.html