Bug 1253527 - ostree+anaconda: unable to shutdown - dracut loop rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem
ostree+anaconda: unable to shutdown - dracut loop rm: cannot remove /lib/drau...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: anaconda (Show other bugs)
7.2
Unspecified Unspecified
high Severity high
: rc
: ---
Assigned To: Colin Walters
Release Test Team
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-13 22:25 EDT by Colin Walters
Modified: 2015-11-19 05:03 EST (History)
12 users (show)

See Also:
Fixed In Version: anaconda-21.48.22.50-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1178497
: 1274977 (view as bug list)
Environment:
Last Closed: 2015-11-19 05:03:07 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Screenshot (65.61 KB, image/png)
2015-09-17 17:37 EDT, Colin Walters
no flags Details
untested patch (3.56 KB, patch)
2015-09-18 11:23 EDT, Colin Walters
no flags Details | Diff

  None (edit)
Description Colin Walters 2015-08-13 22:25:43 EDT
> rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem

I'm seeing this trying to use anaconda in Imagefactory to generate an Atomic Host cloud image.
Comment 2 Harald Hoyer 2015-08-19 10:09:36 EDT
(In reply to Colin Walters from comment #0)
> > rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem
> 
> I'm seeing this trying to use anaconda in Imagefactory to generate an Atomic
> Host cloud image.

so /run is mounted readonly??
Comment 3 Colin Walters 2015-08-21 14:20:37 EDT
(In reply to Harald Hoyer from comment #2)
> (In reply to Colin Walters from comment #0)
> > > rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem
> > 
> > I'm seeing this trying to use anaconda in Imagefactory to generate an Atomic
> > Host cloud image.
> 
> so /run is mounted readonly??

Presumably, but I haven't been able to get in at the right point to debug this yet.
Comment 4 Colin Walters 2015-09-17 12:03:30 EDT
This is still broken for me; I suspect it has something to do with:

* systemd tries to unmount everything and mounts the remaining read-only

(from dracut-shutdown-service.8.asc).

It might be systemd not understanding some of the ostree setup, such as the /sysroot mount.

One thing honestly that would make all of this simpler is to go to "read-only for all filesystem at shutdown", not "unmounted".  XFS for example has `xfs_freeze`.  Not sure offhand if ext4 or btrfs etc. have equivalents.

It looks like there's an existing rd.break=pre-shutdown, so I'll try that.
Comment 5 Colin Walters 2015-09-17 17:37:18 EDT
Created attachment 1074586 [details]
Screenshot

In rd.break=pre-shutdown
Comment 6 Harald Hoyer 2015-09-18 06:11:41 EDT
hmm, is this a side effect of the "read-only root support" ?

AFAIK, /run is not remounted ro by systemd. Otherwise, you would see these kind of things on every system.

Can you please screenshot the contents of 

# cat /proc/self/mountinfo
Comment 7 Colin Walters 2015-09-18 08:55:50 EDT
BTW anyone else can grab RHELAH-Continuous ISO installers here:

http://aos-ostree.rhev-ci-vms.eng.rdu2.redhat.com/rhel-atomic/7.2-continuous/images/installer/

The way I've been debugging it is to edit the kernel command line via TAB at the boot prompt to add rd.break=pre-shutdown, or just Alt-F2 in Anaconda before rebooting to see the state there.
Comment 8 Colin Walters 2015-09-18 09:47:24 EDT
I think I figured this out.  For some reason, tmpfs mounts all share state.  So if any tmpfs mount is remounted ro, *all* are.  And anaconda+ostree sets up mounts in /mnt/sysimage/ostree/deploy/$x/dev etc.

Cleaning these up before shutdown should fix it.  That'd be a change to rpmostreepayload, should be low risk.

Better long term would be changing Anaconda to use a mount namespace (containers) for %post.
Comment 9 Brian Lane 2015-09-18 11:17:00 EDT
Do you have a patch?
Comment 10 Colin Walters 2015-09-18 11:23:35 EDT
Created attachment 1074977 [details]
untested patch
Comment 12 Chris Lumens 2015-09-21 15:20:58 EDT
While I don't see any obvious problems with the patch in comment #10, what I am curious about is:  Why is it suddenly our problem that the system can't reboot with something mounted RO?  Why isn't that the rest of the system's problem?  What happens if someone manually sets this same situation up post-install?
Comment 13 Colin Walters 2015-09-21 17:25:53 EDT
(In reply to Chris Lumens from comment #12)
> While I don't see any obvious problems with the patch in comment #10, what I
> am curious about is:  Why is it suddenly our problem that the system can't
> reboot with something mounted RO?  Why isn't that the rest of the system's
> problem?  What happens if someone manually sets this same situation up
> post-install?

This bug seems to be specific to the combination of anaconda + ostree, it doesn't happen on the installed system.  I think it has something to do with the way we're leaving mounts alive in the target root with rpmostreepayload, which is a payload-specific issue in Anaconda.

But to answer your question, were we're talking about the initramfs which is just an in-memory filesystem, there's no real reason to have it remounted read-only, same with /run - it's transient state.  Even "stateless" systems have /tmp mounted writable, everything would break if it wasn't.
Comment 14 Chris Lumens 2015-09-22 10:11:23 EDT
Okay, on second glance I do see one problem:  We should be unmounting in the reverse order that we mount things.  That's just in case you mount /a and then /a/b.  Aside from that, if this patch works for you, I have no problems merging it in.
Comment 15 Colin Walters 2015-09-22 15:54:11 EDT
Upstream PR: https://github.com/rhinstaller/anaconda/pull/370

I tried a few things here but I couldn't really do this without touching generic Anaconda code, sorry.  However I think it's *very* low risk as we just have a new method with a default `pass`.
Comment 16 Colin Walters 2015-09-22 15:55:40 EDT
Should we clone this bug for Fedora to get a saner fix upstream?
Comment 17 Chris Lumens 2015-09-23 10:10:10 EDT
We don't strictly need a bug for anaconda-master if we can talk about it on a mailing list, or in a github PR, or wherever.  But there's no harm in making a new bug either.
Comment 19 Alexander Todorov 2015-09-28 04:24:54 EDT
(In reply to Colin Walters from comment #0)
> > rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem
> 
> I'm seeing this trying to use anaconda in Imagefactory to generate an Atomic
> Host cloud image.

I'm also seeing this after installing from installation.iso and trying to shut down the system.
Comment 20 Colin Walters 2015-10-02 13:32:37 EDT
(In reply to Alexander Todorov from comment #19)
> (In reply to Colin Walters from comment #0)
> > > rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem
> > 
> > I'm seeing this trying to use anaconda in Imagefactory to generate an Atomic
> > Host cloud image.
> 
> I'm also seeing this after installing from installation.iso and trying to
> shut down the system.

With which installer?  Isn't this fixed now?
Comment 21 Alexander Todorov 2015-10-05 04:42:10 EDT
During comment #19 we had an installer.iso with an older anaconda. Moving to VERIFIED now.
Comment 22 errata-xmlrpc 2015-11-19 05:03:07 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2284.html

Note You need to log in before you can comment on or make changes to this bug.