Bug 851970

Summary: Unable to umount partition as long there are running services that have PrivateTmp set that where started after the partition mount
Product: [Fedora] Fedora Reporter: GV <rhel>
Component: systemdAssignee: Michal Schmidt <mschmidt>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 17CC: andrei.ilie, esandeen, frank, fsimonce, github, hgbug, johannbg, lnykryn, lpoetter, mads, metherid, mihai, mschmidt, msekleta, notting, plautrba, rhel, stefanrin, systemd-maint, vpavlin
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-05 06:53:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description GV 2012-08-27 07:33:15 UTC
Description of problem:
I'm unable to umount an xfs partition as long there are running services that have PrivateTmp set.

Version-Release number of selected component (if applicable):
all versions of systemd in F17 (I think F16 too).

How reproducible:
# mount | grep /saves
/dev/sdb1 on /saves type xfs (rw,seclabel,nosuid,nodev,noexec,relatime,attr2,noquota)
# umount /saves
# xfs_repair /dev/sdb1
xfs_repair: /dev/sdb1 contains a mounted filesystem

fatal error -- couldn't initialize XFS library

# service ntpd stop
# service mysqld stop
# service httpd stop
# xfs_repair /dev/sdb1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
....
Phase 7 - verify and correct link counts...
done

Actual results:
Partition is not really unmounted.

Expected results:
Partition should be unmounted.

Comment 1 GV 2012-08-27 07:39:07 UTC
Forgot to mention: partition has to be mounted when services are starting.

Comment 2 Stefan Ring 2012-08-27 17:09:52 UTC
It's not limited to XFS though. I can reproduce with ext4 as well. ext4 will say "mounted filesystem with ordered data mode" or something like that when it's doing a real mount, whereas it will keep quiet during a half mount (the case where the file system is still open, but not attached to the directory hierarchy). That's the way to detect this anomaly.

Comment 3 Bill Nottingham 2012-08-27 19:37:02 UTC
I suspect this is due to shared namespaces?

Comment 4 Stefan Ring 2012-08-28 07:28:22 UTC
A workable solution would be to remove all mounts from private namespaces which are not mentioned in /etc/fstab. Additionally, a special flag like _net could be added to fstab with the same meaning, i.e. "do not mount in private namespace".

Comment 5 Lennart Poettering 2012-09-14 13:34:47 UTC
I am not following here. Are you suggesting that mounts have been inherited from the host to the specific service, but the umounts are not inherited as well?

The mounts have happened before the service has been started, but the umounts are attempted while the service is still running?

My guess this is simply the effect of MS_PRIVATE vs. MS_SLAVE for the PrivateTmp namespace. In F18 we default to MS_SHARED for the host and MS_SLAVE for service namespaces and nspawn. THat should fix the issue.

Comment 6 Stefan Ring 2012-09-14 13:43:19 UTC
(In reply to comment #5)
> I am not following here. Are you suggesting that mounts have been inherited
> from the host to the specific service, but the umounts are not inherited as
> well?

I don't know how an umount can be inherited.

> The mounts have happened before the service has been started, but the
> umounts are attempted while the service is still running?

Yes, because I have started the service manually, and I need to umount the filesystem before initiating a shutdown.

Comment 7 Fedora Update System 2012-09-20 19:54:22 UTC
systemd-190-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/systemd-190-1.fc18

Comment 8 Stefan Ring 2012-09-21 07:22:22 UTC
I cannot reproduce it with F18 anyway, even with the previous systemd (188-3.fc18). It's easily reproducible on F17, though.

Comment 9 Fedora Update System 2012-09-22 06:35:35 UTC
Package systemd-191-2.fc18, rtkit-0.11-3.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-191-2.fc18 rtkit-0.11-3.fc18'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-14581/rtkit-0.11-3.fc18,systemd-191-2.fc18
then log in and leave karma (feedback).

Comment 10 Fedora Update System 2012-09-28 00:16:00 UTC
Package glibc-2.16-17.fc18, systemd-192-1.fc18, selinux-policy-3.11.1-23.fc18, rtkit-0.11-3.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing glibc-2.16-17.fc18 systemd-192-1.fc18 selinux-policy-3.11.1-23.fc18 rtkit-0.11-3.fc18'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-14581/selinux-policy-3.11.1-23.fc18,rtkit-0.11-3.fc18,systemd-192-1.fc18,glibc-2.16-17.fc18
then log in and leave karma (feedback).

Comment 11 Fedora Update System 2012-10-01 20:07:40 UTC
Package glibc-2.16-17.fc18, rtkit-0.11-3.fc18, systemd-193-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing glibc-2.16-17.fc18 rtkit-0.11-3.fc18 systemd-193-1.fc18'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-14581/rtkit-0.11-3.fc18,systemd-193-1.fc18,glibc-2.16-17.fc18
then log in and leave karma (feedback).

Comment 12 GV 2012-10-08 06:24:04 UTC
How exactly is this bug fixed in Fedora 17? Fedora 18 is still alpha. How exactly fixing systemd in Fedora 18 is this going to help us in Fedora 17?

Is there a new policy I'm not aware of so bugs in one Fedora release will be fixed in the next version?

Comment 13 Bill Nottingham 2012-10-16 12:25:32 UTC
They will get fixed, just not always at the same time, and since it requires backporting fixes (rather than copying the new version over wholesale) it might not be entirely trivial.

Michal - is there any reason why the fix mentioned in comment #5 can't be pulled back?

Comment 14 GV 2012-10-16 15:09:42 UTC
I'm sorry, but this bug has been closed with "CLOSED NEXTRELEASE". What do you expect me to say?

Comment 15 Michal Schmidt 2012-10-16 15:50:11 UTC
(In reply to comment #13)
> Michal - is there any reason why the fix mentioned in comment #5 can't be
> pulled back?

The reason is simple. I do not understand well enough how the filesystem namespace stuff works, so I am afraid of backporting the patches that touch it.

So in order to fix this bug in F17 and not break anything, which of these upstream patches do we need? And are there some others (e.g. for nspawn)?:

commit b3ac5f8cb98757416d8660023d6564a7c411f0a0
Author: Lennart Poettering <lennart>
Date:   Mon Aug 6 18:28:42 2012 +0200

    mount-setup: change system mount propagation to shared by default

commit 4bfa638d43c05e8db052cd55818765bb3575a405
Author: Dave Reisner <dreisner>
Date:   Fri Aug 10 11:02:03 2012 -0400

    shutdown: recursively mark root as private before pivot
    
commit f47fc35555565c4b161c2e44b357b4dbaf3a997d
Author: Lennart Poettering <lennart>
Date:   Sun Aug 12 01:29:41 2012 +0200

    switch-root: remount to MS_PRIVATE

commit ac0930c892bc7979b4c9bc2a52e5e844650b025d
Author: Lennart Poettering <lennart>
Date:   Mon Aug 13 15:27:04 2012 +0200

    namespace: rework namespace support
    
commit 1e41be20158a6d982c34cea20e66ff271302abc5
Author: Lennart Poettering <lennart>
Date:   Mon Aug 13 16:25:03 2012 +0200

    nspawn,namespaces: make sure we recursively bind mount things in
    
commit 8caf9d6836c3ed5b7bb4c1ea8dea5241a634c298
Author: Lennart Poettering <lennart>
Date:   Mon Aug 13 16:30:10 2012 +0200

    umount: MS_MGC_VAL is so 90s

Comment 16 Stefan Ring 2012-11-05 15:17:40 UTC
This is especially painful when the filesystem is mounted before logging into a desktop session via gdm, because apparently that causes some services to start automagically, in my case (with F17, again):

udevd
cupsd
udisks2
colord
colord-sane

After umounting the filesystem and logging out, I restarted all of them in this order, and only after restarting colord-sane (of all things) did the filesystem really get unmounted.

Comment 17 Andrei ILIE 2012-11-27 15:09:39 UTC
Confirming this bug against 64bit Fedora 17 (kernel-v3.6.7 / systemd-v44)...


# kpartx -a disk-image.dd

# mount /dev/mapper/loop0p2 /mnt/P2

    <work, work, work>

# umount /mnt/P2

# losetup --detach-all
losetup: /dev/loop0: detach failed: Device or resource busy

Comment 18 Eric Sandeen 2012-12-13 21:43:03 UTC
F17 is still broken.

I just burned several hours re-triaging this weirdness for bug #886030, which sucks, since the solution was known months ago.

Michal, if you can't fix it yourself, please pull in Lennart?  Just giving up is a poor solution for the current release of Fedora.  :/

Comment 19 Lennart Poettering 2012-12-19 14:41:29 UTC
I am a bit afraid of making MS_SHARED the default in F17 now simply because the implications in all details are not clear to me. We did this change early in F18 to see if this is an OK choice. So far most things ended up being OK, but it took us a while to get to that point (for example, MS_SHARED breaks pivot_root()). 

I figure we have two options now:

A) bite the bullet and make MS_SHARED the default in F17, too. (Michal's list of patches looks quite complete)

B) disable PrivateTmp= on F17.

Option A) looks a bit ncier to me, maybe we should do that after some prolonged  testing in bodhi?

Comment 20 Michal Schmidt 2012-12-19 14:44:26 UTC
Lennart, thanks for looking at the list of commits.
I'll make a build later tonight and put it to F17 updates-testing.

Comment 21 Lennart Poettering 2012-12-19 14:48:21 UTC
Reopening for F17.

Comment 22 Eric Sandeen 2012-12-19 16:33:08 UTC
Thanks guys!

-Eric

Comment 23 Fedora Update System 2012-12-19 22:08:47 UTC
systemd-44-23.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/systemd-44-23.fc17

Comment 24 Fedora Update System 2012-12-20 03:19:07 UTC
Package systemd-44-23.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-44-23.fc17'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-20703/systemd-44-23.fc17
then log in and leave karma (feedback).

Comment 25 GV 2012-12-20 12:04:13 UTC
umount works fine now. Finally!
Thank you.

Comment 26 Josh Boyer 2013-01-03 16:39:37 UTC
*** Bug 886030 has been marked as a duplicate of this bug. ***

Comment 27 Fedora Update System 2013-01-05 06:53:52 UTC
systemd-44-23.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.