Bug 825869 - named-chroot systemd unit doesn't work
named-chroot systemd unit doesn't work
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: bind (Show other bugs)
17
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Tomas Hozza
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-28 16:25 EDT by Tom Hughes
Modified: 2012-09-22 23:28 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-09-22 23:28:17 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Tom Hughes 2012-05-28 16:25:31 EDT
Description of problem:

The named-chroot systemd service unit doesn't work as the bind mounts appear to go missing due to the PrivateTmp option.

I believe that systemd is proving a separate private tmp (and hence filesystem namespace) for each ExecStartPre script and then for the main process, and hence the bind mounts done by the ExecStartPre vanish when it finishes and the main process doesn't see them.

Changing PrivateTmp to false fixes the problem.

I guess this may need to be moved to systemd to consider whether this behaviour is correct, but in any case a private tmp is of limited use when we're going to be running chroot anyway, with a different tmp as a result.

Version-Release number of selected component (if applicable):

bind-chroot-9.9.0-4.fc17

How reproducible:

Every time.

Steps to Reproduce:
1. Start named-chroot.service
2. Observe that it has used a default configuration
3. Observe that /var/named/chroot/etc/named.conf is empty
  
Actual results:

It doesn't work.

Expected results:

It works.
Comment 1 Sam Varshavchik 2012-06-09 12:17:37 EDT
Seeing the same bug here.

[root@shorty etc]# pwd
/var/named/chroot/etc
[root@shorty etc]# ls -al
total 24
drwxr-x---. 5 root named 4096 Jun  9 12:05 .
drwxr-x---. 7 root named 4096 Jun  9 12:05 ..
-rw-r--r--  1 root root  3519 Jun  9 12:05 localtime
drwxr-x---  2 root named 4096 May 24 08:59 named
drwxr-x---. 4 root named 4096 Jun  9 12:05 pki
-rw-r--r--  1 root root     0 Jun  9 12:02 rndc.key
drwxr-xr-x  6 root root  4096 Jun  3 08:35 .svn
[root@shorty etc]# systemctl start named-chroot.service
[root@shorty etc]# ls -al
total 24
drwxr-x---. 5 root named 4096 Jun  9 12:06 .
drwxr-x---. 7 root named 4096 Jun  9 12:05 ..
-rw-r--r--  1 root root  3519 Jun  9 12:05 localtime
drwxr-x---  2 root named 4096 May 24 08:59 named
-rw-r--r--  1 root root     0 Jun  9 12:06 named.conf
-rw-r--r--  1 root root     0 Jun  9 12:06 named.iscdlv.key
-rw-r--r--  1 root root     0 Jun  9 12:06 named.rfc1912.zones
-rw-r--r--  1 root root     0 Jun  9 12:06 named.root.key
drwxr-x---. 4 root named 4096 Jun  9 12:05 pki
-rw-r--r--  1 root root     0 Jun  9 12:06 rndc.key
drwxr-xr-x  6 root root  4096 Jun  3 08:35 .svn

Boom.

After turning off PrivateTmp, everything gets loopback-mounted properly:

[root@shorty etc]# ls -al
total 44
drwxr-x---. 5 root named 4096 Jun  9 12:14 .
drwxr-x---. 7 root named 4096 Jun  9 12:05 ..
-rw-r--r--  1 root root  3519 Jun  9 12:05 localtime
drwxr-x---. 2 root named 4096 May 24 08:59 named
-rw-r--r--  1 root root  2802 Mar 14 23:02 named.conf
-rw-r--r--  1 root named 2389 May 24 08:59 named.iscdlv.key
-rw-r-----  1 root named  931 Jun 21  2007 named.rfc1912.zones
-rw-r--r--  1 root named  487 Jul 19  2010 named.root.key
drwxr-x---. 4 root named 4096 Jun  9 12:05 pki
-rw-r-----  1 root named  132 Sep  5  2011 rndc.key
drwxr-xr-x  6 root root  4096 Jun  3 08:35 .svn
Comment 2 Scott Shambarger 2012-07-02 16:13:30 EDT
Still a problem in 9.9.1-2.P1.

Could this be a container issue affected by PrivateTmp
Comment 3 Scott Shambarger 2012-07-02 18:08:41 EDT
OK, I tracked down this problem in systemd... 

1) core/service.c:service_spawn() is called for ExecStartPre, which calls
2) core/execute.c:exec_spawn() and if there's a r/w, r/o, inaccessable or PrivateTmp options, it calls:
3) core/namespace.c:setup_namespace(), which performs mount(...MS_SLAVE), whose comment says "so that nothing mounted in the namespace shows up in the parent"

If then creates the /tmp/systemd-namespace-xxxx directories, and pivot_root's into it.  When the command in ExecStartPre is called ("/usr/libexec/setup-named-chroot.sh /var/named/chroot on" in this case), it's in a nice chroot jail where mounts can't be propagated to the parent, and are in /tmp/systemd-namespace... anyway.

This wouldn't be a deal killer, except... when it's time to spawn /usr/sbin/named, the whole process starts again at (1) core/service.c:service_spawn() -- and a brand new temp directory is created and pivot_root'ed to.  The old mounts aren't there (they, I'm guessing, became trapped in setup-named-chroot.sh own jail... and never unmounted -- wonder if these expire with the slave mount, or if they are trapped when the sun don't shine forever?)

I'm guessing the correct behavior for systemd would be for it to create the /tmp/systemd-namespace* once for the service instance, and retain it between ExecStartPre, ExecStart, and ExecStartPost.

However... there's an easy, valid fix here.  Since named is performing it's own chroot post-exec, there's really no reason for PrivateTmp to ever be true here, as it really serves no purpose other than to expose the limitation in systemd.

I suggest the following, simple patch:

--- named-chroot.service    2012-06-04 07:09:31.000000000 -0700
+++ named-chroot.service    2012-07-02 13:07:09.999489687 -0700
@@ -23,7 +23,7 @@ ExecReload=/bin/sh -c '/usr/sbin/rndc re
 ExecStop=/bin/sh -c '/usr/sbin/rndc stop > /dev/null 2>&1 || /bin/kill -TERM $MAINPID'
 ExecStopPost=/usr/libexec/setup-named-chroot.sh /var/named/chroot off
 
-PrivateTmp=true
+PrivateTmp=false
 TimeoutSec=25
 
 [Install]
Comment 4 Scott Shambarger 2012-07-19 14:29:41 EDT
I'm a little surprised more people haven't been clamoring for this to be fixed.... I guess chroot+selinux is a uncommon combination.

I also noticed that whenever I reboot, the get the following logged:

umount: /var/named/chroot/var/named: target is busy

I've run lsof and fuser on it when this is the case, and nothing is using it, so I'm guessing there are some mounts trapped in a now inaccessible container.

Has anyone tracked this problem down?

(BTW, this happens even if I set PrivateTmp=false)
Comment 5 Tomas Hozza 2012-08-08 07:54:14 EDT
Hi Scott.

I fixed problem with chroot related to PrivateTmp. You can find packages including fix here http://koji.fedoraproject.org/koji/buildinfo?buildID=346630 or you can wait for next bind update.

However I'm not able to reproduce your problem with umount. If you are still having problems with umount, please fill a new bug report for it.
Comment 6 Scott Shambarger 2012-08-09 18:07:52 EDT
Sounds great, I look forward to the update appearing on testing so I provide karma :)

Re umount issue, I believe that was related to a kernel bug fixed in 3.5: http://www.spinics.net/lists/linux-fsdevel/msg55572.html

I'm not seeing it atm, so it's possible that's fixed too.
Comment 7 Sam Varshavchik 2012-08-20 08:38:11 EDT
This bug still exists in bind-chroot-9.9.1-5.P2.fc17.x86_64, but results in different breakage.

With bind-chroot-9.9.1-5.P2.fc17.x86_64, bind works only as a recursive resolver. bind does not see /var/named/chroot/var/named/named.conf, or anything in /var/named/chroot/var/named, for that matter, and fails to load any zones, or apply any settings in named.conf.

Again, removing PrivateTmp from named-chroot.service fixes it. PrivateTmp is still broken.
Comment 8 Tom Hughes 2012-08-20 08:41:31 EDT
The fix is in bind-9.9.1-8.P2.fc17 so of course 9.9.1-5.P2 still fails!

Incidentally the fix is to remove PrivateTmp from named-chroot.service, or rather to change it to false.
Comment 9 Fedora Update System 2012-09-13 11:49:36 EDT
bind-9.9.1-9.P3.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/bind-9.9.1-9.P3.fc17
Comment 10 Fedora Update System 2012-09-17 13:39:16 EDT
Package bind-9.9.1-9.P3.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing bind-9.9.1-9.P3.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-14106/bind-9.9.1-9.P3.fc17
then log in and leave karma (feedback).
Comment 11 Fedora Update System 2012-09-22 23:28:17 EDT
bind-9.9.1-9.P3.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.