Bug 504596

Summary: dnssec files are not present in chroot
Product: [Fedora] Fedora Reporter: Jan ONDREJ <ondrejj>
Component: bindAssignee: Adam Tkac <atkac>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 11CC: atkac, braden, dyoung, k.georgiou, matt.castelein, ovasik, paul, pwouters, raytodd
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: 0.1.0-0.2.a1.fc11 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 523682 (view as bug list) Environment:
Last Closed: 2009-07-03 19:42:03 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jan ONDREJ 2009-06-08 12:53:58 UTC
Description of problem:
After update from F10 to F11, dnssec is automatically configured. When trying to start named, error is printed, because dnssec files have not been present in chroot. Please update bind-chroot scripts to update chroot by copying dnssec files.

Version-Release number of selected component (if applicable):
bind-9.6.1-0.4.rc1.fc11.i586
bind-chroot-9.6.1-0.4.rc1.fc11.i586


How reproducible:
always

Steps to Reproduce:
1. update bind
2. service bind restart

  
Actual results:
[root@work etc]# /etc/init.d/named restart
Stopping named:                                            [  OK  ]
Starting named: 
Error in named configuration:
/etc/named.conf:45: open: /etc/named.dnssec.keys: file not found
                                                           [FAILED]


Expected results:
[root@work etc]# /etc/init.d/named restart
Stopping named:                                            [  OK  ]
Starting named:                                            [  OK  ]


Additional info:
You should copy /etc/named.dnssec.keys and also all files under /etc/pki/dnssec-keys directory.

Comment 1 Adam Tkac 2009-06-10 10:28:09 UTC
bind-chroot scripts (especially bind-chroot-admin script) no longer exists because chroot is deprecated in favor of SELinux. The best solution will be not to enable DNSSEC automatically on configurations with chroot environment, probably.

Comment 2 Adam Tkac 2009-06-10 14:26:12 UTC
*** Bug 505052 has been marked as a duplicate of this bug. ***

Comment 3 Fedora Update System 2009-06-18 17:47:48 UTC
dnsperf-1.0.1.0-9.fc11,bind-9.6.1-1.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/dnsperf-1.0.1.0-9.fc11,bind-9.6.1-1.fc11

Comment 4 Ray Todd Stevens 2009-06-18 20:11:47 UTC
This needs to have a higher priority.   

This not only prevents the dnssec stuff from working, it also prevents the entire bind system from working.   Basically this error means that when the system is upgraded you are left with a dead box if the box contains a chroot name server.

This can take an entire network down if you update your name servers.

Comment 5 Ray Todd Stevens 2009-06-18 20:15:23 UTC
Also as far as using selinux, I have yet to find a server that it will actually run on when the server is running in production.   To many applications still don't support it.

Did spend two days on one server recently trying to relabel things to make it work.

Comment 6 Paul Howarth 2009-06-18 20:24:48 UTC
I run SELinux enforcing on all of my production servers and have little trouble with it these days.

Anyway, regarding the chroot, here's what I did for my F-11 boxes (fresh installs, data backed up and restored):


Install BIND
------------
# yum install bind-chroot

Basic config
------------

* Add ROOTDIR=/var/named/chroot to /etc/sysconfig/named

* Restore /var/named/chroot/etc/named.conf (merge new things e.g. DNSSEC from /etc/named.conf)

* Restore /var/named/chroot/etc/rndc.{key,conf}

* Restore /var/named/chroot/var/named/data/*

Move DNSSEC keys into chroot
----------------------------

# cd /etc/pki
# mkdir -p /var/named/chroot/etc/pki
# mv dnssec-keys ../../../var/named/chroot/etc/pki/
# ln -s ../../../var/named/chroot/etc/pki/dnssec-keys .

Move standard configs into chroot
---------------------------------

# cd /etc
# mv named.conf named.conf.orig
# rm rndc.key rndc.conf
# mv named.dnssec.keys named.rfc1912.zones /var/named/chroot/etc/
# ln -s ../var/named/chroot/etc/named.conf .
# ln -s ../var/named/chroot/etc/named.dnssec.keys .
# ln -s ../var/named/chroot/etc/named.rfc1912.zones .
# ln -s ../var/named/chroot/etc/rndc.* .
# cd /var/named
# mv named.ca named.empty named.localhost named.loopback chroot/var/named/
# ln -s chroot/var/named/named.ca .
# ln -s chroot/var/named/named.empty .
# ln -s chroot/var/named/named.localhost .
# ln -s chroot/var/named/named.loopback .

Start the server
----------------

# chkconfig named on
# service named start

Works for me, with SELinux too :-)

Comment 7 Ray Todd Stevens 2009-06-18 20:25:20 UTC
It seems like this new setup is dumping additional files in etc, when they should be dropped into /etc/named

Comment 8 Ray Todd Stevens 2009-06-18 20:42:34 UTC
I kind of expected to be able to run it in enforcing mode by now, but so far that has not happened.

I do run selinux in permissive mode on most of my servers, and have been trying to get to no messages in the log.   I do have two servers I can't even do that in that they will generate so many audit messages so fast it messes with the server.

Part of the problem is that it doesn't seem to understand apache accessing files in the home/user/public_html directory.

It also seems positive that my database server either should have no access to files, or should not be accessed by the apache server.

We also run a separate volume off of the root with rather large web sites on the server, and it appears positive that /www should not be accessed by apache, and even if you mark these to be accessed by apache, it seems to magically mark them back.

Our backup system is an even bigger problem, in that it is positive that the backup system should not access the drive, and that it certainly should not be permitted to do restores.

Comment 9 Adam Tkac 2009-06-19 09:32:22 UTC
First, the best approach how to deal with httpd + SELinux problems is to read documentation and if you think you have everything configured correctly then open a separate bug report against httpd or selinux-policy component, please.

After little discussion about this problem I decided to use approach proposed by original reporter.

Crucial files which are needed to run named in chroot environment are mounted (mount --bind) to chroot automatically during named startup. It should simplify maintenance of chroot. Details are written in updated /etc/sysconfig/named. Would it be possible to test updated packages, please? They are currently in -testing repository (comment #3). Thanks.

Comment 10 Paul Howarth 2009-06-19 11:56:38 UTC
(In reply to comment #8)
> I kind of expected to be able to run it in enforcing mode by now, but so far
> that has not happened.
> 
> I do run selinux in permissive mode on most of my servers, and have been trying
> to get to no messages in the log.   I do have two servers I can't even do that
> in that they will generate so many audit messages so fast it messes with the
> server.
> 
> Part of the problem is that it doesn't seem to understand apache accessing
> files in the home/user/public_html directory.

Should be fixed if you do:
# setsebool -P httpd_enable_homedirs=1
and fix the contexts of home/user/public_html
(see "man httpd_selinux")

> It also seems positive that my database server either should have no access to
> files, or should not be accessed by the apache server.

Should be helped by:
# setsebool -P httpd_can_network_connect_db=1

> We also run a separate volume off of the root with rather large web sites on
> the server, and it appears positive that /www should not be accessed by apache,
> and even if you mark these to be accessed by apache, it seems to magically mark
> them back.

Fedora 11 has new support for "File Context Equivalency", which can help with this - see http://danwalsh.livejournal.com/27571.html

> Our backup system is an even bigger problem, in that it is positive that the
> backup system should not access the drive, and that it certainly should not be
> permitted to do restores.  

Hmm, strange. I'd expect backup systems to run unconfined by SELinux and haven't seen any issues like that with either bacula or EMC Networker, both of which I use regularly.

I'd suggest posting individual issues in separate posts to fedora-selinux-list and whittling them down one at a time - I'm sure you can get there in the end.

Comment 11 Paul Howarth 2009-06-19 11:58:54 UTC
(In reply to comment #9)
> Crucial files which are needed to run named in chroot environment are mounted
> (mount --bind) to chroot automatically during named startup. It should simplify
> maintenance of chroot. Details are written in updated /etc/sysconfig/named.
> Would it be possible to test updated packages, please? They are currently in
> -testing repository (comment #3). Thanks.  

I'll give this a try when it's pushed to -testing (probably later today).

Comment 12 Fedora Update System 2009-06-19 13:43:40 UTC
dnsperf-1.0.1.0-9.fc11, bind-9.6.1-1.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update dnsperf bind'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-6766

Comment 13 Paul Howarth 2009-06-19 15:44:04 UTC
The updated package isn't quite right here. In order for it to bind-mount /var/named into the chroot, it requires $ROOTDIR/var/named to be empty. However, as we would normally expect $ROOTDIR to be /var/named/chroot if we're using the bind-chroot package, it won't work because that package includes the directories /var/named/chroot/var/named/{data,dynamic,slaves} and the presence of those directories inhibits the bind-mounting of /var/named.

Easiest fix is probably just to not include those 3 directories in the bind-chroot package any more. Having removed them myself manually, it all sprang into life OK.

Comment 14 Jan ONDREJ 2009-06-24 07:41:03 UTC
(In reply to comment #13)
> Easiest fix is probably just to not include those 3 directories in the
> bind-chroot package any more. Having removed them myself manually, it all
> sprang into life OK.  

Yes, same problem for me, but each machine administrator can this fix once and then it will work well. I don't know about a safe way, how to deal with this.

I think this package should go into stable. :)

Comment 15 Paul Howarth 2009-06-24 08:00:16 UTC
It'll work well until the next bind update that's applied without the bind mounts in place, and then the three problem directories will be back and bind won't work when it's next started.

Comment 16 Adam Tkac 2009-06-24 11:31:48 UTC
(In reply to comment #15)
> It'll work well until the next bind update that's applied without the bind
> mounts in place, and then the three problem directories will be back and bind
> won't work when it's next started.  

Those files (chroot/var/named/{data,dynamic,slaves}) are currently marked as "%ghost" in specfile. It means if they are exist then they are preserved and if they don't exist they are not created during package update.

But you are right they should be completely excluded from bind-chroot. It will effectivelly mean they will be erased during update if they are empty.

Comment 17 Matt Castelein 2009-06-24 17:55:28 UTC
(In reply to comment #12)
>  If you want to test the update, you can install it with 
>  su -c 'yum --enablerepo=updates-testing update dnsperf bind'.

I would do that, but since bind isn't working because of the bug, yum isn't working either!

Comment 18 Matt Castelein 2009-06-24 18:15:07 UTC
I copied what I *thought* were the relevant files into the chroot, but now yum isn't working anymore with:

Error: Cannot retrieve repository metadata (repomd.xml) for repository: fedora. Please verify its path and try again
 
What happened!?

Comment 19 Matt Castelein 2009-06-24 18:39:48 UTC
# yum check-update
Loaded plugins: protectbase
Error: Cannot retrieve repository metadata (repomd.xml) for repository: fedora. Please verify its path and try again
# yum --disablerepo=fedora check-update
Loaded plugins: protectbase
Could not retrieve mirrorlist http://mirrors.rpmfusion.org/mirrorlist?repo=free-fedora-11&arch=x86_64 error was
[Errno 4] IOError: <urlopen error [Errno -2] Name or service not known>
Error: Cannot retrieve repository metadata (repomd.xml) for repository: rpmfusion-free. Please verify its path and try again

What did I break?

Comment 20 Matt Castelein 2009-06-24 18:59:56 UTC
This is still a bind problem.  If I do nslookup mirrors.fedoraproject.org, I
will get:

Server:         127.0.0.1
Address:        127.0.0.1#53

** server can't find mirrors.fedoraproject.org: NXDOMAIN

If I do a dig, I am getting either the correct answer, or no servers could be
reached, or "bus error"

I think bind is still broken...

Comment 21 Matt Castelein 2009-06-24 19:11:28 UTC
Changing dnssec-enable yes; and dnssec-validation yes; both to "no" in named.conf has got my system working for the time being.  What a huge PITA.

Comment 22 Fedora Update System 2009-06-27 02:57:27 UTC
bind-dyndb-ldap-0.1.0-0.2.a1.fc11, dnsperf-1.0.1.0-10.fc11, bind-9.6.1-2.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update bind-dyndb-ldap dnsperf bind'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-6766

Comment 23 Fedora Update System 2009-07-03 19:41:52 UTC
bind-dyndb-ldap-0.1.0-0.2.a1.fc11, dnsperf-1.0.1.0-10.fc11, bind-9.6.1-2.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 24 Kostas Georgiou 2009-07-26 18:07:22 UTC
Looking at /etc/init.d/named (bind-9.6.1-3) there is an error at:

# mount it only if it is not present in chroot or it is empty
if ! [ -e "$ROOTDIR$all" ] ||
       [ `ls -s "$ROOTDIR$all" | awk '{ print $1 };'` -eq 0 ]; then
....

ls -s does NOT return zero for an empty file so empty files never get
bind mounted in the chroot, something like the following should do the
trick  

# mount it only if it is not present in chroot or it is empty
if ! [ -e "$ROOTDIR$all" ] ||
           [ `stat -c"%s" "$ROOTDIR$all"` -eq 0 ]; then
.....

Comment 25 Adam Tkac 2009-09-15 15:18:12 UTC
(In reply to comment #24)
> Looking at /etc/init.d/named (bind-9.6.1-3) there is an error at:
> 
> # mount it only if it is not present in chroot or it is empty
> if ! [ -e "$ROOTDIR$all" ] ||
>        [ `ls -s "$ROOTDIR$all" | awk '{ print $1 };'` -eq 0 ]; then
> ....
> 
> ls -s does NOT return zero for an empty file so empty files never get
> bind mounted in the chroot, something like the following should do the
> trick  
> 
> # mount it only if it is not present in chroot or it is empty
> if ! [ -e "$ROOTDIR$all" ] ||
>            [ `stat -c"%s" "$ROOTDIR$all"` -eq 0 ]; then
> .....  

Hm, are you sure the file in your case is really empty? On my system ls works fine:

[atkac@f11 ~]$ touch asd
[atkac@f11 ~]$ ls -s asd 
0 asd

Comment 26 Kostas Georgiou 2009-09-16 12:39:56 UTC
Yes  :(

$ rm -f ~/test; touch ~/test; ls -s ~/test
4 ~/test

$ stat -c "%B %b %s"  ~/test
512 8 0

My guess is that since the filesystems were created long ago with the old default inode size (128?) an extra block is allocated to store selinux/xattrs, since ls -s reports allocated size and not filesize it is correct to return 4 in my case even for an empty file.

Comment 27 Adam Tkac 2009-09-16 12:54:02 UTC
(In reply to comment #26)
> Yes  :(
> 
> $ rm -f ~/test; touch ~/test; ls -s ~/test
> 4 ~/test
> 
> $ stat -c "%B %b %s"  ~/test
> 512 8 0
> 
> My guess is that since the filesystems were created long ago with the old
> default inode size (128?) an extra block is allocated to store selinux/xattrs,
> since ls -s reports allocated size and not filesize it is correct to return 4
> in my case even for an empty file.  

Let's track this issue as a separate one - bug #523682.