Bug 505754

Summary: Upgrade to F11 breaks named.conf
Product: [Fedora] Fedora Reporter: Michael McLagan <mmclagan>
Component: bindAssignee: Adam Tkac <atkac>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: low    
Version: 12CC: atkac, bill-bugzilla.redhat.com, braden, bugzilla.redhat.com, flint42, gczarcinski, jan.kratochvil, nerijus, oron, ovasik, pwouters, redhat-bugzilla, warthog9
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: bind-9.6.2-5.P2.fc12 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-07-22 11:54:18 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 566840, 566841    
Bug Blocks: 566585    
Attachments:
Description Flags
/etc/named.conf before the breakage
none
minimal patch to prevent some of the breakage
none
Small patch to allow disabling of dnssec in named itself none

Description Michael McLagan 2009-06-13 15:07:13 UTC
Description of problem:

Upgrading to bind-9.6.1-0.4.rc1.fc11.i586 inserts text randomly in configuration file, breaking the local config.

Version-Release number of selected component (if applicable):

bind-9.6.1-0.4.rc1.fc11.i586

How reproducible:

Always

Steps to Reproduce:
1. Use a non distribution named.conf
2. Install the RPM
3. Try to restart named
  
Actual results:

Jun 13 09:19:14 io named: /etc/named.conf:19: missing ';' before 'yes'
Jun 13 09:19:14 io named: /etc/named.conf:20: missing ';' before 'yes'
Jun 13 09:19:14 io named: /etc/named.conf:21: missing ';' before '.'
Jun 13 09:19:14 io named: /etc/named.conf:21: missing ';' before 'trust-anchor'
Jun 13 09:19:14 io named: /etc/named.conf:21: missing ';' before 'dlv.isc.org.'

[root@xx /] service named start
Starting named: 
Error in named configuration:
/etc/named.conf:19: missing ';' before 'yes'
/etc/named.conf:20: missing ';' before 'yes'
/etc/named.conf:21: missing ';' before '.'
/etc/named.conf:21: missing ';' before 'trust-anchor'
/etc/named.conf:21: missing ';' before 'dlv.isc.org.'
                                                           [FAILED]

Expected results:

[root@xx /]# service named start
Starting named:                                            [  OK  ]

Additional info:

--- named.conf.bak	2008-06-16 15:57:54.000000000 -0400
+++ named.conf	2009-06-12 11:01:29.000000000 -0400
@@ -13,17 +13,20 @@
 
 	datasize		64M;
 	notify			yes;
 	recursion		yes;
 	minimal-responses	yes;
 	forward			first;
 
 	forwarders	{ 192.168.0.1;
-	};
+	dnssec-enable yes;
+	dnssec-validation yes;
+	dnssec-lookaside . trust-anchor dlv.isc.org.;
+};
 	sortlist	{
 		{
 		localhost;
 		localnets;
 		};
 		{
 		localnets;
 		};
@@ -75,8 +78,10 @@
 
 zone "bl" IN {
 	type master;
 	file "bl.hosts";
 };
 
 // Any local additions to the basic config
 include "/etc/named.local";
+include "/etc/named.dnssec.keys";
+include "/etc/pki/dnssec-keys/dlv/dlv.isc.org.conf";

Comment 1 Michael McLagan 2009-06-13 15:16:54 UTC
[root@xxxxx etc]# diff -u -U3 named.conf.bak named.conf
--- named.conf.bak      2008-06-16 16:01:55.000000000 -0400
+++ named.conf  2009-06-12 10:04:02.000000000 -0400
@@ -21,7 +21,10 @@
        sortlist {
                { localhost; localnets; };
                { localnets; };
-       };
+       dnssec-enable yes;
+       dnssec-validation yes;
+       dnssec-lookaside . trust-anchor dlv.isc.org.;
+};
 
         // Restrict access to permitted hosts
         allow-query {
@@ -80,3 +83,5 @@
 // Each host has it's own key file
 include "/etc/rndc.key";
 
+include "/etc/named.dnssec.keys";
+include "/etc/pki/dnssec-keys/dlv/dlv.isc.org.conf";

Comment 2 Michael McLagan 2009-06-13 15:35:46 UTC
It looks like the culprit is actually something that bind brought with it (I upgraded via YUM):

 dnssec-conf              noarch   1.20-2.fc11                  distrib    85 k

Seems that /etc/sysconfig/dnssec defaults to ON and every time you try to restart named it runs dnssec-configure which screws around with the named.conf and makes the above messes.

[root@xxxxxxx ~]# dir /etc/named*
-rw-r--r-- 1 root root  1304 Jun 16  2008 /etc/named.conf
-rw-r----- 1 root named  793 Jun 30  2008 /etc/named.conf.rpmnew
-rw-r--r-- 1 root root  4626 Jun 13 11:20 /etc/named.dnssec.keys
-rw-r--r-- 1 root root   561 Apr 27  2008 /etc/named.local
-rw-r----- 1 root named  931 Jun 21  2007 /etc/named.rfc1912.zones
[root@xxxxxxx ~]# ./trial
ERROR: syntax check for named-checkconf /etc/named.conf failed:/etc/named.conf:19: missing ';' before 'yes'
/etc/named.conf:20: missing ';' before 'yes'
/etc/named.conf:21: missing ';' before '.'
/etc/named.conf:21: missing ';' before 'trust-anchor'
/etc/named.conf:21: missing ';' before 'dlv.isc.org.'
[root@xxxxxxx ~]# dir /etc/named*
-rw-r--r-- 1 root root  1481 Jun 13 11:33 /etc/named.conf
-rw-r--r-- 1 root root  1304 Jun 16  2008 /etc/named.conf.bak
-rw-r----- 1 root named  793 Jun 30  2008 /etc/named.conf.rpmnew
-rw-r--r-- 1 root root  4626 Jun 13 11:33 /etc/named.dnssec.keys
-rw-r--r-- 1 root root   561 Apr 27  2008 /etc/named.local
-rw-r----- 1 root named  931 Jun 21  2007 /etc/named.rfc1912.zones
[root@xxxxxxx ~]# cat trial
#!/bin/bash

# Taken from /etc/init.d/named

[ -r /etc/sysconfig/dnssec ] && . /etc/sysconfig/dnssec

[ -x /usr/sbin/dnssec-configure ] && [ -r /etc/named.conf ] && \
  [ /etc/sysconfig/dnssec -nt /etc/named.conf ] && \
   /usr/sbin/dnssec-configure -b --norestart --dnssec="$DNSSEC" --dlv="$DLV"

Comment 3 Paul Wouters 2009-06-15 13:09:45 UTC
Can someone attach or mail me a full named.conf file before it gets mangled?

Comment 4 Jan Kratochvil 2009-06-22 09:22:37 UTC
Created attachment 348869 [details]
/etc/named.conf before the breakage

(In reply to comment #3)
> Can someone attach or mail me a full named.conf file before it gets mangled?  

This is a stripped down version of what it broke for me.

The config file parsing by the oversimplified python script cannot work.

Comment 5 Adam Tkac 2009-06-22 11:44:27 UTC
As a temporary workaround please don't use multiline { }; statements in the options statement in your named.conf.

Comment 6 Paul Wouters 2009-06-25 23:39:20 UTC
This has been fixed in dnssec-conf 1.22 which i will release today.

Comment 7 Gene Czarcinski 2009-09-19 18:10:36 UTC
*** Bug 510290 has been marked as a duplicate of this bug. ***

Comment 8 Gene Czarcinski 2009-09-19 18:22:50 UTC
This update has not appeared.  See --

https://bugzilla.redhat.com/show_bug.cgi?id=510290
https://bugzilla.redhat.com/show_bug.cgi?id=523973

Comment 9 Gene Czarcinski 2009-09-19 19:36:01 UTC
This comment was replicated from 510290.

I have been looking into this problem.  The problems are with the
/usr/sbin/dnssec-configure program which is part of the dnssec-conf package. 
This program is written in python and I am not python literate so a patch/fix
will need to be done by someone else.

I have looked into the upstream creator/maintainer and there is no advertized
means of reporting bugs ... the source is made available and that is it.

You do not need a real name server to work on this ... just a Fedora 11 (or 12)
system.  Install the bind and dnssec-conf packages.  Then create a valid
/etc/named.conf file (I used /etc/named.named.bak as my base).  I added a
"forwarders" statement to the configuration.  I started named to make sure my
/etc/named.conf file was good ... it was.  To do my testing, I did not
start/run named but simply used the following command:
  /usr/sbin/dnssec-configure -b --norestart --dnssec=off --dlv=off

The problem is that dnssec-configure does not properly parse the "options"
statement.  While named (see named-checkconf) allows statements and
sub-statements to be on multiple lines, dnssec-configure does not.

Specific problems:

1.  An option statement is not recognized unless it is immediately followed by
a left brace ("{") on the same line.  As far as I can determine, additional
left braces in substatements are ignored.

2. The program then looks for dnssec-<whatever> statements on separate physical
lines OR the next right brace ("}") which it assumes is the end of the options
statement.  This is wrong as it could be a sub-statement such as forwarders
with a list of IP addresses and a right brace on a separate line for
readability.

3. Given the apparent simplistic nature of parsing used by dnssec-configure,
there are likely other ways that this program could screw up your
/etc/named.conf.

Current workaround ... "don't do that ... it hurts"

That is, make sure that the options statement is immediately followed by a left
brace and that no right braces are on a line by themselves.  For multi line
statements, put the "};" at the end of the last item in the list.

BTW, I am a bit bothered that the Fedora package maintainer has not said
anything concerning this bug.  I am not expecting him to fix this ... it needs
to be fixed upstream.

Comment 10 Paul Wouters 2009-09-19 19:39:38 UTC
It has been more of a time issue. Sorry for delaying on this.

I had changed the code to use pyparsing instead, but ran into a bug in
pyparsing not parsing some of my named.conf testcases properly. I'll try
to bring in a fix during the weekend. If there is anyone with pyparsing
experience, please drop me a line.

Comment 11 Gene Czarcinski 2009-10-09 22:47:38 UTC
ping ... what is the status on fixing this problem?

Comment 12 Paul Wouters 2009-10-14 19:38:52 UTC
I just built dnssec-conf-1.22-1 in the devel branch. Could people give this a test run? I know it does not fix all named.conf issues, but it should hopefully address the most common ones.

Comment 13 Gene Czarcinski 2009-10-14 22:22:01 UTC
OK, I downloaded dnssec-conf-1.22-1.fc13.noarch.rpm from koji and installed it on an F12-rawhide system.  I then ran dnssec-configure against the /etc/named.conf file which had originally prompted this BZ report.  It worked fine. [I first ran the test against the 1.21 version and, of course, it clobbered the named.conf file]

I assume that the next step is to build the rpm for F11 and put it into updates-testing.  At that time, I will be happy to give this a test on my production name server although, given how it worked on my test, I do not see any problem.

I also assume that you will build 1.22 for F12-rawhide and get it in when you can.  I expect that this update will be in post F12beta but before F12 GA ... the sooner it is in F12-rawhide, the more testing it will get.  Is there anything you need me to do with respect to this?

Comment 14 Adam Tkac 2009-10-30 11:53:16 UTC
*** Bug 529947 has been marked as a duplicate of this bug. ***

Comment 15 Bill McGonigle 2010-02-09 09:47:02 UTC
I just had this break a machine triggered by the update of:

Feb 05 03:44:28 Updated: 32:bind-libs-9.6.1-10.P3.fc11.x86_64
Feb 05 03:44:32 Updated: 32:bind-9.6.1-10.P3.fc11.x86_64
Feb 05 03:45:07 Updated: 32:bind-utils-9.6.1-10.P3.fc11.x86_64
Feb 05 03:46:10 Updated: 32:bind-chroot-9.6.1-10.P3.fc11.x86_64

Since the code in question is breaking production machines in the middle of the night as it is, packaging the fix is probably lower risk than leaving the status quo.

Comment 16 George R. Kasica 2010-02-09 10:29:24 UTC
Simlar issues here on FC12 all of a sudden (why else would I be here at 0419 CST).

Feb  9 03:47:49 saturn named: /etc/named.conf:22: missing ';' before 'yes'
Feb  9 03:47:49 saturn named: /etc/named.conf:23: missing ';' before 'yes'
Feb  9 03:47:49 saturn named: /etc/named.conf:24: missing ';' before '.'
Feb  9 03:47:49 saturn named: /etc/named.conf:24: missing ';' before 'trust-anch
or'
Feb  9 03:47:49 saturn named: /etc/named.conf:24: missing ';' before 'dlv.isc.or
g.'


Problem fixed here by removing the ofending lines as I never put them there in the first place...

        dnssec-enable yes{;
        dnssec-validation yes{;
        dnssec-lookaside . trust-anchor dlv.isc.org.{;


include "/etc/pki/dnssec-keys//named.dnssec.keys";
include "/etc/pki/dnssec-keys//dlv/dlv.isc.org.conf";

Comment 17 Oron Peled 2010-02-09 22:11:37 UTC
Created attachment 389858 [details]
minimal patch to prevent some of the breakage

1. This simple patch solved the most common /etc/named.conf breakage:
   option {
       forwarders {
            blahblah;
       };
   };

2. On F-12 dnssec-conf-1.21-7:
   * Build Date: Fri 05 Feb 2010 10:30:16 PM IST
   * After it broke my DNS server, I found bugzilla has this issue
     all over the place since June 2009 (e.g: bugzilla 505754)
   * This is TOTALLY irresponsible pushing the *KNOWN BUGY* version
     for a key infrastructure component.

3. Other important fixes (but maybe item 4. below, would make them
   unnecessary):
   * first rule in data processing -- NEVER rewrite operational data
     before testing them. The correct sequence:
       - write to temporary file
       - test the temporary file
       - If all is well, rename (atomically)
     For dnssec-configure, it looks like checkBindConfig() should get the
     filename as a parameter and be called from UpdateConfigFile on the
     temporary file. Actual renaming should be done only if everything is
     good.

   * If dnssec-configure rewrite /etc/named.conf, it should add relevant
     comments. E.g:
     - In the top of the file. Something like:
       ---- cut -----------------------------------------
        # Modified by dnssec-configure: Tue Feb 9 23:43:05 IST 2010
       ---- cut -----------------------------------------

     - Every added block of lines, delimited:
       ---- cut -----------------------------------------
        # INIT -- dnssec-configure added
        dnssec-enable no;
        dnssec-validation no;
        // dnssec-lookaside . trust-anchor dlv.isc.org.;
        # FINI -- dnssec-configure added
       ---- cut -----------------------------------------

4. Even better plan is a re-org so we don't edit /etc/named.conf
   (that's why we have "include" ...)

   * In bind package:
     - Ship a default /etc/named.conf:
       ---- cut -----------------------------------------
       options {
            ...
       include /etc/named/options-dnssec.conf
       };
       ---- cut -----------------------------------------

      - Make sure it is config(noreplace) -- killing production
        systems is a big no-no!

      - Ship a default (no-op) /etc/named/options-dnssec.conf:
       ---- cut -----------------------------------------
       # All manual edits will be lost.
       # Please read /etc/sysconfig/dnssec
       dnssec-enable no;
       dnssec-validation no;
       ---- cut -----------------------------------------

      - Now a simple 'sed -i' in /etc/init.d/named can toggle 'no' <-> 'yes'
        according to /etc/sysconfig/dnssec. No complex parsing.
        If there's no /etc/sysconfig/dnssec (dnssec-config isn't installed)
        it should obviously default to 'no'.

    * In dnssec-conf package:
      - Add a comment about the logic in /etc/sysconfig/dnssec

    * Save some kittens...

Comment 18 Oron Peled 2010-02-09 22:24:39 UTC
Linked to this bug from two related ones (so they see the attached patch):
 - Bug 548851
 - Bug 563232

Comment 19 John 'Warthog9' Hawley 2010-02-12 18:19:17 UTC
Created attachment 390551 [details]
Small patch to allow disabling of dnssec in named itself

This patch should actually honour the ability to disable dnssec in named itself.  Right now dnssec will modify the named.conf file regardless, just because the checks in named's initrd script never check to see if dnssec is actually enabled.

I would propose fixing this particular issue *Immediately*, I'm *REALLY* sick and tired of having production servers have their named.conf files get squashed despite being read only:

-r--r--r-- 1 root root 7.7K 2010-02-12 17:25 named.conf

and then not having any useful name server since their failed to start because of dnssec-conf's inability to deal with not breaking existing configuration files.

Comment 20 John 'Warthog9' Hawley 2010-02-18 16:20:57 UTC
I'm actually going to bump this priority, even when "disabled" dnssec squashes the named.conf config file.  This basically means my dns servers are failing randomly as they get restarted for whatever reason (logrotate likely).  This is beyond acceptable and I've started having to resort to more extreme measures to keep dns running as expected.

diff from when dnssec is marked as "off" in /etc/sysconfig/dnssec

===================================================================
RCS file: RCS/named.conf,v
retrieving revision 1.98
diff -r1.98 named.conf
39c39,41
<         };
---
> 	dnssec-enable no;
> 	dnssec-validation no;
> };
53d54
< include "/etc/pki/dnssec-keys//dlv/dlv.isc.org.conf";
===================================================================

Which as you can imagine, puts the disabling lines in the wrong spot and causes the restart to fail - yet again.  Really, this needs to get resolved and it's obviously not.  If it can't be trivially resolved in the short term, pull it's requirement as a required package for named so that those of us actually running name servers aren't getting affected by this.

Comment 21 Bill McGonigle 2010-02-18 22:10:17 UTC
It looks like we've got at least:

1) /etc/sysconfig/dnssec respect issues
2) brace matching
3) use of include files
4) options parsing

as separate problems (each of them leading to the same bind-breakage symptom) co-mingled in this bug.  Gene, Oron, G.H., could you guys file bugs for each of the un-represented problems you've highlighted and set them to block 'dnssecmods' (bug 566585) so we can get a clear handle on these?

Comment 22 Bill McGonigle 2010-02-19 17:52:15 UTC
sorry, I meant 'J.H.' in comment# 21.  Duh on me.

Comment 23 Adam Tkac 2010-03-01 12:56:20 UTC
*** Bug 567023 has been marked as a duplicate of this bug. ***

Comment 24 Bug Zapper 2010-04-27 14:52:09 UTC
This message is a reminder that Fedora 11 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 11.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '11'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 11's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 11 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 25 Bill McGonigle 2010-04-27 16:10:06 UTC
Maintainer, please bump this to f12.

Comment 26 Adam Tkac 2010-04-28 09:48:16 UTC
(In reply to comment #25)
> Maintainer, please bump this to f12.    

Done. Note this problem is not present in Fedora 13 (because dnssec-conf is no longer needed there) which contains many DNSSEC related improvements.

I'm not sure if this problem will be ever fixed in Fedora 12 because the real fix will require to write complex grammar for named.conf which is time-consuming task and it will be a benefit only for several F12 users. Thus, at least from my perspective, the best solution will be to close this bug as WONTFIX.

Comment 27 Bill McGonigle 2010-04-29 18:36:59 UTC
Understood, but if it can't be fixed, the problem code should be backed out.  Some people are still going to be in a position where this will break them.

Comment 28 Penelope Fudd 2010-07-13 07:39:48 UTC
Hah.  Yes, it broke me today.

I've got a Fedora 11 system with 'yum -y update' in cron.  Named stopped working, but was not noticed for a couple of weeks due to DNS caching.  Further yum updates didn't fix this because name resolution wasn't working.  And even when /etc/resolv.conf was edited to point to a working name server, there is no update for this because Fedora 11 is beyond its end-of-life.

Wow, nice landmine for Fedora 11 users!

Comment 29 Adam Tkac 2010-07-22 11:54:18 UTC
This issue is no longer present since bind-9.6.2-5.P2.fc12 (which is already in stable repo) because this version obsoleted dnssec-conf package.

For Fedora 11 there is no way how to fix this issue because it is EOL and updates are disabled.

Comment 30 Penelope Fudd 2010-07-22 16:17:11 UTC
Well, can someone at least remove the broken F11 package from the repository, or increment the epoch number of the last working F11 package?

Thanks.