Bug 1278082 - bind doesn't start to due wrong systemd startup script
bind doesn't start to due wrong systemd startup script
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: bind (Show other bugs)
7.3
All Linux
high Severity medium
: rc
: ---
Assigned To: Tomáš Hozza
Petr Sklenar
Marc Muehlfeld
: Patch, Regression
: 1294731 (view as bug list)
Depends On:
Blocks: 1203710 1295396 1313485 1289025 1295829
  Show dependency treegraph
 
Reported: 2015-11-04 11:37 EST by Tomáš Hozza
Modified: 2017-04-25 20:50 EDT (History)
11 users (show)

See Also:
Fixed In Version: bind-9.9.4-33.el7
Doc Type: Bug Fix
Doc Text:
The *named-chroot* service now starts correctly Due to a regression, the "-t /var/named/chroot" option was omitted in the `named-chroot.service` file. As a consequence, if the `/etc/named.conf` file was missing, the *named-chroot* service failed to start. Additionally, if different `named.conf` files existed in the `/etc/` and `/var/named/chroot/etc/` directories, the "named-checkconf" utility incorrectly checked the one in the changed-root directory when the service was started. With this update, the option in the service file has been added and the *named-chroot* service now works correctly.
Story Points: ---
Clone Of: 1277820
Environment:
Last Closed: 2016-11-03 21:25:14 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Tomáš Hozza 2015-11-04 11:37:33 EST
+++ This bug was initially created as a clone of Bug #1277820 +++

Description of problem:
The checking of the zone doesn't work correctly on startup

Version-Release number of selected component (if applicable):
bind-chroot-9.10.3-1.fc23.x86_64

How reproducible:
Start bind ind chroot environment

Steps to Reproduce:
1. Create config in chroot environment with a zone file

Actual results:
Doesn't start

Expected results:
To start

Additional info:
Problem is missing chroot directory for zone file checking.
nano -w /usr/lib/systemd/system/named-chroot.service
#ExecStartPre=/bin/bash -c 'if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z /etc/named.conf; else echo "Checking of zone files is disabled"; fi'
ExecStartPre=/bin/bash -c 'if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -t /var/named/chroot -z /etc/named.conf; else echo "Checking of zone files is disabled"; fi'
systemctl daemon-reload
systemctl restart named-chroot

Please fix it urgently, it is a release showstopper

--- Additional comment from Tomas Hozza on 2015-11-04 10:46:51 CET ---

When using bind-chroot package, everything is prepared in a way so that you don't have to think about the chroot directory. This means that all zone files and configuration is bind-mounted automatically (by named-chroot-setup.service) into the chroot environment. This happens ONLY if the files and directories are NOT present inside the chroot environment before the bind-mount happens.

This means that you should keep the configuration in the standard places (/etc/named.conf, /var/named/*, etc.) and then just start the named-chroot.service.

If you copy the configuration into the chroot directory, then you'll have to modify the service files yourself. I'm not going to change the service files, since everything works when used in a way I described.

For more information, please refer to the "Running BIND in a chroot environment" section of https://docs.fedoraproject.org/en-US/Fedora/22/html/Networking_Guide/sec-bind-namedconf.html

--- Additional comment from Gerhard Wiesinger on 2015-11-04 11:18:12 CET ---

OK, config was from a system before the automatismus was implemented and it had to be done manually. Worked well with FC22. Moved to standard bind config. Works well now. Thnx.

--- Additional comment from Gerhard Wiesinger on 2015-11-04 11:47:25 CET ---

BTW: Doesn't it look like a bug then in FC22, because that changed from FC22 to FC23
/usr/lib/systemd/system/named-chroot.service
ExecStartPre=/usr/sbin/named-checkconf -t /var/named/chroot -z /etc/named.conf

--- Additional comment from Tomas Hozza on 2015-11-04 17:26:18 CET ---

(In reply to Gerhard Wiesinger from comment #3)
> BTW: Doesn't it look like a bug then in FC22, because that changed from FC22
> to FC23
> /usr/lib/systemd/system/named-chroot.service
> ExecStartPre=/usr/sbin/named-checkconf -t /var/named/chroot -z
> /etc/named.conf

Good catch. Will fix it soon...
Comment 6 Bill McGonigle 2015-12-16 06:51:35 EST
It looks like the buggy version went out last night - we got -29 via yum updates overnight and our existing bind-chroot servers broke:

  Sep 03 23:03:59 Updated: bind-chroot.x86_64 32:9.9.4-18.el7_1.5
  Dec 16 01:48:53 Updated: bind-chroot.x86_64 32:9.9.4-29.el7

  Dec 16 02:12:14 ns1 bash: /etc/named.conf:251: open: /etc/named-slaves-internal.conf: file not found

Adding '-t "$ROOTDIR"' to /usr/lib/systemd/system/named-chroot.service resolved the issue (I can't see bug 1236475).

19c19
< ExecStartPre=/bin/bash -c 'if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z /etc/named.conf; else echo "Checking of zone files is disabled"; fi'
---
> ExecStartPre=/bin/bash -c 'if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -t "$ROOTDIR" -z /etc/named.conf; else echo "Checking of zone files is disabled"; fi'
Comment 7 alienth@gmail.com 2015-12-17 00:02:59 EST
This went out with Centos, as well. We explicitly remove the normal /etc/named.conf, so this bug resulted in our named-chroot failing to start as it couldn't run checkconf.
Comment 8 Tomáš Hozza 2015-12-17 06:09:24 EST
Please see the bug description for the workarounds, which are trivial.
Comment 9 Bill McGonigle 2016-01-01 20:04:29 EST
The workarounds are certainly trivial - the trick is each minor bind update clobbers them, and pretty soon a geodiverse cluster of nameservers is a pain to keep updated and up.

Is there any chance that this fix (removing non-chrooted /etc/named.conf has been upstream guidance for a long time) will be included in an update soon?   I'd rather not chattr +i all the .service files, but if it's going to be a while (say, 7.3), I've had N x 3 fixes to redeploy so far so we'll need to and back that out later.
Comment 10 Tomáš Hozza 2016-01-04 03:44:27 EST
(In reply to Bill McGonigle from comment #9)
> The workarounds are certainly trivial - the trick is each minor bind update
> clobbers them, and pretty soon a geodiverse cluster of nameservers is a pain
> to keep updated and up.

There is not really a need to modify /usr/lib/systemd/system/named-chroot.service manually after each update. You should not really touch that file at all. Instead you should keep the modified service file in /etc/systemd/system/. For more information, please follow [1].

> Is there any chance that this fix (removing non-chrooted /etc/named.conf has
> been upstream guidance for a long time) will be included in an update soon? 
> I'd rather not chattr +i all the .service files, but if it's going to be a
> while (say, 7.3), I've had N x 3 fixes to redeploy so far so we'll need to
> and back that out later.

I can not make any promises on when this issue will be fixed. If this issue is critical to you, please contact the Red Hat support.

Thank you.

[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/sect-Managing_Services_with_systemd-Unit_Files.html#sect-Managing_Services_with_systemd-Unit_File_Modify
Comment 11 Tomáš Hozza 2016-01-04 03:45:24 EST
*** Bug 1294731 has been marked as a duplicate of this bug. ***
Comment 12 Steven Haigh 2016-01-04 03:59:56 EST
(In reply to Tomas Hozza from comment #10)
> (In reply to Bill McGonigle from comment #9)
> > The workarounds are certainly trivial - the trick is each minor bind update
> > clobbers them, and pretty soon a geodiverse cluster of nameservers is a pain
> > to keep updated and up.
> 
> There is not really a need to modify
> /usr/lib/systemd/system/named-chroot.service manually after each update. You
> should not really touch that file at all. Instead you should keep the
> modified service file in /etc/systemd/system/. For more information, please
> follow [1].

Sorry, but I disagree with your logic here. You *WANT* an updated named-chroot.service to be installed over any custom service file in /etc/systemd/system.

Making non-obvious changes to unit files that have the same name in different locations is a very bad idea.

As this bug has now polluted CentOS and Scientific Linux, it should probably get fixed sooner rather than later - as we're already 2 months into it being a known bug....
Comment 13 Tomáš Hozza 2016-01-04 04:26:41 EST
You don't have to copy the whole file, but rather do change just to the ExecStartPre part. You can use systemd unit drop-in for this purpose.

$ mkdir /etc/systemd/system/named-chroot.service.d
$ vim /etc/systemd/system/named-chroot.service.d/exec-start-pre.conf

# exec-start-pre.conf content
[Service]
ExecStartPre=
ExecStartPre=/usr/sbin/named-checkconf -t /var/named/chroot -z /etc/named.conf

$ init q

You can verify the service file with "systemctl show named-chroot"

Also you can always inspect all the changed done to the unit files on your system by calling "systemd-delta".
Comment 16 Bill McGonigle 2016-01-28 12:01:24 EST
Thanks for the recipe, @Tomas.  Since I commented earlier I just want to verify that your suggestion WFM - I've added it to my bind-chroot puppet module for EL7 and pushed it out widely, to good effect.
Comment 18 Steven Haigh 2016-03-17 06:42:05 EDT
Hi all,

Lots of blocks & things on this bug - yet it still appears on the latest bind packages that have recently been pushed - once again breaking countless nameservers on the net....

Are we holding back on this fix for a specific reason?

Common sense would seem to dictate that such as simple fix should have been pushed months ago.
Comment 19 Steven Haigh 2016-03-17 07:50:09 EDT
For what its worth, this is the commit that caused all this headache:

https://git.centos.org/commit/rpms!bind.git/d56ed2d3a2736a07a09c268f3b2607cca8f1b6ca
Comment 20 Tomáš Hozza 2016-03-17 10:09:19 EDT
Hi Steven.

The bug is approved for RHEL-7.3 update, so there is high possibility it will be fixed in the next RHEL-7 minor update. We already know what caused this unfortunate regression. We also know the fix and there are couple of workarounds mentioned in this bug.

Thank you for your patience.
Comment 30 errata-xmlrpc 2016-11-03 21:25:14 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2233.html

Note You need to log in before you can comment on or make changes to this bug.