Bug 1292611

Summary: Assertion failure in resolver.c: REQUIRE((((fctx->finds).head == ((void *)0)) ? isc_boolean_true : isc_boolean_false))
Product: Red Hat Enterprise Linux 6 Reporter: Nick Urbanik <nicku>
Component: bindAssignee: Tomáš Hozza <thozza>
Status: CLOSED NOTABUG QA Contact: qe-baseos-daemons
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.7CC: fweimer, huzaifas, jwright, thozza
Target Milestone: rcKeywords: Reopened, Security
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-04 08:26:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1293250    

Description Nick Urbanik 2015-12-17 22:31:43 UTC 
Description of problem:

Bind failed with 11 REQUIRE failures in a period of a few hours.

Version-Release number of selected component (if applicable):

9.8.2-0.37.rc1.el6_7.4.x86_64

How reproducible:

Has not occurred since 16 December 2015; have not identified the trigger.

Steps to Reproduce:
1. 
2.
3.

Actual results:

BIND died 11 times (on a number of different servers)

Expected results:

BIND should not die unless asked to stop.

Additional info:

Here are excerpts from the /var/named/chroot/var/log/messages files:
16-Dec-2015 03:33:25.000 client: client 10.206.88.22#11109: recursive-clients soft limit exceeded (9904/9900/10000), aborting oldest query
16-Dec-2015 03:33:25.940 general: resolver.c:3123: REQUIRE((((fctx->finds).head == ((void *)0)) ? isc_boolean_true : isc_boolean_false)) failed
16-Dec-2015 03:33:25.941 general: exiting (due to assertion failure)
16-Dec-2015 06:29:47.000 client: client 49.199.24.156#43970: recursive-clients soft limit exceeded (9909/9900/10000), aborting oldest query
16-Dec-2015 06:29:47.521 general: resolver.c:3123: REQUIRE((((fctx->finds).head == ((void *)0)) ? isc_boolean_true : isc_boolean_false)) failed
16-Dec-2015 06:29:47.521 general: exiting (due to assertion failure)
16-Dec-2015 06:29:08.000 client: client 10.204.43.127#5035: recursive-clients soft limit exceeded (9907/9900/10000), aborting oldest query
16-Dec-2015 06:29:08.556 general: resolver.c:3123: REQUIRE((((fctx->finds).head == ((void *)0)) ? isc_boolean_true : isc_boolean_false)) failed
16-Dec-2015 06:29:08.556 general: exiting (due to assertion failure)
16-Dec-2015 13:23:39.002 client: client 110.21.91.221#53551: recursive-clients soft limit exceeded (9901/9900/10000), aborting oldest query
16-Dec-2015 13:23:39.817 general: resolver.c:3123: REQUIRE((((fctx->finds).head == ((void *)0)) ? isc_boolean_true : isc_boolean_false)) failed
16-Dec-2015 13:23:39.817 general: exiting (due to assertion failure)
16-Dec-2015 13:54:45.000 client: client 10.204.44.161#28244: recursive-clients soft limit exceeded (9901/9900/10000), aborting oldest query
16-Dec-2015 13:54:45.592 general: resolver.c:3123: REQUIRE((((fctx->finds).head == ((void *)0)) ? isc_boolean_true : isc_boolean_false)) failed
16-Dec-2015 13:54:45.592 general: exiting (due to assertion failure)
16-Dec-2015 15:44:36.000 client: client 49.195.170.22#18817: recursive-clients soft limit exceeded (9901/9900/10000), aborting oldest query
16-Dec-2015 15:44:36.716 general: resolver.c:3123: REQUIRE((((fctx->finds).head == ((void *)0)) ? isc_boolean_true : isc_boolean_false)) failed
16-Dec-2015 15:44:36.716 general: exiting (due to assertion failure)
16-Dec-2015 06:27:36.000 client: client 10.204.247.33#14995: recursive-clients soft limit exceeded (9901/9900/10000), aborting oldest query
16-Dec-2015 06:27:36.613 general: resolver.c:3123: REQUIRE((((fctx->finds).head == ((void *)0)) ? isc_boolean_true : isc_boolean_false)) failed
16-Dec-2015 06:27:36.614 general: exiting (due to assertion failure)
16-Dec-2015 06:50:06.000 client: client 49.195.168.80#59498: recursive-clients soft limit exceeded (9909/9900/10000), aborting oldest query
16-Dec-2015 06:50:06.376 general: resolver.c:3123: REQUIRE((((fctx->finds).head == ((void *)0)) ? isc_boolean_true : isc_boolean_false)) failed
16-Dec-2015 06:50:06.376 general: exiting (due to assertion failure)
16-Dec-2015 11:27:51.008 client: client 58.111.135.165#58059: recursive-clients soft limit exceeded (9901/9900/10000), aborting oldest query
16-Dec-2015 11:27:51.195 general: resolver.c:3123: REQUIRE((((fctx->finds).head == ((void *)0)) ? isc_boolean_true : isc_boolean_false)) failed
16-Dec-2015 11:27:51.195 general: exiting (due to assertion failure)
16-Dec-2015 12:54:55.002 client: client 10.204.10.25#10198: recursive-clients soft limit exceeded (9901/9900/10000), aborting oldest query
16-Dec-2015 12:54:55.765 general: resolver.c:3123: REQUIRE((((fctx->finds).head == ((void *)0)) ? isc_boolean_true : isc_boolean_false)) failed
16-Dec-2015 12:54:55.765 general: exiting (due to assertion failure)
16-Dec-2015 13:47:52.001 client: client 1.40.136.246#58047: recursive-clients soft limit exceeded (9901/9900/10000), aborting oldest query
16-Dec-2015 13:47:52.285 general: resolver.c:3123: REQUIRE((((fctx->finds).head == ((void *)0)) ? isc_boolean_true : isc_boolean_false)) failed
16-Dec-2015 13:47:52.285 general: exiting (due to assertion failure)

I originially thought this might be bug 1291186, and this comment added there indicates that there is an upstream patch for this bug that could be back ported:

Tomas Hoger 2015-12-17 03:09:45 EST

Upstream commit as applied to 9.9.8:

https://source.isc.org/cgi-bin/gitweb.cgi?p=bind9.git;a=commitdiff;h=07970f1c57b61a06f71f0db0ce103bf31ea8f1c9

Note that we pay Red Hat for support.
Comment 3 Huzaifa S. Sidhpurwala 2015-12-21 08:50:51 UTC 
Why is bind 9.8.2-0.37.rc1.el6_7.4.x86_64 being used here? Latest version of bind in rhel-6.8 is 9.8.2-0.44.rc1.el6

I see a few CVEs being resolved between the version used by customer and the latest version we ship
Comment 4 Huzaifa S. Sidhpurwala 2015-12-21 08:57:24 UTC 
(In reply to Huzaifa S. Sidhpurwala from comment #3)
> Why is bind 9.8.2-0.37.rc1.el6_7.4.x86_64 being used here? Latest version of
> bind in rhel-6.8 is 9.8.2-0.44.rc1.el6
> 
> I see a few CVEs being resolved between the version used by customer and the
> latest version we ship

Actually scratch that ^

Can you try using the latest bind available in rhel-6.7 ie 
bind-9.8.2-0.37.rc1.el6_7.5.x86_64.rpm which was issued via 
https://rhn.redhat.com/errata/RHSA-2015-2655.html


thanks!
Comment 5 Joe Wright 2015-12-21 18:24:15 UTC 
Has the upstream commit specified been applied to the ERRATA package you linked?
Comment 6 Nick Urbanik 2015-12-22 00:02:13 UTC 
(In reply to Huzaifa S. Sidhpurwala from comment #4)
> (In reply to Huzaifa S. Sidhpurwala from comment #3)
> > Why is bind 9.8.2-0.37.rc1.el6_7.4.x86_64 being used here?

I immediately installed the update you mention below; that was the version that was installed when the attack took place.

> Can you try using the latest bind available in rhel-6.7 ie 
> bind-9.8.2-0.37.rc1.el6_7.5.x86_64.rpm which was issued via 
> https://rhn.redhat.com/errata/RHSA-2015-2655.html

Yes, I installed this package immediately after the failures on all our DNS servers; we have had no recurrence of the REQUIRE causing termination of named since.  That does not necessarily mean that the problem is resolved.
Comment 7 Nick Urbanik 2015-12-22 00:03:35 UTC 
(In reply to Joe Wright from comment #5)
> Has the upstream commit specified been applied to the ERRATA package you
> linked?

This question needs an answer; I hope it has, but that does not appear to be the case from the changelog.
Comment 8 Nick Urbanik 2015-12-22 00:07:21 UTC 
(In reply to Nick Urbanik from comment #6)
> I immediately installed the update you mention below; that was the version
> that was installed when the attack took place.

Disambiguation: When the crashes happened, I had 9.8.2-0.37.rc1.el6_7.4.x86_64 installed.  Shortly after, I found that Red Hat had made bind-9.8.2-0.37.rc1.el6_7.5.x86_64 available, and installed that.  It is for this reason that I raised the bug against 9.8.2-0.37.rc1.el6_7.4.x86_64.
Comment 9 Huzaifa S. Sidhpurwala 2015-12-22 02:56:27 UTC 
(In reply to Nick Urbanik from comment #8)
> (In reply to Nick Urbanik from comment #6)
> > I immediately installed the update you mention below; that was the version
> > that was installed when the attack took place.
> 
> Disambiguation: When the crashes happened, I had
> 9.8.2-0.37.rc1.el6_7.4.x86_64 installed.  Shortly after, I found that Red
> Hat had made bind-9.8.2-0.37.rc1.el6_7.5.x86_64 available, and installed
> that.  It is for this reason that I raised the bug against
> 9.8.2-0.37.rc1.el6_7.4.x86_64.

Patches are applied only on top of latest versions. That being said, i understand that this is no longer a security issue. Please direct any such questions you have to Red Hat support, via a support ticket. 

Thanks!
Comment 10 Nick Urbanik 2015-12-22 05:56:34 UTC 
(In reply to Huzaifa S. Sidhpurwala from comment #9)

> Patches are applied only on top of latest versions.

As I said earlier, we are on the latest version of bind for RHEL 6, bind-9.8.2-0.37.rc1.el6_7.5.x86_64.

> That being said, i understand that this is no longer a security issue.

How do you understand that?  I did not say that it is not; do you have any evidence that the security issue is addressed by bind-9.8.2-0.37.rc1.el6_7.5.x86_64?

> Please direct any such questions you have to Red Hat support, via a
> support ticket.

I will raise a support ticket, but am puzzled by what you have said.
Comment 11 Huzaifa S. Sidhpurwala 2015-12-22 10:21:58 UTC 
I am sorry, did i misinterpret?

"
Yes, I installed this package immediately after the failures on all our DNS servers; we have had no recurrence of the REQUIRE causing termination of named since.  That does not necessarily mean that the problem is resolved."

Does the above not mean, that after applying "bind-9.8.2-0.37.rc1.el6_7.5.x86_64." your issue (for which this flaw is filed) is no longer observed.

Do you still see the crash?

If so, could you please provide a stack-trace which will debug the problem.