Bug 972376 - rate-limit-patch and flooding logs
rate-limit-patch and flooding logs
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: bind (Show other bugs)
17
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Tomáš Hozza
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-08 22:39 EDT by Harald Reindl
Modified: 2013-08-01 08:29 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-01 08:29:13 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Harald Reindl 2013-06-08 22:39:37 EDT
first: *THANK YOU* for the rate-limit patch

but the logging should be rate-limited to

the lines above are all for the same authoritative domain
there are also such blocks from the ame sources for other domains
this has the potential to fill the disk and should be somehow reduced
without lose the general information for which IP rate-limits took action

09-Jun-2013 04:35:31.820 query-errors: client 209.205.76.125#32957 (*****): rate limit slip response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.821 query-errors: client 209.205.76.125#32957 (*****): rate limit drop response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.821 query-errors: client 209.205.76.125#32957 (*****): rate limit slip response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.821 query-errors: client 209.205.76.125#32957 (*****): rate limit drop response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.822 query-errors: client 209.205.76.125#32957 (*****): rate limit slip response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.822 query-errors: client 209.205.76.125#32957 (*****): rate limit drop response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.822 query-errors: client 209.205.76.125#32957 (*****): rate limit slip response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.823 query-errors: client 209.205.76.125#32957 (*****): rate limit slip response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.823 query-errors: client 209.205.76.125#32957 (*****): rate limit drop response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.823 query-errors: client 209.205.76.125#32957 (*****): rate limit slip response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.823 query-errors: client 209.205.76.125#32957 (*****): rate limit drop response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.823 query-errors: client 209.205.76.125#32957 (*****): rate limit drop response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.824 query-errors: client 209.205.76.125#32957 (*****): rate limit slip response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.824 query-errors: client 209.205.76.125#32957 (*****): rate limit drop response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.824 query-errors: client 209.205.76.125#32957 (*****): rate limit slip response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.824 query-errors: client 209.205.76.125#32957 (*****): rate limit slip response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.824 query-errors: client 209.205.76.125#32957 (*****): rate limit drop response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.825 query-errors: client 209.205.76.125#32957 (*****): rate limit drop response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.825 query-errors: client 209.205.76.125#32957 (*****): rate limit drop response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.825 query-errors: client 209.205.76.125#32957 (*****): rate limit slip response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.825 query-errors: client 209.205.76.125#32957 (*****): rate limit drop response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.825 query-errors: client 209.205.76.125#32957 (*****): rate limit slip response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
09-Jun-2013 04:35:31.827 query-errors: client 209.205.76.125#32957 (*****): rate limit slip response to 209.205.76.0/24 for ***** IN ANY  (0001a45b)
Comment 1 Tomáš Hozza 2013-06-10 03:59:09 EDT
Can you please write me what version of BIND and RRL patch are you using
(named -v)? I suppose it is the latest one in stable. If so, can you please
try bind-9.9.3 from updates-testing? It includes the latest version of RRL patch
which should have somehow improved logging.

If it won't help I'll discuss this issue with upstream.

Thank you!
Comment 2 Harald Reindl 2013-06-10 05:47:50 EDT
Hi

bind-9.9.3-3.P1.fc17.x86_64

i deployed it last week because i am now 3 weeks at vacation
and enabled the rate-control config on sunday night

 rate-limit  
 {
  responses-per-second 5;
  window               10;
 };
Comment 3 Tomáš Hozza 2013-06-17 05:40:46 EDT
I discussed this with the upstream. I assume these errors are not from
syslog, but from BIND own log files. RRL messages about dropped queries
are logged "info" severity level. You can limit the log size by adding
a limit to the channel that receives the query-errors category. Upstream
suggested something like:

category query-errors {
        info-log;
};

channel info-log {
        file "log/info" versions 10 size 10m;
};

Please try to limit your logging channel and let me know if it helps.
Comment 4 Tomáš Hozza 2013-06-18 03:27:01 EDT
(In reply to Tomas Hozza from comment #3)
> category query-errors {
>         info-log;
> };

There should be also "rate-limit" category available.
Comment 5 Harald Reindl 2013-06-18 09:48:04 EDT
i give it a try after my vacation. not that trivial to test since the named.conf is genreated from a database-backend.............

however, if you look at my initial post you see *a lot* of identical lines for the same IP within the same second and we are far away froma real amplification attack - i fear if that would happen named is self-DOSed with logging and log-rotating!
Comment 6 Harald Reindl 2013-06-18 09:56:34 EDT
one last comment:

the rate-control patch IMHO should not produce more than 3-5 log-lines in the same second, yes you would lose informations but in the c ase for it was written wich is a "DNS amplification attack" you do not want teh full log nor have you time to read it 

maybe it would be possible to buffer these logs in a configurable amount of RAM and write them delayed to the logs like http://httpd.apache.org/docs/current/mod/mod_log_config.html#bufferedlogs and in this case ignoring the exact timestamp and store the one from the moment it is written a line could look like while the "XXX times" would be a interesting value

"09-Jun-2013 04:35:31.825 query-errors: 500 times: client 209.205.76.125 rate limit drop response to 209.205.76.0/24 for ***** IN ANY"

one of my webservers was last year target of a DDOS and hence what is going on in such situations is hardly logable nor can anybody read it
Comment 7 Harald Reindl 2013-06-19 20:13:36 EDT
well, i tested this on my homeserver

"category rate-limit" works not as expected

"rate_limit.log" contains only one line while the rest still goes to the genereal log, but however *than you* for the size-option which i did not know in fact after many years of maintain production dns-servers *shame* 

[root@srv-rhsoft:~/named/data]$ cat rate_limit.log
20-Jun-2013 02:09:03.199 rate-limit: limit responses to 127.0.0.0/24 for localhost IN A  (081a4862)
_____________________________________________

-rw-rw---- 1 named named 5,2K 2013-06-20 02:09 named.log
-rw-rw---- 1 named named  100 2013-06-20 02:09 rate_limit.log
_____________________________________________

logging
{
 channel default_log
 {
  file                 "data/named.log" versions 1 size 10m;
  severity             dynamic;
  print-time           yes;
  print-category       yes;
 };
 channel transfer_log
 {
  file                 "data/transfer.log" versions 1 size 10m;
  severity             dynamic;
  print-time           yes;
  print-category       yes;
 };
 channel rate_limit_log
 {
  file                 "data/rate_limit.log" versions 1 size 10m;
  severity             dynamic;
  print-time           yes;
  print-category       yes;
 };
 channel lame_servers_log
 { 
  file                 "data/lame_servers.log" versions 1 size 10m;
  severity             dynamic;
  print-time           yes;
  print-category       yes;
 };
 category default      {default_log;};
 category resolver     {default_log;};
 category security     {default_log;};
 category xfer-in      {transfer_log;};
 category xfer-out     {transfer_log;};
 category config       {default_log;};
 category queries      {default_log;};
 category notify       {default_log;};
 category database     {default_log;};
 category rate-limit   {rate_limit_log;};
 category lame-servers {lame_servers_log;};
};
Comment 8 Harald Reindl 2013-06-19 20:19:15 EDT
in fact most of the noise goes to "query-errors" and not "rate-limit"

20-Jun-2013 02:17:17.609 query-errors: client 127.0.0.1#64258 (localhost): rate limit slip response to 127.0.0.0/24 for localhost IN A  (081a4862)
20-Jun-2013 02:17:17.623 query-errors: client 127.0.0.1#20101 (localhost): rate limit drop response to 127.0.0.0/24 for localhost IN A  (081a4862)
20-Jun-2013 02:17:18.677 query-errors: client 127.0.0.1#14696 (localhost): rate limit slip response to 127.0.0.0/24 for localhost IN A  (081a4862)
20-Jun-2013 02:17:18.684 query-errors: client 127.0.0.1#44717 (localhost): rate limit drop response to 127.0.0.0/24 for localhost IN A  (081a4862)
20-Jun-2013 02:17:19.740 query-errors: client 127.0.0.1#25976 (localhost): rate limit slip response to 127.0.0.0/24 for localhost IN A  (081a4862)
20-Jun-2013 02:17:19.754 query-errors: client 127.0.0.1#55496 (localhost): rate limit drop response to 127.0.0.0/24 for localhost IN A  (081a4862)
20-Jun-2013 02:17:20.807 query-errors: client 127.0.0.1#62600 (localhost): rate limit slip response to 127.0.0.0/24 for localhost IN A  (081a4862)
20-Jun-2013 02:17:20.822 query-errors: client 127.0.0.1#57450 (localhost): rate limit drop response to 127.0.0.0/24 for localhost IN A  (081a4862)
Comment 9 Fedora End Of Life 2013-07-04 00:02:12 EDT
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 10 Fedora End Of Life 2013-08-01 08:29:17 EDT
Fedora 17 changed to end-of-life (EOL) status on 2013-07-30. Fedora 17 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.