Bug 1314379 - The cobblerd logrotate config file can completely disable the whole logrotate system.
The cobblerd logrotate config file can completely disable the whole logrotate...
Status: CLOSED ERRATA
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Server (Show other bugs)
unspecified
All Linux
high Severity high
: ---
: ---
Assigned To: Jan Dobes
Lukáš Hellebrandt
:
Depends On:
Blocks: sat58-errata
  Show dependency treegraph
 
Reported: 2016-03-03 08:25 EST by Thomas Gardner
Modified: 2017-12-13 02:58 EST (History)
4 users (show)

See Also:
Fixed In Version: cobbler-2.0.7-69-sat
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-13 02:58:38 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Thomas Gardner 2016-03-03 08:25:41 EST
Description of problem:

Very simply, the logrotate config file for cobblerd contains (among other stuffs):

------------------------------------------------------------------------
   postrotate
      if [ -f /var/lock/subsys/cobblerd ]; then
         /etc/init.d/cobblerd condrestart > /dev/null
      fi
   endscript
------------------------------------------------------------------------

Apparently under some circumstances (and I don't know what those are), that conditional restart will hang around indefinitely, with it's stderr still connected to the stdin of an awk that the logrotate system spawns.  The end result is that one instance of logrotate will get hung up and never finish.  The next time "run-parts" goes to run it, it sees that the last one hasn't finished, and will refuse to start it again.  Without manual intervention, it will continue to refuse to run any more logrotate instances, forever.

Again, I don't know what is causing the conditional restart to hang around for this particular customer (the sfdc case is 01546883 if you want to reopen it and try to figure it out, that might be a good plan, too), but I found that if you just change that part of the logrotate config file to:

------------------------------------------------------------------------
   postrotate
      if [ -f /var/lock/subsys/cobblerd ]; then
         /etc/init.d/cobblerd condrestart > /dev/null 2>&1
      fi
   endscript
------------------------------------------------------------------------

(just redirect stderr as well), it at least keeps it from hanging up the whole logrotate system.

Version-Release number of selected component (if applicable):

cobbler-2.0.7-67 has the same thing in its logrotate file.

How reproducible:

Not very.  It only happened once in a while for this customer, but at least we found a way to prevent it.

Steps to Reproduce:
1.
2.
3.

Actual results:

Rotating the cobblerd log files hangs up the logrotate system so that NOTHING gets rotated.

Expected results:

We should be able to rotate the cobblerd log files without completely crippling the entire logrotate system.

Additional info:
Comment 1 Tomas Lestach 2016-03-04 07:49:13 EST
Note: Checking cobbler upstream I see the same as we us in downstream cobbler ...

config/cobblerd_rotate-   weekly
config/cobblerd_rotate-   postrotate
config/cobblerd_rotate:      if [ -f /var/lock/subsys/cobblerd ]; then
config/cobblerd_rotate-         /etc/init.d/cobblerd condrestart > /dev/null
config/cobblerd_rotate-      fi
Comment 2 Thomas Gardner 2016-03-04 08:26:08 EST
(In reply to Tomas Lestach from comment #1)
> Note: Checking cobbler upstream I see the same as we us in downstream
> cobbler ...

Yeah, I don't doubt that.  Neither do I doubt that there was probably something else wrong in the user's cobblerd configuration that made this happen in the first place.  I know nothing about cobblerd, I'm a BaseOS guy.  When I got involved all the guy knew was that his logrotate wasn't working.  Nonetheless, having this as an added safety net against this phenomenon doesn't seem like a bad thing, does it?
Comment 3 Thomas Gardner 2016-04-14 15:55:19 EDT
Is there some objection to simply redirecting stderr to /dev/null along with stdout?  It fixes a pretty serious issue for this customer, and I don't recall seeing anyone come up with a possible down side to this.  What's the hold up?
Comment 6 Jan Dobes 2017-09-27 07:37:49 EDT
I'm not able to reproduce this bug but I don't see any problem with adding 2>&1 at the end, newer cobbler is doing it anyway.

fixed in cobbler.git(release20):

6a30b629e9bee1736d960cab6eab2f6686942b2e
Comment 9 Lukáš Hellebrandt 2017-11-09 07:43:25 EST
Verified with cobbler-2.0.7-69.

Verified sanity-only: logrotate works when called repeatedly. Also, the fix only contains current cobbler rotate config from upstream cobbler and it helped the customer.

Note: Logrotate waits for the condrestart task to complete anyway so I don't know how this could help. When I put sleep into condrestart, the logrotate takes <sleep_length> more time to finish.
Also, we are getting rid of the error output which seems regressionish, but after consulting this with devel and considering this is current upstream code, we've decided to keep the fix.
Comment 12 errata-xmlrpc 2017-12-13 02:58:38 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3445

Note You need to log in before you can comment on or make changes to this bug.