Bug 888225 - nginx log rotation fails
Summary: nginx log rotation fails
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: nginx
Version: 18
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Nobody's working on this, feel free to take it
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-12-18 10:24 UTC by Neil Darlow
Modified: 2020-11-05 10:05 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-12-28 03:51:19 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Neil Darlow 2012-12-18 10:24:37 UTC
Description of problem:

Due to an incorrect signal specification, nginx continues to log to the rotated logfile.

--signal=USR1 should be changed to --signal=SIGUSR1

Additionally, the nginx documentation states that, if logging configuration directives contain variables, the log directory should be writeable by the child process uid (nginx on Fedora).

Ideally /var/log/nginx and the files within should be owned by nginx. This will require the addition of: create 0644 nginx or similar to /etc/logrotate.d/nginx.

There might also be a case for creation of a nginx_log_t SELinux fcontext type to be applied to /var/log/nginx and its contained files.

Version-Release number of selected component (if applicable):

nginx-1.2.6-1

How reproducible:

Every time

Steps to Reproduce:
1. Install and configure nginx
2. Access some web content to create a non-zero length logfile
3. Wait for log rotation
4. Access some more web content
5. Observe that logging continues to the rotated logfile not the newly created one
  
Actual results:

Logging continues to the rotated logfile

Expected results:

With a correctly configured systemctl kill --signal=SIGUSR1 nginx.service logging after rotation performs as expected.

Additional info:

nginx log handling requires review and overhaul.

Comment 1 Jamie Nguyen 2012-12-18 11:50:17 UTC
I couldn't replicate your issues with USR1 and SIGUSR1. Both seem to work the same for me. Also Lennart's blog post [1] suggests that HUP and be used in place of SIGHUP so I don't think it matters much. However, SIGUSR1 is indeed the correct name for the signal, so it's probably best to change it to that anyway to avoid confusion. Please let us know if this fixes your issue.

I do agree that "create 0644 nginx root" should be added to the logrotate configuration.

Builds to follow shortly...

Comment 2 Fedora Update System 2012-12-18 11:58:16 UTC
nginx-1.2.6-2.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/nginx-1.2.6-2.fc18

Comment 3 Fedora Update System 2012-12-18 12:05:18 UTC
nginx-1.0.15-7.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/nginx-1.0.15-7.fc17

Comment 4 Neil Darlow 2012-12-18 12:08:00 UTC
Hi Jamie,

Thanks for the swift action. I haven't seen Lennart's blog post on this subject but the systemctl manpage has this to say:

       --signal=, -s
           When used with kill, choose which signal to send to selected
           processes. Must be one of the well known signal specifiers such as
           SIGTERM, SIGINT or SIGSTOP. If omitted defaults to SIGTERM.

I am following fc18 updates-testing so should pick-up the new package soon and will report back my findings ASAP.

Regards,
Neil Darlow

Comment 5 Fedora Update System 2012-12-18 21:24:30 UTC
Package nginx-1.2.6-2.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing nginx-1.2.6-2.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-20553/nginx-1.2.6-2.fc18
then log in and leave karma (feedback).

Comment 6 Jeremy Hinegardner 2012-12-18 21:37:59 UTC
With regards to the /var/log/nginx directory being owned by the nginx user, that should not be allowed. I originally had /var/log/nginx owned by the nginx user, and I changed it to be owned by root based on feedback from Alexander Peslyak (http://en.wikipedia.org/wiki/Solar_Designer).

See the changelog.

* Mon Feb 15 2010 Jeremy Hinegardner <jeremy at hinegardner dot org> - 0.6.39-4
- change directory ownership of log dir to root:root

There is an issue with nginx and log rotation that I do not know if it has been resolved or not.

Nginx opens the log files before it gives up root. This means that if the log directory is owned by nginx user then there is the possibility for a symlink attack.

So, if you do not want a symlink attack to possibly take place then you have to leave them owned by root.

The other side is that when the sigusr1 is sent to reopen the log files, the nginx process is now running as the nginx user, and it does not have the permissions to open the files since they are owned by root.

I decided to stay on the secure side of things.

I have not kept up to speed on the changes in nginx with respect to the log file opening and its security that is going on.

I would say, the only way you should change the /ver/log/nginx directory to be owned by nginx is if:

1) nginx opens the log files AFTER giving up root permissions
2) the log files can now be opened with some sort of umask so you can have a common group between root and nginx 
3) nginx has been changed to make sure that the log files are owned by it and are not symlinks to other files.

-jeremy

Comment 7 Jamie Nguyen 2012-12-18 22:11:44 UTC
Just to mention that in the package builds I uploaded today, the /var/log/nginx directory is still root:root owned.

Comment 8 Neil Darlow 2012-12-19 08:14:20 UTC
Hi Jamie,

I noticed that logfiles are set by: create 0644 root nginx

This means that they will be readable by nginx but not writeable.

Would a solution be?

1) create 0664 root nginx
2) Use SELinux policy to only permit nginx rw access to /var/log/nginx/*log

Comment 9 Neil Darlow 2012-12-19 09:26:58 UTC
Hi,

With the new update, things are still broken. The situation is:

1) A new logfile /var/log/nginx/access.log is created
2) The existing logfile is rotated as /var/log/access.log-20121219
3) The rotated logfile is not compressed
4) An e-mail is received with content:

Subject: Anacron job 'cron.daily' on aspire.darlow.co.uk

/etc/cron.daily/logrotate:

Failed to issue method call: Access denied
error: error running shared postrotate script for '/var/log/nginx/*log '

Comment 10 Jamie Nguyen 2012-12-19 09:42:10 UTC
Hi Neil,

I can't seem to replicate your issues.

- using fresh f18, updates-testing enabled, selinux enabled
- using fresh install of nginx-1.2.6-2
- systemctl start nginx.service
- echo hello >> /var/log/nginx/access.log
- echo hello >> /var/log/nginx/error.log
- logrotate -f /etc/logrotate.conf
- ls -l:
    nginx:nginx access.log
    root:root   access.log.20121219.gz
    nginx:nginx error.log
    root:root   error.log.20121219.gz

No errors for me. Similar story on f17.

I will change logrotate script to "create 0644 nginx nginx" as that seems more correct, although I don't think that in itself will change anything. The master process (which runs as root) changes the ownership of the log files anyway.

Unfortunately, I'm not sure how to replicate your issue.

Comment 11 Neil Darlow 2012-12-19 09:48:03 UTC
Hi Jamie,

I found that when I run logrotate manually, as root, it works. Does /etc/logrotate.d/nginx execute as non-root or perform its file manipulations as the uid and gid specified in create?

I think there is a difference between manual invocation and cron invocation.

Regards,
Neil Darlow

Comment 12 Jamie Nguyen 2012-12-19 09:55:44 UTC
AFAIK, cron runs as root, and therefore should run logrotate as root.

Have a look at the comments here:
http://ask.metafilter.com/117371/Logrotate-cronjob-fails-but-running-at-commandline-works

Do any of the other files in your /etc/logrotate.d/ folder have a missing "endscript" line or otherwise unterminated script?

Comment 13 Fedora Update System 2012-12-19 09:59:50 UTC
nginx-1.2.6-3.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/nginx-1.2.6-3.fc18

Comment 14 Fedora Update System 2012-12-19 10:00:38 UTC
nginx-1.0.15-8.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/nginx-1.0.15-8.fc17

Comment 15 Neil Darlow 2012-12-19 11:13:04 UTC
(In reply to comment #12)
> AFAIK, cron runs as root, and therefore should run logrotate as root.
> 
> Have a look at the comments here:
> http://ask.metafilter.com/117371/Logrotate-cronjob-fails-but-running-at-
> commandline-works
> 
> Do any of the other files in your /etc/logrotate.d/ folder have a missing
> "endscript" line or otherwise unterminated script?

No but there is a clue in the Anacron mail. I did a Google on: "Failed to issue method call: Access denied" and all results indicate that is was output from systemctl.

So, for some reason the systemctl kill command is failing which would be consistent with what I am seeing with logging still occurring to the rotated file. This, in turn, implies that nginx did not receive the SIGUSR1.

Coincidentally, executing: systemctl kill --signal=SIGUSR1 nginx.service as non-root causes exactly that error to be output.

So, going up the chain we have: systemctl --> logrotate --> cron.daily --> run-parts --> anacron --> cron

Any thoughts on whether any of these could be triggering the failure?

Comment 16 Jamie Nguyen 2012-12-20 10:57:35 UTC
On closer inspection, SELinux policy needs to be amended:

https://bugzilla.redhat.com/show_bug.cgi?id=889151

Comment 17 Fedora Update System 2012-12-28 03:51:21 UTC
nginx-1.0.15-8.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 18 Fedora Update System 2013-01-12 00:30:57 UTC
nginx-1.2.6-3.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.