Bug 728465 - Fedora EC2: httpd test failed with symbolic error in /var/log/messages and systemd became inactive.
Summary: Fedora EC2: httpd test failed with symbolic error in /var/log/messages and s...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: httpd
Version: 15
Hardware: Unspecified
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Joe Orton
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-08-05 08:46 UTC by Sachin Ghai
Modified: 2012-08-07 19:53 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-08-07 19:52:59 UTC


Attachments (Terms of Use)
mod_systemd.c (1.90 KB, text/plain)
2011-09-07 11:53 UTC, Jan Kaluža
no flags Details

Description Sachin Ghai 2011-08-05 08:46:56 UTC
Description of problem:
I was testing the community ami "ami-2abf4443" on amazon and while performing the httpd related tests, I got following:

[root@localhost ec2-user]# git clone git://fedorapeople.org/home/fedora/athmane/public_git/testcases-scripts.git
Cloning into testcases-scripts...
remote: Counting objects: 17, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 17 (delta 4), reused 0 (delta 0)
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (4/4), done.
[root@localhost ec2-user]# cd testcases-scripts/httpd/
[root@localhost httpd]# ./runtests.sh 
Starting httpd (via systemctl):                            [  OK  ]
Running ./httpd_basic_auth.sh - httpd basic authentication test.
Adding password for user test
Reloading httpd configuration (via systemctl):  Job failed. See system logs and 'systemctl status' for details.
                                                           [FAILED]
==> [ PASS ]
Running ./httpd_php.sh - httpd handle PHP test
==> [ PASS ]
Running ./httpd_servehtml.sh - httpd serve html page test.
==> [ PASS ]
Running ./httpd_servehtml_ssl.sh - httpd serve html page test over SSL.
==> [ PASS ]
Running ./httpd_vhost.sh - httpd virtual host test.
Reloading httpd configuration (via systemctl):             [  OK  ]
==> [ PASS ]


tail /var/log/messages says:
===============================
Aug  5 07:34:20 localhost yum[1297]: Installed: php-common-5.3.6-2.fc15.x86_64
Aug  5 07:34:20 localhost yum[1297]: Installed: apr-1.4.5-1.fc15.x86_64
Aug  5 07:34:20 localhost yum[1297]: Installed: apr-util-1.3.12-1.fc15.x86_64
Aug  5 07:34:20 localhost yum[1297]: Installed: httpd-tools-2.2.17-10.fc15.1.x86_64
Aug  5 07:34:20 localhost yum[1297]: Installed: apr-util-ldap-1.3.12-1.fc15.x86_64
Aug  5 07:34:22 localhost systemd[1]: Reloading.
Aug  5 07:34:22 localhost systemd-getty-generator[1322]: Failed to create symlink from /lib/systemd/system/serial-getty@.service to /run/systemd/generator/getty.target.wants/serial-getty@hvc0.service: File exists
Aug  5 07:34:22 localhost systemd[1]: /lib/systemd/system-generators/systemd-getty-generator exited with exit status 1.
Aug  5 07:34:22 localhost yum[1297]: Installed: httpd-2.2.17-10.fc15.1.x86_64
Aug  5 07:34:23 localhost yum[1297]: Installed: php-cli-5.3.6-2.fc15.x86_64
Aug  5 07:34:23 localhost yum[1297]: Installed: php-pdo-5.3.6-2.fc15.x86_64
Aug  5 07:34:24 localhost yum[1297]: Installed: mysql-libs-5.5.14-2.fc15.x86_64
Aug  5 07:34:24 localhost yum[1297]: Installed: php-mysql-5.3.6-2.fc15.x86_64
Aug  5 07:34:24 localhost yum[1297]: Installed: php-5.3.6-2.fc15.x86_64
Aug  5 07:34:24 localhost yum[1297]: Installed: 1:mod_ssl-2.2.17-10.fc15.1.x86_64
Aug  5 07:34:26 localhost systemd[1]: Reloading.
Aug  5 07:34:26 localhost systemd-getty-generator[1350]: Failed to create symlink from /lib/systemd/system/serial-getty@.service to /run/systemd/generator/getty.target.wants/serial-getty@hvc0.service: File exists
Aug  5 07:34:26 localhost systemd[1]: /lib/systemd/system-generators/systemd-getty-generator exited with exit status 1.
Aug  5 07:34:27 localhost systemd[1]: httpd.service: control process exited, code=exited status=7



[root@localhost httpd]# systemctl status systemd.service
systemd.service
	  Loaded: error
	  Active: inactive (dead)


Version-Release number of selected component (if applicable):
"ami-2abf4443" 

How reproducible:
always

Steps to Reproduce:
1. git clone git://fedorapeople.org/home/fedora/athmane/public_git/testcases-scripts.git
2. Execute runtest.sh
3.
  
Actual results:
Running ./httpd_basic_auth.sh - httpd basic authentication test.
Adding password for user test
Reloading httpd configuration (via systemctl):  Job failed. See system logs and 'systemctl status' for details.
                                  


Expected results:
All tests should pass.

Additional info:

Comment 1 Lennart Poettering 2011-08-30 23:39:29 UTC
So, the systemd-getty-generator output is just noise and can safely be ignored (fixed now in git). 

It appears as if reloading of apache via the init script failed with exit code 7. That might not be a bug at all. Reassigning to Apache.

Comment 2 Jan Kaluža 2011-09-06 08:41:05 UTC
"reload" exits with exit code 7 if httpd is not running (httpd process with the pid stored in httpd pidfile does not exist).

I have checked your tests and the problem is that httpd needs some time to fork and write its pid to pidfile. If you put "sleep 2" after "service httpd start" in "0-install_httpd.sh", it will *probably* work as expected ("probably", because 2 seconds don't have to be enough everytime... proper solution is to use some loop to check if httpd started completely and is ready to serve pages).

I will close this as NOTABUG, because it doesn't seem to be httpd problem, but test problem.

Comment 3 Michal Schmidt 2011-09-06 09:09:37 UTC
(In reply to comment #2)
> I have checked your tests and the problem is that httpd needs some time to fork
> and write its pid to pidfile.

If that's so, it is a bug. A forking daemon must exit the original process _after_ the pidfile has been written. For all the details, see:
http://0pointer.de/public/systemd-man/daemon.html

Comment 4 Joe Orton 2011-09-06 10:14:51 UTC
Do many daemons actually behave like that?  I picked exim and mysqld at random and it looks like they both behave similarly to httpd - fork(), exit() in the parent, write pid file in the child.

Comment 5 Jan Kaluža 2011-09-06 10:52:42 UTC
I have to admit httpd really acts like Michal pointed out. I'm not so familiar with code so far, but I'm trying to find out particular place where it writes PID and if it's really in child process.

Comment 6 Jan Kaluža 2011-09-06 10:57:57 UTC
Hm, forget the first sentence. I wanted to say it does *not* act like it should according to http://0pointer.de/public/systemd-man/daemon.html.

Comment 7 Joe Orton 2011-09-06 12:00:47 UTC
Yes; httpd is not unusual in how it daemonizes, and I am fairly sceptical when I see people:

a) inventing new "rules" which are so far removed from existing practice, and then
b) declaring that existing practice is "buggy"

It might be better to have httpd run with -DFOREGROUND -DNO_DETACH so the parent does not fork at all - I believe systemd will cope with this, or possibly need a tweak to the service file, and then it avoids the race with the exit + the new parent writing the pid file.

Comment 8 Michal Schmidt 2011-09-06 15:01:36 UTC
(In reply to comment #7)
> Yes; httpd is not unusual in how it daemonizes, and I am fairly sceptical when
> I see people:
> 
> a) inventing new "rules" which are so far removed from existing practice, and
> then
> b) declaring that existing practice is "buggy"

The problem with not following the rule is that the system gets false information about when the daemon is started and ready.

Even before systemd came, wasn't the initscript susceptible to the very same problem when "start" and "reload" were called in a quick succession?

Unfortunately, "existing practice" and "buggy" are not mutually exclusive.

> It might be better to have httpd run with -DFOREGROUND -DNO_DETACH so the
> parent does not fork at all - I believe systemd will cope with this, or
> possibly need a tweak to the service file, and then it avoids the race with the
> exit + the new parent writing the pid file.

Yes, systemd can handle non-forking services. For correctness there still should be a way to tell when the service is started and ready. There are two possibilities:
 1. The service gets patched to add socket activation. Then it can be assumed
    to be ready (to accept connections) at all times and it can use
    "Type=simple".
 2. The service gets patched to call sd_notify() when it's ready. The unit file
    will use "Type=notify". Socket activation is optional.

Comment 9 Michal Schmidt 2011-09-06 15:23:09 UTC
(In reply to comment #0)
> [root@localhost httpd]# systemctl status systemd.service
> systemd.service
>    Loaded: error
>    Active: inactive (dead)

There's no such thing as "systemd.service". You probably meant to check the status of "httpd.service".

Comment 10 Joe Orton 2011-09-06 15:44:41 UTC
(In reply to comment #8)
> The problem with not following the rule is that the system gets false
> information about when the daemon is started and ready.

There is no "false information", that is silly.  All you are saying is that systemd's (entirely arbitrary) expectation about what the exit code means is completely different from the actual behaviour.

Jan is looking into moving the service to the non-forked type.

Comment 11 Jan Kaluža 2011-09-07 06:10:06 UTC
Or maybe writing module to support sd_notify(). I will try both and see.

Comment 12 Michal Schmidt 2011-09-07 10:38:13 UTC
(In reply to comment #10)
> (In reply to comment #8)
> > The problem with not following the rule is that the system gets false
> > information about when the daemon is started and ready.
> 
> There is no "false information", that is silly.  All you are saying is that
> systemd's (entirely arbitrary) expectation about what the exit code means is
> completely different from the actual behaviour.

The expectation is not arbitrary. It is the only reasonable interpretation of the exit code. Otherwise the exit code would be meaningless.

I already mentioned the problem is not caused by systemd. The problem shows up even when systemd is not involved. For a proof, try this:

[root@alice ~]# cd /etc/init.d
[root@alice init.d]# export SYSTEMCTL_SKIP_REDIRECT=1
[root@alice init.d]# ./httpd start; ./httpd reload
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using ::1 for ServerName
                                                           [  OK  ]
Reloading httpd:                                           [FAILED]
[root@alice init.d]# echo $?
7

Comment 13 Jan Kaluža 2011-09-07 11:49:05 UTC
Well, for systemd, it can be partly fixed by mod_systemd I'm attaching. It's compromise between patching httpd and doing nothing and it fixes "start;reload" sequence using systemctl.

-DFOREGROUND -DNO_DETACH didn't work correctly for me and I think there would be problem with graceful restarts. Unfortunately I can't say if this can be fixed upstream or not.

Comment 14 Jan Kaluža 2011-09-07 11:53:56 UTC
Created attachment 521848 [details]
mod_systemd.c

Comment 15 Fedora End Of Life 2012-08-07 19:53:02 UTC
This message is a notice that Fedora 15 is now at end of life. Fedora
has stopped maintaining and issuing updates for Fedora 15. It is
Fedora's policy to close all bug reports from releases that are no
longer maintained. At this time, all open bugs with a Fedora 'version'
of '15' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we were unable to fix it before Fedora 15 reached end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora, you are encouraged to click on
"Clone This Bug" (top right of this page) and open it against that
version of Fedora.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping


Note You need to log in before you can comment on or make changes to this bug.