Bug 809360 - sd_notifyf doesn't work as expected for httpd
sd_notifyf doesn't work as expected for httpd
Product: Fedora
Classification: Fedora
Component: systemd (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Michal Sekletar
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2012-04-03 04:06 EDT by Jan Kaluža
Modified: 2013-01-15 17:49 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2013-01-15 17:49:45 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
mod_systemd (7.35 KB, application/gzipped-tar)
2012-04-03 04:06 EDT, Jan Kaluža
no flags Details
simple reproducer (1.41 KB, text/plain)
2012-09-06 05:11 EDT, Jan Kaluža
no flags Details
reproducer for second problem (1.56 KB, text/plain)
2012-09-06 06:14 EDT, Jan Kaluža
no flags Details

  None (edit)
Description Jan Kaluža 2012-04-03 04:06:35 EDT
Created attachment 574773 [details]

Description of problem:
I'm trying to use sd_notifyf in httpd to overcome issues with the PID file creation (see Bug #728465) and possibly start using some more advanced systemd features.

To achieve it, I'm trying to write mod_systemd httpd module (attached). It works like this:

1. It calls getpid() to get and store the PID in main httpd process. The pid_file is not written that time yet.
2. When the Apache forks and creates the children processes, it calls sd_notifyf with the PID of main process from the child process. The reason why to do it here is that pid_file is already written that time.

The problem is that if I run "httpd" command without systemd, I see the following in the httpd log meaning that it got to my systemd related code (please ignore logging debug messages as "error", it's only for my convenience):

[Mon Apr 02 15:16:05.678043 2012] [core:notice] [pid 24112:tid 140255251499008] SELinux policy enabled; httpd running as context unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[Mon Apr 02 15:16:05.679088 2012] [suexec:notice] [pid 24112:tid 140255251499008] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Apr 02 15:16:06.000579 2012] [ssl:notice] [pid 24112:tid 140255251499008] AH01886: SSL FIPS mode disabled
[Mon Apr 02 15:16:06.030958 2012] [:error] [pid 24112:tid 140255251499008] AH00521: init_module
[Mon Apr 02 15:16:06.051324 2012] [auth_digest:notice] [pid 24113:tid 140255251499008] AH01757: generating secret for digest authentication ...
[Mon Apr 02 15:16:07.000118 2012] [lbmethod_heartbeat:notice] [pid 24113:tid 140255251499008] AH02282: No slotmem from mod_heartmonitor
[Mon Apr 02 15:16:07.000315 2012] [ssl:notice] [pid 24113:tid 140255251499008] AH01886: SSL FIPS mode disabled
[Mon Apr 02 15:16:07.043374 2012] [:error] [pid 24113:tid 140255251499008] AH00521: systemd_post_config
[Mon Apr 02 15:16:07.043424 2012] [:error] [pid 24113:tid 140255251499008] AH00521: got pid 24113
[Mon Apr 02 15:16:07.045062 2012] [:error] [pid 24115:tid 140255251499008] AH00521: using pid 24113

But if I start it using systemd, it looks like httpd is stopped/killed before it even tries to call sd_notify:

[Mon Apr 02 15:22:33.448722 2012] [core:notice] [pid 24437:tid 139748586375168] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Mon Apr 02 15:22:33.449712 2012] [suexec:notice] [pid 24437:tid 139748586375168] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Apr 02 15:22:34.000216 2012] [ssl:notice] [pid 24437:tid 139748586375168] AH01886: SSL FIPS mode disabled
[Mon Apr 02 15:22:34.035757 2012] [:error] [pid 24437:tid 139748586375168] AH00521: init_module
[Mon Apr 02 15:22:34.055171 2012] [auth_digest:notice] [pid 24438:tid 139748586375168] AH01757: generating secret for digest authentication ...

Service file I currently use looks like this:
> [Unit]
> Description=The Apache HTTP Server
> After=syslog.target network.target remote-fs.target nss-lookup.target
> [Service]
> Type=notify
> PIDFile=/var/run/httpd/httpd.pid
> EnvironmentFile=/etc/sysconfig/httpd
> ExecStart=/usr/sbin/httpd $OPTIONS
> ExecReload=/usr/sbin/httpd $OPTIONS -t
> ExecReload=/usr/sbin/httpd -HUP $MAINPID
> ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop
> PrivateTmp=true
> NotifyAccess=all
> [Install]
> WantedBy=multi-user.target

Version-Release number of selected component (if applicable):
systemd 37-17.fc16

Steps to Reproduce:
1. Compile and install attached mod_systemd and enable it in httpd's configuration file.
2. Start httpd using systemd

Maybe it's caused by my misconfiguration or misunderstanding of the way how it's supposed to work. But I was trying to configure it differently for long time, so I think it's worth reporting.
Comment 1 Jan Kaluža 2012-09-06 05:09:17 EDT
Adding reproducer. I believe this should work with sd_notifyf, shouldn't it?
Comment 2 Jan Kaluža 2012-09-06 05:11:14 EDT
Created attachment 610232 [details]
simple reproducer
Comment 3 Jan Kaluža 2012-09-06 06:13:36 EDT
There is also variation of this reproducer showing different problem. If I keep parent process running and try to update STATUS from child process, it is not reflected in systemctl status output. I still see "Status: "Processing requests..." there.

From the documentation I would expect it will work even called from the child:
If the unset_environment parameter is non-zero sd_notify() will unset the $NOTIFY_SOCKET environment variable before returning (regardless whether the function call itself succeeded or not). Further calls to sd_notify() will then fail, but the variable is no longer inherited by child processes.
Comment 4 Jan Kaluža 2012-09-06 06:14:21 EDT
Created attachment 610248 [details]
reproducer for second problem
Comment 5 Jan Kaluža 2012-09-06 06:22:35 EDT
For the first reproducer I would expect that systemd will wait some time for sd_notify call and it will not care from which children process it comes. I thought that's exactly why sd_notify exists. Currently it kills children when main process terminates.
Comment 6 Lennart Poettering 2012-09-14 11:04:28 EDT
Hmm, you set Type=notify, but you actually have a forking process, right?

You need to set Type=forking in that case, otherwise systemd will assume your main process is dead and terminate you immediately. You also need to set NotifyAccess=all if it is not the original process that sends off the message
Comment 7 Jan Kaluža 2013-01-15 03:17:56 EST
Hm, I think we could close this bug. I remember having discussion about this with someone from systemd community and find out the current systemd behaviour is good.

In httpd we've fixed it by using -DFOREGROUND httpd option, so httpd does not fork and Type=notify can be used.

Feel free to close this bug.
Comment 8 Lennart Poettering 2013-01-15 17:49:45 EST
Cool, thanks!

Note You need to log in before you can comment on or make changes to this bug.