Bug 809360

Summary: sd_notifyf doesn't work as expected for httpd
Product: [Fedora] Fedora Reporter: Jan Kaluža <jkaluza>
Component: systemdAssignee: Michal Sekletar <msekleta>
Status: CLOSED WORKSFORME QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: johannbg, jorton, jskarvad, lpoetter, metherid, msekleta, notting, plautrba, systemd-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-15 22:49:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
mod_systemd
none
simple reproducer
none
reproducer for second problem none

Description Jan Kaluža 2012-04-03 08:06:35 UTC
Created attachment 574773 [details]
mod_systemd

Description of problem:
I'm trying to use sd_notifyf in httpd to overcome issues with the PID file creation (see Bug #728465) and possibly start using some more advanced systemd features.

To achieve it, I'm trying to write mod_systemd httpd module (attached). It works like this:

1. It calls getpid() to get and store the PID in main httpd process. The pid_file is not written that time yet.
2. When the Apache forks and creates the children processes, it calls sd_notifyf with the PID of main process from the child process. The reason why to do it here is that pid_file is already written that time.

The problem is that if I run "httpd" command without systemd, I see the following in the httpd log meaning that it got to my systemd related code (please ignore logging debug messages as "error", it's only for my convenience):

[Mon Apr 02 15:16:05.678043 2012] [core:notice] [pid 24112:tid 140255251499008] SELinux policy enabled; httpd running as context unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[Mon Apr 02 15:16:05.679088 2012] [suexec:notice] [pid 24112:tid 140255251499008] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Apr 02 15:16:06.000579 2012] [ssl:notice] [pid 24112:tid 140255251499008] AH01886: SSL FIPS mode disabled
[Mon Apr 02 15:16:06.030958 2012] [:error] [pid 24112:tid 140255251499008] AH00521: init_module
[Mon Apr 02 15:16:06.051324 2012] [auth_digest:notice] [pid 24113:tid 140255251499008] AH01757: generating secret for digest authentication ...
[Mon Apr 02 15:16:07.000118 2012] [lbmethod_heartbeat:notice] [pid 24113:tid 140255251499008] AH02282: No slotmem from mod_heartmonitor
[Mon Apr 02 15:16:07.000315 2012] [ssl:notice] [pid 24113:tid 140255251499008] AH01886: SSL FIPS mode disabled
[Mon Apr 02 15:16:07.043374 2012] [:error] [pid 24113:tid 140255251499008] AH00521: systemd_post_config
[Mon Apr 02 15:16:07.043424 2012] [:error] [pid 24113:tid 140255251499008] AH00521: got pid 24113
[Mon Apr 02 15:16:07.045062 2012] [:error] [pid 24115:tid 140255251499008] AH00521: using pid 24113

But if I start it using systemd, it looks like httpd is stopped/killed before it even tries to call sd_notify:

[Mon Apr 02 15:22:33.448722 2012] [core:notice] [pid 24437:tid 139748586375168] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Mon Apr 02 15:22:33.449712 2012] [suexec:notice] [pid 24437:tid 139748586375168] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Apr 02 15:22:34.000216 2012] [ssl:notice] [pid 24437:tid 139748586375168] AH01886: SSL FIPS mode disabled
[Mon Apr 02 15:22:34.035757 2012] [:error] [pid 24437:tid 139748586375168] AH00521: init_module
[Mon Apr 02 15:22:34.055171 2012] [auth_digest:notice] [pid 24438:tid 139748586375168] AH01757: generating secret for digest authentication ...

Service file I currently use looks like this:
> [Unit]
> Description=The Apache HTTP Server
> After=syslog.target network.target remote-fs.target nss-lookup.target
> 
> [Service]
> Type=notify
> PIDFile=/var/run/httpd/httpd.pid
> EnvironmentFile=/etc/sysconfig/httpd
> ExecStart=/usr/sbin/httpd $OPTIONS
> ExecReload=/usr/sbin/httpd $OPTIONS -t
> ExecReload=/usr/sbin/httpd -HUP $MAINPID
> ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop
> PrivateTmp=true
> NotifyAccess=all
> 
> [Install]
> WantedBy=multi-user.target


Version-Release number of selected component (if applicable):
systemd 37-17.fc16

Steps to Reproduce:
1. Compile and install attached mod_systemd and enable it in httpd's configuration file.
2. Start httpd using systemd


Maybe it's caused by my misconfiguration or misunderstanding of the way how it's supposed to work. But I was trying to configure it differently for long time, so I think it's worth reporting.

Comment 1 Jan Kaluža 2012-09-06 09:09:17 UTC
Adding reproducer. I believe this should work with sd_notifyf, shouldn't it?

Comment 2 Jan Kaluža 2012-09-06 09:11:14 UTC
Created attachment 610232 [details]
simple reproducer

Comment 3 Jan Kaluža 2012-09-06 10:13:36 UTC
There is also variation of this reproducer showing different problem. If I keep parent process running and try to update STATUS from child process, it is not reflected in systemctl status output. I still see "Status: "Processing requests..." there.

From the documentation I would expect it will work even called from the child:
If the unset_environment parameter is non-zero sd_notify() will unset the $NOTIFY_SOCKET environment variable before returning (regardless whether the function call itself succeeded or not). Further calls to sd_notify() will then fail, but the variable is no longer inherited by child processes.

Comment 4 Jan Kaluža 2012-09-06 10:14:21 UTC
Created attachment 610248 [details]
reproducer for second problem

Comment 5 Jan Kaluža 2012-09-06 10:22:35 UTC
For the first reproducer I would expect that systemd will wait some time for sd_notify call and it will not care from which children process it comes. I thought that's exactly why sd_notify exists. Currently it kills children when main process terminates.

Comment 6 Lennart Poettering 2012-09-14 15:04:28 UTC
Hmm, you set Type=notify, but you actually have a forking process, right?

You need to set Type=forking in that case, otherwise systemd will assume your main process is dead and terminate you immediately. You also need to set NotifyAccess=all if it is not the original process that sends off the message

Comment 7 Jan Kaluža 2013-01-15 08:17:56 UTC
Hm, I think we could close this bug. I remember having discussion about this with someone from systemd community and find out the current systemd behaviour is good.

In httpd we've fixed it by using -DFOREGROUND httpd option, so httpd does not fork and Type=notify can be used.

Feel free to close this bug.

Comment 8 Lennart Poettering 2013-01-15 22:49:45 UTC
Cool, thanks!