Bug 1126955 - Reloading haproxy fails frequently
Summary: Reloading haproxy fails frequently
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: haproxy
Version: 20
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Ryan O'Hara
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-05 17:25 UTC by Ben Nemec
Modified: 2014-08-19 07:08 UTC (History)
4 users (show)

Fixed In Version: haproxy-1.5.3-2.fc20
Clone Of:
Environment:
Last Closed: 2014-08-19 07:08:19 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1352336 0 None None None Never

Description Ben Nemec 2014-08-05 17:25:26 UTC
Description of problem: Running "systemctl reload haproxy" while haproxy is running causes haproxy to fail on a pretty regular basis.


Version-Release number of selected component (if applicable): 1.5.2-1.fc20


How reproducible: Not always, but often


Steps to Reproduce:
1. yum install haproxy
2. systemctl start haproxy
3. systemctl reload haproxy

Actual results: The haproxy service dies


Expected results: The haproxy service gracefully restarts with a new configuration, handing off any existing connections to the new process.


Additional info: This was initially noticed in the linked external bug.

Example of what I'm seeing on a basic Fedora VM:

[root@test system]# systemctl restart haproxy
[root@test system]# systemctl status haproxy
haproxy.service - HAProxy For TCP And HTTP Based Applications
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
   Active: active (running) since Tue 2014-08-05 17:11:00 UTC; 3s ago
  Process: 838 ExecReload=/bin/bash -c exec /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf $MAINPID (code=exited, status=0/SUCCESS)
  Process: 846 ExecStart=/usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid (code=exited, status=0/SUCCESS)
  Process: 844 ExecStartPre=/usr/sbin/haproxy -c -q -f /etc/haproxy/haproxy.cfg (code=exited, status=0/SUCCESS)
 Main PID: 847 (haproxy)
   CGroup: /system.slice/haproxy.service
           └─847 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /run/hap...

Aug 05 17:11:00 test systemd[1]: Started HAProxy For TCP And HTTP Based App...s.
Hint: Some lines were ellipsized, use -l to show in full.
[root@test system]# systemctl reload haproxy
[root@test system]# systemctl status haproxy
haproxy.service - HAProxy For TCP And HTTP Based Applications
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
   Active: failed (Result: signal) since Tue 2014-08-05 17:11:11 UTC; 7s ago
  Process: 852 ExecReload=/bin/bash -c exec /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf $MAINPID (code=exited, status=0/SUCCESS)
  Process: 846 ExecStart=/usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid (code=exited, status=0/SUCCESS)
  Process: 844 ExecStartPre=/usr/sbin/haproxy -c -q -f /etc/haproxy/haproxy.cfg (code=exited, status=0/SUCCESS)
 Main PID: 853 (code=killed, signal=KILL)

Aug 05 17:11:00 test systemd[1]: Started HAProxy For TCP And HTTP Based App...s.
Aug 05 17:11:11 test systemd[1]: Reloading HAProxy For TCP And HTTP Based A...s.
Aug 05 17:11:11 test systemd[1]: Reloaded HAProxy For TCP And HTTP Based Ap...s.
Aug 05 17:11:11 test systemd[1]: haproxy.service: main process exited, code...LL
Aug 05 17:11:11 test systemd[1]: Unit haproxy.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.
[root@test system]#


haproxy log from that time:
Aug 05 17:10:30 test systemd[1]: Starting HAProxy For TCP And HTTP Based Applications...
Aug 05 17:10:30 test systemd[1]: Started HAProxy For TCP And HTTP Based Applications.                                                                                                                                                        
Aug 05 17:10:32 test systemd[1]: Reloading HAProxy For TCP And HTTP Based Applications.                                                                                                                                                      
Aug 05 17:10:32 test systemd[1]: Reloaded HAProxy For TCP And HTTP Based Applications.                                                                                                                                                       
Aug 05 17:10:32 test systemd[1]: haproxy.service: main process exited, code=killed, status=9/KILL                                                                                                                                            
Aug 05 17:10:32 test systemd[1]: Unit haproxy.service entered failed state.

Comment 1 Ben Nemec 2014-08-05 17:50:52 UTC
A relevant discussion from the haproxy list: http://www.serverphorums.com/read.php?10,649689

We probably need to make use of the haproxy-systemd-wrapper to properly implement reload in the service file.  Note the example service file in the change:

@@ -0,0 +1,11 @@
+[Unit]
+Description=HAProxy Load Balancer
+After=network.target
+
+[Service]
+ExecStart=@SBINDIR@/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
+ExecReload=/usr/bin/kill -USR2 $MAINPID
+Restart=always
+
+[Install]
+WantedBy=multi-user.target

Comment 3 Fedora Update System 2014-08-06 19:05:00 UTC
haproxy-1.5.3-2.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/haproxy-1.5.3-2.fc20

Comment 4 Fedora Update System 2014-08-07 15:32:35 UTC
Package haproxy-1.5.3-2.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing haproxy-1.5.3-2.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-9215/haproxy-1.5.3-2.fc20
then log in and leave karma (feedback).

Comment 5 Fedora Update System 2014-08-19 07:08:19 UTC
haproxy-1.5.3-2.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.