This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1464869 - Configuration reload (SIGHUP) opens two (never closed) new pipes in subprocess
Configuration reload (SIGHUP) opens two (never closed) new pipes in subprocess
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: keepalived (Show other bugs)
7.3
x86_64 Linux
urgent Severity urgent
: rc
: ---
Assigned To: Ryan O'Hara
Brandon Perkins
: Regression, ZStream
Depends On:
Blocks: 1477907 1477883
  Show dependency treegraph
 
Reported: 2017-06-26 02:46 EDT by yann.morice
Modified: 2017-08-04 02:25 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1477883 1477907 (view as bug list)
Environment:
Last Closed: 2017-08-03 10:39:49 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description yann.morice 2017-06-26 02:46:57 EDT
Description of problem: 

At each live reload of configuration (<=> send of SIGHUP signal), the main keepalived subprocess opens two new pipes. These two pipes are never closed until maximum of open files reached and full restart of keepalived.

The problem is that this is widely used by openstack to live update configuration of routers (floating-ips, etc...) and do the router go off-line after too many open files...

The problem was introduced in Keepalived version 1.2.13-9.el7_3 to deal with bug #1429880. Previous version 1.2.13-8.el7 was OK.

Version-Release number of selected component (if applicable): 1.2.13-9.el7_3

Steps to Reproduce:

1. If we run a simple config using vrrp only :

# keepalived -P -f /etc/keepalived/keepalived.conf

vrrp_instance VR_1 {
    state BACKUP
    interface eth0
    virtual_router_id 1
    priority 50
    garp_master_delay 60
    nopreempt
    advert_int 2
    track_interface {
        eth0
    }
    virtual_ipaddress {
        169.254.0.2/24 dev eth0
    }
}

2. We have then two new processes : 

# ps -eafwww |grep keepalived
root     13722     1  0 09:00 ?        00:00:00 keepalived -P -f /etc/keepalived/keepalived-simple.conf
root     13723 13722  0 09:00 ?        00:00:00 keepalived -P -f /etc/keepalived/keepalived-simple.conf

3. First one has two pipes  :

# lsof |grep 13722|grep pipe|wc -l
2

4. Second one has four pipes at the beginning :
# lsof |grep 13723|grep pipe|wc -l
4

5. If we do SIGHUP to the main process to live update configuration :
# kill -HUP 13722

6. and do again :
# lsof |grep 13723|grep pipe|wc -l
6

Actual results:

We have now six pipes... (+2 pipes at each SIGHUP in fact)

Expected results:

With the version 1.2.13-8.el7, doing the same, the second process stay at only two pipes across all SIGHUPs.

Additional info: 

I suspect this to be caused by SIGHUP signal from configuration changes (add of floating ip) as a similar router with no config changes stay ok.

In fact, I think the bug was introduced is in lib/signals.c. I don't see any mechanism to replace signal_handler_destroy for closing these two pipes. This piece of code is still in the trunk (and latest versions) of keepalived. I think these four lines should not be deleted :

-       close(signal_pipe[1]);
-       close(signal_pipe[0]);
-       signal_pipe[1] = -1;
-       signal_pipe[0] = -1;

For information (but not directly linked), we never use the code in #ifdef HAVE_PIPE2 because set of variable in configure is not propagated in Makefiles*. In fact, if we do strings /usr/sbin/keepalived |grep pipe on installed keepalived, we got pipe (instead of pipe2). I suspect we should have something like DEFS	 = @DFLAGS@ -D@SNMP_SUPPORT@ @DEFS@ in lib/Makefile.in to do this.
Comment 2 Ryan O'Hara 2017-08-02 08:48:08 EDT
Since we rebased in 7.4, this bug only needs to be fixed in 7.3.z.
Comment 4 Ryan O'Hara 2017-08-02 09:30:26 EDT
Just to show that this problem does not exist in RHEL 7.4:

# cat /etc/redhat-release && rpm -q keepalived
Red Hat Enterprise Linux Server release 7.4 (Maipo)
keepalived-1.3.5-1.el7.x86_64

# keepalived -P -f /etc/keepalived/keepalived.conf
# ps -eafwww | grep keepalived
root     13321     1  0 08:26 ?        00:00:00 keepalived -P -f /etc/keepalived/keepalived.conf
root     13322 13321  0 08:26 ?        00:00:00 keepalived -P -f /etc/keepalived/keepalived.conf

# lsof | grep 13321 | grep pipe | wc -l
2
# lsof | grep 13322 | grep pipe | wc -l
2

Now SIGHUP the parent process:

# cat /var/run/keepalived.pid 
13321
# kill -HUP 13321

# lsof | grep 13321 | grep pipe | wc -l
2
# lsof | grep 13322 | grep pipe | wc -l
2

This is correct.
Comment 6 Oneata Mircea Teodor 2017-08-03 03:07:39 EDT
This bug has been copied as 7.3 z-stream (EUS) bug #1477883 and now must be
resolved in the current update release, set blocker flag.
Comment 8 Ryan O'Hara 2017-08-03 10:39:49 EDT
Setting as CLOSED/CURRENTRELEASE since the rebase in RHEL 7.4 solved this problem. Fixing only for z-stream.

Note You need to log in before you can comment on or make changes to this bug.