Bug 1762624

Unable to reproduce this. Please attach the keepalived.conf file.

Can we confirm that the rpms are being upgraded, from the previous engineering comments, I couldn't locate this.
This issue only happens after upgrading the ipset libraries.


--> Before 

[root@localhost ~]# rpm -qa | egrep "ipset|keepalive|haproxy"
haproxy-1.5.18-9.el7.x86_64
keepalived-1.3.5-16.el7.x86_64
ipset-6.38-2.el7.x86_64     ---> Old Version
ipset-libs-6.38-2.el7.x86_64  ---> Old Version



---> After

This issue can be seen once we get to ipset-libs 7.1

(In reply to jnikolak from comment #5)
> 
> Can we confirm that the rpms are being upgraded, from the previous
> engineering comments, I couldn't locate this.
> This issue only happens after upgrading the ipset libraries.
> 
> 
> --> Before 
> 
> [root@localhost ~]# rpm -qa | egrep "ipset|keepalive|haproxy"
> haproxy-1.5.18-9.el7.x86_64
> keepalived-1.3.5-16.el7.x86_64
> ipset-6.38-2.el7.x86_64     ---> Old Version
> ipset-libs-6.38-2.el7.x86_64  ---> Old Version
> 
> 
> 
> ---> After
> 
> This issue can be seen once we get to ipset-libs 7.1

Yes, the ipset rpms are being upgraded. Please attach the config file.

I have a similar problem but with Red Hat 8.

Hi Cezar,

(In reply to cezar.fernandes from comment #15)
> I have a similar problem but with Red Hat 8.

Can you describe the problem in more detail? If you are using RHEL 8.0, it's
possible that you are hitting bz1708735.

- Red Hat Version:
 
Red Hat Enterprise Linux release 8.1 (Ootpa)
 
 
- Keepalived version:
 
keepalived-2.0.10-4.el8_0.2.x86_64
 
- Ipset version:
 
ipset-7.1-1.el8.x86_64
ipset-libs-7.1-1.el8.x86_64
 
 
 
- /var/log/messages
 
Jan 16 13:35:27 HOSTNAME Keepalived_vrrp[5535]: Unable to load ipset library - libipset.so.11: cannot open shared object file: No such file or directory
 
 
- find / -name libipset*
/usr/lib64/libipset.so.13
/usr/lib64/libipset.so.13.1.0
/usr/share/man/man3/libipset.3.gz

Hi Stefano,
We are getting the same issue in a clean Red Hat 7.7 installation. Steps to reproduce:

Process
yum install ipset.x86_64 ipset-libs.x86_64
yum install keepalived
systemctl enable keepalived
systemctl start keepalived 

Versions:
[root@~]# rpm -qa | grep ipset
ipset-devel-7.1-1.el7.x86_64
ipset-7.1-1.el7.x86_64
ipset-libs-7.1-1.el7.x86_64

[root@~]# rpm -qa | grep keepalived
keepalived-1.3.5-16.el7.x86_64
[root@~]#

find / -name libipset*
/usr/lib64/libipset.so.13
/usr/lib64/libipset.so.13.1.0
/usr/share/man/man3/libipset.3.gz
#


Log:

Keepalived_vrrp[14461]: Unable to load ipset library - libipset.so.11: cannot open shared object file: No such file or directory
Keepalived_vrrp[14461]: VRRP_Instance(VI_1) removing protocol VIPs.
Keepalived_vrrp[14461]: VRRP_Instance(VI_1) removing protocol iptable drop rule

Best regards,
Andrés Leal

Hi Stefano,
I have found a workaround. The problem comes with the vrrp_strict parameter. I commented it out and it's working.

Coould you provide any final solution?

Thanks and best regards,
Andrés Leal

Hi Cezar,

(In reply to cezar.fernandes from comment #18)
> Jan 16 13:35:27 HOSTNAME Keepalived_vrrp[5535]: Unable to load ipset library
> - libipset.so.11: cannot open shared object file: No such file or directory

Thanks for providing additional information. I filed bz1796541 and Cc'ed
you on it.

Hi Andrés,

(In reply to andres.leal from comment #20)
> Hi Stefano,
> I have found a workaround. The problem comes with the vrrp_strict parameter.
> I commented it out and it's working.

Thanks for providing this information. In general, yes, configuring
keepalived in any way that doesn't functionally require ipset
bindings will allow keepalived to start.

> Coould you provide any final solution?

At this stage, I think the way to go is to rebuild keepalived against
the current (libipset.so.13) version of libipset from RHEL 7.7, if
VRRP functionality is needed. You could rebuild the package from
source for the moment being.

Please mind that I can't commit to a particular solution or fix date
at this moment. If you need one, I'd suggest that you file a customer
case for this.

I did some poking around today and noticed that keepalived has two dlopen calls for libipset. Here is the code:

        /* Attempt to open the ipset library */
        if (!(libipset_handle = dlopen("libipset.so", RTLD_NOW)) &&
            !(libipset_handle = dlopen(IPSET_LIB_NAME, RTLD_NOW))) {
                log_message(LOG_INFO, "Unable to load ipset library - %s", dlerror());
                return false;
        }

The first call simply tries to open "lipipset.so". If that fails, it tries to open IPSET_LIB_NAME, which is set when the configure script is run (ie. at build time). So that explains why it is failing -- there is no "libipset.so" symlink on RHEL and the last build found libipset.so.11, which no longer exists.

I looked at RHEL7.7, RHEL7.6 and RHEL8.1 and none have the "lipipset.so" symlink. Fedora does:

#cat /etc/redhat-release 
Fedora release 31 (Thirty One)
# ls -al /usr/lib64/libipset.*
lrwxrwxrwx. 1 root root     18 Aug 19 05:18 /usr/lib64/libipset.so -> libipset.so.13.1.0
lrwxrwxrwx. 1 root root     18 Aug 19 05:18 /usr/lib64/libipset.so.13 -> libipset.so.13.1.0
-rwxr-xr-x. 1 root root 197984 Aug 19 05:18 /usr/lib64/libipset.so.13.1.0

Anyone know why? Keepalived would have no problem opening the shared object if this link existed. Just curious why we provide the link on Fedora but not RHEL.

We will need to fix this for 7.9, 7.8.z and 7.7.z. Setting this bug to rhel-7.9 so we can fix it there first, then clone z-stream bugs.

All in all, although you add the link to version 1.13 there are some 1.11 symbols that are being used by keepalived:

 Failed to dynamic link an ipset function - /lib64/libipset.so: undefined symbol: ipset_session_error

It's not a compilation issue, but a coding problem. I mean keepalived-1.3.5 has been coded using ipset 1.11 libraries.

(In reply to andres.leal from comment #27)
> All in all, although you add the link to version 1.13 there are some 1.11
> symbols that are being used by keepalived:
> 
>  Failed to dynamic link an ipset function - /lib64/libipset.so: undefined
> symbol: ipset_session_error
> 
> It's not a compilation issue, but a coding problem. I mean keepalived-1.3.5
> has been coded using ipset 1.11 libraries.

Interesting. Let me get this straight -- if you manually create a link named "libipset.so" that points to "libipset.so.13" you will get the undefined symbol? I've not yet gotten that far in my testing as the last bit of work I did was at the end of the day yesterday.

This makes me wonder if rebuilding keepalived against the latest ipset-libs is going to work at all. Will look into this shortly.

This should be fixed by rhbz#1818891.

RHEL 8 has similar problem for:
keepalived 2.0.10-4.el8_0.2
ipset 7.1-1.el8

...
Jun  6 01:17:54 prv-node33 Keepalived_vrrp[18124]: Registering gratuitous ARP shared channel
Jun  6 01:17:54 prv-node33 Keepalived_vrrp[18124]: Unable to load ipset library - libipset.so.11: cannot open shared object file: No such file or directory
...

The issue on Red Hat Enterprise Linux 8 was resolved in bug 1796541.  Please update Keepalived to keepalived-2.0.10-10.el8 per Bug Fix Advisory
RHBA-2020:1753: https://access.redhat.com/errata/RHBA-2020:1753

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (keepalived bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4015