Bug 700115

DB2 should be compiled to load a specific ABI version of libdat2, so that it can fail to start if that ABI is not available, rather than randomly crashing during execution because the ABI exported by libdat2.so no longer matches what DB2 was compiled against.

The automatic dependency mechanism of rpm should pull in whatever version of dapl or compat-dapl provides libdat2.so.2, making it difficult to install a broken DB2 configuration.

In the future, the ABI of libdat2 may change, in which case libdat2.so.2 will be moved to compat-dapl, and dapl will contain libdat2.so.3.  dapl-devel will contain a libdat2.so that is a symlink to libdat2.so.3.  Any program that attempts to dynamically load libdat2.so and expects libdat2.so.2 semantics will crash as soon as it attempts to use those semantics on a symbol whose ABI changed in libdat2.so.3

------- Comment From mladen.com 2011-05-05 09:21 EDT-------
We would like clarification as to whether this is truly working as designed
(ie. not a bug) or not.

If we understood correctly, you are encouraging DB2 to rely on libdat2.so.2
instead?

Although it *may* be the correct thing to do from DB2's side, on the other hand
both RHEL 5.5 and SLES 11.1 have the libdat2.so symlink - both releases having
symlink libdat2.so pointing to libdat2.so.2.0.0.

To us, its an obvious change from RedHat's RHEL5.5 model - why is it different
now in RHEL6.1?  Was this a concrete, strategic decision made to go this route?  Are there other packages that exhibit similar behaviour?

Otherwise it looks like a regression to us.

This also does not seem like a good trend from a usability point of view.

1. Every new Redhat release that has this lib updated, applications like DB2
will need more effort and take longer to support.
2. Customers cannot upgrade redhat and have old applications running without
performing manual installation of rpms containing this lib from the old redhat
images, which is not an obvious and easy step.  libstdc++.so.5 requirement is a
similar example.

Its best if we could have this for RHEL 6.1, but if that doesn't happen (and
looks like we're quite late in the cycle), at least if we can commitment to fix
this in RHEL 6.2 timeframe would help.

Can we have the mirrored bugzilla defect re-opened please?

Re-opening to await a response to the questions raised.

(In reply to comment #3)
> ------- Comment From mladen.com 2011-05-05 09:21 EDT-------
> We would like clarification as to whether this is truly working as designed
> (ie. not a bug) or not.
> 
> If we understood correctly, you are encouraging DB2 to rely on libdat2.so.2
> instead?
> 
> Although it *may* be the correct thing to do from DB2's side, on the other hand
> both RHEL 5.5 and SLES 11.1 have the libdat2.so symlink - both releases having
> symlink libdat2.so pointing to libdat2.so.2.0.0.

I can't speak for SLES, but both RHEL5 and RHEL6 include the libdat2.so symlink only in the -devel package.  This is intentional and correct.  And if SLES includes the libdat2.so symlink in the base package and not in their devel package then it is certainly a bug.  There is a very solid reason why this is the case.  The version numbers on libraries are used to denote compatibility, with it being standard that a minor point update is backwards compatible, while a major point update is not.  In this case, the 2 in libdat2 has nothing to do with the library version and merely specifies that the library implements the DAT 2.0 specification.  The library version, therefore, is not indicated *at all* in the libdat2.so file name.  You must include at least the first number of the library version to catch the library's major version number (aka, libdat2.so.2 is the minimum necessary in this case).

Now, the reason that the .so symlink is only included in the -devel package is specifically because it *doesn't* include the version number, and that's what you want when you are compiling your program.  You want compilation to use the latest installed library, this is accomplished by having a generic symlink from a non-versioned library name to the latest library, and by including the symlink in the -devel package you insure that the library itself and the include headers the application is using to build against are always 100% in sync.

The fact that you are using dlopen() to open the library instead of linking against the library when compiling means that you are skipping the version management that the dynamic linker performs for you.  When you do that, you are responsible for performing your own version management.  Using dlopen() instead of linking against libdat2.so when compiling also means you are circumventing the automated rpm dependency creation as well, so you will need to manually install dapl on target machines in order to use it, rpm will not automatically bring it in for you.

The fact that you use dlopen() without performing any version management is a serious bug in your application and in the event that we upgrade libdat2.so from version 2 to 3, your application will most likely crash both immediately and spectacularly.  It will require a recompile of your application or a downgrade of the dapl2 rpms in order to fix the problem.  It will not be possible, as it is normally done, for us to include a compat-dapl2-2 package that allows your application to continue working without a recompile and with the later libraries installed.

> To us, its an obvious change from RedHat's RHEL5.5 model - why is it different
> now in RHEL6.1?

It is *not* different.  You are mistaken.  If you log into a rhel5 machine and run the command:

rpm -qf /usr/lib*/libdat2.so

you will see that the files are owned by the -devel package on rhel5 just like they are on rhel6.  The only difference between the two versions is that under rhel5 the -devel package of dapl is installed by default and under rhel6 it is not.

>  Was this a concrete, strategic decision made to go this route?
>  Are there other packages that exhibit similar behaviour?

Yes.  *EVERY* package on rhel6 does this.  Any time we find a .so symlink not in a -devel package, we fix it.

> Otherwise it looks like a regression to us.
> 
> This also does not seem like a good trend from a usability point of view.
> 
> 1. Every new Redhat release that has this lib updated, applications like DB2
> will need more effort and take longer to support.

This is simply not true.  The fact that you dlopen() the library instead of simply linking against the library keeps both rpm and the ld linker from doing the version management for you automatically like they do for everything else.  

> 2. Customers cannot upgrade redhat and have old applications running without
> performing manual installation of rpms containing this lib from the old redhat
> images, which is not an obvious and easy step.  libstdc++.so.5 requirement is a
> similar example.

Again, this only happens if rpm and the ld linker is being circumvented.

The normal rpm build process goes something like this:

Your application is built, and in the process of being built you link against whatever shared libraries you need.

In the packaging process, rpm runs ldd on your application and records all of the shared libraries necessary for your application to link properly, their version, their arch, etc.  It records this information in the rpm header for the package.

During install of the package, yum and rpm read the headers, and check to make sure all the necessary libraries are installed on the system, including they check for matching versions, arch, etc.  If any are missing, then they query the rhn server for a package that provides the necessary library, and add it to the rpm install transaction as a dependency.

Once installed, any attempt to upgrade a library that the package depends on to a later version will fail if there is not another package in the same upgrade that also provides a backwards compatible library upgrade.  However, yum can only perform this check on libraries listed in the rpm header as requirements for the package.

When you dlopen() the library instead of linking against it, the library does not show up in the output of ldd.  As a result, all of the processing that happens via rpm and yum to make sure that you have the proper libraries installed is never performed on those dlopen()ed libraries.  Therefore your application must perform that work itself.  And all the problems you speak of are exactly what you get when you use dlopen() and circumvent the normal rpm/yum/ld.so library and dependency tracking built into the OS without doing the required work of tracking dependencies in your dlopen()ed libraries.

> Its best if we could have this for RHEL 6.1, but if that doesn't happen (and
> looks like we're quite late in the cycle), at least if we can commitment to fix
> this in RHEL 6.2 timeframe would help.
> 
> Can we have the mirrored bugzilla defect re-opened please?

------- Comment From mladen.com 2011-05-05 23:40 EDT-------
Thank you for that terrific, detailed response.  Let me take that back to the team - please keep this ticket open another day or two for us to review and decide our next steps.  This was very helpful!

------- Comment From mladen.com 2011-05-06 15:51 EDT-------
Hi,

We've been discussing this issue in house trying to see what we should be doing on our end - again, thanks for the explanations.

However, you mentioned we should install the dapl-devel rpm - this seems to be missing in the RHEL 6 repository.  We could only see these:

compat-dapl-1.2.15-2.2.el6.x86_64.rpm
dapl-2.0.25-5.2.el6.x86_64.rpm

Further,

On RHEL 5.5, we have this:

[root@coralm225 lib64]# yum groupinfo 'OpenFabrics Enterprise Distribution'
Loaded plugins: rhnplugin, security
Setting up Group Process

Group: OpenFabrics Enterprise Distribution
Description: Components used for high performance networking and clustering, such as Infiniband and RDMA.
Default Packages:
compat-dapl
compat-dapl-devel
compat-dapl-utils
dapl
dapl-devel ===> dapl-devel is installed in case of RHEL5.5 , but seen the Infiniband Support package in case of RHEL 6.0 doesn't have this
dapl-utils
... more ...

On RHEL 6.1, we have this:
yum groupinfo 'Infiniband Support'
Loaded plugins: product-id, refresh-packagekit, rhnplugin, subscription-manager
Updating Red Hat repositories.
Setting up Group Process

Group: Infiniband Support
Description: Software designed for supporting clustering and grid connectivity
using RDMA-based InfiniBand and iWARP fabrics.
Mandatory Packages:
libibcm
libibverbs
libibverbs-utils
librdmacm
librdmacm-utils
rdma
Default Packages:
dapl
ibsim
ibutils
libcxgb3
libibmad
libibumad
libipathverbs
libmlx4
libmthca
rds-tools
Optional Packages:
compat-dapl
infiniband-diags
libibcommon
mstflint
opensm
perftest
qperf
srptools

dapl and compat-dapl are in this list.  No mention of any dapl-devel rpms - can you let us know where the dapl-devel rpm can be found and if it will be available on default installations?

Devel packages are for software development only and are not appropriate for installation on client or server machines.

------- Comment From mladen.com 2011-05-06 17:01 EDT-------
Although we agree the devel packages shouldn't typically be installed, while we're still testing it helps to get our hands on the devel package temporarily.  We can't seem to find it.

In the list of available packages, there are many devel packages - just wondering where this one might be located.

For example:
GConf2-devel-2.28.0-6.el6.x86_64.rpm
ORBit2-devel-2.14.17-3.1.el6.x86_64.rpm
PyQt4-devel-4.6.2-8.el6.x86_64.rpm
...
and many others.

Thanks!

One of the changes between rhel5 and rhel6 is that the actual repo that we put software packages into in the rhn backend changed.  There used to only be a single backend repo in rhel5 (for the base product anyway, there were always additional repos for additional channels) and now there are multiple backend repos in rhel6 just for the base product.  Admittedly, I'm not familiar with them since I always grab packages internally and the repos are irrelevant to me.  But, IIRC, the Optional software repo is where what you are looking for is probably at.

------- Comment From mladen.com 2011-06-06 11:31 EDT-------
We have made DB2 completely dependent on libdat2.so.2 instead of libdat2.so as per the recommendations made in this bugzilla thread.

Please close this ticket. Thanks.