Bug 1267438

we are getting the same issue when we for reference tested using unixodbc to connect to our mariadb so the issue is not specific to the unixodbc->freetds connection.

freeradius-unixODBC.x86_64              3.0.4-6.el7
unixODBC.x86_64                         2.3.1-10.el7
mysql-connector-odbc.x86_64             5.2.5-6.el7

I tried manually build som version of the unixODBC and found something.
This seems to do the trick for this problem:
# ./configure  --with-included-ltdl

It might be an incompability between the installed libtool-ltdl packages and unixODBC....

Created attachment 1093399 [details]
a patch to the build specs that resolves to issue in this ticket

just a oneline patch to resolve the issue
see my previous comments

The package built fine with the proposed patch.

(In reply to Voja Molan from comment #0)
> Steps to Reproduce:
> 1. yum -y install freeradius-unixODBC
> 2. install freetds from EPEL for example
> 3. configure UnixODBC+FreeTDS

I don't know how to use FreeTDS, the configuration is important thing to have
a look at WRT this issue;  because then we can see what are the real reasons
unixODBC refuses to load the plugin /usr/lib64/libtdsodbc.so.0.

Note that compared to other unixODBC plugins, this is packaged differently
(versioned "library", rather than unversioned plugin).  Probably there is
yet another difference?

> 4. verify that connection to database works with command "isql"
> 5. configure FreeRADIUS to connect to MSSQL database using UnixODBC
> 6. on starting FreeRADIUS (`radiusd -X` for example), error when connecting
> to database:
> 
> rlm_sql_unixodbc: 01000 [unixODBC][Driver Manager]Can't open lib
> '/usr/lib64/libtdsodbc.so.0' : ??????: undefined symbol: get_vtable

Not necessarily unixODBC or especially in libtool-ltdl issue (no bug-report
against system version of ltdl so far).

It should be easy to tell what's wrong with linking if we had a reproducer.

(In reply to Göran Thyni from comment #3)
> I tried manually build som version of the unixODBC and found something.
> This seems to do the trick for this problem:
> # ./configure  --with-included-ltdl
> 
> It might be an incompability between the installed libtool-ltdl packages and
> unixODBC....

This is risky, we provide other ODBC plugins in RHEL which already work with
unixODBC correctly.  Changing (already very brittle) linking in unixODBC
could mean that other plugins will not work anymore.  If we really did
this, other plugins need to be carefully tested...  but that sounds wrong,
I prefer real fix.

Because other plugins work, looks like we should fix fredts, but to help you
with that we need reproducer.

Not reproducible by Tomáš nor by me on RHEL7.

I in particular got:

  # isql -v MSSQLTestServer
  [S1000][unixODBC][FreeTDS][SQL Server]Unable to connect to data source
  [08S01][unixODBC][FreeTDS][SQL Server]Unable to connect: Adaptive Server
  is unavailable or does not exist
  [ISQL]ERROR: Could not SQLConnect

Having this configuration (based on [1]) in odbcinst.ini:

  [FreeTDS]
  Description=v0.63 with protocol v8.0
  Driver=/usr/lib64/libtdsodbc.so.0
  UsageCount=1

And this in odbc.ini:

  [MSSQLTestServer]
  Driver    = FreeTDS
  Description = Northwind sample database
  Trace     = No
  Server    = 192.168.1.25
  Port      = 1433
  Database  = Northwind

The error is expected, because I don't have the Server on 192.168.1.25
running, but it looks the linking works fine.

  # rpm -q freetds unixODBC
  freetds-0.95.81-1.el7.x86_64
  unixODBC-2.3.1-11.el7.x86_64

The original report says:
    Can't open lib '/usr/lib64/libtdsodbc.so.0' : ??????: undefined symbol:
    get_vtable

But I checked that 'isql' correctly goes through this code path, without
dlopen errors.  So some linking issues might be on freeradius side?  But it
really looks unlikely that this is unixODBC issue (maybe some CentOS only
issue).

[1] http://www.unixodbc.org/doc/FreeTDS.html

Nikolai, could you help with reproducer?  Or possibly there's something really
obvious here.. dunno.

-> freeradius dlopens /usr/lib64/freeradius/rlm_sql_unixodbc.so
-> rlm_sql_unixodbc.so links against  libodbc.so.2
-> libodbc.so.2 dlopens /usr/lib64/libtdsodbc.so.0.0.0  (this is proven to work
   with isql, but doesn't work with freeradius)

What looks interesting is that 'ldd /usr/lib64/freeradius/rlm_sql_unixodbc.so'
says the freeradius module links against ltdl:

    ldd /usr/lib64/libtdsodbc.so.0.0.0  | grep dl
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fb7fc841000)
    libltdl.so.7 => /lib64/libltdl.so.7 (0x00007fb7fbe52000)

But I fail to see lbitool-ltdl-devel in freeradius.spec.  I'm not sure how
that's possible.  I also see that there is configure option:

   --with-system-libltdl

But that's not used.  Is there a reason not to use system's ltdl, or is it
used due to some magic?

Typo in comment #27, I should write:
  $ ldd /usr/lib64/freeradius/rlm_sql_unixodbc.so | grep dl
  libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x00007f1b7bedf000)
  libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f1b7bcdb000)

Pavel, one thing first: FreeRADIUS doesn't require libltdl, it's libodbc:

[nkondras@localhost ~]$ readelf -d /usr/lib64/freeradius/rlm_sql_unixodbc.so |grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libodbc.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
[nkondras@localhost ~]$ readelf -d /usr/lib64/libodbc.so.2|grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libltdl.so.7]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

There is a problem when linking all of this with the radiusd:

[nkondras@localhost raddb]$ LD_PRELOAD=/usr/lib64/freeradius/rlm_sql_unixodbc.so:/usr/sbin/radiusd isql -v MSSQLTestServer
[01000][unixODBC][Driver Manager]Can't open lib '/usr/lib64/libtdsodbc.so.0' : ����: undefined symbol: get_vtable
[ISQL]ERROR: Could not SQLConnect

Continuing investigation.

In fact, just the radiusd binary is enough to trigger it:

[nkondras@localhost raddb]$ LD_PRELOAD=/usr/sbin/radiusd isql -v MSSQLTestServer
[01000][unixODBC][Driver Manager]Can't open lib '/usr/lib64/libtdsodbc.so.0' : ����: undefined symbol: get_vtable
[ISQL]ERROR: Could not SQLConnect

It seems there's some garbage in the output above. Valgrind notices it as well:

[nkondras@localhost raddb]$ LD_PRELOAD=/usr/sbin/radiusd valgrind isql -v MSSQLTestServer
==22362== Memcheck, a memory error detector
==22362== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==22362== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==22362== Command: isql -v MSSQLTestServer
==22362== 
==22362== Invalid read of size 8
==22362==    at 0x4009C30: _dl_lookup_symbol_x (in /usr/lib64/ld-2.17.so)
==22362==    by 0x5C9E378: do_sym (in /usr/lib64/libc-2.17.so)
==22362==    by 0x574D0D3: dlsym_doit (in /usr/lib64/libdl-2.17.so)
==22362==    by 0x400EFF3: _dl_catch_error (in /usr/lib64/ld-2.17.so)
==22362==    by 0x574D5BC: _dlerror_run (in /usr/lib64/libdl-2.17.so)
==22362==    by 0x574D127: dlsym (in /usr/lib64/libdl-2.17.so)
==22362==    by 0x5301B0F: ??? (in /usr/lib64/libltdl.so.7.3.0)
==22362==    by 0x52FE966: lt_dlpreload_open (in /usr/lib64/libltdl.so.7.3.0)
==22362==    by 0x50A0BC0: ??? (in /usr/lib64/libodbc.so.2.0.0)
==22362==    by 0x50A3AF6: SQLConnect (in /usr/lib64/libodbc.so.2.0.0)
==22362==    by 0x4028A8: ??? (in /usr/bin/isql)
==22362==    by 0x5B8DB34: (below main) (in /usr/lib64/libc-2.17.so)
==22362==  Address 0x91c6b38 is 568 bytes inside an unallocated block of size 3,348,192 in arena "client"
==22362== 
[01000][unixODBC][Driver Manager]Can't open lib '/usr/lib64/libtdsodbc.so.0' : �'0: undefined symbol: get_vtable
[ISQL]ERROR: Could not SQLConnect
==22362== 
==22362== HEAP SUMMARY:
==22362==     in use at exit: 86,429 bytes in 136 blocks
==22362==   total heap usage: 653 allocs, 517 frees, 813,281 bytes allocated
==22362== 
==22362== LEAK SUMMARY:
==22362==    definitely lost: 256 bytes in 2 blocks
==22362==    indirectly lost: 65,696 bytes in 4 blocks
==22362==      possibly lost: 0 bytes in 0 blocks
==22362==    still reachable: 20,477 bytes in 130 blocks
==22362==         suppressed: 0 bytes in 0 blocks
==22362== Rerun with --leak-check=full to see details of leaked memory
==22362== 
==22362== For counts of detected and suppressed errors, rerun with: -v
==22362== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Also nothing actually looks up the "get_vtable" symbol, and there is no such symbol defined anywhere in the installed libraries. There are only 'dlopen_LTX_get_vtable' and 'preopen_LTX_get_vtable', which are resolved fine.

The debug output from ld shows the error occurring in the middle of looking up other symbols:

[nkondras@localhost raddb]$ LD_DEBUG=all LD_PRELOAD=/usr/sbin/radiusd isql -v MSSQLTestServer |& grep -C5 'get_vtable (fatal)'
     22372:     symbol=_dl_sym;  lookup in file=/lib64/libreadline.so.6 [0]
     22372:     symbol=_dl_sym;  lookup in file=/lib64/libdl.so.2 [0]
     22372:     symbol=_dl_sym;  lookup in file=/lib64/libpthread.so.0 [0]
     22372:     symbol=_dl_sym;  lookup in file=/lib64/libc.so.6 [0]
     22372:     binding file /lib64/libdl.so.2 [0] to /lib64/libc.so.6 [0]: normal symbol `_dl_sym' [GLIBC_PRIVATE]
     22372:     �g*�: error: symbol lookup error: undefined symbol: get_vtable (fatal)
     22372:     symbol=lt__error_string;  lookup in file=isql [0]
     22372:     symbol=lt__error_string;  lookup in file=/usr/sbin/radiusd [0]
     22372:     symbol=lt__error_string;  lookup in file=/lib64/libodbc.so.2 [0]
     22372:     symbol=lt__error_string;  lookup in file=/lib64/libltdl.so.7 [0]
     22372:     binding file /lib64/libltdl.so.7 [0] to /lib64/libltdl.so.7 [0]: normal symbol `lt__error_string'

The garbage is also output in the debug output, and it is output in the place of the shared library name.

So something seems really messed up with the linker.

Here is a backtrace which is trying to lookup the "get_vtable" symbol:

#0  _dl_lookup_symbol_x (undef_name=0x7ffff750d1ba "get_vtable", undef_map=0x612f10, ref=0x7fffffffc650, symbol_scope=0x613298, version=0x0, type_class=0, flags=2, skip_map=0x0) at dl-lookup.c:716
#1  0x00007ffff6c11379 in do_sym (handle=0x612f10, name=0x7ffff750d1ba "get_vtable", who=0x7ffff750bb10 <loader_init_callback+16>, vers=vers@entry=0x0, flags=flags@entry=2) at dl-sym.c:178
#2  0x00007ffff6c1186d in _dl_sym (handle=<optimized out>, name=<optimized out>, who=<optimized out>) at dl-sym.c:283
#3  0x00007ffff70bd0d4 in dlsym_doit (a=a@entry=0x7fffffffc850) at dlsym.c:50
#4  0x00007ffff7debff4 in _dl_catch_error (objname=0x613060, errstring=0x613068, mallocedp=0x613058, operate=0x7ffff70bd0c0 <dlsym_doit>, args=0x7fffffffc850) at dl-error.c:177
#5  0x00007ffff70bd5bd in _dlerror_run (operate=operate@entry=0x7ffff70bd0c0 <dlsym_doit>, args=args@entry=0x7fffffffc850) at dlerror.c:163
#6  0x00007ffff70bd128 in __dlsym (handle=<optimized out>, name=<optimized out>) at dlsym.c:70
#7  0x00007ffff750bb10 in loader_init_callback (handle=<optimized out>) at libltdl/ltdl.c:180
#8  0x00007ffff7508967 in lt_dlpreload_open (originator=originator@entry=0x7ffff750cd0a "libltdl", func=func@entry=0x7ffff750bb00 <loader_init_callback>) at libltdl/loaders/preopen.c:361
#9  0x00007ffff750a0f7 in lt_dlinit () at libltdl/ltdl.c:252
#10 0x00007ffff771cbc1 in __connect_part_one (connection=connection@entry=0x60f2b0, driver_lib=driver_lib@entry=0x7fffffffdc30 "/usr/lib64/libtdsodbc.so.0", driver_name=driver_name@entry=0x7fffffffe020 "FreeTDS", warnings=warnings@entry=0x7fffffffd2e8) at SQLConnect.c:1092
#11 0x00007ffff771faf7 in SQLConnect (connection_handle=0x60f2b0, server_name=server_name@entry=0x7fffffffe881 "MSSQLTestServer", name_length1=name_length1@entry=-3, user_name=user_name@entry=0x0, name_length2=name_length2@entry=-3, authentication=authentication@entry=0x0, name_length3=name_length3@entry=-3)
    at SQLConnect.c:3932
#12 0x00000000004028a9 in OpenDatabase (phEnv=0x607258 <hEnv>, phDbc=0x607250 <hDbc>, szPWD=0x0, szUID=0x0, szDSN=0x7fffffffe881 "MSSQLTestServer") at isql.c:556
#13 main (argc=<optimized out>, argv=<optimized out>) at isql.c:177

The undef_map argument to dl_lookup_symbol_x seems to be messed up, at least the pointers inside it don't seem to make sense and the l_name field points to garbage, which is printed in the error message later.

I ran out of time and energy trying to figure this out and will leave this for you now to try, since you have a reproducer now.

Thanks a lot for having a look, Nikolai!

(In reply to Nikolai Kondrashov from comment #29)
> Pavel, one thing first: FreeRADIUS doesn't require libltdl, it's libodbc:

Right, ldd shows transitive dpes too.  Readelf -d is the right tool.

Based on ./configure --help output, it however looks that (some part of)
the code actually depends on libltdl, but for some reason radius package
"bundles" libltld instead of using the system one.  I'll check this
hypothesis once I'm in the office (that requires some gdb dance), but that
would mean we have (at least) two ltdl libraries (one static bundled in
freeradius and one dynamic loaded as transitive dependency).  Such likning
issues could lead to nasty runtime bugs.

> Also nothing actually looks up the "get_vtable" symbol, and there is no such
> symbol defined anywhere in the installed libraries. There are only
> 'dlopen_LTX_get_vtable' and 'preopen_LTX_get_vtable', which are resolved
> fine.

I think 'get_vtable' is libtool's internal symbol.  We now have clear
reproducer, so we should be able to tell what's wrong easily.  Thanks again!

(In reply to Pavel Raiskup from comment #35)
> for some reason radius package "bundles" libltld instead of using the
> system one

That's not truth, sorry.  Looks like freeradius project switched from libtool
ltdl implementation into "re-implementing" approach.  I haven't checked for
details, neither I know the motivation behind.  But at least it looks like
_at least_ those symbols shouldn't be exported:

 # readelf -s /usr/sbin/radiusd  | grep lt_dl
 532: 0000000000021a90     5 FUNC    GLOBAL DEFAULT   13 lt_dlerror
 569: 0000000000021a60     5 FUNC    GLOBAL DEFAULT   13 lt_dlsym
 606: 0000000000021a70    19 FUNC    GLOBAL DEFAULT   13 lt_dlclose
 627: 0000000000021810   590 FUNC    GLOBAL DEFAULT   13 lt_dlopenext

The problem is that, e.g. in Nikolai's backtrace (frame #7),
loader_init_callback calls 'lt_dlsym' function ... but instead of calling
"real" libtool's lt_dlsym function, unixODBC mistakenly calls __dlsym()
function (glibc) directly.  That's probably because freeradius defines
lt_dlsym as a thin wrapper around dlsym() call -- and that's optimized into
direct __dlsym() call.  Which means 'lt_dlsym' is "hijacked" by freeradius.

Nikolai, it sounds like we should fix freeradius to not collide with commonly
used library libltdl.so.*.  Could we rename the symbols (to something like
fr_dl* == FreeRadius), or could we somehow hide the symbols for linker?
Ideally in upstream?  I'm open to other ideas.

Thank you for your research, Pavel. Yes, I expected some trickery from FreeRADIUS. I'll look into it this week.

Asked FreeRADIUS upstream about this: https://github.com/FreeRADIUS/freeradius-server/issues/1277#issuecomment-258118716

FreeRADIUS upstream already fixed this on github, thanks for such quick
turn-around Nikolai and upstream!

Turns out this is not bug in unixODBC, and the attached fix thus won't be
applied (and updated unixODBC won't be shipped).  The reason is that
such update could lead to other linking issues in future;  we also should
in general always avoid bundling ltdl into RHEL/Fedora packages
(libtool upstream is kind of thinking about removing the "bundling" ltdl
functionality completely).

I'm going to clone this bug against FreeRADIUS project, and close this
NOTABUG.