Bug 1478129 - rpc.idmapd fails to initialize libnfsidmap settings due to clashing conf file parsing routines
rpc.idmapd fails to initialize libnfsidmap settings due to clashing conf file...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: nfs-utils (Show other bugs)
25
All Linux
unspecified Severity high
: ---
: ---
Assigned To: Steve Dickson
Fedora Extras Quality Assurance
: Patch
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-03 12:05 EDT by Scott Mayhew
Modified: 2017-11-15 15:15 EST (History)
9 users (show)

See Also:
Fixed In Version: nfs-utils-2.1.1-6.rc6.fc26
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-11-15 15:15:09 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Scott Mayhew 2017-08-03 12:05:19 EDT
Description of problem:

Spun off from bug 1477231:

Close: the issue is in libnfsidmap actually. 

Note that rpc.idmapd is from nfs-utils (with patches carried by Fedora [0]). Upstream [1] defines conf_init to be `void conf_init(void)` (in support/nfs/conffile.c:373). However, Fedora ships a patch (nfs-utils-2.1.2-rc4.patch?) which updates this to be `void conf_init(const char *conf_file)` (in supporf/nfs/conffile.c:529 after patches), replacing the global variable conf_path with an explicit passed-in argument. This is what rpc.idmapd is built against, and contains guards to replace conf_path with a default if conf_path is NULL, (and further, utils/idmapd/idmapd.c contains code to set it appropriately as well, so it should never be NULL).

==HOWEVER==

Note that rpc.idmapd links against libnfsidamp:

[root@nfs-server usr]# ldd /usr/sbin/rpc.idmapd
        linux-vdso.so.1 (0x00007ffffab5a000)
        libevent-2.0.so.5 => /lib64/libevent-2.0.so.5 (0x00007fc8a80ae000)
        libnfsidmap.so.0 => /lib64/libnfsidmap.so.0 (0x00007fc8a7ea6000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fc8a7ad5000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc8a78b6000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fc8a76b2000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fc8a7497000)
        /lib64/ld-linux-x86-64.so.2 (0x0000561818709000)


In libnfsidmap, if you pull the RPM [2], it defines, in "cfg.c", a function conf_init which closely mirrors upstream's conf_init (an older version?) in that it uses conf_path and much of the same structure. However, this code has not been updated to reflect the new conf_init that rpc.idmapd is using. 

So, conf_path (the global) is NULL, and conf_init from the libnfsidmap package is being used:

# LD_DEBUG="bindings" ldd -r /usr/sbin/rpc.idmapd 2>&1 | grep 'conf_init'
       959:     binding file /lib64/libnfsidmap.so.0 [0] to /lib64/libnfsidmap.so.0 [0]: normal symbol `conf_init'
       959:     binding file /usr/sbin/rpc.idmapd [0] to /lib64/libnfsidmap.so.0 [0]: normal symbol `conf_init'


Hence, we get the error above, and rpc.idmapd crashes after first mount.


[0]: https://dl.fedoraproject.org/pub/fedora/linux/releases/26/Everything/source/tree/Packages/n/nfs-utils-2.1.1-5.rc4.fc26.src.rpm
[1]: https://www.kernel.org/pub/linux/utils/nfs-utils/2.1.1/nfs-utils-2.1.1.tar.xz
[2]: https://dl.fedoraproject.org/pub/fedora/linux/releases/26/Everything/source/tree/Packages/l/libnfsidmap-0.27-1.fc26.src.rpm



Version-Release number of selected component (if applicable):
libnfsidmap-0.27-1.fc26
Comment 1 Scott Mayhew 2017-08-03 12:11:38 EDT
nfs-utils and libnfsidmap originally had the same configuration file parsing routines.  The ones in nfs-utils changed in this commit:

commit 7b33590fee9515e49cda460285acc5d347a800b5
Author: Justin Mitchell <jumitche@rehat.com>
Date:   Fri May 5 11:03:20 2017 -0400

    config: Remove the conf_path global
    
    Working towards an nfs.conf library and API for system config tools,
    first step, replace the conf_path global with a parameter to conf_init
    
    Signed-off-by: Justin Mitchell <jumitche@rehat.com>
    Signed-off-by: Steve Dickson <steved@redhat.com>

which causes the breakage.  Eventually when Justin finishes the nfs.conf library we can just yank the conf file parsing functions from libnfsidmap and just have libnfsidmap use the functions from the nfs.conf library.  But for now we should just make sure the ones in libnfsidmap aren't externally visible.
Comment 2 Scott Mayhew 2017-08-03 12:17:01 EDT
Really the only visible functions from libnfsidmap should be the nfs4_* functions.  I added symbol versioning to configure.ac & Makefile.am to do that.

[root@fedora26 ~]# nm -D /lib64/libnfsidmap.so.0.3.0|grep ' T '
0000000000001dd0 T nfs4_cleanup_name_mapping
0000000000002680 T nfs4_get_default_domain
0000000000002960 T nfs4_gid_to_group_owner
00000000000027f0 T nfs4_gid_to_name
0000000000002d20 T nfs4_group_owner_to_gid
0000000000002eb0 T nfs4_gss_princ_to_grouplist
0000000000003140 T nfs4_gss_princ_to_grouplist_ex
0000000000002d70 T nfs4_gss_princ_to_ids
0000000000002ff0 T nfs4_gss_princ_to_ids_ex
0000000000001e80 T nfs4_init_name_mapping
0000000000002ac0 T nfs4_name_to_gid
00000000000029a0 T nfs4_name_to_uid
0000000000002cd0 T nfs4_owner_to_uid
0000000000003290 T nfs4_set_debug
00000000000026c0 T nfs4_uid_to_name
0000000000002920 T nfs4_uid_to_owner

However now I'm getting this error:

Aug  3 11:20:09 fedora26 rpc.idmapd: rpc.idmapd: libnfsidmap: Unable to load plugin: /lib64/libnfsidmap/nsswitch.so: undefined symbol: get_default_domain
Aug  3 11:20:09 fedora26 rpc.idmapd: rpc.idmapd: libnfsidmap: requested translation method, 'nsswitch', is not available
Aug  3 11:20:09 fedora26 rpc.idmapd: rpc.idmapd: Unable to create name to user id mappings.

So get_default_domain() and probably get_local_realms() need to be shuffled around a bit.
Comment 3 Scott Mayhew 2017-08-07 15:25:54 EDT
http://marc.info/?l=linux-nfs&m=150213294824944&w=2
Comment 4 Scott Mayhew 2017-08-07 15:28:16 EDT
Changing component to nfs-utils since the above patch addresses this in nfs-utils.  We can always open another bug if we want to hide libnfsidmap's config routines.
Comment 5 Anthony Messina 2017-09-01 22:10:17 EDT
If this is the same thing that causes "rpc.idmapd: conf_reinit: open ("(null)", O_RDONLY) failed", and since the patch is applied upstream, can we get an updated nfs-utils package built for F26?

rpc.idmapd[670]: rpc.idmapd: conf_reinit: open ("(null)", O_RDONLY) failed
rpc.idmapd[671]: main: event_dispatch returns errno 9 (Bad file descriptor)

Then a manual restart of nfs-idmapd.service is required.
Comment 6 RobbieTheK 2017-09-11 14:31:57 EDT
(In reply to Anthony Messina from comment #5)
> If this is the same thing that causes "rpc.idmapd: conf_reinit: open
> ("(null)", O_RDONLY) failed", and since the patch is applied upstream, can
> we get an updated nfs-utils package built for F26?
> 
> rpc.idmapd[670]: rpc.idmapd: conf_reinit: open ("(null)", O_RDONLY) failed
> rpc.idmapd[671]: main: event_dispatch returns errno 9 (Bad file descriptor)
> 
> Then a manual restart of nfs-idmapd.service is required.

Also getting this on our Fedora 26:
Sep 11 14:28:12 dsm systemd[1]: Starting NFSv4 ID-name mapping service...
Sep 11 14:28:12 dsm rpc.idmapd[25618]: rpc.idmapd: conf_reinit: open ("(null)", O_RDONLY) failed

restart does not help.

I see a few of these on the slave server:
Sep 11 14:02:02 mandelbrot ypxfr[29224]: ypproc_master_2: RPC: Unable to receive; errno = No route to host
Sep 11 14:02:02 mandelbrot ypxfr[29224]: ypxfr: YP server/map db error
Sep 11 14:02:02 mandelbrot ypserv[28128]: ypxfr_callback: RPC: Unable to receive; errno = No route to host
Sep 11 14:03:32 mandelbrot ypxfr[29288]: ypproc_master_2: RPC: Unable to receive; errno = No route to host
Sep 11 14:03:32 mandelbrot ypxfr[29288]: ypxfr: YP server/map db error
Sep 11 14:03:32 mandelbrot ypserv[28128]: ypxfr_callback: RPC: Unable to receive; errno = No route to host
Comment 7 Steve Dickson 2017-10-16 08:01:32 EDT
commit 8aea4a1c1da563437be5a6103ae459045e371f96
Author: Scott Mayhew <smayhew@redhat.com>
Date:   Wed Aug 23 14:55:20 2017 -0400

    idmapd: fix object order in idmapd_LDADD
Comment 8 Fedora Update System 2017-11-02 13:22:37 EDT
nfs-utils-2.1.1-6.rc6.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9e682ac8e2
Comment 9 Fedora Update System 2017-11-03 10:24:00 EDT
nfs-utils-2.1.1-6.rc6.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-9e682ac8e2
Comment 10 Fedora Update System 2017-11-15 15:15:09 EST
nfs-utils-2.1.1-6.rc6.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.