Bug 1929936
| Summary: | "stack smashing detected" in relation to libnss_systemd.so | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Jan Pokorný [poki] <fedora> |
| Component: | glibc | Assignee: | Carlos O'Donell <codonell> |
| Status: | CLOSED UPSTREAM | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | rawhide | CC: | aoliva, arjun, codonell, dj, fedoraproject, filbranden, flepied, fweimer, gsuckevi, kasong, law, lnykryn, mcermak, mfabian, msekleta, pfrankli, rth, security-response-team, sipoyare, ssahani, s, systemd-maint, yuwatana, zbyszek, z |
| Target Milestone: | --- | Keywords: | Security |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-03-16 13:49:57 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
re [comment 0]: > (- also of interest, but off-topic here: what a funky move trying to > resolve standard wildcard '*' as a user, maybe for internal defense, > but nonetheless) Ah, with bash(1) dusted off, I see what's going on here ... eventually pointing to the dnf completion script bug that was lurking 6 years without anyone noticing: https://github.com/rpm-software-management/dnf/pull/1730 So if nothing else, at least something positive born in this accident. Thank you for the beautiful bug report ;) I don't see anything wrong in the code of nss-systemd. I added a test to invoke those entry points directly: https://github.com/systemd/systemd/pull/18982. I was suspecting that we do something wrong with the scratch buffer sizing, but this doesn't seem to be the case. When compiled with address sanitizer, for i in {0..100}; do SYSTEMD_TEST_NSS_BUFSIZE=$i build/test-nss-users systemd 64591;done doesn't yield any invalid access. (64591 is a dynamic user). But I see something suspicious in here: > #5 0x00007f790e78de88 in pthread_getname_np (th=2, buf=0x7fffe34db6d0 "", len=0) at pthread_getname.c:38 pthread_getname_np (pthread_t th, char *buf, size_t len) { ... int fd = __open64_nocancel (fname, O_RDONLY); if (fd == -1) return errno; int res = 0; ssize_t n = TEMP_FAILURE_RETRY (__read_nocancel (fd, buf, len)); if (n < 0) res = errno; else { if (buf[n - 1] == '\n') buf[n - 1] = '\0'; else if (n == len) res = ERANGE; else buf[n] = '\0'; } ... } I'm not sure how __read_nocancel works exactly, but at least with normal read this would be a problem. Maybe __read_nocancel has different rules… > In the absence of any errors, or if read() does not check for errors, a read() with a count of > 0 returns zero and has no other effects. buf[n-1] and buf[0] are both illegal accesses. I'll reassign this to glibc for investigation. This is a result of an internal GLIBC_PRIVATE change: loading the new libpthread into an old installed glibc results in crashes because of the ABI mismatch. This is one reasons why I have been advising NSS module authors not to link against libpthread. I know I have not been successful with that, so I'm working on removal of libpthread from upstream glibc: https://sourceware.org/pipermail/libc-alpha/2021-March/123674.html Found some more relevant stuff, not that I'd be any much wiser :-) https://bugzilla.redhat.com/show_bug.cgi?id=1369130 + https://github.com/SSSD/sssd/issues/4189 + https://github.com/SSSD/sssd/commit/d2f93542650c2f9613043acfa8e2f368972a70cd https://sourceware.org/bugzilla/show_bug.cgi?id=20500#c3 (please, lift the security flag if/when no longer needed) As mentioned in comment 3, this is a known issue around glibc updates and delayed loading of libpthread. We are tackling this upstream by removing libpthread as a separate shared object. This work is currently targeted at glibc 2.34 (Fedora 35). Thanks to all for having a look and for considering the ways out.
My takeaway here is that I shall rather leave glibc for the equivalent
of an "offline" update rather than on-the-go one as was the triggering
precondition here.
That is, to opportunistically update it once a reboot is encountered
anyway, in a minimal start-up environment and followed with another
reboot round [*1], i.e., what I already do with kernel updates.
I know that this *offline* way of updating keeps popping up as warmly
recommended but there is little actual guidance/enforcement in the
respective RPM ecosystem tooling itself, AFAIK.
For the rest of the package base, the "online" way may be mostly
conflict-free [*2], but with glibc in particular, it is apparently better
to err on the safe side for the time being.
[*1] sadly, I can imagine triggering the same bash completion sequence
in this minimalistic (VT only) environment as well
[*2] given the static conflicts are properly expressed and guarded on
the RPM level, which sadly, leaves a lot to be desired too, e.g.,
[bug 1930110], [bug 1933274] and
https://pagure.io/releng/issue/9539#comment-718052
|
Note on order of events: 1. happily booted to the user's WM session 2. in a terminal, "su -" promoted as root, updated glibc 2.32.9000-{16 -> 29} along with some more programs in a batch (so the was glibc updated without a reboot, if it might affect anything} Few days in, in said terminal (likely the very same, so the one that must have been running with previous glibc, I ran "dnf update lcms<TAB>" and all I got was a smashed stack :-/ And the last suspect in the backtrace is libnss_systemd.so, hence reporting it here. At least I didn't move with systemd version during this boot, was systemd-247.3-1.fc34.x86_64 from the very boot stage. Enter the slightly annotated backtrace: > #0 __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:49 > #1 0x00007f790efa4954 in __GI_abort () at abort.c:100 > ap = {{ > gp_offset = 24, > fp_offset = 0, > overflow_arg_area = 0x7fffe34dba60, ^^^^^^^^^^^^^^ > reg_save_area = 0x7fffe34db9f0 ^^^^^^^^^^^^^^ > }} - marked addresses relate to the affected space referred further > > #2 0x00007f790effdaf7 in __libc_message (action=<optimized out>, fmt=<optimized out>) > at ../sysdeps/posix/libc_fatal.c:155 > #3 0x00007f790f08f59a in __GI___fortify_fail (msg=0x7f790f110207 "stack smashing detected") at fortify_fail.c:26 > #4 0x00007f790f08f566 in __stack_chk_fail () at stack_chk_fail.c:24 > #5 0x00007f790e78de88 in pthread_getname_np (th=2, buf=0x7fffe34db6d0 "", len=0) at pthread_getname.c:38 - could not find where from libnss_systemd.so pthread_getname_np would get called, other thant from pthread_once, which is exactly a recently added code: https://github.com/systemd/systemd/commit/3339381f22fd7fe6c07b67c630c32eab30fa9d27 > #6 0x00007f790ec3b6a4 in ?? () from /usr/lib64/libnss_systemd.so.2 - cannot examine this frame: "No function contains program counter for selected frame" > #7 0x00007fffe34dbca8 in ?? () > #8 0x00007fffe34dbb80 in ?? () > #9 0x00007fffe34dbb90 in ?? () - above frames #7 - #9 may be the actual consequence of a smashed stack (instruction pointer values overwritten?) > #10 0x00007f790f1b86c1 in _dl_lookup_symbol_x ( > undef_name=0x180000000 <error: Cannot access memory at address 0x180000000>, undef_map=0x560e513ee000, > ref=0x560e513ee368, symbol_scope=0x560e513ee368, version=0x560e513b1f00, type_class=1, flags=0, > skip_map=0x7fffe34dbde0) at dl-lookup.c:855 ^^^^^^^^^^^^^^ - many things suspicious about the undef_name argument above, but may be the actual consequence of a smashed stack - if "skip_map" can be trusted, 0x7fffe34dbde0 is pretty close to 0x7fffe34dbb90 from #9 ("info proc map" doesn't indicate anything about this address space, though) > #11 __GI__dl_catch_exception (exception=<optimized out>, operate=<optimized out>, > args=<optimized out>) at dl-error-skeleton.c:182 > #12 dl_open_worker (a=0x7fffe34dbef0) at dl-open.c:783 > #13 __GI__dl_catch_exception (exception=0x7fffe34dbed0, > operate=0x7f790f1c1490 <dl_open_worker>, args=0x7fffe34dbef0) at dl-error-skeleton.c:208 > #14 _dl_open (file=0x7fffe34dbed0 "", mode=-2147483646, > caller_dlopen=0x7f790f0a2901 <nss_load_library+353>, nsid=-2, argc=1, argv=0x7fffe34df798, env=0x560e511bcd30) > at dl-open.c:864 > argv=0x7fffe34df798, env=0x560e511bcd30) at dl-open.c:864 > args = { > file = 0x7fffe34dc180 "libnss_systemd.so.2", ^^^^^^^^^^^^^^ > mode = -2147483646, > caller_dlopen = 0x7f790f0a2901 <nss_load_library+353>, > map = 0x560e513ee000, > nsid = 0, > original_global_scope_pending_adds = 0, > libc_already_loaded = true, > argc = 1, > argv = 0x7fffe34df798, ^^^^^^^^^^^^^^ > env = 0x560e511bcd30 > } > exception = { > objname = 0x0, > errstring = 0x7f790f006819 <malloc_consolidate+297>, > message_buffer = 0x7f790f142a60 <main_arena+96> > } > errcode = <optimized out> > __PRETTY_FUNCTION__ = "_dl_open" > #15 do_dlopen (ptr=ptr@entry=0x7fffe34dc140) at dl-libc.c:96 > #16 __GI__dl_catch_exception (exception=exception@entry=0x7fffe34dc0c0, > operate=0x7f790f0ba010 <do_dlopen>, args=0x7fffe34dc140) at dl-error-skeleton.c:208 > #17 __GI__dl_catch_error (objname=0x7fffe34dc118, errstring=0x7fffe34dc120, > mallocedp=0x7fffe34dc117, operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:227 > #18 dlerror_run (operate=operate@entry=0x7f790f0ba010 <do_dlopen>, > args=args@entry=0x7fffe34dc140) at dl-libc.c:46 > #19 __GI___libc_dlopen_mode (name=<optimized out>, mode=<optimized out>) at dl-libc.c:195 > #20 nss_load_library (ni=ni@entry=0x560e50f24a70) at nsswitch.c:351 > #21 __GI___nss_lookup_function (ni=0x560e50f24a70, fct_name=<optimized out>, > fct_name@entry=0x7f790f10ebec "getpwnam_r") at nsswitch.c:448 > #22 __GI___nss_next2 (ni=0x7fffe34dc2a8, fct_name=0x7f790f10ebec "getpwnam_r", fct2_name=0x0, > fctp=0x7fffe34dc2b0, status=<optimized out>, all_values=<optimized out>) at nsswitch.c:243 > #23 __getpwnam_r (name=0x560e513ccf90 "*", resbuf=0x7f790f146280 <resbuf>, > buffer=0x560e513b07d0 "systemd-oom", buflen=1024, result=<optimized out>) at ../nss/getXXbyYY_r.c:385 - interestingly enough, but may be just a coincidence, it seems the previously getpwnam(3)-resolved user was "systemd-oom", although no idea where this would come from other than another "systemd NSS trip" (- also of interest, but off-topic here: what a funky move trying to resolve standard wildcard '*' as a user, maybe for internal defense, but nonetheless) > #24 getpwnam (name=name@entry=0x560e513ccf90 "*") at ../nss/getXXbyYY.c:134 > #25 tilde_expand_word (filename=0x560e513e60a0 "~*") at lib/readline/./tilde.c:386 > #26 tilde_expand_word (filename=filename@entry=0x560e513e60a0 "~*") at lib/readline/./tilde.c:336 > #27 tilde_expand (string=0x560e513c72b2 "") at lib/readline/./tilde.c:234 > #28 bash_tilde_expand (s=0x560e513c72b0 "~*", assign_p=<optimized out>) > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/general.c:1211 > #29 expand_word_internal (word=<optimized out>, quoted=<optimized out>, isexp=<optimized out>, > contains_dollar_at=<optimized out>, expanded_something=<optimized out>) > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/subst.c:10290 > #30 call_expand_word_internal (e=0x0, c=0x0, i=0, q=0, w=0x560e513cdf50) > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/subst.c:3726 > #31 cond_expand_word (special=1, w=0x560e513cdf50) at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/subst.c:3689 > #32 cond_expand_word (w=0x560e513cdf50, special=1) at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/subst.c:3676 > #33 execute_cond_node (cond=cond@entry=0x560e513e7c30) > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/execute_cmd.c:3892 > #34 execute_cond_command (cond_command=0x560e513e7c30) > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/execute_cmd.c:3993 > #35 execute_command_internal (command=0x560e513e4550, asynchronous=<optimized out>, pipe_in=-1, > pipe_out=<optimized out>, fds_to_close=<optimized out>) > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/execute_cmd.c:1064 > #36 execute_command (command=0x560e513e4550) > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/execute_cmd.c:399 [nothing interesting, permutation of the surrounding functions over and over] > #78 execute_connection (fds_to_close=0x560e513b1070, pipe_out=-1, pipe_in=-1, asynchronous=0, > command=0x560e513ca800) at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/execute_cmd.c:2718 > #79 execute_command_internal (command=0x560e513ca800, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, > fds_to_close=0x560e513b1070) at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/execute_cmd.c:1032 > #80 execute_command_internal (command=0x560e511b0c60, asynchronous=<optimized out>, > pipe_in=-1, pipe_out=-1, fds_to_close=0x560e513b1070) > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/execute_cmd.c:1024 > #81 execute_function (var=var@entry=0x560e513d84d0, words=words@entry=0x560e513bde80, > flags=flags@entry=0, fds_to_close=fds_to_close@entry=0x560e513b1070, async=async@entry=0, > subshell=subshell@entry=0) at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/execute_cmd.c:5110 > #82 execute_shell_function (var=0x560e513d84d0, words=0x560e513bde80) > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/execute_cmd.c:5169 > #83 gen_shell_function_matches (cs=0x560e513edb00, cs=0x560e513edb00, nw=<optimized out>, > foundp=<synthetic pointer>, cw=<optimized out>, lwords=0x560e513e1ea0, ind=-481435440, > line=0x560e50f49be0 "dnf update lcms", text=0x560e50f49b30 "lcms", cmd=0x560e513b8970 "dnf") > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/pcomplete.c:1155 > #84 gen_compspec_completions (cs=cs@entry=0x560e513edb00, cmd=cmd@entry=0x560e513b8970 "dnf", > word=word@entry=0x560e50f49b30 "lcms", start=start@entry=0, end=end@entry=15, foundp=foundp@entry=0x7fffe34de2ec) > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/pcomplete.c:1422 > #85 gen_progcomp_completions (ocmd=ocmd@entry=0x560e513b8970 "dnf", > cmd=cmd@entry=0x560e513b8970 "dnf", word=word@entry=0x560e50f49b30 "lcms", start=start@entry=0, > end=end@entry=15, foundp=foundp@entry=0x7fffe34de2ec, retryp=0x7fffe34de2e8, lastcs=0x7fffe34de2f0) > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/pcomplete.c:1594 > #86 programmable_completions (cmd=0x560e513b8970 "dnf", word=0x560e50f49b30 "lcms", start=0, > end=<optimized out>, foundp=0x7fffe34de384) at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/pcomplete.c:1649 > #87 attempt_shell_completion (text=0x560e50f49b30 "lcms", start=11, end=15) > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/bashline.c:1665 > #88 gen_completion_matches (text=text@entry=0x560e50f49b30 "lcms", start=start@entry=11, > end=end@entry=15, our_func=0x560e4eebb130 <rl_filename_completion_function>, found_quote=<optimized out>, > quote_char=<optimized out>) at lib/readline/complete.c:1222 > #89 rl_complete_internal (what_to_do=9) at lib/readline/complete.c:2031 > #90 _rl_dispatch_subseq (key=<optimized out>, map=<optimized out>, got_subseq=<optimized out>) > at lib/readline/readline.c:934 > #91 _rl_dispatch (map=<optimized out>, key=<optimized out>) at lib/readline/readline.c:880 > #92 readline_internal_char () at lib/readline/readline.c:692 > #93 readline_internal_charloop () at lib/readline/readline.c:741 > #94 readline_internal () at lib/readline/readline.c:753 > #95 readline (prompt=<optimized out>) at lib/readline/readline.c:432 > #96 yy_readline_get () at ./parse.y:1488 > #97 yy_getc () at ./parse.y:1422 > #98 shell_getc (remove_quoted_newline=1) at ./parse.y:2358 > #99 read_token (command=<optimized out>) at ./parse.y:3290 > #100 read_token (command=0) at ./parse.y:3254 > #101 yylex () at ./parse.y:2797 > #102 yyparse () at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/y.tab.c:1824 > #103 parse_command () at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/eval.c:348 > #104 read_command () at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/eval.c:392 > #105 reader_loop () at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/eval.c:138 > #106 main (argc=1, argv=0x7fffe34df798, env=0x7fffe34df7a8) > at /usr/src/debug/bash-5.1.0-2.fc34.x86_64/shell.c:821 No idea if this helps but I figure having something reported is better than nothing, if there are more occurrences in the future, for instance. Let me know if I can dig something more from the core.