Bug 1636633
Summary: | FreeIPA server deployment fails due to crash of ipa-submit and/or kernel/xfs locking issue and/or crash of named-pkcs11 | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Adam Williamson <awilliam> | ||||||
Component: | krb5 | Assignee: | Robbie Harwood <rharwood> | ||||||
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
Severity: | urgent | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | rawhide | CC: | abokovoy, awilliam, ipa-maint, jan.kratochvil, jcholast, jforbes, jhrozek, j, kevin, klember, labbott, mattdm, mboddu, mboddu, nalin, npmccallum, pvoborni, rcritten, rharwood, robatino, sbose, sergiodj, sgallagh, ssorce | ||||||
Target Milestone: | --- | ||||||||
Target Release: | --- | ||||||||
Hardware: | x86_64 | ||||||||
OS: | Linux | ||||||||
Whiteboard: | AcceptedBlocker | ||||||||
Fixed In Version: | krb5-1.16.1-21.fc29 krb5-1.16.1-21.fc28 | Doc Type: | If docs needed, set a value | ||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2018-10-12 16:17:12 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | 1633089, 1671505 | ||||||||
Bug Blocks: | 1517013, 1574713 | ||||||||
Attachments: |
|
Description
Adam Williamson
2018-10-06 01:29:28 UTC
Created attachment 1490966 [details]
tarball of /var/log from the failed test
Wow, this crash is *contagious* - if you try to run it through gdb...gdb crashes. Core was generated by `/usr/libexec/certmonger/ipa-submit'. Program terminated with signal SIGABRT, Aborted. #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 50 return ret; (gdb) set logging on Copying output to gdb.txt. (gdb) thread apply all bt full Segmentation fault (core dumped) Here's the best I can do: (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007f3cb8daf895 in __GI_abort () at abort.c:79 #2 0x00007f3cb8daf769 in __assert_fail_base (fmt=0x7f3cb8f16eb0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7f3cb9336f25 "r == 0", file=0x7f3cb9336f08 "../../../include/k5-thread.h", line=376, function=<optimized out>) at assert.c:92 #3 0x00007f3cb8dbd9f6 in __GI___assert_fail (assertion=assertion@entry=0x7f3cb9336f25 "r == 0", file=file@entry=0x7f3cb9336f08 "../../../include/k5-thread.h", line=line@entry=376, function=function@entry=0x7f3cb9336ff0 <__PRETTY_FUNCTION__.5598> "k5_mutex_lock") at assert.c:101 #4 0x00007f3cb92d0c95 in k5_mutex_lock (m=0x55b7f90409b8) at ../../../include/k5-thread.h:376 #5 0x00007f3cb92d16ef in k5_mutex_lock (m=0x55b7f90409b8) at ../../../include/k5-thread.h:376 #6 k5_cc_mutex_lock (context=context@entry=0x55b7f9051b20, m=m@entry=0x55b7f90409b8) at ccbase.c:460 Segmentation fault (core dumped) Similar story with the named-pkcs11 crash, backtracing it also crashes gdb: (gdb) bt full #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 set = {__val = {16387, 140069578765808, 140069712060800, 140069739891035, 140071695908864, 140069578765808, 140069578765808, 140069578765808, 140069578765808, 140069578765898, 140069578765908, 140069578765808, 140069578765908, 0, 0, 0}} pid = <optimized out> tid = <optimized out> ret = <optimized out> #1 0x00007f6487118895 in __GI_abort () at abort.c:79 save_stage = 1 act = {__sigaction_handler = {sa_handler = 0x7f647d7c75f0, sa_sigaction = 0x7f647d7c75f0}, sa_mask = { __val = {0, 140069553831968, 140069712058688, 0, 0, 0, 140069712060248, 21474836480, 140069712060096, 140069741017680, 140069741002416, 0, 9654014966396037376, 140069740987315, 0, 140069741002416}}, sa_flags = -2031231224, sa_restorer = 0x7f6486eddf25} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x00007f6487118769 in __assert_fail_base (fmt=0x7f648727feb0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7f6486eddf25 "r == 0", file=0x7f6486eddf08 "../../../include/k5-thread.h", line=376, function=<optimized out>) at assert.c:92 str = 0x7f647d7c75f0 "p\235y}d\177" total = 4096 #3 0x00007f64871269f6 in __GI___assert_fail (assertion=assertion@entry=0x7f6486eddf25 "r == 0", file=file@entry=0x7f6486eddf08 "../../../include/k5-thread.h", line=line@entry=376, function=function@entry=0x7f6486eddff0 <__PRETTY_FUNCTION__.5598> "k5_mutex_lock") at assert.c:101 No locals. #4 0x00007f6486e77c95 in k5_mutex_lock (m=0x7f647d799d78) at ../../../include/k5-thread.h:376 r = <optimized out> __PRETTY_FUNCTION__ = "k5_mutex_lock" #5 0x00007f6486e786ef in k5_mutex_lock (m=0x7f647d799d78) at ../../../include/k5-thread.h:376 r = 22 r = <optimized out> __PRETTY_FUNCTION__ = "k5_mutex_lock" #6 k5_cc_mutex_lock (context=context@entry=0x7f647d7a3ff0, m=m@entry=0x7f647d799d78) at ccbase.c:460 No locals. Segmentation fault (core dumped) Let's add some gdb maintainers to the CC party! Created attachment 1490969 [details]
tarball containing the crash coredumps (the whole abrt directory for each, in fact)
The failure in both cases is an assertion in annobin in glibc within libkrb5. Thus moving to krb5 component -- I guess there are gaps which annobin detects and which need to be fixed at krb5 side. This looks like possibly the same bug as https://bugzilla.redhat.com/show_bug.cgi?id=1633089 . I think I may have fixed it in krb5-1.16.1-20 ; can you try with that? it'll happen automatically with the next successful Rawhide compose. That build seems to be broken for the releases it was submitted as an update for. See openQA failures: FEDORA-2018-fde188a269 - krb5-1.16.1-20.fc28 https://openqa.fedoraproject.org/tests/290634 https://openqa.fedoraproject.org/tests/290622 FEDORA-2018-0408a72d39 - krb5-1.16.1-20.fc29 https://openqa.fedoraproject.org/tests/290599 https://openqa.fedoraproject.org/tests/290611 The logs - see "Logs & Assets" tab - show very similar problems to those reported in this bug, multiple crashes running through libkrb, e.g.: { "signal": 6 , "executable": "/usr/libexec/certmonger/ipa-submit" , "stacktrace": [ { "crash_thread": true , "frames": [ { "address": 139810602745515 , "build_id": "4677f12c31fa6c18c9925096f32e71afd4c71327" , "build_id_offset": 224939 , "function_name": "raise" , "file_name": "/lib64/libc.so.6" } , { "address": 139810602657209 , "build_id": "4677f12c31fa6c18c9925096f32e71afd4c71327" , "build_id_offset": 136633 , "function_name": "abort" , "file_name": "/lib64/libc.so.6" } , { "address": 139810602656913 , "build_id": "4677f12c31fa6c18c9925096f32e71afd4c71327" , "build_id_offset": 136337 , "function_name": "_nl_load_domain.cold.0" , "file_name": "/lib64/libc.so.6" } , { "address": 139810602714642 , "build_id": "4677f12c31fa6c18c9925096f32e71afd4c71327" , "build_id_offset": 194066 , "file_name": "/lib64/libc.so.6" } , { "address": 139810626620997 , "build_id": "a97d31f22a47cfc282aacaca99672bf4f3984900" , "build_id_offset": 196165 , "file_name": "/lib64/libkrb5.so.3" } , { "address": 139810626623647 , "build_id": "a97d31f22a47cfc282aacaca99672bf4f3984900" , "build_id_offset": 198815 , "function_name": "k5_cc_mutex_lock" , "file_name": "/lib64/libkrb5.so.3" } , { "address": 139810626662252 , "build_id": "a97d31f22a47cfc282aacaca99672bf4f3984900" , "build_id_offset": 237420 , "function_name": "krb5_mcc_close" , "file_name": "/lib64/libkrb5.so.3" } , { "address": 139810626636547 , "build_id": "a97d31f22a47cfc282aacaca99672bf4f3984900" , "build_id_offset": 211715 , "function_name": "realm_choose" , "file_name": "/lib64/libkrb5.so.3" } , { "address": 139810626632749 , "build_id": "a97d31f22a47cfc282aacaca99672bf4f3984900" , "build_id_offset": 207917 , "function_name": "krb5_cc_select" , "file_name": "/lib64/libkrb5.so.3" } , { "address": 139810547787593 , "build_id": "19dc0f068054682836700e0df269e7002f2d7abb" , "build_id_offset": 141129 , "function_name": "kg_cred_resolve" , "file_name": "/lib64/libgssapi_krb5.so.2" } , { "address": 139810547824754 , "build_id": "19dc0f068054682836700e0df269e7002f2d7abb" , "build_id_offset": 178290 , "function_name": "krb5_gss_init_sec_context_ext" , "file_name": "/lib64/libgssapi_krb5.so.2" } , { "address": 139810547828982 , "build_id": "19dc0f068054682836700e0df269e7002f2d7abb" , "build_id_offset": 182518 , "function_name": "krb5_gss_init_sec_context" , "file_name": "/lib64/libgssapi_krb5.so.2" } , { "address": 139810547736235 , "build_id": "19dc0f068054682836700e0df269e7002f2d7abb" , "build_id_offset": 89771 , "function_name": "gss_init_sec_context" , "file_name": "/lib64/libgssapi_krb5.so.2" } , { "address": 139810519338711 , "build_id": "d4421f649df90b99f09feaafb5e2c16d51490966" , "build_id_offset": 11991 , "file_name": "/usr/lib64/sasl2/libgssapiv2.so" } , { "address": 139810593379985 , "build_id": "3dd431575f610904df6e7f6f708931118693bc2c" , "build_id_offset": 38545 , "function_name": "sasl_client_step" , "file_name": "/lib64/libsasl2.so.3" } , { "address": 139810593380378 , "build_id": "3dd431575f610904df6e7f6f708931118693bc2c" , "build_id_offset": 38938 , "function_name": "sasl_client_start" , "file_name": "/lib64/libsasl2.so.3" } , { "address": 139810629588027 , "build_id": "83bfd387acf651d85146865b67720f35f5b804cf" , "build_id_offset": 95291 , "function_name": "ldap_int_sasl_bind" , "file_name": "/lib64/libldap-2.4.so.2" } , { "address": 139810629602012 , "build_id": "83bfd387acf651d85146865b67720f35f5b804cf" , "build_id_offset": 109276 , "function_name": "ldap_sasl_interactive_bind" , "file_name": "/lib64/libldap-2.4.so.2" } , { "address": 139810629602554 , "build_id": "83bfd387acf651d85146865b67720f35f5b804cf" , "build_id_offset": 109818 , "function_name": "ldap_sasl_interactive_bind_s" , "file_name": "/lib64/libldap-2.4.so.2" } , { "address": 94513532705626 , "build_id": "9d767fd01f98945efad349bb9b3ead7116899d94" , "build_id_offset": 23386 , "file_name": "/usr/libexec/certmonger/ipa-submit" } , { "address": 94513532707468 , "build_id": "9d767fd01f98945efad349bb9b3ead7116899d94" , "build_id_offset": 25228 , "file_name": "/usr/libexec/certmonger/ipa-submit" } , { "address": 94513532701410 , "build_id": "9d767fd01f98945efad349bb9b3ead7116899d94" , "build_id_offset": 19170 , "file_name": "/usr/libexec/certmonger/ipa-submit" } ] } ] } Unfortunately, the F29 update was pushed stable with positive karma from folks who clearly didn't test FreeIPA, or look at the automated test results :( That means this is now a clear F29 Final blocker. This is probably the same bug as https://bugzilla.redhat.com/show_bug.cgi?id=1633089 . Well...it looks the same as the crash in https://bugzilla.redhat.com/show_bug.cgi?id=1633089#c26 , yes. But if I'm following things correctly, that's not the same as the *initial* problem in that bug. The initial problem in that bug was a crash that occurred at a later point in a specific test suite. You then investigated and attempted to fix that crash with the -20 build. But it seems like the build that attempts to fix it introduces this *other* crash during initial FreeIPA deployment - so when your -20 build which tried to fix #1633089 showed up in Rawhide on 2018-10-02, then later in F28 and F29 updates-testing yesterday, it *introduced* this crash to Fedora. krb5-1.16.1-21.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-b1048ce2df krb5-1.16.1-21.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-e2b9bccef0 Note, -21 should just revert to the state of -19, before Robbie tried to fix 1633089 and this crash showed up. 1633089 will still likely be present in it, but hopefully this crash will not be present and openQA tests will pass. krb5-1.16.1-21.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-e2b9bccef0 hey folks, can we get some blocker votes on this? it's a pretty obvious blocker and the fix is ready but I cannot push it stable until it's accepted... +1 given adam's comments. +1 blocker +1 blocker +1 Blocker That's +4, setting accepted. krb5-1.16.1-21.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report. krb5-1.16.1-21.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report. |