Description of problem: Running ipa-server-install ends during ipa.service restart with segfault in /usr/sbin/ipactl. Version-Release number of selected component (if applicable): freeipa-server-4.8.1-2.fc32.x86_64 How reproducible: Deterministic. Steps to Reproduce: 1. ipa-server-install -U -r EXAMPLE.TEST -a Secret123 -p Secret123 2. systemctl status ipa 3. /usr/sbin/ipactl start Actual results: [6/10]: starting directory server [7/10]: upgrading server [8/10]: stopping directory server [9/10]: restoring configuration [10/10]: starting directory server Done. Restarting the KDC Configuring client side components This program will set up FreeIPA client. Version 4.8.1 Using existing certificate '/etc/ipa/ca.crt'. Client hostname: freeipa.example.test Realm: EXAMPLE.TEST DNS Domain: example.test IPA Server: freeipa.example.test BaseDN: dc=example,dc=test Configured sudoers in /etc/nsswitch.conf Configured /etc/sssd/sssd.conf /usr/lib/python3.8/site-packages/yubico/yubikey_usb_hid.py:288: SyntaxWarning: "is" with a literal. Did you mean "=="? if mode is 'nand': /usr/lib/python3.8/site-packages/yubico/yubikey_usb_hid.py:294: SyntaxWarning: "is" with a literal. Did you mean "=="? elif mode is 'and': /usr/lib/python3.8/site-packages/yubico/yubikey_usb_hid.py:306: SyntaxWarning: "is" with a literal. Did you mean "=="? if mode is 'nand': /usr/lib/python3.8/site-packages/yubico/yubikey_config.py:478: SyntaxWarning: "is" with a literal. Did you mean "=="? if slot is 1: /usr/lib/python3.8/site-packages/yubico/yubikey_config.py:483: SyntaxWarning: "is" with a literal. Did you mean "=="? elif slot is 2: Systemwide CA database updated. Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub Could not update DNS SSHFP records. SSSD enabled Configured /etc/openldap/ldap.conf Configured /etc/ssh/ssh_config Configured /etc/ssh/sshd_config Configuring example.test as NIS domain. Client configuration complete. The ipa-client-install command was successful Please add records in this file to your DNS system: /tmp/ipa.system.records.h1z1_7jt.db CalledProcessError(Command ['/bin/systemctl', 'restart', 'ipa.service'] returned non-zero exit status 1: 'Job for ipa.service failed because a fatal signal was delivered causing the control process to dump core.\nSee "systemctl status ipa.service" and "journalctl -xe" for details.\n') The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information Segmentation fault (core dumped) # systemctl status ipa ● ipa.service - Identity, Policy, Audit Loaded: loaded (/usr/lib/systemd/system/ipa.service; enabled; vendor preset: disabled) Active: failed (Result: core-dump) since Mon 2019-09-02 08:55:57 UTC; 9s ago Process: 8908 ExecStart=/usr/sbin/ipactl start (code=dumped, signal=SEGV) Main PID: 8908 (code=dumped, signal=SEGV) CPU: 1.993s Sep 02 08:55:56 freeipa.example.test ipactl[8908]: Starting krb5kdc Service Sep 02 08:55:56 freeipa.example.test ipactl[8908]: Starting kadmin Service Sep 02 08:55:56 freeipa.example.test ipactl[8908]: Starting httpd Service Sep 02 08:55:56 freeipa.example.test ipactl[8908]: Starting ipa-custodia Service Sep 02 08:55:56 freeipa.example.test ipactl[8908]: Starting pki-tomcatd Service Sep 02 08:55:56 freeipa.example.test ipactl[8908]: Starting ipa-otpd Service Sep 02 08:55:57 freeipa.example.test systemd[1]: ipa.service: Main process exited, code=dumped, status=11/SEGV Sep 02 08:55:57 freeipa.example.test systemd[1]: ipa.service: Failed with result 'core-dump'. Sep 02 08:55:57 freeipa.example.test systemd[1]: Failed to start Identity, Policy, Audit. Sep 02 08:55:57 freeipa.example.test systemd[1]: ipa.service: Consumed 1.993s CPU time. # /usr/sbin/ipactl start Existing service file detected! Assuming stale, cleaning and proceeding Starting Directory Service Starting krb5kdc Service Starting kadmin Service Starting named Service Starting httpd Service Starting ipa-custodia Service Starting pki-tomcatd Service Starting ipa-otpd Service Starting ipa-dnskeysyncd Service ipa: INFO: The ipactl command was successful Segmentation fault (core dumped) Expected results: No error during ipa-server-install, ipa service fine and dandy, ipactl not ending with segfault after it said that it was successful. Additional info: This is with python3-3.8.0~b4-1.fc32.x86_64, so it has the fix for bug 1745450.
Simpler reproducer is to run following command on machine with installed freeIPA sh# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful Segmentation fault (core dumped) sh# coredumpctl info PID: 30832 (ipactl) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Mon 2019-09-02 07:13:40 EDT (3min 43s ago) Command Line: /usr/bin/python3 -E /usr/sbin/ipactl status Executable: /usr/bin/python3.8 Control Group: /user.slice/user-0.slice/session-4.scope Unit: session-4.scope Slice: user-0.slice Session: 4 Owner UID: 0 (root) Boot ID: 462e056a512a4811ab1526cd06588a8b Machine ID: 8e803693219b48718133e6400b902c88 Hostname: kvm-07-guest22.testrelm.test Storage: /var/lib/systemd/coredump/core.ipactl.0.462e056a512a4811ab1526cd06588a8b.30832.1567422820000000000000.lz4 Message: Process 30832 (ipactl) of user 0 dumped core. Stack trace of thread 30832: #0 0x00007f503a75154f _PyFunction_Vectorcall (libpython3.8.so.1.0) #1 0x00007f503a71f6b0 object_vacall (libpython3.8.so.1.0) #2 0x00007f503a71f9e1 PyObject_CallFunctionObjArgs (libpython3.8.so.1.0) #3 0x00007f503a7ad7c3 handle_callback (libpython3.8.so.1.0) #4 0x00007f503a76fbf7 PyObject_ClearWeakRefs (libpython3.8.so.1.0) #5 0x00007f5038b59abc ctypedescr_dealloc (_cffi_backend.cpython-38-x86_64-linux-gnu.so) #6 0x00007f5038b56825 cfield_dealloc (_cffi_backend.cpython-38-x86_64-linux-gnu.so) #7 0x00007f503a712c65 dict_dealloc (libpython3.8.so.1.0) #8 0x00007f5038b59b25 ctypedescr_dealloc (_cffi_backend.cpython-38-x86_64-linux-gnu.so) #9 0x00007f503a713058 tupledealloc (libpython3.8.so.1.0) #10 0x00007f503a713058 tupledealloc (libpython3.8.so.1.0) #11 0x00007f503a784b76 subtype_dealloc (libpython3.8.so.1.0) #12 0x00007f503a712c65 dict_dealloc (libpython3.8.so.1.0) #13 0x00007f503a706a4b cell_dealloc (libpython3.8.so.1.0) #14 0x00007f503a713058 tupledealloc (libpython3.8.so.1.0) #15 0x00007f503a7071c2 func_clear (libpython3.8.so.1.0) #16 0x00007f503a713f84 collect.constprop.0 (libpython3.8.so.1.0) #17 0x00007f503a81a8c2 _PyGC_CollectNoFail (libpython3.8.so.1.0) #18 0x00007f503a81ab74 PyImport_Cleanup (libpython3.8.so.1.0) #19 0x00007f503a81af56 Py_FinalizeEx (libpython3.8.so.1.0) #20 0x00007f503a81b088 Py_Exit (libpython3.8.so.1.0) #21 0x00007f503a81b0cf handle_system_exit (libpython3.8.so.1.0) #22 0x00007f503a81b219 PyErr_PrintEx (libpython3.8.so.1.0) #23 0x00007f503a6ffb42 PyRun_SimpleFileExFlags.cold (libpython3.8.so.1.0) #24 0x00007f503a81c36f Py_RunMain (libpython3.8.so.1.0) #25 0x00007f503a81c559 Py_BytesMain (libpython3.8.so.1.0) #26 0x00007f503a98f193 __libc_start_main (libc.so.6) #27 0x000056378b68808e _start (python3.8)
Could you try to take a look and try to investigate whether bug in freeipa/python3.8/cffi?
From Christian's report earlier today on IRC: ------- So turns out the segfault on 3.8 is a bizarre edge case that involves cyclic GC, cleanup on shutdown, weak references, and closure functions. It's triggered by the new vector call feature. Victor came up with a workaround that no longer triggeres the bug for us. He is currently looking into proper solutions. It looks like both Python and cffi have to change. cffi does some magic with non-Python linked list and borrowed references, too. It's all messy. -------
Thanks for reporting, Jan. It's a duplicate of RHBH#1747901. You can find more details on upstream bug https://bugs.python.org/issue38006 *** This bug has been marked as a duplicate of bug 1747901 ***