Bug 1747913 - Running ipa-server-install ends during ipa.service restart with segfault
Summary: Running ipa-server-install ends during ipa.service restart with segfault
Keywords:
Status: CLOSED DUPLICATE of bug 1747901
Alias: None
Product: Fedora
Classification: Fedora
Component: freeipa
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: IPA Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-02 09:03 UTC by Jan Pazdziora
Modified: 2019-09-05 07:13 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-05 07:13:48 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Jan Pazdziora 2019-09-02 09:03:57 UTC
Description of problem:

Running ipa-server-install ends during ipa.service restart with segfault in /usr/sbin/ipactl.

Version-Release number of selected component (if applicable):

freeipa-server-4.8.1-2.fc32.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. ipa-server-install -U -r EXAMPLE.TEST -a Secret123 -p Secret123
2. systemctl status ipa
3. /usr/sbin/ipactl start

Actual results:

  [6/10]: starting directory server
  [7/10]: upgrading server
  [8/10]: stopping directory server
  [9/10]: restoring configuration
  [10/10]: starting directory server
Done.
Restarting the KDC
Configuring client side components
This program will set up FreeIPA client.
Version 4.8.1

Using existing certificate '/etc/ipa/ca.crt'.
Client hostname: freeipa.example.test
Realm: EXAMPLE.TEST
DNS Domain: example.test
IPA Server: freeipa.example.test
BaseDN: dc=example,dc=test

Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
/usr/lib/python3.8/site-packages/yubico/yubikey_usb_hid.py:288: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if mode is 'nand':
/usr/lib/python3.8/site-packages/yubico/yubikey_usb_hid.py:294: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif mode is 'and':
/usr/lib/python3.8/site-packages/yubico/yubikey_usb_hid.py:306: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if mode is 'nand':
/usr/lib/python3.8/site-packages/yubico/yubikey_config.py:478: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if slot is 1:
/usr/lib/python3.8/site-packages/yubico/yubikey_config.py:483: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif slot is 2:
Systemwide CA database updated.
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Could not update DNS SSHFP records.
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring example.test as NIS domain.
Client configuration complete.
The ipa-client-install command was successful

Please add records in this file to your DNS system: /tmp/ipa.system.records.h1z1_7jt.db
CalledProcessError(Command ['/bin/systemctl', 'restart', 'ipa.service'] returned non-zero exit status 1: 'Job for ipa.service failed because a fatal signal was delivered causing the control process to dump core.\nSee "systemctl status ipa.service" and "journalctl -xe" for details.\n')
The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information
Segmentation fault (core dumped)

# systemctl status ipa
● ipa.service - Identity, Policy, Audit
   Loaded: loaded (/usr/lib/systemd/system/ipa.service; enabled; vendor preset: disabled)
   Active: failed (Result: core-dump) since Mon 2019-09-02 08:55:57 UTC; 9s ago
  Process: 8908 ExecStart=/usr/sbin/ipactl start (code=dumped, signal=SEGV)
 Main PID: 8908 (code=dumped, signal=SEGV)
      CPU: 1.993s

Sep 02 08:55:56 freeipa.example.test ipactl[8908]: Starting krb5kdc Service
Sep 02 08:55:56 freeipa.example.test ipactl[8908]: Starting kadmin Service
Sep 02 08:55:56 freeipa.example.test ipactl[8908]: Starting httpd Service
Sep 02 08:55:56 freeipa.example.test ipactl[8908]: Starting ipa-custodia Service
Sep 02 08:55:56 freeipa.example.test ipactl[8908]: Starting pki-tomcatd Service
Sep 02 08:55:56 freeipa.example.test ipactl[8908]: Starting ipa-otpd Service
Sep 02 08:55:57 freeipa.example.test systemd[1]: ipa.service: Main process exited, code=dumped, status=11/SEGV
Sep 02 08:55:57 freeipa.example.test systemd[1]: ipa.service: Failed with result 'core-dump'.
Sep 02 08:55:57 freeipa.example.test systemd[1]: Failed to start Identity, Policy, Audit.
Sep 02 08:55:57 freeipa.example.test systemd[1]: ipa.service: Consumed 1.993s CPU time.

# /usr/sbin/ipactl start
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting httpd Service
Starting ipa-custodia Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
Segmentation fault (core dumped)

Expected results:

No error during ipa-server-install, ipa service fine and dandy, ipactl not ending with segfault after it said that it was successful.

Additional info:

This is with python3-3.8.0~b4-1.fc32.x86_64, so it has the fix for bug 1745450.

Comment 1 Lukas Slebodnik 2019-09-02 11:17:53 UTC
Simpler reproducer is to run following command on machine with installed freeIPA

sh# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
Segmentation fault (core dumped)

sh# coredumpctl info
           PID: 30832 (ipactl)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Mon 2019-09-02 07:13:40 EDT (3min 43s ago)
  Command Line: /usr/bin/python3 -E /usr/sbin/ipactl status
    Executable: /usr/bin/python3.8
 Control Group: /user.slice/user-0.slice/session-4.scope
          Unit: session-4.scope
         Slice: user-0.slice
       Session: 4
     Owner UID: 0 (root)
       Boot ID: 462e056a512a4811ab1526cd06588a8b
    Machine ID: 8e803693219b48718133e6400b902c88
      Hostname: kvm-07-guest22.testrelm.test
       Storage: /var/lib/systemd/coredump/core.ipactl.0.462e056a512a4811ab1526cd06588a8b.30832.1567422820000000000000.lz4
       Message: Process 30832 (ipactl) of user 0 dumped core.
                
                Stack trace of thread 30832:
                #0  0x00007f503a75154f _PyFunction_Vectorcall (libpython3.8.so.1.0)
                #1  0x00007f503a71f6b0 object_vacall (libpython3.8.so.1.0)
                #2  0x00007f503a71f9e1 PyObject_CallFunctionObjArgs (libpython3.8.so.1.0)
                #3  0x00007f503a7ad7c3 handle_callback (libpython3.8.so.1.0)
                #4  0x00007f503a76fbf7 PyObject_ClearWeakRefs (libpython3.8.so.1.0)
                #5  0x00007f5038b59abc ctypedescr_dealloc (_cffi_backend.cpython-38-x86_64-linux-gnu.so)
                #6  0x00007f5038b56825 cfield_dealloc (_cffi_backend.cpython-38-x86_64-linux-gnu.so)
                #7  0x00007f503a712c65 dict_dealloc (libpython3.8.so.1.0)
                #8  0x00007f5038b59b25 ctypedescr_dealloc (_cffi_backend.cpython-38-x86_64-linux-gnu.so)
                #9  0x00007f503a713058 tupledealloc (libpython3.8.so.1.0)
                #10 0x00007f503a713058 tupledealloc (libpython3.8.so.1.0)
                #11 0x00007f503a784b76 subtype_dealloc (libpython3.8.so.1.0)
                #12 0x00007f503a712c65 dict_dealloc (libpython3.8.so.1.0)
                #13 0x00007f503a706a4b cell_dealloc (libpython3.8.so.1.0)
                #14 0x00007f503a713058 tupledealloc (libpython3.8.so.1.0)
                #15 0x00007f503a7071c2 func_clear (libpython3.8.so.1.0)
                #16 0x00007f503a713f84 collect.constprop.0 (libpython3.8.so.1.0)
                #17 0x00007f503a81a8c2 _PyGC_CollectNoFail (libpython3.8.so.1.0)
                #18 0x00007f503a81ab74 PyImport_Cleanup (libpython3.8.so.1.0)
                #19 0x00007f503a81af56 Py_FinalizeEx (libpython3.8.so.1.0)
                #20 0x00007f503a81b088 Py_Exit (libpython3.8.so.1.0)
                #21 0x00007f503a81b0cf handle_system_exit (libpython3.8.so.1.0)
                #22 0x00007f503a81b219 PyErr_PrintEx (libpython3.8.so.1.0)
                #23 0x00007f503a6ffb42 PyRun_SimpleFileExFlags.cold (libpython3.8.so.1.0)
                #24 0x00007f503a81c36f Py_RunMain (libpython3.8.so.1.0)
                #25 0x00007f503a81c559 Py_BytesMain (libpython3.8.so.1.0)
                #26 0x00007f503a98f193 __libc_start_main (libc.so.6)
                #27 0x000056378b68808e _start (python3.8)

Comment 2 Lukas Slebodnik 2019-09-04 20:50:27 UTC
Could you try to take a look and try to investigate whether bug in freeipa/python3.8/cffi?

Comment 3 Alexander Bokovoy 2019-09-04 21:00:08 UTC
From Christian's report earlier today on IRC:

-------

So turns out the segfault on 3.8 is a bizarre edge case that involves cyclic GC, cleanup on shutdown, weak references, and closure functions. It's triggered by the new vector call feature.
Victor came up with a workaround that no longer triggeres the bug for us. He is currently looking into proper solutions.
It looks like both Python and cffi have to change. cffi does some magic with non-Python linked list and borrowed references, too.
It's all messy.

-------

Comment 4 Christian Heimes 2019-09-05 07:13:48 UTC
Thanks for reporting, Jan.

It's a duplicate of RHBH#1747901. You can find more details on upstream bug https://bugs.python.org/issue38006

*** This bug has been marked as a duplicate of bug 1747901 ***


Note You need to log in before you can comment on or make changes to this bug.