Bug 1745450

Summary:	ipa-server-install fails with RuntimeError: fork not supported for subinterpreters
Product:	[Fedora] Fedora	Reporter:	Jan Pazdziora (Red Hat) <jpazdziora>
Component:	freeipa	Assignee:	IPA Maintainers <ipa-maint>
Status:	CLOSED RAWHIDE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	abokovoy, awilliam, cheimes, contribs, fcami, ipa-maint, jcholast, jhrozek, lslebodn, pvoborni, rcritten, robatino, ssorce, twoerner, vstinner
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	openqa
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-09-02 08:53:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1745894
Bug Blocks:	1705303

Description Jan Pazdziora (Red Hat) 2019-08-26 07:40:01 UTC

Description of problem:

Running ipa-server-install end in Configuring client side components phase with error and segfault.

Version-Release number of selected component (if applicable):

freeipa-server-4.8.1-2.fc32.x86_64
in Fedora-Cloud-Base-Rawhide-20190825.n.0.x86_64.qcow2 VM

How reproducible:

Deterministic.

Steps to Reproduce:
1. ipa-server-install -U -r EXAMPLE.TEST -p Secret123 -a Secret123

Actual results:

  [7/10]: upgrading server
  [8/10]: stopping directory server
  [9/10]: restoring configuration
  [10/10]: starting directory server
Done.
Restarting the KDC
Configuring client side components
This program will set up FreeIPA client.
Version 4.8.1

Using existing certificate '/etc/ipa/ca.crt'.
Client hostname: freeipa.example.test
Realm: EXAMPLE.TEST
DNS Domain: example.test
IPA Server: freeipa.example.test
BaseDN: dc=example,dc=test

Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
cannot connect to 'https://freeipa.example.test/ipa/json': Internal Server Error
The ipa-client-install command failed. See /var/log/ipaclient-install.log for more information
Configuration of client side components failed!
The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information
Segmentation fault (core dumped)

Expected results:

No error.

Additional info:

The /var/log/ipaclient-install.log ends with

2019-08-26T07:31:51Z DEBUG New HTTP connection (freeipa.example.test)
2019-08-26T07:31:51Z DEBUG HTTP connection destroyed (freeipa.example.test)
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/ipaclient/remote_plugins/__init__.py", line 126, in get_package
    plugins = api._remote_plugins
AttributeError: 'API' object has no attribute '_remote_plugins'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/ipalib/rpc.py", line 727, in single_request
    raise ProtocolError(
xmlrpc.client.ProtocolError: <ProtocolError for freeipa.example.test/ipa/json: 500 Internal Server Error>
2019-08-26T07:31:51Z DEBUG Destroyed connection context.rpcclient_140366869366912
2019-08-26T07:31:51Z DEBUG   File "/usr/lib/python3.8/site-packages/ipapython/admintool.py", line 179, in execute
    return_value = self.run()
  File "/usr/lib/python3.8/site-packages/ipapython/install/cli.py", line 340, in run
    return cfgr.run()
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 360, in run
    return self.execute()
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 386, in execute
    for rval in self._executor():
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.8/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.8/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.8/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.8/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 655, in _configure
    next(executor)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.8/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 515, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.8/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.8/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.8/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.8/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.8/site-packages/ipapython/install/common.py", line 65, in _install
    for unused in self._installer(self.parent):
  File "/usr/lib/python3.8/site-packages/ipaclient/install/client.py", line 3842, in main
    install(self)
  File "/usr/lib/python3.8/site-packages/ipaclient/install/client.py", line 2555, in install
    _install(options)
  File "/usr/lib/python3.8/site-packages/ipaclient/install/client.py", line 2867, in _install
    api.finalize()
  File "/usr/lib/python3.8/site-packages/ipalib/plugable.py", line 741, in finalize
    self.__do_if_not_done('load_plugins')
  File "/usr/lib/python3.8/site-packages/ipalib/plugable.py", line 429, in __do_if_not_done
    getattr(self, name)()
  File "/usr/lib/python3.8/site-packages/ipalib/plugable.py", line 621, in load_plugins
    for package in self.packages:
  File "/usr/lib/python3.8/site-packages/ipalib/__init__.py", line 949, in packages
    ipaclient.remote_plugins.get_package(self),
  File "/usr/lib/python3.8/site-packages/ipaclient/remote_plugins/__init__.py", line 134, in get_package
    plugins = schema.get_package(server_info, client)
  File "/usr/lib/python3.8/site-packages/ipaclient/remote_plugins/schema.py", line 553, in get_package
    schema = Schema(client)
  File "/usr/lib/python3.8/site-packages/ipaclient/remote_plugins/schema.py", line 402, in __init__
    fingerprint, ttl = self._fetch(client, ignore_cache=read_failed)
  File "/usr/lib/python3.8/site-packages/ipaclient/remote_plugins/schema.py", line 427, in _fetch
    schema = client.forward(u'schema', **kwargs)['result']
  File "/usr/lib/python3.8/site-packages/ipalib/rpc.py", line 1188, in forward
    raise NetworkError(uri=server, error=e.errmsg)

2019-08-26T07:31:51Z DEBUG The ipa-client-install command failed, exception: NetworkError: cannot connect to 'https://freeipa.example.test/ipa/json': Internal Server Error
2019-08-26T07:31:51Z ERROR cannot connect to 'https://freeipa.example.test/ipa/json': Internal Server Error
2019-08-26T07:31:51Z ERROR The ipa-client-install command failed. See /var/log/ipaclient-install.log for more information

The /var/log/httpd/error_log ends with

[Mon Aug 26 07:31:51.754632 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768] mod_wsgi (pid=15932): Failed to exec Python script file '/usr/share/ipa/wsgi.py'.
[Mon Aug 26 07:31:51.754701 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768] mod_wsgi (pid=15932): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
[Mon Aug 26 07:31:51.754968 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768] Traceback (most recent call last):
[Mon Aug 26 07:31:51.755039 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/share/ipa/wsgi.py", line 44, in <module>
[Mon Aug 26 07:31:51.755052 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from ipalib import api
[Mon Aug 26 07:31:51.755066 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib/python3.8/site-packages/ipalib/__init__.py", line 919, in <module>
[Mon Aug 26 07:31:51.755076 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from ipalib import plugable
[Mon Aug 26 07:31:51.755089 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib/python3.8/site-packages/ipalib/plugable.py", line 41, in <module>
[Mon Aug 26 07:31:51.755100 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from ipalib import errors
[Mon Aug 26 07:31:51.755112 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib/python3.8/site-packages/ipalib/errors.py", line 109, in <module>
[Mon Aug 26 07:31:51.755122 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from ipalib.text import ngettext as ungettext
[Mon Aug 26 07:31:51.755135 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib/python3.8/site-packages/ipalib/text.py", line 139, in <module>
[Mon Aug 26 07:31:51.755145 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from ipalib.request import context
[Mon Aug 26 07:31:51.755158 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib/python3.8/site-packages/ipalib/request.py", line 28, in <module>
[Mon Aug 26 07:31:51.755212 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from ipalib.base import ReadOnly, lock
[Mon Aug 26 07:31:51.755232 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib/python3.8/site-packages/ipalib/base.py", line 26, in <module>
[Mon Aug 26 07:31:51.755243 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from ipalib.constants import NAME_REGEX, NAME_ERROR
[Mon Aug 26 07:31:51.755275 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib/python3.8/site-packages/ipalib/constants.py", line 27, in <module>
[Mon Aug 26 07:31:51.755284 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from ipapython.dn import DN
[Mon Aug 26 07:31:51.755295 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib/python3.8/site-packages/ipapython/dn.py", line 425, in <module>
[Mon Aug 26 07:31:51.755304 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     import cryptography.x509
[Mon Aug 26 07:31:51.755315 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib64/python3.8/site-packages/cryptography/x509/__init__.py", line 8, in <module>
[Mon Aug 26 07:31:51.755324 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from cryptography.x509.base import (
[Mon Aug 26 07:31:51.755335 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib64/python3.8/site-packages/cryptography/x509/base.py", line 16, in <module>
[Mon Aug 26 07:31:51.755345 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from cryptography.x509.extensions import Extension, ExtensionType
[Mon Aug 26 07:31:51.755356 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib64/python3.8/site-packages/cryptography/x509/extensions.py", line 13, in <module>
[Mon Aug 26 07:31:51.755365 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from asn1crypto.keys import PublicKeyInfo
[Mon Aug 26 07:31:51.755376 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib/python3.8/site-packages/asn1crypto/keys.py", line 22, in <module>
[Mon Aug 26 07:31:51.755385 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from ._elliptic_curve import (
[Mon Aug 26 07:31:51.755396 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib/python3.8/site-packages/asn1crypto/_elliptic_curve.py", line 51, in <module>
[Mon Aug 26 07:31:51.755405 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from ._int import inverse_mod
[Mon Aug 26 07:31:51.755415 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib/python3.8/site-packages/asn1crypto/_int.py", line 56, in <module>
[Mon Aug 26 07:31:51.755425 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     from ._perf._big_num_ctypes import libcrypto
[Mon Aug 26 07:31:51.755436 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib/python3.8/site-packages/asn1crypto/_perf/_big_num_ctypes.py", line 35, in <module>
[Mon Aug 26 07:31:51.755445 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     libcrypto_path = find_library(b'crypto' if sys.version_info < (3,) else 'crypto')
[Mon Aug 26 07:31:51.755456 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib64/python3.8/ctypes/util.py", line 311, in find_library
[Mon Aug 26 07:31:51.755465 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     return _findSoname_ldconfig(name) or \\
[Mon Aug 26 07:31:51.755476 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib64/python3.8/ctypes/util.py", line 276, in _findSoname_ldconfig
[Mon Aug 26 07:31:51.755485 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     with subprocess.Popen(['/sbin/ldconfig', '-p'],
[Mon Aug 26 07:31:51.755496 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib64/python3.8/subprocess.py", line 829, in __init__
[Mon Aug 26 07:31:51.755525 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     self._execute_child(args, executable, preexec_fn, close_fds,
[Mon Aug 26 07:31:51.755539 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]   File "/usr/lib64/python3.8/subprocess.py", line 1608, in _execute_child
[Mon Aug 26 07:31:51.755548 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768]     self.pid = _posixsubprocess.fork_exec(
[Mon Aug 26 07:31:51.755567 2019] [wsgi:error] [pid 15932:tid 139742924355328] [remote 192.168.100.211:37768] RuntimeError: fork not supported for subinterpreters

Comment 1 François Cami 2019-08-26 07:48:27 UTC

Looks like https://bugs.python.org/issue34651

Comment 2 Christian Heimes 2019-08-26 08:44:08 UTC

The issue is caused by changes described in https://bugs.python.org/issue34651. I have created a new BPO https://bugs.python.org/issue37951 to track the issue.

Victor, please take a look. This issue breaks and blocks FreeIPA on Python 3.8 / Fedora rawhide.

Comment 3 Adam Williamson 2019-08-30 18:51:42 UTC

Yeah, openQA hits this too (I just didn't get to reporting it till today as I've been prioritizing F31).

https://openqa.fedoraproject.org/tests/438706#next_previous (fails each compose since 0828.n.0, which is when Python 3.8 landed).

This is an obvious F32 Beta blocker, per Basic criterion "It must be possible to configure a Fedora Server system installed according to the above criteria as a FreeIPA domain controller, using the official deployment tools provided in the distribution FreeIPA packages. Once deployed, the system must handle multiple client enrolments and unenrolments, and client authentication via Kerberos. The web UI must be available..." - https://fedoraproject.org/wiki/Basic_Release_Criteria#FreeIPA_server_requirements

Comment 4 Christian Heimes 2019-08-30 21:47:16 UTC

Fix has landed in master and 3.8, https://github.com/python/cpython/commit/98d90f745d35d5d07bffcb46788b50e05eea56c6

3.8b4 with the fix was tagged yesterday and release today. The Python maintenance team is super fast. I expect to see 3.8b4 in rawhide early next week.

Comment 5 Christian Heimes 2019-09-02 08:53:52 UTC

python3-3.8.0~b4-1.fc32.x86_64 fixed the issue.