During FreeIPA server deployment on Fedora 26 and Fedora Rawhide, the 'ipa-custodia' service keeps failing, with these errors: Mar 07 12:52:57 ipa001.domain.local systemd[1]: ipa-custodia.service: Service hold-off time over, scheduling restart. Mar 07 12:52:58 ipa001.domain.local system-python[9430]: detected unhandled Python exception in '/usr/sbin/custodia' Mar 07 12:52:58 ipa001.domain.local custodia[9430]: Traceback (most recent call last): Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "/usr/lib/python3.6/site-packages/custodia/server/__init__.py", line 86, in _create_plugin Mar 07 12:52:58 ipa001.domain.local custodia[9430]: handler = _load_plugin_class(menu, handler_name) Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "/usr/lib/python3.6/site-packages/custodia/server/__init__.py", line 73, in _load_plugin_class Mar 07 12:52:58 ipa001.domain.local custodia[9430]: m = importlib.import_module(module) Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module Mar 07 12:52:58 ipa001.domain.local custodia[9430]: return _bootstrap._gcd_import(name[level:], package, level) Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "<frozen importlib._bootstrap>", line 978, in _gcd_import Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "<frozen importlib._bootstrap>", line 961, in _find_and_load Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "<frozen importlib._bootstrap>", line 936, in _find_and_load_unlocked Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "<frozen importlib._bootstrap>", line 978, in _gcd_import Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "<frozen importlib._bootstrap>", line 961, in _find_and_load Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "<frozen importlib._bootstrap>", line 936, in _find_and_load_unlocked Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "<frozen importlib._bootstrap>", line 978, in _gcd_import Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "<frozen importlib._bootstrap>", line 961, in _find_and_load Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "<frozen importlib._bootstrap>", line 948, in _find_and_load_unlocked Mar 07 12:52:58 ipa001.domain.local custodia[9430]: ModuleNotFoundError: No module named 'ipapython' Mar 07 12:52:58 ipa001.domain.local custodia[9430]: During handling of the above exception, another exception occurred: Mar 07 12:52:58 ipa001.domain.local custodia[9430]: Traceback (most recent call last): Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "/usr/lib/python3.6/site-packages/custodia/server/__init__.py", line 181, in parse_config Mar 07 12:52:58 ipa001.domain.local custodia[9430]: config[menu][name] = _create_plugin(parser, s, menu) Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "/usr/lib/python3.6/site-packages/custodia/server/__init__.py", line 91, in _create_plugin Mar 07 12:52:58 ipa001.domain.local custodia[9430]: '[%r]: %s' % (e, handler_name)) Mar 07 12:52:58 ipa001.domain.local custodia[9430]: ValueError: Invalid format for "handler" option [ModuleNotFoundError("No module named 'ipapython'",)]: ipapython.secrets.kem.IPAKEMKeys Mar 07 12:52:58 ipa001.domain.local custodia[9430]: During handling of the above exception, another exception occurred: Mar 07 12:52:58 ipa001.domain.local custodia[9430]: Traceback (most recent call last): Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "/usr/sbin/custodia", line 11, in <module> Mar 07 12:52:58 ipa001.domain.local custodia[9430]: load_entry_point('custodia==0.3.0', 'console_scripts', 'custodia')() Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "/usr/lib/python3.6/site-packages/custodia/server/__init__.py", line 198, in main Mar 07 12:52:58 ipa001.domain.local custodia[9430]: config = parse_config(args) Mar 07 12:52:58 ipa001.domain.local custodia[9430]: File "/usr/lib/python3.6/site-packages/custodia/server/__init__.py", line 183, in parse_config Mar 07 12:52:58 ipa001.domain.local custodia[9430]: raise RuntimeError(menu, name, e) Mar 07 12:52:58 ipa001.domain.local custodia[9430]: RuntimeError: ('authorizers', 'kemkeys', ValueError('Invalid format for "handler" option [ModuleNotFoundError("No module named \'ipapython\'",)]: ipapython.secrets.kem.IPAKEMKeys',)) Mar 07 12:52:58 ipa001.domain.local systemd[1]: ipa-custodia.service: Main process exited, code=exited, status=1/FAILURE Mar 07 12:52:58 ipa001.domain.local systemd[1]: ipa-custodia.service: Unit entered failed state. Mar 07 12:52:58 ipa001.domain.local systemd[1]: ipa-custodia.service: Failed with result 'exit-code'. The problem is pretty obvious: the 'ipapython' library is missing. This library is provided by python3-ipalib, but python3-custodia does not depend on python3-ipalib (either the 0.2.0 build in F26, or the 0.3.0 build in Rawhide). Proposing as at least a freeze exception for F26 Alpha, it may be a blocker (I'm not sure how critical ipa-custodia is to FreeIPA functionality).
python3-custodia does not depend on python3-ipalib. Only the specialized Custodia instance of FreeIPA loads an additional plugin that is provided by ipapython and ipaserver. The ipa-custodia is critical for replica install. For FreeIPA 4.4 and Custodia 0.2, ipa-custodia must be run with Python 2. Once https://github.com/freeipa/freeipa/pull/517 has landed, it's all handled by FreeIPA internally.
Better?
Christian, I don't get how PR 517 should fix F26. Custodia 0.3 is built only for Fedora 27 and not Fedora 26. So neither FreeIPA 4.4 nor 4.5(master) will be fixed by it and it will only says that we are missing dependencies. Or do you plan to add Custodia 0.3 to Fedora 26?
Can someone say whether this is a critical issue - i.e. whether FreeIPA will fail to work correctly without ipa-custodia? I'm not sure what it's for. Thanks!
I think it is a critical issue. Christian is traveling right now, so his response will be delayed.
Adam, which version of Custodia and FreeIPA are you running? The Fedora 26 package should still default to Python 2. I don't understand why your system runs Custodia with Python 3. $ rpm -qa custodia freeipa-server freeipa-server-4.4.3-8.fc26.x86_64 custodia-0.2.0-3.fc26.noarch $ head -n1 /usr/sbin/custodia #!/usr/bin/python2 $ grep ExecStart /usr/lib/systemd/system/ipa-custodia.service ExecStart=/usr/sbin/custodia /etc/ipa/custodia/custodia.conf
Discussed in today's Blocker Review meeting. It'd be really good to get a fix in for this before Alpha release. Please update the bug if this issue is more severe than we currently suspect.
Note, in further discussion between me and sgallagh, it transpires that this issue may not affect F26. It may only affect 0.3.0-1+, which are only in Rawhide. F26 tests have been running into *other* issues, so we haven't reached this point in the F26 tests yet. Once the other issues are resolved, I'll check if this is really affect current F26.
This indeed doesn't seem to affect F26 (at present), so switching to Rawhide and dropping Alpha FE status.
Upstream ticket: https://pagure.io/freeipa/issue/5825
Fixed upstream ipa-4-5: https://pagure.io/freeipa/c/403263df7a3be61086c87c5577698cf32a912065 master: https://pagure.io/freeipa/c/f5bf5466eda0de2a211b4f2682e5c50b82577701
freeipa-4.4.4-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0d0ec3e0af
freeipa-4.4.4-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-0d0ec3e0af
custodia-0.3.1-2.fc26 freeipa-4.4.4-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0d0ec3e0af
custodia-0.3.1-2.fc26, freeipa-4.4.4-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-0d0ec3e0af
custodia-0.3.1-2.fc26, freeipa-4.4.4-2.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.