Bug 2007664 - Fedora Python imports lots of additional modules on startup
Summary: Fedora Python imports lots of additional modules on startup
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: abrt
Version: 35
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: abrt
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedFreezeException
Depends On:
Blocks: F35FinalFreezeException
TreeView+ depends on / blocked
 
Reported: 2021-09-24 14:09 UTC by Christian Heimes
Modified: 2021-10-07 23:31 UTC (History)
15 users (show)

Fixed In Version: abrt-2.14.6-4.fc34 abrt-2.14.6-9.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-07 23:31:47 UTC
Type: Bug


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github abrt abrt pull 1580 0 None open Use lazy imports in abrt_exception_handler3 2021-09-24 15:48:39 UTC

Description Christian Heimes 2021-09-24 14:09:57 UTC
Description of problem:
A Python interpreter on Fedora imports lots of additional modules and packages. On my Fedora system a fresh interpreter has 104 entries in sys.modules. For comparision an upstream Python 3.9 process as well as Python 3.9 in podman container image only has 33 entries.

Any additional imports slows down startup and has a negative impact on general performance. Especially command line tools are affected.

The largest culprict is python3-abrt-addon. It increases the amount of loaded modules by 57 (!) to 90 total.


Version-Release number of selected component (if applicable):
python3-3.9.7-1.fc34.x86_64
python3-abrt-addon-2.14.6-3.fc34.noarch

How reproducible:
always

Steps to Reproduce:
1. python3 -c "import sys; print(len(sys.modules))"
2. python3 -c "import sys; print(sorted(sys.modules))"

Actual results:
104

['__future__', '__main__', '_abc', '_bootlocale', '_codecs', '_collections', '_collections_abc', '_datetime', '_frozen_importlib', '_frozen_importlib_external', '_functools', '_heapq', '_imp', '_io', '_locale', '_operator', '_posixsubprocess', '_signal', '_sitebuiltins', '_sre', '_stat', '_string', '_thread', '_uuid', '_warnings', '_weakref', '_weakrefset', 'abc', 'abrt_exception_handler3', 'atexit', 'bkr', 'builtins', 'codecs', 'collections', 'collections.abc', 'contextlib', 'copyreg', 'datetime', 'encodings', 'encodings.aliases', 'encodings.latin_1', 'encodings.utf_8', 'enum', 'errno', 'functools', 'genericpath', 'google', 'grp', 'heapq', 'importlib', 'importlib._bootstrap', 'importlib._bootstrap_external', 'importlib.abc', 'importlib.machinery', 'importlib.util', 'io', 'itertools', 'keyword', 'linecache', 'logging', 'marshal', 'math', 'operator', 'os', 'os.path', 'paste', 'platform', 'posix', 'posixpath', 'pwd', 're', 'reprlib', 'ruamel', 'select', 'selectors', 'signal', 'site', 'sphinxcontrib', 'sre_compile', 'sre_constants', 'sre_parse', 'stat', 'string', 'subprocess', 'sys', 'syslog', 'systemd', 'systemd._journal', 'systemd._reader', 'systemd.id128', 'systemd.journal', 'threading', 'time', 'token', 'tokenize', 'traceback', 'types', 'typing', 'typing.io', 'typing.re', 'uuid', 'warnings', 'weakref', 'zipimport']


Expected results:

I expect a smaller number of modules that gets import by default

Additional info:

$ podman run -ti fedora:34

# python3 -c "import sys; print(len(sys.modules))"
33
# python3 -c "import sys; print(sorted(sys.modules))"
['__main__', '_abc', '_codecs', '_collections_abc', '_frozen_importlib', '_frozen_importlib_external', '_imp', '_io', '_signal', '_sitebuiltins', '_stat', '_thread', '_warnings', '_weakref', 'abc', 'builtins', 'codecs', 'encodings', 'encodings.aliases', 'encodings.latin_1', 'encodings.utf_8', 'genericpath', 'io', 'marshal', 'os', 'os.path', 'posix', 'posixpath', 'site', 'stat', 'sys', 'time', 'zipimport']

# dnf install -y python3-abrt-addon

# python3 -c "import sys; print(len(sys.modules))"
90
# python3 -c "import sys; print(sorted(sys.modules))"
['__future__', '__main__', '_abc', '_bootlocale', '_codecs', '_collections', '_collections_abc', '_datetime', '_frozen_importlib', '_frozen_importlib_external', '_functools', '_heapq', '_imp', '_io', '_locale', '_operator', '_posixsubprocess', '_signal', '_sitebuiltins', '_sre', '_stat', '_string', '_thread', '_uuid', '_warnings', '_weakref', '_weakrefset', 'abc', 'abrt_exception_handler3', 'atexit', 'builtins', 'codecs', 'collections', 'collections.abc', 'contextlib', 'copyreg', 'datetime', 'encodings', 'encodings.aliases', 'encodings.latin_1', 'encodings.utf_8', 'enum', 'errno', 'functools', 'genericpath', 'grp', 'heapq', 'io', 'itertools', 'keyword', 'linecache', 'logging', 'marshal', 'math', 'operator', 'os', 'os.path', 'platform', 'posix', 'posixpath', 'pwd', 're', 'reprlib', 'select', 'selectors', 'signal', 'site', 'sre_compile', 'sre_constants', 'sre_parse', 'stat', 'string', 'subprocess', 'sys', 'syslog', 'systemd', 'systemd._journal', 'systemd._reader', 'systemd.id128', 'systemd.journal', 'threading', 'time', 'token', 'tokenize', 'traceback', 'types', 'uuid', 'warnings', 'weakref', 'zipimport']

Comment 1 Charalampos Stratakis 2021-09-24 14:35:41 UTC
Changing the component to abrt to get feedback from the maintainers.

Comment 2 Miro Hrončok 2021-09-24 14:58:37 UTC
/usr/lib/python3.9/site-packages/abrt3.pth has:

import abrt_exception_handler3


/usr/lib/python3.9/site-packages/abrt_exception_handler3.py has:

import sys
import os

from systemd import journal

I think ideally, abrt_exception_handler3 could lazy import:



Even this helps a lot:

def syslog(msg):
    """Log message to system logger (journal)"""
    from systemd import journal
    journal.send(msg)



On rawhide:

 - no python3-abrt-addon installed: 32 modules
 - python3-abrt-addon installed: 85 modules
 - python3-abrt-addon with the modification from above: 33 modules

The plugin could even do the right thing and module-level importing os, but that currently does not make any difference. My diff:

@@ -21,13 +21,11 @@
 """
 
 import sys
-import os
 
-from systemd import journal
 
 def syslog(msg):
     """Log message to system logger (journal)"""
-
+    from systemd import journal
     journal.send(msg)
 
 
@@ -68,6 +66,8 @@
 
 
 def write_dump(tb_text, tb):
+    import os
+
     if sys.argv[0][0] == "/":
         executable = os.path.abspath(sys.argv[0])
     else:
@@ -117,7 +117,7 @@
         # Restore original exception handler
         sys.excepthook = sys.__excepthook__  # pylint: disable-msg=E1101
 
-        import errno
+        import errno, os
 
         # Ignore Ctrl-C
         # SystemExit rhbz#636913 -> this exception is not an error

Comment 3 Miro Hrončok 2021-09-24 15:07:15 UTC
Christian, as for the other 14 modules imported on your workstation:

 'bkr',
 'google',
 'importlib',
 'importlib._bootstrap',
 'importlib._bootstrap_external',
 'importlib.abc',
 'importlib.machinery',
 'importlib.util',
 'paste',
 'ruamel',
 'sphinxcontrib',
 'typing',
 'typing.io',
 'typing.re'

I believe those are caused by some old-school namespace packages with files like /usr/lib/python3.9/site-packages/ruamel.yaml-0.16.6-py3.9-nspkg.pth:

import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('ruamel',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('ruamel', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('ruamel', [os.path.dirname(p)])));m = m or sys.modules.setdefault('ruamel', types.ModuleType('ruamel'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)


Not much we can do about those except talking to upstream not to do this anymore.

Comment 4 Miro Hrončok 2021-09-24 15:11:29 UTC
> The plugin could even do the right thing and module-level importing os...

Correction: And **stop** module-level importing os...

Comment 5 Miro Hrončok 2021-09-24 15:48:39 UTC
Upstream PR: https://github.com/abrt/abrt/pull/1580

Comment 6 Matej Grabovsky 2021-09-27 10:14:37 UTC
Pull request merged. The change will appear in the next release of abrt.

Comment 7 Miro Hrončok 2021-09-27 10:18:52 UTC
(In reply to Matej Grabovsky from comment #6)
> The change will appear in the next release of abrt.


Will that happen in time for Fedora 35 final freeze? If not, I'd rather see that backported.

Comment 8 Matej Grabovsky 2021-09-27 10:26:59 UTC
OK, I'll push a backport for active releases.

Comment 9 Miro Hrončok 2021-09-27 10:33:59 UTC
Thanks!

Comment 10 Fedora Update System 2021-09-27 11:01:45 UTC
FEDORA-2021-db079c289a has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-db079c289a

Comment 11 Fedora Update System 2021-09-29 02:23:01 UTC
FEDORA-2021-db079c289a has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-db079c289a`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-db079c289a

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 14 Fedora Update System 2021-10-06 17:47:01 UTC
FEDORA-2021-db079c289a has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 15 Miro Hrončok 2021-10-06 18:48:17 UTC
This has still not reached F35. I'd really like to see it on the media.

Comment 16 Fedora Update System 2021-10-06 18:48:32 UTC
FEDORA-2021-418a00501f has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-418a00501f

Comment 17 Adam Williamson 2021-10-07 23:19:17 UTC
+3 in https://pagure.io/fedora-qa/blocker-review/issue/519 , marking accepted.

Comment 18 Fedora Update System 2021-10-07 23:31:47 UTC
FEDORA-2021-418a00501f has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.