Bug 1791761

Summary: elements fails to build with Python 3.9: Segmentation fault in %check
Product: [Fedora] Fedora Reporter: Miro Hrončok <mhroncok>
Component: elementsAssignee: Petr Viktorin (pviktori) <pviktori>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: a.alvarezayllon, cstratak, Hubert.Degaudenzi, mhroncok, pviktori, vstinner
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-30 11:05:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1785415    

Description Miro Hrončok 2020-01-16 12:41:19 UTC
elements fails to build with Python 3.9.0a2.

+ /builddir/build/BUILDROOT/elements-5.8-4.fc32.x86_64//usr/bin/CreateElementsProject --help
usage: CreateElementsProject [-h] [-d project_name version] [-n] [-e] [-s]
                             [-y] [--config-file CONFIG_FILE]
                             [--log-file LOG_FILE] [--log-level LOG_LEVEL]
                             project-name project-version

This script creates an <Elements> project in your current directory(by default)
Generic Options:
  --config-file CONFIG_FILE
                        Name of a configuration file
  --log-file LOG_FILE   Name of a log file
  --log-level LOG_LEVEL
                        Log level: FATAL, ERROR, WARN, INFO (default), DEBUG
  --version             show program's version number and exit
/var/tmp/rpm-tmp.quucqt: line 33:  3330 Segmentation fault      (core dumped) /builddir/build/BUILDROOT/elements-5.8-4.fc32.x86_64//usr/bin/CreateElementsProject --help

For the build logs, see:

For all our attempts to build elements with Python 3.9, see:

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9:

Let us know here if you have any questions.

Python 3.9 will be included in Fedora 33. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.9.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Alejandro Alvarez 2020-01-16 14:47:56 UTC

I can reproduce, and I know where the issue is being triggered, but I am not sure it is a bug on this library, though:

So, there is a class called ElementsKernel.Temporary.Environment, which is to be used as os.environ, with the difference that it automatically resets the environment when is destroyed/exited: it defines __exit__ and __del__


os.environ["FOO"] = "zzz"
with Environment() as env:
    env["FOO"] = "yyy"
    print(os.environ["FOO"]) # print yyy
print(os.environ["FOO"]) # print zzz

CreateElementsProject uses it itself, on a class that is instantiated just at the beginning.

If I comment out the __del__ method, there is no segfault.

So it looks to me as if Python is calling this __del__, which tries to reset the environment, which seems to be somehow already freed (or pointing somewhere that has been).
This is consistent with the backtrace I get

#1  0x00007ffff7c86041 in posix_putenv_garbage_setitem (
    value=b'PATH=/home/aalvarez/Work/Projects/Elements/develop/build.x86_64-fc32-gcc92-dbg/scripts:/home/aalvarez/Work/Projects/Elements/develop/build.x86_64-fc32-gcc92-dbg/bin:/home/aalvarez/Work/Projects/Elements/develop/ElementsExamples/scripts:/home/aalvarez/Work/Projects/Elements/develop/ElementsKernel/scripts:/usr/sbin:/usr/bin:/sbin:/bin', name=b'PATH')
    at /usr/src/debug/python3-3.9.0~a2-1.fc32.x86_64/Modules/posixmodule.c:10159
#2  os_putenv_impl (module=<optimized out>, 
    value=b'/home/aalvarez/Work/Projects/Elements/develop/build.x86_64-fc32-gcc92-dbg/scripts:/home/aalvarez/Work/Projects/Elements/develop/build.x86_64-fc32-gcc92-dbg/bin:/home/aalvarez/Work/Projects/Elements/develop/ElementsExamples/scripts:/home/aalvarez/Work/Projects/Elements/develop/ElementsKernel/scripts:/usr/sbin:/usr/bin:/sbin:/bin', name=b'PATH')
    at /usr/src/debug/python3-3.9.0~a2-1.fc32.x86_64/Modules/posixmodule.c:10159
#3  os_putenv (module=<optimized out>, args=<optimized out>, nargs=<optimized out>) at /usr/src/debug/python3-3.9.0~a2-1.fc32.x86_64/Modules/clinic/posixmodule.c.h:6115

I am guessing this is coming from this commit?

Comment 2 Miro Hrončok 2020-01-16 16:01:41 UTC
(In reply to Alejandro Alvarez from comment #1)
> I am guessing this is coming from this commit?
> https://github.com/python/cpython/commit/
> b3966639d28313809774ca3859a347b9007be8d2#diff-
> a6f29e907cbb5fffd44d453bcd7b77d5


Comment 3 Petr Viktorin (pviktori) 2020-01-16 17:21:14 UTC
It definitely looks like something I should look into. Unfortunately I'll not get to this bug until the last week of January. Let me know if you need it fixed sooner.

Alejandro, thanks for the investigation!

Comment 4 Petr Viktorin (pviktori) 2020-01-17 20:32:01 UTC
It looks similar to https://bugs.python.org/issue38076

Comment 5 Victor Stinner 2020-01-20 12:28:50 UTC
I'm unable to reproduce the issue. I hacked Python to display the value of an environment variable at exit after its internal posix_putenv_garbage dictionary is cleared: getenv() doesn't crash. Even if the memory has been filled with 0xBB byte pattern, even if the memory has been released, getenv() does not always crash: it seems still possible to access the memory sometimes.

Anyway, I understood the issue: Python releases the memory but it doesn't tell the glibc that the variable has been deleted.

I reported the bug to Python upstream https://bugs.python.org/issue39395 and I proposed https://github.com/python/cpython/pull/18078 to fix it.

Comment 6 Petr Viktorin (pviktori) 2020-01-30 11:05:26 UTC
The newest build with Python 3.9.0~a3 succeeded!

Thank you, Victor and Alejandro, for looking into this issue.

Comment 7 Victor Stinner 2020-01-30 12:49:20 UTC
> Thank you, Victor and Alejandro, for looking into this issue.

You're welcome. FYI I fixed Python by modifying os.putenv() implementation: it's now implemented with setenv() instead of putenv(). putenv() has a really strange behavior: the caller has to handle the memory, whereas Python released this memory at exit.