Bug 1791761 - elements fails to build with Python 3.9: Segmentation fault in %check
Summary: elements fails to build with Python 3.9: Segmentation fault in %check
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: elements
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Petr Viktorin
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON39
TreeView+ depends on / blocked
 
Reported: 2020-01-16 12:41 UTC by Miro Hrončok
Modified: 2020-01-31 08:00 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-30 11:05:26 UTC
Type: Bug


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Python 39395 0 None None None 2020-01-20 12:28:50 UTC

Description Miro Hrončok 2020-01-16 12:41:19 UTC
elements fails to build with Python 3.9.0a2.

+ /builddir/build/BUILDROOT/elements-5.8-4.fc32.x86_64//usr/bin/CreateElementsProject --help
usage: CreateElementsProject [-h] [-d project_name version] [-n] [-e] [-s]
                             [-y] [--config-file CONFIG_FILE]
                             [--log-file LOG_FILE] [--log-level LOG_LEVEL]
                             [--version]
                             project-name project-version

This script creates an <Elements> project in your current directory(by default)
(...snip...)
Generic Options:
  --config-file CONFIG_FILE
                        Name of a configuration file
  --log-file LOG_FILE   Name of a log file
  --log-level LOG_LEVEL
                        Log level: FATAL, ERROR, WARN, INFO (default), DEBUG
  --version             show program's version number and exit
/var/tmp/rpm-tmp.quucqt: line 33:  3330 Segmentation fault      (core dumped) /builddir/build/BUILDROOT/elements-5.8-4.fc32.x86_64//usr/bin/CreateElementsProject --help



For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01148386-elements/

For all our attempts to build elements with Python 3.9, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/elements/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Let us know here if you have any questions.

Python 3.9 will be included in Fedora 33. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.9.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Alejandro Alvarez 2020-01-16 14:47:56 UTC
Hi,

I can reproduce, and I know where the issue is being triggered, but I am not sure it is a bug on this library, though:

So, there is a class called ElementsKernel.Temporary.Environment, which is to be used as os.environ, with the difference that it automatically resets the environment when is destroyed/exited: it defines __exit__ and __del__

i.e


os.environ["FOO"] = "zzz"
with Environment() as env:
    env["FOO"] = "yyy"
    print(os.environ["FOO"]) # print yyy
print(os.environ["FOO"]) # print zzz



CreateElementsProject uses it itself, on a class that is instantiated just at the beginning.

If I comment out the __del__ method, there is no segfault.

So it looks to me as if Python is calling this __del__, which tries to reset the environment, which seems to be somehow already freed (or pointing somewhere that has been).
This is consistent with the backtrace I get

#1  0x00007ffff7c86041 in posix_putenv_garbage_setitem (
    value=b'PATH=/home/aalvarez/Work/Projects/Elements/develop/build.x86_64-fc32-gcc92-dbg/scripts:/home/aalvarez/Work/Projects/Elements/develop/build.x86_64-fc32-gcc92-dbg/bin:/home/aalvarez/Work/Projects/Elements/develop/ElementsExamples/scripts:/home/aalvarez/Work/Projects/Elements/develop/ElementsKernel/scripts:/usr/sbin:/usr/bin:/sbin:/bin', name=b'PATH')
    at /usr/src/debug/python3-3.9.0~a2-1.fc32.x86_64/Modules/posixmodule.c:10159
#2  os_putenv_impl (module=<optimized out>, 
    value=b'/home/aalvarez/Work/Projects/Elements/develop/build.x86_64-fc32-gcc92-dbg/scripts:/home/aalvarez/Work/Projects/Elements/develop/build.x86_64-fc32-gcc92-dbg/bin:/home/aalvarez/Work/Projects/Elements/develop/ElementsExamples/scripts:/home/aalvarez/Work/Projects/Elements/develop/ElementsKernel/scripts:/usr/sbin:/usr/bin:/sbin:/bin', name=b'PATH')
    at /usr/src/debug/python3-3.9.0~a2-1.fc32.x86_64/Modules/posixmodule.c:10159
#3  os_putenv (module=<optimized out>, args=<optimized out>, nargs=<optimized out>) at /usr/src/debug/python3-3.9.0~a2-1.fc32.x86_64/Modules/clinic/posixmodule.c.h:6115


I am guessing this is coming from this commit?
https://github.com/python/cpython/commit/b3966639d28313809774ca3859a347b9007be8d2#diff-a6f29e907cbb5fffd44d453bcd7b77d5

Comment 2 Miro Hrončok 2020-01-16 16:01:41 UTC
(In reply to Alejandro Alvarez from comment #1)
> I am guessing this is coming from this commit?
> https://github.com/python/cpython/commit/
> b3966639d28313809774ca3859a347b9007be8d2#diff-
> a6f29e907cbb5fffd44d453bcd7b77d5

Petr?

Comment 3 Petr Viktorin 2020-01-16 17:21:14 UTC
It definitely looks like something I should look into. Unfortunately I'll not get to this bug until the last week of January. Let me know if you need it fixed sooner.

Alejandro, thanks for the investigation!

Comment 4 Petr Viktorin 2020-01-17 20:32:01 UTC
It looks similar to https://bugs.python.org/issue38076

Comment 5 Victor Stinner 2020-01-20 12:28:50 UTC
I'm unable to reproduce the issue. I hacked Python to display the value of an environment variable at exit after its internal posix_putenv_garbage dictionary is cleared: getenv() doesn't crash. Even if the memory has been filled with 0xBB byte pattern, even if the memory has been released, getenv() does not always crash: it seems still possible to access the memory sometimes.

Anyway, I understood the issue: Python releases the memory but it doesn't tell the glibc that the variable has been deleted.

I reported the bug to Python upstream https://bugs.python.org/issue39395 and I proposed https://github.com/python/cpython/pull/18078 to fix it.

Comment 6 Petr Viktorin 2020-01-30 11:05:26 UTC
The newest build with Python 3.9.0~a3 succeeded!

Thank you, Victor and Alejandro, for looking into this issue.

Comment 7 Victor Stinner 2020-01-30 12:49:20 UTC
> Thank you, Victor and Alejandro, for looking into this issue.

You're welcome. FYI I fixed Python by modifying os.putenv() implementation: it's now implemented with setenv() instead of putenv(). putenv() has a really strange behavior: the caller has to handle the memory, whereas Python released this memory at exit.


Note You need to log in before you can comment on or make changes to this bug.