Bug 1715016 - clang fails to build with Python 3.8
Summary: clang fails to build with Python 3.8
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: python3
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Miro Hrončok
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 1728067
Blocks: PYTHON38
TreeView+ depends on / blocked
 
Reported: 2019-05-29 11:38 UTC by Miro Hrončok
Modified: 2019-09-02 08:25 UTC (History)
16 users (show)

Fixed In Version: python3-3.8.0~b4-1.fc32, clang-8.0.0-4.fc32
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-31 18:36:48 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Python 37140 None None None 2019-06-03 09:23:07 UTC

Description Miro Hrončok 2019-05-29 11:38:08 UTC
clang 8.0.0-3.fc30 fails to build with Python 3.8.0a4+:

[211/212] cd /builddir/build/BUILD/cfe-8.0.0.src/bindings/python && /usr/bin/cmake -E env CLANG_LIBRARY_PATH=/builddir/build/BUILD/cfe-8.0.0.src/_build/lib64 /usr/bin/python3 -m unittest discover
FAILED: bindings/python/tests/CMakeFiles/check-clang-python 
cd /builddir/build/BUILD/cfe-8.0.0.src/bindings/python && /usr/bin/cmake -E env CLANG_LIBRARY_PATH=/builddir/build/BUILD/cfe-8.0.0.src/_build/lib64 /usr/bin/python3 -m unittest discover
free(): double free detected in tcache 2
Child aborted
ninja: build stopped: subcommand failed.
+ false

Full logs at https://copr.fedorainfracloud.org/coprs/g/python/python3.8/build/916505/

Comment 1 serge_sans_paille 2019-05-31 17:08:22 UTC
I've been able to track this down to clang's handling of CxString. For some reason, the destructor of one of those is called twice upon __del__, so there may be some sharing here?

Comment 2 serge_sans_paille 2019-06-02 12:33:27 UTC
Bug introduced in cpython by commit 32119e10b792ad7ee4e5f951a2d89ddbaf111cc5, esp. the changes in Modules/_ctypes/_ctypes.c.

Comment 3 Miro Hrončok 2019-06-02 14:14:20 UTC
Is that a CPython bug or a juts something clang should be adapted to?

Comment 4 serge_sans_paille 2019-06-02 15:11:45 UTC
Hard to say. The problem triggers in clang code base around that function 

    https://github.com/llvm-mirror/clang/blob/release_80/bindings/python/clang/cindex.py#L230

Basically `from_result` is called through the `errcheck` field from `ctypes`.
The error is raised because `conf.lib.clang_getCString(res)` creates sharing between a field of `res` and the newly created object.

So my take on this would be: it's clang's fault, but I'd be happy if you could shed light on why it used to work prioro to that cpython commit?

Comment 5 Miro Hrončok 2019-06-02 22:37:54 UTC
> So my take on this would be: it's clang's fault, but I'd be happy if you could shed light on why it used to work prioro to that cpython commit?

I'm sorry but I don't know the answer for this. Perhaps Petr might know.

Comment 6 serge_sans_paille 2019-06-03 08:58:24 UTC
Last findings, in the following

```
    if (self->b_size > sizeof(void*)) {
        void *new_ptr = PyMem_Malloc(self->b_size);
        if (new_ptr == NULL)
            return NULL;
        memcpy(new_ptr, self->b_ptr, self->b_size);
        copied_self = (CDataObject *)PyCData_AtAddress(
            (PyObject *)Py_TYPE(self), new_ptr);
        copied_self->b_needsfree = 1;`
    } else {
        copied_self = self;
        Py_INCREF(copied_self);
    }
```

from ``StructUnionType_paramfunc``, if ``self->b_ptr`` contains pointer, the ``memcpy`` creates sharing, and this is dangerous: if a ``__del__`` happens to free the original pointer, we end up with a dangling reference in ``new_ptr``. As far as I can tell, this is what happens in the clang bindings code.

Comment 7 Petr Viktorin 2019-06-03 09:08:37 UTC
I'm not familiar with this code, and I won't be able to focus on this today :(
The deadline for the Python 3.8 beta1 release is in 3 hours. I'll report this upstream; maybe someone will have time to help.

Comment 8 serge_sans_paille 2019-06-18 11:37:06 UTC
Bug opened upstream https://bugs.python.org/issue37140

Comment 9 Miro Hrončok 2019-07-10 12:15:25 UTC
This is now blocked on python3-lit, bz1728067.

Comment 10 Ben Cotton 2019-08-13 19:04:18 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

Comment 11 Miro Hrončok 2019-08-14 22:24:28 UTC
The coordinated rebuild of Python 3.8 has started in the `f32-python` side tag.

If you figure out how to rebuild this package, please don't rebuild it in regular rawhide, but use the side tag instead:

    on branch master:
    $ fedpkg build --target=f32-python

To wait for a build to show up in the side tag, do:

    $ koji wait-repo f32-python --build=<nvr>

Where <nvr> is name-version-release of the source package, e.g. python-foo-1.1-2.fc32.

An updated mock config is posted at:
http://copr.fedorainfracloud.org/coprs/g/python/python3.8/

Note that it will take a while before the essential packages are rebuilt, so don't expect all your dependencies to be available right away.

Thanks. Let us know if you need up to date info, or if you have any questions.



PS this message is mass posted to all the bugs that block the PYTHON38 bug. If this is also a Fedora 31 FTBFS bug and you manage to fix it, you can do a f31 build as usual:

    on branch f31:
    $ fedpkg build

Comment 12 serge_sans_paille 2019-08-20 13:53:11 UTC
Package needed a build without check, then a build with check.

python-lit-0.8.0-7.fc32 is now available in f32-python

Comment 13 serge_sans_paille 2019-08-21 08:29:12 UTC
Miro, I'm still experiencing the same error (grep "double free" in https://kojipkgs.fedoraproject.org//work/tasks/6134/37196134/build.log) even with updated f32-python. I see that https://bugs.python.org/issue37140  / https://github.com/python/cpython/pull/13796 has not been merged yet, so we'll have to wait

Comment 14 Miro Hrončok 2019-08-21 09:00:55 UTC
Victor, could you help us make this happen for b4?

Comment 15 Miro Hrončok 2019-08-21 16:35:12 UTC
The f32-python side tag has been merged. In order to rebuild the package, do it in regular rawhide, but please wait until python3-3.8 is tagged:

  $ koji wait-repo f32-build --build python3-3.8.0~b3-3.fc32


If your built already started in f32-python, after it is finished, please tag it to rawhide with:

  $ koji tag-build f32-pending <nvr>

For example:

  $ koji tag-build f32-pending libreoffice-6.3.0.4-3.fc32

Thanks!

(This comment is mass posted to all bugzillas blocking the PYTHON38 tracking bug.)

Comment 16 Miro Hrončok 2019-08-21 17:29:16 UTC
(Python 3.8 has landed in the rawhide buildroot.)

Comment 17 Victor Stinner 2019-08-30 12:33:25 UTC
I wrote a fix upstream:
https://bugs.python.org/issue37140#msg350874

It will be part of the next Python 3.8.0rc1 release. If the fix is needed earlier in Fedora Rawhide, we may backport the fix downstream, or disable the Python binding in clang, until python3 is upgrade to Python 3.8.0rc1 in Rawhide.

Comment 18 Miro Hrončok 2019-08-30 13:02:40 UTC
I'll backport it.

Comment 20 Victor Stinner 2019-08-30 15:16:52 UTC
> https://src.fedoraproject.org/rpms/python3/pull-request/132

Oh wow, Miro already included my fix as part of his PR to update Python 3.8 to 3.8.0beta4. Miro: you are amazing and magic!

Comment 21 Victor Stinner 2019-09-02 08:25:27 UTC
Thanks Miro for including the ctypes fix.

I tested bug.py attached to https://bugs.python.org/issue37140 I modified the script: LIBCLANG_FILENAME = '/usr/lib64/libclang.so.8'. On an up to date Fedora Rawhide, the script does no long crash!

* python3-3.8.0~b4-1.fc32.x86_64
* clang-libs-8.0.0-4.fc32.x86_64


Note You need to log in before you can comment on or make changes to this bug.