Bug 1417870 - [abrt] system-python: strlen(): system-python killed by signal 11
Summary: [abrt] system-python: strlen(): system-python killed by signal 11
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: libdnf
Version: rawhide
Hardware: x86_64
OS: Unspecified
low
unspecified
Target Milestone: ---
Assignee: rpm-software-management
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:42ce4b38346841f156bce3ef083...
: 1418172 1421797 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-31 10:42 UTC by Igor Gnatenko
Modified: 2017-07-03 09:09 UTC (History)
26 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-10 15:02:54 UTC
Type: ---


Attachments (Terms of Use)
File: backtrace (164.56 KB, text/plain)
2017-01-31 10:42 UTC, Igor Gnatenko
no flags Details
File: cgroup (263 bytes, text/plain)
2017-01-31 10:42 UTC, Igor Gnatenko
no flags Details
File: core_backtrace (20.34 KB, text/plain)
2017-01-31 10:42 UTC, Igor Gnatenko
no flags Details
File: dso_list (10.64 KB, text/plain)
2017-01-31 10:42 UTC, Igor Gnatenko
no flags Details
File: environ (290 bytes, text/plain)
2017-01-31 10:42 UTC, Igor Gnatenko
no flags Details
File: exploitable (82 bytes, text/plain)
2017-01-31 10:42 UTC, Igor Gnatenko
no flags Details
File: limits (1.29 KB, text/plain)
2017-01-31 10:42 UTC, Igor Gnatenko
no flags Details
File: maps (48.99 KB, text/plain)
2017-01-31 10:42 UTC, Igor Gnatenko
no flags Details
File: open_fds (1.40 KB, text/plain)
2017-01-31 10:42 UTC, Igor Gnatenko
no flags Details
File: proc_pid_status (1.23 KB, text/plain)
2017-01-31 10:42 UTC, Igor Gnatenko
no flags Details
File: var_log_messages (28 bytes, text/plain)
2017-01-31 10:42 UTC, Igor Gnatenko
no flags Details
Valgrind output (110.55 KB, text/plain)
2017-02-07 14:14 UTC, Vít Ondruch
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1247405 0 unspecified CLOSED [abrt] dnf: free(): python3.4 killed by SIGABRT 2021-02-22 00:41:40 UTC

Internal Links: 1247405

Description Igor Gnatenko 2017-01-31 10:42:13 UTC
Description of problem:
  Upgrading   : glibc-common-2.24.90-29.fc26.x86_64                                                                                                                                                           1/58 
  Upgrading   : glibc-langpack-en-2.24.90-29.fc26.x86_64                                                                                                                                                      2/58 
  Upgrading   : glibc-2.24.90-29.fc26.x86_64                                                                                                                                                                  3/58 
  Upgrading   : libgcc-7.0.1-0.3.fc26.x86_64                                                                                                                                                                  4/58 
  Upgrading   : libstdc++-7.0.1-0.3.fc26.x86_64                                                                                                                                                               5/58 
  Upgrading   : libcrypt-nss-2.24.90-29.fc26.x86_64                                                                                                                                                           6/58 
  Upgrading   : system-python-libs-3.6.0-9.fc26.x86_64                                                                                                                                                        7/58 
  Upgrading   : python3-libs-3.6.0-9.fc26.x86_64                                                                                                                                                              8/58 
  Upgrading   : python3-3.6.0-9.fc26.x86_64                                                                                                                                                                   9/58 
  Upgrading   : systemd-libs-232-11.fc26.x86_64                                                                                                                                                              10/58 
  Upgrading   : systemd-232-11.fc26.x86_64                                                                                                                                                                   11/58 
  Upgrading   : systemd-pam-232-11.fc26.x86_64                                                                                                                                                               12/58 
  Upgrading   : libstdc++-devel-7.0.1-0.3.fc26.x86_64                                                                                                                                                        13/58 
  Upgrading   : cpp-7.0.1-0.3.fc26.x86_64                                                                                                                                                                    14/58 
  Upgrading   : libgomp-7.0.1-0.3.fc26.x86_64                                                                                                                                                                15/58 
  Upgrading   : glibc-headers-2.24.90-29.fc26.x86_64                                                                                                                                                         16/58 
  Upgrading   : glibc-devel-2.24.90-29.fc26.x86_64                                                                                                                                                           17/58 
  Upgrading   : gcc-7.0.1-0.3.fc26.x86_64                                                                                                                                                                    18/58 
  Upgrading   : libtool-2.4.6-15.fc26.x86_64                                                                                                                                                                 19/58 
  Upgrading   : gcc-gdb-plugin-7.0.1-0.3.fc26.x86_64                                                                                                                                                         20/58 
  Upgrading   : gcc-c++-7.0.1-0.3.fc26.x86_64                                                                                                                                                                21/58 
  Upgrading   : systemd-devel-232-11.fc26.x86_64                                                                                                                                                             22/58 
  Upgrading   : systemd-udev-232-11.fc26.x86_64                                                                                                                                                              23/58 
  Upgrading   : systemd-container-232-11.fc26.x86_64                                                                                                                                                         24/58 
  Upgrading   : python3-tkinter-3.6.0-9.fc26.x86_64                                                                                                                                                          25/58 
  Upgrading   : python3-devel-3.6.0-9.fc26.x86_64                                                                                                                                                            26/58 
  Upgrading   : system-python-3.6.0-9.fc26.x86_64                                                                                                                                                            27/58 
  Upgrading   : swig-3.0.12-1.fc26.x86_64                                                                                                                                                                    28/58 
  Upgrading   : glibc-all-langpacks-2.24.90-29.fc26.x86_64                                                                                                                                                   29/58 
  Cleanup     : python3-devel-3.6.0-7.fc26.x86_64                                                                                                                                                            30/58 
  Cleanup     : systemd-devel-232-10.fc26.x86_64                                                                                                                                                             31/58 
  Cleanup     : glibc-all-langpacks-2.24.90-28.fc26.x86_64                                                                                                                                                   32/58 
  Cleanup     : libtool-2.4.6-13.fc26.x86_64                                                                                                                                                                 33/58 
  Cleanup     : gcc-gdb-plugin-6.3.1-2.fc26.x86_64                                                                                                                                                           34/58 
  Cleanup     : systemd-container-232-10.fc26.x86_64                                                                                                                                                         35/58 
  Cleanup     : systemd-udev-232-10.fc26.x86_64                                                                                                                                                              36/58 
  Cleanup     : systemd-pam-232-10.fc26.x86_64                                                                                                                                                               37/58 
  Cleanup     : systemd-232-10.fc26.x86_64                                                                                                                                                                   38/58 
  Cleanup     : gcc-c++-6.3.1-2.fc26.x86_64                                                                                                                                                                  39/58 
  Cleanup     : gcc-6.3.1-2.fc26.x86_64                                                                                                                                                                      40/58 
  Cleanup     : glibc-devel-2.24.90-28.fc26.x86_64                                                                                                                                                           41/58 
  Cleanup     : swig-3.0.11-2.fc26.x86_64                                                                                                                                                                    42/58 
  Cleanup     : system-python-3.6.0-7.fc26.x86_64                                                                                                                                                            43/58 
  Cleanup     : python3-tkinter-3.6.0-7.fc26.x86_64                                                                                                                                                          44/58 
  Cleanup     : python3-3.6.0-7.fc26.x86_64                                                                                                                                                                  45/58 
  Cleanup     : systemd-libs-232-10.fc26.x86_64                                                                                                                                                              46/58 
  Cleanup     : python3-libs-3.6.0-7.fc26.x86_64                                                                                                                                                             47/58 
  Cleanup     : glibc-headers-2.24.90-28.fc26.x86_64                                                                                                                                                         48/58 
  Cleanup     : libstdc++-devel-6.3.1-2.fc26.x86_64                                                                                                                                                          49/58 
  Cleanup     : libstdc++-6.3.1-2.fc26.x86_64                                                                                                                                                                50/58 
  Cleanup     : system-python-libs-3.6.0-7.fc26.x86_64                                                                                                                                                       51/58 
  Cleanup     : libcrypt-nss-2.24.90-28.fc26.x86_64                                                                                                                                                          52/58 
  Cleanup     : cpp-6.3.1-2.fc26.x86_64                                                                                                                                                                      53/58 
  Cleanup     : libgomp-6.3.1-2.fc26.x86_64                                                                                                                                                                  54/58 
  Cleanup     : glibc-common-2.24.90-28.fc26.x86_64                                                                                                                                                          55/58 
  Cleanup     : glibc-langpack-en-2.24.90-28.fc26.x86_64                                                                                                                                                     56/58 
  Cleanup     : glibc-2.24.90-28.fc26.x86_64                                                                                                                                                                 57/58 
  Cleanup     : libgcc-6.3.1-2.fc26.x86_64                                                                                                                                                                   58/58 



After update dnf started segfaulting.

Version-Release number of selected component:
system-python-3.6.0-9.fc26

Additional info:
reporter:       libreport-2.9.0
backtrace_rating: 4
cmdline:        /usr/libexec/system-python /bin/dnf update
crash_function: strlen
executable:     /usr/libexec/system-python
journald_cursor: s=ed16930a9faf4aab9ce12497a4806035;i=1e91e;b=55e0b70ed062445082c1983b7e23d3fa;m=6419577c;t=5476171e03d2e;x=d250fa17bc62a583
kernel:         4.10.0-0.rc5.git1.1.fc26.x86_64
rootdir:        /
runlevel:       N 5
type:           CCpp
uid:            0

Truncated backtrace:
Thread no. 1 (10 frames)
 #0 strlen at ../sysdeps/x86_64/strlen.S:106
 #1 __regexec at regexec.c:243
 #2 parse_reldep_str at /usr/src/debug/libdnf-f9b798cadb6821f9cffd5c0331578b3f7c19d699/libdnf/hy-iutil.c:700
 #3 reldep_from_str at /usr/src/debug/libdnf-f9b798cadb6821f9cffd5c0331578b3f7c19d699/libdnf/hy-iutil.c:724
 #4 reldep_from_pystr at /usr/src/debug/libdnf-f9b798cadb6821f9cffd5c0331578b3f7c19d699/python/hawkey/iutil-py.c:324
 #5 pyseq_to_reldeplist at /usr/src/debug/libdnf-f9b798cadb6821f9cffd5c0331578b3f7c19d699/python/hawkey/iutil-py.c:246
 #6 filter at /usr/src/debug/libdnf-f9b798cadb6821f9cffd5c0331578b3f7c19d699/python/hawkey/query-py.c:261
 #7 PyCFunction_Call at /usr/src/debug/Python-3.6.0/Objects/methodobject.c:114
 #8 do_call_core at /usr/src/debug/Python-3.6.0/Python/ceval.c:5053
 #9 _PyEval_EvalFrameDefault at /usr/src/debug/Python-3.6.0/Python/ceval.c:3357

Comment 1 Igor Gnatenko 2017-01-31 10:42:25 UTC
Created attachment 1246125 [details]
File: backtrace

Comment 2 Igor Gnatenko 2017-01-31 10:42:26 UTC
Created attachment 1246126 [details]
File: cgroup

Comment 3 Igor Gnatenko 2017-01-31 10:42:29 UTC
Created attachment 1246127 [details]
File: core_backtrace

Comment 4 Igor Gnatenko 2017-01-31 10:42:31 UTC
Created attachment 1246128 [details]
File: dso_list

Comment 5 Igor Gnatenko 2017-01-31 10:42:33 UTC
Created attachment 1246129 [details]
File: environ

Comment 6 Igor Gnatenko 2017-01-31 10:42:34 UTC
Created attachment 1246130 [details]
File: exploitable

Comment 7 Igor Gnatenko 2017-01-31 10:42:36 UTC
Created attachment 1246131 [details]
File: limits

Comment 8 Igor Gnatenko 2017-01-31 10:42:38 UTC
Created attachment 1246132 [details]
File: maps

Comment 9 Igor Gnatenko 2017-01-31 10:42:40 UTC
Created attachment 1246133 [details]
File: open_fds

Comment 10 Igor Gnatenko 2017-01-31 10:42:42 UTC
Created attachment 1246134 [details]
File: proc_pid_status

Comment 11 Igor Gnatenko 2017-01-31 10:42:43 UTC
Created attachment 1246135 [details]
File: var_log_messages

Comment 12 Igor Gnatenko 2017-01-31 10:55:28 UTC
Removing /var/cache/dnf fixes problem.

* dnf --disablerepo=\* --enablerepo=rawhide update
-> doesn't crash
* dnf --disablerepo=\* --enablerepo=rawhide --enablerepo=kdudka-covscan update
-> doesn't crash
* dnf --disablerepo=\* --enablerepo=rawhide --enablerepo=kdudka-covscan --enablerepo=rpmfusion-free-rawhide update
-> crashes in horrible fire

Comment 13 Igor Gnatenko 2017-01-31 11:08:14 UTC
* dnf --refresh update
-> doesn't really refresh cache, but fixes problem

Comment 14 Igor Gnatenko 2017-01-31 11:08:46 UTC
* removing *.solv and *.solvx files doesn't fix problem

Comment 15 Igor Gnatenko 2017-01-31 11:11:51 UTC
* removing packages.db fixes problem

Comment 16 Igor Gnatenko 2017-01-31 11:23:07 UTC
Downgrade from glib -29 to -28 fixes problem.

 glibc                                                    x86_64                                      2.24.90-28.fc26                                      @commandline                                      3.4 M
 glibc-all-langpacks                                      x86_64                                      2.24.90-28.fc26                                      @commandline                                      7.0 M
 glibc-common                                             x86_64                                      2.24.90-28.fc26                                      @commandline                                      878 k
 glibc-devel                                              x86_64                                      2.24.90-28.fc26                                      @commandline                                      962 k
 glibc-headers                                            x86_64                                      2.24.90-28.fc26                                      @commandline                                      512 k
 glibc-langpack-en                                        x86_64                                      2.24.90-28.fc26                                      @commandline                                      288 k
 libcrypt-nss                                             x86_64                                      2.24.90-28.fc26                                      @commandline                                       50 k

Comment 17 Florian Weimer 2017-01-31 12:03:19 UTC
This looks like a use-after-free issue in libdnf or its Python bindings.  Reassigning.

Comment 18 Kalev Lember 2017-01-31 16:08:10 UTC
Igor, if you still have the files around that make dnf crash, it might be worth running it under valgrind to see if it can pinpoint any use-after-free issues.

Comment 19 Igor Gnatenko 2017-01-31 16:12:11 UTC
(In reply to Kalev Lember from comment #18)
> Igor, if you still have the files around that make dnf crash, it might be
> worth running it under valgrind to see if it can pinpoint any use-after-free
> issues.
https://ignatenkobrain.fedorapeople.org/dnf-cache.tar.xz

Comment 20 Vít Ondruch 2017-02-03 13:13:56 UTC
Not sure this is glibc issue, since I get bug 1418172 with glibc-2.24.90-26.fc26.x86_64

Comment 21 Vít Ondruch 2017-02-03 13:16:13 UTC
Actually there seems to be similar issue on F25:

https://retrace.fedoraproject.org/faf/reports/1505192/

Comment 22 Vít Ondruch 2017-02-03 13:16:58 UTC
*** Bug 1418172 has been marked as a duplicate of this bug. ***

Comment 23 Honza Silhan 2017-02-06 12:52:58 UTC
It's hard to reproduce. to avoid these situations again (to actually know you need to restart system), please use tracer plugin form dnf-plugins-extras. 

DNF should probably take information from updateinfo metadata and report to user that package requires restarting.

Comment 24 Vít Ondruch 2017-02-06 13:17:12 UTC
(In reply to Honza Silhan from comment #23)
It seems that you know what is the reason behind this issue, so could you please enlighten me what is the problem here?

As I said, in my case I got the error with way older version of glibc then Igor used and I am pretty sure that I updated just a few packages non of which is running on the background. So why I should restart anything?

Comment 25 Vít Ondruch 2017-02-07 14:14:56 UTC
Created attachment 1248405 [details]
Valgrind output

Some observations.

1) It is interesting, that in this case, the "dnf update" crashes only with the "--refresh" parameter. It won't crash otherwise.

2) The log is in czech, since I was not able to reproduce the error with LANG=C.utf-8.

3) I was not able to reproduce this when running under Valgrind, nor GDB



$ rpm -q glibc
glibc-2.24.90-29.fc26.x86_64

$ rpm -q dnf
dnf-2.0.0-2.fc26.noarch

$ rpm -q libdnf
libdnf-0.7.0-0.7gitf9b798c.fc26.x86_64

$ rpm -q python3-hawkey
python3-hawkey-0.7.0-0.7gitf9b798c.fc26.x86_64

Comment 26 Kalev Lember 2017-02-07 15:13:14 UTC
There's a bunch of invalid memory accesses (use-after-free) in the valgrind log. Could someone who understands the hawkey python bindings look at those please? I think that fixing those should fix the crash.

Comment 27 Vít Ondruch 2017-02-08 14:12:04 UTC
So from the Vagrant output, it all begins in reldep_from_pystr [1] and this is the implementation:

~~~
DnfReldep *
reldep_from_pystr(PyObject *o, DnfSack *sack)
{
    DnfReldep *reldep = NULL;
    const char *reldep_str = NULL;
    PyObject *tmp_py_str = NULL;

    reldep_str = pycomp_get_string(o, &tmp_py_str);
    if (reldep_str == NULL)
        return NULL;
    Py_XDECREF(tmp_py_str);

    reldep = reldep_from_str(sack, reldep_str);
    return reldep;
}
~~~

The string is allocated by "pycomp_get_string". In its description is written:

~~~
/**
 * bytes, basic string or unicode string in Python 2/3 to c string converter,
 * you need to call Py_XDECREF(tmp_py_str) after usage of returned string
 */
~~~

The "Py_XDECREF" is called indeed, but *prior* the string is used. So now I can only guess, that once the reference is decreased, the GC kicks in in some cases and cleans up the memory and later, when the reldep_from_str is called, it migh crash.

So this [2] is PR which might fix the SEGFAULT (and it fixes another similar looking place). But:

1. I have not tested the patch at all.
2. I don't have reproducer at my hand, but the shortest path seems to be something like properly modified test_reldep_list [3].

I would really appreciate if somebody else (more skilled in Python and DNF) could find the right reproducer and provide regression test prior this gets merged.


[1]: https://github.com/rpm-software-management/libdnf/blob/master/python/hawkey/iutil-py.c#L313
[2]: https://github.com/rpm-software-management/libdnf/pull/255
[3]: https://github.com/rpm-software-management/libdnf/blob/master/python/hawkey/tests/tests/test_query.py#L189

Comment 28 Vít Ondruch 2017-02-08 14:15:09 UTC
(In reply to Vít Ondruch from comment #27)
> Vagrant

s/Vagrant/Valgrind/ of course ... to my defense, I faced this issue during work on Vagrant :D

Comment 29 Igor Gnatenko 2017-02-15 11:10:40 UTC
*** Bug 1421797 has been marked as a duplicate of this bug. ***

Comment 30 Robin Powell 2017-02-15 17:18:15 UTC
Confirmed latest libdnf works; thanks!

Comment 31 Kamil Dudka 2017-04-10 09:49:35 UTC
Igor, could you please refer to the actual commit that fixed this bug and set the Fixed In Version field accordingly?

Comment 32 Igor Gnatenko 2017-07-03 09:09:56 UTC
(In reply to Kamil Dudka from comment #31)
> Igor, could you please refer to the actual commit that fixed this bug and
> set the Fixed In Version field accordingly?

1. It's linked from Vit's comment above
2. I have absolutely no idea with which version it got fixed


Note You need to log in before you can comment on or make changes to this bug.