Hide Forgot
Description of problem: Upgrading : glibc-common-2.24.90-29.fc26.x86_64 1/58 Upgrading : glibc-langpack-en-2.24.90-29.fc26.x86_64 2/58 Upgrading : glibc-2.24.90-29.fc26.x86_64 3/58 Upgrading : libgcc-7.0.1-0.3.fc26.x86_64 4/58 Upgrading : libstdc++-7.0.1-0.3.fc26.x86_64 5/58 Upgrading : libcrypt-nss-2.24.90-29.fc26.x86_64 6/58 Upgrading : system-python-libs-3.6.0-9.fc26.x86_64 7/58 Upgrading : python3-libs-3.6.0-9.fc26.x86_64 8/58 Upgrading : python3-3.6.0-9.fc26.x86_64 9/58 Upgrading : systemd-libs-232-11.fc26.x86_64 10/58 Upgrading : systemd-232-11.fc26.x86_64 11/58 Upgrading : systemd-pam-232-11.fc26.x86_64 12/58 Upgrading : libstdc++-devel-7.0.1-0.3.fc26.x86_64 13/58 Upgrading : cpp-7.0.1-0.3.fc26.x86_64 14/58 Upgrading : libgomp-7.0.1-0.3.fc26.x86_64 15/58 Upgrading : glibc-headers-2.24.90-29.fc26.x86_64 16/58 Upgrading : glibc-devel-2.24.90-29.fc26.x86_64 17/58 Upgrading : gcc-7.0.1-0.3.fc26.x86_64 18/58 Upgrading : libtool-2.4.6-15.fc26.x86_64 19/58 Upgrading : gcc-gdb-plugin-7.0.1-0.3.fc26.x86_64 20/58 Upgrading : gcc-c++-7.0.1-0.3.fc26.x86_64 21/58 Upgrading : systemd-devel-232-11.fc26.x86_64 22/58 Upgrading : systemd-udev-232-11.fc26.x86_64 23/58 Upgrading : systemd-container-232-11.fc26.x86_64 24/58 Upgrading : python3-tkinter-3.6.0-9.fc26.x86_64 25/58 Upgrading : python3-devel-3.6.0-9.fc26.x86_64 26/58 Upgrading : system-python-3.6.0-9.fc26.x86_64 27/58 Upgrading : swig-3.0.12-1.fc26.x86_64 28/58 Upgrading : glibc-all-langpacks-2.24.90-29.fc26.x86_64 29/58 Cleanup : python3-devel-3.6.0-7.fc26.x86_64 30/58 Cleanup : systemd-devel-232-10.fc26.x86_64 31/58 Cleanup : glibc-all-langpacks-2.24.90-28.fc26.x86_64 32/58 Cleanup : libtool-2.4.6-13.fc26.x86_64 33/58 Cleanup : gcc-gdb-plugin-6.3.1-2.fc26.x86_64 34/58 Cleanup : systemd-container-232-10.fc26.x86_64 35/58 Cleanup : systemd-udev-232-10.fc26.x86_64 36/58 Cleanup : systemd-pam-232-10.fc26.x86_64 37/58 Cleanup : systemd-232-10.fc26.x86_64 38/58 Cleanup : gcc-c++-6.3.1-2.fc26.x86_64 39/58 Cleanup : gcc-6.3.1-2.fc26.x86_64 40/58 Cleanup : glibc-devel-2.24.90-28.fc26.x86_64 41/58 Cleanup : swig-3.0.11-2.fc26.x86_64 42/58 Cleanup : system-python-3.6.0-7.fc26.x86_64 43/58 Cleanup : python3-tkinter-3.6.0-7.fc26.x86_64 44/58 Cleanup : python3-3.6.0-7.fc26.x86_64 45/58 Cleanup : systemd-libs-232-10.fc26.x86_64 46/58 Cleanup : python3-libs-3.6.0-7.fc26.x86_64 47/58 Cleanup : glibc-headers-2.24.90-28.fc26.x86_64 48/58 Cleanup : libstdc++-devel-6.3.1-2.fc26.x86_64 49/58 Cleanup : libstdc++-6.3.1-2.fc26.x86_64 50/58 Cleanup : system-python-libs-3.6.0-7.fc26.x86_64 51/58 Cleanup : libcrypt-nss-2.24.90-28.fc26.x86_64 52/58 Cleanup : cpp-6.3.1-2.fc26.x86_64 53/58 Cleanup : libgomp-6.3.1-2.fc26.x86_64 54/58 Cleanup : glibc-common-2.24.90-28.fc26.x86_64 55/58 Cleanup : glibc-langpack-en-2.24.90-28.fc26.x86_64 56/58 Cleanup : glibc-2.24.90-28.fc26.x86_64 57/58 Cleanup : libgcc-6.3.1-2.fc26.x86_64 58/58 After update dnf started segfaulting. Version-Release number of selected component: system-python-3.6.0-9.fc26 Additional info: reporter: libreport-2.9.0 backtrace_rating: 4 cmdline: /usr/libexec/system-python /bin/dnf update crash_function: strlen executable: /usr/libexec/system-python journald_cursor: s=ed16930a9faf4aab9ce12497a4806035;i=1e91e;b=55e0b70ed062445082c1983b7e23d3fa;m=6419577c;t=5476171e03d2e;x=d250fa17bc62a583 kernel: 4.10.0-0.rc5.git1.1.fc26.x86_64 rootdir: / runlevel: N 5 type: CCpp uid: 0 Truncated backtrace: Thread no. 1 (10 frames) #0 strlen at ../sysdeps/x86_64/strlen.S:106 #1 __regexec at regexec.c:243 #2 parse_reldep_str at /usr/src/debug/libdnf-f9b798cadb6821f9cffd5c0331578b3f7c19d699/libdnf/hy-iutil.c:700 #3 reldep_from_str at /usr/src/debug/libdnf-f9b798cadb6821f9cffd5c0331578b3f7c19d699/libdnf/hy-iutil.c:724 #4 reldep_from_pystr at /usr/src/debug/libdnf-f9b798cadb6821f9cffd5c0331578b3f7c19d699/python/hawkey/iutil-py.c:324 #5 pyseq_to_reldeplist at /usr/src/debug/libdnf-f9b798cadb6821f9cffd5c0331578b3f7c19d699/python/hawkey/iutil-py.c:246 #6 filter at /usr/src/debug/libdnf-f9b798cadb6821f9cffd5c0331578b3f7c19d699/python/hawkey/query-py.c:261 #7 PyCFunction_Call at /usr/src/debug/Python-3.6.0/Objects/methodobject.c:114 #8 do_call_core at /usr/src/debug/Python-3.6.0/Python/ceval.c:5053 #9 _PyEval_EvalFrameDefault at /usr/src/debug/Python-3.6.0/Python/ceval.c:3357
Created attachment 1246125 [details] File: backtrace
Created attachment 1246126 [details] File: cgroup
Created attachment 1246127 [details] File: core_backtrace
Created attachment 1246128 [details] File: dso_list
Created attachment 1246129 [details] File: environ
Created attachment 1246130 [details] File: exploitable
Created attachment 1246131 [details] File: limits
Created attachment 1246132 [details] File: maps
Created attachment 1246133 [details] File: open_fds
Created attachment 1246134 [details] File: proc_pid_status
Created attachment 1246135 [details] File: var_log_messages
Removing /var/cache/dnf fixes problem. * dnf --disablerepo=\* --enablerepo=rawhide update -> doesn't crash * dnf --disablerepo=\* --enablerepo=rawhide --enablerepo=kdudka-covscan update -> doesn't crash * dnf --disablerepo=\* --enablerepo=rawhide --enablerepo=kdudka-covscan --enablerepo=rpmfusion-free-rawhide update -> crashes in horrible fire
* dnf --refresh update -> doesn't really refresh cache, but fixes problem
* removing *.solv and *.solvx files doesn't fix problem
* removing packages.db fixes problem
Downgrade from glib -29 to -28 fixes problem. glibc x86_64 2.24.90-28.fc26 @commandline 3.4 M glibc-all-langpacks x86_64 2.24.90-28.fc26 @commandline 7.0 M glibc-common x86_64 2.24.90-28.fc26 @commandline 878 k glibc-devel x86_64 2.24.90-28.fc26 @commandline 962 k glibc-headers x86_64 2.24.90-28.fc26 @commandline 512 k glibc-langpack-en x86_64 2.24.90-28.fc26 @commandline 288 k libcrypt-nss x86_64 2.24.90-28.fc26 @commandline 50 k
This looks like a use-after-free issue in libdnf or its Python bindings. Reassigning.
Igor, if you still have the files around that make dnf crash, it might be worth running it under valgrind to see if it can pinpoint any use-after-free issues.
(In reply to Kalev Lember from comment #18) > Igor, if you still have the files around that make dnf crash, it might be > worth running it under valgrind to see if it can pinpoint any use-after-free > issues. https://ignatenkobrain.fedorapeople.org/dnf-cache.tar.xz
Not sure this is glibc issue, since I get bug 1418172 with glibc-2.24.90-26.fc26.x86_64
Actually there seems to be similar issue on F25: https://retrace.fedoraproject.org/faf/reports/1505192/
*** Bug 1418172 has been marked as a duplicate of this bug. ***
It's hard to reproduce. to avoid these situations again (to actually know you need to restart system), please use tracer plugin form dnf-plugins-extras. DNF should probably take information from updateinfo metadata and report to user that package requires restarting.
(In reply to Honza Silhan from comment #23) It seems that you know what is the reason behind this issue, so could you please enlighten me what is the problem here? As I said, in my case I got the error with way older version of glibc then Igor used and I am pretty sure that I updated just a few packages non of which is running on the background. So why I should restart anything?
Created attachment 1248405 [details] Valgrind output Some observations. 1) It is interesting, that in this case, the "dnf update" crashes only with the "--refresh" parameter. It won't crash otherwise. 2) The log is in czech, since I was not able to reproduce the error with LANG=C.utf-8. 3) I was not able to reproduce this when running under Valgrind, nor GDB $ rpm -q glibc glibc-2.24.90-29.fc26.x86_64 $ rpm -q dnf dnf-2.0.0-2.fc26.noarch $ rpm -q libdnf libdnf-0.7.0-0.7gitf9b798c.fc26.x86_64 $ rpm -q python3-hawkey python3-hawkey-0.7.0-0.7gitf9b798c.fc26.x86_64
There's a bunch of invalid memory accesses (use-after-free) in the valgrind log. Could someone who understands the hawkey python bindings look at those please? I think that fixing those should fix the crash.
So from the Vagrant output, it all begins in reldep_from_pystr [1] and this is the implementation: ~~~ DnfReldep * reldep_from_pystr(PyObject *o, DnfSack *sack) { DnfReldep *reldep = NULL; const char *reldep_str = NULL; PyObject *tmp_py_str = NULL; reldep_str = pycomp_get_string(o, &tmp_py_str); if (reldep_str == NULL) return NULL; Py_XDECREF(tmp_py_str); reldep = reldep_from_str(sack, reldep_str); return reldep; } ~~~ The string is allocated by "pycomp_get_string". In its description is written: ~~~ /** * bytes, basic string or unicode string in Python 2/3 to c string converter, * you need to call Py_XDECREF(tmp_py_str) after usage of returned string */ ~~~ The "Py_XDECREF" is called indeed, but *prior* the string is used. So now I can only guess, that once the reference is decreased, the GC kicks in in some cases and cleans up the memory and later, when the reldep_from_str is called, it migh crash. So this [2] is PR which might fix the SEGFAULT (and it fixes another similar looking place). But: 1. I have not tested the patch at all. 2. I don't have reproducer at my hand, but the shortest path seems to be something like properly modified test_reldep_list [3]. I would really appreciate if somebody else (more skilled in Python and DNF) could find the right reproducer and provide regression test prior this gets merged. [1]: https://github.com/rpm-software-management/libdnf/blob/master/python/hawkey/iutil-py.c#L313 [2]: https://github.com/rpm-software-management/libdnf/pull/255 [3]: https://github.com/rpm-software-management/libdnf/blob/master/python/hawkey/tests/tests/test_query.py#L189
(In reply to Vít Ondruch from comment #27) > Vagrant s/Vagrant/Valgrind/ of course ... to my defense, I faced this issue during work on Vagrant :D
*** Bug 1421797 has been marked as a duplicate of this bug. ***
Confirmed latest libdnf works; thanks!
Igor, could you please refer to the actual commit that fixed this bug and set the Fixed In Version field accordingly?
(In reply to Kamil Dudka from comment #31) > Igor, could you please refer to the actual commit that fixed this bug and > set the Fixed In Version field accordingly? 1. It's linked from Vit's comment above 2. I have absolutely no idea with which version it got fixed