Bug 1600917

Summary: libdnf Segmentation fault after update
Product: [Fedora] Fedora Reporter: lukasz.tylski
Component: libdnfAssignee: rpm-software-management
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 29CC: bugzilla, djuran, dmach, giampaolo.ferradini, jmracek, ken, mattias.ellert, mluscon, nathanael, prd-fedora, rpm-software-management, samuel.rakitnican, thib, tomek
Target Milestone: ---Keywords: Reopened, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-29 07:47:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description lukasz.tylski 2018-07-13 11:36:35 UTC
Description of problem:

After update - every dnf command finished witch segmentation fault


Version-Release number of selected component (if applicable):

libdnf-0.15.2-1.fc29.x86_64

How reproducible:

always

Steps to Reproduce:
1.run dnf list / dnf update ...

Actual results:

dnf -vvvv list

Loaded plugins: builddep, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repograph, repomanage, reposync, system-upgrade
DNF version: 3.0.2
cachedir: /var/cache/dnf
repo: using cache for: rawhide
not found deltainfo for: Fedora - Rawhide - Developmental packages for the next Fedora release
not found updateinfo for: Fedora - Rawhide - Developmental packages for the next Fedora release
rawhide: using metadata from Wed Jul 11 10:49:33 2018.
Last metadata expiration check: 0:14:55 ago on Fri Jul 13 13:20:38 2018.
Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/rawhide-2d95c80a1fa0a67d
modulemd-prioritizer-error-quark: No module objects have been added to the prioritizer. Use modulemd_prioritizer_add() first. (0)
Segmentation fault (core dump)


Expected results:

packages list

Additional info:

gdb bt

#0  0x00007fffe735ab0b in libdnf::TransactionItem::saveReplacedBy() () at /lib64/libdnf.so.2
#1  0x00007fffe736742a in libdnf::swdb_private::Transaction::saveItems() () at /lib64/libdnf.so.2
#2  0x00007fffe7360f62 in libdnf::Transformer::transformTrans(std::shared_ptr<SQLite3>, std::shared_ptr<SQLite3>) () at /lib64/libdnf.so.2
#3  0x00007fffe736265e in libdnf::Transformer::transform() () at /lib64/libdnf.so.2
#4  0x00007fffe734c03b in libdnf::Swdb::Swdb(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () at /lib64/libdnf.so.2
#5  0x00007fffe05c8f2f in _wrap_new_Swdb () at /usr/lib64/python3.7/site-packages/libdnf/_transaction.so
#6  0x00007ffff7987643 in PyCFunction_Call () at /lib64/libpython3.7m.so.1.0
#7  0x00007ffff7a29645 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#8  0x00007ffff79666a8 in _PyEval_EvalCodeWithName () at /lib64/libpython3.7m.so.1.0
#9  0x00007ffff796794b in _PyFunction_FastCallDict () at /lib64/libpython3.7m.so.1.0
#10 0x00007ffff7977526 in _PyObject_Call_Prepend () at /lib64/libpython3.7m.so.1.0
#11 0x00007ffff79c6301 in slot_tp_init () at /lib64/libpython3.7m.so.1.0
#12 0x00007ffff79db479 in _PyObject_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#13 0x00007ffff7a28c00 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#14 0x00007ffff796776a in _PyFunction_FastCallDict () at /lib64/libpython3.7m.so.1.0
#15 0x00007ffff798774e in property_descr_get () at /lib64/libpython3.7m.so.1.0
#16 0x00007ffff7965713 in _PyObject_GenericGetAttrWithDict () at /lib64/libpython3.7m.so.1.0
#17 0x00007ffff7a23349 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#18 0x00007ffff79af26a in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#19 0x00007ffff7a22f91 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#20 0x00007ffff796776a in _PyFunction_FastCallDict () at /lib64/libpython3.7m.so.1.0
#21 0x00007ffff798774e in property_descr_get () at /lib64/libpython3.7m.so.1.0
#22 0x00007ffff7965713 in _PyObject_GenericGetAttrWithDict () at /lib64/libpython3.7m.so.1.0
#23 0x00007ffff7a23349 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#24 0x00007ffff79666a8 in _PyEval_EvalCodeWithName () at /lib64/libpython3.7m.so.1.0
#25 0x00007ffff79af411 in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#26 0x00007ffff7a231f9 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#27 0x00007ffff79af26a in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#28 0x00007ffff7a231f9 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#29 0x00007ffff79666a8 in _PyEval_EvalCodeWithName () at /lib64/libpython3.7m.so.1.0
#30 0x00007ffff79af411 in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#31 0x00007ffff7a22f91 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#32 0x00007ffff79af26a in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#33 0x00007ffff7a22f91 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#34 0x00007ffff79af26a in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#35 0x00007ffff7a22f91 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#36 0x00007ffff79af26a in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#37 0x00007ffff7a231f9 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#38 0x00007ffff79af26a in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#39 0x00007ffff7a231f9 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#40 0x00007ffff79666a8 in _PyEval_EvalCodeWithName () at /lib64/libpython3.7m.so.1.0
#41 0x00007ffff79af411 in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#42 0x00007ffff7a231f9 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#43 0x00007ffff79666a8 in _PyEval_EvalCodeWithName () at /lib64/libpython3.7m.so.1.0
#44 0x00007ffff79af411 in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#45 0x00007ffff7a24007 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#46 0x00007ffff79666a8 in _PyEval_EvalCodeWithName () at /lib64/libpython3.7m.so.1.0
#47 0x00007ffff7967623 in PyEval_EvalCodeEx () at /lib64/libpython3.7m.so.1.0
#48 0x00007ffff796764b in PyEval_EvalCode () at /lib64/libpython3.7m.so.1.0
#49 0x00007ffff7a98d1e in run_mod () at /lib64/libpython3.7m.so.1.0
#50 0x00007ffff7a9ab46 in PyRun_FileExFlags () at /lib64/libpython3.7m.so.1.0
#51 0x00007ffff7a9c008 in PyRun_SimpleFileExFlags () at /lib64/libpython3.7m.so.1.0
#52 0x00007ffff7a9d9cc in pymain_main.constprop () at /lib64/libpython3.7m.so.1.0
#53 0x00007ffff7a9e370 in _Py_UnixMain () at /lib64/libpython3.7m.so.1.0
#54 0x00007ffff6b1b3b3 in __libc_start_main () at /lib64/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--c
#55 0x000055555555478a in _start ()
(gdb) bt
#0  0x00007fffe735ab0b in libdnf::TransactionItem::saveReplacedBy() () at /lib64/libdnf.so.2
#1  0x00007fffe736742a in libdnf::swdb_private::Transaction::saveItems() () at /lib64/libdnf.so.2
#2  0x00007fffe7360f62 in libdnf::Transformer::transformTrans(std::shared_ptr<SQLite3>, std::shared_ptr<SQLite3>) () at /lib64/libdnf.so.2
#3  0x00007fffe736265e in libdnf::Transformer::transform() () at /lib64/libdnf.so.2
#4  0x00007fffe734c03b in libdnf::Swdb::Swdb(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () at /lib64/libdnf.so.2
#5  0x00007fffe05c8f2f in _wrap_new_Swdb () at /usr/lib64/python3.7/site-packages/libdnf/_transaction.so
#6  0x00007ffff7987643 in PyCFunction_Call () at /lib64/libpython3.7m.so.1.0
#7  0x00007ffff7a29645 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#8  0x00007ffff79666a8 in _PyEval_EvalCodeWithName () at /lib64/libpython3.7m.so.1.0
#9  0x00007ffff796794b in _PyFunction_FastCallDict () at /lib64/libpython3.7m.so.1.0
#10 0x00007ffff7977526 in _PyObject_Call_Prepend () at /lib64/libpython3.7m.so.1.0
#11 0x00007ffff79c6301 in slot_tp_init () at /lib64/libpython3.7m.so.1.0
#12 0x00007ffff79db479 in _PyObject_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#13 0x00007ffff7a28c00 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#14 0x00007ffff796776a in _PyFunction_FastCallDict () at /lib64/libpython3.7m.so.1.0
#15 0x00007ffff798774e in property_descr_get () at /lib64/libpython3.7m.so.1.0
#16 0x00007ffff7965713 in _PyObject_GenericGetAttrWithDict () at /lib64/libpython3.7m.so.1.0
#17 0x00007ffff7a23349 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#18 0x00007ffff79af26a in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#19 0x00007ffff7a22f91 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#20 0x00007ffff796776a in _PyFunction_FastCallDict () at /lib64/libpython3.7m.so.1.0
#21 0x00007ffff798774e in property_descr_get () at /lib64/libpython3.7m.so.1.0
#22 0x00007ffff7965713 in _PyObject_GenericGetAttrWithDict () at /lib64/libpython3.7m.so.1.0
#23 0x00007ffff7a23349 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#24 0x00007ffff79666a8 in _PyEval_EvalCodeWithName () at /lib64/libpython3.7m.so.1.0
#25 0x00007ffff79af411 in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#26 0x00007ffff7a231f9 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#27 0x00007ffff79af26a in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#28 0x00007ffff7a231f9 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#29 0x00007ffff79666a8 in _PyEval_EvalCodeWithName () at /lib64/libpython3.7m.so.1.0
#30 0x00007ffff79af411 in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#31 0x00007ffff7a22f91 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#32 0x00007ffff79af26a in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#33 0x00007ffff7a22f91 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#34 0x00007ffff79af26a in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#35 0x00007ffff7a22f91 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#36 0x00007ffff79af26a in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#37 0x00007ffff7a231f9 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#38 0x00007ffff79af26a in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#39 0x00007ffff7a231f9 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#40 0x00007ffff79666a8 in _PyEval_EvalCodeWithName () at /lib64/libpython3.7m.so.1.0
#41 0x00007ffff79af411 in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#42 0x00007ffff7a231f9 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#43 0x00007ffff79666a8 in _PyEval_EvalCodeWithName () at /lib64/libpython3.7m.so.1.0
#44 0x00007ffff79af411 in _PyFunction_FastCallKeywords () at /lib64/libpython3.7m.so.1.0
#45 0x00007ffff7a24007 in _PyEval_EvalFrameDefault () at /lib64/libpython3.7m.so.1.0
#46 0x00007ffff79666a8 in _PyEval_EvalCodeWithName () at /lib64/libpython3.7m.so.1.0
#47 0x00007ffff7967623 in PyEval_EvalCodeEx () at /lib64/libpython3.7m.so.1.0
#48 0x00007ffff796764b in PyEval_EvalCode () at /lib64/libpython3.7m.so.1.0
#49 0x00007ffff7a98d1e in run_mod () at /lib64/libpython3.7m.so.1.0
#50 0x00007ffff7a9ab46 in PyRun_FileExFlags () at /lib64/libpython3.7m.so.1.0
#51 0x00007ffff7a9c008 in PyRun_SimpleFileExFlags () at /lib64/libpython3.7m.so.1.0
#52 0x00007ffff7a9d9cc in pymain_main.constprop () at /lib64/libpython3.7m.so.1.0
#53 0x00007ffff7a9e370 in _Py_UnixMain () at /lib64/libpython3.7m.so.1.0
#54 0x00007ffff6b1b3b3 in __libc_start_main () at /lib64/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--
#55 0x000055555555478a in _start ()


dmesg

[17031.485307] dnf[38295]: segfault at 40 ip 00007f0d0e082b0b sp 00007ffc56dcaaa0 error 4 in libdnf.so.2[7f0d0dfce000+13c000]
[17031.485320] Code: 00 49 8b 6c 24 68 4d 8b 6c 24 70 4c 39 ed 0f 84 83 00 00 00 0f 1f 00 48 8b 45 00 49 8b 54 24 40 be 01 00 00 00 48 8b 7c 24 38 <4c> 8b 70 40 e8 dc 94 f9 ff 89 c3 85 c0 0f 85 97 00 00 00 48 8b 7c

Comment 1 lukasz.tylski 2018-07-25 08:04:12 UTC
Solved:

Renaming/removing /var/lib/dnf/history works.

Comment 2 srakitnican 2018-09-18 19:34:10 UTC
Did you just remove history or was dnf fixed to work with older history format?

Comment 3 Tomasz Torcz 2018-09-25 04:24:39 UTC
I'm reopening this bug. I've just upgraded (f28→f29) and DNF started crashing.
Removing /var/lib/dnf/history fixes this, but we can't expect users to do that.
DNF have to be fixed to work with (or ignore) old history files.

dnf-3.5.1-1.fc29.noarch
libdnf-0.19.1-3.fc29.x86_64

Comment 4 Tomasz Torcz 2018-09-25 04:30:17 UTC
My DNF history/ dir causing crashes:

https://pipebreaker.pl/z/dnf_history.tar.bz2

Comment 5 Jaroslav Mracek 2018-09-25 06:36:15 UTC
Please can you confirm that problem still persist with dnf-3.6.0? It is available from copr repository (dnf copr enable rpmsoftwaremanagement/dnf-nightly)?

Comment 6 Tomasz Torcz 2018-09-25 14:12:13 UTC
Hmm. So I've upgraded dnf from COPR, restored history/ dir and did an upgrade. Seems fine on the first run. But I've noticed that history got tossed away:

$ dnf history list
ID     | Command line             | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
     3 | upgrade                  | 2018-09-25 16:01 | Upgrade        |    8   
     2 | upgrade dnf              | 2018-09-25 15:59 | Upgrade        |   11   
     1 | upgrade https://kojipkgs | 2018-09-25 06:22 | Upgrade        |    1   

dnf-3.6.1-0.36g7232477b.fc29.noarch

Comment 7 Jaroslav Mracek 2018-09-25 19:15:04 UTC
Ok, thanks for the report. Then I believed that the problem is fixed with dnf-3.6.1

Comment 8 srakitnican 2018-09-25 19:45:20 UTC
No comment on history disappearing, is that made on purpose? I am personally not very happy about that.

Comment 9 Nathanael Noblet 2018-09-25 20:56:48 UTC
*** Bug 1632922 has been marked as a duplicate of this bug. ***

Comment 10 srakitnican 2018-09-25 21:40:35 UTC
Tried dnf-3.6.1 with my particular case bug#1598590, history is preserved and there are no database errors on dnf operations.

Comment 11 Nathanael Noblet 2018-09-26 15:23:26 UTC
@srakitnican 

I'm curious - how do you update dnf to 3.6.1 without losing history when dnf itself won't work?

Comment 12 Jaroslav Mracek 2018-09-26 15:27:07 UTC
Normally db conversion works fine, but it looks like that your original DB was corrupted.

Comment 13 Tomasz Torcz 2018-09-26 15:51:05 UTC
You have my original DB in #c4.
But, hmm, conversion you say? I did the following:
1. Moved /var/lib/dnf/history/ directory to the side (so I have non-crashing dnf)
2. Installed some package (iwd) from koji, to verify it's working
3. enabled copr
4. Upgraded dnf (using version from copr)
5. Moved original history/ dir back
6. Did and upgrade to verify dnf work.

#c12 got me thinking - is there a conversion going at step 4? Does it mean when I get my original history back in 5, there was already converted DB somewhere and my original wasn't use any longer?
If yes, then at 6 I wasn't testing if my original DB crash new dnf. How do I undo the conversion?

Comment 14 srakitnican 2018-09-27 06:46:53 UTC
(In reply to Nathanael Noblet from comment #11)
> @srakitnican 
> 
> I'm curious - how do you update dnf to 3.6.1 without losing history when dnf
> itself won't work?

With rpm. It is necessary to provide all the dependencies manually, though. A bit of an tedious process if there are many changes. Alternatively, another option which might have worked in my case is to do the update with an older dnf from another release with the help of --installroot option.

Comment 15 Jaroslav Mracek 2018-09-27 11:37:32 UTC
To trigger db conversion to new format please delete /var/lib/dnf/history.sqlite. Hope that it helps.

Comment 16 Kenneth Topp 2018-10-06 00:20:54 UTC
this should make the fedora 29 release notes.

Comment 17 Tomasz Torcz 2018-10-06 08:21:21 UTC
Reopening, my re-test in #c6 was with empty history. Now, when I deleted history.sqlite and put my original (#c4) history/ dir back, dnf crashes:

$ LANG=C LC_ALL=C dnf upgrade
Last metadata expiration check: 0:10:50 ago on Sat Oct  6 10:08:45 2018.
Naruszenie ochrony pamięci (zrzut pamięci)

(the translated message is segmentation fault, core dumped). There is no more information, only this in dmesg:

[30559.851045] dnf[49241]: segfault at 40 ip 00007f7d23b17cfb sp 00007ffee51ec2a0 error 4 in libdnf.so.2[7f7d23aa4000+e6000]
[30559.851053] Code: 00 49 8b 6c 24 68 4d 8b 6c 24 70 4c 39 ed 0f 84 83 00 00 00 0f 1f 00 48 8b 45 00 49 8b 54 24 40 be 01 00 00 00 48 8b 7c 24 38 <4c> 8b 70 40 e8 bc 6a f9 ff 89 c3 85 c0 0f 85 97 00 00 00 48 8b 7c 

$ rpm -q dnf libdnf
dnf-3.6.1-15gaa71348c.fc29.noarch
libdnf-0.20.0-12gd58bb2aa.fc29.x86_64

Comment 18 Tomasz Torcz 2019-01-27 12:59:46 UTC
I see 1632177 is fixed.
Is this one fixed, too? Can I disable dnf COPR and get back to regular repos?https://www.eea.europa.eu/themes/air/air-quality-index

Comment 19 Jaroslav Mracek 2019-01-29 07:47:44 UTC
I believe that the problem is fixed in dnf-4.0.9 with libdnf-0:0.22.3. If the problem reappear please open the new bug, because already multiple issues are discussed here.

Comment 20 giampaolo ferradini 2019-08-09 11:23:54 UTC
Is this bug causing just a DNF crash, or can it cause the entire system to crash?

I am running DNF 4.2.5 on a Fedora 29 KDE, and the entire system froze in the middle of my last update. Not even [Alt + F2/3/etc] worked, nor mouse moving on the screen.
After a reset of the mainboard the system seems to run fine, but running `dnf update` gives no update necessary.

Am a bit of a noob, and came here from the docs, but I can follow instructions to provide logs or else, in case.

My system does come from multiple upgrades.