Bug 1460303

Summary: [abrt] system-python: __env_in_api(): system-python killed by SIGSEGV
Product: [Fedora] Fedora Reporter: Dominik 'Rathann' Mierzejewski <dominik>
Component: libdbAssignee: Petr Kubat <pkubat>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 25CC: awilliam, bkabrda, chris, cstratak, eric, hhorak, ishcherb, jstanek, jsynacek, mcyprian, mhonek, mhroncok, pkubat, pmatilai, pviktori, rkuska, rmatos, roth, tomspur, torsava
Target Milestone: ---Keywords: CommonBugs
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
URL: https://retrace.fedoraproject.org/faf/reports/bthash/647e50c01260d75ced62728e902b43182e8c5427
Whiteboard: abrt_hash:0427554d9d0af41664826f8af823e26ff59949b0; https://fedoraproject.org/wiki/Common_F26_bugs#libdb-rebuilddb
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-01 12:40:10 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
File: backtrace
none
File: cgroup
none
File: core_backtrace
none
File: dso_list
none
File: environ
none
File: exploitable
none
File: limits
none
File: maps
none
File: mountinfo
none
File: namespaces
none
File: open_fds
none
File: proc_pid_status
none
File: var_log_messages none

Description Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:06 UTC
Description of problem:
dnf update (containing libdb-5.3.28-21.fc25 in the transation)

Version-Release number of selected component:
system-python-3.5.3-6.fc25

Additional info:
reporter:       libreport-2.8.0
backtrace_rating: 4
cmdline:        /usr/libexec/system-python /bin/dnf update
crash_function: __env_in_api
executable:     /usr/libexec/system-python
global_pid:     9905
kernel:         4.11.3-200.fc25.x86_64
pkg_fingerprint: 4089 D8F2 FDB1 9C98
pkg_vendor:     Fedora Project
runlevel:       N 5
type:           CCpp
uid:            0

Truncated backtrace:
Thread no. 1 (10 frames)
 #0 __env_in_api at ../../src/env/env_failchk.c:328
 #1 __env_failchk_int at ../../src/env/env_failchk.c:82
 #2 __env_failchk_pp at ../../src/env/env_failchk.c:58
 #3 db_init at backend/db3.c:504
 #4 db3_dbiOpen at backend/db3.c:841
 #5 pkgdbOpen at rpmdb.c:149
 #6 openDatabase at rpmdb.c:575
 #7 rpmdbOpen at rpmdb.c:605
 #8 rpmtsOpenDB at rpmts.c:96
 #9 rpmts_Match at rpmts-py.c:648

Comment 1 Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:16 UTC
Created attachment 1286452 [details]
File: backtrace

Comment 2 Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:18 UTC
Created attachment 1286453 [details]
File: cgroup

Comment 3 Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:20 UTC
Created attachment 1286454 [details]
File: core_backtrace

Comment 4 Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:22 UTC
Created attachment 1286455 [details]
File: dso_list

Comment 5 Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:24 UTC
Created attachment 1286456 [details]
File: environ

Comment 6 Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:26 UTC
Created attachment 1286457 [details]
File: exploitable

Comment 7 Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:27 UTC
Created attachment 1286458 [details]
File: limits

Comment 8 Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:30 UTC
Created attachment 1286459 [details]
File: maps

Comment 9 Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:32 UTC
Created attachment 1286460 [details]
File: mountinfo

Comment 10 Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:34 UTC
Created attachment 1286461 [details]
File: namespaces

Comment 11 Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:35 UTC
Created attachment 1286462 [details]
File: open_fds

Comment 12 Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:37 UTC
Created attachment 1286463 [details]
File: proc_pid_status

Comment 13 Dominik 'Rathann' Mierzejewski 2017-06-09 15:51:39 UTC
Created attachment 1286464 [details]
File: var_log_messages

Comment 14 Dominik 'Rathann' Mierzejewski 2017-06-09 15:52:54 UTC
Fortunately, rpm --rebuilddb fixes it.

Comment 15 Eric COUTURIER-GUILLAUME 2017-06-09 17:27:06 UTC
Similar problem has been detected:

Unable to complete system update

reporter:       libreport-2.8.0
backtrace_rating: 4
cmdline:        /usr/libexec/system-python /bin/dnf update --refresh
crash_function: __env_in_api
executable:     /usr/libexec/system-python
global_pid:     43698
kernel:         4.11.3-200.fc25.x86_64
package:        system-python-3.5.3-6.fc25
pkg_fingerprint: 4089 D8F2 FDB1 9C98
pkg_vendor:     Fedora Project
reason:         system-python killed by SIGSEGV
runlevel:       N 5
type:           CCpp
uid:            0

Comment 16 Adam Williamson 2017-06-09 23:26:36 UTC
*** Bug 1460379 has been marked as a duplicate of this bug. ***

Comment 17 Panu Matilainen 2017-06-12 09:10:02 UTC
This crash is *after* the libdb update which ended up in dnf crashing first time, right? 

The backtrace shows dnf is in early init stage here, reading config file and figuring distro version, which it certainly should not be doing when it's just about to close shop for the day after printing out "Complete!". 

Switching component to libdb as that's where it's crashing.

Comment 18 Petr Kubat 2017-06-13 07:01:53 UTC
Most of the discussion for this takes place over in https://bugzilla.redhat.com/show_bug.cgi?id=1394862 as the fix for that bug introduced this failure.

Comment 19 Dominik 'Rathann' Mierzejewski 2017-06-13 23:22:17 UTC
(In reply to Panu Matilainen from comment #17)
> This crash is *after* the libdb update which ended up in dnf crashing first
> time, right? 
> 
> The backtrace shows dnf is in early init stage here, reading config file and
> figuring distro version, which it certainly should not be doing when it's
> just about to close shop for the day after printing out "Complete!". 
> 
> Switching component to libdb as that's where it's crashing.

Correct, this is the crash that happened when I tried to run `dnf update' again afterwards. abrt wasn't able to get a usable stack trace from the first crash in the final stage of the libdb update.

Comment 20 Dominik 'Rathann' Mierzejewski 2017-06-13 23:29:45 UTC
It looks like this happened to me because I have the tracer plugin installed:

python3-dnf-plugins-extras-tracer-0.0.12-4.fc25.noarch
python3-tracer-0.6.12-4.fc25.noarch
tracer-common-0.6.12-4.fc25.noarch

Comment 21 Petr Kubat 2017-06-27 07:55:32 UTC
Thanks Dominik for pointing out the issue is with the tracer plugin.

After looking into this some more I have found out this issue will not be possible to mitigate unfortunately. The segfault happens because a posttransaction dnf plugin tries to access the rpmdb in some way and rpmdb's environment is rebuilt along the way.

The segfault happening past the first crash is a result of libdb's fault management system that tries to log any errors hit when inside libdb's functions. However, as the older version of libdb (with older internal structure representation) is the one that creates the fault report and the new version of libdb is the one that reads it, another segfault is hit when trying to access the details of the first hang. Any further hangs should provide the user with proper error messages.

Fortunately this only becomes an issue when upgrading from an older release of libdb (older than libdb-5.3.28-21) and only once as release 21 (and newer) introduce additional locks on the environment to make sure any such crashes cannot happen (the plugin will still fail when unable to access the environment though).

I will leave this bug open for now so that people can find this information more easily.

Comment 22 Petr Kubat 2017-11-01 12:40:10 UTC
This should hopefully be already fixed by band-aids applied to rpm and libsolv.