Bug 1366004 - [abrt] setroubleshoot-server: service.py:647:_message_cb:SystemError: <built-in function isinstance> returned a result with an error set
Summary: [abrt] setroubleshoot-server: service.py:647:_message_cb:SystemError: <built-...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: setroubleshoot
Version: 25
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Petr Lautrbach
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:3d39da3bb52ff728109278f9b5c...
: 1316617 1429711 1462261 1493092 (view as bug list)
Depends On:
Blocks: F26FinalBlocker 1419245
TreeView+ depends on / blocked
 
Reported: 2016-08-10 17:45 UTC by Chris Murphy
Modified: 2022-03-03 05:48 UTC (History)
305 users (show)

Fixed In Version: setroubleshoot-3.3.12-3.fc25
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-09-28 23:52:30 UTC
Type: ---


Attachments (Terms of Use)
File: backtrace (836 bytes, text/plain)
2016-08-10 17:45 UTC, Chris Murphy
no flags Details
File: environ (88 bytes, text/plain)
2016-08-10 17:45 UTC, Chris Murphy
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Python 23571 0 None None None 2017-05-25 12:47:19 UTC

Description Chris Murphy 2016-08-10 17:45:39 UTC
Version-Release number of selected component:
setroubleshoot-server-3.3.10-2.fc25

Additional info:
reporter:       libreport-2.7.2
cmdline:        /usr/bin/python3 -Es /usr/sbin/setroubleshootd -f ''
executable:     /usr/sbin/setroubleshootd
kernel:         4.8.0-0.rc1.git0.1.fc25.x86_64
pkg_vendor:     Fedora Project
runlevel:       N 5
type:           Python3
uid:            991

Truncated backtrace:
service.py:647:_message_cb:SystemError: <built-in function isinstance> returned a result with an error set

SystemExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.5/site-packages/dbus/service.py", line 647, in _message_cb
    if not isinstance(message, MethodCallMessage):
SystemError: <built-in function isinstance> returned a result with an error set

Local variables in innermost frame:
connection: <dbus._dbus.SystemBus (system) at 0x7f9da9e77ca8>
message: <dbus.lowlevel.MethodCallMessage path: /org/fedoraproject/Setroubleshootd, iface: org.fedoraproject.SetroubleshootdIface, member: avc dest: org.fedoraproject.Setroubleshootd>
self: <setroubleshoot.server.SetroubleshootdDBusObject at /org/fedoraproject/Setroubleshootd at 0x7f9da706dcf8>

Potential duplicate: bug 1352209

Comment 1 Chris Murphy 2016-08-10 17:45:44 UTC
Created attachment 1189773 [details]
File: backtrace

Comment 2 Chris Murphy 2016-08-10 17:45:45 UTC
Created attachment 1189774 [details]
File: environ

Comment 3 Christian Stadelmann 2016-08-28 16:26:23 UTC
Confirmed. Happens on a live Workstation iso image booted in virt-manager.

Comment 4 Christian Stadelmann 2016-08-29 08:06:42 UTC
Proposing as final blocker bug because this crasher is violating "SELinux and crash notifications" final release criterion: 
> There must be no SELinux denial notifications or crash notifications on boot of or during installation from a release-blocking live image, or at first login after a default install of a release-blocking desktop.

Source: https://fedoraproject.org/wiki/Fedora_25_Final_Release_Criteria#SELinux_and_crash_notifications

Comment 5 Geoffrey Marr 2016-08-29 19:53:47 UTC
Discussed during the 2016-08-29 blocker review meeting: [1]

The decision to classify this bug as a RejectedBlocker was made as the bug does not seem to crash on boot like originally reported and is also hard to reproduce reliably. If more details surrounding this bug arise and so warrant reclassification, it should be reclassified as a Blocker.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2016-08-29/f25-blocker-review.2016-08-29-16.00.txt

Comment 6 Chris Murphy 2016-08-30 01:17:33 UTC
OK this is easy to reproduce afterall.
1. Boot Fedora-Workstation-Live-x86_64-25_Alpha-2.iso in a VM.
2. Launch Abrt GUI (Problem Reporting app)

Result: This crash is listed in the System tab. But there was no Gnome notification for it. Happens every boot.

If I launch SELinuxTroubleshooter, it launches. If I go to Problem Reporting again, there's been another crash of setroubleshoot-server.

Comment 7 Adam Williamson 2016-08-30 19:20:01 UTC
yeah, I also see it in abrt-gui just after boot (of today's Rawhide Workstation live) but with no notification.

Comment 8 Jan Vlug 2016-09-03 08:02:01 UTC
Related to bug 1316617?

Comment 9 Petr Viktorin 2016-09-05 13:40:06 UTC
I'll investigate this from the Python point of view.

Comment 10 Petr Viktorin 2016-09-05 13:42:37 UTC
*** Bug 1316617 has been marked as a duplicate of this bug. ***

Comment 11 Petr Viktorin 2016-09-05 17:43:56 UTC
I have a theory about the cause of this bug.

Generally, functions in the Python C API either return a value (PyObject*), or "set an exception" and return NULL. Exiting with a valid value while an exception is set is an error, and this situation is generally checked at the C->Python boundary after a C function returns. It is *not* checked when a function is called.

Setting an exception is done with one of the PyErr_* functions, but ultimately the effect is setting a thread-local pointer to the "current exception".
This "current exception" pointer is cleared before any Python function is called.


Now, I'm not at all familiar with the dbus code, but from Connection__register_object_path in conn_methods.c, I intuit that "_object_path_message" is registered as a callback somewhere in libdbus. The callback calls DBusPyConnection_HandleMessage which then calls (via PyObject_CallFunctionObjArgs) the Python method _message_cb of dbus.service.Object.

What I'm not clear on is: when can the callback be triggered?

My hunch is that it can be triggered when other Python code is running. Specifically, it might happen that when some Python code is currently handling an exception (so the thread-local "current exception" pointer is set), _message_cb is called. The first thing in _message_cb is a call to isinstance(), which succeeds very quickly (it's a pointer compare and return in this case -- no error handling involved). So isinstance() returns a valid value, but since an exception is set from before, Python raises "SystemError: <built-in function isinstance> returned a result with an error set" even though isinstance wasn't the function to set the error.

A question for python-dbus devs: Is that scenario plausible?

Comment 12 Simon McVittie 2016-09-06 11:45:30 UTC
(In reply to Petr Viktorin from comment #11)
> What I'm not clear on is: when can the callback be triggered?

While the main loop (application-supplied, but in practice usually the GLib main loop) is being iterated. Which main loop does setroubleshoot use?

> My hunch is that it can be triggered when other Python code is running.
> Specifically, it might happen that when some Python code is currently
> handling an exception (so the thread-local "current exception" pointer is
> set), _message_cb is called.

I can't see how this could happen unless something iterates the (GLib?) main loop while an exception is being propagated (and before it is caught), which would seem wrong?

(Catching an exception, as in an except: block, uses PyErr_Fetch() which clears the thread-local "current exception" pointer, so merely spinning the main loop in an except: block or the C equivalent would not be enough to cause this problem, AFAICS.)

Comment 13 Simon McVittie 2016-09-06 11:57:00 UTC
(In reply to Petr Viktorin from comment #11)
> My hunch is that it can be triggered when other Python code is running.

Someone who can reproduce this could test this theory as follows: near the beginning of _object_path_message(), after taking the GIL but before doing anything else, use PyErr_Fetch(). If the resulting error objects are non-NULL, log to stderr or something, but do not clear them or release their references (this might require copying them - I'm not sure, my CPython is very rusty). Just before releasing the GIL, use PyErr_Restore() to put the previous exception back.

I would hope that random libraries like dbus-python are not expected to do this as a routine thing whenever C code calls into Python, but the traceback from the pre-existing exception (if there is one) would at least point you towards whatever it was that set the previous exception.

Comment 14 Brian J. Murrell 2016-09-29 10:58:00 UTC
In case anyone missed it, this happens on F24 as reported in the duplicate bug #1316617.  So when this is fixed, an F24 backport needs to be done.

Comment 15 Vernon Trust 2016-10-05 07:23:36 UTC Comment hidden (spam)
Comment 16 Simon McVittie 2016-10-05 10:18:03 UTC
(In reply to Simon McVittie from comment #13)
> (In reply to Petr Viktorin from comment #11)
> > My hunch is that it can be triggered when other Python code is running.
> 
> Someone who can reproduce this could test this theory as follows: near the
> beginning of _object_path_message(), after taking the GIL but before doing
> anything else, use PyErr_Fetch(). If the resulting error objects are
> non-NULL, log to stderr or something, but do not clear them or release their
> references (this might require copying them - I'm not sure, my CPython is
> very rusty). Just before releasing the GIL, use PyErr_Restore() to put the
> previous exception back.

Has anyone who can reproduce this issue tried this?

Comment 17 Christopher Tubbs 2016-10-22 06:04:13 UTC
I saw this crash today. I checked `ausearch --raw | audit2why` to see what happened and saw this around the same time as the crash:

type=AVC msg=audit(1477114536.376:2980): avc:  denied  { sys_ptrace } for  pid=26000 comm="abrt-hook-ccpp" capability=19  scontext=system_u:system_r:abrt_dump_oops_t:s0 tcontext=system_u:system_r:abrt_dump_oops_t:s0 tclass=cap_userns permissive=0

I also saw that almost all the google-chrome tabs I was running had crashed.

Side note: this is probably unrelated, but I noticed that audit2why seems to display '1d 0a' as a single multi-byte character instead of a group separator followed by a newline.

Comment 18 Peter Gervase 2017-02-13 13:08:56 UTC
Similar problem has been detected:

I've been unable to login to kde for the past few days and so I just tried logging into gnome. Things seem OK except I got a few of these problems to report.

reporter:       libreport-2.9.0
cmdline:        /usr/bin/python3 -Es /usr/sbin/setroubleshootd -f ''
crash_function: sighandler
exception_type: SystemExit
executable:     /usr/sbin/setroubleshootd
kernel:         4.10.0-0.rc7.git1.1.fc26.x86_64
package:        setroubleshoot-server-3.3.11-2.fc26
reason:         service.py:647:_message_cb:SystemError: <built-in function isinstance> returned a result with an error set
runlevel:       N 5
type:           Python3
uid:            984

Comment 19 Christian Stadelmann 2017-03-06 16:56:18 UTC
Please make this bug a prioritized bug by adding "PrioritizedBug" to the "Keywords" field.

Rationale: This bug has happened >20k times on F25 and almost 40k times on F24, the bug report has a dozen duplicates, …

Comment 20 Michael Catanzaro 2017-03-06 18:20:53 UTC
Note that we must use the Whiteboard field for this. What is the difference between the fields? Who knows.

Comment 21 Jan Kurik 2017-03-23 16:16:05 UTC
This bug is tracked as Prioritized Bug: https://fedoraproject.org/wiki/Fedora_Program_Management/Prioritized_bugs_and_issues

Comment 22 Steve Grubb 2017-04-04 13:06:11 UTC
Also, there are a few thousand occurrences/month being tracked as bug 1419245.

Comment 23 Mike Simms 2017-05-18 17:23:59 UTC
Similar problem has been detected:

logged into freshly installed F27 MATE Spin desktop session and the alerts just happened. The installation media passed the boot time media test
nightly build 17th May 2017

reporter:       libreport-2.9.1
cmdline:        /usr/bin/python3 -Es /usr/sbin/setroubleshootd -f ''
crash_function: sighandler
exception_type: SystemExit
executable:     /usr/sbin/setroubleshootd
kernel:         4.12.0-0.rc1.git1.1.fc27.x86_64
package:        setroubleshoot-server-3.3.11-3.fc26
reason:         service.py:647:_message_cb:SystemError: <built-in function isinstance> returned a result with an error set
runlevel:       N 5
type:           Python3
uid:            991

Comment 24 Mike Simms 2017-05-19 12:19:43 UTC
Similar problem has been detected:

just logging into desktop session of rawhide mate

reporter:       libreport-2.9.1
cmdline:        /usr/bin/python3 -Es /usr/sbin/setroubleshootd -f ''
crash_function: sighandler
exception_type: SystemExit
executable:     /usr/sbin/setroubleshootd
kernel:         4.12.0-0.rc1.git1.1.fc27.x86_64
package:        setroubleshoot-server-3.3.11-3.fc26
reason:         service.py:647:_message_cb:SystemError: <built-in function isinstance> returned a result with an error set
runlevel:       N 5
type:           Python3
uid:            991

Comment 25 Jan Kurik 2017-05-25 08:40:48 UTC
I would like to ask for a status update. Is there any progress on this bug ?
This bug is on the list of Prioritized bugs https://fedoraproject.org/wiki/Fedora_Program_Management/Prioritized_bugs_and_issues and a fix is considered as important for Fedora users.

Thanks,
Jan

Comment 26 Petr Viktorin 2017-05-25 08:48:44 UTC
Is there a way to reliably reproduce this?

Comment 27 Christian Stadelmann 2017-05-25 08:52:29 UTC
(In reply to Petr Viktorin from comment #26)
> Is there a way to reliably reproduce this?

I don't know one, because for me this crasher sporadically happens without any user interaction, just after logging in to my GNOME/Wayland session.

Comment 28 Charalampos Stratakis 2017-05-25 12:38:25 UTC
Pasting here the comment from the duplicate as it is important information.

(In reply to Simon McVittie from comment #14)
> (In reply to leigh scott from comment #12)
> > (In reply to Petr Lautrbach from comment #9)
> > > It seems to be a bug in dbus-python package or lower.
> > 
> > Is this a dbus-python issue?
> 
> Something is going wrong at the interface between Python and C, and I'm not
> sure how it would be dbus-python's fault.
> 
> dbus-python has defined a type named MethodCallMessage, and is doing this:
> 
> def _message_cb(self, connection, message):
>     if not isinstance(message, MethodCallMessage):
>         return
> 
> and the SystemError we're getting is effectively an assertion failure from
> Python, saying that something is using Python's C APIs wrong: functions are
> meant to do exactly one of returning a value or raising an exception, but in
> this case isinstance() has done both. isinstance is a built-in part of the
> Python interpreter.
> 
> The SystemError has the exception raised by isinstance() attached to it as a
> chained exception; it would be useful if abrt could log that.
> 
> I don't know how this can happen; it's possible that this is a result of a
> bug in the C parts of dbus-python, or in any other C code used by
> setroubleshoot. People who work on the Python interpreter itself would
> perhaps know more.
> 
> If this was reproducible, then a developer who can reproduce it could attach
> a debugger to the Python interpreter and break on the line in
> _Py_CheckFunctionResult() where this SystemError is thrown; details of the
> backtrace and error state at that point might be enlightening.

Unfortunately we do not have a reproducer for this issue, and we are not familiar with the dbus-python codebase, however IMO this should be investigated from the dbus-python point of view and people familiar with its codebase and the way it interfaces with the Python's interpreter C API.

Python-SIG can provide some help but we do not have much information to move this issue forward and it doesn't seem to be an interpreter issue from the comments posted so far (and I don't why it's being pointed out as such).

Reassigning to dbus-python.

Comment 29 Charalampos Stratakis 2017-05-25 12:47:19 UTC
(Possibly) relevant upstream issue: https://bugs.python.org/issue23571

Comment 30 Charalampos Stratakis 2017-05-25 12:54:35 UTC
A possible way to retrieve more information on the issue, would be to provide a build of dbus-python which uses the python3-debug binary and try to capture a traceback, which I suppose would be easier on user's systems that sporadically face this issue.

Would it be possible to provide such a build, maybe through COPR?

Comment 31 Christian Stadelmann 2017-05-26 12:00:54 UTC
(In reply to Christian Stadelmann from comment #27)
> (In reply to Petr Viktorin from comment #26)
> > Is there a way to reliably reproduce this?
> 
> I don't know one, because for me this crasher sporadically happens without
> any user interaction, just after logging in to my GNOME/Wayland session.

Sometimes it also happens on a running session. No user interaction required either.

Comment 32 Christian Stadelmann 2017-05-26 12:04:07 UTC
Related bug: https://bugzilla.redhat.com/show_bug.cgi?id=1429711 (probably a duplicate)

Comment 33 liam 2017-05-26 12:40:43 UTC
I'm not sure if it's relevant, but this has also been happening on a headless CentOS 7 Server machine

Comment 34 Steve Grubb 2017-05-26 14:18:57 UTC
My reading of the bug is that something happens where setroubleshoot decides to exit (server.py)

def sighandler(signum, frame):
    log_debug("received signal=%s" % signum)
    import setroubleshoot.config as config
    if signum == signal.SIGHUP:
        log_debug("reloading configuration file")
        config.config_init()
        return
    import sys
    sys.exit()

During exit, dbus does a callback (service.py)

    def _message_cb(self, connection, message):
        if not isinstance(message, MethodCallMessage):
            return

Then isinstance returns an exception. The documentation for isinstance says:  

If classinfo is not a class, type, or tuple of classes, types, and such tuples, a TypeError exception is raised.

Could this be a shutdown race where MethodCallMessage has been torn down but the interface is still active? Is simply calling sys.exit() in the troubleshooter OK or should it do something to deactivate dbus first?

Comment 35 Simon McVittie 2017-05-26 15:06:45 UTC
(In reply to Steve Grubb from comment #34)
> Then isinstance returns an exception.

No, it seems to be worse than that: isinstance() returns an exception *and* a result. It is an invariant of the CPython[1] APIs that it is not meant to do that under any circumstances (any method call is meant to return exactly one of an exception *or* a result), hence the SystemError.

If the offending method was something implemented in dbus-python's C code then I would agree that it was potentially a dbus-python bug, but dbus-python is not the component that implements the isinstance() builtin - that's part of CPython.

(If the offending method was something implemented in dbus-python's C code, I would probably still be asking for a C-level backtrace. I can't do much with C bugs based on only a Python backtrace.)

[1] when I say CPython I mean the C reference implementation of Python, so Fedora's python package

> Could this be a shutdown race where MethodCallMessage has been torn down but
> the interface is still active? Is simply calling sys.exit() in the
> troubleshooter OK or should it do something to deactivate dbus first?

I would have expected this to be fine. I mean, you can forcibly unexport objects if you want to, but once you've called sys.exit() I would expect that the main loop (presumably the GLib main loop, but nobody has confirmed that yet?) should exit rather than continuing to be dispatched, which means dbus-python should no longer have any opportunity to call into Python code from its C code.

Looking at the stack trace on Bug #1429711, it might be helpful to issue the sys.exit() call from a deferred callback (a GLib idle, if you're using the GLib main loop). That would maybe avoid some re-entrancy? Or you could use the PyGI binding for g_unix_signal_add() to get a similar practical effect, deferring handling of the POSIX signal until the next time you iterate the main loop - that's much safer and more predictable than POSIX-style asynchronous signals.

In the case of fatal POSIX signals (all other than SIGHUP) it would probably be safer still if you just didn't try to catch the POSIX signal at all, and let it kill your process in the normal way. The best signal handling is the signal handler whose implementation is 0 lines long :-)

https://bugzilla.redhat.com/show_bug.cgi?id=1366004#c13 contains a suggestion for what a developer who experiences this bug could do to provide more useful information. Capturing a C backtrace would also be valuable.

I do not run SELinux or Fedora, so I am not in a position to debug this.

Comment 36 leigh scott 2017-05-27 07:09:54 UTC
(In reply to Charalampos Stratakis from comment #30)
> A possible way to retrieve more information on the issue, would be to
> provide a build of dbus-python which uses the python3-debug binary 

Is this what you want?

mkdir python3-build; pushd python3-build
%configure PYTHON=python3-debug PYTHON_LIBS='-lpython%{python3_version}dm' PYTHON_CONFIG=python%{python3_version}dm-config
make %{?_smp_mflags}
popd



> and try to capture a traceback, which I suppose would be easier on user's systems
> that sporadically face this issue.

I have zero interest in investigating this issue


> Would it be possible to provide such a build, maybe through COPR?

Done

https://copr.fedorainfracloud.org/coprs/leigh123linux/dbus-python/

Comment 37 Charalampos Stratakis 2017-05-29 13:32:49 UTC
(In reply to leigh scott from comment #36)

> mkdir python3-build; pushd python3-build
> %configure PYTHON=python3-debug PYTHON_LIBS='-lpython%{python3_version}dm'
> PYTHON_CONFIG=python%{python3_version}dm-config
> make %{?_smp_mflags}
> popd
> 

Yes that should work.
> 
> I have zero interest in investigating this issue

Ok?

> https://copr.fedorainfracloud.org/coprs/leigh123linux/dbus-python/

Thanks!

Comment 38 Charalampos Stratakis 2017-05-29 13:37:07 UTC
(In reply to Christian Stadelmann from comment #27)
> (In reply to Petr Viktorin from comment #26)
> > Is there a way to reliably reproduce this?
> 
> I don't know one, because for me this crasher sporadically happens without
> any user interaction, just after logging in to my GNOME/Wayland session.

Would you be able to install python3-debug and python3-dbus from the provided copr repository and see if you can manage to get the crash again? That should provide us a more informative traceback.

Comment 39 Christian Stadelmann 2017-05-30 06:57:16 UTC
(In reply to Charalampos Stratakis from comment #38)
> (In reply to Christian Stadelmann from comment #27)
> > (In reply to Petr Viktorin from comment #26)
> > > Is there a way to reliably reproduce this?
> > 
> > I don't know one, because for me this crasher sporadically happens without
> > any user interaction, just after logging in to my GNOME/Wayland session.
> 
> Would you be able to install python3-debug and python3-dbus from the
> provided copr repository and see if you can manage to get the crash again?
> That should provide us a more informative traceback.

Done. Waiting for the crash to happen again.

Comment 40 Christian Stadelmann 2017-05-30 10:59:20 UTC
(In reply to leigh scott from comment #36)
> https://copr.fedorainfracloud.org/coprs/leigh123linux/dbus-python/

This COPR ships broken packages. After running `# dnf copr enable leigh123linux/dbus-python` followed by `# dnf upgrade`, it seems like all Python3 applications using dbus crash on start with this backtrace:


types.py:6:<module>:ImportError: /usr/lib64/python3.6/site-packages/_dbus_bindings.so: undefined symbol: _Py_RefTotal

Traceback (most recent call last):
  File "/usr/bin/gnome-abrt", line 63, in <module>
    from gnome_abrt.dbus_problems import (get_standard_problems_source,
  File "/usr/lib64/python3.6/site-packages/gnome_abrt/dbus_problems.py", line 22, in <module>
    import dbus
  File "/usr/lib64/python3.6/site-packages/dbus/__init__.py", line 77, in <module>
    import dbus.types as types
  File "/usr/lib64/python3.6/site-packages/dbus/types.py", line 6, in <module>
    from _dbus_bindings import (
ImportError: /usr/lib64/python3.6/site-packages/_dbus_bindings.so: undefined symbol: _Py_RefTotal


This issue affects gnome-abrt, firewalld, dnfdaemon and probably some more packages.

So I rolled back the upgrade and disabled the copr.

Comment 41 Charalampos Stratakis 2017-05-30 15:31:57 UTC
(In reply to Christian Stadelmann from comment #40)
> (In reply to leigh scott from comment #36)
> > https://copr.fedorainfracloud.org/coprs/leigh123linux/dbus-python/
> 
> This COPR ships broken packages. After running `# dnf copr enable
> leigh123linux/dbus-python` followed by `# dnf upgrade`, it seems like all
> Python3 applications using dbus crash on start with this backtrace:
> 
> 
> types.py:6:<module>:ImportError:
> /usr/lib64/python3.6/site-packages/_dbus_bindings.so: undefined symbol:
> _Py_RefTotal
> 
> Traceback (most recent call last):
>   File "/usr/bin/gnome-abrt", line 63, in <module>
>     from gnome_abrt.dbus_problems import (get_standard_problems_source,
>   File "/usr/lib64/python3.6/site-packages/gnome_abrt/dbus_problems.py",
> line 22, in <module>
>     import dbus
>   File "/usr/lib64/python3.6/site-packages/dbus/__init__.py", line 77, in
> <module>
>     import dbus.types as types
>   File "/usr/lib64/python3.6/site-packages/dbus/types.py", line 6, in
> <module>
>     from _dbus_bindings import (
> ImportError: /usr/lib64/python3.6/site-packages/_dbus_bindings.so: undefined
> symbol: _Py_RefTotal
> 
> 
> This issue affects gnome-abrt, firewalld, dnfdaemon and probably some more
> packages.
> 
> So I rolled back the upgrade and disabled the copr.

Indeed, tried it myself, it is an issue with the linker and the flags used for the configure script.

I forked the copr repo to try to fix the builds, however the python3 tests are failing so I disabled them for the time being.

Could you try to use this repo?

https://copr.fedorainfracloud.org/coprs/cstratak/dbus-python/

Comment 42 Charalampos Stratakis 2017-05-30 15:46:43 UTC
I'll work on providing some custom cpython builds as well so we can get some more info out of the isinstance function.

Comment 43 Bill Gianopoulos 2017-05-30 16:00:07 UTC
Just so people understand the importance about this (because perhaps it seems trivial) as far as I know what is happening here is some serious issue is happening but in the course of trying to report it this error occurs in the reporting process which is not that important.  What IS important is as far as I have been able to determine the original issue DOES NOT get reported when this occurs.

Comment 44 Fedora Admin XMLRPC Client 2017-05-30 17:11:16 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 45 Fedora Admin XMLRPC Client 2017-05-31 13:26:18 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 46 Christian Stadelmann 2017-06-06 11:48:16 UTC
Update:
I have not seen this crasher in a week with python3-debug installed. Is it possible, that installing python3-debug makes this bug go away? (Heisenbug)

I am running Fedora 26. On Retrace, it still gets crashes:
https://retrace.fedoraproject.org/faf/reports/1518755/

Comment 47 Charalampos Stratakis 2017-06-14 15:37:04 UTC
pviktori provided a patch [0] that might give us some more information on the issue at hand.

Currently I am building python3 for F25 as well as F26 in copr. You can enable this repository to try it out: https://copr.fedorainfracloud.org/coprs/cstratak/py3_dbus/

Christian would you be able to uninstall python3-debug, sync the dbus packages (dbus-python and python3-dbus) with the fedora builds and install python3 from the provided repository?

[0] https://github.com/encukou/cpython/commit/43c04ce09f73c87d23b7e4c4fe9da5081a8af2fe

Comment 48 George R. Goffe 2017-06-28 10:02:22 UTC
Similar problem has been detected:

This system is a freshly installed fedora 27 x86_64 (rawhide). There are numerous problemş this is just onė of them.

Ihad just started KD

reporter:       libreport-2.9.1
cmdline:        /usr/bin/python3 -Es /usr/sbin/setroubleshootd -f ''
crash_function: sighandler
exception_type: SystemExit
executable:     /usr/sbin/setroubleshootd
kernel:         4.12.0-0.rc6.git3.1.fc27.x86_64
package:        setroubleshoot-server-3.3.12-1.fc27
reason:         service.py:647:_message_cb:SystemError: <built-in function isinstance> returned a result with an error set
runlevel:       N 3
type:           Python3
uid:            989

Comment 49 Fedora Admin XMLRPC Client 2017-07-05 20:35:57 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 50 Peter Gervase 2017-08-01 11:37:51 UTC
Similar problem has been detected:

I have been trying to get vpn working. I've been updating / downgrading rpms like plasma-nm-openvpn and NetworkManager-openvpn-profiles. I don't recall anything specifically for this abrt though.

reporter:       libreport-2.9.1
cmdline:        /usr/bin/python3 -Es /usr/sbin/setroubleshootd -f ''
crash_function: sighandler
exception_type: SystemExit
executable:     /usr/sbin/setroubleshootd
kernel:         4.12.0-0.rc2.git3.1.fc27.x86_64
package:        setroubleshoot-server-3.3.11-3.fc26
reason:         service.py:647:_message_cb:SystemError: <built-in function isinstance> returned a result with an error set
runlevel:       N 5
type:           Python3
uid:            984

Comment 51 Peter Gervase 2017-08-08 11:55:54 UTC
Similar problem has been detected:

I turned on my computer this morning. I went to a console login rather than gui, so taking a quick peek around, I ran `journalctl`. I saw some selinux messages,
[root@t440s ~]# journalctl -S today| grep sealert| awk '{print $1 = "", $2 = "", $3 = ""; print}'| sort | uniq
  
   t440s.gahzinia.com dbus-daemon[839]: [system] Activating service name='org.fedoraproject.Setroubleshootd' requested by ':1.388' (uid=1001 pid=1367 comm="/usr/bin/python3 -Es /usr/bin/sealert -l cce27af0-" label="unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023") (using servicehelper)
   t440s.gahzinia.com setroubleshoot[1100]: SELinux is preventing accounts-daemon from using the dac_read_search capability. For complete SELinux messages run: sealert -l cce27af0-6e6d-4698-8d3e-fd64626c7d4b
   t440s.gahzinia.com setroubleshoot[1100]: SELinux is preventing sm-notify from using the dac_read_search capability. For complete SELinux messages run: sealert -l 8042f71d-c1c5-4f7b-9e89-6a25e2b820d1
   t440s.gahzinia.com setroubleshoot[1237]: SELinux is preventing accounts-daemon from using the dac_read_search capability. For complete SELinux messages run: sealert -l cce27af0-6e6d-4698-8d3e-fd64626c7d4b
   t440s.gahzinia.com setroubleshoot[1237]: SELinux is preventing login from using the dac_read_search capability. For complete SELinux messages run: sealert -l 8b9c8bff-60c8-4359-b9c8-9cb6bdf01a22
   t440s.gahzinia.com setroubleshoot[1237]: SELinux is preventing unix_chkpwd from using the dac_read_search capability. For complete SELinux messages run: sealert -l 731f106e-18c9-4c95-b624-de0242019356
   t440s.gahzinia.com setroubleshoot[1237]: SELinux is preventing unix_chkpwd from using the dac_read_search capability. For complete SELinux messages run: sealert -l d5442137-472b-4ed3-84e7-b3bc67468837
   t440s.gahzinia.com setroubleshoot[1369]: SELinux is preventing systemd from 'read, write' accesses on the chr_file /dev/dri/card0. For complete SELinux messages run: sealert -l 22eafd95-9c96-4796-be03-3e148894f7b7
   t440s.gahzinia.com setroubleshoot[1369]: SELinux is preventing systemd from 'read, write' accesses on the chr_file /dev/input/event10. For complete SELinux messages run: sealert -l 52b495c5-b183-4aaa-a1f6-b71adc53d93a
   t440s.gahzinia.com setroubleshoot[1369]: SELinux is preventing systemd from 'read, write' accesses on the chr_file /dev/input/event1. For complete SELinux messages run: sealert -l 52b495c5-b183-4aaa-a1f6-b71adc53d93a
   t440s.gahzinia.com setroubleshoot[1369]: SELinux is preventing systemd from 'read, write' accesses on the chr_file /dev/input/event20. For complete SELinux messages run: sealert -l 52b495c5-b183-4aaa-a1f6-b71adc53d93a
   t440s.gahzinia.com setroubleshoot[1369]: SELinux is preventing systemd from 'read, write' accesses on the chr_file /dev/input/event2. For complete SELinux messages run: sealert -l 52b495c5-b183-4aaa-a1f6-b71adc53d93a
   t440s.gahzinia.com setroubleshoot[1369]: SELinux is preventing systemd from 'read, write' accesses on the chr_file /dev/input/event3. For complete SELinux messages run: sealert -l 52b495c5-b183-4aaa-a1f6-b71adc53d93a
   t440s.gahzinia.com setroubleshoot[1369]: SELinux is preventing systemd from 'read, write' accesses on the chr_file /dev/input/event4. For complete SELinux messages run: sealert -l 52b495c5-b183-4aaa-a1f6-b71adc53d93a
   t440s.gahzinia.com setroubleshoot[1369]: SELinux is preventing systemd from 'read, write' accesses on the chr_file /dev/input/event5. For complete SELinux messages run: sealert -l 52b495c5-b183-4aaa-a1f6-b71adc53d93a
   t440s.gahzinia.com setroubleshoot[1369]: SELinux is preventing systemd from 'read, write' accesses on the chr_file /dev/input/event6. For complete SELinux messages run: sealert -l 52b495c5-b183-4aaa-a1f6-b71adc53d93a
   t440s.gahzinia.com setroubleshoot[1369]: SELinux is preventing systemd from 'read, write' accesses on the chr_file /dev/input/event7. For complete SELinux messages run: sealert -l 52b495c5-b183-4aaa-a1f6-b71adc53d93a
   t440s.gahzinia.com setroubleshoot[1369]: SELinux is preventing systemd from 'read, write' accesses on the chr_file /dev/input/event8. For complete SELinux messages run: sealert -l 52b495c5-b183-4aaa-a1f6-b71adc53d93a
   t440s.gahzinia.com setroubleshoot[1369]: SELinux is preventing systemd from 'read, write' accesses on the chr_file /dev/input/event9. For complete SELinux messages run: sealert -l 52b495c5-b183-4aaa-a1f6-b71adc53d93a
   t440s.gahzinia.com setroubleshoot[1369]: SELinux is preventing systemd-tmpfile from using the dac_read_search capability. For complete SELinux messages run: sealert -l 0f6e2c86-722f-48e3-9ec9-3b8d26571ddc
[root@t440s ~]# 

and so from the console I ran 
$ sealert -l cce27af0-6e6d-4698-8d3e-fd64626c7d4b > sealert
$ startx
When I got logged in to the gui, I got this notice from abrt.

reporter:       libreport-2.9.1
cmdline:        /usr/bin/python3 -Es /usr/sbin/setroubleshootd -f ''
crash_function: sighandler
exception_type: SystemExit
executable:     /usr/sbin/setroubleshootd
kernel:         4.13.0-0.rc3.git4.1.fc27.x86_64
package:        setroubleshoot-server-3.3.12-1.fc27
reason:         service.py:647:_message_cb:SystemError: <built-in function isinstance> returned a result with an error set
runlevel:       N 5
type:           Python3
uid:            988

Comment 52 Paul Whalen 2017-08-21 20:44:53 UTC
Similar problem has been detected:

Log in to Xfce spin, crash shortly after.

reporter:       libreport-2.9.1
cmdline:        /usr/bin/python3 -Es /usr/sbin/setroubleshootd -f ''
crash_function: sighandler
exception_type: SystemExit
executable:     /usr/sbin/setroubleshootd
kernel:         4.13.0-0.rc5.git1.1.fc27.armv7hl
package:        setroubleshoot-server-3.3.12-3.fc27
reason:         service.py:647:_message_cb:SystemError: <built-in function isinstance> returned a result with an error set
runlevel:       unknown
type:           Python3
uid:            990

Comment 53 Matthew Miller 2017-09-13 14:41:02 UTC
So, this seems to have gotten lost in the shuffle somehow, but is still important. I don't know who can fix it, but "Alternative GTK desktop environment SIG" doesn't sound right. I'm reassigning back to setroubleshoot as a first stab at routing this correctly.

Comment 54 Petr Lautrbach 2017-09-14 06:21:02 UTC
Thanks all for the investigation.

The following patch [1] based on your comments about sys.exit and POSIX signal seems to fix it [1].

Testing builds should be available soon in my COPR repo [2]

[1] https://pagure.io/setroubleshoot/c/af2aa6c0491cb719d4be2ce60bc79c0541d18a44?branch=master

[2] https://copr.fedorainfracloud.org/coprs/plautrba/setroubleshoot/

Comment 55 Charalampos Stratakis 2017-09-14 09:09:27 UTC
(In reply to Petr Lautrbach from comment #54)
> Thanks all for the investigation.
> 
> The following patch [1] based on your comments about sys.exit and POSIX
> signal seems to fix it [1].
> 
> Testing builds should be available soon in my COPR repo [2]
> 
> [1]
> https://pagure.io/setroubleshoot/c/
> af2aa6c0491cb719d4be2ce60bc79c0541d18a44?branch=master
> 
> [2] https://copr.fedorainfracloud.org/coprs/plautrba/setroubleshoot/

That is great news! Thanks for that.



Could people who experience the issue install the builds from copr and test them?

Comment 56 robert fairbrother 2017-09-14 09:32:57 UTC
Similar problem has been detected:

I havent been able to update selinux policy targeted

reporter:       libreport-2.8.0
cmdline:        /usr/bin/python3 -Es /usr/sbin/setroubleshootd -f ''
event_log:      2017-09-14-03:31:16> ('report_uReport' completed successfully)
executable:     /usr/sbin/setroubleshootd
kernel:         4.12.9-200.fc25.i686+PAE
package:        setroubleshoot-server-3.3.12-1.fc25
pkg_fingerprint: 4089 D8F2 FDB1 9C98
pkg_vendor:     Fedora Project
reason:         service.py:647:_message_cb:SystemError: <built-in function isinstance> returned a result with an error set
runlevel:       N 5
type:           Python3
uid:            991

Comment 57 Petr Lautrbach 2017-09-15 05:53:59 UTC
*** Bug 1429711 has been marked as a duplicate of this bug. ***

Comment 58 Petr Lautrbach 2017-09-15 05:59:46 UTC
*** Bug 1462261 has been marked as a duplicate of this bug. ***

Comment 59 Fedora Update System 2017-09-15 09:49:45 UTC
setroubleshoot-3.3.12-4.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-a3eb5dffa6

Comment 60 Fedora Update System 2017-09-15 10:41:46 UTC
setroubleshoot-3.3.12-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d82b89fe72

Comment 61 Fedora Update System 2017-09-15 10:58:00 UTC
setroubleshoot-3.3.12-2.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-be0959ff2c

Comment 62 Fedora Update System 2017-09-15 16:53:29 UTC
setroubleshoot-3.3.12-4.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-a3eb5dffa6

Comment 63 Fedora Update System 2017-09-16 04:23:27 UTC
setroubleshoot-3.3.12-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d82b89fe72

Comment 64 Fedora Update System 2017-09-16 05:28:57 UTC
setroubleshoot-3.3.12-2.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-be0959ff2c

Comment 65 Fedora Update System 2017-09-19 07:13:47 UTC
setroubleshoot-3.3.12-5.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-a3eb5dffa6

Comment 66 Fedora Update System 2017-09-19 07:18:06 UTC
setroubleshoot-3.3.12-3.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-be0959ff2c

Comment 67 Fedora Update System 2017-09-19 07:20:00 UTC
setroubleshoot-3.3.12-3.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d82b89fe72

Comment 68 Petr Lautrbach 2017-09-19 15:14:36 UTC
*** Bug 1493092 has been marked as a duplicate of this bug. ***

Comment 69 Fedora Update System 2017-09-19 23:26:09 UTC
setroubleshoot-3.3.12-5.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-a3eb5dffa6

Comment 70 Fedora Update System 2017-09-20 00:21:06 UTC
setroubleshoot-3.3.12-3.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d82b89fe72

Comment 71 Fedora Update System 2017-09-20 01:19:37 UTC
setroubleshoot-3.3.12-3.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-be0959ff2c

Comment 72 Matthew Miller 2017-09-27 14:19:36 UTC
Can people please test this update and provide karma in bodhi? 

Note that if you did so when these updates first appeared that there is a newer update which supersedes that and so re-testing is required. Thank you!

Comment 73 Petr Lautrbach 2017-09-27 14:27:06 UTC
Note that I've already submitted all free updates for f25, f26 and f27 to stable this morning (8am CEST).

Comment 74 Fedora Update System 2017-09-28 19:54:46 UTC
setroubleshoot-3.3.12-3.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 75 Fedora Update System 2017-09-28 23:52:30 UTC
setroubleshoot-3.3.12-3.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 76 Fedora Update System 2017-09-30 06:31:41 UTC
setroubleshoot-3.3.12-5.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 78 Alex Gambler 2020-01-23 09:36:22 UTC
(In reply to Vernon Trust from comment #15)
> Thanks, Brian. When I tried to fix that, Python's  "SystemError: <built-in
> function is instance> returned an error set result as well. And it wasn't a
> function to set the error, just like Petr said. Python is new to me, I'm
> used to Java, but I'm trying to apply something new.
> Trying to make my start-up https://casinority.com/za/ clear
> Vernon


Thank you Vernon too, I just fixed the same errors

Comment 80 mabel 2021-04-05 18:55:00 UTC Comment hidden (spam)
Comment 81 Brian J. Murrell 2021-04-05 19:02:17 UTC
Can somebody please cancel this @mabel account?  It's clearly for nothing other than spamming OT crap to tickets.

https://stopforumspam.com/ipcheck/103.92.212.25 which is in the list of evidence found at https://www.google.com/search?q=mabelgoodrich%40gmail.com&oq=mabelgoodrich%40gmail.com&aqs=chrome..69i57j69i58.1159j0j7&sourceid=chrome&ie=UTF-8

Being the victim of this spammer, you should also report it to abuse@gmail.com and get that account closed down.

Comment 82 Kamil Páral 2021-04-06 09:51:42 UTC
Brian, it has been marked as spam, see the comment. More info here in 5) and 6):
https://bugzilla.redhat.com/page.cgi?id=faq.html#getting-help-spam1

You can also email the Bugzilla admins (see above) if you feel a further action is required.

Comment 83 SalmanAlmas 2021-07-08 14:49:35 UTC Comment hidden (spam)
Comment 84 SalmanAlmas 2021-07-12 12:06:30 UTC Comment hidden (spam)
Comment 85 Prabhu 2021-08-22 10:33:56 UTC Comment hidden (spam)
Comment 87 Alice G. Sutton 2022-02-25 05:41:24 UTC Comment hidden (spam)

Note You need to log in before you can comment on or make changes to this bug.