This is Python 3.8.0b1 on aarch64: ====================================================================== FAIL: test_NULL_ob_type (test.test_gdb.PrettyPrintTests) Ensure that a PyObject* with NULL ob_type is handled gracefully ---------------------------------------------------------------------- Traceback (most recent call last): File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 537, in test_NULL_ob_type self.assertSane('id(42)', File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 508, in assertSane self.get_gdb_repr(source, File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 273, in get_gdb_repr gdb_output = self.get_stack_trace(source, breakpoint=BREAKPOINT_FN, File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 253, in get_stack_trace self.assertEqual(unexpected_errlines, []) AssertionError: Lists differ: ['Missing separate debuginfo for /lib/ld-l[975 chars]bug"] != [] First list contains 12 additional elements. First extra element 0: 'Missing separate debuginfo for /lib/ld-linux-aarch64.so.1' Diff is 1099 characters long. Set self.maxDiff to None to see it. Other test failures with the same problem: FAIL: test_NULL_ptr (test.test_gdb.PrettyPrintTests) FAIL: test_builtin_method (test.test_gdb.PrettyPrintTests) FAIL: test_builtins_help (test.test_gdb.PrettyPrintTests) FAIL: test_bytes (test.test_gdb.PrettyPrintTests) FAIL: test_corrupt_ob_type (test.test_gdb.PrettyPrintTests) FAIL: test_corrupt_tp_flags (test.test_gdb.PrettyPrintTests) FAIL: test_corrupt_tp_name (test.test_gdb.PrettyPrintTests) FAIL: test_dicts (test.test_gdb.PrettyPrintTests) FAIL: test_exceptions (test.test_gdb.PrettyPrintTests) FAIL: test_frames (test.test_gdb.PrettyPrintTests) FAIL: test_frozensets (test.test_gdb.PrettyPrintTests) FAIL: test_getting_backtrace (test.test_gdb.PrettyPrintTests) FAIL: test_int (test.test_gdb.PrettyPrintTests) FAIL: test_lists (test.test_gdb.PrettyPrintTests) FAIL: test_modern_class (test.test_gdb.PrettyPrintTests) FAIL: test_selfreferential_dict (test.test_gdb.PrettyPrintTests) FAIL: test_selfreferential_list (test.test_gdb.PrettyPrintTests) FAIL: test_selfreferential_new_style_instance (test.test_gdb.PrettyPrintTests) FAIL: test_selfreferential_old_style_instance (test.test_gdb.PrettyPrintTests) FAIL: test_sets (test.test_gdb.PrettyPrintTests) FAIL: test_singletons (test.test_gdb.PrettyPrintTests) FAIL: test_strings (test.test_gdb.PrettyPrintTests) FAIL: test_subclassing_list (test.test_gdb.PrettyPrintTests) FAIL: test_subclassing_tuple (test.test_gdb.PrettyPrintTests) FAIL: test_truncation (test.test_gdb.PrettyPrintTests) FAIL: test_tuples (test.test_gdb.PrettyPrintTests) FAIL: test_down_at_bottom (test.test_gdb.StackNavigationTests) FAIL: test_up_at_top (test.test_gdb.StackNavigationTests) FAIL: test_threads (test.test_gdb.PyBtTests) FAILED (failures=30, skipped=16) 1 test failed again: test_gdb == Tests result: FAILURE then FAILURE == This is ppc64le: ====================================================================== FAIL: test_NULL_ob_type (test.test_gdb.PrettyPrintTests) Ensure that a PyObject* with NULL ob_type is handled gracefully ---------------------------------------------------------------------- Traceback (most recent call last): File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 537, in test_NULL_ob_type self.assertSane('id(42)', File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 508, in assertSane self.get_gdb_repr(source, File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 273, in get_gdb_repr gdb_output = self.get_stack_trace(source, breakpoint=BREAKPOINT_FN, File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 253, in get_stack_trace self.assertEqual(unexpected_errlines, []) AssertionError: Lists differ: ['Missing separate debuginfo for /lib64/ld[965 chars]bug"] != [] First list contains 12 additional elements. First extra element 0: 'Missing separate debuginfo for /lib64/ld64.so.2' Diff is 1089 characters long. Set self.maxDiff to None to see it. Other test failures with the same problem: FAIL: test_NULL_ptr (test.test_gdb.PrettyPrintTests) FAIL: test_builtin_method (test.test_gdb.PrettyPrintTests) FAIL: test_builtins_help (test.test_gdb.PrettyPrintTests) FAIL: test_bytes (test.test_gdb.PrettyPrintTests) FAIL: test_corrupt_ob_type (test.test_gdb.PrettyPrintTests) FAIL: test_corrupt_tp_flags (test.test_gdb.PrettyPrintTests) FAIL: test_corrupt_tp_name (test.test_gdb.PrettyPrintTests) FAIL: test_dicts (test.test_gdb.PrettyPrintTests) FAIL: test_exceptions (test.test_gdb.PrettyPrintTests) FAIL: test_frames (test.test_gdb.PrettyPrintTests) FAIL: test_frozensets (test.test_gdb.PrettyPrintTests) FAIL: test_getting_backtrace (test.test_gdb.PrettyPrintTests) FAIL: test_int (test.test_gdb.PrettyPrintTests) FAIL: test_lists (test.test_gdb.PrettyPrintTests) FAIL: test_modern_class (test.test_gdb.PrettyPrintTests) FAIL: test_selfreferential_dict (test.test_gdb.PrettyPrintTests) FAIL: test_selfreferential_list (test.test_gdb.PrettyPrintTests) FAIL: test_selfreferential_new_style_instance (test.test_gdb.PrettyPrintTests) FAIL: test_selfreferential_old_style_instance (test.test_gdb.PrettyPrintTests) FAIL: test_sets (test.test_gdb.PrettyPrintTests) FAIL: test_singletons (test.test_gdb.PrettyPrintTests) FAIL: test_strings (test.test_gdb.PrettyPrintTests) FAIL: test_subclassing_list (test.test_gdb.PrettyPrintTests) FAIL: test_subclassing_tuple (test.test_gdb.PrettyPrintTests) FAIL: test_truncation (test.test_gdb.PrettyPrintTests) FAIL: test_tuples (test.test_gdb.PrettyPrintTests) FAIL: test_down_at_bottom (test.test_gdb.StackNavigationTests) FAIL: test_up_at_top (test.test_gdb.StackNavigationTests) FAIL: test_threads (test.test_gdb.PyBtTests) FAILED (failures=30, skipped=16) 1 test failed again: test_gdb == Tests result: FAILURE then FAILURE ==
Created attachment 1581664 [details] root.log from Koji (aarch64)
Created attachment 1581665 [details] build.log from Koji (aarch64)
Created attachment 1581666 [details] root.log from Koji (ppc64le)
Created attachment 1581667 [details] build.log from Koji (ppc64le) Logs attached. Koji scratch build at https://koji.fedoraproject.org/koji/taskinfo?taskID=35612158
The spec file has: # Expensive optimizations (mainly, profile-guided optimizations) %ifarch %{ix86} x86_64 %bcond_without optimizations %else # On some architectures, the optimized build takes tens of hours, possibly # longer than Koji's 24-hour timeout. Disable optimizations here. %bcond_with optimizations %endif So neither aarch64 or ppc64le has profile-guided optimizations, but there are other arches (armv7hl, s390x) where they are also not enabled.
Dependency changes since last build: glibc-devel 2.29.9000-25.fc31 2.29.9000-28.fc31 mesa-libGL-devel 19.1.0~rc4-1.fc31 19.1.0-1.fc31 libuuid-devel 2.34-0.2.fc31 2.34-1.fc31 systemtap-sdt-devel 4.2-0.20190513git... 4.2-0.20190613git... libappstream-glib 0.7.15-1.fc31 0.7.15-3.fc31 gdb 8.3.50.20190601-15.fc31 8.3.50.20190610-16.fc31 redhat-rpm-config 130-1.fc31 132-1.fc31 gcc-c++ 9.1.1-1.fc31 9.1.1-2.fc31 glibc-all-langpacks 2.29.9000-25.fc31 2.29.9000-28.fc31 bluez-libs-devel 5.50-7.fc31 5.50-8.fc31
Will try to BR glibc-debuginfo to workaround this, but I'm not sure that is even possible. Also, running a python3 scratchbuild to see if there is a similar issue.
Same problem with the python3 package: https://koji.fedoraproject.org/koji/taskinfo?taskID=35613210
(In reply to Miro Hrončok from comment #7) > Will try to BR glibc-debuginfo to workaround this, but I'm not sure that is > even possible. DEBUG util.py:585: BUILDSTDERR: No matching package to install: 'glibc-debuginfo'
https://apps.fedoraproject.org/koschei/package/python3 says: glibc-devel was updated from 2.29.9000-27.fc31 to 2.29.9000-28.fc31 Florian, any idea what is going on here?
(Oour other Python versions appear to skip test_gdb either entirely or at least on power and aarch64.)
(In reply to Miro Hrončok from comment #9) > (In reply to Miro Hrončok from comment #7) > > Will try to BR glibc-debuginfo to workaround this, but I'm not sure that is > > even possible. > > DEBUG util.py:585: BUILDSTDERR: No matching package to install: > 'glibc-debuginfo' This is a missing Koji feature: <https://pagure.io/koji/issue/540> (In reply to Miro Hrončok from comment #10) > https://apps.fedoraproject.org/koschei/package/python3 says: > > glibc-devel was updated from 2.29.9000-27.fc31 to 2.29.9000-28.fc31 > > Florian, any idea what is going on here? There's no explicit change in the glibc spec file related to debugging information. I will try to get an aarch64 or ppc64le system to investigate. In general, you will need to run glibc-debuginfo to run such tests.
I now also got this on x86_64: ====================================================================== FAIL: test_locals_after_up (test.test_gdb.PyLocalsTests) ---------------------------------------------------------------------- Traceback (most recent call last): BUILDSTDERR: File "/builddir/build/BUILD/Python-3.7.3/Lib/test/test_gdb.py", line 964, in test test_gdb failed BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Da5jiz (%check) test_locals_after_up cmds_after_breakpoint=['py-up', 'py-up', 'py-locals']) File "/builddir/build/BUILD/Python-3.7.3/Lib/test/test_gdb.py", line 250, in get_stack_trace self.assertEqual(unexpected_errlines, []) AssertionError: Lists differ: ['Missing separate debuginfo for /lib64/ld[976 chars]bug"] != [] First list contains 12 additional elements. First extra element 0: 'Missing separate debuginfo for /lib64/ld-linux-x86-64.so.2' Diff is 1100 characters long. Set self.maxDiff to None to see it. ----------------------------------------------------------------------
(In reply to Miro Hrončok from comment #5) > So neither aarch64 or ppc64le has profile-guided optimizations, but there > are other arches (armv7hl, s390x) where they are also not enabled. But there is: %ifarch %{arm} s390x -x test_gdb So it can be that the test fails on non-profile-guided optimizations builds. On profile-guided optimizations builds, it is skipped already by logic in upstream tests.
We can fix this by adding to the ignored warnings. Unfortunately, the "dnf" line is Fedora-specific, so it can't go upstream like this: diff -U3 -r Python-3.7.3-orig/Lib/test/test_gdb.py Python-3.7.3/Lib/test/test_gdb.py --- Python-3.7.3-orig/Lib/test/test_gdb.py 2019-03-25 21:21:05.000000000 +0100 +++ Python-3.7.3/Lib/test/test_gdb.py 2019-06-19 15:39:49.312801739 +0200 @@ -230,6 +230,9 @@ 'BFD: ', # ignore all warnings 'warning: ', + # ignore "Missing separate debuginfo" warnings + 'Missing separate debuginfo for ', + "Try: dnf --enablerepo='*debug*' install ", ) for line in errlines: if not line:
Ah, got it: There's an undocumented setting in GDB, `build-id-verbose` that can disable the "Missing separate debuginfo" messages. Given that these tests tend to match exact wording, I don't think an undocumented setting would be a problem. Victor, what do you think? diff -U3 -r Python-3.7.3-orig/Lib/test/test_gdb.py Python-3.7.3/Lib/test/test_gdb.py --- Python-3.7.3-orig/Lib/test/test_gdb.py 2019-03-25 21:21:05.000000000 +0100 +++ Python-3.7.3/Lib/test/test_gdb.py 2019-06-19 16:11:58.443775063 +0200 @@ -172,6 +172,12 @@ # #0 0x00003fffb7dd1798 in PyObject_Print ... 'set print address off', + # Ignore "Missing separate debuginfo" messages, + # which typically carry platform-specific instructions + # (like `debuginfo-install` or `dnf debuginfo install`) + # so they can't be ignored in "ignore_patterns" below. + 'set build-id-verbose 0', + 'run'] # GDB as of 7.4 onwards can distinguish between the
I filed Bug #1722178 to ask if this is a deliberate change in GDB.
(In reply to Florian Weimer from comment #12) > I will try to get an aarch64 or ppc64le system to investigate. In general, > you will need to run glibc-debuginfo to run such tests. Sorry, I can seem to get a matching Fedora machine right now, but I see this issue has already been diagnosed as something not caused by glibc.
Petr: > We can fix this by adding to the ignored warnings. Unfortunately, the "dnf" line is Fedora-specific, so it can't go upstream like this: (...) Honestly, I don't understand why test_gdb is so strict about stderr: it sounds more like a bug that I wanted to fix for years in test_gdb. So I proposed a change upstream: * https://bugs.python.org/issue37362 * https://github.com/python/cpython/pull/14287 That should fix this issue, but also any new "unexpected" message on stderr.
We are good for now. For leftovers, let's follow on them in the upstream ticket.