Bug 1721483

Created attachment 1581664 [details]
root.log from Koji (aarch64)

Created attachment 1581665 [details]
build.log from Koji (aarch64)

Created attachment 1581666 [details]
root.log from Koji (ppc64le)

Created attachment 1581667 [details]
build.log from Koji (ppc64le)

Logs attached. Koji scratch build at https://koji.fedoraproject.org/koji/taskinfo?taskID=35612158

The spec file has:

# Expensive optimizations (mainly, profile-guided optimizations)
%ifarch %{ix86} x86_64
%bcond_without optimizations
%else
# On some architectures, the optimized build takes tens of hours, possibly
# longer than Koji's 24-hour timeout. Disable optimizations here.
%bcond_with optimizations
%endif

So neither aarch64 or ppc64le has profile-guided optimizations, but there are other arches (armv7hl, s390x) where they are also not enabled.

Dependency changes since last build:

 glibc-devel
    2.29.9000-25.fc31
    2.29.9000-28.fc31

 mesa-libGL-devel
    19.1.0~rc4-1.fc31
    19.1.0-1.fc31

 libuuid-devel
    2.34-0.2.fc31
    2.34-1.fc31

 systemtap-sdt-devel
    4.2-0.20190513git...
    4.2-0.20190613git...

 libappstream-glib
    0.7.15-1.fc31
    0.7.15-3.fc31

 gdb
    8.3.50.20190601-15.fc31
    8.3.50.20190610-16.fc31

 redhat-rpm-config
    130-1.fc31
    132-1.fc31

 gcc-c++
    9.1.1-1.fc31
    9.1.1-2.fc31

 glibc-all-langpacks
    2.29.9000-25.fc31
    2.29.9000-28.fc31

 bluez-libs-devel
    5.50-7.fc31
    5.50-8.fc31

Will try to BR glibc-debuginfo to workaround this, but I'm not sure that is even possible.

Also, running a python3 scratchbuild to see if there is a similar issue.

Same problem with the python3 package: https://koji.fedoraproject.org/koji/taskinfo?taskID=35613210

(In reply to Miro Hrončok from comment #7)
> Will try to BR glibc-debuginfo to workaround this, but I'm not sure that is
> even possible.

DEBUG util.py:585:  BUILDSTDERR: No matching package to install: 'glibc-debuginfo'

https://apps.fedoraproject.org/koschei/package/python3 says:

glibc-devel was updated from 2.29.9000-27.fc31 to 2.29.9000-28.fc31

Florian, any idea what is going on here?

(Oour other Python versions appear to skip test_gdb either entirely or at least on power and aarch64.)

(In reply to Miro Hrončok from comment #9)
> (In reply to Miro Hrončok from comment #7)
> > Will try to BR glibc-debuginfo to workaround this, but I'm not sure that is
> > even possible.
> 
> DEBUG util.py:585:  BUILDSTDERR: No matching package to install:
> 'glibc-debuginfo'

This is a missing Koji feature: <https://pagure.io/koji/issue/540>

(In reply to Miro Hrončok from comment #10)
> https://apps.fedoraproject.org/koschei/package/python3 says:
> 
> glibc-devel was updated from 2.29.9000-27.fc31 to 2.29.9000-28.fc31
> 
> Florian, any idea what is going on here?

There's no explicit change in the glibc spec file related to debugging information.

I will try to get an aarch64 or ppc64le system to investigate.  In general, you will need to run glibc-debuginfo to run such tests.

I now also got this on x86_64:

======================================================================
FAIL: test_locals_after_up (test.test_gdb.PyLocalsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
BUILDSTDERR:   File "/builddir/build/BUILD/Python-3.7.3/Lib/test/test_gdb.py", line 964, in test test_gdb failed
BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Da5jiz (%check)
test_locals_after_up
    cmds_after_breakpoint=['py-up', 'py-up', 'py-locals'])
  File "/builddir/build/BUILD/Python-3.7.3/Lib/test/test_gdb.py", line 250, in get_stack_trace
    self.assertEqual(unexpected_errlines, [])
AssertionError: Lists differ: ['Missing separate debuginfo for /lib64/ld[976 chars]bug"] != []
First list contains 12 additional elements.
First extra element 0:
'Missing separate debuginfo for /lib64/ld-linux-x86-64.so.2'
Diff is 1100 characters long. Set self.maxDiff to None to see it.
----------------------------------------------------------------------

(In reply to Miro Hrončok from comment #5)
> So neither aarch64 or ppc64le has profile-guided optimizations, but there
> are other arches (armv7hl, s390x) where they are also not enabled.


But there is:


 %ifarch %{arm} s390x
    -x test_gdb

So it can be that the test fails on non-profile-guided optimizations builds. On profile-guided optimizations builds, it is skipped already by logic in upstream tests.

We can fix this by adding to the ignored warnings. Unfortunately, the "dnf" line is Fedora-specific, so it can't go upstream like this:


diff -U3 -r Python-3.7.3-orig/Lib/test/test_gdb.py Python-3.7.3/Lib/test/test_gdb.py
--- Python-3.7.3-orig/Lib/test/test_gdb.py	2019-03-25 21:21:05.000000000 +0100
+++ Python-3.7.3/Lib/test/test_gdb.py	2019-06-19 15:39:49.312801739 +0200
@@ -230,6 +230,9 @@
             'BFD: ',
             # ignore all warnings
             'warning: ',
+            # ignore "Missing separate debuginfo" warnings
+            'Missing separate debuginfo for ',
+            "Try: dnf --enablerepo='*debug*' install ",
             )
         for line in errlines:
             if not line:

Ah, got it: There's an undocumented setting in GDB, `build-id-verbose` that can disable the "Missing separate debuginfo" messages.
Given that these tests tend to match exact wording, I don't think an undocumented setting would be a problem. Victor, what do you think?


diff -U3 -r Python-3.7.3-orig/Lib/test/test_gdb.py Python-3.7.3/Lib/test/test_gdb.py
--- Python-3.7.3-orig/Lib/test/test_gdb.py	2019-03-25 21:21:05.000000000 +0100
+++ Python-3.7.3/Lib/test/test_gdb.py	2019-06-19 16:11:58.443775063 +0200
@@ -172,6 +172,12 @@
                     # #0 0x00003fffb7dd1798 in PyObject_Print ...
                     'set print address off',
 
+                    # Ignore "Missing separate debuginfo" messages,
+                    # which typically carry platform-specific instructions
+                    # (like `debuginfo-install` or `dnf debuginfo install`)
+                    # so they can't be ignored in "ignore_patterns" below.
+                    'set build-id-verbose 0',
+
                     'run']
 
         # GDB as of 7.4 onwards can distinguish between the

I filed Bug #1722178 to ask if this is a deliberate change in GDB.

(In reply to Florian Weimer from comment #12)
> I will try to get an aarch64 or ppc64le system to investigate.  In general,
> you will need to run glibc-debuginfo to run such tests.

Sorry, I can seem to get a matching Fedora machine right now, but I see this issue has already been diagnosed as something not caused by glibc.

Petr:
> We can fix this by adding to the ignored warnings. Unfortunately, the "dnf" line is Fedora-specific, so it can't go upstream like this: (...)

Honestly, I don't understand why test_gdb is so strict about stderr: it sounds more like a bug that I wanted to fix for years in test_gdb. So I proposed a change upstream:

* https://bugs.python.org/issue37362
* https://github.com/python/cpython/pull/14287

That should fix this issue, but also any new "unexpected" message on stderr.

We are good for now. For leftovers, let's follow on them in the upstream ticket.