Created attachment 1865926 [details] Full log from a failing build Description of problem: Since updating to 20211102.0 in the development version of the distribution, we see frequent flaky failures of CordzInfoStatisticsTest.ThreadSafety in absl_cordz_info_statistics_test on the ppc64le architecture. When the test fails, a message like > [mutex.cc : 1682] RAW: Mutex unlocked when destroyed or not locked: v=0x0 is printed. Version-Release number of selected component (if applicable): 20211102.0-1.fc37 How reproducible: Steps to Reproduce: 1. fedpkg scratch-build --srpm --arches ppc64le Actual results: Always succeeds. Expected results: Frequently (roughly 10–80% of the time) fails with the described error. Additional info: I reported this upstream as https://github.com/abseil/abseil-cpp/issues/1132. It seems likely that this is a real problem, but I am not sure we can solve it downstream. It would be easy enough to skip the test on ppc64le, if that’s the way we want to go: > %ifarch ppc64le > # CordzInfoStatisticsTest.ThreadSafety flaky failure on ppc64le > # https://github.com/abseil/abseil-cpp/issues/1132 > # > # 144/182 Test #150: absl_cordz_info_statistics_test > ....................Subprocess aborted***Exception: 0.31 sec > # Running main() from gmock_main.cc > # [==========] Running 14 tests from 1 test suite. > # [----------] Global test environment set-up. > # [----------] 14 tests from CordzInfoStatisticsTest > # [ RUN ] CordzInfoStatisticsTest.Flat > # [ OK ] CordzInfoStatisticsTest.Flat (0 ms) > # … > # [ RUN ] CordzInfoStatisticsTest.ThreadSafety > # [mutex.cc : 1682] RAW: Mutex unlocked when destroyed or not locked: v=0x0 > sed -r -i 's/\bThreadSafety\b/DISABLED_&/' \ > absl/strings/internal/cordz_info_statistics_test.cc > %endif
It looks like this could actually be fixed by disabling LTO on ppc64le. I think that’s an appropriate workaround. (Of course, problems that show up with LTO enabled are usually actually some kind of undefined behavior upstream, but it’s not always easy to find the root cause.) I am also planning to disable LTO on s390x, since it fixes the (reliable) failure in Symbolize.SymbolizeWithMultipleMaps in absl_symbolize_test on that architecture (https://github.com/abseil/abseil-cpp/issues/1133). I’m still inclined to leave LTO enabled on other architectures as long as we don’t observe any specific issues on them.
I take it back. The problem is still present with LTO. It *might* be rarer, or I might have just gotten (un)lucky.
In doing ten or so scratch builds to look at this issue, I observed a couple of other flaky failures. Since these are relatively infrequent, I’m not trying to look into them or report them upstream right now. ----- On aarch64: 10/182 Test #13: absl_sysinfo_test ..................................***Failed 0.06 sec Running main() from /builddir/build/BUILD/googletest-release-1.11.0/googletest/src/gtest_main.cc [==========] Running 4 tests from 1 test suite. [----------] Global test environment set-up. [----------] 4 tests from SysinfoTest [ RUN ] SysinfoTest.NumCPUs [ OK ] SysinfoTest.NumCPUs (0 ms) [ RUN ] SysinfoTest.NominalCPUFrequency /builddir/build/BUILD/abseil-cpp-20211102.0/absl/base/internal/sysinfo_test.cc:54: Failure Expected equality of these values: NominalCPUFrequency() Which is: 3e+09 1.0 Which is: 1 CPU frequency detection was fixed! Please update unittest. ----- On s390x (without LTO): Start 81: absl_random_mocking_bit_gen_test 80/182 Test #81: absl_random_mocking_bit_gen_test ...................***Failed 0.05 sec Running main() from /builddir/build/BUILD/googletest-release-1.11.0/googletest/src/gtest_main.cc [==========] Running 22 tests from 7 test suites. [----------] Global test environment set-up. [----------] 9 tests from BasicMocking [ RUN ] BasicMocking.AllDistributionsAreOverridable [ OK ] BasicMocking.AllDistributionsAreOverridable (0 ms) [ RUN ] BasicMocking.OnDistribution GMOCK WARNING: Uninteresting mock function call - taking default action specified at: /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:82: Function call: Call(1, 1000000) Returns: 20 NOTE: You can safely ignore the above warning unless this call should not happen. Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call. See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details. GMOCK WARNING: Uninteresting mock function call - taking default action specified at: /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:87: Function call: Call(0, 100) Returns: 5 NOTE: You can safely ignore the above warning unless this call should not happen. Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call. See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details. GMOCK WARNING: Uninteresting mock function call - taking default action specified at: /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:92: Function call: Call(1) Returns: 42 NOTE: You can safely ignore the above warning unless this call should not happen. Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call. See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details. GMOCK WARNING: Uninteresting mock function call - taking default action specified at: /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:97: Function call: Call(1) Returns: 500 NOTE: You can safely ignore the above warning unless this call should not happen. Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call. See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details. GMOCK WARNING: Uninteresting mock function call - taking default action specified at: /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:101: Function call: Call(1e-06) Returns: true NOTE: You can safely ignore the above warning unless this call should not happen. Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call. See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details. /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:105: Failure Expected: (absl::Zipf<int>(gen, 1000000, 2.0, 1.0)) != (1221), actual: 1221 vs 1221 GMOCK WARNING: Uninteresting mock function call - taking default action specified at: /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:106: Function call: Call(1000000, 2, 1) Returns: 1221 NOTE: You can safely ignore the above warning unless this call should not happen. Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call. See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details. GMOCK WARNING: Uninteresting mock function call - taking default action specified at: /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:111: Function call: Call(0, 1) Returns: 0.001 NOTE: You can safely ignore the above warning unless this call should not happen. Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call. See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details. GMOCK WARNING: Uninteresting mock function call - taking default action specified at: /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:116: Function call: Call(0, 1000000, 2) Returns: 2040 NOTE: You can safely ignore the above warning unless this call should not happen. Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call. See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details. [ FAILED ] BasicMocking.OnDistribution (0 ms) [ RUN ] BasicMocking.GMockMatchers GMOCK WARNING: Uninteresting mock function call - taking default action specified at: /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:125: Function call: Call(1000000, 2, 1) Returns: 1221 NOTE: You can safely ignore the above warning unless this call should not happen. Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call. See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details. [ OK ] BasicMocking.GMockMatchers (0 ms) [ RUN ] BasicMocking.OverridesWithMultipleGMockExpectations [ OK ] BasicMocking.OverridesWithMultipleGMockExpectations (0 ms) [ RUN ] BasicMocking.DefaultArgument GMOCK WARNING: Uninteresting mock function call - taking default action specified at: /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:145: Function call: Call(1) Returns: 200 NOTE: You can safely ignore the above warning unless this call should not happen. Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call. See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details. GMOCK WARNING: Uninteresting mock function call - taking default action specified at: /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:145: Function call: Call(1) Returns: 200 NOTE: You can safely ignore the above warning unless this call should not happen. Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call. See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details. [ OK ] BasicMocking.DefaultArgument (0 ms) [ RUN ] BasicMocking.MultipleGenerators [ OK ] BasicMocking.MultipleGenerators (0 ms) [ RUN ] BasicMocking.MocksNotTrigeredForIncorrectTypes [ OK ] BasicMocking.MocksNotTrigeredForIncorrectTypes (0 ms) [ RUN ] BasicMocking.FailsOnUnsatisfiedMocks [ OK ] BasicMocking.FailsOnUnsatisfiedMocks (0 ms) [ RUN ] BasicMocking.WillByDefaultWithArgs GMOCK WARNING: Uninteresting mock function call - taking default action specified at: /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:331: Function call: Call(1.7) Returns: 17 NOTE: You can safely ignore the above warning unless this call should not happen. Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call. See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details. GMOCK WARNING: Uninteresting mock function call - taking default action specified at: /builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:331: Function call: Call(0.03) Returns: 0 NOTE: You can safely ignore the above warning unless this call should not happen. Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call. See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details. [ OK ] BasicMocking.WillByDefaultWithArgs (0 ms) [----------] 9 tests from BasicMocking (0 ms total)
For an update on the SysinfoTest.NominalCPUFrequency issue, see discussion in https://src.fedoraproject.org/rpms/abseil-cpp/pull-request/3. That test is now disabled in our package, and will be removed in future upstream releases.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 37 development cycle. Changing version to 37.
I can’t reproduce this, even on Fedora 37, with abseil-cpp-20230802.0; see https://github.com/abseil/abseil-cpp/issues/1132#issuecomment-1675180517. I’m going to consider this probably-fixed-upstream in that release.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 39 development cycle. Changing version to 39.
FEDORA-2023-b5fb7bba8c has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-b5fb7bba8c
FEDORA-2023-b5fb7bba8c has been pushed to the Fedora 39 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-b5fb7bba8c` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-b5fb7bba8c See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-b5fb7bba8c has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report.