Bug 2064025

Summary: Flaky build failures on ppc64le due to CordzInfoStatisticsTest.ThreadSafety
Product: [Fedora] Fedora Reporter: Ben Beasley <code>
Component: abseil-cppAssignee: Ben Beasley <code>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 39CC: code, richmattes
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: abseil-cpp-20230802.0-1.fc39 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-09-18 00:16:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2229015    
Bug Blocks: 1071880    
Attachments:
Description Flags
Full log from a failing build none

Description Ben Beasley 2022-03-14 21:40:35 UTC 
Created attachment 1865926 [details]
Full log from a failing build

Description of problem:

Since updating to 20211102.0 in the development version of the distribution, we see frequent flaky failures of CordzInfoStatisticsTest.ThreadSafety in absl_cordz_info_statistics_test on the ppc64le architecture. When the test fails, a message like
> [mutex.cc : 1682] RAW: Mutex unlocked when destroyed or not locked: v=0x0
is printed.

Version-Release number of selected component (if applicable): 20211102.0-1.fc37


How reproducible:


Steps to Reproduce:
1. fedpkg scratch-build --srpm --arches ppc64le

Actual results:

Always succeeds.

Expected results:

Frequently (roughly 10–80% of the time) fails with the described error.

Additional info:

I reported this upstream as https://github.com/abseil/abseil-cpp/issues/1132.

It seems likely that this is a real problem, but I am not sure we can solve it downstream.

It would be easy enough to skip the test on ppc64le, if that’s the way we want to go:

> %ifarch ppc64le
> # CordzInfoStatisticsTest.ThreadSafety flaky failure on ppc64le
> # https://github.com/abseil/abseil-cpp/issues/1132
> # 
> # 144/182 Test #150: absl_cordz_info_statistics_test > ....................Subprocess aborted***Exception:   0.31 sec
> # Running main() from gmock_main.cc
> # [==========] Running 14 tests from 1 test suite.
> # [----------] Global test environment set-up.
> # [----------] 14 tests from CordzInfoStatisticsTest
> # [ RUN      ] CordzInfoStatisticsTest.Flat
> # [       OK ] CordzInfoStatisticsTest.Flat (0 ms)
> #   …
> # [ RUN      ] CordzInfoStatisticsTest.ThreadSafety
> # [mutex.cc : 1682] RAW: Mutex unlocked when destroyed or not locked: v=0x0
> sed -r -i 's/\bThreadSafety\b/DISABLED_&/' \
>     absl/strings/internal/cordz_info_statistics_test.cc
> %endif
Comment 1 Ben Beasley 2022-03-15 18:01:11 UTC 
It looks like this could actually be fixed by disabling LTO on ppc64le. I think that’s an appropriate workaround. (Of course, problems that show up with LTO enabled are usually actually some kind of undefined behavior upstream, but it’s not always easy to find the root cause.)

I am also planning to disable LTO on s390x, since it fixes the (reliable) failure in Symbolize.SymbolizeWithMultipleMaps in absl_symbolize_test on that architecture (https://github.com/abseil/abseil-cpp/issues/1133).

I’m still inclined to leave LTO enabled on other architectures as long as we don’t observe any specific issues on them.
Comment 2 Ben Beasley 2022-03-15 18:12:53 UTC 
I take it back. The problem is still present with LTO. It *might* be rarer, or I might have just gotten (un)lucky.
Comment 3 Ben Beasley 2022-03-15 18:33:55 UTC 
In doing ten or so scratch builds to look at this issue, I observed a couple of other flaky failures. Since these are relatively infrequent, I’m not trying to look into them or report them upstream right now.

-----

On aarch64:

 10/182 Test  #13: absl_sysinfo_test ..................................***Failed    0.06 sec
Running main() from /builddir/build/BUILD/googletest-release-1.11.0/googletest/src/gtest_main.cc
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from SysinfoTest
[ RUN      ] SysinfoTest.NumCPUs
[       OK ] SysinfoTest.NumCPUs (0 ms)
[ RUN      ] SysinfoTest.NominalCPUFrequency
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/base/internal/sysinfo_test.cc:54: Failure
Expected equality of these values:
  NominalCPUFrequency()
    Which is: 3e+09
  1.0
    Which is: 1
CPU frequency detection was fixed! Please update unittest.

-----

On s390x (without LTO):

        Start  81: absl_random_mocking_bit_gen_test
 80/182 Test  #81: absl_random_mocking_bit_gen_test ...................***Failed    0.05 sec
Running main() from /builddir/build/BUILD/googletest-release-1.11.0/googletest/src/gtest_main.cc
[==========] Running 22 tests from 7 test suites.
[----------] Global test environment set-up.
[----------] 9 tests from BasicMocking
[ RUN      ] BasicMocking.AllDistributionsAreOverridable
[       OK ] BasicMocking.AllDistributionsAreOverridable (0 ms)
[ RUN      ] BasicMocking.OnDistribution
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:82:
    Function call: Call(1, 1000000)
          Returns: 20
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:87:
    Function call: Call(0, 100)
          Returns: 5
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:92:
    Function call: Call(1)
          Returns: 42
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:97:
    Function call: Call(1)
          Returns: 500
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:101:
    Function call: Call(1e-06)
          Returns: true
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:105: Failure
Expected: (absl::Zipf<int>(gen, 1000000, 2.0, 1.0)) != (1221), actual: 1221 vs 1221
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:106:
    Function call: Call(1000000, 2, 1)
          Returns: 1221
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:111:
    Function call: Call(0, 1)
          Returns: 0.001
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:116:
    Function call: Call(0, 1000000, 2)
          Returns: 2040
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
[  FAILED  ] BasicMocking.OnDistribution (0 ms)
[ RUN      ] BasicMocking.GMockMatchers
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:125:
    Function call: Call(1000000, 2, 1)
          Returns: 1221
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
[       OK ] BasicMocking.GMockMatchers (0 ms)
[ RUN      ] BasicMocking.OverridesWithMultipleGMockExpectations
[       OK ] BasicMocking.OverridesWithMultipleGMockExpectations (0 ms)
[ RUN      ] BasicMocking.DefaultArgument
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:145:
    Function call: Call(1)
          Returns: 200
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:145:
    Function call: Call(1)
          Returns: 200
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
[       OK ] BasicMocking.DefaultArgument (0 ms)
[ RUN      ] BasicMocking.MultipleGenerators
[       OK ] BasicMocking.MultipleGenerators (0 ms)
[ RUN      ] BasicMocking.MocksNotTrigeredForIncorrectTypes
[       OK ] BasicMocking.MocksNotTrigeredForIncorrectTypes (0 ms)
[ RUN      ] BasicMocking.FailsOnUnsatisfiedMocks
[       OK ] BasicMocking.FailsOnUnsatisfiedMocks (0 ms)
[ RUN      ] BasicMocking.WillByDefaultWithArgs
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:331:
    Function call: Call(1.7)
          Returns: 17
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:331:
    Function call: Call(0.03)
          Returns: 0
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
[       OK ] BasicMocking.WillByDefaultWithArgs (0 ms)
[----------] 9 tests from BasicMocking (0 ms total)
Comment 4 Ben Beasley 2022-03-16 23:58:37 UTC 
For an update on the SysinfoTest.NominalCPUFrequency issue, see discussion in https://src.fedoraproject.org/rpms/abseil-cpp/pull-request/3. That test is now disabled in our package, and will be removed in future upstream releases.
Comment 5 Ben Cotton 2022-08-09 13:13:51 UTC 
This bug appears to have been reported against 'rawhide' during the Fedora Linux 37 development cycle.
Changing version to 37.
Comment 6 Ben Beasley 2023-08-12 14:17:39 UTC 
I can’t reproduce this, even on Fedora 37, with abseil-cpp-20230802.0; see https://github.com/abseil/abseil-cpp/issues/1132#issuecomment-1675180517. I’m going to consider this probably-fixed-upstream in that release.
Comment 7 Fedora Release Engineering 2023-08-16 07:05:20 UTC 
This bug appears to have been reported against 'rawhide' during the Fedora Linux 39 development cycle.
Changing version to 39.
Comment 8 Fedora Update System 2023-09-16 21:02:05 UTC 
FEDORA-2023-b5fb7bba8c has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-b5fb7bba8c
Comment 9 Fedora Update System 2023-09-17 02:03:32 UTC 
FEDORA-2023-b5fb7bba8c has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-b5fb7bba8c`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-b5fb7bba8c

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 10 Fedora Update System 2023-09-18 00:16:11 UTC 
FEDORA-2023-b5fb7bba8c has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.