Description of problem: The upstream Ceph project developers have found that Ceph crashes semi-consistently with the recent update of gperftools-libs from 2.7 to 2.8 in EPEL 8. gperftools-libs-2.7-6.el8 has been stable in EPEL 8 for a long time. When we updated to 2.8 in https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-dd6932436d , we started seeing crashes in gperftools in Ceph's testing automated framework. Version-Release number of selected component (if applicable): gperftools-2.8-1.el8 How reproducible: fairly reproducible given enough stress-testing over time Steps to Reproduce: See the bug reports at https://tracker.ceph.com/issues/49240 and https://tracker.ceph.com/issues/49387 Actual results: Ceph crashes in gperftools Expected results: Ceph does not crash in gperftools Additional info: I'm planning to reset back to gperftools-libs-2.7-6.el8 and bump the Epoch and Release in order to resolve this. Note Fedora 33 or 34 is likely to be the default branching point for EPEL 9. Fedora 34 has gperftools 2.9 (bz 1931259).
In bug 1933756, Casey mentioned this is resolved in 2.8.1, so we could update EPEL 8 to that version instead.
After discussion in #ceph-devel, I'm going to push to 2.7 with an epoch bump to epel8. Given that no other distros besides Fedora have updated to 2.8 and some other upstream maintenance factors, we need wider testing at this point.
I would prefer that we not go backwards here. If this is fixed in 2.8.1, that would be an appropriate update.
debian sid/bullseye currently has 2.8.1 OpenSUSE tumbleweed also has at 2.8.1 at present
debian sid/bullseye currently has 2.8.1, ubuntu hirsute (21/04) currently has 2.8.1, OpenSUSE tumbleweed also has at 2.8.1 at present.
I'm concerned with going to 2.8.1 since there may be further bugs that haven't appeared in ceph's stress testing yet. In the past we've seen several issues with updates to tcmalloc in ceph. All the distros using 2.8.1 are bleeding edge/unreleased, so it has little real-world exposure. What benefit to updating to 2.8.1 is there? It seems high risk, for little reward to me.
What's odd is that 2.8 is in F33, but AFAICS no such bugs have been raised against it. If we revert to 2.7, then the P&Z fixes will need to be backported, and ceph.spec updated yet again to reflect that.
Yes, we would need to backport P&Z fixes to 2.7-6 to avoid regressing on those arches. Several engineers from multiple teams did testing on gperftools 2.8. They reported their results in Bodhi at https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-dd6932436d . The problem is that this testing was not heavy enough stress testing to shake out this particular issue, and in the absence of further information, we're now stuck wondering about unknown-unknowns. Next time we have to update gperftools in EPEL, we'll coordinate longer testing with Teuthology in the main upstream ceph lab.
Filed: https://src.fedoraproject.org/rpms/gperftools/pull-request/3
FEDORA-EPEL-2021-0eda4297eb has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-0eda4297eb
FEDORA-EPEL-2021-0eda4297eb has been pushed to the Fedora EPEL 8 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-0eda4297eb See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2021-0eda4297eb has been pushed to the Fedora EPEL 8 stable repository. If problem still persists, please make note of it in this bug report.