Description of problem: After the leap second on June 30, 2012, load spikes were noticed in userspace. After some debugging it was noticed that futexes were timing out which was causing CPU loads to increase dramatically. [FWIW: I noticed this myself yesterday night. My firefox suddenly consumed 98.9% of the CPU shortly after the leap second. Resetting the date and restarting firefox resolved the problem. Version-Release number of selected component (if applicable): 2.6.32-279 How reproducible: Unknown at this time. Probably fairly high. Steps to Reproduce: A reproducer is available here, http://marc.info/?l=linux-kernel&m=134113615122011&w=2 Actual results: userspace programs consume ~100% of CPU time because of futex timeouts. Expected results: Futexes should not timeout. Additional info: http://marc.info/?l=linux-kernel&m=134113577921904&w=2 has an RFC patch and reproducer attached.
Working on a backport now. Backport is likely to depend on fix for bug 836748. P.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux release for currently deployed products. This request is not yet committed for inclusion in a release.
The current situation is as follows: A patchset has been posted upstream, http://marc.info/?l=linux-kernel&m=134138316402296&w=2 which has an Acked-by: me. I've tested this patchset on an upstream kernel using the following tests: 1. A leap second test I wrote (but which is VERY similar to) 2. http://marc.info/?l=linux-kernel&m=134116789230177&w=2, and 3. http://marc.info/?l=linux-kernel&m=134116789230177&w=2 with and without the "-s" option. So far all tests have been successful. I am now doing a wider test for RHEL6 with a signficant backport of patches to the kernel/time/timekeeping.c code + the patches from upstream. This testing is ongoing, however, thus far boot testing has not picked up any issues. A smaller patchset has been identified as well that will resolve only the leapsecond issue, but still leaves the clock code susceptible to some other smaller races and issues. I've decided to go with the larger set for the sake of completion -- besides, we're changing core code already and I don't see any reason not to broaden the changes at this point. Watch here for further BZ updates. P.
upstream kernel testing .... [root@intel-canoepass-05 tmp]# uname -a Linux intel-canoepass-05.lab.bos.redhat.com 3.4.4-5.fc17.x86_64 #1 SMP Thu Jul 5 20:20:59 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux [root@intel-canoepass-05 tmp]# ./leap-a-day -s Setting time to Sun Jul 22 20:00:00 2012 Scheduling leap second for Sun Jul 22 20:00:00 2012 Sun Jul 22 19:59:57 2012 + 500283 us TIME_INS Sun Jul 22 19:59:58 2012 + 521 us TIME_INS Sun Jul 22 19:59:58 2012 + 500770 us TIME_INS Sun Jul 22 19:59:59 2012 + 1011 us TIME_INS Sun Jul 22 19:59:59 2012 + 501289 us TIME_INS Sun Jul 22 19:59:59 2012 + 6806 us TIME_OOP Sun Jul 22 19:59:59 2012 + 506946 us TIME_OOP Sun Jul 22 20:00:00 2012 + 7143 us TIME_WAIT Sun Jul 22 20:00:00 2012 + 507274 us TIME_WAIT Sun Jul 22 20:00:01 2012 + 7516 us TIME_WAIT Sun Jul 22 20:00:01 2012 + 507652 us TIME_WAIT Sun Jul 22 20:00:02 2012 + 7898 us TIME_WAIT Note: hrtimer early expiration failure observed. Leap complete ............................................................................. Modified kernel with upstream patches: [root@intel-canoepass-05 tmp]# uname -a Linux intel-canoepass-05.lab.bos.redhat.com 3.5.0-rc6+ #2 SMP Wed Jul 11 14:51:10 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux [root@intel-canoepass-05 tmp]# [root@intel-canoepass-05 tmp]# ./leap-a-day -s This runs continuously. Press ctrl-c to stop Setting time to speed up testing Setting time to Wed Jul 11 20:00:00 2012 Scheduling leap second for Wed Jul 11 20:00:00 2012 Something woke us up, returning to sleep Wed Jul 11 19:59:50 2012 + 746240 us TIME_OK Wed Jul 11 19:59:51 2012 + 246487 us TIME_INS Wed Jul 11 19:59:51 2012 + 746702 us TIME_INS Wed Jul 11 19:59:52 2012 + 246965 us TIME_INS Wed Jul 11 19:59:52 2012 + 747181 us TIME_INS Wed Jul 11 19:59:53 2012 + 247454 us TIME_INS Wed Jul 11 19:59:53 2012 + 747677 us TIME_INS Wed Jul 11 19:59:54 2012 + 247885 us TIME_INS Wed Jul 11 19:59:54 2012 + 748096 us TIME_INS Wed Jul 11 19:59:55 2012 + 248371 us TIME_INS Wed Jul 11 19:59:55 2012 + 748623 us TIME_INS Wed Jul 11 19:59:56 2012 + 248886 us TIME_INS Wed Jul 11 19:59:56 2012 + 749087 us TIME_INS Wed Jul 11 19:59:57 2012 + 249357 us TIME_INS Wed Jul 11 19:59:57 2012 + 749597 us TIME_INS Wed Jul 11 19:59:58 2012 + 249793 us TIME_INS Wed Jul 11 19:59:58 2012 + 750003 us TIME_INS Wed Jul 11 19:59:59 2012 + 250274 us TIME_INS Wed Jul 11 19:59:59 2012 + 750494 us TIME_INS Wed Jul 11 19:59:59 2012 + 250728 us TIME_OOP Wed Jul 11 19:59:59 2012 + 750938 us TIME_OOP Wed Jul 11 20:00:00 2012 + 251203 us TIME_WAIT Wed Jul 11 20:00:00 2012 + 751426 us TIME_WAIT Wed Jul 11 20:00:01 2012 + 251631 us TIME_WAIT Wed Jul 11 20:00:01 2012 + 751845 us TIME_WAIT Wed Jul 11 20:00:02 2012 + 252118 us TIME_WAIT Leap complete
Created attachment 597663 [details] Current upstream test for hrtimer expiration
I've run several systems with a modified upstream kernel and the futex patchset and haven't seen any failures in 18+ hours. P.
Upstream patches are currently in tip. P.
I've put together a set of patches (that depend on BZ 836748) and have started testing across a large set of systems using the test case previously provided in this BZ. I will update the BZ with testing results, and the patches after my initial testing is complete. P.
(In reply to comment #2) > This request was evaluated by Red Hat Product Management for > inclusion in a Red Hat Enterprise Linux release. Product > Management has requested further review of this request by > Red Hat Engineering, for potential inclusion in a Red Hat > Enterprise Linux release for currently deployed products. > This request is not yet committed for inclusion in a release. Is there an estimated time for inclusion of this as an official errata? There are several RHEL customers wanting this errata to come out. Scott McCarty Solutions Architect
Scott, There should not be any urgency surrounding this errata as no other leap seconds are currently scheduled and they are typically announced _years_ in advance. http://en.wikipedia.org/wiki/Leap_second Please inform your customers that Engineering is working on a stable and well-tested solution, and that a fix will be in RHEL6.4. P.
Prarit, I appreciate the response. The leap second insertion is released in Bulletin C on the IERS website, which is typically 5 months before the June or December of a leap second. Our customers all have tickets open which cannot be closed until they have guaranteed that a fix is in place for their systems. These two facts combined create risk for operations folks and their managers are particularly uncomfortable closing the ticket without closing the loop. This is why a definite release for this patch is so critical. I hope that helps clarify. I have been educating our customer's operations teams on the RHEL6.4 release. I have explained to them that it would not be released z-stream because the time system is such a critical piece of the kernel. Best Regards Scott M
Created attachment 603549 [details] RHEL PATCH 1/7
Created attachment 603550 [details] RHEL PATCH 2/7
Created attachment 603551 [details] RHEL PATCH 3/7
Created attachment 603552 [details] RHEL PATCH 4/7
Created attachment 603553 [details] RHEL PATCH 5/7
Created attachment 603554 [details] RHEL PATCH 6/7
Created attachment 603555 [details] RHEL PATCH 7/7
Patch(es) available on kernel-2.6.32-298.el6
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0496.html