Bug 444321 - yum update hangs -- FUTEX_WAIT
yum update hangs -- FUTEX_WAIT
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: rpm (Show other bugs)
9
All Linux
low Severity low
: ---
: ---
Assigned To: Panu Matilainen
Fedora Extras Quality Assurance
:
: 450705 485780 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-04-26 23:54 EDT by Chris Ricker
Modified: 2014-01-21 18:02 EST (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-07-14 12:12:03 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
rpmdb_stat -CA (3.69 KB, application/octet-stream)
2008-06-09 05:58 EDT, Timon
no flags Details

  None (edit)
Description Chris Ricker 2008-04-26 23:54:17 EDT
yum-3.2.14-10.fc9.noarch
kernel-2.6.25-8.fc9.x86_64

On a current rawhide x86_64 box, yum update hangs:

[root@scratchmonkey ~]# yum clean all
Loaded plugins: refresh-packagekit
Cleaning up Everything
[root@scratchmonkey ~]# yum update
Loaded plugins: refresh-packagekit
livna-development                                        | 2.1 kB     00:00    
primary.sqlite.bz2                                       | 118 kB     00:00     
fedora                                                   | 2.4 kB     00:00    
primary.sqlite.bz2                                       | 7.6 MB     00:01     
pptp-stable                                              | 1.9 kB     00:00    
primary.sqlite.bz2                                       | 5.2 kB     00:00    
adobe-linux-i386                                         |  951 B     00:00    
primary.xml.gz                                           |  10 kB     00:00    
adobe-linu: ################################################## 17/17
updates                                                  | 2.4 kB     00:00    
primary.sqlite.bz2                                       | 7.6 MB     00:02     
Setting up Update Process

strace shows it sitting forever on

futex(0x7fc2fa534078, FUTEX_WAIT, 2, NULL
Comment 1 Bug Zapper 2008-05-14 06:14:44 EDT
Changing version to '9' as part of upcoming Fedora 9 GA.
More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 2 Timon 2008-05-29 14:19:18 EDT
it's very old bug, I watch them on fc5, fc6, fc7, f8 and now on f9.
check bug 145021 which still present.
what information I need to catch to help to resolve this problem?

[timon@localhost ~]$ ps aux | grep yum
root      7303  0.0  4.0  61436 20828 ?        S    May29   0:01 /usr/bin/python
/usr/share/PackageKit/helpers/yum/get-update-detail.py cronie;1.0-6.fc9;i386;updates
timon     8157  0.0  0.1   5068   760 pts/2    S+   00:17   0:00 grep yum
[timon@localhost ~]$ sudo strace -p 7303
Process 7303 attached - interrupt to quit
futex(0xb7d9e0d0, FUTEX_WAIT, 2, NULL

Comment 3 Jeff Johnson 2008-05-29 22:27:49 EDT
There are two causes of FUTEX_WAIT that need to be separated if you wish a solution.

The 2 causes are

1) stale lock in Berkeley DB environment left registered when process exited
without unregistering the lock in the dbenv. A registered but abandoned lock
can be displayed/verified by running (see Berkely db_stat documentation):
    cd /var/lib/rpm && /use/lib/rpm/rpmdb_stat -CA
and checking whether the processes/threads that own the lock(s) are still active.
FWIW, the check (and the cleanup) are largely automated for about 1.5 years now.

2) A stale FUTEX caused by process/thread termination without releasing the FUTEX.
This is a much rarer condition, and a more complicated diagnosis. There is now a
mechansim to handle reaping stale FUTEX's called"robust mutexes" that can clear
a stale FUTEX. The problem is that clearing a stale FUTEX is not sufficient, one
also has to deal with whatever state the lock was protecting in order to clear.

Luckily (for rpm) there is little important state that needs to be cleared in a rpmdb,
so reaping a stale FUTEX prevents the futex syscall "hang".

(aside) Note that there are 2nd order (if you will) consequences from cleanup of a
failed rpm install transaction that can/will lead to, e.g., duplicate package
entries in a rpmdb, but that's a different, higeher level, issue. FYI a patch for
rpm + Berkeley DB to use "robust mutexes" to reap stale FUTEX's has existed for
about 1.5 years now as well, but is rather tricky to deploy reliably since a functional
"robust mitex" environment has both kernel & glibc version dependencies, and the
patch is specific to rpm, because of the manner that rpm access a Berkeley DB.

So which case are you: 1) registered-but-abandoned dbenv lock 2) stale FUTEX?

I've seen lots of 1) cases, but have only seen/heard of maybe a handful of stale FUTEX's
personally, they appear to be quite rare. But YMMV, everyone's does.
Comment 4 Timon 2008-06-09 05:58:11 EDT
Created attachment 308673 [details]
rpmdb_stat -CA
Comment 5 Timon 2008-06-09 05:59:26 EDT
i catch it again. so, i get rpmdb_stat, but can't understand which numbers i
need to check. see attach: #4
Comment 6 Jeff Johnson 2008-06-09 17:16:02 EDT
For context (note that you are using an older version than 4.7.25 of Berkeley DB):

The db_stat output is described here:
    http://www.oracle.com/technology/documentation/berkeley-db/db/utility/db_stat.html

The locking subsystem is here:
    http://www.oracle.com/technology/documentation/berkeley-db/db/ref/lock/intro.html

The code (or similar) that is displaying the info is available here, see __lock_printlock():
    http://src.opensolaris.org/source/xref/webstack/webstack/trunk/src/db-4.5.20/lock/lock_stat.c

This looks like a stale futex to me because
    1) There are only READ locks. In normal Concurrent Data Store operation, there's
          either a single WRITE'r or ultiple READers. So this is not a "normal" operation.
    2) All the nerdy hex information being displayed is identical, which is what I would expect
          when a common (and stale) kernel futex is the cause of a "hang".

Disclaimer: I have _NOT_verified whether your output is actually a stale futex (or not), just made
a reasoned guess given the supplied db_stat spew. There may well be other interpretations better
than mine.

Note that there is a patch to use "robust mutexes" that would clear stale futexes in rpm
available somewhere in this thread:
    https://lists.dulug.duke.edu/pipermail/rpm-devel/2006-November/001853.html

I continue to "sabotage" Fedora bugs ;-)

Comment 7 Pascal Parois 2008-11-01 15:16:10 EDT
I have the same problem with yum (3.2.19-3) but only on the 2 last kernel (2.6.26.6-79.fc9.i386 and 2.6.26.5-45). It's working with an older kernel like 2.6.26.6-55

[...]
pread64(10, "\0\0\0\0\1\0\0\0\336:\0\0\337:\0\0\335:\0\0\1\0\346\17\0\7b7c91a"..., 4096, 61726720) = 4096
pread64(10, "\0\0\0\0\1\0\0\0\335:\0\0\336:\0\0\334:\0\0\1\0\346\17\0\7c8a29e"..., 4096, 61722624) = 4096
pread64(10, "\0\0\0\0\1\0\0\0\334:\0\0\335:\0\0\333:\0\0\1\0\346\17\0\7\000799f6"..., 4096, 61718528) = 4096
pread64(10, "\0\0\0\0\1\0\0\0\333:\0\0\334:\0\0\332:\0\0\1\0\346\17\0\0076b0e7e"..., 4096, 61714432) = 4096
futex(0xb7a7b850, FUTEX_WAIT, 2, NULL

I have a similar problem on an other computer (fedora 9 as weel but x86_64)
I only have trouble on the installation of the kernel, not for any other packages (so far):

rt_sigprocmask(SIG_SETMASK, NULL, [CHLD], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
futex(0x21614e8, FUTEX_WAIT_PRIVATE, 2, NULL
^C <unfinished ...>
[root@localhost pascal]# erreur: %post(kernel-2.6.26.6-79.fc9.x86_64) scriptlet failed, signal 2

Although this time it's not completely stuck, I can install the kernel via rpm if I have patience, it takes about 10min.
Comment 8 Panu Matilainen 2009-01-31 04:36:42 EST
*** Bug 450705 has been marked as a duplicate of this bug. ***
Comment 9 Panu Matilainen 2009-03-18 09:47:49 EDT
*** Bug 485780 has been marked as a duplicate of this bug. ***
Comment 10 Bug Zapper 2009-06-09 20:28:16 EDT
This message is a reminder that Fedora 9 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 9.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '9'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 9's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 9 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 11 Bug Zapper 2009-07-14 12:12:03 EDT
Fedora 9 changed to end-of-life (EOL) status on 2009-07-10. Fedora 9 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.