Red Hat Bugzilla – Bug 113968
glibc 2.3.2, /lib64/tls library bug in pthread_rwlock_init
Last modified: 2007-11-30 17:07:00 EST
From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20020903 Description of problem: We observed a problem when running MySQL 4.0.17 application on RH AS 3.0, insert statements locked up on a very small table. Platform CPU info: vendor_id : AuthenticAMD model name : AMD Opteron(tm) Processor 244 cpu MHz : 1792.138 cache size : 1024 KB kernel info: 2.4.21-4.0.1.EL #1 Thu Oct 23 01:24:43 EDT 2003 x86_64 GNU/Linux During debugging we found out that the problem occurred with MySQL my_rwlock_init() code, which was used from /lib64/tls/ library, as the binary was dynamic. It seemed that sometimes the argument had uninitialized values when it was passed to the locking function. This caused the lock to be ignored and further code was executed, which was wrong. We found out that just by moving /lib64/tls to /lib64/tls.disabled fixed the problem. MySQL server was then using the corresponding shared libraries from elsewhere on the system. Version-Release number of selected component (if applicable): glibc-2.3.2-95.6 How reproducible: Always Steps to Reproduce: 1. Install MySQL 4.0.17 (or higher), must be a 64bit version, made for Linux AMD 64bit. 2. Create a table with a key field in it, not primary key. CREATE TABLE t1 (i int, key (i)); 3. Run simultaneous inserts to the table, into the above key field INSERT INTO t1 VALUES (1); INSERT INTO t1 VALUES (2); etc. Inserts must be done via separate threads. Actual Results: If two or more inserts have been run close enough to each other, one can notice the problem with 'mysqladmin processlist', which says that the first thread is in state 'update' while the others are locked. The threads will stay at this state for ever. Expected Results: Insert statements should have gone through. Additional info: The exact place in the mysql code where the problem occurs is at myisam/mi_open.c, around line 434, which looks like this: for (i=0; i<keys; i++) VOID(my_rwlock_init(&share->key_root_lock[i], NULL)); my_rwlock_init is defined as #define my_rwlock_init(A,B) pthread_rwlock_init((A),(B)), which is a system function call, which is defined in and used from /lib64/tls/libpthread-0.60.so We suspect that the bug is somewhere in this library. Just by disabling this library things will work. Regards, Jani
This issue has also been seen with MySQL 3 rpm packages as released by Red Hat within the Extra's channel for AS 3 AMD64. A resolution on this would be real nice ! :)
This has nothing to do with Red Hat Application Server. I'm sending it over to RHEL for further investigation.
Tom... please take a look at this. It may very well be a glibc bug, but try to reproduce first.
I was not able to reproduce this on an AMD64 4-way box running RHEL3, using MySQL 4.0.17 built from source. I made a file containing 10000 insert commands and fed it into two mysql processes simultaneously in two shell windows. The performance was pretty abysmal (a single mysql could run the file in about a second, but two parallel instances would take 13 to 15 seconds to finish), but I observed no lockup in several dozen trials. Can anyone suggest what I must do differently to see the failure? Note that if the problem occurs only with MySQL-supplied binaries, and not in a source build, I'd be inclined to blame the variant version of glibc that they use in their binary packages.
I've continued to try to reproduce this, without success. I've tried MySQL 4.0.17 built from MySQL AB's SRPM, and I've tried the mysql-server-3.23.58-1 RPM from the RHN Extras channel as suggested by rackspace. No luck. I am using a more recent kernel: 2.4.21-9.ELsmp #1 SMP Thu Jan 8 16:52:31 EST 2004 x86_64 x86_64 x86_64 GNU/Linux but other than that I think I am testing the same software and hardware as the complainant. The only other possibility that comes to mind is that my testing methodology might be wrong. As mentioned above, I'm feeding script files containing many INSERT commands to mysql clients running in parallel. I tried 2,3,4,5 clients (this is a 4-way machine btw). Doesn't lock up. Unless someone can tell me how to reproduce this, I'm going to have to close it WORKSFORME.
I will try to get a backtrace of where MySQL hangs. We were testing this on RedHat machine with MySQL compiled there, so it's definitely a RedHat problem. However, I understand that you need more information about the problem before you can check the code or debug it. I will discuss with our contact if I can login to the machine and get more information about the bug. I'd appreciate if you can hold this report in status 'NEEDINFO' for now, I will get back to you asap. Regards, Jani
I'm going to set this in state NEEDINFO until I hear more. Jani, if you file followup info please remember to change the state back to ASSIGNED.
Rackspace sent me login info and reproduction instructions for their own x86_64 machine running Taroon Update 1. I have confirmed that following their recipe it is possible to get mysql to freeze up with a stack trace pointing to pthread_rwlock_wrlock. Obviously I can't put the login details in this bugzilla entry, but will send them by private mail to anyone who needs to look into this. It does seem to be a pthread issue.
I am relabeling this as a glibc issue. I'm not sure who bugzilla will reassign it to (maybe Jakub?) but whoever wants to look at it please contact me by mail for info about reproducing the problem.
This looks very much like: http://sources.redhat.com/ml/libc-hacker/2004-02/msg00019.html which is fixed in glibc-2.3.2-95.12 and later (U2 beta includes glibc-2.3.2-95.17).
Ah, very interesting. [looks...] The Rackspace machine that shows the problem is running glibc-2.3.2-95.6. I will ask them to update and see if problem goes away. Thanks.
ftp://people.redhat.com/jakub/glibc/2.3.2-95.17/
Ping!? I want to close this bug if there is nothing more to report.
I've not heard any more complaints from Rackspace, so let's assume the issue is gone. They can reopen if not...