Red Hat Bugzilla – Bug 75128
MySQL hangs/crashes after glibc upgrade
Last modified: 2016-11-24 07:42:38 EST
From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.0) Gecko/20020606 Description of problem: When connecting to a MySQL server remotely after the latest glibc upgrade, MySQL hangs and crashes as follows; 021004 18:36:02 mysqld started /usr/local/mysql/libexec/mysqld: ready for connections Number of processes running now: 1 mysqld process hanging, pid 31607 - killed 021004 18:36:30 mysqld restarted This does not affect local connections, but only remote connections. This has also been replicated on RedHat 7.3. This problem is seen with the RedHat MySQL RPM's as well as other versions of MySQL (custom RPM's). It seems to die before it even checks the tables. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1.Install redhat 7.1 / 7.3 2.Install glibc (.40) update 3.Install MySQL 4.Connect to MySQL from a remote machine (or locally to the IP, not to 127.0.0.1) Actual Results: MySQL hung then crashed, resulting in the following in the err log; 021004 18:36:02 mysqld started /usr/local/mysql/libexec/mysqld: ready for connections Number of processes running now: 1 mysqld process hanging, pid 31607 - killed 021004 18:36:30 mysqld restarted /usr/local/mysql/libexec/mysqld: ready for connections Expected Results: MySQL should have allowed / disallowed connections based on the table contents, however it never got as far as to check tables Additional info: This has been replicated on several Intel based servers running a variety of redhat 7.1 and 7.3. The problem only occured after installing glibc. Recompiling MySQL didn't cure the problem.
downgrading glibc to 2.2.5-39 fixed the problem
Also replicated in RH7.2
I get a slight change when I rebuilt mysql under glibc 2.2.5-40 on rhl 7.3. The error changes to error 2013 - but the mysqld does not die. Could it in be related to this: http://www.mysql.com/doc/en/Packet_too_large.html -sv
might be worth checking the mit-pthreads includes in mysql - looks like they define and work with the nameres maxpacket size in there and it might be a test they are doing based on it being 1024 not 65536 which the glibc -40 patches redefine it to be. so that might be why its dying unhappily.
The problem is actually that MySQL uses insanely low RLIMIT_STACK and does gethostbyname from such threads. There are many other places in glibc which use alloca for 32K etc., so it is weird MySQL did not get caught by this far earlier. A patch to deal with low RLIMIT_STACK while not slowing programs down in the normal case is in the works.
This is also reproducable with RH 7.1 on an alpha ev56. The same alpha also gave a haltcode 07 which may be unrelated.
Another extremely kludgey and temporary workaround: adding IP/hostnames to /etc/hosts for any host which will be connecting to MySQL also keeps it from crashing. Of course, there's an easy DoS attack possible by simply attempting to connect to MySQL's port from a host that is not in /etc/hosts, but it does ameliorate the problem slightly until you can recompile.
Another workaround that appears to more or less work so far: put "set-variable = thread_stack=1M" into /etc/my.cnf under [mysqld] if this won't make resources get out of hand on your server(s). Actually 256K instead of 1M seems to work as well. YMMV.
Neither workaround works on my system (RH7.3). I run local DNS off the same box and it appears to override any setting I make in /etc/hosts. The thread_stack option did nothing. I guess I'll have to await an errata release :-(
The thread_stack fixed worked perfect for me. However I had to set it during start up for mysql to get it to work. safe_mysqld --set-variable thread_stack=256k&
*** Bug 75616 has been marked as a duplicate of this bug. ***
I've put fixed rpms (i386/i686/SRPMS for 7.x only) to people.redhat.com/jakub/glibc/errata/ Those packages have not been through QA yet, though in the initial testing seems to cure the problem. If you have any problems with them please let me know ASAP.
I certainly can't find the putative fixed RPMs at /jakub/glibc/errata/, which appears to be empty (permissions?). It would sure be great if this got fixed - not only has this bug hosed the snort setup on my network, compromising my network security, but many other people have MySQL problems which are essentially identical to this bug, cf. bugs 74943, 75290, and 75297.
did you go to ftp://people.redhat.com/jakup/glibc/errata/ I found them there. also - you can always roll the glibc update back out.
*** Bug 75684 has been marked as a duplicate of this bug. ***
ftp://people.redhat.com/jakub/glibc/errata/ works well [b not p]
*** Bug 75507 has been marked as a duplicate of this bug. ***
{aside: display of this bug points out a bug in Bugzilla, not handling wide display stuff well). This fix is desperately needed. Problem with up2date is that it's easy to use to apply "approved" fixes, but when handling a large number of machines it's quite painful to do updates from "pre-release" patches. We've got MySQL daemons crashing all over the place due to this, on RH7.0 and RH7.2 machines. We did find adding the host names in question to /etc/hosts works around the problem, as it avoids the "fixed" versions of the resolver routines.
If someone could provide details of which version WAS in play just before this one (i.e. there've been several updates to glibc, and I really don't know which one was previously installed) that'd be a help. Are the preceeding RPMs kept somewhere on systems? Seems like a rollback mechanism is proving to be necessary. Much as I dislike working with Microsoft stuff, I note that in XP they've added some facilities to roll back driver updates and such in the event a new version causes things to break. Guess it's time for the Linux folks to consider the same. Until the fix for the present version of glibc has cleared QA, I think I'd like to go ahead and roll back. Any pointers to doing this for RH7.0 and RH7.2 would be greatly appreciated.
Wide format of this page is caused by long CC: field. Too many people would like to know final solution. Backing up old packages is provided by up2date in RH 8.0 (AFAIK). As there is update for up2date for 7.x releases, this could be posiible to have this improvements handy now. But I did not find appropriate switches for command line inside man page and thus don't know if I'm right.
glibc -42 WORKSFORME on 7.3. Now, RH really needs to push these through testing and get them out for 7.x and 8.0 ASAP.
Another "ME TOO" style response. (RHL 7.3 smp i686 box). glibc 2.2.5-40 is fooed, but .39 and Jakub's .42 are fine. I'll stay with .42. Many thanks.
*** Bug 76111 has been marked as a duplicate of this bug. ***
*** Bug 76242 has been marked as a duplicate of this bug. ***
*** Bug 75379 has been marked as a duplicate of this bug. ***
*** Bug 76082 has been marked as a duplicate of this bug. ***
*** Bug 76565 has been marked as a duplicate of this bug. ***
WORKSFORME too. I had to restart whole server to be able to login via ssh and sendmail stopped receiving mails too. All programs which using glibc should be stopped and started again to use new liraries.
Note: This same Bug affects Red Hat Linux 6.2 as well. There are currently no 6.2 glibc packages available. The suggested set-variable=thread_stack=256k does not work around it. I will try setting it during startup next. If packages for 6.2 can be respun as well, that would be most helpfull. Note that rsync is also affected by this issue: Works: rsync -r --compress nmg@eva.host.nl::wtvinvoer/cmp /var/directory Fails: rsync --compress nmg@eva.host.nl::wtvinvoer/cmp/* /var/directory/cmp/ With a Connection reset by peer message. This was something that also was affected by the glibc upgraded. This was between 2 7.3 machines.
Been 8 days since the last update to this bug (and that was only to add more people to the massive CC list). Two weeks since it was reported test versions worked properly. I guess we were all hoping RedHat would place some priority on getting a fix to a busted errata into RHN in short order, but that appears to not be the case. We are using /etc/hosts entries to work around the bad code (assumes your nsswitch.com is checked before DNS). With a large number of systems now running the bad version of glibc, it would take many hours to apply new RPMs by hand. Since it appeared weeks ago that fixed versions were entering QA, we held off, assuming RHN would have updates shortly. It would be helpful if this bug were updated by the folks working on fixing and/or testing the fix.
Hear hear. It seems only security issues get immediate erratas. Most other bugs go un-errata'd. Even, now it seems, super popular bugs like this one that is probably affecting thousands of sites! The rpm and rhn system are wonderful, but do have their drawbacks... namely time delays.
Jakub's .42 version has worked great on our test systems. Unfortunately, we're prohibited by company security policies from installing test fixes on production machines, unless there's an immediate visible danger to leaving off the test fix. If RedHat can get this through QA testing and approved it would be a tremendous help to us. I wouldn't suggest rushing it, but an update here and a little more expeditious effort on completing their QA testing would be a very good thing. Since the original errata that caused this was a security update, I hope we can get this issue moving with the same sense of urgency.
An errata has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2002-197.html
*** Bug 76631 has been marked as a duplicate of this bug. ***
Hello, I am having this issue with redhat 8.0 and MySql. I can telnet localhost 3306 or mysql with no issue. I get immediately disconnect if I telnet valid ip address 3306 or mysql with -h Valid Ip address. I have run up2date and am current. Is this the same issue? Thanks, Tom Morgan
I have not seen an update or comment on whether this is being addressed for red hat 8.0. What is the status? If I added the remote clients to /etc/hosts on the MySql box they were able to connect. Thanks, Tom Morgan
Upgrading from 7.3 to 8.0 brings the bug back. Any chance of updates for glibc-2.2.93-5, since 8.0 contains many packages which won't work with glibc-2.2.5-42. Please re-OPEN
I opened bug 78990 to track the issue for RedHat 8.0. That way we at least have an OPEN bug to refer to. RedHat folks: feel free to re-open 75128 and close 78990 if you agree to fix the issue.
This issue is being tracked as bug 77467 for 8.0.
This bug reappeared after the upgrading mysql from version 3.23.54a (rpm provided by mysq.com) to version 4.0.8-gamma (binary provided by mysql.com as well). Using redhat 7.1 glibc 2.2.4-31. Mysql 4.0.8 stopped crashing when I added --skip-name-resolve in the mysql startup... This server has glibc 2.2.4-31 installed for quite a while, and mysql 3.23.54a was working just fine with it.
It was actually a bug in mysql 4.0.8. Upgrading to version 4.0.9 fix the problem
using redhat 8.0 with latest redhat supplied version of mysql and all updates as of 01-30-2003, this problem still exists, but 2 workarounds exist: add: set-variable = thread_stack=256K to /etc/my.cnf [mysql] section - make sure sure /etc/rc.d/init.d/mysqld contains: "/usr/bin/safe_mysqld --defaults-file=/etc/my.cnf" or add ips and hostnames of connecting hosts to /etc/hosts.
Try this at mysql prompt: mysql> USE mysql; mysql> GRANT ALL ON *.* TO root@"%" IDENTIFIED BY 'yourpassword'; mysql> FLUSH PRIVILEGES; This worked for me. Edinelson Keiji Shimokawa.
*** Bug 75297 has been marked as a duplicate of this bug. ***