Bug 75128 - MySQL hangs/crashes after glibc upgrade
Summary: MySQL hangs/crashes after glibc upgrade
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Linux
Classification: Retired
Component: glibc
Version: 7.1
Hardware: i386
OS: Linux
high
high
Target Milestone: ---
Assignee: Jakub Jelinek
QA Contact: Brian Brock
URL:
Whiteboard:
: 75297 75379 75507 75616 75684 76082 76111 76242 76565 76631 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2002-10-04 18:10 UTC by Mike Bartling
Modified: 2016-11-24 12:42 UTC (History)
49 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2002-11-07 00:57:56 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2002:197 0 high SHIPPED_LIVE : Updated glibc packages fix vulnerabilities in resolver 2002-09-10 04:00:00 UTC

Description Mike Bartling 2002-10-04 18:10:45 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.0) Gecko/20020606

Description of problem:
When connecting to a MySQL server remotely after the latest glibc upgrade, MySQL
hangs and crashes as follows;

021004 18:36:02  mysqld started
/usr/local/mysql/libexec/mysqld: ready for connections

Number of processes running now: 1
mysqld process hanging, pid 31607 - killed
021004 18:36:30  mysqld restarted

This does not affect local connections, but only remote connections.
This has also been replicated on RedHat 7.3. This problem is seen with the
RedHat MySQL RPM's as well as other versions of MySQL (custom RPM's). It seems
to die before it even checks the tables.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1.Install redhat 7.1 / 7.3
2.Install glibc (.40) update
3.Install MySQL
4.Connect to MySQL from a remote machine (or locally to the IP, not to 127.0.0.1)
	

Actual Results:  MySQL hung then crashed, resulting in the following in the err log;

021004 18:36:02  mysqld started
/usr/local/mysql/libexec/mysqld: ready for connections

Number of processes running now: 1
mysqld process hanging, pid 31607 - killed
021004 18:36:30  mysqld restarted
/usr/local/mysql/libexec/mysqld: ready for connections



Expected Results:  MySQL should have allowed / disallowed connections based on
the table contents, however it never got as far as to check tables

Additional info:

This has been replicated on several Intel based servers running a variety of
redhat 7.1 and 7.3. The problem only occured after installing glibc. Recompiling
MySQL didn't cure the problem.

Comment 1 Mike Bartling 2002-10-04 18:17:44 UTC
downgrading glibc to 2.2.5-39 fixed the problem

Comment 2 Mike Bird 2002-10-04 19:12:54 UTC
Also replicated in RH7.2

Comment 3 Seth Vidal 2002-10-05 20:27:44 UTC
I get a slight change when I rebuilt mysql under glibc 2.2.5-40 on rhl 7.3. The
error changes to error 2013 - but the mysqld does not die.

Could it in be related to this:

http://www.mysql.com/doc/en/Packet_too_large.html

-sv


Comment 4 Seth Vidal 2002-10-05 20:57:57 UTC
might be worth checking the mit-pthreads includes in mysql - looks like they
define and work with the nameres maxpacket size in there and it might be a test
they are doing based on it being 1024 not 65536 which the glibc -40 patches
redefine it to be. so that might be why its dying unhappily.




Comment 5 Jakub Jelinek 2002-10-05 22:38:49 UTC
The problem is actually that MySQL uses insanely low RLIMIT_STACK and does
gethostbyname from such threads. There are many other places in glibc which
use alloca for 32K etc., so it is weird MySQL did not get caught by this
far earlier. A patch to deal with low RLIMIT_STACK while not slowing programs
down in the normal case is in the works.

Comment 6 Jacco Ligthart 2002-10-08 16:06:56 UTC
This is also reproducable with RH 7.1 on an alpha ev56.
The same alpha also gave a haltcode 07 which may be unrelated.

Comment 7 Need Real Name 2002-10-09 00:25:36 UTC
Another extremely kludgey and temporary workaround: adding IP/hostnames to
/etc/hosts for any host which will be connecting to MySQL also keeps it from
crashing. Of course, there's an easy DoS attack possible by simply attempting to
connect to MySQL's port from a host that is not in /etc/hosts, but it does
ameliorate the problem slightly until you can recompile.

Comment 8 Need Real Name 2002-10-09 01:19:39 UTC
Another workaround that appears to more or less work so far: put "set-variable =
thread_stack=1M" into /etc/my.cnf under [mysqld] if this won't make resources
get out of hand on your server(s). Actually 256K instead of 1M seems to work as
well. YMMV.

Comment 9 Trevor Cordes 2002-10-09 10:41:35 UTC
Neither workaround works on my system (RH7.3).  I run local DNS off the same box
and it appears to override any setting I make in /etc/hosts.  The thread_stack
option did nothing.

I guess I'll have to await an errata release :-(

Comment 10 Need Real Name 2002-10-09 15:11:08 UTC
The thread_stack fixed worked perfect for me.  However I had to set it during start up for mysql to 
get it to work.  

safe_mysqld --set-variable thread_stack=256k&

Comment 11 Jakub Jelinek 2002-10-10 20:24:45 UTC
*** Bug 75616 has been marked as a duplicate of this bug. ***

Comment 12 Jakub Jelinek 2002-10-10 20:26:43 UTC
I've put fixed rpms (i386/i686/SRPMS for 7.x only) to people.redhat.com/jakub/glibc/errata/
Those packages have not been through QA yet, though in the initial testing
seems to cure the problem.
If you have any problems with them please let me know ASAP.

Comment 13 Joe Christy 2002-10-11 00:01:52 UTC
I certainly can't find the putative fixed RPMs at /jakub/glibc/errata/, which
appears to be empty (permissions?).

It would sure be great if this got fixed - not only has this bug hosed the snort
setup on my network, compromising my network security, but many other people
have MySQL problems which are essentially identical to this bug, cf. bugs 74943,
75290, and 75297.

Comment 14 Seth Vidal 2002-10-11 00:12:20 UTC
did you go to ftp://people.redhat.com/jakup/glibc/errata/


I found them there.

also - you can always roll the glibc update back out.



Comment 15 Jakub Jelinek 2002-10-11 06:56:52 UTC
*** Bug 75684 has been marked as a duplicate of this bug. ***

Comment 16 Milan Kerslager 2002-10-11 07:23:05 UTC
ftp://people.redhat.com/jakub/glibc/errata/ works well [b not p]

Comment 17 Jakub Jelinek 2002-10-11 08:02:36 UTC
*** Bug 75507 has been marked as a duplicate of this bug. ***

Comment 18 Daniel Senie 2002-10-15 02:11:54 UTC
{aside: display of this bug points out a bug in Bugzilla, not handling wide 
display stuff well).

This fix is desperately needed. Problem with up2date is that it's easy to use 
to apply "approved" fixes, but when handling a large number of machines it's 
quite painful to do updates from "pre-release" patches.

We've got MySQL daemons crashing all over the place due to this, on RH7.0 and 
RH7.2 machines.

We did find adding the host names in question to /etc/hosts works around the 
problem, as it avoids the "fixed" versions of the resolver routines.

Comment 19 Daniel Senie 2002-10-15 02:17:07 UTC
If someone could provide details of which version WAS in play just before this 
one (i.e. there've been several updates to glibc, and I really don't know 
which one was previously installed) that'd be a help.

Are the preceeding RPMs kept somewhere on systems? Seems like a rollback 
mechanism is proving to be necessary. Much as I dislike working with Microsoft 
stuff, I note that in XP they've added some facilities to roll back driver 
updates and such in the event a new version causes things to break. Guess it's 
time for the Linux folks to consider the same.

Until the fix for the present version of glibc has cleared QA, I think I'd 
like to go ahead and roll back. Any pointers to doing this for RH7.0 and RH7.2 
would be greatly appreciated.



Comment 20 Milan Kerslager 2002-10-15 07:38:09 UTC
Wide format of this page is caused by long CC: field. Too many people would like
to know final solution.

Backing up old packages is provided by up2date in RH 8.0 (AFAIK). As there is
update for up2date for 7.x releases, this could be posiible to have this
improvements handy now. But I did not find appropriate switches for command line
inside man page and thus don't know if I'm right.

Comment 21 Tom Wood 2002-10-16 02:38:02 UTC
glibc -42 WORKSFORME on 7.3.  Now, RH really needs to push these through testing
and get them out for 7.x and 8.0 ASAP.

Comment 22 Nic Doye 2002-10-16 17:10:39 UTC
Another "ME TOO" style response. (RHL 7.3 smp i686 box).

glibc 2.2.5-40 is fooed, but .39 and Jakub's .42 are fine. I'll stay with .42.

Many thanks.



Comment 23 Jakub Jelinek 2002-10-18 20:20:23 UTC
*** Bug 76111 has been marked as a duplicate of this bug. ***

Comment 24 Jakub Jelinek 2002-10-18 20:21:34 UTC
*** Bug 76242 has been marked as a duplicate of this bug. ***

Comment 25 Jakub Jelinek 2002-10-18 20:45:42 UTC
*** Bug 75379 has been marked as a duplicate of this bug. ***

Comment 26 Jakub Jelinek 2002-10-18 20:47:12 UTC
*** Bug 76082 has been marked as a duplicate of this bug. ***

Comment 27 Jakub Jelinek 2002-10-23 14:42:46 UTC
*** Bug 76565 has been marked as a duplicate of this bug. ***

Comment 28 Milan Kerslager 2002-10-23 16:18:44 UTC
WORKSFORME too.

I had to restart whole server to be able to login via ssh and sendmail stopped
receiving mails too. All programs which using glibc should be stopped and
started again to use new liraries.

Comment 29 Need Real Name 2002-10-23 21:01:49 UTC
Note: This same Bug affects Red Hat Linux 6.2 as well.

There are currently no 6.2 glibc packages available.
The suggested set-variable=thread_stack=256k does not work around it. I will try
setting it during startup next.

If packages for 6.2 can be respun as well, that would be most helpfull.

Note that rsync is also affected by this issue:
Works: rsync -r --compress nmg.nl::wtvinvoer/cmp /var/directory

Fails: rsync --compress nmg.nl::wtvinvoer/cmp/* /var/directory/cmp/
With a Connection reset by peer message. This was something that also was
affected by the glibc upgraded. This was between 2 7.3 machines.


Comment 30 Daniel Senie 2002-11-01 04:31:53 UTC
Been 8 days since the last update to this bug (and that was only to add more 
people to the massive CC list). Two weeks since it was reported test versions 
worked properly. I guess we were all hoping RedHat would place some priority 
on getting a fix to a busted errata into RHN in short order, but that appears 
to not be the case.

We are using /etc/hosts entries to work around the bad code (assumes your 
nsswitch.com is checked before DNS). With a large number of systems now 
running the bad version of glibc, it would take many hours to apply new RPMs 
by hand. Since it appeared weeks ago that fixed versions were entering QA, we 
held off, assuming RHN would have updates shortly.

It would be helpful if this bug were updated by the folks working on fixing 
and/or testing the fix.

Comment 31 Trevor Cordes 2002-11-01 05:53:02 UTC
Hear hear.  It seems only security issues get immediate erratas.  Most other 
bugs go un-errata'd.  Even, now it seems, super popular bugs like this one 
that is probably affecting thousands of sites!

The rpm and rhn system are wonderful, but do have their drawbacks... namely 
time delays.


Comment 32 Need Real Name 2002-11-05 20:38:30 UTC
Jakub's .42 version has worked great on our test systems. Unfortunately, we're
prohibited by company security policies from installing test fixes on production
machines, unless there's an immediate visible danger to leaving off the test fix.

If RedHat can get this through QA testing and approved it would be a tremendous
help to us. I wouldn't suggest rushing it, but an update here and a little more
expeditious effort on completing their QA testing would be a very good thing.

Since the original errata that caused this was a security update, I hope we can
get this issue moving with the same sense of urgency.

Comment 33 Bill Nottingham 2002-11-07 00:57:57 UTC
An errata has been issued which should help the problem described in this bug report. 
This report is therefore being closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, please follow the link below. You may reopen 
this bug report if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2002-197.html


Comment 34 Neil Sedley 2002-11-07 08:14:57 UTC
*** Bug 76631 has been marked as a duplicate of this bug. ***

Comment 35 Tom Morgan 2002-11-15 16:13:15 UTC
Hello,
I am having this issue with redhat 8.0 and MySql.
I can telnet localhost 3306 or mysql with no issue.
I get immediately disconnect if I telnet valid ip address 3306 or mysql with -h 
Valid Ip address.
I have run up2date and am current.

Is this the same issue?

Thanks,
Tom Morgan

Comment 36 Tom Morgan 2002-11-19 14:26:20 UTC
I have not seen an update or comment on whether this is being addressed for red 
hat 8.0.  What is the status?
If I added the remote clients to /etc/hosts on the MySql box they were able to 
connect.

Thanks,
Tom Morgan

Comment 37 Mike Bird 2002-12-04 13:25:47 UTC
Upgrading from 7.3 to 8.0 brings the bug back.  Any chance of updates for
glibc-2.2.93-5, since 8.0 contains many packages which won't work with
glibc-2.2.5-42.

Please re-OPEN


Comment 38 Daniel Senie 2002-12-04 13:52:45 UTC
I opened bug 78990 to track the issue for RedHat 8.0. That way we at least 
have an OPEN bug to refer to. RedHat folks: feel free to re-open 75128 and 
close 78990 if you agree to fix the issue.

Comment 39 Joe Orton 2002-12-12 12:19:33 UTC
This issue is being tracked as bug 77467 for 8.0.

Comment 40 Gustavo 2003-01-11 19:26:00 UTC
This bug reappeared after the upgrading mysql from version 3.23.54a (rpm
provided by mysq.com) to version 4.0.8-gamma (binary provided by mysql.com as well).

Using redhat 7.1 glibc 2.2.4-31.

Mysql 4.0.8 stopped crashing when I added --skip-name-resolve in the mysql
startup... 

This server has glibc 2.2.4-31 installed for quite a while, and mysql 3.23.54a
was working just fine with it.


Comment 41 Gustavo 2003-01-23 17:42:38 UTC
It was actually a bug in mysql 4.0.8.
Upgrading to version 4.0.9 fix the problem

Comment 42 Ken A. 2003-01-30 19:51:39 UTC
using redhat 8.0 with latest redhat supplied version of mysql and all updates 
as of 01-30-2003, this problem still exists, 
but 2 workarounds exist: 
add: set-variable = thread_stack=256K 
to /etc/my.cnf [mysql] section
- make sure sure /etc/rc.d/init.d/mysqld contains:
 "/usr/bin/safe_mysqld --defaults-file=/etc/my.cnf"

or 

add ips and hostnames of connecting hosts to /etc/hosts.


Comment 43 Edinelson Keiji Shimokawa 2003-02-18 15:58:51 UTC
Try this at mysql prompt:

mysql> USE mysql;
mysql> GRANT ALL ON *.* TO root@"%" IDENTIFIED BY 'yourpassword';
mysql> FLUSH PRIVILEGES;

This worked for me.

Edinelson Keiji Shimokawa.


Comment 44 Patrick Macdonald 2003-03-12 20:36:17 UTC
*** Bug 75297 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.