Red Hat Bugzilla – Bug 160704
squid child processes exit with signal 6.. squid crashes
Last modified: 2007-11-30 17:07:18 EST
Description of problem:
I upgraded squid to this version released in RHSA-2005:415-16. Syslog starts
logging the following message repeatedly every 1 minute or so to /var/log/messages:
Squid Parent: child process 17865 exited due to signal 6
Googling that indicates it is a fatal error on the child process and everyone
says to check the cache.log for the cause. I have been unable to find a
resolution Googling for the messages in my cache.log.
Checking the cache.log file shows squid restarting repeatedly and trying to
rebuild the cache. Eventually the parent process dies leaving it's lock file
and pid file on the filesystem which must be deleted to restart the squid
service. Squid *does* appear to function for about an hour after a fresh start
before the parent process dies with too many errors logged to syslog
(squid: Exiting due to repeated, frequent failures).
I checked the squid log file sizes to make sure they were not over the 2 gig
limit.. not the problem.
I purged the cache directories off disk and started squid fresh. Problem
Note: *I am reporting this bug as a security level issue since this version is
intended to fix security issues with previous versions of Squid.* I had to roll
back Squid to a previous version to get my web caches working again.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. upgrade squid to version squid-2.5.STABLE6-3.4E.9:7.x86_64.
2. service squid start.
3. squid crashes repeatedly.
squid child processes exit with signal 6.. squid crashes
Squid runs normally and does not crash.
See attachment showing cache.log snippet.
Note: I am submitting this with Severity = security since this version of Squid
was released to fix security issues in previous versions.
Created attachment 115559 [details]
Please attach your squid.conf file so I can try to reproduce the problem here.
Created attachment 115578 [details]
squid configuration file
Attached squid.conf as requested
I have the same problem on i686 (Pentium4). Using strace I found the following
18082 write(2, "(squid): rfc1035.c:417: rfc1035RRUnpack: Assertion `(*off) <=
sz\' failed.\n", 74) = 74
I downgraded to 2.5.STABLE6-3.4E.5 and squid works ok now.
I'm having the exact same problem.
Here's an example DNS reply packet which squid received:
(retrieved from a strace) before quiting (assert failure and ABRT) with the
above message (also from strace, why isn't this logged to any file?)
Could you please check STABLE6-3.4E.6 version? src.rpm is here:
I upgraded to STABLE6-3.4E.6 as requested on both my sibling caches.
It is not crashing every few minutes anymore, but I am seeing the
following logged to my syslog on one of the machines:
Jul 11 15:55:19 lnxwc2 (squid): xstrdup: tried to dup a NULL pointer!
Jul 11 15:55:21 lnxwc2 squid: Squid Parent: child process 17158
exited due to signal 6
Jul 11 15:55:24 lnxwc2 squid: Squid Parent: child process 26795
*Note I did not re-init my disk caches since I didn't see anything
that looked like a significant problem in the cache logs after the
I'm having the exact same problem from squid-2.5.STABLE3-6.3E.9.i386.rpm to
(In reply to comment #7)
> I upgraded to STABLE6-3.4E.6 as requested on both my sibling caches.
> It is not crashing every few minutes anymore, but I am seeing the
> following logged to my syslog on one of the machines:
> Jul 11 15:55:19 lnxwc2 (squid): xstrdup: tried to dup a NULL pointer!
> Jul 11 15:55:21 lnxwc2 squid: Squid Parent: child process 17158
> exited due to signal 6
> Jul 11 15:55:24 lnxwc2 squid: Squid Parent: child process 26795
> *Note I did not re-init my disk caches since I didn't see anything
> that looked like a significant problem in the cache logs after the
Could you check it with strace?
A bug for assertion is here:
Here is a new testing package:
Could you check it? I removed two patches from STABLE6-3.4E.6...
Created attachment 116994 [details]
snippet from strace of xstrdup() fatal: tried to dup a null pointer in squid child
This is a snippet from the strace on a squid child that aborted due to the
fatal xstrdup() call.
Created attachment 116995 [details]
Full strace from xstrdup() null pointer (bzipped because of large file size).
Full strace of the xstrdup() fatal call. File is big (15MB) so I have
compressed it with bzip2.
(In reply to comment #11)
> Here is a new testing package:
> Could you check it? I removed two patches from STABLE6-3.4E.6...
I rolled both boxes to this version and the xstrdup() error persists on both.
It's not enough to crash the squid parent, but aborts the children several times
Thanks for testing. Could you please check this package?
It's a new package with all upstreams fixes...
Upgraded to squid-2.5.STABLE10-2.src.rpm which solves the xstrdup() problem and
squid seems stable. I started getting reports of random access denied issues
from users. Tested this with several pages and squid would refuse pages with
access denied by cache messages at random intervals. Refreshing the page five
or six times would eventually render the page correctly.
Rolled back to the test2 version for now.
Could you please check the original squid-2.5.STABLE6-3.4E.9:7.x86_64? Add
please "debug_options ALL,9" to /etc/squid/squid.conf, restart it and attach
/var/log/squid/cache.log file after some crashes. But be careful, this file may
be very big...
(In reply to comment #17)
> Could you please check the original squid-2.5.STABLE6-3.4E.9:7.x86_64? Add
> please "debug_options ALL,9" to /etc/squid/squid.conf, restart it and attach
> /var/log/squid/cache.log file after some crashes. But be careful, this file may
> be very big...
Sure. I will have to hold off for about a week however. We are right in the
middle of registration and this is the first week of classes, so I'll need to
keep things stable for now until we're past all the typical start of semester
tech issues. I'll touch base in a week or so with some debugging information.
Oh, sure :-) Thanks for your help.
I'm requesting the priority of this bug be increased to high because it is
almost 3 months old and is now effecting our organization on an almost daily
1. squid crashes
2. entries in the cache.log file match those in attachment id 115559
3. no one is able to access the internet through squid until an administrator
deletes and recreates the swap directories and restarts squid.
The problem is also not limited to X86_64 hardware, we are using i686.
Executives and admins at our orgnization are becoming increasingly annoyed due
to the issue and lack of prompt response from RedHat and it is creating and
unfavorable opinion of Red Hat and Red Hat Enterprise Linux within our
This sounds similar to
Could you please create and attach a strack trace when squid crashes? How to is
here - http://people.redhat.com/stransky/squid.html
(In reply to comment #22)
> Could you please create and attach a strack trace when squid crashes? How to is
> here - http://people.redhat.com/stransky/squid.html
I've installed squid-2.5.STABLE6-3.4E.12.dumps.src.rpm on one of my caches.
Will advise when I get a core and stack trace.
Thanks. btw. I slightly updated how-to page, squid for test needs to be run as
"#/usr/sbin/squid -NCDd1", not with "service squid start". It's because the
latter perform some clean up before shutdown.
Created attachment 118763 [details]
output of /usr/sbin/squid -NCDd 1
(In reply to comment #24)
> Thanks. btw. I slightly updated how-to page, squid for test needs to be run as
> "#/usr/sbin/squid -NCDd1", not with "service squid start". It's because the
> latter perform some clean up before shutdown.
I'm running from the shell as described. ulimit is unlimited. issued ulimit -c
unlimited as well. Still not seeing cores after several tries. Looks like
we're aborting before it gets a chance to crash.
See attachment with id=118763
(In reply to comment #26)
> I'm running from the shell as described. ulimit is unlimited. issued ulimit -c
> unlimited as well. Still not seeing cores after several tries. Looks like
> we're aborting before it gets a chance to crash.
> See attachment with id=118763
Great, it looks like a dupe of this issue:
People report some problems with the upstream patch, so I'll make a package with
this patch and publish it for testing...
Want to make these go away?:
> Jul 11 15:55:19 lnxwc2 (squid): xstrdup: tried to dup a NULL pointer!
Compare this with the one in the rpm. They are not the same...
Fixed mine 2 weeks ago and have not seen xstrdup error since.
(In reply to comment #28)
This thread is a little crowded so you can open a new bug for this issue if you
have this problem.
I have up2dated to new release of squid 3.4E.11.i386.rpm and got squid craching.
Config file and debug (ALL,9) log file (cache.log ~ 2Gb) gzipped and can be
found at http://ftp.mstuca.ru/uploads/squid/cache.log.bz2
(In reply to comment #30)
> I have up2dated to new release of squid 3.4E.11.i386.rpm and got squid craching.
> Config file and debug (ALL,9) log file (cache.log ~ 2Gb) gzipped and can be
> found at http://ftp.mstuca.ru/uploads/squid/cache.log.bz2
Could you provide the cache.log file generated with debug (ALL,1) too?
Due to PIE gdb can't read symbols from the debug package. If your squid crashes
(and it isn't a problem with assertion) and you can't obtain a stack-trace there
are new packages which aren't compiled with PIE:
The new testing binaries (19.assert) are here:
Created attachment 119670 [details]
squid.2.5.Stable11 squid.spec with updated build and config patches
IMHO, Red Hat needs to release a clean rebuild of squid rpms from the upstream
After being totally frustrated by Red Hat's poor response in fixing this issue,
I decided to do it myself.
1. downloaded and installed the squid-2.5.STABLE6-3.src.rpm
2. downloaded 2.5.STABLE11 source bz2 archive to /usr/src/redhat/SOURCES
3. Removed all back ported patch references from the squid.spec file and added
entry for the new /etc/squid/cachemgr.conf file (see attachment)
4. Rebuilt the redhat build.patch and config.patch files used to configure the
Makefiles so they would apply to the new version
5. Built the RPM
I've been running this custom package on 3 machines for a week and haven't seen
this problem and quite a few other minor issues, nor I have encountered any
If Red Hat wants to keep me and its other customers, they need to provide the
quality and support we paid for when paid for a RHEL subscription. If I'm going
to have to do this much work to diagnose and fix a buggy package, i might as
well use a free disribution and apply updates by compiling the new version of
the authors source code.
Interestingly enough, this is the same way the STABLE11-2 package for FC4 is
created except FC4 contains also contains the delay pool patch and a few other
patches targeted for STABLE12.
So how about it RedHat? Please, give us the updates and support we paid for!
I'm going to propose this for RHEL-3 and RHEL-4.
I would also appreciate that, as it seems that a different approach to resolving
this issue is warranted. I'm willing to test a release candidate under RHEL3.
Okay, I'll prepare the upstream packages for testing.
There are requests for update to current upstream, so you can write your comment
here. (Bug 170390, Bug 170392)
Page with packages from upstream and hopefully fixed packages for RHEL3/4 is here:
I am using squid-2.5.STABLE6-3.4E.11. The following URL consistently crashes
squid with the same error referred to in the summary:
The version squid-2.5.STABLE3-6.3E.14.RC1 was rebuilt and right now is working
without craching from the 18 of October.
I too have been running squid-2.5.STABLE3-6.3E.14.RC1 on both my production
caches since 10/18/2005 with great success. No more crashing and no more signal
Dave R: Try the Release candidate packages on Martin's site. They are working
fine for that URL you posted.
Thanks everybody :)
squid-2.5.STABLE3-6.3E.14.RC1 is for RHEL3.
squid-2.5.STABLE6-3.4E.11.RC1 is for RHEL4.
This bug is against RHEL4.
Are you guys running the RHEL3 version of squid under RHEL4?
(In reply to comment #49)
> squid-2.5.STABLE3-6.3E.14.RC1 is for RHEL3.
> squid-2.5.STABLE6-3.4E.11.RC1 is for RHEL4.
> This bug is against RHEL4.
> Are you guys running the RHEL3 version of squid under RHEL4?
I'm not, I just can't copy/paste :)
Please allow me to correct my last comment (#48).
I'm running squid-2.5.STABLE6-3.4E.11.RC1 on RHEL4 with success. The version I
posted in comment #48 is incorrect.
Sorry for any confusion.
The changelog from squid-2.5.STABLE6-3.4E.11.RC1 simply states:
- fix for #160704
What is the actual patch, Martin?
Created attachment 120555 [details]
Patch for RHEL4 is here
The new release-candidate packages for RHEL3/4 are available here:
*** Bug 171169 has been marked as a duplicate of this bug. ***
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.
Internal Status set to 'Resolved'
Status set to: Closed by Client
Resolution set to: 'RHEL 4 U4'
This event sent from IssueTracker by uthomas