Bug 721365

Summary: [abrt] squid-7:3.1.12-1.fc14: death: Process /usr/sbin/squid was killed by signal 6 (SIGABRT)
Product: [Fedora] Fedora Reporter: Joao Carlos Mendes Luis <jonny>
Component: squidAssignee: Jiri Skala <jskala>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 14CC: aglotov, ccom.nujs, henrik, jonathansteffan, jskala, redhat
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Unspecified   
Whiteboard: abrt_hash:12a9578e604e4cb58097b45b548c582fd2a87075
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-11-29 13:37:00 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
File: backtrace none

Description Joao Carlos Mendes Luis 2011-07-14 11:59:25 UTC 
abrt version: 1.1.18
architecture: i686
Attached file: backtrace, 19875 bytes
cmdline: (squid) -f /etc/squid/squid.conf
component: squid
Attached file: coredump, 55787520 bytes
crash_function: death
executable: /usr/sbin/squid
kernel: 2.6.35.13-92.fc14.i686.PAE
package: squid-7:3.1.12-1.fc14
rating: 4
reason: Process /usr/sbin/squid was killed by signal 6 (SIGABRT)
release: Fedora release 14 (Laughlin)
time: 1310644432
uid: 0

comment
-----
Strange things I have in my configuration:
- IPv6 setup
- URL redirector
- Transparent redirector (configured, but in use anymore)

How to reproduce
-----
1. Squid is running
2. Suddenly, it dies: "Squid Parent: child process XXX exited due to signal 6 with status 0"
3. Happens many times a day
Comment 1 Joao Carlos Mendes Luis 2011-07-14 11:59:29 UTC 
Created attachment 512873 [details]
File: backtrace
Comment 2 Jiri Skala 2011-10-03 08:08:56 UTC 
May I have your conf file?

Thanks, Jiri
Comment 3 Henrik Nordström 2011-10-03 12:35:26 UTC 
And what is said in cache.log?
Comment 4 João Carlos Mendes Luís 2011-10-03 14:52:04 UTC 
I am currently running squid-3.1.15-1.fc14.i686

Logs from the last month do not show anymore this problem, but last week I had to clean the disk cache, due to some inconsistencies, and since then, I had no more problems.

Not sure if the problems are related, though.  Anyway, here is an excerpt from cache.log:

2011/09/24 17:58:01| Starting Squid Cache version 3.1.15 for i386-redhat-linux-gnu...
2011/09/24 17:58:01| Process ID 952
2011/09/24 17:58:01| With 1024 file descriptors available
2011/09/24 17:58:01| Initializing IP Cache...
2011/09/24 17:58:01| DNS Socket created at [::], FD 7
2011/09/24 17:58:01| DNS Socket created at 0.0.0.0, FD 8
2011/09/24 17:58:01| Adding domain home.jonny.eng.br from /etc/resolv.conf
2011/09/24 17:58:01| Adding nameserver 127.0.0.1 from /etc/resolv.conf
2011/09/24 17:58:01| helperOpenServers: Starting 5/5 'redir.pl' processes
2011/09/24 17:58:01| User-Agent logging is disabled.
2011/09/24 17:58:01| Referer logging is disabled.
2011/09/24 17:58:01| Unlinkd pipe opened on FD 23
2011/09/24 17:58:01| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2011/09/24 17:58:01| Store logging disabled
2011/09/24 17:58:01| Swap maxSize 33554432 + 32768 KB, estimated 2583630 objects
2011/09/24 17:58:01| Target number of buckets: 129181
2011/09/24 17:58:01| Using 131072 Store buckets
2011/09/24 17:58:01| Max Mem  size: 32768 KB
2011/09/24 17:58:01| Max Swap size: 33554432 KB
2011/09/24 17:58:01| Version 1 of swap file with LFS support detected...
2011/09/24 17:58:01| Rebuilding storage in /var/spool/squid (CLEAN)
2011/09/24 17:58:01| Using Least Load store dir selection
2011/09/24 17:58:01| Set Current Directory to /var/spool/squid
2011/09/24 17:58:01| Loaded Icons.
2011/09/24 17:58:01| Accepting  HTTP connections at [::]:3128, FD 26.
2011/09/24 17:58:01| Accepting  intercepted HTTP connections at 0.0.0.0:3129, FD 27.
2011/09/24 17:58:01| HTCP Disabled.
2011/09/24 17:58:01| Squid plugin modules loaded: 0
2011/09/24 17:58:01| Adaptation support is off.
2011/09/24 17:58:01| Ready to serve requests.
2011/09/24 17:58:01| Store rebuilding is 0.81% complete
2011/09/24 17:58:02| Done reading /var/spool/squid swaplog (505404 entries)
2011/09/24 17:58:02| Finished rebuilding storage from disk.
2011/09/24 17:58:02|    505404 Entries scanned
2011/09/24 17:58:02|         0 Invalid entries.
2011/09/24 17:58:02|         0 With invalid flags.
2011/09/24 17:58:02|    505404 Objects loaded.
2011/09/24 17:58:02|         0 Objects expired.
2011/09/24 17:58:02|         0 Objects cancelled.
2011/09/24 17:58:02|         0 Duplicate URLs purged.
2011/09/24 17:58:02|         0 Swapfile clashes avoided.
2011/09/24 17:58:02|   Took 1.56 seconds (324402.60 objects/sec).
2011/09/24 17:58:02| Beginning Validation Procedure
2011/09/24 17:58:02|   524288 Entries Validated so far.
2011/09/24 17:58:02|   Completed Validation Procedure
2011/09/24 17:58:02|   Validated 1010827 Entries
2011/09/24 17:58:02|   store_swap_size = 29194244
2011/09/24 17:58:03| storeLateRelease: released 0 objects
2011/09/24 18:02:09| WARNING: swapfile header inconsistent with available data
FATAL: Received Segment Violation...dying.
2011/09/24 18:02:09| storeDirWriteCleanLogs: Starting...
2011/09/24 18:02:09| WARNING: Closing open FD   26
2011/09/24 18:02:09|     65536 entries written so far.
2011/09/24 18:02:09|    131072 entries written so far.
2011/09/24 18:02:09|    196608 entries written so far.
2011/09/24 18:02:09|    262144 entries written so far.
2011/09/24 18:02:09|    327680 entries written so far.
2011/09/24 18:02:09|    393216 entries written so far.
2011/09/24 18:02:09|    458752 entries written so far.
2011/09/24 18:02:09|   Finished.  Wrote 505569 entries.
2011/09/24 18:02:09|   Took 0.10 seconds (5130961.20 entries/sec).
CPU Usage: 3.426 seconds = 1.844 user + 1.583 sys
Maximum Resident Size: 293344 KB
Page faults with physical i/o: 0
Memory usage for squid via mallinfo():
        total space in arena:   64816 KB
        Ordinary blocks:        64675 KB     30 blks
        Small blocks:               0 KB      1 blks
        Holding blocks:          5308 KB      7 blks
        Free Small blocks:          0 KB
        Free Ordinary blocks:     140 KB
        Total in use:           69983 KB 108%
        Total free:               140 KB 0%
2011/09/24 18:02:13| Starting Squid Cache version 3.1.15 for i386-redhat-linux-gnu...
2011/09/24 18:02:13| Process ID 1097
2011/09/24 18:02:13| With 1024 file descriptors available
Comment 5 Jiri Skala 2011-10-04 07:27:05 UTC 
The first findings:

- the source of troubles are url_rewrite* options. to be continued ...
Comment 6 Henrik Nordström 2011-10-04 08:00:36 UTC 
the backtrace is cache swapin related, not url_rewrite.
Comment 7 João Carlos Mendes Luís 2011-10-04 14:52:35 UTC 
Henrik,

    I've sent in private my squid.conf to Jiri, maybe he has seen something interesting there.  I'll send it to you too.

    But I agree with you.  Taking a further look into the backtrace, and to the troubles I had with squid-3.1.15-1.fc14.i686, probably some check is missing in the Swap Validation routines.

    This machine is my desktop and also my lab.  It crashes sometimes, and the filesystem may be left in an inconsistent state that squid3 is not able to deal with.

    Thanks in advance for your help!
Comment 8 Jiri Skala 2011-10-04 15:12:00 UTC 
Yes, the c#5 is nonsense. I've written the comment based only on the first experience with the conf file.

I agree with the possible filesystem issue.
Comment 9 Jiri Skala 2011-11-29 13:37:00 UTC 
I'm decided to close this bug in accordance with your private mail from October 4 2011. The decision is based on oncoming EOL of F14.
If the bug occurs again on higher Fedora release I'll await reopening the bug.
Comment 10 zeus 2013-03-16 11:55:30 UTC 
Hello

We run squid here at the university on an IBMx3650 M3 with 32 GB of RAM on a RAID0 ( 4/146GB with two disks paired and mirrored)

There was a power surge at the university this afternoon and the computer did not shutdown normally. None of the machines however suffered any damage as a result of the surge (the central powerbackup however did). When the machine was turned on, squid started up but incessantly kept crashing with 100%cpu usage with the following error at cache.log

2013/03/16 16:35:03| WARNING: swapfile header inconsistent with available data
FATAL: Received Segment Violation...dying.
2013/03/16 16:35:03| storeDirWriteCleanLogs: Starting...
2013/03/16 16:35:03| WARNING: Closing open FD   17
2013/03/16 16:35:03|     65536 entries written so far.
2013/03/16 16:35:03|    131072 entries written so far.
2013/03/16 16:35:03|    196608 entries written so far.
2013/03/16 16:35:03|    262144 entries written so far.
2013/03/16 16:35:03|    327680 entries written so far.
2013/03/16 16:35:03|    393216 entries written so far.
2013/03/16 16:35:03|    458752 entries written so far.
2013/03/16 16:35:03|    524288 entries written so far.
2013/03/16 16:35:03|    589824 entries written so far.
2013/03/16 16:35:03|    655360 entries written so far.
2013/03/16 16:35:03|    720896 entries written so far.
2013/03/16 16:35:03|    786432 entries written so far.
2013/03/16 16:35:03|   Finished.  Wrote 833156 entries.
2013/03/16 16:35:03|   Took 0.24 seconds (3470037.48 entries/sec).
CPU Usage: 14.707 seconds = 12.713 user + 1.994 sys
Maximum Resident Size: 1110848 KB
Page faults with physical i/o: 0
Memory usage for squid via mallinfo():
        total space in arena:  291852 KB
        Ordinary blocks:       291711 KB     50 blks
        Small blocks:               0 KB      5 blks
        Holding blocks:         15260 KB     10 blks
        Free Small blocks:          0 KB
        Free Ordinary blocks:     140 KB
        Total in use:          306971 KB 105%
        Total free:               141 KB 0%

Any help will be much appreciated! Thank you
Comment 11 zeus 2013-03-16 12:18:07 UTC 
UPDATE:

So we looked into squid's cache folder and found that there was a 'swap.state' and a 'swap.state.last-clean'. We proceeded by removing both this files in a hope that squid would recreate them. This didn't happen and squid created a new 'swap.state' and a 'swap.state.new'. But the crashes persisted.

So we decided to remove the cache folder entirely and use the squid -z command to recreate the cache. This did the trick it seems, squid is running fine as before.

Also, forgot to mention this earlier but we're running RHEL 6.4 (Santiago)

Thank you