Description of problem: I'm seeing squid continuously starting/restarting/crashing in my log files. The logpriority is at warning, so I'm assuming this is not a normal occurrence. This is on an smp Dell 1655 blade server. Can anyone tell me what might be causing this? The logs are showing a child process exiting due to signal 6. man 7 signal shows that signal 6 is SIGABRT 6 Core Abort signal from abort(3) Log messages: Feb 18 05:31:18 webc1 squid[13403]: storeDirWriteCleanLogs: Starting...Feb 18 05:31:18 webc1 squid[13403]: WARNING: Closing open FD 11 Feb 18 05:31:18 webc1 squid[13403]: 65536 entries written so far. Feb 18 05:31:18 webc1 squid[13403]: 131072 entries written so far. Feb 18 05:31:18 webc1 squid[13403]: 196608 entries written so far. Feb 18 05:31:18 webc1 squid[13403]: 262144 entries written so far. Feb 18 05:31:18 webc1 squid[13403]: 327680 entries written so far. Feb 18 05:31:18 webc1 squid[13403]: Finished. Wrote 360001 entries. Feb 18 05:31:18 webc1 squid[13403]: Took 0.3 seconds (1087258.3entries/sec). Feb 18 05:31:18 webc1 squid[9313]: Squid Parent: child process 13403exited due to signal 6 Feb 18 05:31:21 webc1 squid[13407]: Starting Squid Cache version2.5.STABLE3 for i686-pc-linux-gnu... Feb 18 05:31:21 webc1 squid[13407]: Process ID 13407Feb 18 05:31:21 webc1 squid[13407]: With 1024 file descriptors available Feb 18 05:31:21 webc1 squid[13407]: DNS Socket created at 0.0.0.0, port32780, FD 5 2004/02/19 17:10:27| Starting Squid Cache version 2.5.STABLE4 for i686-pc-linux-gnu... 2004/02/19 17:10:27| Process ID 13474 2004/02/19 17:10:27| With 1024 file descriptors available 2004/02/19 17:10:27| DNS Socket created at 0.0.0.0, port 32786, FD 5 2004/02/19 17:10:27| Adding nameserver 10.10.20.100 from /etc/resolv.conf 2004/02/19 17:10:27| Adding nameserver 10.10.20.101 from /etc/resolv.conf 2004/02/19 17:10:27| Adding nameserver 10.10.20.102 from /etc/resolv.conf 2004/02/19 17:10:27| Adding nameserver 10.10.20.103 from /etc/resolv.conf 2004/02/19 17:10:27| Unlinkd pipe opened on FD 10 2004/02/19 17:10:27| Swap maxSize 33554432 KB, estimated 2581110 objects 2004/02/19 17:10:27| Target number of buckets: 129055 2004/02/19 17:10:27| Using 131072 Store buckets 2004/02/19 17:10:27| Max Mem size: 524288 KB 2004/02/19 17:10:27| Max Swap size: 33554432 KB 2004/02/19 17:10:27| Store logging disabled 2004/02/19 17:10:27| Rebuilding storage in /var/spool/squid (DIRTY) 2004/02/19 17:10:27| Using Round Robin store dir selection 2004/02/19 17:10:27| Current Directory is / 2004/02/19 17:10:27| Loaded Icons. 2004/02/19 17:10:27| Accepting HTTP connections at 0.0.0.0, port 80, FD 11. 2004/02/19 17:10:27| Accepting ICP messages at 0.0.0.0, port 3130, FD 13. 2004/02/19 17:10:27| Accepting SNMP messages on port 3401, FD 14. 2004/02/19 17:10:27| WCCP Disabled. 2004/02/19 17:10:27| Ready to serve requests. 2004/02/19 17:10:27| Configuring Sibling webc1.nocout.navy.mil/80/3130 2004/02/19 17:10:27| Store rebuilding is 0.8% complete FATAL: Received Segment Violation...dying. 2004/02/19 17:10:33| Not currently OK to rewrite swap log. 2004/02/19 17:10:33| storeDirWriteCleanLogs: Operation aborted. CPU Usage: 6.010 seconds = 2.900 user + 3.110 sys Maximum Resident Size: 0 KB Page faults with physical i/o: 516 Memory usage for squid via mallinfo(): total space in arena: 32474 KB Ordinary blocks: 32456 KB 2 blks Small blocks: 0 KB 0 blks Holding blocks: 716 KB 2 blks Free Small blocks: 0 KB Free Ordinary blocks: 18 KB Total in use: 33172 KB 102% Total free: 18 KB 0% Feb 19 17:10:27 webc2 squid[13474]: Using Round Robin store dir selection Feb 19 17:10:27 webc2 squid[13474]: Current Directory is / Feb 19 17:10:27 webc2 squid[13474]: Loaded Icons. Feb 19 17:10:27 webc2 squid[13474]: Accepting HTTP connections at 0.0.0.0, port 80, FD 11. Feb 19 17:10:27 webc2 squid[13474]: Accepting ICP messages at 0.0.0.0, port 3130, FD 13. Feb 19 17:10:27 webc2 squid[13474]: Accepting SNMP messages on port 3401, FD 14. Feb 19 17:10:27 webc2 squid[13474]: WCCP Disabled. Feb 19 17:10:27 webc2 squid[13474]: Ready to serve requests. Feb 19 17:10:27 webc2 squid[13474]: Configuring Sibling webc1.nocout.navy.mil/80/3130 Feb 19 17:10:27 webc2 squid[13474]: Store rebuilding is 0.8% complete Feb 19 17:10:30 webc2 squid[13474]: access 1077210630.391 667 205.72.1.21 TCP_MISS/304 267 GET http://images.bestbuy.com/BestBuy_US/images/global/misc/gl_stripes_top_333399.gif - TIMEOUT_DIRECT/63.215.124.23 image/gif Feb 19 17:10:31 webc2 squid[13474]: access 1077210631.306 658 205.106.99.21 TCP_MISS/200 2241 GET http://thumbs.ebaystatic.com/pict/24604322966464.jpg - DIRECT/64.124.29.189 image/jpeg Feb 19 17:10:31 webc2 squid[13474]: access 1077210631.382 664 205.72.1.21 TCP_MISS/304 240 GET http://us.i1.yimg.com/us.yimg.com/i/us/nws/main5.gif - DIRECT/63.215.124.46 image/gif Feb 19 17:10:32 webc2 squid[13474]: access 1077210632.122 692 205.106.99.21 TCP_MISS/200 1879 GET http://thumbs.ebaystatic.com/pict/24606385106464.jpg - TIMEOUT_DIRECT/64.124.29.188 image/jpeg Feb 19 17:10:32 webc2 squid[13474]: access 1077210632.199 1048 205.106.145.42 TCP_MISS/200 5626 GET http://www.epicgals.com/am33/affiliates/ - DIRECT/66.228.210.150 text/html Feb 19 17:10:32 webc2 squid[13474]: access 1077210632.211 686 205.72.1.21 TCP_MISS/302 263 GET http://www.sharperimage.com/ionicbreeze - TIMEOUT_DIRECT/64.41.222.72 text/html Feb 19 17:10:32 webc2 squid[13474]: access 1077210632.268 678 205.106.99.21 TCP_MISS/200 2445 GET http://www.ebay.com/sfpromo/allthingsautoV2.html - TIMEOUT_DIRECT/66.135.192.88 text/html Feb 19 17:10:32 webc2 squid[13474]: access 1077210632.345 623 205.72.1.21 TCP_MISS/200 943 GET http://www.sonystyle.com/intershoproot/eCS/Store/en/imagesOnline/header/hdr_btn_support_hi.gif - DIRECT/129.33.21.28 image/gif Feb 19 17:10:32 webc2 squid[13474]: access 1077210632.766 2033 205.72.1.21 TCP_MISS/200 12475 GET http://images.overstock.com/f/102/3117/8h/www.overstock.com/images/products/muze/dvd/205745.JPG - DIRECT/63.215.124.45 image/jpeg Feb 19 17:10:32 webc2 squid[13474]: access 1077210632.901 677 205.106.99.21 TCP_MISS/200 2053 GET http://thumbs.ebaystatic.com/pict/24606201056464.jpg - TIMEOUT_DIRECT/64.124.29.220 image/jpeg Feb 19 17:10:32 webc2 squid[13474]: access 1077210632.962 643 205.72.1.21 TCP_MISS/200 2934 GET http://www.biotimeinc.com/ProductSub.gif - TIMEOUT_DIRECT/216.205.146.188 image/gif Feb 19 17:10:33 webc2 squid[13474]: access 1077210633.081 1055 205.106.145.42 TCP_MISS/200 6029 GET http://www.epicgals.com/am33/affiliates/images/011t.jpg - TIMEOUT_DIRECT/66.228.210.148 image/jpeg Feb 19 17:10:33 webc2 squid[13474]: access 1077210633.282 671 205.72.1.21 TCP_MISS/304 199 GET http://us.js1.yimg.com/us.yimg.com/lib/hdr/ygma.css - DIRECT/166.90.208.153 text/css Feb 19 17:10:33 webc2 squid[13474]: access 1077210633.384 1059 205.106.145.42 TCP_MISS/200 5525 GET http://www.epicgals.com/am33/affiliates/images/003t.jpg - TIMEOUT_DIRECT/66.228.210.150 image/jpeg Feb 19 17:10:33 webc2 squid[13474]: access 1077210633.500 684 205.72.1.21 TCP_MISS/302 851 GET http://www.sharperimage.com/ionicbreeze/ - TIMEOUT_DIRECT/64.41.222.72 text/html Feb 19 17:10:33 webc2 squid[13474]: Not currently OK to rewrite swap log. Feb 19 17:10:33 webc2 squid[13474]: storeDirWriteCleanLogs: Operation aborted. Feb 19 17:10:33 webc2 squid[13200]: Squid Parent: child process 13474 exited due to signal 6 Feb 19 17:10:33 webc2 squid[13200]: Exiting due to repeated, frequent failures Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Start squid on 1655 2. Load under normal load 3. Watch logs and see failures and crashes Actual results: See logs in Description Expected results: Normal occurance as not shown from logs in Description Additional info:
Can you attach your squid.conf file to this so I'll have a chance of debugging it? The "FATAL: Received Segment Violation...dying." certainly appears to be an error, but without the config file, I can't even try to replicate it.
http_port 80 icp_port 3130 cache_peer webc1.nocout.xxxx.xxx sibling 80 3130 hierarchy_stoplist cgi-bin ? cache_mem 512 MB cache_swap_low 72 cache_swap_high 75 maximum_object_size 32768 KB minimum_object_size 0 KB cache_dir diskd /var/spool/squid 32768 32 1024 store_dir_select_algorithm round-robin cache_store_log none debug_options ALL,1 ftp_user ftpuser acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl webc1.nocout.xxxx.xxx src 10.10.20.113/255.255.255.255 acl nocman src 10.10.20.120/255.255.255.255 acl snmppublic snmp_community pubinner acl Safe_ports port 80 21 443 563 70 210 1025-65535 http_access deny !Safe_ports http_access allow all icp_access allow webc1.nocout.xxxx.xxx icp_access deny all miss_access allow all cache_mgr webmaster cachemgr_passwd secret all cache_effective_user squid cache_effective_group squid visible_hostname webc2.ior.xxxx.xxx httpd_accel_host virtual httpd_accel_port 80 httpd_accel_with_proxy on httpd_accel_uses_host_header on dns_testnames infosec.xxxx.xxx spawar.xxxx.xxx logfile_rotate 5 buffered_logs on always_direct allow nocman snmp_port 3401 snmp_access allow snmppublic localhost snmp_access deny all memory_pools on log_icp_queries off ipcache_size 4192 fqdncache_size 4192
It would also be helpful if you could enable core file dumps and setuid cores. If you could attach the core here it would be helpful in debugging this. Thanks!
Created attachment 98238 [details] patch to try fix a few shortcommings... Based on the last core that jason (woodward) sent me regarding this bug, This patch should clear up a few things. The core file submitted clearly indicated that the segfault occured inside logfilePrintf(). My guess is that some function was calling this with an improperly constructed format string, or an invalid requisite argument. I was unfortunately unable to find the problem, but I was able to relieve the segfault by rewriting the logfilePrintf function as a macro in this patch. I understand this is a completely undesireable fix, but since this group is under a deadline, I thought this might help track down the real problem, or at least patch it up until a better fix is constructed. Also, this patch corrects something else that I noticed while investigating this problem. The assert macro is used quite a bit in this app, but since it runs a child as a daemon, any error messages reported due to failed asserts get lost. This patch also adds a wrapper around assert to copy an error message into syslog anytime an assert fails. While I'm writing here, can I ask why squid is built with -fPIE and -pie turned on? This causes it to be reported as a shared object, and makes for difficult debugging, since you have to manually load it at the appropriate address in the debugger.
Created attachment 98692 [details] allow syslogging to work The attached patch seems to fix our syslog issues on top of the fix the last patch gave us.
While testing one setup we also got this error message. While googling around I found suggestion that upgrading to STABLE4 may solve the problem: http://www.mail-archive.com/squid-users@squid- cache.org/msg08444.html By the way our test was performed on x86_64 dual Opteron system.
I get this under RHEL3U5. It started when I upgraded from: squid-2.5.STABLE3-6.3E.8 to: squid-2.5.STABLE3-6.3E.13 I backed down to: squid-2.5.STABLE3-6.3E.9 And the problem has cleared up.
Could you please create and attach a strack trace when squid crashes? How to is here - http://people.redhat.com/stransky/squid.html
Due to PIE gdb can't read symbols from the debug package. If your squid crashes (and it isn't a problem with assertion) and you can't obtain a stack-trace there are new packages which aren't compiled with PIE: http://people.redhat.com/stransky/squid.html
*** This bug has been marked as a duplicate of 165367 ***