From Bugzilla Helper: User-Agent: Mozilla/4.72 [en] (X11; U; Linux 2.2.14-5.0 i686) Description of problem: TCP services don't respond. ICMP still works. I did an snmpwalk, it got halfway through but then hang. It actually connects to the port, just the services don't interact. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Just connect to the port 2. 3. Actual Results: Services hang Expected Results: normal network function Additional info:
Here are some log entries for you. Jan 25 13:18:57 netops kernel: <1>Unable to handle kernel paging request at virtual address ff344b40 Jan 25 13:18:57 netops kernel: printing eip: Jan 25 13:18:57 netops kernel: c0142b41 Jan 25 13:18:57 netops kernel: *pde = 00000000 Jan 25 13:18:57 netops kernel: Oops: 0000 Jan 25 13:18:57 netops kernel: CPU: 0 Jan 25 13:18:57 netops kernel: EIP: 0010:[d_lookup+173/252] Not tainted Jan 25 13:18:57 netops kernel: EIP: 0010:[<c0142b41>] Not tainted Jan 25 13:18:57 netops kernel: EFLAGS: 00010246 Jan 25 13:18:57 netops kernel: EIP is at d_lookup [kernel] 0xad Jan 25 13:18:57 netops kernel: eax: 00000000 ebx: d77753c0 ecx: 0000001f edx: 0000001f Jan 25 13:18:57 netops kernel: esi: ff344b40 edi: c1f30000 ebp: dff60218 esp: d76cbee8 Jan 25 13:18:57 netops kernel: ds: 0018 es: 0018 ss: 0018 Jan 25 13:18:57 netops kernel: Process la-larrd.pl (pid: 29512, stackpage=d76cb000) Jan 25 13:18:57 netops kernel: Stack: dff60218 c1f30000 800f49bf 0000001f d76cbf54 800f49bf d76cbf9c c1f3001f Jan 25 13:18:57 netops kernel: c013a54a c1902ac0 d76cbf54 d76cbf54 c013ac8f c1902ac0 d76cbf54 00000000 Jan 25 13:18:57 netops kernel: 00000009 00000000 00000000 c01239c6 c1c87d60 00000282 df8e54e0 00001000 Jan 25 13:18:57 netops kernel: Call Trace: [cached_lookup+14/72] cached_lookup [kernel] 0xe Jan 25 13:18:57 netops kernel: Call Trace: [<c013a54a>] cached_lookup [kernel] 0xe Jan 25 13:18:57 netops kernel: [path_walk+1375/2008] path_walk [kernel] 0x55f Jan 25 13:18:57 netops kernel: [<c013ac8f>] path_walk [kernel] 0x55f Jan 25 13:18:57 netops kernel: [unmap_fixup+98/300] unmap_fixup [kernel] 0x62 Jan 25 13:18:57 netops kernel: [<c01239c6>] unmap_fixup [kernel] 0x62 Jan 25 13:18:57 netops kernel: [__user_walk+56/80] __user_walk [kernel] 0x38 Jan 25 13:18:57 netops kernel: [<c013b394>] __user_walk [kernel] 0x38 Jan 25 13:18:57 netops kernel: [sys_stat64+18/104] sys_stat64 [kernel] 0x12 Jan 25 13:18:57 netops kernel: [<c013850e>] sys_stat64 [kernel] 0x12 Jan 25 13:18:57 netops kernel: [sys_munmap+52/60] sys_munmap [kernel] 0x34 Jan 25 13:18:57 netops kernel: [<c0123d80>] sys_munmap [kernel] 0x34 Jan 25 13:18:57 netops kernel: [system_call+51/56] system_call [kernel] 0x33 Jan 25 13:18:57 netops kernel: [<c0106e07>] system_call [kernel] 0x33 Jan 25 13:18:57 netops kernel: Jan 25 13:18:57 netops kernel: Jan 25 13:18:57 netops kernel: Code: f3 a6 0f 97 c2 0f 92 c0 38 c2 75 2b ff 03 8b 03 48 75 1a ff Jan 25 14:13:59 netops kernel: <3>swap_free: Bad swap offset entry ff000000 Jan 25 14:13:59 netops kernel: swap_free: Bad swap offset entry ff000000 Jan 25 14:13:59 netops last message repeated 2 times Jan 25 14:16:50 netops sshd(pam_unix)[16712]: session closed for user bambenek Jan 25 14:16:50 netops su(pam_unix)[16755]: session closed for user ids Jan 25 14:19:24 netops kernel: swap_free: Unused swap offset entry 0000ff00 Jan 25 14:28:56 netops last message repeated 8 times Jan 25 14:28:56 netops kernel: swap_free: Unused swap offset entry 0000ff00 Jan 25 14:28:56 netops kernel: swap_free: Bad swap offset entry ff000000 Jan 25 14:28:56 netops kernel: swap_free: Bad swap offset entry ff000000 Jan 25 14:28:56 netops kernel: swap_free: Unused swap offset entry 0000ff00 Jan 25 14:28:56 netops kernel: swap_free: Unused swap offset entry 0000ff00 Jan 25 14:28:56 netops kernel: swap_free: Bad swap offset entry ff000000 Jan 25 14:28:56 netops kernel: swap_free: Bad swap offset entry ff000000 Jan 25 14:28:56 netops kernel: swap_free: Unused swap offset entry 0000ff00 Jan 25 14:28:56 netops kernel: swap_free: Unused swap offset entry 0000ff00 Jan 25 14:28:56 netops kernel: swap_free: Bad swap offset entry ff000000 Jan 25 14:28:56 netops kernel: swap_free: Bad swap offset entry ff000000 Jan 25 14:28:56 netops kernel: swap_free: Unused swap offset entry 0000ff00 Jan 25 14:28:56 netops kernel: swap_free: Unused swap offset entry 0000ff00 Jan 25 14:28:56 netops kernel: swap_free: Bad swap offset entry ff000000 Jan 25 14:28:56 netops kernel: swap_free: Bad swap offset entry ff000000 Jan 25 20:17:02 netops ucd-snmp[734]: [smux_accept] accepted fd 7 from 12.109.78.107:1898 Jan 25 20:23:19 netops ucd-snmp[734]: [smux_accept] accepted fd 7 from 12.109.78.107:1636 Jan 25 20:23:24 netops ucd-snmp[734]: [smux_accept] accepted fd 7 from 12.109.78.107:1644 Jan 25 20:23:28 netops ucd-snmp[734]: [smux_accept] accepted fd 7 from 12.109.78.107:1664 Jan 25 20:27:31 netops ucd-snmp[734]: Received SNMP packet(s) from 12.161.227.227 Jan 28 09:21:47 netops kernel: apm: busy: Unable to enter requested state
Doesn't look like a MySQL bug to me.... which kernel is this?
Kernel is 2.4.9-21
Hmmm Any idea which modules are loaded ?
Here is the output from lsmod Module Size Used by eepro100 17040 1 ide-scsi 7904 0 scsi_mod 94648 1 [ide-scsi] ide-cd 26560 0 cdrom 29568 0 [ide-cd] usb-uhci 20708 0 (unused) usbcore 49920 1 [usb-uhci] ext3 59808 5 jbd 39076 5 [ext3]
Here is RPM listing: redhat-logos-1.1.3-1 cracklib-2.7-12 dosfstools-2.7-1 gdbm-1.8.0-10 ksymoops-2.4.1-1 mktemp-1.5-11 perl-5.6.0-17 setserial-2.17-4 netconfig-0.8.11-7 termcap-11.0.1-10 crontabs-1.10-1 groff-1.17.2-3 ncurses-5.2-12 ed-0.2-21 gawk-3.1.0-3 grub-0.90-11 openssl-0.9.6b-8 raidtools-0.90-23 sed-3.02-10 sysklogd-1.4.1-4 dev-3.2-5 mkbootdisk-1.4.2-3 utempter-0.5.2-6 words-2-17 cyrus-sasl-1.5.24-20 kudzu-0.99.23-1 openldap-2.0.11-13 rpm-4.0.3-1.03 apmd-3.0final-34 lokkit-0.50-6 vixie-cron-3.0.1-63 ghostscript-fonts-5.50-3 libxml2-2.4.2-1 mpage-2.5.1-7 perl-HTML-Tagset-3.03-3 perl-Parse-Yapp-1.04-3 perl-XML-Encoding-1.01-2 perl-libxml-enno-1.02-5 pnm2ppa-1.04-2 alchemist-1.0.18-1 XFree86-libs-4.1.0-3Xaw3d-1.5-10 Xconfigurator-4.9.39-1 XFree86-ISO8859-15-75dpi-fonts-4.1.0-3 XFree86-xdm-4.1.0-3 esound-0.2.22-5 aspell-0.33.7-1 libogg-1.0rc2-1 awesfx-0.4.3a-8 cdparanoia-alpha9.8-2 mikmod-3.1.6-12 sox-12.17.1-4 bind-utils-9.1.3-4 gnupg-1.0.6-3 ncftp-3.0.3-6 nss_ldap-172-2 pam_krb5-1.46-1 rdate-1.0-8 rsh-0.17-5 telnet-0.17-20 wget-1.7-3 isdn4k-utils-3.1-46 ppp-2.4.1-2 fetchmail-5.9.0-1 pine-4.33-15 sysstat-4.0.1-2 cpp-2.96-98 mt-st-0.6-1 sgml-common-0.5-7 docbook-dtd41-sgml-1.0-10 docbook-utils-0.6.9-2 tamago-4.0.6-5 lsof-4.51-2 procinfo-18-2 stat-2.5-2 units-1.55-10 automake-1.4p5-2 bzip2-devel-1.0.1-4 curl-devel-7.8-1 db3-devel-3.2.9-4 e2fsprogs-devel-1.23-2 gd-devel-1.8.4-4 gpm-devel-1.19.3-20 krbafs-devel-1.0.9-2 libogg-devel-1.0rc2-1 libtiff-devel-3.5.5-13 ltrace-0.3.10-7 ncurses-devel-5.2-12 pam-devel-0.75-14 pcre-devel-3.4-2 readline-devel-4.2-2 newt-devel-0.50.33-1 VFlib2-devel-2.25.1-20 XFree86-SVGA-3.3.6-42 openssh-2.9p2-12 glibc-common-2.2.4-19.3 mrtg-2.9.6-6 iptables-1.2.4-2 mysql-devel-3.23.41-1 php-mysql-4.0.6-7 kernel-2.4.9-21 mailcap-2.1.6-1 setup-2.5.7-1 basesystem-7.0-2 bdflush-1.5-17 chkconfig-1.2.24-1 ttfonts-ja-1.0-6 fortune-mod-1.0-16 libtiff-3.5.5-13db1-1.85-7 db3-3.2.9-4 e2fsprogs-1.23-2 file-3.35-2 glib-1.2.10-5 iputils-20001110-6 losetup-2.11g-5 mingetty-0.9.4-18 net-tools-1.60-3 pcre-3.4-2 popt-1.6.3-1.03 reiserfs-utils-3.x.0j-2 shadow-utils-20000902-4 newt-0.50.33-1 ntsysv-1.2.24-1 syslinux-1.52-2 libtermcap-2.0.8-28 bzip2-1.0.1-4 hotplug-2001_04_24-11 libstdc++-2.96-98 logrotate-3.5.9-1 info-4.0b-3 diffutils-2.7.2-2 fileutils-4.1-4 findutils-4.1.7-1 grep-2.4.2-7 dhcpcd-1.3.18pl8-13 gzip-1.3-15 man-1.5i2-6 procmail-3.21-1 psmisc-20.1-2 readline-4.2-2 rootfiles-7.2-1 console-tools-19990829-36 slocate-2.6-1 tar-1.13.19-6 textutils-2.0.14-2 mount-2.11g-5 authconfig-4.1.19-1 cyrus-sasl-md5-1.5.24-20 gpm-1.19.3-20 passwd-0.64.1-7 krb5-libs-1.2.2-13 sendmail-8.11.6-3 zlib-1.1.3-24 util-linux-2.11f-9 ipchains-1.3.10-10 pciutils-2.1.8-23 timeconfig-3.2.2-1 anacron-2.3-17 freetype-2.0.3-7 gmp-3.1.1-4 libpng-1.0.12-2 libxslt-1.0.1-3 m4-1.4.1-5 nkf-1.92-6 perl-Digest-MD5-2.13-1 perl-HTML-Parser-3.25-2 perl-MIME-Base64-2.12-6 perl-Storable-0.6.11-6 perl-libwww-perl-5.53-3 perl-XML-Grove-0.46alpha-3 perl-libxml-perl-0.07-5 perl-XML-Dumper-0.4-5 foomatic-1.1-0.20010905.1 psutils-1.17-13 python-1.5.2-35 PyXML-0.6.5-4 watanabe-vf-1.0-5 VFlib2-2.25.1-20 chkfontpath-1.9.5-2 urw-fonts-2.0-12 printconf-0.3.44-1 gtk+-1.2.10-11 libjpeg-6b-16 gdk-pixbuf-0.11.0-8 ttfonts-1.0-4 XFree86-4.1.0-3xinitrc-3.20-1 xloadimage-4.1-21 arts-2.2-11 tcl-8.3.3-65 pspell-0.12.2-3 aumix-2.7-5 xinetd-2.3.3-1 libvorbis-1.0rc2-2 efax-0.9-9 cdda2wav-1.10-4 cdp-0.33-21 cdrecord-1.10-4 libao-0.8.0-1 mkisofs-1.10-4 playmidi-2.4-16 sndconfig-0.65.2-1 autofs-3.1.7-21 cipe-1.4.5-6 ftp-0.17-12 krbafs-1.0.9-2 micq-0.4.6.p1-2 nfs-utils-0.3.1-13 nscd-2.2.4-13 openldap-clients-2.0.11-13 pidentd-3.0.14-1 radvd-0.6.2pl4-1 rmt-0.4b22-6 rhn_register-2.7.2-7.x.2 rusers-0.17-12 sendmail-cf-8.11.6-3 stunnel-3.19-1 tcp_wrappers-7.6-19 traceroute-1.4a12-1 up2date-2.7.2-7.x.6 whois-1.0.9-1 yp-tools-2.5-1 lockdev-1.0.0-14 minicom-1.83.1-16 rp-pppoe-3.2-3 wvdial-1.41-15 links-0.96-2 rsh-server-0.17-5 rwall-server-0.17-10 talk-server-0.17-12 ypserv-1.3.12-2 curl-7.8-1 mm-1.1.3-1 rdist-6.1.5-16 ucd-snmp-utils-4.2.1-7 docbook-dtd30-sgml-1.0-10 docbook-dtd40-sgml-1.0-11 openjade-1.3-17 perl-SGMLSpm-1.03ii-4 sgml-tools-1.0.9-12 emacs-nox-20.7-41 arpwatch-2.1a11-9 dump-0.4b22-6 lv-4.49.4-3 pinfo-0.6.1-2 rsync-2.4.6-5 shapecfg-2.2.12-7 tree-1.2-13 xdelta-1.1.1-11 autoconf-2.13-14 binutils-2.11.90.0.8-9 byacc-1.9-19 cdecl-2.5-20 ctags-5.0.1-2 cvs-1.11.1p1-3 db1-devel-1.85-7 db3-utils-3.2.9-4 diffstat-1.28-1 esound-devel-0.2.22-5 flex-2.5.4a-15 gdb-5.0rh-15 gettext-0.10.38-7 gcc-2.96-98 gmp-devel-3.1.1-4 gsm-devel-1.0.10-3 krb5-devel-1.2.2-13 lclint-2.5q-4 lockdev-devel-1.0.0-14 man-pages-1.39-2 ncompress-4.2.4-24 njamd-0.8.1-2 openssl-devel-0.9.6b-8 patch-2.5.4-10 openssh-server-2.9p2-12 glibc-2.2.4-19.3 wu-ftpd-2.6.1-20 kernel-BOOT-2.4.9-21 kernel-headers-2.4.9-21 kudzu-devel-0.99.23-1 pmake-1.45-4 rcs-5.7-15 rpm-build-4.0.3-1.03 slang-devel-1.4.4-4 strace-4.3-2 texinfo-4.0b-3 zlib-devel-1.1.3-24 XFree86-compat-modules-3.3.6-42 apache-1.3.22-2 apache-manual-1.3.22-2 openssh-askpass-2.9p2-12 glibc-profile-2.2.4-19.3 tripwire-2.3.1-5 iptables-ipv6-1.2.4-2 mysqlclient9-3.23.22-6 mysql-server-3.23.41-1 php-manual-4.0.6-7 libpcap-0.6.2-9 modutils-2.4.10-1 indexhtml-7.2-1 filesystem-2.1.6-2 bzip2-libs-1.0.1-4 db2-2.4.14-7 eject-2.0.9-2 hdparm-4.1-2 mailx-8.1.1-22 parted-1.4.16-8 pwdb-0.61.1-3 MAKEDEV-3.2-5 cpio-2.4.2-23 at-3.1.8-20 ash-0.3.7-2 less-358-21 procps-2.0.7-11 redhat-release-7.2-1 kbdconfig-1.9.14-1 tcsh-6.10-6 mkinitrd-3.2.6-1 time-1.7-14 vim-minimal-5.8-7 pam-0.75-14 cyrus-sasl-plain-1.5.24-20 sh-utils-2.0.11-5 SysVinit-2.78-19 initscripts-6.40-1 quota-3.01pre9-3 expat-1.95.1-7 groff-perl-1.17.2-3 LPRng-3.7.4-28 perl-DateManip-5.39-5 perl-libnet-1.0703-6 perl-URI-1.12-5 perl-XML-Parser-2.30-7 perl-XML-Twig-2.02-2 a2ps-4.13b-15 4Suite-0.11-2 XFree86-xfs-4.1.0-3 ghostscript-6.51-12 lesstif-0.92.32-6 switchdesk-3.9.7-1 Mesa-3.4.2-7 XFree86-75dpi-fonts-4.1.0-3 XFree86-twm-4.1.0-3 audiofile-0.2.1-2 libtool-libs-1.4-8 portmap-4.0-38 make-3.79.1-8 cdlabelgen-1.5.0-9 gsm-1.0.10-3 mpg321-0.1.5-1 vorbis-1.0rc2-1 finger-0.17-9 slang-1.4.4-4 setuptool-1.8-2 bash-2.05-8 iproute-2.2.4-14 libjpeg-devel-6b-16 libstdc++-devel-2.96-98 libtermcap-devel-2.0.8-28 libtool-1.4-8 nmh-1.0.4-9 sharutils-4.2.1-8 finger-server-0.17-9 mkxauth-1.7-16 XFree86-100dpi-fonts-4.1.0-3 XFree86-ISO8859-15-100dpi-fonts-4.1.0-3 XFree86-tools-4.1.0-3 lilo-21.4.4-14 mouseconfig-4.23-1 tmpwatch-2.8-2 vim-common-5.8-7 which-2.12-3 cracklib-dicts-2.7-12 logwatch-2.1.1-3 nmap-2.54BETA22-3 python-xmlrpc-1.5.0-1 rpm-python-4.0.3-1.03 rwho-0.17-11 talk-0.17-12 ucd-snmp-4.2.1-7 ypbind-1.8-1 lrzsz-0.12.20-10 statserial-1.1-23 mutt-1.2.5i-17 metamail-2.7-28 rusers-server-0.17-12 telnet-server-0.17-20 gd-1.8.4-4 tcpdump-3.6.2-9 docbook-dtd31-sgml-1.0-10 docbook-style-dsssl-1.64-3 emacs-20.7-41 bc-1.06-5 pax-1.5-4 screen-3.9.9-3 symlinks-1.2-13 audiofile-devel-0.2.1-2 bison-1.28-7 cdrecord-devel-1.10-4 cyrus-sasl-devel-1.5.24-20 dev86-0.15.5-1 expat-devel-1.95.1-7 gdbm-devel-1.8.0-10 gcc-g77-2.96-98 indent-2.2.6-2 libao-devel-0.8.0-1 gcc-c++-2.96-98 libvorbis-devel-1.0rc2-2 mm-devel-1.1.3-1 openldap-devel-2.0.11-13 pciutils-devel-2.1.8-23 python-devel-1.5.2-35 rpm-devel-4.0.3-1.03 swig-1.1p5-10 glibc-devel-2.2.4-19.3 sudo-1.6.4-0.7x.2 mysql-3.23.41-1 php-4.0.6-7 snort-mysql-1.8.3-5snort kernel-doc-2.4.9-21 libpng-devel-1.0.12-2 apache-devel-1.3.22-2 openssh-clients-2.9p2-12
This looks like memory corruption in some of the kernel's internal caches (the directory entry cache in this case.) The swap map messages: Jan 25 14:28:56 netops kernel: swap_free: Bad swap offset entry ff000000 Jan 25 14:28:56 netops kernel: swap_free: Unused swap offset entry 0000ff00 also indicate that there is memory getting corrupted: in this case it's page tables. Such memory corruption can be caused by _anything_. In my own experience, swap map errors are most often associated by hardware memory (cache or dram) failures. Is this reproducible? I don't mean, does the tcp connect fail more than once after the bug occurs --- once you have got a kernel "oops" message, there may well be parts of the kernel which have died and which are simply never going to return to active service until you reboot. So, can you reproduce this after a reboot? In cases of apparently random memory corruption, it is rarely easy to identify the cause unless you can reproduce the problem. You may want to visit www.memtest86.com for a PC memory testing program.
It does continue to happen. I've noticed that during compilation of otherwise normal programs, gcc will segfault and ask me to submit a full bug report. There isn't any rhyme or reason as to why gcc segfaults though. I thought this was originally the "noathlon" thing, now I'm not so sure.
Random oopses, GCC segfaults and page table corruption taken together all point to hardware memory corruptions. GCC segfaults are one of the primary indications of such hardware problems. memtest is a good place to start looking for them, but it does not find all possible faults, so you may just need to replace hardware components until you find the real cause. Sometimes just reseating the memory is enough to cure problems like this.
Incidentally, the "noathlon" problem that has been discussed recently only affects AGP systems, and manifests itself as openGL lockups or corruption of memory actively in use by the AGP system (ie. graphics memory). It is not expected to cause instability outside of AGP applications such as the X server or openGL applications. I don't think it's the problem here.