Bug 745603 - System hangs on reboot/shutdown/Ctrl-Alt-Del when snoopy running.
Summary: System hangs on reboot/shutdown/Ctrl-Alt-Del when snoopy running.
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: snoopy
Version: 15
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Mosaab Alzoubi
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-10-12 19:51 UTC by mis
Modified: 2015-04-21 18:42 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-08-07 20:04:39 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description mis 2011-10-12 19:51:44 UTC
Description of problem:

System hangs on reboot, shutdown, or Ctrl-Alt-Del when snoopy enabled in ld.so.preload.

Version-Release number of selected component (if applicable): 

snoopy-1.7.10-2.fc15.x86_64

How reproducible:

Every install

Steps to Reproduce:
1. Install a minimal, or desktop install
2. After intial boot, do 'yum install snoopy'
3. Create ld.so.preload with 1 line containing '/lib64/snoopy.so'.
4. Check /var/log/syslog to ensure log entries exist.
5. Try executing 'reboot' or 'shutdown' commands, or do 'Ctrl-Alt-Del'.
  
Actual results:

System starts to shutdown, but then hangs. Must hold power button, or pull plug to get the system shut down or rebooted.

Expected results:

System should shutdown or reboot, depending on which command is executed.

Additional info:

I've tried this numerous times with the minimal install, and a desktop install. I've also tried with both x86 and x64.

Comment 1 Steve Traylen 2011-10-12 20:22:38 UTC
Hi,

I'll try to reproduce later. Is any other information available
such what does snoopy log in /var/log/secure when this happens?

An selinux problem is also possible, is there anything in /var/log/audit/audit.log?

Steve.

Comment 2 mis 2011-10-13 18:34:23 UTC
I need to update a point on "Steps to Reproduce:". After performing step 4, you must reboot the system, in order for snoppy to be activated at bootup. This reboot can be performed successfully. However all subsequent reboots will hang.

I've inspected the /var/log/secure and /var/log/audit/audit.log files, and have been unable to see any additional information that would highlight what is causing this issue. There definately do not appear to be any 'additional' log entries created when this issue is reproduced.

One additional piece of info though... If I let the system sit for long enough, eventually messages similar to the following appear on the console:

[828.410200] audit: *NO* daemon at audit_pid=573
[828.410276] audit: auditd disappeared
[828.410278]

Comment 3 Steve Traylen 2011-10-13 19:30:59 UTC
okay I'll try and reproduce this myself on f15. 

I use this configuration only on RHEL5 and 6 where I don't see this problem.

Steve.

Comment 4 Steve Traylen 2011-10-29 19:46:45 UTC
Can't reproduce in a nutshell.

Please try entering instead of 

/lib64/snoopy.so

in /etc/ld.so.preload

enter
/$LIB/snoopy.so

instead.. 

In fact regardless of anything this should be in the README anyway.
/lib64 is definitely wrong for 32 bit systems.

Steve.

Comment 5 mis 2011-10-31 19:58:05 UTC
Yes, clearly the x86 README.fedora is showing '/lib64/snoopy.so', and this is wrong, but that's not my issue...

Perhaps another point worth mentioning is that I'm using 'biosdevname=0' on the boot command line when I install.

Steps to Reproduce *** x64 ***:

1. Perform a 'minimal' install
2. After intial boot, do 'yum install snoopy'
3. Create ld.so.preload with 1 line containing '/lib64/snoopy.so'.
4. Check /var/log/syslog to ensure log entries exist.
5. Reboot system, in order for snoopy to be loaded on system startup.
6. Log in, and try executing 'reboot' or 'shutdown' commands, or do 'Ctrl-Alt-Del'.

Steps to Reproduce *** x86 ***:

1. Perform a 'minimal' install
2. After intial boot, do 'yum install snoopy'
3. Create ld.so.preload with 1 line containing '/lib/snoopy.so'.
4. Check /var/log/syslog to ensure log entries exist.
5. Reboot system, in order for snoopy to be loaded on system startup.
6. Log in, and try executing 'reboot' or 'shutdown' commands, or do 'Ctrl-Alt-Del'.

Actual results:

System starts to shutdown, but then hangs. Must hold power button, or pull plug
to get the system shut down or rebooted.

Expected results:

System should shutdown or reboot, depending on which command is executed.

Comment 6 mis 2011-11-01 21:06:25 UTC
After a full day of work on this, and numerous installs (both from DVD and netinstall media), and trying several different Computers, we believe that we've discovered something valuable to add to the requirements to reproduce this issue.

If we disable Hyper-Threading, we can reboot successfully. As soon as we turn Hyper-Threading back on, the issue reappears.

The problem now is that we have not determined how to disable Hyper-Threading on our Dual Core Servers.

The long and short is that this issue needs to be fixed, as I don't want to have to disable Hyper-Threading (if I even can) on these servers.

Hopefully this will help point you in the right direction.

Comment 7 Steve Traylen 2011-11-01 22:03:35 UTC
I completely missed this rather important line last time when I checked:
>5. Reboot system, in order for snoopy to be loaded on system startup.

Will try again shortly.

Comment 8 mis 2011-11-11 20:56:45 UTC
Have you been able to reproduce this yet?

Any updates?

Comment 9 Steve Traylen 2011-11-11 21:41:06 UTC
No,
I only have fedora running on VMs and no real hardware.

Opening a bug on the sourceforge project makes sense.

Steve.

Comment 10 mis 2011-11-14 18:56:07 UTC
So are you going to open a bug report with the sourceforge project?

Comment 11 Steve Traylen 2011-11-14 19:26:49 UTC
It's probably better if you do given I am incapable of reproducing this.

Steve.

Comment 12 mis 2011-11-14 20:29:43 UTC
Any likelyhood that version 1.8.0 might resolve? Any status on when the new version will be available as an update?

Comment 13 Steve Traylen 2011-11-14 22:05:27 UTC
fedora16 is 1.8.0 already, for fedora15, I've put some packages here:

http://koji.fedoraproject.org/koji/taskinfo?taskID=3514226

the changelog has nothing obvious.

2011-03-06 - Version  1.8.0
---------------------------
o Feature: syslog facility is now configurable
o Feature: syslog level is now configurable
o Feature: external filter support available
o Feature: single path prefix filtering available

if they do make a difference I'll release of course.

Steve

Comment 14 Bostjan Skufca 2011-11-14 22:38:49 UTC
Hi,

(snoopy maintainer here)

Has anyone else been able to reproduce this issue except mis?

b.

Comment 15 mis 2011-11-24 14:05:24 UTC
Bostjan:

Are you unable to reproduce? I've tried multiple systems/servers, and I have yet to find a system/server where I cannot reproduce.

Comment 16 Bostjan Skufca 2011-11-24 14:19:59 UTC
Not a fedora user nor I have any system at hand, therefore I am asking for additional information about this issue. Any specific process that is stuck at the end of halt process?

Comment 17 mis 2011-11-24 15:37:22 UTC
Unfortunately, I can't perform any interactive debugging after I initiate a reboot/halt, as the system just hangs. I can't log in, or get any interactive session to respond. Just hung...

Sorry...

Let me know if I can provide anything like logs, etc.

Comment 18 George B. Magklaras 2012-04-26 15:23:09 UTC
I can confirm that I can reproduce this in Fedora 15. I am the author of LUARM (http://luarm.sourceforge.net) which uses snoopy as a key component. Here are the steps I perform to reproduce the problem of snoopy-1.7.10 halting the boot of an up-to-date F15 machine. I reproduced the steps successfully on a physical Dell Precision 490 and also on kvm powered x86_64 vm hosted on an up-to-date Fedora 16 x86_64 machine:

1)Boot the system normally.
2)LD_PRELOAD the compiled snoopy.so lib.
3)Attempt to reboot the system, while snoopy is preloaded (active). 

The result is that systemd[1] segfaults from the very early stages of boot. The only way to recover the system is to boot in Rescue mode, remove the preloading of snoopy.so and then all is normal again. 

It's hitting me pretty hard. I do not see the issue in other distros (Debian on 2.6.32-5-amd64). I am in the process of testing whether it occurs with snoopy-1.8.0 and/or other combinations (1.7.0 against RHEL 6).

George Magklaras
http://folk.uio.no/georgios

Comment 19 Steve Traylen 2012-04-26 16:21:37 UTC
Maybe not as a permanent solution but recompiling with the filter option and this filtering out systemd might be interesting.

Comment 20 Bostjan Skufca 2012-04-27 01:22:37 UTC
Just tried with fedora 16 x64 in VM:
- hangs if I click "shutdown" in X environment
- DOES NOT HANG if I open console and enter "shutdown -P now"

About external filters:
PLEASE PLEASE PLEASE do not use external filtering, it's just too dangerous and I've hanged a couple of machines successfully with recursive logging.

I'm planning on updating snoopy with framework for compiled-in filters, which one will be able to choose at compile time (with ./configure option), but unfortunately I am running out of time in general.

@George: Nice work!
But: it hangs for you while starting up? I thought this issue only applies for shutdown/reboot situation.

Comment 21 George B. Magklaras 2012-04-27 10:38:21 UTC
OK to confirm my testing results, I standby my original steps:

1)Boot the system normally.
2)LD_PRELOAD the compiled snoopy.so lib.
3)Attempt to reboot the system, while snoopy is preloaded (active) (*either with shutdown -r now or from the GUI*)

I have now tested with:
i)RHEL 6
ii)Fedora 15
iii)Debian 6.0 

and the two most recent versions of snoopy (1.7.10 and 1.8.0).

and they all reproduce the problem *when snoopy is preloaded* and the system is shutdown/rebooted. 

a)The RHEL 6 does not even shutdown properly:
Screenshot: 
ftp://ftp.no.embnet.org/snoopybug/RHEL6restartfromGUI.png

b)Fedora 15 shuts down properly-ish (I can't see messages indicating a problem during the shutdown phase) but then on the way up:
Screenshot: 
ftp://ftp.no.embnet.org/snoopybug/SNOOPYBUG1.png

c)Debian 6 will not shutdown properly and on the way up it segfaults:
Screenshot:
ftp://ftp.no.embnet.org/snoopybug/debian6startup.png

Hence, my conclusion is that the problem is not specific to the Fedora distro and should be taken with the snoopy logger developer further. 

Note: The software I wrote uses a modified version of snoopy (taking away the filtering functionality and writing to a specific file, not via the syslog channel:
http://luarm.svn.sourceforge.net/viewvc/luarm/snoopy-luarm/snoopy.c?revision=187&view=markup

). I do not think that this is what causing the problem, since I see that people that use the clean version of snoopy are facing problems, but I thought I should mention it anyway. 

I plan to forward the issue to the snoopy logger developers.

George Magklaras
http://folk.uio.no/georgios

Comment 22 Bostjan Skufca 2012-04-27 12:26:03 UTC
George, thanks for the extensive report.

It seems that there is a general problem with systemd and snoopy combination. What exactly, I have no clue.

"I plan to forward the issue to the snoopy logger developers."
I am currently the only one, as snoopy was abandoned years ago I when I found it, it seemed nice and not too demanding to get it working in my environment (slackware). I asked original developers if I can take over and that was it.

About the issue resolution:
- do you have any idea how to exclude systemd from preloading snoopy?

b.

Comment 23 George B. Magklaras 2012-04-27 23:52:15 UTC
(In reply to comment #22)
> George, thanks for the extensive report.
> 
> It seems that there is a general problem with systemd and snoopy combination.
> What exactly, I have no clue.
> 
> "I plan to forward the issue to the snoopy logger developers."
> I am currently the only one, as snoopy was abandoned years ago I when I found
> it, it seemed nice and not too demanding to get it working in my environment
> (slackware). I asked original developers if I can take over and that was it.
> 
> About the issue resolution:
> - do you have any idea how to exclude systemd from preloading snoopy?
> 
> b.

Hi,

I think the issue is more fundamental than systemd and concerns the initial init. The systemd interface is not present in RHEL6 and earlier RHEL versions that use the "traditional" System V way of starting up things, yet the problem is also there.

GM

Comment 24 George B. Magklaras 2012-04-28 12:49:04 UTC
> The systemd interface is not present in RHEL6 and earlier RHEL versions
> that use the "traditional" System V way of starting up things, yet the problem
> is also there.
> 

I can also confirm that I consistently re-produced the problem with RHEL 5.8. So, it seems that init does not like the pre-loading of snoopy for some reason, which is why the LUARM scripts remove the ld.so.preload file every time I stop the monitoring clients. I could also fix the shutdown scripts to remove the preload file to prevent the problem, but that still leaves the possibility of  a system crash/improper shutdown, leaving the ld.so.preload with snoopy in place...

Bostjan, can you address this problem. Steve, are you still active into looking the problem? The problem has been consistently reproduced now, so I need help to understand why this happens.

Cheers,
GM

Comment 25 Steve Traylen 2012-04-28 17:55:26 UTC
> I can also confirm that I consistently re-produced the problem with RHEL 5.8.

You think this is new property 5.8?

We run 10000s of machines with will have gone recently to 5.8, .. not heard anything bad yet
but will try and reproduce again.

In all the above comments you don't mention what you have /etc/ld.so.preload?

Comment 26 George B. Magklaras 2012-04-28 18:13:25 UTC
(In reply to comment #25)

> In all the above comments you don't mention what you have /etc/ld.so.preload?

One line:
/usr/local/lib/snoopy.so

Comment 27 Steve Traylen 2012-04-29 07:04:14 UTC
(In reply to comment #26)
> (In reply to comment #25)
> > In all the above comments you don't mention what you have /etc/ld.so.preload?
> 
> One line:
> /usr/local/lib/snoopy.so

Is that correct? You are working on a 32 bit machine?, why the odd location also.

For comparison we have

$ cat /etc/ld.so.preload 
/$LIB/snoopy.so

So why are you in local? I am also wandering if this is due to a particular partition
layout?

We have / and /usr/ all in the one partition.

 Steve.

Comment 28 George B. Magklaras 2012-04-29 12:30:14 UTC
(In reply to comment #27)
> $ cat /etc/ld.so.preload 
> /$LIB/snoopy.so
> 
> So why are you in local? I am also wandering if this is due to a particular
> partition
> layout?
> 
> We have / and /usr/ all in the one partition.
> 
>  Steve.

I am installing from sources. (In comment 21, I did provide a note that I am using a modified version of the sources, as I need to bypass the syslog path and write to a specific file ). 

Yes, obviously you need to be on the root partition, as you need to correctly preload the shared lib early in the boot process and I have produced versions where you dump it to the /lib64 dir, as done with the .spec file. This is something that Bostjan needs to explain in the README file, to prevent situations where snoopy.so is preloaded. 

However, even when I dumped snoopy.so under /lib64 and preloaded from that path, the problem persisted. Right now, I think I am getting to the bottom of this and conclude that the problem occurs due to the LUARM specific things I was doing. I should be able to update you folks by Monday, Tuesday. 

GM

Comment 29 Fedora End Of Life 2012-08-07 20:04:41 UTC
This message is a notice that Fedora 15 is now at end of life. Fedora
has stopped maintaining and issuing updates for Fedora 15. It is
Fedora's policy to close all bug reports from releases that are no
longer maintained. At this time, all open bugs with a Fedora 'version'
of '15' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we were unable to fix it before Fedora 15 reached end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora, you are encouraged to click on
"Clone This Bug" (top right of this page) and open it against that
version of Fedora.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 30 Bostjan Skufca 2013-04-04 17:27:34 UTC
I think we have found a resolution. New bug report (#948417) has been opened for that:
https://bugzilla.redhat.com/show_bug.cgi?id=948417

Comment 31 Fedora Update System 2015-03-22 16:13:29 UTC
snoopy-2.2.6-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/snoopy-2.2.6-1.fc20

Comment 32 Fedora Update System 2015-03-22 16:18:49 UTC
snoopy-2.2.6-1.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/snoopy-2.2.6-1.fc22

Comment 33 Fedora Update System 2015-03-22 17:32:19 UTC
snoopy-2.2.6-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/snoopy-2.2.6-1.fc21

Comment 34 Fedora Update System 2015-03-22 17:43:22 UTC
snoopy-2.2.6-1.el7 has been submitted as an update for Fedora EPEL 7.
https://admin.fedoraproject.org/updates/snoopy-2.2.6-1.el7

Comment 35 Fedora Update System 2015-03-31 21:51:37 UTC
snoopy-2.2.6-1.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 36 Fedora Update System 2015-03-31 22:00:24 UTC
snoopy-2.2.6-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 37 Fedora Update System 2015-04-21 18:42:01 UTC
snoopy-2.2.6-1.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.