Bug 435637

Summary: udev delays for long periods of time
Product: [Fedora] Fedora Reporter: Josh Boyer <jwboyer>
Component: udevAssignee: Harald Hoyer <harald>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: low Docs Contact:
Priority: low    
Version: rawhideCC: michal, notting
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-03-31 14:15:17 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On:    
Bug Blocks: 235706    
Attachments:
Description Flags
photo of screen after hitting ctrl+alt+del
none
screen a bit after none

Description Josh Boyer 2008-03-02 14:33:16 EST
Description of problem:

When adding 'single' to the kernel command line, upstart doesn't appear to
respond.  The last message that is seen is the udev startup, and then it sits
there doing nothing.

Version-Release number of selected component (if applicable):

upstart-0.3.9-5.fc9.i386

How reproducible:

Always

Steps to Reproduce:
1.  Add 'single' to the kernel command line
2.  Let it boot
3.  Watch it do nothing after udev runs
  
Actual results:

fail

Expected results:

work

Additional info:

Playing around with it a bit, I found that if you hit ctrl+alt+del while it's at
the "hung" state, it will send signals to processes and then drop you into the
shell prompt that you expected it to put you in to begin with.  That's cute.

Doing ctrl+alt+del again does the regular reboot.
Comment 1 Casey Dahlin 2008-03-02 14:47:35 EST
hmm. WFM.

What do you have in event.d that's pertinent? Maybe I fiddled with something on
my install and forgot about it.
Comment 2 Josh Boyer 2008-03-03 09:53:03 EST
Sorry, not quite sure what you're asking here.  /etc/event.d/rc1 simply has:

start on runlevel 1

stop on runlevel

console output
script
        set $(runlevel --set 1 || true)
        if [ "$1" != "unknown" ]; then
            PREVLEVEL=$1
            RUNLEVEL=$2
            export PREVLEVEL RUNLEVEL
        fi

        exec /etc/rc.d/rc 1
end script


I will note that upstart got installed as a dependency for
event-compat-sysv-0.3.9-8.fc9.noarch which got pulled in by something else.  So
I didn't explicitly install it, if that makes a difference.
Comment 3 Bill Nottingham 2008-03-03 11:53:39 EST
What happens if you boot with 'udevinfo' or 'udevtrace'?
Comment 4 Josh Boyer 2008-03-03 23:00:02 EST
(In reply to comment #3)
> What happens if you boot with 'udevinfo' or 'udevtrace'?

Odd.  Adding udevinfo produces a massive amount of spewage from udev, then it
goes to the prompt as expected.  

[2 minutes]

udevtrace doesn't really produce any output, but it still goes to the prompt as
expected.

[2 minutes]

Just having 'single' there without the udev* additions and it simply sits.
Comment 5 Josh Boyer 2008-03-03 23:02:59 EST
(In reply to comment #4)
> Just having 'single' there without the udev* additions and it simply sits.

After letting it sit a bit longer, eventually it says:

Wait timeout.  Will continue in the backgroun[FAILED]

and doesn't go to the prompt.
Comment 6 Josh Boyer 2008-03-03 23:07:35 EST
(In reply to comment #5)
> (In reply to comment #4)
> > Just having 'single' there without the udev* additions and it simply sits.
> 
> After letting it sit a bit longer, eventually it says:
> 
> Wait timeout.  Will continue in the backgroun[FAILED]
> 
> and doesn't go to the prompt.

And now it just did that without 'single' at all.  udev is messing with my mind.

Comment 7 Bill Nottingham 2008-03-05 22:44:10 EST
This appears to be more a udev issue that an upstart issue, but I could be wrong...
Comment 8 Josh Boyer 2008-03-05 22:57:03 EST
I agree.  I'll try again with tomorrow's rawhide and see what happens.
Comment 9 Casey Dahlin 2008-03-10 22:59:57 EDT
Any news on this?
Comment 10 Josh Boyer 2008-03-10 23:38:20 EDT
(In reply to comment #9)
> Any news on this?

I doubt this is an upstart issue.  It's already been moved over to udev.

As for new news, nothing much.  udev still intermittently decided to go to lunch
on my machine and I've not had time to play with it more.
Comment 11 Josh Boyer 2008-03-11 08:01:19 EDT
Created attachment 297599 [details]
photo of screen after hitting ctrl+alt+del

I passed 'single udevinfo' on the command line this morning and got the udev
hang.  Then I hit ctrl+alt+del and took a picture.  This is the screen right
after I did the ctrl+alt+del
Comment 12 Josh Boyer 2008-03-11 08:02:49 EDT
Created attachment 297600 [details]
screen a bit after

The screen a bit later on
Comment 13 Bill Nottingham 2008-03-11 11:18:11 EDT
The ctrl-alt-del thing is logged as 435577.
Comment 14 Harald Hoyer 2008-03-12 14:55:11 EDT
try "udevdebug" instead of "udevinfo"
Comment 15 Josh Boyer 2008-03-12 17:00:23 EDT
(In reply to comment #14)
> try "udevdebug" instead of "udevinfo"

http://picasaweb.google.com/jwboyer/Udev

Pretty pictures from the places udev hung when trying with udevdebug instead.  I
apologize in advance for the crappy quality.  If you can't get to them for some
reason, let me know and I'll just attach them here.

Those were the only places where a noticeable delay of any duration was to be
seen.  The rest of the time udevdebug seemed to spew a bunch of output that
scrolled too fast off of the screen.

Let me know if there's more I can do.
Comment 16 Josh Boyer 2008-03-12 17:05:05 EDT
Update the title of the bug to reflect reality at this point.  The delays have
happened regardless of the 'single' being present or not.
Comment 17 Harald Hoyer 2008-03-12 17:49:33 EDT
seems like problems with scsi and/or usb

which devices do you have?
Comment 18 Josh Boyer 2008-03-12 18:32:26 EDT
Stock Thinkpad z60m.  I played around a bit with power on/off vs. reboot, etc. 
Nothing to really notice there.  Occasionally it won't hang, but it does for the
most part.

Then I played around with having a USB key or SD card in the built-in reader
during booting.  Again, nothing really noticeable with that.

I'm waiting for it to decide to boot so I can give you a smolts.org URL.
Comment 20 Harald Hoyer 2008-03-13 12:50:53 EDT
udev-118-9.fc9 now has support for a "modprobedebug" kernel command line, which
may identify a kernel module causing the delay.

you also may try the new "udevnopersist" kernel command line.
Comment 22 Harald Hoyer 2008-03-31 11:26:21 EDT
any news on this?
Comment 23 Josh Boyer 2008-03-31 12:19:02 EDT
(In reply to comment #22)
> any news on this?

Sorry.  It's magically gone away with either the newer udev or a newer kernel. 
I tried for an afternoon last week to get it to hang through various
reboot/power-off,power-on combinations and it doesn't happen any longer.

Bill, have you had any kind of recreates?
Comment 24 Bill Nottingham 2008-03-31 12:58:18 EDT
No, the virt image where I saw it seems to work now. 
Comment 25 Josh Boyer 2008-03-31 13:58:09 EDT
I think we can either close this as WORKSFORME or RAWHIDE.  If it happens again,
I'll try to capture debug output with the new options and reopen.
Comment 26 Michal Jaegermann 2008-03-31 16:14:04 EDT
Hm, I was just caught by a multiminute udev delay with
kernel-2.6.25-0.172.rc7.git4.fc9.x86_64 and udev-118-11.fc9.x86_64.
It eventually decided to continue when I was practically ready to hit
a power button.

I tried to reproduce that with "modprobedebug" but without any
success.  One thing is that was "not in a mood" and I did not get
those delays.  The other is that "modprobedebug" produces only
the following on my screen:

Starting udev:
/sbin/start_udev: line 278: find: command not found
Triggering Rest

Line 278 is a closing "fi" of an an "if" starting on line 224.
The problem is really here:

    259                 for i in $(find /sys -name modalias); do
    260                     modules=$(cat $i)
    261                     modprobe -a -v -q $modules
    262                     wait_for_queue $udevtimeout
    263                 done
    264                 echo "Triggering Rest"

'find' is /usr/bin/find and /usr may be not available at all at
this point (as is the case on my system).  I guess that /bin/awk
is already present and "for" could loop through a list produced
in awk with a help of 'stat' and 'chdir' functions.  Not even that
difficult (although not worth to bother if it is not going to
be used).

Do you still want this closed?  It does not look that I can reproduce
the issue on demand.