> - Disk identifier: 0x00000000 => Is this normal ?
> - Disk /dev/mapper/vg_epf-lv_root doesn't contain a valid partition table => Is
> this normal ?

Yes, both are normal. The disk identifier and the partition table are stored in the Master Boot Record. Logical volumes do not usually contain a MBR.

Have you tried waiting for about 5 minutes? There's a chance that whatever is hanging may timeout and the boot would continue.

Does it react to CTRL+ALT+DEL when it's apparently frozen?

Yes i have.

CTRL+ALT+DEL is frozen
CAPS LOCK Also
like dead .. i can no more switch screen with CTRL+Shift...

Add "sysrq_always_enabled" to the kernel boot parameters and see if the system reboots when the Alt+SysRq+B combo is pressed.

Well as soon as i get back (probably in a few hours) i will proceed to your test. But imho the system is frozen (else the CAPS LOCK key light would be responsive).

What else should i do provide you more informations and advance with diagnostic ? Did you ever experiment this ? What do you think could be the problem ?

first thing: remove "rhgb quiet" from the kernel command line to see more messages

Replaced "rhgb quiet" with rq_always_enabled
=> got all the messages
boot freezes after line :
"(C0) PCI Express found at mem dc000000, IRQ 17, node addr 00:10:18:a6:aa:d2"

CTRL+ALT+DEL has no effect as expected.

After this i made it again with the same parameters replaced => This time the system accepted to boot.

Have you got an idea of what is happening ?
Do you need any log file ?

Would it be possible to upgrade severity / priority ?

I have a similar system at customer site under Fedora Core 13 with similar hardware (PowerEdge T110 + RD1000 backup) which sometimes would not reboot, never found why but it might be the same issue.

(In reply to comment #6)
> Replaced "rhgb quiet" with rq_always_enabled

It's 'sysrq_always_enabled'. I don't know if you just mistyped it here in the bugzilla comment or in the actually used command line too.

> CTRL+ALT+DEL has no effect as expected.

And Alt+SysRq+B ?

> boot freezes after line :
> "(C0) PCI Express found at mem dc000000, IRQ 17, node addr 00:10:18:a6:aa:d2"

If you repeat the experiment and manage to reproduce the hang, will the hang happen at this same line again?
Could you take a picture of it so we can see more of the preceding lines?

Anyway, this looks more like a kernel/hw issue. Reassigning.

yes sorry i misstyped it in the post only.

Alt+SysRq+B => I will try when i get back to the server in exactly 4hours.

Same line yes as i did it twice but i will verify it again and also take a picture.

Is there anything else i could do to provide you some more details / logs / debug ?

(In reply to comment #10)
> Is there anything else i could do to provide you some more details / logs /
> debug ?

After a successful boot, get the output of
 lspci -nnvvv
 dmesg

Created attachment 510887 [details]
lspci -nnvvv   [when boot works]

Created attachment 510889 [details]
dmesg     [ When boot works ]

I made it again , this time frozen just after :

14.928582 bnx2 0000:02:00.0: eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem da000000, IRQ 16, node addr 00:10:18:a6:aa:d0
14.929113 bnx2 0000:02:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
******** Frozen Here ********
******** Alt+SysRq+B IS UNRESPONSIVE

Comparing to the working dmesg it is just before setting latency timer to 64
and before eth2 detection (eth1+eth2) is the dual broadcom card.

But this is also just before systemd-fsck on /dev/sdb1 ..
and still believe the problem might be arround here on disks...

At start when i used only the ESC key during the boot sequence it was hanging after displaying message : "Starting File System Check on
/dev/disk/by-uuid/f816092f-c39c-49c2-b0b1-f09f72b1b930"

Thanx for help.

Finally i removed the dual NIC card and booted kernel with rq_always_enabled and without rhgb quiet and it went frozen @ Starting File System Check on
/dev/disk/by-uuid/f816092f-c39c-49c2-b0b1-f09f72b1b930. Please find attached screenshot "Boot-Still-Crashes-Without-2nd_NIC.jpg". Hope this can help.
Could you please upgrade severity ?

Kind Regards
Philippe.

Created attachment 511049 [details]
Boot-Still-Crashes-Without-2nd_NIC

You could try blacklisting some modules to see if one of them is responsible
for the hang.
The NMI watchdog might help point out the fault:
http://www.mjmwired.net/kernel/Documentation/nmi_watchdog.txt

For other debugging tips see:

https://fedoraproject.org/wiki/Common_kernel_problems#Diagnosing_.22My_machine_locked_up.22

https://fedoraproject.org/wiki/Common_kernel_problems#Crashes.2FHangs

Replacing rhgb quiet with nmi_watchdog=1 || nmi_watchdog=2 , the problem remains the same and kernel steel freezes :(

Do you have an idea of what happens regarding the whole thread ?
As this would occur for anyone using this Dell Box same config and default install ..

I am really stuck & disappointed at this point. Please tell me precisely what i need to do / what could help to solve this ?

If this can help i can use a serial cable to a second box for capturing kernel messages. Will we see more messages than we already have ?

Regards
Philippe

(In reply to comment #18)
> Replacing rhgb quiet with nmi_watchdog=1 || nmi_watchdog=2 , the problem
> remains the same and kernel steel freezes :(

Just a clarification: The NMI watchdog is not supposed to fix anything. Its purpose is merely to produce a kernel stack trace when there's a hang with
IRQs disabled.

> Do you have an idea of what happens regarding the whole thread ?

Not yet.

> As this would occur for anyone using this Dell Box same config and default
> install ..

We don't know if that's the case.

> I am really stuck & disappointed at this point. Please tell me precisely what
> i need to do / what could help to solve this ?
> 
> If this can help i can use a serial cable to a second box for capturing kernel
> messages. Will we see more messages than we already have ?

There is a chance there might be something interesting logged to the serial console, but I cannot promise you that.

I suggest you to try a few things before you get the other box:
 - Blacklisting of modules non-essential for bootup (I can see bnx2, tg3,
   dcdbas, ...). To get a list of modules use 'lsmod' when you get
   a successful boot. To blacklist modules, write their names on separate lines
   in /etc/modprobe.d/local.conf:  blacklist <modulename>
   The idea is to narrow the problem down to specific module. If you can get a
   reliable boot after the blacklisting, try to modprobe the modules back
   manually and see if any of them causes the lockup.
   Alternatively you can boot with "init=/bin/sh" (hopefully at least such a
   minimal system will start reliably) and then try loading various modules
   with modprobe to see if any of the causes the lockup.
 - Some of the tips from the links in comment #17 (at the very least try
   acpi=off, nolapic, noapic)

With Nmi watchdog i expected the NMI handler generates an oops and kills the process. Have a 'controlled crash' (and the resulting kernel messages) like mentionned in the link Documentation ; and this was not the case ; kernel steel crashed / get unresponsive / without any new message. It was just feedback on this.
Still talking on NMI , where/how should i get the kernel stack trace you talk about ?

I will try blacklisting bnx2, tg3, dcdbas and give you feedback on this.

If the problem has something to do with mpt2sas , how should i deal with this ad it will not be possible to exclude it ?

(In reply to comment #20)
> Still talking on NMI , where/how should i get the kernel stack trace you talk
> about ?

It would be printed on the console. Since it isn't, the system is apparently even deader than the NMI watchdog can handle.

> I will try blacklisting bnx2, tg3, dcdbas and give you feedback on this.
> 
> If the problem has something to do with mpt2sas , how should i deal with this
> ad it will not be possible to exclude it ?

Is there no possibility to connect only a SATA disk to the AHCI controller and setup a test system on that?

After removing ipv6, bnx2, tg3, dcdbas the system seems to boot always.
Now trying to narrow excluded modules ..

Short resume of tests :
- Boot will NEVER freeze when blacklisting only (bnx2+tg3)
- Boot will often freeze when blacklisting only tg3
- Boot will often freeze when blacklisting only bnx2
- Keep in mind boot was still crashing without dual NIC (bnx2) : see Comment 15
- Tried booting with acpi=off noapic nolapic , had no positive effect

- Attached Com DebugLog Results :
SerialLog-BootStuckWithOnlyBnx2Enabled.txt
SerialLog-BootStuckWithOnlyTg3Enabled.txt

Created attachment 511231 [details]
BootStuckWithOnlyBnx2Enabled

Created attachment 511232 [details]
ServeurTest\SerialLog-BootStuckWithOnlyTg3Enabled

In addition probing modules after boot never lead to freeze.

i tryed disabling embedded NIC in the BIOS and boot with modules not blacklisted
=> No Effect

Also tryed set all IRQ to value "default" (which means they will be choosen by BIOS) instead of the fixed value the system came with.
=> No effect

In a few days/week i should have been delivering the working system to customer.

As i am not really used to bugzilla , i wonder how long it might take to solve this issue ? Even if it is difficult , could you give me an idea on schedule and how things will occur now ?

How long it might take if something had to be fixed in the kernel. (which i hope will not be necessary)

Regards

This morning my other customer using almost the same PE110 server confirmed the server was often not rebooting properly.
This server uses only tg3 module as it has 2 Broadcom BCM5721 & BCM5722 NIC.

Moreover the server OS (Fedora 13-kernel 2.6.34.8-68) sometimes hangs after 3 weeks in production.
The hardware has already been verified by Dell & Diagnostic tools.

Just in case it could help.
I am using many Fedora on other Dell servers ie PowerEdge R710 / PE 2950 / PE 2850 without any problem untill now.

I think we are in the same issue as :
https://bugzilla.redhat.com/show_bug.cgi?id=552288

Do you have an idea on why am i still having boot crash with BootStuckWithOnlyBnx2Enabled.txt and even with onboard NIC disabled in Bios ?

Sorry my fault, not the same issue for bug 552288, only tg3 related.

(In reply to comment #23)
> [   15.790196] bnx2 0000:02:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
> [   15.820592] bnx2 0000:02:0

Hmm, a hang in the middle of a printk, that's an interesting moment.

Can you also try blacklisting only dcdbas? The module can produce System Management Interrupts. They are handled fully by the BIOS, out the operating system's control. There may be a bug in the BIOS showing itself depending on the timing and other operations going on in the system.

(In reply to comment #26)
> In a few days/week i should have been delivering the working system to
> customer.
> 
> As i am not really used to bugzilla , i wonder how long it might take to solve
> this issue ? Even if it is difficult , could you give me an idea on schedule
> and how things will occur now ?

Sorry, I cannot give you any schedule. Fedora is a volunteer community-supported distribution. No warranty, no promises, no service agreements.
For users needing commercial support there's Red Hat Enterprise Linux.

> Hmm, a hang in the middle of a printk, that's an interesting moment
I did that twice. The second time , the second line was complete. Might be the whole message had no time to be completly sent to the port.

> Can you also try blacklisting only dcdbas?
I'll do that.
Also i am trying to build module tg3 from broadcom sources tg3-3.116j.tar.gz
As i don't know really what to do.
Anything more to suggest i should work on / try ?


> Sorry, I cannot give you any schedule ..
Sure i know those rules , i'm also trying to contribute by this thread and using fedora on production environnement.
I am just asking , regarding your experience if you think we will be able to solve it in let's say a week or two, or if you think i should switch to another OS. Which i would hate.

- Re-Enabled integrated NIC into the BIOS
- upgraded tg3 driver from 3.116 to 3.116j
- blacklisted only dcdbas

=> 1st reboot OK
=> 2nd reboot KO after IRQ 17 as before

tryed again with only both blacklist dcdbas && blacklist tg3
the system booted once and then the 5/6 next times it would always freeze.
I used rescue to set back blacklist on tg3 && bnx2.

Tests done after server boot with blacklist on both module for tg3 && bnx2 :

-  running command "rmmod dcdbas ; rmmod bnx2 ; rmmod tg3; modprobe dcdbas ; modprobe bnx2 ; modprobe tg3"
will not fail. (done 50 times less than 1 second before next launch)

- whereas running command "rmmod bnx2 ; rmmod tg3; modprobe bnx2 ; modprobe tg3"
crashed after 3 or 4 attempts

- did "rmmod dcdbas" and re-run command "rmmod bnx2 ; rmmod tg3; modprobe bnx2 ; modprobe tg3"
crashed after more than 40 attempts (less than 1 second before next launch)

- just to verify it again i removed blacklist for tg3 && tg2 and put only blacklist dcdbas
=> System freeze at startup as in the past
 => reverted back to blacklist on both module for tg3 && bnx2

- running command "rmmod dcdbas ; rmmod bnx2 ; rmmod tg3; modprobe dcdbas ; modprobe tg3 ; modprobe bnx2"
will not fail. (done 50 times less than 1 second before next launch)
=> it seems that the module load order has no effect

- after this tried running command "rmmod bnx2 ; rmmod tg3; modprobe tg3 ; modprobe bnx2"     (with dcdbas loaded)
would not fail for 50 attempts

=> did "rmmod dcdbas" and re-run command  "rmmod bnx2 ; rmmod tg3; modprobe tg3 ; modprobe bnx2"
crashed after 6 attempts


I am not sure if we can really conclude :
- it is obvious that things get better when dcdbas is probed before. Without it the freeze is happening fast.
- finally it seems that loading order of tg3 & bnx2 is not that important

Hope this will help.
Do you need something more ?

http://kyle.fedorapeople.org/kernel-2.6.40-0.rc5.git0.1.fc16.x86_64.rpm

Try booting this kernel and see if the problem persists in the latest upstream?

Hi kyle , thanx for your interest.

I installed 2.6.40 kernel and booted with modules dcdbas, tg3 & bnx2 disabled.

- On the first call of "rmmod dcdbas ; rmmod bnx2 ; rmmod tg3; modprobe dcdbas ;
modprobe tg3 ; modprobe bnx2" the OS got frozen immediately same behaviour.

- And also "rmmod dcdbas ; rmmod bnx2 ; rmmod tg3; modprobe dcdbas ;
modprobe bnx2 ; modprobe tg3" will not fail except if run continuously (without a minimum pause) after something like 30 attempts

Did you try putting "sleep 1" after loading each module?

Yes i tried and it was working better with specific order and sleep 5

Just one shot in the dark: does adding "biosdevname=0" to the kernel command line help?

I found a PowerEdge T110 in our test lab, provisioned it with Fedora 15. Rebooted it three times and ran hundreds of iterations of
 rmmod bnx2; rmmod tg3; modprobe bnx2; modprobe tg3
and
 rmmod bnx2; rmmod tg3; modprobe tg3; modprobe bnx2
with and without dcdbas loaded. There was no hang.

There are some HW differences compared to Philippe's configuration:
 - a different SAS controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS [1000:0058], mptsas driver (not mpt2sas)
 - a few extra NICs (Intel, igb)

- Finally i switched to Centos 6 , i couldnt not get OpenManage to detect the H200 SAS controller whereas all the services were running fine. The freeze problem is gone uneder Centos , if you need some log to compare tell me. 

- FYI i installed the system at least 5 times from different manners (From DVDROM and From kickstart file + inet download) and everytime i had the problem, whatever the manner was. (i believe its specific to H200+both NICs)
and i still do not understand why disabling the embedded tg3 NIC in the bios had no effect !

- As i mentionned early i have another of this PE100 which look like yours :
01:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)
with mptsas , it has not the freeze on boot problem. But the system freeze also after 1 or 2 weeks. Its running Fedora 13 + 2.6.34.9-69.fc13.i686.PAE.
Dell tested the complete hardware , there was no problem. This server is not doing much : serving files (smb) + pppoe connexion.
Freeze looks the same , no error in the logs, power and fans running, OS frozen.

PE100 is the only "server" hardware i get problem running on Fedora.
No problem with R710 , PE2950, PE2850, PE750

Best Regards,
Philippe

F15 is going EOL in less than a month.  We aren't going to get this fixed.