From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041111 Firefox/1.0 Description of problem: We were previously using kernel-smp-2.4.21-15.0.4.EL under RHAS 3.0 without any problems. However, once we upgraded to kernel-smp-2.4.21-27.0.2.EL, the server would not boot - it failed to mount the root filesystem. It hung after printing the "Freeing unused kernel memory" line to the console, eg: Jan 24 04:36:14 huon kernel: Partition check: Jan 24 04:36:14 huon kernel: sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 > Jan 24 04:36:14 huon kernel: Journalled Block Device driver loaded Jan 24 04:36:14 huon kernel: kjournald starting. Commit interval 5 seconds Jan 24 04:36:14 huon kernel: EXT3-fs: mounted filesystem with ordered data mode. Jan 24 04:36:14 huon kernel: Freeing unused kernel memory: 228k freed We booted back into kernel-smp-2.4.21-15.0.4.EL and used the steps in buzilla 131703 to use the older version of the aacraid driver with 2.4.21-27.0.2. We were then able to boot with kernel 2.4.21-27.0.2. The strange part is we have three servers with the same RAID card (HP NetRAID 4M). Two of them work fine with 2.4.21-27.0.2 SMP, and one didn't and needed the older driver... Version-Release number of selected component (if applicable): kernel-smp-2.4.21-27.0.2.EL How reproducible: Always Steps to Reproduce: 1. Boot server with 2.4.21-27.0.2 kernel, using new aacraid driver. 2. Server does not boot. Actual Results: Server hangs after printing "freeing unused kernel memory" line Expected Results: Server mounts filesystems and boots Additional info:
The messages above seem to indicate that the driver loaded okay, sda was configured, and the ext3 filesystem was mounted. Then the hang occurred. Is this accurate? Notes to self: The default driver in 2.4.21-27.0.2.EL (Update 4) is 1.1.5-2361. The older driver provided in that kernel is aacraid_10102 (v1.1.2). The default driver worked in two of the three servers. The third required the older driver. Mark, would you please take a look at this? Tom
I have never experienced this style of failure in any certification testing. The closest cards we have to the HP NetRAID 4M at hand, are the 5400S, and those are deployed in a 8 processor build server with 6GB of memory running a 2.4.19 kernel with the 1.1.5-2366 driver. I do not have the option to experiment. This server has been rock solid. 1.1.5-2361 is actually a branch of 1.1.5-2340 with the aac_info fix that was introduced in 1.1.5-2364. The differences I have between 1.1.5-2366 and 1.1.5-2340 are mostly cosmetic, however, one of the three following issues were fixed (if we disregard the aac_info_fix already done): - faked AIF needs to call fib_init() later ADPmp70525, panic resulted when the adapter caused a blinkLED timeout. I mark this as 5% odds, reduced because this adapter I believe has no blinkLED state and thus would not enter this code path: diff -ru aacraid-1.1.5-2340/linit.c aacraid-1.1.5-2366/linit.c --- aacraid-1.1.5-2340/linit.c Wed Jun 16 10:56:46 2004 +++ aacraid-1.1.5-2366/linit.c Wed Sep 29 13:04:02 2004 @@ -793,13 +874,13 @@ if (fib && hw_fib) { struct aac_aifcmd * aif; memset(hw_fib, 0, sizeof(struct hw_fib)); - fib_init(fib); memset(fib, 0, sizeof(struct fib)); + fib->hw_fib = hw_fib; + fib->dev = aac; + fib_init(fib); fib->type = FSAFS_NTC_FIB_CONTEXT; fib->size = sizeof (struct fib); - fib->hw_fib = hw_fib; fib->data = hw_fib->data; - fib->dev = aac; aif = (struct aac_aifcmd *)hw_fib->data; aif->command = AifCmdEventNotify; aif->seqnum = 0xFFFFFFFF; - Since kfree has the possibility of switching in some esoteric variants of the kernel, and since the BKL is held during ioctl calls, we are unlocking the fib lock around these system calls. I mark this as 0% chance since no ioctl calls are performed at this point. diff -ru aacraid-1.1.5-2340/commctrl.c aacraid-1.1.5-2366/commctrl.c --- aacraid-1.1.5-2340/commctrl.c Wed Jun 16 11:03:55 2004 +++ aacraid-1.1.5-2366/commctrl.c Fri Sep 24 08:30:10 2004 @@ -342,8 +340,10 @@ /* * Free the space occupied by this copy of the fib. */ + spin_unlock_irq(&dev->fib_lock); kfree(fib->hw_fib); kfree(fib); + spin_lock_irq(&dev->fib_lock); } /* * Remove the Context from the AdapterFibContext List @@ -356,7 +356,9 @@ /* * Free the space occupied by the Context */ + spin_unlock_irq(&dev->fib_lock); kfree(fibctx); + spin_lock_irq(&dev->fib_lock); return 0; } - Some adapters have internal limits of 34 SG elements as a result of a bug of not splitting requests up when cache is disabled when sending them to the CHIM. Created a quirk for the adapters that have this limit. The HP card is one of these adapters, so the odds are *this* is the issue. The patch below is 'complicated' by the fact some additional products (cosmetic) were added, it can be simplified by attaching the AAC_QUIRK_34SG to the offending adapters. diff -ru aacraid-1.1.5-2340/aacraid.h aacraid-1.1.5-2366/aacraid.h --- aacraid-1.1.5-2340/aacraid.h Wed Jun 16 11:34:26 2004 +++ aacraid-1.1.5-2366/aacraid.h Thu Sep 30 05:05:23 2004 @@ -481,7 +493,14 @@ * dma mask such that fib memory will be allocated where the * adapter firmware can get to it. */ -#define AAC_QUIRK_31BIT 1 +#define AAC_QUIRK_31BIT 0x0001 + +/* + * Some adapter firmware, when the raid card's cache is turned off, can not + * split up scatter gathers in order to deal with the limits of the + * underlying CHIM. This limit is 34 scatter gather elements. + */ +#define AAC_QUIRK_34SG 0x0002 /* * The adapter interface specs all queues to be located in the same diff -ru aacraid-1.1.5-2340/linit.c aacraid-1.1.5-2366/linit.c --- aacraid-1.1.5-2340/linit.c Wed Jun 16 10:56:46 2004 +++ aacraid-1.1.5-2366/linit.c Wed Sep 29 13:04:02 2004 @@ -194,18 +201,24 @@ { 0x9005, 0x0285, 0x9005, 0x0292, 0, 0, 27 }, /* AAR-2810SA PCI SATA 8ch (Corsair-8) */ { 0x9005, 0x0285, 0x9005, 0x0293, 0, 0, 28 }, /* AAR-21610SA PCI SATA 16ch (Corsair-16) */ { 0x9005, 0x0285, 0x9005, 0x0294, 0, 0, 29 }, /* ESD SO-DIMM PCI-X SATA ZCR (Prowler) */ - { 0x9005, 0x0285, 0x0E11, 0x0295, 0, 0, 30 }, /* SATA 6Ch (Bearcat) */ - - { 0x9005, 0x0285, 0x1028, 0x0287, 0, 0, 31 }, /* Perc 320/DC*/ - { 0x1011, 0x0046, 0x9005, 0x0365, 0, 0, 32 }, /* Adaptec 5400S (Mustang) */ - { 0x1011, 0x0046, 0x9005, 0x0364, 0, 0, 33 }, /* Adaptec 5400S (Mustang) */ - { 0x1011, 0x0046, 0x9005, 0x1364, 0, 0, 34 }, /* Dell PERC2/QC */ - { 0x1011, 0x0046, 0x103c, 0x10c2, 0, 0, 35 }, /* HP NetRAID-4M */ - - { 0x9005, 0x0285, 0x1028, PCI_ANY_ID, 0, 0, 36 }, /* Dell Catchall */ - { 0x9005, 0x0285, 0x17aa, PCI_ANY_ID, 0, 0, 37 }, /* Legend Catchall */ - { 0x9005, 0x0285, PCI_ANY_ID, PCI_ANY_ID, 0, 0, 38 }, /* Adaptec Catch All */ - { 0x9005, 0x0286, PCI_ANY_ID, PCI_ANY_ID, 0, 0, 39 }, /* Adaptec Rocket Catch All */ + { 0x9005, 0x0285, 0x103C, 0x3227, 0, 0, 30 }, /* AAR-2610SA PCI SATA 6ch */ + { 0x9005, 0x0285, 0x9005, 0x0296, 0, 0, 31 }, /* ASR-2240S */ + { 0x9005, 0x0285, 0x9005, 0x0297, 0, 0, 32 }, /* ASR-4005SAS */ + { 0x9005, 0x0285, 0x1014, 0x02F2, 0, 0, 33 }, /* & IBM 8i */ + { 0x9005, 0x0285, 0x9005, 0x0298, 0, 0, 34 }, /* ASR-4000SAS */ + { 0x9005, 0x0285, 0x9005, 0x0299, 0, 0, 35 }, /* ASR-4800SAS */ + { 0x9005, 0x0285, 0x9005, 0x029A, 0, 0, 36 }, /* ASR-4805SAS */ + + { 0x9005, 0x0285, 0x1028, 0x0287, 0, 0, 37 }, /* Perc 320/DC*/ + { 0x1011, 0x0046, 0x9005, 0x0365, 0, 0, 38 }, /* Adaptec 5400S (Mustang) */ + { 0x1011, 0x0046, 0x9005, 0x0364, 0, 0, 39 }, /* Adaptec 5400S (Mustang) */ + { 0x1011, 0x0046, 0x9005, 0x1364, 0, 0, 40 }, /* Dell PERC2/QC */ + { 0x1011, 0x0046, 0x103c, 0x10c2, 0, 0, 41 }, /* HP NetRAID-4M */ + + { 0x9005, 0x0285, 0x1028, PCI_ANY_ID, 0, 0, 42 }, /* Dell Catchall */ + { 0x9005, 0x0285, 0x17aa, PCI_ANY_ID, 0, 0, 43 }, /* Legend Catchall */ + { 0x9005, 0x0285, PCI_ANY_ID, PCI_ANY_ID, 0, 0, 44 }, /* Adaptec Catch All */ + { 0x9005, 0x0286, PCI_ANY_ID, PCI_ANY_ID, 0, 0, 45 }, /* Adaptec Rocket Catch All */ { 0,} }; MODULE_DEVICE_TABLE(pci, aac_pci_tbl); @@ -216,22 +229,22 @@ * for the card. At that time we can remove the channels from here */ static struct aac_driver_ident aac_drivers[] = { - { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT }, /* PERC 2/Si (Iguana/PERC2Si) */ - { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT }, /* PERC 3/Di (Opal/PERC3Di) */ - { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT }, /* PERC 3/Si (SlimFast/PERC3Si */ - { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT }, /* PERC 3/Di (Iguana FlipChip/PERC3DiF */ - { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT }, /* PERC 3/Di (Viper/PERC3DiV) */ - { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT }, /* PERC 3/Di (Lexus/PERC3DiL) */ - { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 1, AAC_QUIRK_31BIT }, /* PERC 3/Di (Jaguar/PERC3DiJ) */ - { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT }, /* PERC 3/Di (Dagger/PERC3DiD) */ - { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT }, /* PERC 3/Di (Boxster/PERC3DiB) */ - { aac_rx_init, "aacraid", "ADAPTEC ", "catapult ", 2, AAC_QUIRK_31BIT }, /* catapult */ - { aac_rx_init, "aacraid", "ADAPTEC ", "tomcat ", 2, AAC_QUIRK_31BIT }, /* tomcat */ - { aac_rx_init, "aacraid", "ADAPTEC ", "Adaptec 2120S ", 1, AAC_QUIRK_31BIT }, /* Adaptec 2120S (Crusader) */ - { aac_rx_init, "aacraid", "ADAPTEC ", "Adaptec 2200S ", 2, AAC_QUIRK_31BIT }, /* Adaptec 2200S (Vulcan) */ - { aac_rx_init, "aacraid", "ADAPTEC ", "Adaptec 2200S ", 2, AAC_QUIRK_31BIT }, /* Adaptec 2200S (Vulcan-2m) */ - { aac_rx_init, "aacraid", "Legend ", "Legend S220 ", 1, AAC_QUIRK_31BIT }, /* Legend S220 (Legend Crusader) */ - { aac_rx_init, "aacraid", "Legend ", "Legend S230 ", 2, AAC_QUIRK_31BIT }, /* Legend S230 (Legend Vulcan) */ + { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* PERC 2/Si (Iguana/PERC2Si) */ + { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* PERC 3/Di (Opal/PERC3Di) */ + { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* PERC 3/Si (SlimFast/PERC3Si */ + { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* PERC 3/Di (Iguana FlipChip/PERC3DiF */ + { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* PERC 3/Di (Viper/PERC3DiV) */ + { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* PERC 3/Di (Lexus/PERC3DiL) */ + { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 1, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* PERC 3/Di (Jaguar/PERC3DiJ) */ + { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* PERC 3/Di (Dagger/PERC3DiD) */ + { aac_rx_init, "percraid", "DELL ", "PERCRAID ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* PERC 3/Di (Boxster/PERC3DiB) */ + { aac_rx_init, "aacraid", "ADAPTEC ", "catapult ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* catapult */ + { aac_rx_init, "aacraid", "ADAPTEC ", "tomcat ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* tomcat */ + { aac_rx_init, "aacraid", "ADAPTEC ", "Adaptec 2120S ", 1, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* Adaptec 2120S (Crusader) */ + { aac_rx_init, "aacraid", "ADAPTEC ", "Adaptec 2200S ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* Adaptec 2200S (Vulcan) */ + { aac_rx_init, "aacraid", "ADAPTEC ", "Adaptec 2200S ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* Adaptec 2200S (Vulcan-2m) */ + { aac_rx_init, "aacraid", "Legend ", "Legend S220 ", 1, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* Legend S220 (Legend Crusader) */ + { aac_rx_init, "aacraid", "Legend ", "Legend S230 ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* Legend S230 (Legend Vulcan) */ { aac_rx_init, "aacraid", "ADAPTEC ", "Adaptec 3230S ", 2 }, /* Adaptec 3230S (Harrier) */ { aac_rx_init, "aacraid", "ADAPTEC ", "Adaptec 3240S ", 2 }, /* Adaptec 3240S (Tornado) */ @@ -247,17 +260,23 @@ { aac_rx_init, "aacraid", "ADAPTEC ", "AAR-2810SA SATA ", 1 }, /* AAR- 2810SA PCI SATA 8ch (Corsair-8) */ { aac_rx_init, "aacraid", "ADAPTEC ", "AAR-21610SA SATA", 1 }, /* AAR- 21610SA PCI SATA 16ch (Corsair-16) */ { aac_rx_init, "aacraid", "ADAPTEC ", "SO-DIMM SATA ZCR", 1 }, /* ESD SO-DIMM PCI-X SATA ZCR (Prowler) */ - { aac_rx_init, "aacraid", "ADAPTEC ", "SATA 6Channel ", 1 }, /* SATA 6Ch (Bearcat) */ - - { aac_rx_init, "percraid", "DELL ", "PERC 320/DC ", 2, AAC_QUIRK_31BIT }, /* Perc 320/DC*/ - { aac_sa_init, "aacraid", "ADAPTEC ", "Adaptec 5400S ", 4 }, /* Adaptec 5400S (Mustang)*/ - { aac_sa_init, "aacraid", "ADAPTEC ", "AAC-364 ", 4 }, /* Adaptec 5400S (Mustang)*/ - { aac_sa_init, "percraid", "DELL ", "PERCRAID ", 4, AAC_QUIRK_31BIT }, /* Dell PERC2/QC */ - { aac_sa_init, "hpnraid", "HP ", "NetRAID ", 4 }, /* HP NetRAID-4M */ - - { aac_rx_init, "aacraid", "DELL ", "RAID ", 2, AAC_QUIRK_31BIT }, /* Dell Catchall */ - { aac_rx_init, "aacraid", "Legend ", "RAID ", 2, AAC_QUIRK_31BIT }, /* Legend Catchall */ - { aac_rx_init, "aacraid", "ADAPTEC ", "RAID ", 2, AAC_QUIRK_31BIT }, /* Adaptec Catch All */ + { aac_rx_init, "aacraid", "ADAPTEC ", "AAR-2610SA ", 1 }, /* SATA 6Ch (Bearcat) */ + { aac_rx_init, "aacraid", "ADAPTEC ", "ASR-2240S ", 1 }, /* ASR- 2240S */ + { aac_rx_init, "aacraid", "ADAPTEC ", "ASR-4005SAS ", 1 }, /* ASR- 4005SAS */ + { aac_rx_init, "aacraid", "IBM ", "ServeRAID 8i ", 1 }, /* & IBM 8i */ + { aac_rx_init, "aacraid", "ADAPTEC ", "ASR-4000SAS ", 1 }, /* ASR- 4000SAS */ + { aac_rx_init, "aacraid", "ADAPTEC ", "ASR-4800SAS ", 1 }, /* ASR- 4800SAS */ + { aac_rx_init, "aacraid", "ADAPTEC ", "ASR-4805SAS ", 1 }, /* ASR- 4805SAS */ + + { aac_rx_init, "percraid", "DELL ", "PERC 320/DC ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* Perc 320/DC*/ + { aac_sa_init, "aacraid", "ADAPTEC ", "Adaptec 5400S ", 4, AAC_QUIRK_34SG }, /* Adaptec 5400S (Mustang)*/ + { aac_sa_init, "aacraid", "ADAPTEC ", "AAC-364 ", 4, AAC_QUIRK_34SG }, /* Adaptec 5400S (Mustang)*/ + { aac_sa_init, "percraid", "DELL ", "PERCRAID ", 4, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* Dell PERC2/QC */ + { aac_sa_init, "hpnraid", "HP ", "NetRAID ", 4, AAC_QUIRK_34SG }, /* HP NetRAID-4M */ + + { aac_rx_init, "aacraid", "DELL ", "RAID ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* Dell Catchall */ + { aac_rx_init, "aacraid", "Legend ", "RAID ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* Legend Catchall */ + { aac_rx_init, "aacraid", "ADAPTEC ", "RAID ", 2, AAC_QUIRK_31BIT | AAC_QUIRK_34SG }, /* Adaptec Catch All */ { aac_rkt_init, "aacraid", "ADAPTEC ", "RAID ", 2 } /* Adaptec Rocket Catch All */ }; @@ -1319,6 +1435,16 @@ aac_get_adapter_info(aac); /* + * Lets override negotiations and drop the maximum SG limit to 34 + */ + if ((aac_drivers[index].quirks & AAC_QUIRK_34SG) + && (aac->scsi_host_ptr->sg_tablesize > 34)) { + aac->scsi_host_ptr->sg_tablesize = 34; + aac->scsi_host_ptr->max_sectors + = (aac->scsi_host_ptr->sg_tablesize * 8) + 120; + } + + /* * max channel will be the physical channels plus 1 virtual channel * all containers are on the virtual channel 0 * physical channels are address by their actual physical number+1
Tom: you are correct. Going by the boot messages after changing to the older version of the driver, the next line in the boot messages after the hang should be: EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,6), internal journal /dev/sda6 is the / partition.
My earlier statement about the 2.4.21-27.0.2.EL kernel being OK on two of the three servers doesn't look to be correct. While they booted OK, I've filed bug 146974 relating to data corruption under /boot...
This bug is filed against RHEL 3, which is in maintenance phase. During the maintenance phase, only security errata and select mission critical bug fixes will be released for enterprise products. Since this bug does not meet that criteria, it is now being closed. For more information of the RHEL errata support policy, please visit: http://www.redhat.com/security/updates/errata/ If you feel this bug is indeed mission critical, please contact your support representative. You may be asked to provide detailed information on how this bug is affecting you.