Fedora Account System
Red Hat Associate
Red Hat Customer
As requested this is a seperate bug report from 498408. Presently I regularly have this problem with machines during mostly upgrades, but some installs. Will try and send you some sample log file sets.
The problem is that with machines which have multiple drives the boot loader is frequently not being properly installed. You have to then rescue boot and use grub-install to install the boot loader. That in and of itself can be a problem in that some of our machines have video systems that the installer doesn't like and since by default the rescue boot system scans for and tries to setup the video system instead of using default vga this requires a lot of manual actions.
Opening more and more bug reports with no good information does not help anything. Please only open a report if you are prepared to include the log files and information required to allow us to make some use of the report. Marking this bug as NEEDINFO until you can provide the logs.
Will be providing logs from a sampling of the machines experiencing the error, but needed to do this as house keeping to provide a place to put them.
Created attachment 385154 [details] install.log from first machine This machine was a fresh install and came up to the black screen with the blinking cursor in the upper left.
Created attachment 385156 [details] install syslog from first machine
Created attachment 385160 [details] anaconda log from first machine
Created attachment 385166 [details] anaconda syslog from first machine
Created attachment 385167 [details] program log from first machine
Created attachment 385168 [details] storage log from first machine
Created attachment 385188 [details] anaconda log for second macine This machine was upgraded and then went to the grub> prompt. It is a name server and test system. By the way the first machine is basically a fire wall and router for a cable connection.
Created attachment 385191 [details] anaconda syslog for second machine
Created attachment 385192 [details] program log for second machine
Created attachment 385193 [details] storage log for second machine
Created attachment 385194 [details] upgrade log for second machine
Created attachment 385195 [details] upgrade syslog for second server
Created attachment 385200 [details] anaconda log for third machine OK this is the third machine. It was a new install, but interestingly the install log stuff doesn't exist in /root. It is a development application web server. It also was the most annoying, in that grub install did not work and I had to actually go to manually using grub to install the boot loader.
Created attachment 385201 [details] anaconda syslog from third machine
Created attachment 385203 [details] program log from third machine
Created attachment 385204 [details] storage log from third machine
Ray, Where was (is) the bootloader installed on this systems ? Was it installed in the MBR, or in the boot record of a partition ? I'm asking because we have a bug in F-12, where we change the active flag in the partition table, and this causes unbootable systems. Thanks, Hans
Where ever things do this by default. This also has been a bug going back to since you changed /dev/hd0 to /dev/sd0 So is this a bug that appeared in fc12 or one that was in the past that was discovered in fc12? Also if you are saying that the partition was supposed to have a "*" in the boot column under fdisk and didn't, I already checked for that. Those have been disappearing for a while now when you use the software raid system. Yeah I end up spending a great deal of time in the rescue mode whenever I do an upgrade in the last year or so.
Created attachment 385211 [details] anaconda log from forth machine This one also gave the black with the cursor screen. It is a primary application web server.
Created attachment 385212 [details] anaconda syslog from fourth machine
Created attachment 385213 [details] program log from fourth machine
Created attachment 385215 [details] storage log from fourth machine
Created attachment 385216 [details] upgrade log from fourth machine Yeah this was an upgrade. I forgot to tell you that.
Created attachment 385217 [details] upgrade syslog from fourth machine
Ray, Wait a minute (I have not looked at the logs yet), all these systems are using an software raid / mdraid /boot ? Regards, Hans
Nope all are using software raid. Some are using /md0 as the /boot some are using md0 as the root with /boot on the / and some are using /sda1 as the boot.
Created attachment 385221 [details] anaconda log for fith machine This is the fifth machine. It is supposed to be identical to the fourth, and is the backup for the fourrh. Yeah I am going to end here, but I figure this gives a good overview. I ahve a 30 more of these machines I could check specific things on, but figured that uploading all 30 would be serious over kill. I have duplicates of vertually all of these machines. I also have other duplicates I have had no problem with. The exact symptoms don't seem to follow hardware. As an example while the duplicate of this machine gave a black screen failure this one gave a "grub>" failure.
Created attachment 385222 [details] anaconda syslog for fifth machine
Created attachment 385223 [details] program log for fifth machine
Created attachment 385224 [details] upgrade log for fifth machine
Created attachment 385225 [details] storage log for fifth machine
Created attachment 385226 [details] upgrade syslog for fifth server
Would it help if I created a single file of the df command of all five servers?
(In reply to comment #29) > Nope all are using software raid. Some are using /md0 as the /boot some are > using md0 as the root with /boot on the / and some are using /sda1 as the > boot. Hi, Ok, for the case of /boot on /dev/md# all I can say is: yes our grub writing code for that configuration was broken (sorry about that), and it has been almost completely rewritten (and extensively tested) for F-13. So that part of this bug I consider closed->next release. I know this sucks, and I apologize, but with the installer their is little we can do post release to fix issues. Please don't take this as us not caring we do care, but as the code has been rewritten their is little use in debugging the old code as it lives in F-12. Please do test with F-13 alpha when it is out, and if you find any issues file a new bug, and send me a personal mail pointing to it, and I'll make sure any remaining issues get attention from the team member who did the rewrite (Radek Vykydal). However this does not explain the /boot on /dev/sda1 case, so I'm keeping this bug open to track that. Some questions about that: 1) Which machine (as you numbered the logs) has this configuration 2) Can you provide /etc/fstab and grub.conf and for each disk: "parted /dev/sd? p" Output please ?
OK wished I had known about the software raid issue as I have had this one several times and it keeps getting somewhat ignored. In fact the other bug I opened this one from I had mentioned this as being one of the possible issues and that was pretty much ignored. Frankly had I known about the rewrite I would probably not even have opened this one. I would have waited for the next version to see if the boot not on RAID was also fixed then to. OH I can certainly if you are doing a rewrite and addressing it in the rewrite for fc13 that this is not an issue which can get much attention in fc12. I have made the same call and have had to explain that same call to users before. Good news bad news on the boot on sd issue. Two of the machines with this problem are four and five. My goal had been to fully raid them this month but had not gotten to it. Given this text I think I will wait for the next version. I do remember in Netware (which I worked with extensively) it was possible to install fully in RAID 1 (Mirroring in their terms) and then swap the cables and load the boot loader again and have it actually not only stay running with a drive loss, but be able to reboot with either drive failed. If you are contemplating something like this for fedora and need someone to test it, I do have some test machines I could setup and test and would be willing to give it a good workout.
Created attachment 385430 [details] the requested material for server 4 This is the requested information for server 4 This server, and the other one were generated in the day and age when /dev/sda was /dev/hda Also for a long time what is now sdb was sda
Created attachment 385433 [details] The requested information for server 5 Here is for the fifth server
(In reply to comment #38) > OH I can certainly if you are doing a rewrite and addressing it in the rewrite > for fc13 that this is not an issue which can get much attention in fc12. I > have made the same call and have had to explain that same call to users before. > Thanks for understanding! > I do remember in Netware (which I worked with extensively) it was possible to > install fully in RAID 1 (Mirroring in their terms) and then swap the cables and > load the boot loader again and have it actually not only stay running with a > drive loss, but be able to reboot with either drive failed. If you are > contemplating something like this for fedora and need someone to test it, I do > have some test machines I could setup and test and would be willing to give it > a good workout. If you could setup a test once F-13 alpha is released that would be great. This is something we would really like to have fixed for F-13, so testing definitely is welcome. About the issue with the machines with a regular /boot also not booting after upfrade, I'm afraid I need one more bit of information from machine 4 and 5 for that, can you please collect /boot/grub/device.map from both machines and attach it here ? Thanks, Hans
Server 5 # this device map was generated by anaconda (hd0) /dev/sda (hd1) /dev/sdb Server 4 # this device map was generated by anaconda (hd0) /dev/sda (hd1) /dev/sdb Now this is very very very interesting. I had never seen this before. This is basically wrong. The drive connected to the first SATA port is the one that is referenced as sdb. If I pull that drive out the the bios thinks and informs me that the first drive is missing. If I pull out what is drive sda it will be the second drive and pulling it will caused the bios screen to say that the second drive is missing. If I rescue boot and to a fdisk with one of the drives missing this is the logic that I get. But yet here is is telling me that they are in the other order. weird, and probably exactly where our problem is located. Since we are now dealing with the exact same hardware, I figured that I would bring the map from the third server back in here. # file updated by anaconda # this device map was generated by anaconda (hd0) /dev/sdb (hd1) /dev/sda This was quite frankly what I was expecting to find in the other two cases, but .... This one has the same drive pull test results as machines 4 and 5. If nothing else I think we have located the general vicinity of the problem.
Hi, (In reply to comment #42) > Server 5 > > # this device map was generated by anaconda > (hd0) /dev/sda > (hd1) /dev/sdb > > Server 4 > > # this device map was generated by anaconda > (hd0) /dev/sda > (hd1) /dev/sdb > > > Now this is very very very interesting. I had never seen this before. This > is basically wrong. The drive connected to the first SATA port is the one > that is referenced as sdb. If I pull that drive out the the bios thinks and > informs me that the first drive is missing. If I pull out what is drive sda > it will be the second drive and pulling it will caused the bios screen to say > that the second drive is missing. > > If I rescue boot and to a fdisk with one of the drives missing this is the > logic that I get. But yet here is is telling me that they are in the other > order. > > weird, and probably exactly where our problem is located. > Yes, we always put grub on hd0, shuffling the BIOS order if told to boot from another disk (as that is what the BIOS does when you tell it to boot from another disk). So this definitely explains why they are not booting even do they have a plain /boot . As for the why of this likely the BIOS does not export any (usable) EDD data, so we cannot ask it for the order, thus we fallback to just detection order, which in this case is wrong. This could be caused by writing the same image to boot disks, as that would give them the same mbr signature, which has to be unique for detecting BIOS driver order through EDD (which is the only way to detect it). > Since we are now dealing with the exact same hardware, I figured that I would > bring the map from the third server back in here. > > # file updated by anaconda > # this device map was generated by anaconda > (hd0) /dev/sdb > (hd1) /dev/sda > > > This was quite frankly what I was expecting to find in the other two cases, but > .... > > This one has the same drive pull test results as machines 4 and 5. > If it is the exact same hardware you are most likely having mbr's with the same signature (the 4 bytes at 440 bytes offset into the disk) on both disks in the case where this is not working, which could be caused by imaging the 2 disks with an identical image. Anyways, our EDD code has some issues with BIOS RAID sets, but since that is not in use here, I'm going to blame this on either a firmware bug, or identical signatures and close this bug. I'm using the resolution next release, as that will have the re-written mdraid mirror /boot handling.
Interesting this might be an echo on my dmraid problems. Basically once a disk has been in a dmraid array it appears that it can never be removed from that array. If I remember rightly the drives in servers 3 and 5 are from a machine that used dm raid and was upgraded to a higher storage size. Four should have had empty scratch drives. I have a bug open (Bug 542734) for the fact that I keep having problems with drives that have been in a dmraid array retaining this status and fedora giving strange results because of it. I also have had some really interesting results with drives that have had lvm systems on them not losing all of that information when the partition table is reinitialized and resetup. One of the things you guys might want to think about for fedora 13 is some form of program that will take a drive and reset it back to a scratch from the factory setup. That maybe be the problem we are seeing here.
Ray, Did you happen to "zero out" (dd if=/dev/zero of=/dev/sd?) the drives in the machines in question (the ones with a regular /boot, and the bios drive order issues) ?? I'm asking because the *only* way anaconda can figure out the BIOS drive order is through EDD which uses something known as the mbr signature (four random bytes in a standard PC MBR), and if you zero out the mbr, the signature on both disks will be indentical (and this get ignored, as it is useless then). You can circumvent this next time, be using parted's mklabel command after zero-ing out the beginning of the disk, so that a valid mbr (with a random signature) gets put on there. Regards, Hans
p.s. I took a look at the storage logs and no dmraid metadata is being found on the disks of machine 4 and 5, so that does not seem to be a factor.
Mater a fact I did. That was recommended on another bug. I am beginning to believe that a great deal of my problems are related to bugs in the software raid, dmraid, and lvm system or echos of those bugs. On the other hand I also did a fdisk rebuild label thing which I would think is the same thing. I also have had to boot other oses and do a full write because linux refused to write to the first section of the disk.