Bug 696876

Created attachment 492284 [details]
Anaconda Log  for san boot using /dev/mapper/mpatha

Stratus also hit this problem and it is blocking our ability to install RHEL6 for boot from SAN.  This is a regression of functionality that was available in RHEL5.

We found we could work around this by creating /etc/multipath.conf and starting multipath in the kickstart %pre step.

------- Comment From ranittal.ibm.com 2011-05-11 03:38 EDT-------
Hi Redhat,

This defect is still present in rc2, what is the chances this will be fixed for ga?

Thanks,
Ravi

(In reply to comment #6)
> ------- Comment From ranittal.ibm.com 2011-05-11 03:38 EDT-------
> Hi Redhat,
> 
> This defect is still present in rc2, what is the chances this will be fixed for
> ga?
> 
> Thanks,
> Ravi

Hello,

this is not scheduled for 6.1 GA.

Ales Kozumplik

------- Comment From jkenefic.com 2011-05-11 12:21 EDT-------
> Stratus also hit this problem and it is blocking our ability to install RHEL6
> for boot from SAN.  This is a regression of functionality that was available in
> RHEL5.
> We found we could work around this by creating /etc/multipath.conf and starting
> multipath in the kickstart %pre step.

I've attempted this workaround, and while I am able to see the multipath device from tty2, the installer is reporting "no usable disks found" this is going to make kickstart installation on san boot unworkable

I will attach the anaconda log and kickstart file

Created attachment 498333 [details]
Anaconda log from may 11


------- Comment (attachment only) From jkenefic.com 2011-05-11 12:27 EDT-------

Created attachment 498334 [details]
kickstart file from may 11


------- Comment (attachment only) From jkenefic.com 2011-05-11 12:28 EDT-------

I tried a modified kickstart partitioning scheme like the one found in attachment from IBM and got the same "no usable disks found" error.  Changing the "ignoredisk" line from  "--only-use" to "--drives=mapper/mpathx,..." moved the installer past the error message and the install continued.

> changing the kickstart formating stanza to use /dev/disk/by-id/scsi-<id>
> as follows
> 
> ignoredisk --only-use=disk/by-id/scsi-3600a0b80001115960000afef4cc96c4b
> clearpart --all --drives=disk/by-id/scsi-3600a0b80001115960000afef4cc96c4b
> part swap --size=4096 --asprimary
> --ondisk=disk/by-id/scsi-3600a0b80001115960000afef4cc96c4b
> part / --fstype ext3 --grow --size=100 --asprimary 
> --ondisk=disk/by-id/scsi-3600a0b80001115960000afef4cc96c4b

This particular kickstart notation doesn't work with multipath because /dev/disk/by-id/scsi-3600a0b80001115960000afef4cc96c4b in fact points to an arbitrary mpath member with the given id, so it equals to writing e.g.

ignoredisk --only-use=sda

where sda is an mpath member. 

Nonetheless, your request to be able to identify mpaths by their ids is valid and will be dealt with in the bug 677263.

In this bug I will try to make anaconda recognize the "mapper/mpath<x>" notation.

Patch is waiting for a review: https://www.redhat.com/archives/anaconda-devel-list/2011-June/msg00051.html

*** Bug 675322 has been marked as a duplicate of this bug. ***

I just tested the patch from

   https://www.redhat.com/archives/anaconda-devel-list/2011-June/msg00056.html

Using this partitioning section:

ignoredisk --drives=sda,sdc,sde,sdg,sdi
bootloader --location=mbr --append=crashkernel=auto rhgb quiet
clearpart --all --drives=mpatha,sdb,sdd,sdf,sdh
part /boot --fstype ext4 --size=100 --ondisk=mapper/mpatha
part pv.10 --size=1 --grow --ondisk=mapper/mpatha
volgroup jrummy pv.10
logvol swap --fstype swap --name=LogVol01 --vgname=jrummy --size=100
logvol / --fstype ext4 --name=LogVol00 --vgname=jrummy --size=1 --grow

With the above, it works.  However, notice that I use mpatha in clearpart and mapper/mpatha in part.  This is the only way it would work for me.  With mapper/mpatha in clearpart, it fails with 'Specified nonexistent disk mapper/mpatha in clearpart command'. With mpatha in the part command, it fails with 'Specified nonexistent disk mpatha in partition command'.  

Let me know if I can help with further testing, or you need access to my test environment

-John

(In reply to comment #15)
> With the above, it works.  However, notice that I use mpatha in clearpart and
> mapper/mpatha in part.  This is the only way it would work for me.  With
> mapper/mpatha in clearpart, it fails with 'Specified nonexistent disk
> mapper/mpatha in clearpart command'. With mpatha in the part command, it fails
> with 'Specified nonexistent disk mpatha in partition command'.  
> 
> Let me know if I can help with further testing, or you need access to my test
> environment
> 

Very strange. I have this passing allright:

clearpart --all --drives=mpatha,mpathb
ignoredisk --only-use=mpatha,mpathb
part prepboot --fstype=prepboot --asprimary --size=4 --ondisk=mpathb
part swap --asprimary --size=4096 --ondisk=mpatha
part / --fstype=ext3 --grow --asprimary --size=100 --ondisk=mpathb


One thing is clear, we should stick to one naming convention in all commands and that should be without 'mapper/' (because that's how anaconda memorizes device names internally). 

Still, I don't see how your error can happen, so I'd like to take a look. Can you please append updates=http://cobra02.englab.brq.redhat.com/ks/ak/updates/updates.img to the bootline and give me access to the machine?

Ales

*** Bug 712360 has been marked as a duplicate of this bug. ***

Fixed on the rhel6 branch by commit 01697c1e13550f44eab2333704df9ca7c51b52dd.

Created attachment 511504 [details]
Kickstart files and console outputs

Created attachment 511505 [details]
Kickstart files and console outputs - RHEL6.1 Alpha

Created attachment 511506 [details]
Error message

*** Bug 728137 has been marked as a duplicate of this bug. ***

Ales,

It seems we still have something more to fix.

Repo: RHEL6.2-20110814.n.0
Beaker job: https://beaker.engineering.redhat.com/jobs/119783
(You can check log in this page.)

storageqe-05.rhts.eng.bos.redhat.com have 2 LUN via 4 paths each.
mpatha (20090ef1270000000) is the boot LUN.

Kickstart options: ignoredisk --only-use=mpatha

The problems are:

 0. In anaconda-ks.cfg, it show this line:
    ===
    #ignoredisk --only-use=mpatha,sda,sdc,sde,sdg
    ===
    It should not contain sdX which is the member of mpatha.

 1. OS boot into recovery mode after installation. Reason: Incorrect /etc/fstab for /boot partition.
    It got this line in /etc/fstab:
    ===
    UUID=7598c6b0-d93e-44d1-92dc-81a8291013e4 /boot ext4 defaults 1 2
    ===
    UUID is pointing to /dev/sdc1 which is incorrect. It should point to /dev/mapper/mpathap1
    I cannot find /dev/mapper/mpathap1 in /dev/disk/by-uuid/, check this output:
    ===
    (Repair filesystem) 29 # ls -l /dev/disk/by-uuid/
    total 0
    lrwxrwxrwx. 1 root root 10 Aug 14 21:50 7598c6b0-d93e-44d1-92dc-81a8291013e4 -> ../../sdc1
    lrwxrwxrwx. 1 root root 10 Aug 14 21:50 c8811b1b-a01f-4efe-ad07-98dcb95ed7f3 -> ../../dm-3
    lrwxrwxrwx. 1 root root 10 Aug 14 21:50 e2d5d8b2-9325-433d-803c-31c855a8cd5a -> ../../dm-4
    (Repair filesystem) 30 # ls -l /dev/mapper/
    total 0
    lrwxrwxrwx. 1 root root      7 Aug 14 21:50 20090ef1270000000 -> ../dm-0
    lrwxrwxrwx. 1 root root      7 Aug 14 21:50 20090ef1270000000p1 -> ../dm-1
    lrwxrwxrwx. 1 root root      7 Aug 14 21:50 20090ef1270000000p2 -> ../dm-2
    crw-rw----. 1 root root 10, 58 Aug 14 17:50 control
    lrwxrwxrwx. 1 root root      7 Aug 14 21:50 vg_storageqe05-lv_root -> ../dm-3
    lrwxrwxrwx. 1 root root      7 Aug 14 21:50 vg_storageqe05-lv_swap -> ../dm-4
    ===
    As multipath is using O_EXCL which take /dev/sdc exclusively, OS cannot mount /dev/sdc1 as /boot, hence it goes intto recovery mode.

 2. multipath doesn't create '/etc/multipath/wwids' during installation.
    This will cause OS don't have /dev/mapper/mpatha, instead of that we got /dev/mapper/20090ef1270000000
    By default, anaconda should set "user_friendly_names yes" in /etc/multipath.conf.
    OS got correct option in /etc/multipath.conf after installation.

You can got detailed information in http://lab2.rhts.eng.bos.redhat.com/beaker/logs/recipes/246492//console.log
That's how I debug these things above.

Ales,

It seems we still have something more to fix.

Repo: RHEL6.2-20110814.n.0
Beaker job: https://beaker.engineering.redhat.com/jobs/119783
(You can check log in this page.)

storageqe-05.rhts.eng.bos.redhat.com have 2 LUN via 4 paths each.
mpatha (20090ef1270000000) is the boot LUN.

Kickstart options: ignoredisk --only-use=mpatha

The problems are:

 0. In anaconda-ks.cfg, it show this line:
    ===
    #ignoredisk --only-use=mpatha,sda,sdc,sde,sdg
    ===
    It should not contain sdX which is the member of mpatha.

 1. OS boot into recovery mode after installation. Reason: Incorrect /etc/fstab for /boot partition.
    It got this line in /etc/fstab:
    ===
    UUID=7598c6b0-d93e-44d1-92dc-81a8291013e4 /boot ext4 defaults 1 2
    ===
    UUID is pointing to /dev/sdc1 which is incorrect. It should point to /dev/mapper/mpathap1
    I cannot find /dev/mapper/mpathap1 in /dev/disk/by-uuid/, check this output:
    ===
    (Repair filesystem) 29 # ls -l /dev/disk/by-uuid/
    total 0
    lrwxrwxrwx. 1 root root 10 Aug 14 21:50 7598c6b0-d93e-44d1-92dc-81a8291013e4 -> ../../sdc1
    lrwxrwxrwx. 1 root root 10 Aug 14 21:50 c8811b1b-a01f-4efe-ad07-98dcb95ed7f3 -> ../../dm-3
    lrwxrwxrwx. 1 root root 10 Aug 14 21:50 e2d5d8b2-9325-433d-803c-31c855a8cd5a -> ../../dm-4
    (Repair filesystem) 30 # ls -l /dev/mapper/
    total 0
    lrwxrwxrwx. 1 root root      7 Aug 14 21:50 20090ef1270000000 -> ../dm-0
    lrwxrwxrwx. 1 root root      7 Aug 14 21:50 20090ef1270000000p1 -> ../dm-1
    lrwxrwxrwx. 1 root root      7 Aug 14 21:50 20090ef1270000000p2 -> ../dm-2
    crw-rw----. 1 root root 10, 58 Aug 14 17:50 control
    lrwxrwxrwx. 1 root root      7 Aug 14 21:50 vg_storageqe05-lv_root -> ../dm-3
    lrwxrwxrwx. 1 root root      7 Aug 14 21:50 vg_storageqe05-lv_swap -> ../dm-4
    ===
    As multipath is using O_EXCL which take /dev/sdc exclusively, OS cannot mount /dev/sdc1 as /boot, hence it goes intto recovery mode.

 2. multipath doesn't create '/etc/multipath/wwids' during installation.
    This will cause OS don't have /dev/mapper/mpatha, instead of that we got /dev/mapper/20090ef1270000000
    By default, anaconda should set "user_friendly_names yes" in /etc/multipath.conf.
    OS got correct option in /etc/multipath.conf after installation.

You can got detailed information in http://lab2.rhts.eng.bos.redhat.com/beaker/logs/recipes/246492//console.log
That's how I debug these things above.

Should be assigned rather new.

> The problems are:
> 
>  0. In anaconda-ks.cfg, it show this line:
>     ===
>     #ignoredisk --only-use=mpatha,sda,sdc,sde,sdg
>     ===
>     It should not contain sdX which is the member of mpatha.

Nope, there is no problem with explicitly naming all of the member paths to be used. It might not be necessary but it is not a bug and it is much easier to implement it this way.

>  1. OS boot into recovery mode after installation. Reason: Incorrect /etc/fstab
> for /boot partition.
>     It got this line in /etc/fstab:
>     ===
>     UUID=7598c6b0-d93e-44d1-92dc-81a8291013e4 /boot ext4 defaults 1 2
>     ===
>     UUID is pointing to /dev/sdc1 which is incorrect. It should point to
> /dev/mapper/mpathap1

As far as I know, a UUID is a property of a filesystem so it points to the filesystem on /dev/sdc1 as well as the (same) filesystem on /dev/mapper/mpathap1.

I am not sure why you are seeing the recovery mode but it can be related to the new issues there are with multipath installations, see bug 701371. If you don't use the new ignoredisk with multipaths syntax, does that happen too?
 
>  2. multipath doesn't create '/etc/multipath/wwids' during installation.
>     This will cause OS don't have /dev/mapper/mpatha, instead of that we got
> /dev/mapper/20090ef1270000000

Please bug 701371.

Moving to modified. I think we are only hitting other multipath problems here, especially bug 701371.

works well with repo RHEL6.2-20110822.5

Beaker job: https://beaker.engineering.redhat.com/jobs/122949

Created attachment 526110 [details]
Kickstart file


------- Comment (attachment only) From anibalca.ibm.com 2011-10-03 14:32 EDT-------

------- Comment From ranittal.ibm.com 2011-10-04 02:46 EDT-------
Externalising the comment.

Tested on RHEL6.2 beta:

The following error was found while parsing the kickstart configuration file:
???                                                                  ???
??? The following problem occurred on line 21 of the kickstart file: ???
???                                                                  ???
??? Specified nonexistent disk mpathe in partition command

It seems it still doesn't work.

Hello IBM,
Please confirm the version you tested was anaconda-13.21.119-1 or above.

Thank You
Joe Kachuck

------- Comment From vahegde1.ibm.com 2011-10-10 01:13 EDT-------
We have verified on RHEL6.1 Beta ( anaconda-13.21.140-1.el6)

Thanks
Vasant

------- Comment From vahegde1.ibm.com 2011-10-11 08:27 EDT-------
(In reply to comment #31)
> We have verified on RHEL6.1 Beta ( anaconda-13.21.140-1.el6)

Sorry for the typo .. We have verified on RHEL6.2 Beta.

(In reply to comment #47)
> ------- Comment From vahegde1.ibm.com 2011-10-11 08:27 EDT-------
> (In reply to comment #31)
> > We have verified on RHEL6.1 Beta ( anaconda-13.21.140-1.el6)
> Sorry for the typo .. We have verified on RHEL6.2 Beta.

Hi Red Hat, To be clear IBM has verified that the defect is still reproducing on the 6.2 beta at anaconda-13.21.140-1.el6

(In reply to comment #48)
> (In reply to comment #47)
> > ------- Comment From vahegde1.ibm.com 2011-10-11 08:27 EDT-------
> > (In reply to comment #31)
> > > We have verified on RHEL6.1 Beta ( anaconda-13.21.140-1.el6)
> > Sorry for the typo .. We have verified on RHEL6.2 Beta.
> 
> Hi Red Hat, To be clear IBM has verified that the defect is still reproducing
> on the 6.2 beta at anaconda-13.21.140-1.el6

If that's so please open a new bug, attaching the kickstart and all files from /tmp/*log.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1565.html