Bug 597223

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

Steps to reproduce:
1. start graphical installation on s390x, build RHEL6.0-20100527.2
2. at the Specialized storage devices screen, try to add two zFCP drives (one multipath device):

   devno    WWPN               LUN
1) 0.0.A000 0x50050763050B073D 0x4020400100000000
2) 0.0.A001 0x50050763050B473D 0x4020400100000000

3. the first one is added correctly
4. when addidng the second one, traceback is thrown

If the multipath device is specified in CMS config file, the multipath device is recognized without any traceback.

When you hit this exception, can you go to pdb and paste the output of the following:

(pdb) print self._lst
(pdb) up
(pdb) print new_mpaths

When I hit the issue (running vnc installation) and click on Debug button, I see this in ssh terminal:

Entering debugger...
> /usr/lib/anaconda/iw/filter_gui.py(69)__contains__()
-> return item["name"] in iter(self)
(Pdb)

But I can't write anything there, terminal doesn't respond, anaconda in vnc is still running and I can even finish the installation.

How else could I start the debugger to provide you more details?

You're going to need to sit in front of the machine.

Sitting in front of the machine provides exactly the same default line-mode terminal as remotely. There is no framebuffer or VTs on s390x.
With the following patch in place, pdb would be usable on the ssh terminal that was used to login into the install image as user install:
https://www.redhat.com/archives/anaconda-devel-list/2009-October/msg00058.html

Jan, maybe you can try booting with "serial" in the parm file and hope setting the serial flag makes pdb work?

Jan,

Boot with updates=http://dcantrel.fedorapeople.org/updates/597223.img which includes the patch mentioned in comment #7.  Then try doing what Chris asked for in comment #4.

(In reply to comment #4)

Using RHEL6.0-20100523.0, anaconda-13.21.45-1.el6.s390x plus updates.img to get HEAD of rhel6-branch plus some private fixes to loader and some python stuff for other bugs, including patch of comment 7:

(Pdb) print self._lst
[]
(Pdb) up
> /usr/lib/anaconda/iw/filter_gui.py(438)<lambda>()
-> mpaths = filter(lambda d: d not in self._cachedMPaths, new_mpaths)
(Pdb) print new_mpaths
*** NameError: name 'new_mpaths' is not defined
(Pdb) print self._cachedMPaths
<FilterWindow.NameCache object at 0x81d89cc0>
(Pdb) up
> /usr/lib/anaconda/iw/filter_gui.py(438)_add_advanced_clicked()
-> mpaths = filter(lambda d: d not in self._cachedMPaths, new_mpaths)
(Pdb) print new_mpaths
[[{'ID_WWN_WITH_EXTENSION': '0x6005076303ffc56200000000000010ee', 'ID_PART_TABLE_TYPE': 'dos', 'ID_REVISION': '.280', 'ID_PATH': 'ccw-0.0.3c1b-zfcp-0x500507630300c562:0x401040ee00000000', 'ID_VENDOR_ENC': 'IBM\\x20\\x20\\x20\\x20\\x20', 'ANACBIN': '/usr/sbin', 'ID_VENDOR': 'IBM', 'ID_SERIAL': '36005076303ffc56200000000000010ee', 'DEVTYPE': 'disk', 'ID_BUS': 'scsi', 'ID_MPATH_NAME': 'mpatha', 'ID_FS_TYPE': 'multipath_member', 'DEVLINKS': '/dev/block/8:0 /dev/disk/by-id/scsi-36005076303ffc56200000000000010ee /dev/disk/by-path/ccw-0.0.3c1b-zfcp-0x500507630300c562:0x401040ee00000000 /dev/disk/by-id/wwn-0x6005076303ffc56200000000000010ee', 'SUBSYSTEM': 'block', 'ID_MODEL': '2107900', 'ID_SERIAL_SHORT': '6005076303ffc56200000000000010ee', 'sysfs_path': '/devices/css0/0.0.0015/0.0.3c1b/host0/rport-0:0-21/target0:0:21/0:0:21:1089355792/block/sda', 'ID_TYPE': 'disk', 'ID_WWN_VENDOR_EXTENSION': '0x00000000000010ee', 'MINOR': '0', 'symlinks': ['/dev/block/8:0', '/dev/disk/by-id/scsi-36005076303ffc56200000000000010ee', '/dev/disk/by-path/ccw-0.0.3c1b-zfcp-0x500507630300c562:0x401040ee00000000', '/dev/disk/by-id/wwn-0x6005076303ffc56200000000000010ee'], 'MAJOR': '8', 'ID_WWN': '0x6005076303ffc562', 'name': 'sda', 'DEVPATH': '/devices/css0/0.0.0015/0.0.3c1b/host0/rport-0:0-21/target0:0:21/0:0:21:1089355792/block/sda', 'UDEV_LOG': '3', 'DEVNAME': 'sda', 'MPATH_GOT_HERE': '2', 'ID_SCSI_SERIAL': '759246110EE', 'ID_MODEL_ENC': '2107900\\x20\\x20\\x20\\x20\\x20\\x20\\x20\\x20\\x20', 'ID_SCSI': '1'}, {'ID_WWN_WITH_EXTENSION': '0x6005076303ffc56200000000000010ee', 'ID_PART_TABLE_TYPE': 'dos', 'ID_REVISION': '.280', 'ID_PATH': 'ccw-0.0.3c5b-zfcp-0x500507630313c562:0x401040ee00000000', 'ID_VENDOR_ENC': 'IBM\\x20\\x20\\x20\\x20\\x20', 'ANACBIN': '/usr/sbin', 'ID_VENDOR': 'IBM', 'ID_SERIAL': '36005076303ffc56200000000000010ee', 'DEVTYPE': 'disk', 'ID_BUS': 'scsi', 'ID_MPATH_NAME': 'mpatha', 'ID_FS_TYPE': 'multipath_member', 'DEVLINKS': '/dev/block/8:32 /dev/disk/by-id/scsi-36005076303ffc56200000000000010ee /dev/disk/by-path/ccw-0.0.3c5b-zfcp-0x500507630313c562:0x401040ee00000000 /dev/disk/by-id/wwn-0x6005076303ffc56200000000000010ee', 'SUBSYSTEM': 'block', 'ID_MODEL': '2107900', 'ID_SERIAL_SHORT': '6005076303ffc56200000000000010ee', 'sysfs_path': '/devices/css0/0.0.0035/0.0.3c5b/host1/rport-1:0-27/target1:0:27/1:0:27:1089355792/block/sdc', 'ID_TYPE': 'disk', 'ID_WWN_VENDOR_EXTENSION': '0x00000000000010ee', 'MINOR': '32', 'symlinks': ['/dev/block/8:32', '/dev/disk/by-id/scsi-36005076303ffc56200000000000010ee', '/dev/disk/by-path/ccw-0.0.3c5b-zfcp-0x500507630313c562:0x401040ee00000000', '/dev/disk/by-id/wwn-0x6005076303ffc56200000000000010ee'], 'MAJOR': '8', 'ID_WWN': '0x6005076303ffc562', 'name': 'sdc', 'DEVPATH': '/devices/css0/0.0.0035/0.0.3c5b/host1/rport-1:0-27/target1:0:27/1:0:27:1089355792/block/sdc', 'UDEV_LOG': '3', 'DEVNAME': 'sdc', 'MPATH_GOT_HERE': '2', 'ID_SCSI_SERIAL': '759246110EE', 'ID_MODEL_ENC': '2107900\\x20\\x20\\x20\\x20\\x20\\x20\\x20\\x20\\x20', 'ID_SCSI': '1'}]]

Looks like this is a list instead of a hash table.

# cat /etc/multipath.conf 
# multipath.conf written by anaconda

defaults {
        user_friendly_names yes
}
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z]"
        devnode "^dcssblk[0-9]*"
        device {
                vendor "DGC"
                product "LUNZ"
        }
        device {
                vendor "IBM"
                product "S/390.*"
        }
        # don't count normal SATA devices as multipaths
        device {
                vendor  "ATA"
        }
        # don't count 3ware devices as multipaths
        device {
                vendor  "3ware"
        }
        device {
                vendor  "AMCC"
        }
        # nor highpoint devices
        device {
                vendor  "HPT"
        }
}
multipaths {
}

# multipath -ll -v3
May 30 02:49:03 | ram0: device node name blacklisted
May 30 02:49:03 | ram1: device node name blacklisted
May 30 02:49:03 | ram2: device node name blacklisted
May 30 02:49:03 | ram3: device node name blacklisted
May 30 02:49:03 | ram4: device node name blacklisted
May 30 02:49:03 | ram5: device node name blacklisted
May 30 02:49:03 | ram6: device node name blacklisted
May 30 02:49:03 | ram7: device node name blacklisted
May 30 02:49:03 | ram8: device node name blacklisted
May 30 02:49:03 | ram9: device node name blacklisted
May 30 02:49:03 | ram10: device node name blacklisted
May 30 02:49:03 | ram11: device node name blacklisted
May 30 02:49:03 | ram12: device node name blacklisted
May 30 02:49:03 | ram13: device node name blacklisted
May 30 02:49:03 | ram14: device node name blacklisted
May 30 02:49:03 | ram15: device node name blacklisted
May 30 02:49:03 | loop0: device node name blacklisted
May 30 02:49:03 | loop1: device node name blacklisted
May 30 02:49:03 | loop2: device node name blacklisted
May 30 02:49:03 | loop3: device node name blacklisted
May 30 02:49:03 | loop4: device node name blacklisted
May 30 02:49:03 | loop5: device node name blacklisted
May 30 02:49:03 | loop6: device node name blacklisted
May 30 02:49:03 | loop7: device node name blacklisted
May 30 02:49:03 | dasdb: not found in pathvec
May 30 02:49:03 | dasdb: mask = 0x5
May 30 02:49:03 | dasdb: dev_t = 94:4
May 30 02:49:03 | dasdb: size = 14424480
May 30 02:49:03 | dasdb: subsystem = ccw
May 30 02:49:03 | dasdb: vendor = IBM
May 30 02:49:03 | dasdb: product = S/390 DASD ECKD
May 30 02:49:03 | dasdb: h:b:t:l = 0:0:60188:0
May 30 02:49:03 | dasdb: get_state
May 30 02:49:03 | dasdb: path checker = directio (controller setting)
May 30 02:49:03 | dasdb: checker timeout = 300000 ms (internal default)
May 30 02:49:03 | dasdb: path offline
May 30 02:49:03 | sda: not found in pathvec
May 30 02:49:03 | sda: mask = 0x5
May 30 02:49:03 | sda: dev_t = 8:0
May 30 02:49:03 | sda: size = 10485760
May 30 02:49:03 | sda: subsystem = scsi
May 30 02:49:03 | sda: vendor = IBM
May 30 02:49:03 | sda: product = 2107900
May 30 02:49:03 | sda: rev = .280
May 30 02:49:03 | sda: h:b:t:l = 0:0:21:1089355792
May 30 02:49:03 | sda: tgt_node_name = 0x5005076303ffc56
May 30 02:49:03 | sda: get_state
May 30 02:49:03 | loading /lib64/multipath/libchecktur.so checker
May 30 02:49:03 | sda: path checker = tur (controller setting)
May 30 02:49:03 | sda: checker timeout = 300000 ms (internal default)
May 30 02:49:03 | sda: state = running
May 30 02:49:03 | sda: state = 3
May 30 02:49:03 | sdb: not found in pathvec
May 30 02:49:03 | sdb: mask = 0x5
May 30 02:49:03 | sdb: dev_t = 8:16
May 30 02:49:03 | sdb: size = 10485760
May 30 02:49:03 | sdb: subsystem = scsi
May 30 02:49:03 | sdb: vendor = IBM
May 30 02:49:03 | sdb: product = 2107900
May 30 02:49:03 | sdb: rev = .280
May 30 02:49:03 | sdb: h:b:t:l = 0:0:21:1089421328
May 30 02:49:03 | sdb: tgt_node_name = 0x5005076303ffc56
May 30 02:49:03 | sdb: get_state
May 30 02:49:03 | sdb: path checker = tur (controller setting)
May 30 02:49:03 | sdb: checker timeout = 300000 ms (internal default)
May 30 02:49:03 | sdb: state = running
May 30 02:49:03 | sdb: state = 3
May 30 02:49:03 | sdc: not found in pathvec
May 30 02:49:03 | sdc: mask = 0x5
May 30 02:49:03 | sdc: dev_t = 8:32
May 30 02:49:03 | sdc: size = 10485760
May 30 02:49:03 | sdc: subsystem = scsi
May 30 02:49:03 | sdc: vendor = IBM
May 30 02:49:03 | sdc: product = 2107900
May 30 02:49:03 | sdc: rev = .280
May 30 02:49:03 | sdc: h:b:t:l = 1:0:27:1089355792
May 30 02:49:03 | sdc: tgt_node_name = 0x5005076303ffc56
May 30 02:49:03 | sdc: get_state
May 30 02:49:03 | sdc: path checker = tur (controller setting)
May 30 02:49:03 | sdc: checker timeout = 300000 ms (internal default)
May 30 02:49:03 | sdc: state = running
May 30 02:49:03 | sdc: state = 3
===== paths list =====
uuid hcil              dev   dev_t pri dm_st chk_st vend/prod/rev       dev_st
     0:0:60188:0       dasdb 94:4  -1  undef faulty IBM,S/390 DASD ECKD runnin
     0:0:21:1089355792 sda   8:0   -1  undef ready  IBM,2107900         runnin
     0:0:21:1089421328 sdb   8:16  -1  undef ready  IBM,2107900         runnin
     1:0:27:1089355792 sdc   8:32  -1  undef ready  IBM,2107900         runnin

# lsscsi -t
[0:0:21:1089355792]disk    fc:0x500507630300c562,0x656500  /dev/sda 
[0:0:21:1089421328]disk    fc:0x500507630300c562,0x656500  /dev/sdb 
[1:0:27:1089355792]disk    fc:0x500507630313c562,0x656000  /dev/sdc 
# lszfcp -DV
/sys/devices/css0/0.0.0015/0.0.3c1b/0x500507630300c562/0x401040ee00000000
/sys/bus/ccw/drivers/zfcp/0.0.3c1b/host0/rport-0:0-21/target0:0:21/0:0:21:1089355792
/sys/devices/css0/0.0.0015/0.0.3c1b/0x500507630300c562/0x401040ef00000000
/sys/bus/ccw/drivers/zfcp/0.0.3c1b/host0/rport-0:0-21/target0:0:21/0:0:21:1089421328
/sys/devices/css0/0.0.0035/0.0.3c5b/0x500507630313c562/0x401040ee00000000
/sys/bus/ccw/drivers/zfcp/0.0.3c5b/host1/rport-1:0-27/target1:0:27/1:0:27:1089355792

Created attachment 418252 [details]
print new_mpaths

In reply to comment #4, with updates img from comment #8

Well, the second value being returned by identifyMultipaths is a list of list, and not just a list.  I need to do some work on stashing the right things into the name cache.

*** Bug 602206 has been marked as a duplicate of this bug. ***

Created attachment 427873 [details]
Attached traceback automatically from anaconda.

I don't have the hardware to test this, so there's likely to be a little back-and-forth on this one.  However, I have what looks like the basis of a fix if you could try it out.  I'll attach an updates.img for testing as well as the patch itself so everyone else can see what I'm going for here.

Created attachment 428560 [details]
patch to fix adding new multipath devices

And the updates image is updates=http://clumens.fedorapeople.org/597223.img

Thanks for the updates image, Chris.
Good news - there is no traceback when both paths to FCP LUN are added via GUI. Multipath LUN is recognized and user can see it at the Multipath tab. Also installation is working fine (I stopped it during installing of packages).

One issue is there: when I add both paths and the multipath LUN is shown, I still see the first path at the "Other SAN Devices" tab together with the multipath LUN. Anaconda should hide/remove the first path and show only the multipath LUN. This is only when adding paths via GUI. When paths are specified in CMS config file:
FCP_1="0.0.A000 0x500507630503C73D 0x4020400100000000"
FCP_2="0.0.A100 0x500507630513C73D 0x4020400100000000"
then the "Other SAN Devices" tab contains only the multipath LUN.

Hm, not sure if that is related to this bug or not.  It's suspicious, though.

Can you attach some screenshots of what you mean?  Also, the output of udevadm info --export-db after you have used the GUI to add the LUNs would be handy.

Created attachment 429981 [details]
FCP defined in CMS config file

This is correct behaviour, one multipath device at both "Multipath Devices" and Other SAN Devices" tabs.

Created attachment 429984 [details]
FCPs added via GUI

This is when adding FCP LUNs via GUI. See the "Other SAN Devices" tab. There is a multipath LUN (60:05:...) and one single path FCP (ccw-0.0.a000-zfcp-...). Device ccw-0.0.a000-zfcp... is the first path of the multipath LUN.

Created attachment 429986 [details]
udevadm info --export-db

Created attachment 430065 [details]
Attached traceback automatically from anaconda.

Okay, let me restate the above in very simple words so I make sure I understand you.

You have two FCP devices that are part of a multipath.
You add the first via the "Add Advanced Storage" button and it's put into its proper place in the UI.
You add the second via the same button, and then both the multipath device and the first new storage device are shown in the UI.

Correct?

I think I have a fix for that situation.  Can you please try again with the same updates location?

*** Bug 612353 has been marked as a duplicate of this bug. ***

http://clumens.fedorapeople.org/597223.img   Does not fix the issue for FCoE multipath that was reported in Bug 612353.

Anaconda log attached.

Created attachment 430429 [details]
anaconda logs from 597223.img

Ross - that's because you are testing anaconda-13.21.50 with an updates image built on top of anaconda-12.21.56.  There are changes in 13.21.52 that are not reflected in your tree that would require further files in the updates.img.

In short, you can't really use the above updates image unless you are using a tree with the absolute latest anaconda in it.

Could we get an updates image with a fix that works with Snapshot7?

Multipath support is required for EMC I&QP.

Jan - I wonder if you could walk me through how you're testing this so I could do it myself and save a bunch of these long round trips for stupid mistakes in the patch?

Jan - on second thought, let's deal with this as a new bug report.  Other people are seeing the initial traceback (see bug 612535 duped to this one) so I'd like to get that fix committed.  Relatively fewer people are going to hit the second problem of having the UI populated incorrectly when using the "Add Advanced" button for multipath.  I'll continue to work on the second problem, of course.

Does that work for you?

Hello,

I agree with you, please fix the original bug and I'll try to reproduce the second one (Jan is on PTO for 2 weeks starting today). I'll create a new bug if I'm successful.

Thanks.

I'm no longer able to reproduce the initial bug report with anaconda-13.21.60-1.el6, build RHEL6.0-20100722.0. Moving this bug to VERIFIED.

There is a new bug report (bug 618535) to fix the second issue from comment 19.

Chris, ping me on IRC, I can show you how to test the second issue yourself.

Unable to reproduce with RHEL6.0-20100722.0-Server-x86_64-DVD1.iso

I am able to add two FCoE interfaces and install to a multipath LUN (mpatha).

Thanks.

Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.