This BZ is to address the following portion of the original BZ (dracut not mkinitrd) 


> - Patch mkinitrd (for root devices only) / initscripts (for all other devices)
> to:
>    Unmask and wait for appearance of devices needed which are enumerated in
> already existing config files such as modprobe.conf (options dasd_mod
> dasd=...), zfcp.conf (1st column) and ifcfg-* (SUBCHANNELS=...) where device
> numbers are listed (doing so automatically handles all cases of device
> configuration during installation, post-installation or manual editing of
> config files)

This needs to be filed as an mkinitrd bug if it hasn't already.

My suggestion would be to introduce three new scripts znet_cio_free, dasd_cio_free, and zfcp_cio_free, that can be called by both dracut (for root-fs device dependencies) and sysV-init/upstart (for non-rootfs devices).
All three scripts parse their corresponding config files from /etc and dynamically free each device bus ID by means of /proc/cio_ignore. After the scripts have done their job, the already existing configuration mechanisms just work as they have been doing ever since.

For dracut, the scripts should be called probably before returning from the respective module parse function of the dracut command line, since the removal of devices from the cio blacklist has to happen before udev or anybody else tries to do something with those device bus IDs.
For sysV-init/upstart, the scripts should be called at an appropriate place and before udev is started or udev is triggered respectively.

znet_cio_free should parse /etc/sysconfig/network-scripts/ifcfg-* similar to /lib/udev/ccw_init.

dasd_cio_free should parse $(modprobe --showconfig | grep "options[[:space:]]\+dasd_mod") [see anaconda's loader/linuxrc.s390:parse_dasd] and /etc/dasd.conf [see dracut's modules.d/95dasd/dasdconf.sh].

zfcp_cio_free should parse /etc/zfcp.conf [see s390utils-base's /sbin/zfcpconf.sh].

Since both sysV-init and dracut use the same config files, they can transparently call the three new scripts. Also the user is free to manually modify the config files without having to rerun mkinitrd or dracut-gencmdline. Additionally, system-config tools may simply write config files without modifications except for the case where they dynamically configure s390 devices including setting online. For the latter case, they need support to dynamically free devices from the cio blacklist before trying to configure those devices.

Created attachment 375017 [details]
script for unblocking DASD and ZFCP devices

with added dasd_cio_free and zfcp_cio_free symlinks it should work for DASD and ZFCP devices

Comment on attachment 375017 [details]
script for unblocking DASD and ZFCP devices

>#!/bin/sh
>
># unblock devices listed in various config files
>#
># it uses dasd and zfcp config file
># config file syntax:
># deviceno   options
># or
># deviceno   WWPN   FCPLUN
>#
>
>DASDCONFIG=/etc/dasd.conf
>ZFCPCONFIG=/etc/zfcp.conf
>BLACKLIST=/proc/cio_ignore
>PATH=/bin:/usr/bin:/sbin:/usr/sbin
>
>CMD=`basename $0`
>if [ $CMD = "dasd_cio_free" ]; then
>    CONFIG=$DASDCONFIG
>    MODE=dasd
>elif [ $CMD = "zfcp_cio_free" ]; then
>    CONFIG=$ZFCPCONFIG
>    MODE=zfcp
>else
>    echo "Unknown alias '$CMD'"
>    exit 1
>fi

Switch case construct instead of if-elif chain?

>
>if [ ! -f $BLACKLIST ]; then

Could we give the user a hint why we exit here?

>    exit 2
>fi
>
>if [ $MODE = "dasd" -o $MODE = "zfcp" ]; then
>    # process the config file
>    if [ -f "$CONFIG" ]; then
>        while read line; do
>	    case $line in
>		\#*) ;;
>		*)
>		    [ -z "$line" ] && continue
>		    set $line
>		    DEVICE=$1
>		    echo "freeing device $DEVICE"
>		    echo "free $DEVICE" > $BLACKLIST 2> /dev/null

Writing to $BLACKLIST can return with an errno. I don't know how we should show such error cases. They might be syntax errors from users in the config files and reporting would help the user.

>		    ;;
>	    esac
>	done < $CONFIG
>    fi
>fi
>
>if [ $MODE = "dasd" ]; then
>    # process the device list defined as option for the dasd module
>    DEVICES=`modprobe --showconfig | grep "options[[:space:]]\+dasd_mod" | cut -d "=" -f 2`

echo "freeing devices $DEVICES"

>    echo "free $DEVICES" > $BLACKLIST 2> /dev/null

[1] Chapter 3, Setting up the DASD device driver, Module parameters, page 34

Valid example entry in modprobe.conf:
options   dasd_mod  eer_pages=5 dasd=nopav,100(ro:diag),105-10f,0.0.beed-0.0.beef(erplog),nofcx,0.0.1234(erplog:failfast),0.1.5678

The above script code using grep and cut is not specific enough in parsing:
5 dasd

The idea to make use of /proc/cio_ignore's capability to eat sequences of device ranges and just take them from modprobe.conf seems nice. In order to get something that is digestible by
[1] Chapter 44, cio_ignore, page 513,
we could use the following, if we really wanted to do everything with sed:
DEVICES=$(modprobe --showconfig | grep "options[[:space:]]\+dasd_mod" | sed -e 's/.*[[:space:]]dasd=\([^[:space:]]*\).*/\1/' -e 's/([^)]*)//g' -e 's/nopav\|nofcx\|autodetect\|probeonly//g' -e 's/,,/,/g' -e 's/^,//' -e 's/,$//')

>fi

[1] Device Drivers, Features, and Commands (kernel 2.6.31) - SC33-8411-03, September 2009
http://www.ibm.com/developerworks/linux/linux390/documentation_dev.html
http://download.boulder.ibm.com/ibmdl/pub/software/dw/linux390/docu/lk31dd03.pdf

Created attachment 376864 [details]
script for unblocking DASD and ZFCP devices v2

updated script uploaded, thanks for the comments, Steffen

changes:
- use case instead of if-elif chain
- print some message what's going on when run with --verbose
- check error code after writing to /proc/cio_ignore
- use the sed command from Steffen to parse the dasd module options

Attachment 376864 [details] looks good. Thanks a lot for the update, Dan.
Assuming this has been or will be function tested successfully, let's integrate it into sysV-init/upstart and dracut.
Please note, that this is so far only sufficient to get access to non-network disk drives. We still need support for the znet_cio_free part for network to function, most notably for ssh login (and potentially for network (root-)fs).

I am aware of the fact that this script solves only 2/3 of the problem. The znet_cio_free script is still in development, but will be ready in few days. And I tend to making it also a part of the "device_cio_free" script, though the data it should process are quite different.

Created attachment 377744 [details]
script for unblocking DASD and ZFCP and ZNET devices v3

new version that adds
- support for freeing networking channels
- verbose output enabled with command line switch

Comment on attachment 377744 [details]
script for unblocking DASD and ZFCP and ZNET devices v3

Thanks! Basically it looks good, I only have a few minor comments below.

>if [ $MODE = "znet" ]; then
>    # process the config file
>    if [ -f "$CONFIG" ]; then
>        while read line; do
>	    case $line in
>		\#*) ;;
>		*)
>		    [ -z "$line" ] && continue
>		    # grep 2 or 3 channels from beginning of each line
>		    DEVICES=$(echo $line | egrep -i -o "^([0-9]\.[0-9]\.[a-f0-9]+,){1,2}([0-9]\.[0-9]\.[a-f0-9]+)")

I'm not sure about matching case-insensitively. In fact, the drivers only accept lower case letters for xdigits. Unfortunately, it's a trap users fall in, when copying&pasting devnos from z/VM, which writes out upper case letters in xdigits. But then again, we could transform whatever the user specifies to lower case. Unfortunately, this is not the only place where such a more flexible acceptance scheme would have to be implemented to work all the way through. I guess what I'm trying to say is, that we should probably accept either lower case only or mixed case and transform it to lower case everywhere to be consistent.

>		    if [ $DEVICES ]; then
>			[ $VERBOSE ] && echo "Freeing device(s) $DEVICES"
>			echo "free $DEVICES" > $BLACKLIST 2> /dev/null || echo "Error: can't free device(s) $DEVICES"
>		    fi
>		    ;;
>	    esac
>	done < $CONFIG
>    fi
>    # process channels from network interface configurations
>    for line in $(egrep -i -h "^[[:space:]]*SUBCHANNELS=([0-9]\.[0-9]\.[a-f0-9]+,){1,2}([0-9]\.[0-9]\.[a-f0-9]+)([[:space:]]+#|[[:space:]]*$)" /etc/sysconfig/network-scripts/ifcfg-* 2> /dev/null)

Here, I would have the same note as above regarding case-insensitive matching.

It would be nice, if the regex also matched if the user quoted (single or double) the right hand side of the assignment to the SUBCHANNELS variable. Anaconda does not write out quotes here, but the user might add them when manually editing an ifcfg file.

>    do
>	eval $line

Is this always safe, no matter what the content of $line is, or do we need to escape the dollar sign for variable expansion?:
eval \$line
(I had to do the latter in function ask() of anaconda's linuxrc.s390 but I'm not sure this even works here as well.)

>        [ $VERBOSE ] && echo "Freeing device(s) $SUBCHANNELS"
>	echo "free $SUBCHANNELS" > $BLACKLIST 2> /dev/null  || echo "Error: can't free device(s) $SUBCHANNELS"
>    done
>fi

(In reply to comment #12)
> (From update of attachment 377744 [details])
> Thanks! Basically it looks good, I only have a few minor comments below.
> 
> >if [ $MODE = "znet" ]; then
> >    # process the config file
> >    if [ -f "$CONFIG" ]; then
> >        while read line; do
> >	    case $line in
> >		\#*) ;;
> >		*)
> >		    [ -z "$line" ] && continue
> >		    # grep 2 or 3 channels from beginning of each line
> >		    DEVICES=$(echo $line | egrep -i -o "^([0-9]\.[0-9]\.[a-f0-9]+,){1,2}([0-9]\.[0-9]\.[a-f0-9]+)")
> 
> I'm not sure about matching case-insensitively. In fact, the drivers only
> accept lower case letters for xdigits. Unfortunately, it's a trap users fall
> in, when copying&pasting devnos from z/VM, which writes out upper case letters
> in xdigits. But then again, we could transform whatever the user specifies to
> lower case. Unfortunately, this is not the only place where such a more
> flexible acceptance scheme would have to be implemented to work all the way
> through. I guess what I'm trying to say is, that we should probably accept
> either lower case only or mixed case and transform it to lower case everywhere
> to be consistent.

new version has the input lowercased before going to search the regex, "-i" option is removed

> >		    if [ $DEVICES ]; then
> >			[ $VERBOSE ] && echo "Freeing device(s) $DEVICES"
> >			echo "free $DEVICES" > $BLACKLIST 2> /dev/null || echo "Error: can't free device(s) $DEVICES"
> >		    fi
> >		    ;;
> >	    esac
> >	done < $CONFIG
> >    fi
> >    # process channels from network interface configurations
> >    for line in $(egrep -i -h "^[[:space:]]*SUBCHANNELS=([0-9]\.[0-9]\.[a-f0-9]+,){1,2}([0-9]\.[0-9]\.[a-f0-9]+)([[:space:]]+#|[[:space:]]*$)" /etc/sysconfig/network-scripts/ifcfg-* 2> /dev/null)
> 
> Here, I would have the same note as above regarding case-insensitive matching.

I keep the "-i" option, because the lowercasing is done after the SUBCHANNEL variable is read and before going to the ?cio_ignore"
 
> It would be nice, if the regex also matched if the user quoted (single or
> double) the right hand side of the assignment to the SUBCHANNELS variable.
> Anaconda does not write out quotes here, but the user might add them when
> manually editing an ifcfg file.

added, the regex will catch also strings with mismatched or missing quotes, but an error will be issued later
 
> >    do
> >	eval $line
> 
> Is this always safe, no matter what the content of $line is, or do we need to
> escape the dollar sign for variable expansion?:
> eval \$line
> (I had to do the latter in function ask() of anaconda's linuxrc.s390 but I'm
> not sure this even works here as well.)

the "line" variable contains only the string "SUBCHANNELS=...." grepped from the config files, so I think it's correct to use "eval $line"
 
> >        [ $VERBOSE ] && echo "Freeing device(s) $SUBCHANNELS"
> >	echo "free $SUBCHANNELS" > $BLACKLIST 2> /dev/null  || echo "Error: can't free device(s) $SUBCHANNELS"
> >    done
> >fi

Created attachment 378191 [details]
script for unblocking DASD and ZFCP and ZNET devices v4

(In reply to comment #14)
> Created an attachment (id=378191) [details]
> script for unblocking DASD and ZFCP and ZNET devices v4  

Thank you for the update. Regarding the unmasking the script is very good.
However, I was reminded of the part "waiting" for the appearance of devices that have just been unmasked.

Since udev and its rules rely on devices being sensed completely, *_cio_free must wait for the appearance of devices after unmasking them. Since we do not really know if device bus IDs will actually be sensed after unmasking them (they may just not be configured in the system at all), this probably has to be done with some timeout mechanism and polling for the device appearance on the ccw bus.
See also https://bugzilla.redhat.com/show_bug.cgi?id=463544#c12.

Steffen, thanks for reminding the "wait" part, but I am not sure where it should belong and also how the whole device initialization process actually looks like. Can you, please, give me some pointers or write it here?

- lets start with all devices ignored
- issue "free <device>" to /proc/cio_ignore
- a bit corresponding to a device is switched in the kernel structure and a rescan is started by css_schedule_reprobe()

And my questions are
- when does an udev event apear that leads to the (dasd,ccw,zfcp)conf.sh scripts
- how can I check device is fully initialized

Dan,

here's the flow of kernel functions following a blacklist free operation.
(* denotes deferred work).

# echo free > /proc/cio_ignore
cio_ignore_write
blacklist_parse_proc_parameters
css_schedule_reprobe
css_schedule_eval_all_unreg

        (work item is scheduled)

*css_slow_path_func
slow_eval_unknown_fn
css_evaluate_new_subchannel
css_probe_device
css_register_subchannel
css_sch_device_register
device_register

        After this, there is an object in sysfs for the
        subchannel (the future parent of the device we're
        looking for), but no kobject event will be
        generated as it is suppressed until later.

css_probe
io_subchannel_probe
css_schedule_eval

        (work item is scheduled)

*css_slow_path_func
slow_eval_known_fn
css_evaluate_known_subchannel
io_subchannel_sch_event
sch_create_and_recog_new_device
io_subchannel_recog
ccw_device_recognition
ccw_device_sense_id_start

        (I/O takes place)

*ccw_device_sense_id_done
ccw_device_recog_done
io_subchannel_recog_done
ccw_device_sched_todo

        (work item is scheduled)

*ccw_device_todo
io_subchannel_register

        Here, the KOBJ_ADD event for the subchannel is
        generated.

ccw_device_register
device_add

        After this, there is an object in sysfs for the
        device and a KOBJ_ADD event for the device is
        generated.

ccw_device_probe

        At this point, device type specific setup
        continues, e.g. DASD will add its own sysfs
        attributes and try to set the device online when
        a dasd= module parameter has been specified.
        Once the probe function finishes (assuming no
        module parameters were specified), the device is
        fully initialized.

I'm assuming that the *conf.sh scripts are run when the second UEVENT is processed. As you can see, there is a theoretical race condition between the second UEVENT and the completion of ccw_device_probe, but this condition also exists for the *conf.sh scripts so I guess that can be disregarded.

The biggest problem here is that you cannot wait for "cio_ignore processing has finished, but there are no new devices". That's where timeout processing comes into play. In a worst case scenario, both I/O and deferred kernel work can delay processing for an almost arbitrary amount of time, so there's no perfect timeout period. It might be a good idea to make this value user configurable somehow. Good starting values could be 10 to 15 seconds.

Here's my suggestion for the resulting wait logic:
        echo free <device id> > /proc/cio_ignore
        while /sys/bus/ccw/devices/<device id>/online does not exist and
              retry count hasn't been exhausted; do
                decrease retry count
                # maybe inform user about the reason for the delay when
                # we're waiting for more than 3 seconds
                wait for ADD UEVENT for any device or 1 second
        done

If you're issuing multiple blacklist free requests, it might be a good idea to do the waiting once for all device IDs at the end, or otherwise booting could be delayed for 1 second per freed device.

As a side note: we're currently working on a new kernel feature that will offer a replacement for this user space wait logic in the form of a single command: echo > /proc/cio_settle. When this command returns, processing for all previous blacklist operations is guaranteed to have finished.

(In reply to comment #18)
> As a side note: we're currently working on a new kernel feature that will offer
> a replacement for this user space wait logic in the form of a single command:
> echo > /proc/cio_settle. When this command returns, processing for all previous
> blacklist operations is guaranteed to have finished.  

Peter, first thanks a lot for the detailed description, I think it will be useful not only for me.

The new feature won't be available in the RHEL-6 kernels or is a backport planned?

(In reply to comment #19)
> The new feature won't be available in the RHEL-6 kernels or is a backport
> planned?  

This feature is not planned for RHEL-6 - the kernel feature submission deadline for RHEL-6 GA has long passed and even if there was an exception, we're not yet in a state where we can submit the code for inclusion. There is a chance that we'll submit it for one of the RHEL6 updates though.

Ok, it makes sense.

And before I start to reinvent the wheel and because you are the author of the  do you think it's possible to use(In reply to comment #20)
> (In reply to comment #19)
> > The new feature won't be available in the RHEL-6 kernels or is a backport
> > planned?  
> 
> This feature is not planned for RHEL-6 - the kernel feature submission deadline
> for RHEL-6 GA has long passed and even if there was an exception, we're not yet
> in a state where we can submit the code for inclusion. There is a chance that
> we'll submit it for one of the RHEL6 updates though.  

Ok, it makes sense.

And before I start to reinvent the wheel and because you are the author of some tools in s390-tools, do you think it's possible to use some tool directly or as a library of functions? Mainly I am interested in converting of ids to the canonical x.y.zzzz form and iterating thru device ranges.

My plan is to collect the device ids from the whole run of the script and do the availability checks at the end of the script.

(In reply to comment #21)
> And before I start to reinvent the wheel and because you are the author of some
> tools in s390-tools, do you think it's possible to use some tool directly or as
> a library of functions? Mainly I am interested in converting of ids to the
> canonical x.y.zzzz form and iterating thru device ranges.
> 
> My plan is to collect the device ids from the whole run of the script and do
> the availability checks at the end of the script.  

Unfortunately there are currently no tools for converting and iterating device IDs. Perhaps you could have a look at function "check_id()" in the lscss tool for a pointer on how to parse the device ID.

What I do recommend though, is using the cio_ignore tool from the s390-tools package: it will accept all sorts of identifier formats and convert them internally to the format as required by the /proc/cio_ignore interface, e.g.

  cio_ignore --remove 12,78,0.0.0100,8000,7000-7010

will work.

Also when the kernel item I mentioned will be available, we will also patch the cio_ignore tool to use that "settle" interface whenever a change to the blacklist was made.

Created attachment 379649 [details]
 script for unblocking DASD and ZFCP and ZNET devices v5

Hopefully a complete implementation with the waiting part added. Waiting/check times are 0, 1, 3, 6, 10 and 15 seconds and can be configured thru the "for" cycle in the wait_for_device function.

hmm, this script seems to need bash:

$ dash -n ~/Desktop/device_cio_free.sh 
/home/harald/Desktop/device_cio_free.sh: 176: Syntax error: redirection unexpected

and maybe we can replace "egrep" with "grep -E" to save binaries to pull in.

(In reply to comment #25)
> hmm, this script seems to need bash:
> 
> $ dash -n ~/Desktop/device_cio_free.sh 
> /home/harald/Desktop/device_cio_free.sh: 176: Syntax error: redirection
> unexpected

it looks like the whole "waiting" section is dash-incompatible ...

> and maybe we can replace "egrep" with "grep -E" to save binaries to pull in.  

that's doable

Created attachment 382186 [details]
script for unblocking DASD and ZFCP and ZNET devices v6

this is de-bashified version with a fix for device ranges that could be treated wrongly in the previous version

(In reply to comment #27)
> Created an attachment (id=382186) [details]
> script for unblocking DASD and ZFCP and ZNET devices v6
> 
> this is de-bashified version with a fix for device ranges that could be treated
> wrongly in the previous version  

Looks good!

There appears to be a small error though:

    if [ echo "free $DEV" > $BLACKLIST 2> /dev/null ]; then

I'm not a dash expert, but shouldn't it look more like this?:

    if ! echo "free $DEV" > $BLACKLIST 2> /dev/null; then

(In reply to comment #28)
> (In reply to comment #27)
> > Created an attachment (id=382186) [details] [details]
> > script for unblocking DASD and ZFCP and ZNET devices v6
> > 
> > this is de-bashified version with a fix for device ranges that could be treated
> > wrongly in the previous version  
> 
> Looks good!
> 
> There appears to be a small error though:
> 
>     if [ echo "free $DEV" > $BLACKLIST 2> /dev/null ]; then
> 
> I'm not a dash expert, but shouldn't it look more like this?:
> 
>     if ! echo "free $DEV" > $BLACKLIST 2> /dev/null; then  

Thanks for catching this.

And I have a question again - when does the /sys/bus/ccw/devices/<device id> directory appear for an existing device? Is it (almost) immediately? Because there an issue when the scripts unblocks non-existing devices and then it wants to wait on them and that can take a serious amount of time (nr of such devices * 15 sec timeout). So is there something that can be checked before the script starts waiting for the /sys/bus/ccw/devices/<device id>/online file?

Created attachment 382410 [details]
script for unblocking DASD and ZFCP and ZNET devices v7

this version is present in s390utils-1.8.2-7

changes:
- switch to grep -E
- implemented waiting for individual devices in the ranges
- little fixes

(In reply to comment #29)
> And I have a question again - when does the /sys/bus/ccw/devices/<device id>
> directory appear for an existing device? Is it (almost) immediately? Because
> there an issue when the scripts unblocks non-existing devices and then it wants
> to wait on them and that can take a serious amount of time (nr of such devices
> * 15 sec timeout). So is there something that can be checked before the script
> starts waiting for the /sys/bus/ccw/devices/<device id>/online file?  

The /sys directory for a device may take a potentially arbitrary amount of time to appear. In most cases and on modern hardware, it should be there immediately (<<1s). We've seen cases though, where congestion on a path to the device led to delays of more than 15s, though such long times are very rare.

The 100% solution would be to add a kernel interface through which userspace can wait until all pending kernel work related to CCW device recognition is done. That's what we're working on at the moment, but we're not there yet, so for this release we'll have to work around that problem.

I agree that the total waiting time for multiple non-existent devices might be unacceptably high. Is there a way to consolidate the timeout logic for all devices? If a device isn't present 15 seconds after the script started, there's no use in waiting for it any longer.

Also would it be possible to add a kernel(?) parameter which is parsed by the script through which users can influence the timeout value? In case a customer runs into the mentioned congestion problem, such a parameter could be used to work around the problem of their particular setup.

Peter, thanks for the answer and inspiration. I will change the waiting to accumulate the waiting time from individual devices and after a predefined time (say 60 seconds after sending all "free device" requests) is reached no additional time will spent for the waits.

Created attachment 382441 [details]
script for unblocking DASD and ZFCP and ZNET devices v8

this version is present in s390utils-1.8.2-8

changes:
- reworked the waiting logic to use a global timeout
- little fixes

> if [ $MODE = "dasd" ]; then
>     # process the device list defined as option for the dasd module
>     DEVICES=$(modprobe --showconfig | grep "options[[:space:]]\+dasd_mod" | \
> 	sed -e 's/.*[[:space:]]dasd=\([^[:space:]]*\).*/\1/' -e 's/([^)]*)//g' \
> 	-e 's/nopav\|nofcx\|autodetect\|probeonly//g' -e 's/,,/,/g' -e 's/^,//' -e 's/,$//')
> 
>     free_device $DEVICES

This is the only place, where the user may specify a list of device bus IDs or ranges thereof, that correspond to different devices (the tuple with znet represents one ccwgroup and is therefore not considered multiple devices). The cio_ignore backend called by free_devices parses the comma-separated list of ranges and tries to free each list item during parsing. If the user now specified a list with many valid ranges and at least one invalid range, the kernel returns EINVAL for the entire list but might have successfully freed some valid ranges from the whole list already. Hence device_cio_free won't wait for the valid ranges because nothing is appended to ALL_DEVICES.

Therefore, I suggest to split the list here and call free_device for each range separately:

    for DEVRANGE in $(echo $DEVICES | tr ',' ' ')
    do
        free_device $DEVRANGE
    done

I did a quick test with the following, where the 2nd to last range is intentionally invalid:
options dasd_mod dasd=nopav,A00,0.0.0A02,400-402,0.1.0500-0.0.0503,0.0.fff0-0.0.fFfF

Other than that, the script seems very solid now. Thank you, Dan!

Steffen, thanks for your note, updated script is available in s390utils-1.8.2-9.el6

Harald, I was just curious about the integration of device_cio_free into all s390-specific dracut modules and found dracut commit e1603bf7e404c13b4dd7312901e6153f45b8352a. It introduces the following calls:
modules.d/95dasd_mod/parse-dasd-mod.sh: dasd_cio_free
modules.d/95zfcp/parse-zfcp.sh: zfcp_cio_free
modules.d/95znet/parse-ccw.sh: znet_cio_free

They all seem good, thanks. However, I was wondering, if we needed an additional dasd_cio_free at the end of modules.d/95dasd/parse-dasd.sh. I'm not sure the one in parse-dasd-mod.sh is effective, if the user only specified rd_DASD but no rd_DASD_MOD.

(In reply to comment #39)
> Harald, I was just curious about the integration of device_cio_free into all
> s390-specific dracut modules and found dracut commit
> e1603bf7e404c13b4dd7312901e6153f45b8352a. It introduces the following calls:
> modules.d/95dasd_mod/parse-dasd-mod.sh: dasd_cio_free
> modules.d/95zfcp/parse-zfcp.sh: zfcp_cio_free
> modules.d/95znet/parse-ccw.sh: znet_cio_free
> 
> They all seem good, thanks. However, I was wondering, if we needed an
> additional dasd_cio_free at the end of modules.d/95dasd/parse-dasd.sh. I'm not
> sure the one in parse-dasd-mod.sh is effective, if the user only specified
> rd_DASD but no rd_DASD_MOD.    

The one in parse-dasd-mod.sh is always effective, although we might want to make sure parse-dasd.sh is executed before parse-dasd-mod.sh.

After installation with "cio_ignore=all,!0.0.0009" parameter, system doesn't
have network interface working. I did following installation
(RHEL6.0-20100201.4, anaconda-13.21.8-1, s390utils-1.8.2-9, dracut-004-1):

1. make sure you are using network interface in layer2 mode (bug 561017)
2. edit config file, modify for example "LAYER2=1" to "oAYER2=1" - you need to
go interactive and restart zSeries initrd (see bug 558881 comment 6 for more
details)
3. start installation with "cio_ignore=all,!0.0.0009" in parameter file
4. install with default values
5. restart when the installation is done
6. log in and run "ip a" - no eth0 available:

Ýroot@rtt6 etc¨# ip a
ip a  
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00  
    inet 127.0.0.1/8 scope host lo  
    inet6 ::1/128 scope host   
       valid_lft forever preferred_lft forever  

I will append logs from the system.
Moving back to ASSIGNED.

(In reply to comment #41)
> After installation with "cio_ignore=all,!0.0.0009" parameter, system doesn't
> have network interface working. I did following installation
> (RHEL6.0-20100201.4, anaconda-13.21.8-1, s390utils-1.8.2-9, dracut-004-1):
> 
> 1. make sure you are using network interface in layer2 mode (bug 561017)
> 2. edit config file, modify for example "LAYER2=1" to "oAYER2=1" - you need to
> go interactive and restart zSeries initrd (see bug 558881 comment 6 for more
> details)
> 3. start installation with "cio_ignore=all,!0.0.0009" in parameter file
> 4. install with default values
> 5. restart when the installation is done
> 6. log in and run "ip a" - no eth0 available:
> 
> Ýroot@rtt6 etc¨# ip a
> ip a  
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00  
>     inet 127.0.0.1/8 scope host lo  
>     inet6 ::1/128 scope host   
>        valid_lft forever preferred_lft forever  
> 
> I will append logs from the system.
> Moving back to ASSIGNED.    

This is _not_ a dracut bug, if eth0 is not available at all...

dracut does only configure network interfaces, if you access the root device over network.

I'm sorry, I will file a new bug.
Moving back to ON_QA

(In reply to comment #41)
> 1. make sure you are using network interface in layer2 mode (bug 561017)
> 2. edit config file, modify for example "LAYER2=1" to "oAYER2=1" - you need to
> go interactive and restart zSeries initrd (see bug 558881 comment 6 for more
> details)

I'm not sure what this step does. If this means edit the CMS conf file (or parm
file if not using a conf file), wouldn't this step then in fact disable layer2,
i.e. enable layer3 mode and cause bug 561017?

> I will append logs from the system.

The content of /proc/cio_ignore after booting the installed system would be
interesting and I suppose that it'll show the equivalent of
"cio_ignore=all,!0.0.0900".

This is because we have integrated device_cio_free into dracut to handle rootfs
devices. But I'm not sure the other necessary integration of *_cio_free into
sysV-init/initscripts/upstart has been completed (see comment 1 and comment 2).
If not, then all the udev triggered rules to configure s390 devices after the
rootfs has been mounted won't work since the devices are ignored and cannot
send udev events to trigger the rules for configuration (in this case
/lib/udev/ccw_init for network).

If so, maybe a clone for initscripts or the like would help to clearly track this dependency.

(In reply to comment #45)
> (In reply to comment #41)
> > 1. make sure you are using network interface in layer2 mode (bug 561017)
> > 2. edit config file, modify for example "LAYER2=1" to "oAYER2=1" - you need to
> > go interactive and restart zSeries initrd (see bug 558881 comment 6 for more
> > details)
> 
> I'm not sure what this step does. If this means edit the CMS conf file (or parm
> file if not using a conf file), wouldn't this step then in fact disable layer2,
> i.e. enable layer3 mode and cause bug 561017?

zSeries initrd will ask only for LAYER parameter, because there is no LAYER= option in config file, and the rest of parameters are taken from the config file - you don't need to enter all the parameters by hand and you are able to restart configuration questions at the end of zSeries initrd. Bug 561017 was tested with the proper config file.

> The content of /proc/cio_ignore after booting the installed system would be
> interesting and I suppose that it'll show the equivalent of
> "cio_ignore=all,!0.0.0900".

Ýroot@rtt6 proc¨# cat cio_ignore
cat cio_ignore  
0.0.0000-0.0.0008  
0.0.000a-0.0.3025  
0.0.3027-0.0.3125  
0.0.3127-0.0.ffff  
0.1.0000-0.1.ffff  
0.2.0000-0.2.ffff  
0.3.0000-0.3.ffff  

I will do a clone of this bug for initscripts, since I didn't find bug for that.

clone of this bug for initscript part: bug 561339

Created attachment 402311 [details]
init.log

While testing network part of this bug, I found that dracut doesn't bring network interface up. Not sure if it is dracut issue or issue with my setup.

steps I did:
1. install rhel without cio_ignore parameter, install on dasd disks
2. install dracut-network
3. run dracut to rebuild initramfs with network support:
dracut --force /boot/initramfs-2.6.32-19.el6.s390x.img `uname -r`
4. after reboot, modify /etc/zipl.conf and add lines:

[nfs2]
        image=/boot/vmlinuz-2.6.32-19.el6.s390x
        ramdisk=/boot/initramfs-2.6.32-19.el6.s390x.img
        parameters="root=10.16.105.196:/nfs/nfs_root cio_ignore=all,!0.0.0009 rd_CCW=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR ip=10.16.105.197:10.16.105.196:10.16.111.254:255.255.248.0:rtt6.s390.bos.redhat.com:eth0:none rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us rdshell rdinitdebug"

5. run zipl
6. reboot and select "nfs2" to boot
7. dracut fails to boot, eth0 is not up:

dracut:/# ip a
1: lo: <LOOPBACK> mtu 16436 qdisc noop state DOWN  
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

cat /proc/cio_ignore
0.0.0000-0.0.0008 
0.0.000a-0.0.3025 
0.0.3027-0.0.3125 
0.0.3127-0.0.3225 
0.0.3227-0.0.3325 
0.0.3327-0.0.3425 
0.0.3427-0.0.3525 
0.0.3527-0.0.3625 
0.0.3627-0.0.3725 
0.0.3727-0.0.ffff 
0.1.0000-0.1.ffff 
0.2.0000-0.2.ffff 
0.3.0000-0.3.ffff 

init.log is in attachment.

The same configuration without cio_ignore=all,!0.0.0009 bring the network interface up.

DASD and zFCP part of this bug is working fine for me.

cio_ignore=all,!0.0.0009

shouldn't this be: cio_ignore=all,!0.0.0009,!0.0.0a00,!0.0.0a01,!0.0.0a02

for rd_CCW=qeth,0.0.0a00,0.0.0a01,0.0.0a02 to work??

(In reply to comment #49)
> cio_ignore=all,!0.0.0009
> 
> shouldn't this be: cio_ignore=all,!0.0.0009,!0.0.0a00,!0.0.0a01,!0.0.0a02

I think it should be only cio_ignore=all,!0.0.0009 and the rest is freed automatically with *_cio_free
 
> for rd_CCW=qeth,0.0.0a00,0.0.0a01,0.0.0a02 to work??    

If I remember the dracut code correctly this string is copied into /etc/ccw.conf and so there is probably a problem with parsing it in the znet_cio_free script.

with "cio_ignore=all,!0.0.0009,!0.0.0a00,!0.0.0a01,!0.0.0a02", the network ifc
eth0 still doesn't bring up, although /proc/cio_ignore looks fine to me:

cat /proc/cio_ignore
0.0.0000-0.0.0008 
0.0.000a-0.0.09ff 
0.0.0a03-0.0.3025 
0.0.3027-0.0.3125 
0.0.3127-0.0.3225 
0.0.3227-0.0.3325 
0.0.3327-0.0.3425 
0.0.3427-0.0.3525 
0.0.3527-0.0.3625 
0.0.3627-0.0.3725 
0.0.3727-0.0.ffff 
0.1.0000-0.1.ffff 
0.2.0000-0.2.ffff 
0.3.0000-0.3.ffff 

dracut:/# ip a
1: lo: <LOOPBACK> mtu 16436 qdisc noop state DOWN  
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 

Then I tried to run "udevadm trigger" and "ip a":

dracut:/# udevadm trigger
dracut:/# qeth: register layer 2 discipline 
qdio: 0.0.0a02 OSA on SC 2 using AI:1 QEBSM:0 PCI:1 TDD:1 SIGA:RW AO  
qeth 0.0.0a00: MAC address 02:00:00:00:00:07 successfully registered on device
e
th0 
qeth 0.0.0a00: Device is a Guest LAN QDIO card (level: V543) 
with link type GuestLAN QDIO (portname: FOOBAR) 

ip a
1: lo: <LOOPBACK> mtu 16436 qdisc noop state DOWN  
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 
2: eth0: <BROADCAST,MULTICAST> mtu 1492 qdisc noop state DOWN qlen 1000 
    link/ether 02:00:00:00:00:07 brd ff:ff:ff:ff:ff:ff 


Anyway, from line:
rd_CCW=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR
there should be all necessary information for dracut to free network device,
doesn't it? For DASD disks it is not needed to add any additional devices to
cio_ignore= list, this boot item works for DASD:

[linux2]
        image=/boot/vmlinuz-2.6.32-19.el6.s390x
        ramdisk=/boot/initramfs-2.6.32-19.el6.s390x.img
        parameters="root=/dev/disk/by-path/ccw-0.0.3126-part1
rd_DASD=0.0.3126,use_diag=0,readonly=0,erplog=0,failfast=0 rd_NO_LUKS rd_NO_LVM
rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us
cio_ignore=all,!0.0.0009"

(In reply to comment #48)

+ à -e /cmdline/30parse-ccw.sh ¨ 
+ . /cmdline/30parse-ccw.sh 
+ getargs rd_CCW= 
+ local o line found 
+ Ã -z root=10.16.105.196:/nfs/nfs_root cio_ignore=all,!0.0.0009 rd_CCW=qeth,0.0
.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR ip=10.16.105.197:10.16.105.196:
10.16.111.254:255.255.248.0:rtt6.s390.bos.redhat.com:eth0:none rd_NO_LUKS rd_NO_
LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us rds
hell rdinitdebug BOOT_IMAGE=4  ¨ 

+ à rd_CCW=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR = rd_CCW= ¨ 
+ à rd_CCW = rd_CCW ¨ 
+ echo -n qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR 
+ found=1 

+ à -n 1 ¨ 
+ return 0 

$(getargs 'rd_CCW=') returns successfully with hopefully at least one ccw_arg.

+ echo qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR 

modules.d/95znet/parse-ccw.sh correctly appends one line to /etc/ccw.conf
and calls znet_cio_free.

+ znet_cio_free 
ALL_DEVICES='' 

PROBLEM NO. 1:
Yet, znet_cio_free does not find anything in its internal ALL_DEVICES variable
which leads to unwanted output (c.f. bug 570763) but that's not the problem here.

Jan, could you please provide output of the following commands at the rdshell?:

ls -lF /etc/ccw.conf
cat /etc/ccw.conf
ls -laF /sys/bus/ccwgroup/devices/
sh -x znet_cio_free

Dan, I suspect this:

# grep 2 or 3 channels from beginning of each line
DEVICES=$(echo $line | grep -E -i -o "^([0-9]\.[0-9]\.[a-f0-9]+,){1,2}([0-9]\.[0-9]\.[a-f0-9]+)")

I think we should not match from the beginning of the line since it begins
with the driver name, "qeth," in the case here.

(In reply to comment #51)
> dracut:/# udevadm trigger
> dracut:/# qeth: register layer 2 discipline 
> qdio: 0.0.0a02 OSA on SC 2 using AI:1 QEBSM:0 PCI:1 TDD:1 SIGA:RW AO  
> qeth 0.0.0a00: MAC address 02:00:00:00:00:07 successfully registered on device
> e
> th0 
> qeth 0.0.0a00: Device is a Guest LAN QDIO card (level: V543) 
> with link type GuestLAN QDIO (portname: FOOBAR) 

PROBLEM NO. 2:
Even when working around problem 1, udev does not bring up the ccwgroup
correctly and it almost seems as if it needs to be triggered twice.
Wasn't there a similar issues requiring multiple udevadm trigger for network
in anaconda?

> Anyway, from line:
> rd_CCW=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR
> there should be all necessary information for dracut to free network device,
> doesn't it?

Yes, rd_DASD, rd_DASD_MOD, rd_ZFCP, and rd_CCW (which should btw have been named rd_ZNET along with /etc/znet.conf in dracut) contain everything necessary for *_cio_fee, namely the device bus IDs.

(In reply to comment #52)
> Jan, could you please provide output of the following commands at the rdshell?:
> 
> ls -lF /etc/ccw.conf
> cat /etc/ccw.conf
> ls -laF /sys/bus/ccwgroup/devices/
> sh -x znet_cio_free

dracut:/# ls -lF /etc/ccw.conf
-rw-r--r-- 1 root root 57 Mar 24 21:32 /etc/ccw.conf 

dracut:/# cat /etc/ccw.conf
qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR 

dracut:/# ls -laF /sys/bus/ccwgroup/devices/
ls: cannot access /sys/bus/ccwgroup/devices/: No such file or directory

ls -laF /sys/bus/
total 0 
drwxr-xr-x  7 root root 0 Mar 24 21:32 ./ 
drwxr-xr-x 13 root root 0 Mar 24 21:32 ../ 
drwxr-xr-x  4 root root 0 Mar 24 21:32 ccw/ 
drwxr-xr-x  4 root root 0 Mar 24 21:32 css/ 
drwxr-xr-x  4 root root 0 Mar 24 21:32 iucv/ 
drwxr-xr-x  4 root root 0 Mar 24 21:32 platform/
drwxr-xr-x  4 root root 0 Mar 24 21:32 virtio/ 

dracut:/# sh -x znet_cio_free
sh: Can't open znet_cio_free 

dracut:/sbin# sh -x /sbin/znet_cio_free
+ DASDCONFIG=/etc/dasd.conf 
+ ZFCPCONFIG=/etc/zfcp.conf 
+ ZNETCONFIG=/etc/ccw.conf 
+ BLACKLIST=/proc/cio_ignore 
+ VERBOSE= 
+ PATH=/bin:/usr/bin:/sbin:/usr/sbin 
+ ALL_DEVICES= 
+ WAITING_TIMEOUT=60 
+ WAITING_TOTAL=0 
+ basename /sbin/znet_cio_free 
+ CMD=znet_cio_free 
+ CONFIG=/etc/ccw.conf 
+ MODE=znet 
+ Ý 0 -gt 0 ¨ 
+ Ý ! -f /proc/cio_ignore ¨ 
+ Ý znet = dasd -o znet = zfcp ¨ 
+ Ý znet = dasd ¨ 
+ Ý znet = znet ¨ 
+ Ý -f /etc/ccw.conf ¨ 
+ read line 
+ Ý -z qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR ¨ 
+ echo qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR 
+ grep -E -i -o ^(Ý0-9¨\.Ý0-9¨\.Ýa-f0-9¨+,){1,2}(Ý0-9¨\.Ý0-9¨\.Ýa-f0-9¨+) 
+ DEVICES= 
+ free_device 
+ local DEV 
+ Ý -z  ¨ 
+ return 
+ read line 
+ grep -E -i -h ^ÝÝ:space:¨¨*SUBCHANNELS=Ý'"¨?(Ý0-9¨\.Ý0-9¨\.Ýa-f0-9¨+,){1,2}(Ý0
-9¨\.Ý0-9¨\.Ýa-f0-9¨+)Ý'"¨?(ÝÝ:space:¨¨+#|ÝÝ:space:¨¨*$) /etc/sysconfig/network-
scripts/ifcfg-* 
+ OLD_IFS=   
 
+ IFS=, 
+ set 
ALL_DEVICES='' 
BLACKLIST='/proc/cio_ignore' 
BOOT_IMAGE='4' 
CMD='znet_cio_free' 
CONFIG='/etc/ccw.conf' 
DASDCONFIG='/etc/dasd.conf' 
DEVICES='' 
HOME='/' 
IFS=',' 
KEYTABLE='us' 
LANG='en_US.UTF-8' 
MODE='znet' 
OLDPWD='/' 
OLD_IFS='   
' 
OPTIND='1' 
PATH='/bin:/usr/bin:/sbin:/usr/sbin' 
PPID='663' 
PS1='dracut:${PWD}# ' 
PS2='> ' 
PS4='+ ' 
PWD='/sbin' 
SYSFONT='latarcyrheb-sun16' 
TERM='linux' 
VERBOSE='' 
WAITING_TIMEOUT='60' 
WAITING_TOTAL='0' 
ZFCPCONFIG='/etc/zfcp.conf' 
ZNETCONFIG='/etc/ccw.conf' 
line='' 
rd_CCW='qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR' 

Module qeth is not loaded at this moment, is it expected behavior?

cat /proc/modules
dasd_eckd_mod 87759 24 - Live 0x000003e000c04000 
dasd_mod 101128 9 dasd_eckd_mod, Live 0x000003e000b8f000 
dm_mod 104990 0 - Live 0x000003e000b3c000 
nfs 468197 0 - Live 0x000003e000a78000 
lockd 103088 1 nfs, Live 0x000003e000999000 
fscache 65381 1 nfs, Live 0x000003e00094f000 
nfs_acl 3845 1 nfs, Live 0x000003e000926000 
auth_rpcgss 57161 1 nfs, Live 0x000003e000908000 
sunrpc 309132 5 nfs,lockd,nfs_acl,auth_rpcgss, Live 0x000003e000888000

(In reply to comment #53)
> dracut:/# ls -laF /sys/bus/ccwgroup/devices/
> ls: cannot access /sys/bus/ccwgroup/devices/: No such file or directory

OK, this directory does not exist, because no network device driver using ccwgroup has been loaded which is in turn due to problem no. 2 of comment 52.

> dracut:/sbin# sh -x /sbin/znet_cio_free

> + Ý -f /etc/ccw.conf ¨ 
> + read line 
> + Ý -z qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR ¨ 
> + echo qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR 
> + grep -E -i -o ^(Ý0-9¨\.Ý0-9¨\.Ýa-f0-9¨+,){1,2}(Ý0-9¨\.Ý0-9¨\.Ýa-f0-9¨+) 
> + DEVICES= 

Dan, I think this confirms my suspicion that we should not match for beginning of line in above grep expression.

> ALL_DEVICES='' 

> Module qeth is not loaded at this moment, is it expected behavior?

Assuming the VM guest has qeth devices configured and they are no longer ignored by cio_ignore, then udev should have loaded qeth automatically by means of modalias. Additionally, dracut's modules.d/95znet/55-ccw.rules should have triggered modules.d/95znet/ccw_init to establish a ccwgroup and set it online, in order to provide eth0.

My assumption can be confirmed by looking if the following sysfs symlinks exist and point to ../../../../bus/ccw/drivers/qeth:
/sys/bus/ccw/devices/0.0.0a00/driver
/sys/bus/ccw/devices/0.0.0a01/driver
/sys/bus/ccw/devices/0.0.0a02/driver
And to confirm that modalias is there:
cat /sys/bus/ccw/devices/0.0.0a00/modalias

Jan, maybe Harald can figure out what's wrong with regard to problem no. 2 by looking at a udev debug trace (full z/VM console log with rdudevdebug and maybe rdudevinfo boot option).
The anaconda udevadm trigger issue with network I mentioned above was here (maybe this also helps Harald; I haven't understood the cause nor the conclusion):
https://www.redhat.com/archives/anaconda-devel-list/2009-July/msg00387.html
referring to bug 514501 comment 4.

(In reply to comment #54)
> > + Ý -f /etc/ccw.conf ¨ 
> > + read line 
> > + Ý -z qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR ¨ 
> > + echo qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR 
> > + grep -E -i -o ^(Ý0-9¨\.Ý0-9¨\.Ýa-f0-9¨+,){1,2}(Ý0-9¨\.Ý0-9¨\.Ýa-f0-9¨+) 
> > + DEVICES= 
> 
> Dan, I think this confirms my suspicion that we should not match for beginning
> of line in above grep expression.

Right and it's also documented in dracut's man page. It's already fixed in my development version.

> > Module qeth is not loaded at this moment, is it expected behavior?
> 
> Assuming the VM guest has qeth devices configured and they are no longer
> ignored by cio_ignore, then udev should have loaded qeth automatically by means
> of modalias. Additionally, dracut's modules.d/95znet/55-ccw.rules should have
> triggered modules.d/95znet/ccw_init to establish a ccwgroup and set it online,
> in order to provide eth0.
> 
> My assumption can be confirmed by looking if the following sysfs symlinks exist
> and point to ../../../../bus/ccw/drivers/qeth:
> /sys/bus/ccw/devices/0.0.0a00/driver
> /sys/bus/ccw/devices/0.0.0a01/driver
> /sys/bus/ccw/devices/0.0.0a02/driver
> And to confirm that modalias is there:
> cat /sys/bus/ccw/devices/0.0.0a00/modalias

These directories do not exist:
dracut:/# ls /sys/bus/ccw/devices
0.0.0009  0.0.3126  0.0.3326  0.0.3526 0.0.3726
0.0.3026  0.0.3226  0.0.3426  0.0.3626

(In reply to comment #56)
> These directories do not exist:
> dracut:/# ls /sys/bus/ccw/devices
> 0.0.0009  0.0.3126  0.0.3326  0.0.3526 0.0.3726
> 0.0.3026  0.0.3226  0.0.3426  0.0.3626    

Even if you circumvent problem no. 1 by using cio_ignore=all,!0.0.0009,!0.0.0a00-0.0.0a02 at IPL time as you did in comment 51?

with cio_ignore=all,!0.0.0009,!0.0.0a00,!0.0.0a01,!0.0.0a02

dracut:/# ls -l /sys/bus/ccw/devices/0.0.0a00/driver
lrwxrwxrwx 1 root root 0 Mar 25 15:53 /sys/bus/ccw/devices/0.0.0a00/driver -> ..
/../../../bus/ccw/drivers/qeth 
dracut:/# ls -l /sys/bus/ccw/devices/0.0.0a01/driver
lrwxrwxrwx 1 root root 0 Mar 25 15:54 /sys/bus/ccw/devices/0.0.0a01/driver -> ..
/../../../bus/ccw/drivers/qeth 
dracut:/# ls -l /sys/bus/ccw/devices/0.0.0a02/driver
lrwxrwxrwx 1 root root 0 Mar 25 15:54 /sys/bus/ccw/devices/0.0.0a02/driver -> ..
/../../../bus/ccw/drivers/qeth 
dracut:/# cat /sys/bus/ccw/devices/0.0.0a00/modalias
ccw:t1731m01dt1732dm01

Phew, I was already worried something really weird was going on.
rdudevdebug and rdudevinfo should help us shed light onto problem no. 2.

Created attachment 402692 [details]
rdudevinfo

Messages from boot with rdudevinfo parameter.

Could you please replace attachment 402692 [details] with a console log having cio_ignore=all,!0.0.0009,!0.0.0a00,!0.0.0a01,!0.0.0a02
AND rdudevdebug plus rdinitdebug at the beginning of the kernel command line
(and possibly compressed with gzip since it might get large)?

We have to circumvent problem no. 1, otherwise dracut will never see any qeth devices.
rdudevinfo does not give enough info to track all the output.



Independent of that, I see some unrelated oddities in the console log
(which might be worth new bug(s) against dracut):

Kernel command line: root=10.16.105.196:/nfs/nfs_root cio_ignore=all,!0.0.0009 r
d_CCW=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portname=FOOBAR ip=10.16.105.197:
10.16.105.196:10.16.111.254:255.255.248.0:rtt6.s390.bos.redhat.com:eth0:none rd_
NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 K
EYTABLE=us rdshell rdinitdebug rdudevinfo BOOT_IMAGE=4 

Even though the kernel command line does not specify any rd_DASD*, six DASDs seem to be freed automatically from cio_ignore, the dasd device driver gets loaded automatically, and the DASDs set online successfully. This is definitely not intended. The only thing I can imagine is that /etc/dasd.conf (or even /etc/modprobe.conf or /etc/modprobe.d/** with some "options dasd_mod dasd=...") was copied from your installed system into the initramfs even though you correctly did not specify hostonly mode on generating initramfs and modules.d/95dasd/install does only copy the file in hostonly mode.

Also, I'm missing lots of dracut shell trace e.g. "getarg rd_CCW" and all the other dracut args of the above cited kernel command line. The only getarg visible in rdinitdebug output is "getarg rdshell" at the end. But then again, dracut seems to do something with rd_NO_LUKS, rd_NO_LVM, and rd_NO_MD.
Ahhh, maybe rdinitdebug is position dependent and should be moved to the beginning of the kernel command line together with rdudevdebug???

Created attachment 402768 [details]
rdudevdebug rdinitdebug

console log with kernel parameters rdudevdebug rdinitdebug cio_ignore=all,!0.0.0009,!0.0.0a00,!0.0.0a01,!0.0.0a02

The issue reported by Jan in comment 48 actually consists of two sub-problems:

Problem 1 is a bug in s390utils' dasd_cio_free and therefore not related to dracut. However, we have been discussing *_cio_free in this bug here, so problem 1 actually belongs here.

Problem 2 is the same as in bug 561926 comment 26, i.e. related to s390utils' ccw_init (assuming bug 539491 has been fixed) and therefore not related to dracut. Since I don't have a recent compose, I cannot look into it to see which of those dependent bugs have already been fixed and incorporated. I hesitate to make this bug here dependent on 561926 and 539491 since that would weave unrelated stuff together.

### analysis of debug log:

udevd^?185^?: 'ACTION=add' added
udevd^?185^?: 'DEVPATH=/devices/css0/0.0.0000/0.0.0a00' added
udevd^?185^?: 'MODALIAS=ccw:t1731m01dt1732dm01' added
...
udevd^?185^?: 'ACTION=add' added
udevd^?185^?: 'DEVPATH=/devices/css0/0.0.0001/0.0.0a01' added
udevd^?185^?: 'MODALIAS=ccw:t1731m01dt1732dm01' added
...
udevd^?185^?: 'ACTION=add' added
udevd^?185^?: 'DEVPATH=/devices/css0/0.0.0002/0.0.0a02' added
udevd^?185^?: 'MODALIAS=ccw:t1731m01dt1732dm01' added

### Udevadm trigger correctly generates add events for the qeth subchannels.

udevd-work^?223^?: '/lib/udev/ccw_init' (stderr) '/lib/udev/ccw_init: 78: cannot create /sys/bus/ccwgroup/drivers/qeth/group: Directory nonexistent'

udevd-work^?224^?: '/lib/udev/ccw_init' (stderr) '/lib/udev/ccw_init: 78: cannot create /sys/bus/ccwgroup/drivers/qeth/group: Directory nonexistent'

udevd-work^?226^?: '/lib/udev/ccw_init' (stderr) '/lib/udev/ccw_init: 78: cannot ceate /sys/bus/ccwgroup/drivers/qeth/group: Directory nonexistent'

udevd-work^?223^?: '/lib/udev/ccw_init' (stderr) '/lib/udev/ccw_init: 87: cannot create /sys/bus/ccwgroup/drivers/qeth/0.0.0a00/layer2: Directory nonexistent' 
udevd-work^?223^?: '/lib/udev/ccw_init' (stderr) '/lib/udev/ccw_init: 93: cannot create /sys/bus/ccwgroup/drivers/qeth/0.0.0a00/portname: Directory nonexistent'
udevd-work^?223^?: 'ccw_init' returned with exitcode 1
...
udevd-work^?223^?: '/sbin/modprobe -b ccw:t1731m01dt1732dm01' started
...
udevd-work^?223^?: '/sbin/modprobe -b ccw:t1731m01dt1732dm01' returned with exitcode 0

qeth: loading core functions

udevd-work^?224^?: '/lib/udev/ccw_init' (stderr) '/lib/udev/ccw_init: 87: cannot create /sys/bus/ccwgroup/drivers/qeth/0.0.0a00/layer2: Directory nonexistent' 
udevd-work^?224^?: '/lib/udev/ccw_init' (stderr) '/lib/udev/ccw_init: 93: cannot create /sys/bus/ccwgroup/drivers/qeth/0.0.0a00/portname: Directory nonexistent'
udevd-work^?224^?: 'ccw_init' returned with exitcode 1
...
udevd-work^?224^?: '/sbin/modprobe -b ccw:t1731m01dt1732dm01' started
...
udevd-work^?224^?: '/sbin/modprobe -b ccw:t1731m01dt1732dm01' returned with exitcode 0

udevd-work^?226^?: '/lib/udev/ccw_init' (stderr) '/lib/udev/ccw_init: 87: cannot create /sys/bus/ccwgroup/drivers/qeth/0.0.0a00/layer2: Directory nonexistent'
udevd-work^?226^?: '/lib/udev/ccw_init' (stderr) '/lib/udev/ccw_init: 93: cannot create /sys/bus/ccwgroup/drivers/qeth/0.0.0a00/portname: Directory nonexistent'
udevd-work^?226^?: 'ccw_init' returned with exitcode 1
...
udevd-work^?226^?: '/sbin/modprobe -b ccw:t1731m01dt1732dm01' started
...
udevd-work^?226^?: '/sbin/modprobe -b ccw:t1731m01dt1732dm01' returned with exitcode 0

### For all three subchannels 0a00, 0a01, and 0a02 separate udevd workers run in parallel. Unfortunately ccw_init gets triggered before the qeth driver has been loaded successfully. That's simply the wrong order and therefore it doesn't work.

The /etc/ccw.conf parsing is fixed s390utils-1.8.2-15.el6

Hi Dan, the following code fragment of device_cio_free seems to exit early if one of the free for one device type failed, e.g. due to some non-existent device. Suppose, one DASD does not exist or took too long to free and we ran into the timeout, then no ZFCP or network devices would be freed any more.

I think it should still call free for all different device types. It could maybe remember if an error level unequal to zero was returned by one of the three calls and then finally return with an appropriate error level unequal to zero.

if [ $MODE = "all" ]; then
    # shortcut for calling all 3 scripts
    $DIR/dasd_cio_free $ARGS || exit $?
    $DIR/zfcp_cio_free $ARGS || exit $?
    $DIR/znet_cio_free $ARGS || exit $?
    exit 0
fi

(In reply to comment #65)
> Hi Dan, the following code fragment of device_cio_free seems to exit early if
> one of the free for one device type failed, e.g. due to some non-existent
> device. Suppose, one DASD does not exist or took too long to free and we ran
> into the timeout, then no ZFCP or network devices would be freed any more.

Hi Steffen, device_cio_free never exits with a non-zero when there is a problem with a device, it can reach a timeout for some devices, but still returns a zero. It must check also the devices (with a success) that are listed after the timed out ones.

Did you get the new version from the public repo for the s390(x) Fedora utils (http://fedorapeople.org/gitweb?p=sharkcz/public_git/utils.git;a=summary)?

Dan, should this bz be assigned to s390-utils, or stay with dracut?

(In reply to comment #67)
> Dan, should this bz be assigned to s390-utils, or stay with dracut?    

IMHO, this bug here treats both dracut (originally) and s390utils (started later on). Similar to bug 561339, the dracut part should strictly speaking be called "[LTC 6.0 FEAT] 201085: unmask and wait for devices, dracut part" and we still need it to track the dependency on that package. For the s390utils part this bug here already has the corresponding description.

Maybe we could clone this bug for s390utils and then rename the title in this original dracut bug?

(In reply to comment #66)
> Hi Steffen, device_cio_free never exits with a non-zero when there is a problem
> with a device, it can reach a timeout for some devices, but still returns a
> zero. It must check also the devices (with a success) that are listed after the
> timed out ones.

Ah, OK, I missed that. Thanks for pointing out. Everything's fine then.

> Did you get the new version from the public repo for the s390(x) Fedora utils
> (http://fedorapeople.org/gitweb?p=sharkcz/public_git/utils.git;a=summary)?    

Yes, already cloned. Thanks a lot for having shared this on fedora-s390x!

device_cio_free has implicit dependencies on various external tools such as:
echo, sleep, modprobe, grep, printf, seq.

To make things worse, device_cio_free is used in 3 different environments:
1) in anaconda [bug 558881, bug 533492, bug 576015]: dependencies are in anaconda's rpm, upd-instroot, and mk-images (for linuxrc.s390).
2) in dracut [bug 533494]: dependencies are coded in dracut module's install script and probably dracut's rpm in the first place as well.
3) in initscripts/upstart [bug 561339]: dependencies would need to be fetched on install of s390utils-base by rpm requires.

Since dependencies seem only partly explicitly specified, there are cases where some of the external tools are not available and device_cio_free therefore fails,
e.g. bug 558881 comment 30.

Can we make all those dependencies explicit for each environment and/or reduce the number of external tools by using dash builtins where possible?

Wouldn't the least error prone way be to simply have s390utils-base require all the necessary components? As each of the above mentioned components will need to require s390utils-base for s390x anyway that would ensure that existence of the necessary packages, right?

The problem with image generation is distinct from those requirements though. Those are and will always be script related issues, be it anaconda image generation or dracut. So basically only 3) can be really fixed via dependencies, 1) and 2) will always need sync ups with possible changes in device_cio_free requirements i guess.

Or am i missing something here?

Just my $0.02

Thanks & regards, Phil

(In reply to comment #71)
> The problem with image generation is distinct from those requirements though.
> Those are and will always be script related issues, be it anaconda image
> generation or dracut. So basically only 3) can be really fixed via
> dependencies, 1) and 2) will always need sync ups with possible changes in
> device_cio_free requirements i guess.

Exactly, 3) is necessary but not sufficient for 1) and 2).
I just did the overview in comment 70 so we won't forget any place where dependencies occur.

(In reply to comment #70)
> Since dependencies seem only partly explicitly specified, there are cases where
> some of the external tools are not available and device_cio_free therefore
> fails,
> e.g. bug 558881 comment 30.

seq will be installed in initramfs since dracut-004-12.el6

Created attachment 409528 [details]
rdudevdebug rdinitdebug

I'm still unable to boot with rootfs on network device (iSCSI disk) when booting with cio_ignore=all,!0.0.0009 parameter. (It works without the cio_ignore parameter).

Attaching boot log with "cio_ignore=all,!0.0009,!0.0.0a00,!0.0.0a01,!0.0.0a02 rdshell rdudevdebug rdinitdebug" parameter.

Tested on build RHEL6.0-20100422.12 with dracut-004-17.el6, s390utils-1.8.2-18.el6

Moving back to ASSIGNED.

Did anybody address problem 2 of comment 63?
It doesn't look like because I still see those lines:

udevd-work^?176^?: '/lib/udev/ccw_init' (stderr) '/lib/udev/ccw_init: 73: cannot create /sys/bus/ccwgroup/drivers/qeth/group: Directory nonexistent' 
...
udevd-work^?176^?: 'ccw_init' returned with exitcode 1 

Before that is not fixed, it does not make sense to retest.
Jan, you could work around the too late module loading by booting with rdbreak=cmdline or rdbreak=pre-udev and issue "modprobe qeth" in the debug shell and then exit the shell to continue dracut initramfs execution.

===

Is Dan's fix of comment 55 in the compose?
If so, then you don't need the workaround with manually adapting the default cio_ignore anymore, Jan. cio_ignore=all,!0.0009 should just work.

===

FYI: I doubt "ifname=eth0:02:00:00:00:00:05" will work, since OSA layer2 will come up with a random MAC address each time so you cannot match for MAC address. That's why linuxrc pins the first random MAC address in the ifcfg file as MACADDR so it'll continue using the same MAC on each reboot. However, that doesn't help with dracut here.

Created attachment 409723 [details]
use ccw-init and ccw rules from s390utils in dracut

I've committed and built s390utils-1.8.2-19.el6 with the merged ccw_init script that should work in both initramfs and normal startup.

re comment 63:
there is a delay loop before trying to group the channel in the ccw_init script, it should help if it's only a timing issue

re comment 55:
the fix is present in s390utils for some time

Tested many times during RHEL6 Snapshot testing and no issue found with recent trees:

 * root fs on zFCP LUN tested on build RHEL6.0-20100531.1 (dracut-004-20.el6, s390utils-1.8.2-24.el6)
 * root fs on DASD tested on build RHEL6.0-20100615.0 (dracut-004-20.1.el6, s390utils-1.8.2-24.el6)
 * root fs on iSCSI (=network part of this bug) tested on build RHEL6.0-20100615.0 (dracut-004-20.1.el6, s390utils-1.8.2-24.el6)

Moving bug to VERIFIED.

Dan, while looking at bug 607481, I figured that /etc/sysconfig/network-scripts/network-functions ignores ifcfg files with certain name pattersn defined in /etc/init.d/functions with $__sed_discard_ignored_files.
If we wanted to copy the semantics of SUBCHANNELS handling, /sbin/device_cio_free could use the same file ignore pattern, in order to not free devnos from cio_ignore which /lib/udev/ccw_init would never activate.

(In reply to comment #83)
> Dan, while looking at bug 607481, I figured that
> /etc/sysconfig/network-scripts/network-functions ignores ifcfg files with
> certain name pattersn defined in /etc/init.d/functions with
> $__sed_discard_ignored_files.
> If we wanted to copy the semantics of SUBCHANNELS handling,
> /sbin/device_cio_free could use the same file ignore pattern, in order to not
> free devnos from cio_ignore which /lib/udev/ccw_init would never activate.    

good point, Steffen

fixed in commit 5dec421fb and the updated device_cio_free will be included in s390utils-1.8.2-26.el6

Red Hat Enterprise Linux Beta 2 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.