@Jonathan @Prashanth could one of you investigate this scenario?

I'm not a multipath SME so I'm not sure as to the specifics here, but this is unlikely to be an RHCOS-specific bug. We ship the same multipath used in vanilla RHEL.

Is it perhaps missing something in /etc/multipath.conf to have it automatically re-activate previously failed paths?

@Alexander Klein - how did you simulate the failover on the zVM ? are there any commands that you can share so i can try the same?

But i agree that this looks more like a general multipath question/issue rather than RHCOS.

on the zvm side you can just detach the one of the fcp devices with 

#cp det <4 digit fcp device>  

from linux: 

sudo vmcp det 1234


once you attach it again (by 

#cp att 1234 to *   

[* if you are issuing the command from that guest or the guestname to do it remotely]) Linux will automatically detect the fcp device being available again.
Just multipath will not bring it up again.

another way would be to just deconfigure single paths in linux with 

sudo chzdev -d <fcpdevice>:0x<wwpn>:0x<lun> 

and afterward reconfigureing them with 

sudo chzdev -e <fcpdevice>:0x<wwpn>:0x<lun> 

they will come up for the linux but multipath wills till not recognize them.

i just created a valid /etc/multipath.conf (s390x fcp recommendation)

    defaults {
        default_features "1 queue_if_no_path"
        user_friendly_names yes
        path_grouping_policy multibus
        dev_loss_tmo 2147483647
        fast_io_fail_tmo 5
    }

    blacklist {
        devnode '*'
    }

    blacklist_exceptions {
        devnode "^sd[a-z]+"

    }


and this will make the failed paths recover correctly, also the scsi ids will be displayed correctly, so this should be fine as long as the documentation mentions that you have to do this step.

on rhel you would get a default multipath.conf by running  
sudo /sbin/mpathconf --enable
this does work on rhcos too but is not done automatically when enabling multipath so it might be a good idea to do this automatically for a default config and still have the option to enable a custom config via mco.

You can create a multipath.conf with MCO before you enable MP via MCO:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: "multipath"
  name: worker-enablempath-config
spec:
  config:
     ignition:
      version: 3.2.0      
     storage:
       files:
       - contents:
           source: data:text/plain;charset=utf-8;base64,IyBkZXZpY2UtbWFwcGVyLW11bHRpcGF0aCBjb25maWd1cmF0aW9uIGZpbGUKCiMgRm9yIGEgY29tcGxldGUgbGlzdCBvZiB0aGUgZGVmYXVsdCBjb25maWd1cmF0aW9uIHZhbHVlcywgcnVuIGVpdGhlcjoKIyAjIG11bHRpcGF0aCAtdAojIG9yCiMgIyBtdWx0aXBhdGhkIHNob3cgY29uZmlnCgojIEZvciBhIGxpc3Qgb2YgY29uZmlndXJhdGlvbiBvcHRpb25zIHdpdGggZGVzY3JpcHRpb25zLCBzZWUgdGhlCiMgbXVsdGlwYXRoLmNvbmYgbWFuIHBhZ2UuCgpkZWZhdWx0cyB7Cgl1c2VyX2ZyaWVuZGx5X25hbWVzIHllcwoJZmluZF9tdWx0aXBhdGhzIHllcwoJZW5hYmxlX2ZvcmVpZ24gIl4kIgp9CgpibGFja2xpc3RfZXhjZXB0aW9ucyB7CiAgICAgICAgcHJvcGVydHkgIihTQ1NJX0lERU5UX3xJRF9XV04pIgp9CgpibGFja2xpc3Qgewp9Cg==
         filesystem: root
         mode: 0644
         path: /etc/multipath.conf

I used a default configuration, but it is also possible to create another (recommended) config. I labeled my node as "multipath" to have it on a specific node only.

Ok, yes i see this issue too and adding the `/etc/multipath.conf` does work.

@Jonathan Lebon - Could the /etc/multipath.conf be persisted to the real root in the case that the rd.multipath=default kargs is present in dracut? is that a change we could potentially make? and does it make sense?

(In reply to Prashanth Sundararaman from comment #8)
> @Jonathan Lebon - Could the /etc/multipath.conf be persisted to the real
> root in the case that the rd.multipath=default kargs is present in dracut?
> is that a change we could potentially make? and does it make sense?

This is mostly how it's set up: https://github.com/coreos/fedora-coreos-config/blob/6fe1cff5ade8bc7489b6ab542d6f3cb5c92d9082/overlay.d/05core/usr/lib/dracut/modules.d/35coreos-ignition/coreos-teardown-initramfs.sh#L170-L184

But the thing is that this code only kicks in on first boot, while `rd.multipath=default` is now a second boot thing. So the config doesn't get propagated.

But yeah I think it's reasonable to fix this so as to not make the UX too painful. I'll look at hoisting that code out so it runs every boot instead.

@psundara If you have the cycles, would appreciate a sanity-check of that patch on RHCOS with the same s390x scenario. I did confirm it works fine in `cosa run --qemu-multipath` on x86.

(In reply to Jonathan Lebon from comment #10)
> @psundara If you have the cycles, would appreciate a sanity-check
> of that patch on RHCOS with the same s390x scenario. I did confirm it works
> fine in `cosa run --qemu-multipath` on x86.

will test this out today. thank you!!

The patch posted above works. Tested disk failover and when i enabled the device, the path was recognized as active. thanks Jonathan!

(In reply to Prashanth Sundararaman from comment #12)
> The patch posted above works. Tested disk failover and when i enabled the
> device, the path was recognized as active. thanks Jonathan!

Good news!  I'm going to target this BZ for 4.7 with hopes of getting it in the next boot image bump.

(In reply to Micah Abbott from comment #13)
> (In reply to Prashanth Sundararaman from comment #12)
> > The patch posted above works. Tested disk failover and when i enabled the
> > device, the path was recognized as active. thanks Jonathan!
> 
> Good news!  I'm going to target this BZ for 4.7 with hopes of getting it in
> the next boot image bump.

that would be great. Thanks Micah!

@Alexander Klein - Can you please retest with the latest rhcos image: https://releases-rhcos-art.cloud.privileged.psi.redhat.com/?stream=releases/rhcos-4.7-s390x&release=47.83.202102031512-0#47.83.202102031512-0. I tested it and it works fine.

Prashanth verified this in comment #21 using a recent RHCOS 4.7 build

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633