Bug 1904243

+++ This bug was initially created as a clone of Bug #1901021 +++

Description of problem:

According OpenShift 4.6 Official document, the RHCOS worker node's default ISCSI initiator name is in /etc/iscsi/initiatorname.iscsi

https://docs.openshift.com/container-platform/4.6/storage/persistent_storage/persistent-storage-iscsi.html

But we cannot find this file in RHCOS 4.6.1 !!!


[core@worker-1 ~]$ ls -a /etc/iscsi/
.  ..  iscsid.conf

[core@worker-1 ~]$ hostnamectl
   Static hostname: n/a
Transient hostname: worker-1.ocp4-1.example.internal
         Icon name: computer-vm
           Chassis: vm
        Machine ID: ada997b5da89423e8255633f232d8db4
           Boot ID: f4d48fddb7f0499c97dede5fd1ec47c1
    Virtualization: vmware
  Operating System: Red Hat Enterprise Linux CoreOS 46.82.202010091720-0 (Ootpa)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:8::coreos
            Kernel: Linux 4.18.0-193.24.1.el8_2.dt1.x86_64
      Architecture: x86-64

--- Additional comment from kevin on 2020-11-24 10:15:31 UTC ---

In RHCOS 4.5, the initiatorname.iscsi default existed. But in RHCOS 4.6.1, this file is missing!

--- Additional comment from Luca BRUNO on 2020-11-24 11:04:29 UTC ---

Misc notes:
 * The file under /etc should is not supposed be part of the base OS in both RHCOS-4.5 and RHCOS-4.6. The InitatorName is machine-specific, thus it shouldn't be backed into the OS image
 * The RHEL package has some buggy %post logic in the RPM that we have to bypass on CoreOS side, see https://bugzilla.redhat.com/show_bug.cgi?id=1734144 
 * We carry some service units specific to CoreOS in order to generate the initiatorname on first-boot, see https://github.com/openshift/os/blob/a06951cb0b9ec63d79087ca257a2135601e027dd/overlay.d/05rhcos/usr/lib/systemd/system/coreos-generate-iscsi-initiatorname.service
 * There have been some work around this area in the 4.5->4.6 timeframe which may have resulted in a regression, see https://gitlab.cee.redhat.com/coreos/redhat-coreos/-/merge_requests/1098

I would be good to attach here the full journal from this 4.6 node to pinpoint what's going on.

--- Additional comment from Luca BRUNO on 2020-11-24 11:14:53 UTC ---

Ah, I think MR#1098 above renamed the service unit (s/regenerate/generate/), but the `43-manifest-rhcos.preset` content still tries to reference/enable the unit using the old name.

Jonathan, can you maybe have a look here?

Micah, did you verify https://bugzilla.redhat.com/show_bug.cgi?id=1868174 using a patched bootimage?

--- Additional comment from Scott Dodson on 2020-11-24 13:55:32 UTC ---



--- Additional comment from Jonathan Lebon on 2020-11-24 14:25:01 UTC ---

(In reply to Luca BRUNO from comment #3)
> Ah, I think MR#1098 above renamed the service unit (s/regenerate/generate/),
> but the `43-manifest-rhcos.preset` content still tries to reference/enable
> the unit using the old name.

Hmm indeed, you're right. I'm confused now how this worked during testing and verification.
Anyway, will fix this and add a test for it as well.

--- Additional comment from Jonathan Lebon on 2020-11-24 15:08:49 UTC ---

https://github.com/openshift/os/pull/453
https://gitlab.cee.redhat.com/coreos/redhat-coreos/-/merge_requests/1180

--- Additional comment from Benjamin Gilbert on 2020-11-24 16:06:32 UTC ---

Adding to bootimage bump per Jonathan.

--- Additional comment from kevin on 2020-11-24 16:30:39 UTC ---

I have aggregate some log from journal related to iscsi

[root@worker-1 ~]# journalctl  |grep "can't open InitiatorAlias configuration file /etc/iscsi/initiatorname.iscsi" -B 5 -A 5
Nov 24 08:58:12 localhost multipathd[688]: /etc/multipath.conf. See man mpathconf(8) for more details
Nov 24 08:58:12 localhost iscsid[690]: iscsid: can't open InitiatorName configuration file /etc/iscsi/initiatorname.iscsi
Nov 24 08:58:12 localhost iscsid[690]: iscsid: Warning: InitiatorName file /etc/iscsi/initiatorname.iscsi does not exist or does not contain a properly formatted InitiatorName. If using software iscsi (iscsi_tcp or ib_iser) or partial offload (bnx2i or cxgbi iscsi), you may not be able to log into or discover targets. Please create a file /etc/iscsi/initiatorname.iscsi that contains a sting with the format: InitiatorName=iqn.yyyy-mm.<reversed domain name>[:identifier].
Nov 24 08:58:12 localhost iscsid[690]: Example: InitiatorName=iqn.2001-04.com.redhat:fc6.
Nov 24 08:58:12 localhost iscsid[690]: If using hardware iscsi like qla4xxx this message can be ignored.
Nov 24 08:58:12 localhost iscsid[690]: iscsid: can't open InitiatorAlias configuration file /etc/iscsi/initiatorname.iscsi
Nov 24 08:58:12 localhost systemd[1]: Started Open-iSCSI.
Nov 24 08:58:12 localhost systemd[1]: Started Create Volatile Files and Directories.
Nov 24 08:58:12 localhost systemd[1]: Reached target System Initialization.
Nov 24 08:58:12 localhost systemd[1]: Reached target Basic System.
Nov 24 08:58:12 localhost systemd[1]: Starting Ignition (fetch-offline)...
--
Nov 24 09:01:25 localhost multipathd[638]: /etc/multipath.conf. See man mpathconf(8) for more details
Nov 24 09:01:25 localhost iscsid[639]: iscsid: can't open InitiatorName configuration file /etc/iscsi/initiatorname.iscsi
Nov 24 09:01:25 localhost iscsid[639]: iscsid: Warning: InitiatorName file /etc/iscsi/initiatorname.iscsi does not exist or does not contain a properly formatted InitiatorName. If using software iscsi (iscsi_tcp or ib_iser) or partial offload (bnx2i or cxgbi iscsi), you may not be able to log into or discover targets. Please create a file /etc/iscsi/initiatorname.iscsi that contains a sting with the format: InitiatorName=iqn.yyyy-mm.<reversed domain name>[:identifier].
Nov 24 09:01:25 localhost iscsid[639]: Example: InitiatorName=iqn.2001-04.com.redhat:fc6.
Nov 24 09:01:25 localhost iscsid[639]: If using hardware iscsi like qla4xxx this message can be ignored.
Nov 24 09:01:25 localhost iscsid[639]: iscsid: can't open InitiatorAlias configuration file /etc/iscsi/initiatorname.iscsi
Nov 24 09:01:25 localhost systemd[1]: Started Open-iSCSI.
Nov 24 09:01:25 localhost systemd[1]: Started Create Volatile Files and Directories.
Nov 24 09:01:25 localhost systemd[1]: Reached target System Initialization.
Nov 24 09:01:25 localhost systemd[1]: Reached target Basic System.
Nov 24 09:01:25 localhost systemd[1]: Starting dracut initqueue hook...

--- Additional comment from kevin on 2020-11-25 01:59:27 UTC ---

[root@worker-1 ~]# journalctl -u coreos-generate-iscsi-initiatorname.service
-- Logs begin at Tue 2020-11-24 08:58:09 UTC, end at Wed 2020-11-25 01:58:17 UTC. --
-- No entries --

--- Additional comment from Jonathan Lebon on 2020-11-26 16:32:46 UTC ---

Fixed by https://github.com/openshift/installer/pull/4422.

--- Additional comment from Micah Abbott on 2020-11-30 21:32:20 UTC ---

(In reply to Jonathan Lebon from comment #10)
> Fixed by https://github.com/openshift/installer/pull/4422.

This PR has merged, so moving to MODIFIED

--- Additional comment from kevin on 2020-12-01 16:13:17 UTC ---

which OCP 4.6 minor version can be fix ?

--- Additional comment from Micah Abbott on 2020-12-01 21:44:46 UTC ---

(In reply to kevin from comment #12)
> which OCP 4.6 minor version can be fix ?

I would expect it to be part of 4.6.7

--- Additional comment from kevin on 2020-12-02 12:34:56 UTC ---

Thank you for your reply