Bug 1904243 - RHCOS 4.6.1 missing ISCSI initiatorname.iscsi !
Summary: RHCOS 4.6.1 missing ISCSI initiatorname.iscsi !
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RHCOS
Version: 4.7
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: ---
: 4.7.0
Assignee: Jonathan Lebon
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On: 1901021
Blocks: 1186913 1899176
TreeView+ depends on / blocked
 
Reported: 2020-12-03 21:48 UTC by Micah Abbott
Modified: 2021-02-24 15:39 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The service unit which normally regenerates the iSCSI initiator name on first boot had its name in a systemd preset mistyped, causing it to not run. Consequence: The iSCSI initiator name wasn't regenerated on first-boot. Fix: The service unit's name in the preset was corrected. Result: The iSCSI initiator name is now regenerated on first-boot.
Clone Of: 1901021
Environment:
Last Closed: 2021-02-24 15:37:53 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:39:59 UTC

Description Micah Abbott 2020-12-03 21:48:12 UTC
+++ This bug was initially created as a clone of Bug #1901021 +++

Description of problem:

According OpenShift 4.6 Official document, the RHCOS worker node's default ISCSI initiator name is in /etc/iscsi/initiatorname.iscsi

https://docs.openshift.com/container-platform/4.6/storage/persistent_storage/persistent-storage-iscsi.html

But we cannot find this file in RHCOS 4.6.1 !!!


[core@worker-1 ~]$ ls -a /etc/iscsi/
.  ..  iscsid.conf

[core@worker-1 ~]$ hostnamectl
   Static hostname: n/a
Transient hostname: worker-1.ocp4-1.example.internal
         Icon name: computer-vm
           Chassis: vm
        Machine ID: ada997b5da89423e8255633f232d8db4
           Boot ID: f4d48fddb7f0499c97dede5fd1ec47c1
    Virtualization: vmware
  Operating System: Red Hat Enterprise Linux CoreOS 46.82.202010091720-0 (Ootpa)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:8::coreos
            Kernel: Linux 4.18.0-193.24.1.el8_2.dt1.x86_64
      Architecture: x86-64

--- Additional comment from kevin on 2020-11-24 10:15:31 UTC ---

In RHCOS 4.5, the initiatorname.iscsi default existed. But in RHCOS 4.6.1, this file is missing!

--- Additional comment from Luca BRUNO on 2020-11-24 11:04:29 UTC ---

Misc notes:
 * The file under /etc should is not supposed be part of the base OS in both RHCOS-4.5 and RHCOS-4.6. The InitatorName is machine-specific, thus it shouldn't be backed into the OS image
 * The RHEL package has some buggy %post logic in the RPM that we have to bypass on CoreOS side, see https://bugzilla.redhat.com/show_bug.cgi?id=1734144 
 * We carry some service units specific to CoreOS in order to generate the initiatorname on first-boot, see https://github.com/openshift/os/blob/a06951cb0b9ec63d79087ca257a2135601e027dd/overlay.d/05rhcos/usr/lib/systemd/system/coreos-generate-iscsi-initiatorname.service
 * There have been some work around this area in the 4.5->4.6 timeframe which may have resulted in a regression, see https://gitlab.cee.redhat.com/coreos/redhat-coreos/-/merge_requests/1098

I would be good to attach here the full journal from this 4.6 node to pinpoint what's going on.

--- Additional comment from Luca BRUNO on 2020-11-24 11:14:53 UTC ---

Ah, I think MR#1098 above renamed the service unit (s/regenerate/generate/), but the `43-manifest-rhcos.preset` content still tries to reference/enable the unit using the old name.

Jonathan, can you maybe have a look here?

Micah, did you verify https://bugzilla.redhat.com/show_bug.cgi?id=1868174 using a patched bootimage?

--- Additional comment from Scott Dodson on 2020-11-24 13:55:32 UTC ---



--- Additional comment from Jonathan Lebon on 2020-11-24 14:25:01 UTC ---

(In reply to Luca BRUNO from comment #3)
> Ah, I think MR#1098 above renamed the service unit (s/regenerate/generate/),
> but the `43-manifest-rhcos.preset` content still tries to reference/enable
> the unit using the old name.

Hmm indeed, you're right. I'm confused now how this worked during testing and verification.
Anyway, will fix this and add a test for it as well.

--- Additional comment from Jonathan Lebon on 2020-11-24 15:08:49 UTC ---

https://github.com/openshift/os/pull/453
https://gitlab.cee.redhat.com/coreos/redhat-coreos/-/merge_requests/1180

--- Additional comment from Benjamin Gilbert on 2020-11-24 16:06:32 UTC ---

Adding to bootimage bump per Jonathan.

--- Additional comment from kevin on 2020-11-24 16:30:39 UTC ---

I have aggregate some log from journal related to iscsi

[root@worker-1 ~]# journalctl  |grep "can't open InitiatorAlias configuration file /etc/iscsi/initiatorname.iscsi" -B 5 -A 5
Nov 24 08:58:12 localhost multipathd[688]: /etc/multipath.conf. See man mpathconf(8) for more details
Nov 24 08:58:12 localhost iscsid[690]: iscsid: can't open InitiatorName configuration file /etc/iscsi/initiatorname.iscsi
Nov 24 08:58:12 localhost iscsid[690]: iscsid: Warning: InitiatorName file /etc/iscsi/initiatorname.iscsi does not exist or does not contain a properly formatted InitiatorName. If using software iscsi (iscsi_tcp or ib_iser) or partial offload (bnx2i or cxgbi iscsi), you may not be able to log into or discover targets. Please create a file /etc/iscsi/initiatorname.iscsi that contains a sting with the format: InitiatorName=iqn.yyyy-mm.<reversed domain name>[:identifier].
Nov 24 08:58:12 localhost iscsid[690]: Example: InitiatorName=iqn.2001-04.com.redhat:fc6.
Nov 24 08:58:12 localhost iscsid[690]: If using hardware iscsi like qla4xxx this message can be ignored.
Nov 24 08:58:12 localhost iscsid[690]: iscsid: can't open InitiatorAlias configuration file /etc/iscsi/initiatorname.iscsi
Nov 24 08:58:12 localhost systemd[1]: Started Open-iSCSI.
Nov 24 08:58:12 localhost systemd[1]: Started Create Volatile Files and Directories.
Nov 24 08:58:12 localhost systemd[1]: Reached target System Initialization.
Nov 24 08:58:12 localhost systemd[1]: Reached target Basic System.
Nov 24 08:58:12 localhost systemd[1]: Starting Ignition (fetch-offline)...
--
Nov 24 09:01:25 localhost multipathd[638]: /etc/multipath.conf. See man mpathconf(8) for more details
Nov 24 09:01:25 localhost iscsid[639]: iscsid: can't open InitiatorName configuration file /etc/iscsi/initiatorname.iscsi
Nov 24 09:01:25 localhost iscsid[639]: iscsid: Warning: InitiatorName file /etc/iscsi/initiatorname.iscsi does not exist or does not contain a properly formatted InitiatorName. If using software iscsi (iscsi_tcp or ib_iser) or partial offload (bnx2i or cxgbi iscsi), you may not be able to log into or discover targets. Please create a file /etc/iscsi/initiatorname.iscsi that contains a sting with the format: InitiatorName=iqn.yyyy-mm.<reversed domain name>[:identifier].
Nov 24 09:01:25 localhost iscsid[639]: Example: InitiatorName=iqn.2001-04.com.redhat:fc6.
Nov 24 09:01:25 localhost iscsid[639]: If using hardware iscsi like qla4xxx this message can be ignored.
Nov 24 09:01:25 localhost iscsid[639]: iscsid: can't open InitiatorAlias configuration file /etc/iscsi/initiatorname.iscsi
Nov 24 09:01:25 localhost systemd[1]: Started Open-iSCSI.
Nov 24 09:01:25 localhost systemd[1]: Started Create Volatile Files and Directories.
Nov 24 09:01:25 localhost systemd[1]: Reached target System Initialization.
Nov 24 09:01:25 localhost systemd[1]: Reached target Basic System.
Nov 24 09:01:25 localhost systemd[1]: Starting dracut initqueue hook...

--- Additional comment from kevin on 2020-11-25 01:59:27 UTC ---

[root@worker-1 ~]# journalctl -u coreos-generate-iscsi-initiatorname.service
-- Logs begin at Tue 2020-11-24 08:58:09 UTC, end at Wed 2020-11-25 01:58:17 UTC. --
-- No entries --

--- Additional comment from Jonathan Lebon on 2020-11-26 16:32:46 UTC ---

Fixed by https://github.com/openshift/installer/pull/4422.

--- Additional comment from Micah Abbott on 2020-11-30 21:32:20 UTC ---

(In reply to Jonathan Lebon from comment #10)
> Fixed by https://github.com/openshift/installer/pull/4422.

This PR has merged, so moving to MODIFIED

--- Additional comment from kevin on 2020-12-01 16:13:17 UTC ---

which OCP 4.6 minor version can be fix ?

--- Additional comment from Micah Abbott on 2020-12-01 21:44:46 UTC ---

(In reply to kevin from comment #12)
> which OCP 4.6 minor version can be fix ?

I would expect it to be part of 4.6.7

--- Additional comment from kevin on 2020-12-02 12:34:56 UTC ---

Thank you for your reply

Comment 2 Micah Abbott 2020-12-05 16:00:44 UTC
This is pending the merge of the installer PR; setting UpcomingSprint to appease the bots.

Comment 3 Micah Abbott 2020-12-08 21:33:00 UTC
Installer PR is merged, moving to MODIFIED

Comment 5 Micah Abbott 2020-12-09 22:10:41 UTC
Verified with 4.6.0-0.nightly-2020-12-09-162454 / RHCOS 46.82.202012082041-0

```
[core@cosa-devsh ~]$ rpm-ostree status 
State: idle
Deployments:
* ostree://0c0f8983e459f70654a92f1deb03e74814d881c814416f43a8433bc7f91bb1c8
                   Version: 46.82.202012082041-0 (2020-12-08T20:45:12Z)
[core@cosa-devsh ~]$ ls /etc/iscsi/
initiatorname.iscsi  iscsid.conf
[core@cosa-devsh ~]$ cat /etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.1994-05.com.redhat:26bc2b78e478
[core@cosa-devsh ~]$ systemctl status coreos-generate-iscsi-initiatorname
● coreos-generate-iscsi-initiatorname.service - CoreOS Generate iSCSI Initiator Name
   Loaded: loaded (/usr/lib/systemd/system/coreos-generate-iscsi-initiatorname.service; enabled; vendor preset: enabled)
   Active: active (exited) since Wed 2020-12-09 22:06:49 UTC; 2min 23s ago
     Docs: https://bugzilla.redhat.com/show_bug.cgi?id=1493296
           https://bugzilla.redhat.com/show_bug.cgi?id=1687722
  Process: 1422 ExecStart=/usr/bin/sh -c echo "InitiatorName=`/usr/sbin/iscsi-iname`" > /etc/iscsi/initiatorname.iscsi (code=exited, status=0/SUCCESS)
 Main PID: 1422 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 5724)
   Memory: 0B
   CGroup: /system.slice/coreos-generate-iscsi-initiatorname.service

Dec 09 22:06:49 localhost systemd[1]: Starting CoreOS Generate iSCSI Initiator Name...
Dec 09 22:06:49 localhost systemd[1]: Started CoreOS Generate iSCSI Initiator Name.
```

Comment 8 errata-xmlrpc 2021-02-24 15:37:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.