RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1744935 - RHEL 8.1: Unable to perform boot from SAN installation for iSCSI initiator over L2
Summary: RHEL 8.1: Unable to perform boot from SAN installation for iSCSI initiator ov...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: NetworkManager
Version: 8.1
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: 8.0
Assignee: Thomas Haller
QA Contact: Desktop QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-23 09:53 UTC by Manish Rangankar
Modified: 2020-11-04 01:49 UTC (History)
14 users (show)

Fixed In Version: NetworkManager-1.22.8-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-04 01:48:32 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)
iSCSI BFS tmp logs (13.57 MB, application/zip)
2019-08-23 09:53 UTC, Manish Rangankar 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
freedesktop.org Gitlab NetworkManager NetworkManager-ci merge_requests 624 0 None None None 2020-08-21 08:49:38 UTC

Description Manish Rangankar 2019-08-23 09:53:27 UTC 
Created attachment 1607263 [details]
iSCSI BFS tmp logs

Description of problem:
Unable to perform boot from SAN installation for iSCSI initiator over L2.
From the installation /tmp/syslogs, We can see ipaddress gets acquired correctly in the initial stag of installation, but at the end of the installation connection with iSCSI target gets reset because of interface loses its IPaddress.
I see some network manager logs before connection error i.e. iSCSI connection ping timeout (5 sec expiry). The network manager is messing up with ibft associated nic interface causes iscsi connection to timeout

Version-Release number of selected component (if applicable):

How reproducible:


Steps to Reproduce:
  1. HPE Quick AH adapter under HPE Gen10 
  2. Configure iSCSI L2 BFS in HII and make sure target login is successful at pre-boot level 
  3. Mount RHEL-8.1.0-Partner-Pre-Beta ISO 
  4. Add kernel boot parameter 'rd.iscsi.ibft'  during OS installation 
  5. Make sure target login successful with inbox drivers and start installation on the target Boot LUN 
  6. OS installation fails to proceed further and the observation is iBFT interface doesn't have any IP and fails to ping the target

Actual results:
L2 iSCSI Boot from SAN failed during installation

Expected results:
L2 iSCSI Boot from SAN to succeed.

Additional info:

/tmp directory debug logs are attached
Comment 1 Thomas Haller 2019-08-23 10:32:45 UTC 
Hi,

is there any chance to provide debug logs of NetworkManager?

See https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/contrib/fedora/rpm/NetworkManager.conf#n28 for infos about logging.
Basically, configure level=TRACE in /etc/NetworkManager/NetworkManager.conf and reproduce (reboot).
If possible, also disable rate-limiting of journal (see link) and provide full logs.


Thank you.
Comment 2 sudarsana.kalluru 2019-08-26 12:11:35 UTC 
Posting query on behalf of Marvell QA engineer.

Following command is not persistent after the reboot. We need to try it during BFS OS installation, not on the installed OS.
   configure level=TRACE in /etc/NetworkManager/NetworkManager.conf and reproduce (reboot).
Could you please provide instructions on how to collect the required logs during OS installation?
Comment 3 Manish Rangankar 2019-11-21 05:53:16 UTC 
In Comment2 QA folks provided the needed information
Comment 4 Beniamino Galvani 2020-04-02 07:58:29 UTC 
Relevant excerpt from logs:

 06:05:44,503 INFO dracut-initqueue:dhcp: PREINIT ibft0 up
 06:05:44,522 INFO dhclient:DHCPDISCOVER on ibft0 to 255.255.255.255 port 67 interval 6 (xid=0xaf84e91a)
 06:05:44,701 INFO dhclient:DHCPREQUEST on ibft0 to 255.255.255.255 port 67 (xid=0xaf84e91a)
 06:05:44,701 INFO dhclient:DHCPOFFER from 192.168.100.5
 06:05:44,732 INFO dhclient:DHCPACK from 192.168.100.5 (xid=0xaf84e91a)
 06:05:46,760 INFO dhclient:bound to 192.168.100.25 -- renewal in 225 seconds.

 06:06:18,965 INFO NetworkManager:<info>  [1562911578.9651] NetworkManager (version 1.14.0-14.el8) is starting... (for the first time)

 06:06:18,972 DEBUG NetworkManager:<debug> [1562911578.9728] platform: signal: address 4   added: 192.168.100.25/24 lft 568sec pref 568sec lifetime 1-1[568,568] dev 6 src kernel

 06:06:24,085 DEBUG nm-dispatcher:req:3 'up' [ibft0]: environment: DEVICE_IFACE=ibft0
 06:06:24,085 DEBUG nm-dispatcher:req:3 'up' [ibft0]: environment: DEVICE_IP_IFACE=ibft0
 06:06:24,085 DEBUG nm-dispatcher:req:3 'up' [ibft0]: environment: IP4_ADDRESS_0=192.168.100.25/24 192.168.100.5

 06:15:46,032 DEBUG NetworkManager:<debug> [1562912146.0326] platform: signal: address 4 removed: 192.168.100.25/24 lft 0sec pref 0sec lifetime 569-1[568,568] dev 6 src kernel
 06:15:46,033 DEBUG NetworkManager:<debug> [1562912146.0327] device[0x5562139cf6a0] (ibft0): queued IP4 config change

 06:15:52,176 ERR kernel: connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295278592, last ping 4295283712, now 4295288832
 06:15:52,433 WARNING iscsid:Kernel reported iSCSI connection 1:0 error (1022 - ISCSI_ERR_NOP_TIMEDOUT: A NOP has timed out) state (3)

where it can be seen that the initrd gets address 192.168.100.5 via
DHCP with a lifetime of 10 minutes and NetworkManager lets the address
expire, leading to iSCSI errors in accessing the filesystem.

This will be fixed in RHEL 8.2 by commit [1].

[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/f2fe6c03ee3fe5e262cbaa5b898f05bb8e349d0e
Comment 5 Manish Rangankar 2020-04-06 09:18:35 UTC 
Got verification confirmation from our testing team also and they have verified iSCSI IBFT (non-offload) DHCP IPv4 on RHEL-8.2.0-Partner-Snapshot3 with Inbox drivers (qed/qede:8.37.0.20). It works fine.

Thanks,
Manish
Comment 6 Thomas Haller 2020-04-08 15:41:22 UTC 
It seems this is effectively fixed on rhel-8.2.

Note that from rhel-8.3 onwards, RHEL will default to run NetworkManager in dracut. That means, the implementation for setting up networking in early boot changes completely. Of course, the functionality is supposed to still work.

Moving to MODIFIED.
Comment 9 Filip Pokryvka 2020-08-21 08:49:38 UTC 
We have some iSCSI dracut tests covered, but we are not able to test ibft.table with DHCP scenario. We test ibft.table with static adresses and getting iSCSI root from DHCP (dynamic adresses).
Comment 12 errata-xmlrpc 2020-11-04 01:48:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4499
Note You need to log in before you can comment on or make changes to this bug.