Bug 1740827

Can you elaborate on what you mean by 'intropecting'  Are you just trying to chain load a kernel image, or are you doing something else?  If you have the ipxe script you are using that would be helpful

the problems have the same symptom, but different causes.  This bug seems to be dealing with a io error in a driver, whereas bz 1744638 seems to be result of a failure to exec the downloaded image from pxe (suggesting an error in the downloaded image).

As for this bug, I'm still waiting on a response to comment 3

ok, so I presume that its the chain load command thats failing?

Sorry, you never cleared the needinfo flag on this so I didn't see it.  That said, we are dealing with separate problems here, and we should generally not be attaching to bugs where separate drivers are involved.  Though in the most recent case, I believe the customer is out of luck.  The QLogic FastLinQ Ql41000 with pci Vendor ids:
8070
8080
8084
8090

arent supported by ipxe with any specific rom.  Im guessing that this NIC also doesn't support snp, meaning it likely won't work.  The customer should check with hp to see if their EFI firmware implements an SNP interface for this NIC, as that will be the only way it works.

As for the other devices, I assume that SNP was being used in the working case on the OpenConnect NIC?

Why are you clearing needinfo?  I asked you a question in comment 14

No, and you never cleared the needinfo flag when you updated this at the end of october, so I never saw your last update, please be more consistent there.

In response to comment #17, I don't know what you mean by 'both' cards.  As far as I read, you are only dealing with a single card.

In response  to comment #18, you can open a new bug sure, but as noted, I see no support for your card in ipxe, so unless they were using snp, and that broke, I'm not sure what we can do for you.

closing for lack of activity.

Ok, if they are hitting this problem again, lets reset, as this bug has been touched by multiple users with multiple bits of hardware.  Can you confirm for me what the pci ids (vendor/device/subvendor/subdevice) are?  If you provide me those for your customer, I can give you a support statement on the hardware in both RHEL7 and RHEL8

Hi Neil,

Thanks - here is the information requested:

0000:05:00.0]# cat subsystem_vendor
0x103c

0000:05:00.0]# cat vendor
0x10df

0000:05:00.0]# cat device
0x0720

0000:05:00.0]# cat subsystem_device
0x803f

Elf

so, I can confirm that there has never been a specific driver for this device, its always used the undi interface in both RHEL7 and RHEL8.

What that means is, assuming you are still seeing the error in the initial description (repeated here for reference):
Could not open net4: Input/output error (http://ipxe.org/1d6a4a98)
No more network devices.

its the hardware returning this error.

So its likely that this is a firmware bug of some sort that the new version of ipxe is likely triggering

There is one commit in the nii interface that was included with the latest ipxe, that may be involved here (upstream commit  6324227dcaa820436da4acd88ee174d3e26f0d54), which attempts to work around multiple firmware bugs, but may be failing in this case.  We can try a test build to determine if this is related, but I'm not sure what to do about it if this is the problem.  We may be able to introduce a device specific workaround here, but I'm very hesitant to do so, as if this works, it would really point to the firmware on the NIC needing to be corrected.

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=29778816
Thats a scratch build of ipxe with a change to enforce cable detection on the driver (which is what the above referenced commit eliminated for nics in which media was not yet detected).  Please have the customer give it a try.  If it works, we will have to have a discussion about how to go about actually fixing this bug, which may result in engaging with HP directly

Hi,

Customer has tested your brew build, and it works as expected - hopefully this confirms that this commit caused (indirectly if its a bug in the firmware!) the issue.

For now, I'll ask the custoemr to continue usign your build, but it would be good if I could have an update on the way forward - at your convenience.

Thanks again for your help on this,
Elf

well, as noted, this build just confirms that its the emulex firmware that has an issue.  If I had to guess, I would surmise that the error we are hitting is here:

static int nii_initialise_flags ( struct nii_nic *nii, unsigned int flags ) {
...
/* Issue command */
        op = NII_OP ( PXE_OPCODE_INITIALIZE, flags );
        if ( ( stat = nii_issue_cpb_db ( nii, op, &cpb, sizeof ( cpb ),
                                         &db, sizeof ( db ) ) ) < 0 ) {
                rc = -EIO_STAT ( stat );
                DBGC ( nii, "NII %s could not initialise: %s\n",
                       nii->dev.name, strerror ( rc ) );
                goto err_initialize;


I would guess that the system in question is running an older version of EFI firmware and doesn't recognize the PXE_OPFLAGS_INITIALIZE_DO_NOT_DETECT_CABLE flag, as defined in the UEFI spec here:
https://uefi.org/sites/default/files/resources/UEFI_Spec_2_8_final.pdf
in section E.4.7.2

Its likely returning PXE_STATCODE_INVALID_CDB, which we translate into  EIO.

We could hack something in place, like we did here as a test, but the right solution would be to get an EFI firmware update from HP that implemented support for the specified flags.  

What kind of system is this, its possible that an updated EFI firmware already exists

Hi neil,

The servers are HPE ProLiant DL360 Gen9 servers.  Is there any more info you require?

Thanks,
Elf

No, that should be enough.

Searching the HP website indicates that there is both a bios update:
https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_f78e4e6c13a0492bb18246efe1#tab4

And possibly a firmware update for the emulex card (hard to say as they don't list specific pci ids):
https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_1decee97677848a982da78fd28#tab4

The revision history up to the latest version indicates a few pxe fixes in both packages (but are somewhat unclear on their relation to this issue).  I suggest that we upgrade to the latest version of each, and if the issue isn't resolved, we can look at hacking in a workaround to ipxe while the customer brings the issue up with HP (I can provide documentation as to the details of the problem)

Copy that, let me know what the response is once you have it.

This is tough case then.  

I can dev-ack this bz for 7.9, but its very late in the development cycle, and so will need blocker/exception justification.  I'm not sure it will be granted, since RHEL7 is about to enter maintenance mode as well.

If you want to try, please set the exception flag to ? and we will see what happens.

If however, the customer is planning on migrating to RHEL8, we could move this bz to target 8.4 and fix it fairly easily there

Let me know what you want to do

I can't promise it will get in, but yes, if you want to move this to RHEL8, you can retarget this for 8.4 and set the z-stream flag

Ok have set it to 8.2 and set zstream flag - thanks

Neil - was this intended to be reassigned to the sst_virt or stay elsewhere?

I think so, virt is the default component for ipxe

Assigned to Ademar for initial triage per bz process and age of bug created or assigned to virt-maint without triage

Created attachment 1702378 [details]
vaguely hopeful iPXE patch, based on upstream iPXE b76052335788

commit 2de005e9ac399f7064968a2611f266dc86a96700
Author: Laszlo Ersek <lersek>
Date:   Fri Jul 24 19:15:28 2020 +0200

    [efi] perform cable detection at NII initialization on HPE 557SFP+
    
    Commit c0b61bad99ba ("[efi] Work around bugs in Emulex NII driver",
    2015-08-17) added PXE_OPFLAGS_INITIALIZE_DETECT_CABLE to nii_open() for
    working around Emulex NII driver bugs.
    
    That broke some Mellanox drivers, so commit 6324227dcaa8 ("[efi] Skip
    cable detection at initialisation where possible", 2017-03-19) predicated
    PXE_OPFLAGS_INITIALIZE_DETECT_CABLE on the NII driver's *inability* to
    report link status.
    
    This in turn breaks the NII driver on (some?) HPE 557SFP+ cards, as those
    drivers are capable of reporting link status, but they still need cable
    detection.
    
    So check for this card (through PCI vendor / device / subsys vendor /
    subsys device identifiers), and opt for cable detection regardless of link
    status reporting.
    
    Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1740827
    Signed-off-by: Laszlo Ersek <lersek>

Laszlo, yes, I can take this back over

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=30351331

Test build I have queued up to check in (using Laszlos patch).  Elf, can you please give it a try and confirm that it solves your problem?  Thanks!

ping, Elf, please test this out ASAP so I can check it in.

ELF had access to a system that exhibited the bug described here (the i/o error described in the initial comment).  I need you to take my scratch build, and run it on the system in question to ensure that error no longer occurs.

I'm honestly not sure, You would have to ask Elf how he provided it t the customer

In response

The benefit of making this change is that a large customer openstack deployment will continue to function without the need for a massive hardware upgrade

The risk to the release is minimal, as the patch is implmented as a hardware quirk, so it won't affect functionality for any hardware not listed in the quirk (currently on this single card)

Update: the same issue seems to have been addressed, independently, with a spanking new upstream iPXE commit:

ef2c844d01e7 ("[efi] Attempt NII initialisation both with and without cable detection", 2020-08-17)

It might make sense to brew a test RPM for the customer with upstream ef2c844d01e7 backported, replacing my quirk patch from comment 41 / comment 43.

(In reply to Elf Lewis from comment #35)
> Ok have set it to 8.2 and set zstream flag - thanks

I see zstream? but no ZTR set. Do you want this fixed for RHEL 8.2.z? If so, please set ZTR to 8.2.0. 

There is no assignee on this BZ, so I'm not sure who will do the z-stream backport. Erico, Do you know who could take this from the kernel team? Thanks!

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ipxe bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4836