Description of problem: I had an iscsi initiator and target working on Fedora 12 2.6.31.x kernels. When I upgrade the the kernel on the initiator to 2.6.32 it breaks both header digest support and CHAP authentication. Version-Release number of selected component (if applicable): kernel-2.6.32.11-102.fc12.x86_64 How reproducible: Always Steps to Reproduce: 1. Setup an iscsi initiator and target on two Fedora 12 systems 2. Use turn on header digest and CHAP authentication 2. Upgrade the client to a 2.6.32.x kernel 3. Start the initiator Actual results: Fails to authentication and if authentication is disabled, fails to connect with header digest enabled Expected results: Works with both CHAP authentication and header digests Additional info: I tried upgrading the package to open-iscsi-2.0-871. In the process I recreated most of the patches for the newer version. It didn't help, but the older version didn't work either. Much older 2.6.32.x kernel packages had the same issue.
I don't see how just updating the kernel could break that.
- If you do not use CHAP or header digests does it work? If your use one other other does it work? - Could you send the /var/log/messages for the failure? - Did you also update the iscsi-initiator-utils package at the same time? You do not have to. I am just asking in case the rpm stuff got messed up and maybe the the config values got overwritten. - What log in is failing? It is the discovery session login (when you run iscsiadm -m discovery .... or the normal session login (when you run service iscsi start or iscsiadm -m node ... -l?
Yes, it is currently working after I disabled CHAP and header digests. I disabled CHAP first, and then header digests. I only upgraded iscsi-initiator-utils to a newer version in hopes that it would fix the issue. It didn't work either CHAP and Header digests before the upgrade of iscsi-initiator-utils. Discovery works, but normal session login fails. Login failing client side: Apr 18 20:52:10 proton iscsid: Login failed to authenticate with target iqn.2009-03.org.cygnusx-1:storage.md3 Apr 18 20:52:10 proton iscsid: session 1 login rejected: Initiator failed authentication with target Login failing server side: Apr 18 21:07:04 storage tgtd: chap_initiator_auth_check_response(412) No valid user/pass combination for initiator iqn.1994-05.com.fedora:5ee117cd725a found Apr 18 21:07:04 storage tgtd: conn_close(100) connection closed, 0xead178 1 Header digest failing client side: Apr 18 23:14:14 proton iscsid: Kernel reported iSCSI connection 1:0 error (1020) state (3) Apr 18 23:14:17 proton iscsid: connection1:0 is operational after recovery (1 attempts) Header digest failing server side: Apr 18 23:26:28 storage tgtd: iscsi_rx_handler(1988) rx hdr digest error 0x294daf4c calc 0x770a322 Apr 18 23:26:31 storage tgtd: iscsi_rx_handler(1988) rx hdr digest error 0xac09a285 calc 0x8f3ed306 Apr 18 23:26:34 storage tgtd: iscsi_rx_handler(1988) rx hdr digest error 0xdaf53741 calc 0xb2a6d829
(In reply to comment #3) > > Header digest failing server side: > Apr 18 23:26:28 storage tgtd: iscsi_rx_handler(1988) rx hdr digest error > 0x294daf4c calc 0x770a322 > Apr 18 23:26:31 storage tgtd: iscsi_rx_handler(1988) rx hdr digest error > 0xac09a285 calc 0x8f3ed306 > Apr 18 23:26:34 storage tgtd: iscsi_rx_handler(1988) rx hdr digest error > 0xdaf53741 calc 0xb2a6d829 Looks like the initiator box is now screwing up the disgest calculations or there is some corruption of the packets. Did you update the target box? What version of scsi-target-utils is that running? I will set this up here and see if I can replicate the problem. If I can then it could be a kernel problem in the digest calculations.
The server is currently running scsi-target-utils-1.0.1-0.fc12.x86_64. I this I upgraded to see if it would help when I was having this problem. I was running scsi-target-utils-0.9.5-3.fc12.x86_64 before.
I can replicate this here now. I am git bisecting the kernels to find the commit that added the regression. It will take about a day to finish building and testing kernels.
The cause of the regression is this commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=d3305f3407fa3e9452079ec6cc8379067456e4aa CHAP should work fine. The problem is in the digest calculation code. In that patch I moved the iscsi packet's cmdsn modification after we calculate the digest, so when we set it we change what the digest value should be and the target returns a error like it should. I am testing a patch and will attach it when it is done running.
Created attachment 408770 [details] Path to fix digest regression added in 2.6.32. Path to fix digest regression added in 2.6.32. I will be sending upstream and to the fedora kernel list. And when it is merged in a fedora kernel, I think you get automatic update notification in this bugzilla.
Fixed in 2.6.32.12-114
kernel-2.6.32.12-114.fc12 has been submitted as an update for Fedora 12. http://admin.fedoraproject.org/updates/kernel-2.6.32.12-114.fc12
Could it be that RHEL6 beta is also affected by this problem? I've been testing iSCSI with it, and it works fine up until I start enabling CHAP authentication. This is with kernel-2.6.32-19.el6.x86_64.
(In reply to comment #11) > Could it be that RHEL6 beta is also affected by this problem? I've been testing > iSCSI with it, and it works fine up until I start enabling CHAP authentication. > This is with kernel-2.6.32-19.el6.x86_64. RHEL 6 beta is affected. I sent a fix for RHEL6 already. It is not merged yet, but should get it in one of the next snapshots.
kernel-2.6.32.12-115.fc12 has been submitted as an update for Fedora 12. http://admin.fedoraproject.org/updates/kernel-2.6.32.12-115.fc12
kernel-2.6.32.12-115.fc12 has been pushed to the Fedora 12 stable repository. If problems still persist, please make note of it in this bug report.
This message is a reminder that Fedora 12 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 12. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '12'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 12's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 12 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Fedora 12 changed to end-of-life (EOL) status on 2010-12-02. Fedora 12 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. Thank you for reporting this bug and we are sorry it could not be fixed.