Red Hat Bugzilla – Bug 168057
linux-iscsi initiator in rhel4 u2 does not recover from tpgt change via "iscsi reload"
Last modified: 2012-06-20 12:11:05 EDT
Description of problem: The linux-iscsi initiator does not recover well from a change to the tpgt in a session. This can happen in certain upgrading scenarios for our target, and there may be other targets that have similar needs. The change in tpgt does not really affect much other than a number, so in previous versions of the linux-iscsi driver an "iscsi reload" would cause the driver to update the tpgt during the session reconnection process. In the rewrite for RHEL4 though, this functionality was lost. Bugzilla 167946 was filed to track another related issue. I've filed this bugzilla to specifically track the change to tpgt. https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=167946 Version-Release number of selected component (if applicable): iscsi-initiator-utils How reproducible: Every time. Steps to Reproduce: 1. Connect session to target 2. On target side, change tpgt (on netapp filers, you can do this with the "iscsi tpgroup" command) for in progress session - this should terminate the session with the initiator and the initiator should try to reconnect 3. There is nothing you can do right now to recover the session on the initiator side except destroy the session and create a new one Actual results: Driver tries to reconnect the session and continuously fails because the tpgt has changed. Errors seen in /var/log/messages are like: Sep 9 16:31:27 tso kernel: iscsi-sfnet:host47: Portal group tag mismatch, expected 1, received 1001 Sep 9 16:31:27 tso kernel: iscsi-sfnet:host48: Portal group tag mismatch, expected 1, received 1002 Sep 9 16:31:27 tso kernel: iscsi-sfnet:host48: Waiting 1 seconds before next login attempt Sep 9 16:31:27 tso kernel: iscsi-sfnet:host47: Waiting 1 seconds before next login attempt The only way to recover from this is to blow the session away via sysfs. Expected results: Consider having "iscsi reload" tell the driver to pick up the new tpgt and reconnect the session without having to destroy it via sysfs. Additional info: Cut/pasting from previous bugzilla so we can have a context for this bugzilla: Comment #9 From Dave Wysochanski on 2005-09-09 17:39 EST [reply] If you want, I can file another bugzilla to track the tpgt scenario since it's really different than this one. New "iscsi-kill-session" script in attachment of this bugzilla applies to that case, not the general case of "iscsi reload" hanging. Comment #10 From Mike Christie on 2005-09-09 18:09 EST [reply] I think it is becuase of how we match sessions. If the portal group changes and iscsid uses the new value then it will never be able to update the old one. Comment #11 From Mike Christie on 2005-09-09 18:27 EST [reply] I cannot find the mail where we discussed the update problem. But if I remember correctly, we assumed iscsid would stay alive and that it would detect the change and kill the old bad session and start a new one. But if iscsid was not up for some reason then you were out of luck since we could not rebuild iscsids old state reliably. Comment #12 From Mike Christie on 2005-09-11 02:45 EST [reply] actually we did not want to kill the old session for updates since it will fail all the IO in flight and disrupt a bunc of other things. Comment #13 From Mike Christie on 2005-09-11 03:11 EST [reply] Yeah so we discussed this on the list. Krishna removed it and I asked him to add this functionality back. Obviosly he did not. It turns out to add it back now we will have to change the interfacae in U3 which I think would violate some RHEL thing. Problem is, for Cisco's ipv6 addition we will have to update the ioctl too. Tom what can I do in this case? Also Dave, can the targetname ever change? Comment #14 From Dave Wysochanski on 2005-09-11 12:01 EST [reply] Yes, conceivably it can change. But we need to be careful here. From the protocol perspective, I think the right thing to do in the case of a tpgt change (or anything that changes the I_T nexus identifiers), is to treat it as a different session. Here's the definition from RFC 3720: " - I_T nexus: According to [SAM2], the I_T nexus is a relationship between a SCSI Initiator Port and a SCSI Target Port. For iSCSI, this relationship is a session, defined as a relationship between an iSCSI Initiator's end of the session (SCSI Initiator Port) and the iSCSI Target's Portal Group. The I_T nexus can be identified by the conjunction of the SCSI port names; that is, the I_T nexus identifier is the tuple (iSCSI Initiator Name + ',i,'+ ISID, iSCSI Target Name + ',t,'+ Portal Group Tag)." I personally don't know why exactly Cisco didn't want to do that. Being a target vendor, I can guess at why they wanted the initiator to just "look the other way", but I'm not sure that's the right thing to do - seems unsafe for certain conceivable (though perhaps arguably pathalogical) cases. I guess that's why I went ahead and just said maybe we should just add the "iscsi-kill-session" thing in this case (for us, it'll probably be pretty rare, though we need to handle it). I need to ask though for sure. I think at one point we had an upgrade scenario where the tpgt could change even though nothing else changed, but then they went back and made the tpgt's backward compatible across upgrades. I need to check though. This also might be a good question for the ips reflector if we want to be sure.
Created attachment 118697 [details] iscsi-kill-session script
I heard back from someone internally that has dealt with this issue a lot before. As I thought, the right behavior is to tear down the session, fail any I/Os, then start all over. I don't think this was the case in previous linux-iscsi drivers (I/Os would not fail, but would just be resubmitted under the covers w/out LUN map revalidation, etc). This previous behavior was not really technically correct, but is probably practically ok. Here's the precise wording from the developer: My expectation of an initiator when the TPGT changes is for it to recognize that something has changed and re-discover its iSCSI targets and LUNs. My expectation of the target is to close all existing sessions tied to the target portal whose TPGT has changed (which we do). If the initiator doesn't do the right thing, the next best thing it can do is ignore the TPGT change altogether. Ignoring TPGT is not an expected behavior according to the RFC and we shouldn't ask any initiator developers to implement it as such. If possible, we should encourage them to do the right thing.
Just confirmed that on RHEL3 linux-iscsi, an "iscsi reload" will just cause an update of the tpgt, not an invalidation of the LUNs (I had a LUN mounted with I/O going to it, then changed the tpgt on the target, which shutdown the session, initiator was trying to reconnect and couldn't b/c of tpgt change, then issued "iscsi reload", and initiator reconnected and I/O resumed w/out any I/O errors). Looks like if we change the target name though, the LUN gets removed and I/Os will fail (see below). Not sure if you change an IP address if you'll get the tpgt change behavior or the target nodename change behavior - my guess is the former. Sep 12 15:28:46 rachman kernel: iSCSI: session dfff0000 portal group tag mismatch, expected 1001, received 1 Sep 12 15:28:46 rachman kernel: iSCSI: session dfff0000 retrying login to portal 1 at 6744844 Sep 12 15:28:46 rachman kernel: iSCSI: session dfff0000 to iqn.1992-08.com.netapp:sn.50393227.yanni waiting 1 seconds before next login attempt Sep 12 15:28:46 rachman iscsid[28350]: Connected to Discovery Address 10.60.155.94 Sep 12 15:28:46 rachman iscsid[28108]: updating bus 0 target 3 to configuration #2 Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 updating configuration of session dfff0000 to iqn.1992-08.com.netapp:sn.50393227.yanni Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 = iqn.1992-08.com.netapp:sn.50393227.yanni Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 portal 0 = address 10.60.155.94 port 3260 group 1 Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 portal 1 = address 10.60.155.95 port 3260 group 1 Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 portals have changed, failed to find a new portal in portal group 1002, session dfff0000 trying portal 0 group 1 Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 configuration updated at 6744961 while session dfff0000 to iqn.1992-08.com.netapp:sn.50393227.yanni is not established Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 trying to establish session dfff0000 to portal 0, address 10.60.155.94 port 3260 group 1 Sep 12 15:28:49 rachman kernel: iSCSI: bus 0 target 3 established session dfff0000 #3, portal 0, address 10.60.155.94 port 3260 group 1 Sep 12 15:28:49 rachman kernel: iSCSI: session dfff0000 recv_cmd d35a6600, cdb 0x2a, status 0x2, response 0x0, senselen 22, key 06, ASC/ASCQ 29/00, itt 140554 task ce6ec358 to (5 0 3 0), iqn.1992-08.com.netapp:sn.50393227.yanni Sep 12 15:28:49 rachman kernel: iSCSI: Sense f0000600 0000000e 00000000 29000000 0000 Sep 12 15:28:49 rachman kernel: iSCSI: session dfff0000 recv_cmd d35a6600, itt 140554, task ce6ec358 to (5 0 3 0), cdb 0x2a, U underflow, received 0, residual 8192, expected 8192 Sep 12 15:42:50 rachman kernel: iSCSI: session dfff024c target dropping all connections, reconnect min 0 max 0 Sep 12 15:42:50 rachman iscsid[28350]: Connection to Discovery Address 10.60.155.94 closed Sep 12 15:42:50 rachman kernel: iSCSI: session dfff0000 closed by target iqn.1992-08.com.netapp:sn.50393227.yanni at 6829312 Sep 12 15:42:50 rachman kernel: iSCSI: session dfff0000 to iqn.1992-08.com.netapp:sn.50393227.yanni dropped Sep 12 15:42:51 rachman kernel: iSCSI: bus 0 target 3 trying to establish session dfff0000 to portal 0, address 10.60.155.94 port 3260 group 1 Sep 12 15:42:51 rachman kernel: iSCSI: session dfff0000 to iqn.1992-08.com.netapp:sn.50393227.yanni failed to connect, rc -111, Connection refused Sep 12 15:42:51 rachman kernel: iSCSI: session dfff0000 connect failed at 6829358 Sep 12 15:42:51 rachman kernel: iSCSI: session dfff0000 to iqn.1992-08.com.netapp:sn.50393227.yanni waiting 1 seconds before next login attempt Sep 12 15:42:52 rachman kernel: iSCSI: bus 0 target 3 trying to establish session dfff0000 to portal 0, address 10.60.155.94 port 3260 group 1 Sep 12 15:42:52 rachman kernel: iSCSI: session dfff0000 login rejected: initiator error - target not found (02/03) Sep 12 15:42:52 rachman kernel: iSCSI: session dfff0000 giving up at 6829499 Sep 12 15:42:52 rachman kernel: iSCSI: session dfff0000 terminating, failing all SCSI commands Sep 12 15:42:52 rachman kernel: iSCSI: session dfff0000 failing command d35a6600 cdb 0x28 to (5 0 3 0) at 6829512 Sep 12 15:42:52 rachman kernel: iSCSI: session dfff0000 failing command cdb1d600 cdb 0x28 to (5 0 3 0) at 6829512 Sep 12 15:42:52 rachman kernel: iSCSI: session dfff0000 failing command d0831400 cdb 0x2a to (5 0 3 0) at 6829512 Sep 12 15:42:52 rachman kernel: Device 08:30 not ready. Sep 12 15:42:52 rachman kernel: I/O error: dev 08:30, sector 4460560 Sep 12 15:42:53 rachman kernel: iSCSI: session dfff0000 terminating, failing to queue d0831000 cdb 0x28 and any following commands to (5 0 3 0), iqn.1992-08.com.netapp:sn.50393227.yanni Sep 12 15:42:53 rachman kernel: Device 08:30 not ready. Sep 12 15:42:53 rachman kernel: I/O error: dev 08:30, sector 4460768 Sep 12 15:42:53 rachman kernel: Device 08:30 not ready. Sep 12 15:42:53 rachman kernel: I/O error: dev 08:30, sector 61160 Sep 12 15:42:53 rachman kernel: scsi5: remove-single-device 0 3 0 failed, device busy(1). Sep 12 15:42:53 rachman kernel: iSCSI: session dfff0000 error -16 writing 'scsi remove-single-device 5 0 3 0' to /proc/scsi/scsi Sep 12 15:42:53 rachman kernel: Device 08:30 not ready. Sep 12 15:42:53 rachman kernel: I/O error: dev 08:30, sector 61208 Sep 12 15:42:53 rachman kernel: Device 08:30 not ready. Sep 12 15:42:53 rachman kernel: I/O error: dev 08:30, sector 4460560 Sep 12 15:42:53 rachman kernel: Device 08:30 not ready. Sep 12 15:42:53 rachman kernel: I/O error: dev 08:30, sector 4460560 Sep 12 15:42:56 rachman kernel: iSCSI: queuecommand d0831400 failed to find a session for HBA cdb1db00, (5 0 3 0) Sep 12 15:42:56 rachman kernel: Device 08:30 not ready. Sep 12 15:42:56 rachman kernel: I/O error: dev 08:30, sector 61216 Sep 12 15:42:56 rachman kernel: Device 08:30 not ready. Sep 12 15:42:56 rachman kernel: I/O error: dev 08:30, sector 61232 Sep 12 15:43:10 rachman kernel: Device 08:30 not ready. Sep 12 15:43:10 rachman kernel: I/O error: dev 08:30, sector 4419952 Sep 12 15:43:11 rachman kernel: Device 08:30 not ready. Sep 12 15:43:11 rachman kernel: I/O error: dev 08:30, sector 4456448 Sep 12 15:43:11 rachman kernel: Device 08:30 not ready. Sep 12 15:43:11 rachman kernel: I/O error: dev 08:30, sector 6815744
Looks like in RHEL3 as long as the target nodename doesn't change, the IPs or tpgts can change without it affecting I/Os (mainly just an FYI).
Things like ips and tpgts just cause a logout then the commands are internally failed (like they are failed for our erl=0 handling) and setup for a requeue. Beucase that driver does scanning internally it can also probe luns without failing IO to upper layers and causing sd? to change. If you turn on debugging in iscsid you can tell if we are doing a a probe luns on this reconfig (iscsid -d 8) for sure.
I mean fo
I'd be ok with that if it wasn't for the fact that the scanning it does internally is just based on LUN # - if "LUN 0" was mapped before, and "LUN 0" is still mapped, it might be a different LUN but the rhel3 driver (or other drivers) won't recognize it (they do LUN map validation based on bitmaps, not INQUIRY pg 0x83 data or something like that). It's probably mostly a pathalogical case, but something to be aware of for rhel3 and prior drivers. We hit this when doing LUN resizes (in rhel3 if you resize a LUN, then do an "iscsi reload", the kernel will still think the LUN is the old size b/c the iscsi driver just sees that LU's are still mapped at the same #'s so it doesn't tell the kernel anything new).
ok with what? I was only describing the code and I thought we were only trying to fix RHEL4 to do something. Are you also wanting to fix RHEL3?
Sorry - was mainly just pointing out differences between rhel4 and rhel3 - not suggesting we change rhel3 behavior.
Created attachment 118738 [details] messages file snippit showing scsi_lib error after iscsi-kill-session when tpgt changed
In rhel4 I'm just looking for the best way to recover from this scenario. Right now the best I can come up with is just "iscsi stop/start" or something like that. Maybe that's the best answer right now since "iscsi reload" doesn't work like it used to and maybe it's too hard to get the old behavior back. It might be nice to be able to kill the one session and keep going but I'm not sure it's worth the effort - Mike you can probably make the call. Maybe it would be nice to make "iscsi-kill-session" safe and fix errors like the scsi_lib one but then again maybe not.
Changing Component to iscsi-initiator-utils for now. In the future please try to select iscsi-initiator-utils for userspace problems and kernel for driver problems. iSCSI is legacy Component from when it was all bundled in one rpm. Thanks.
Mike, just to let you know we're probably going to go with a workaround on our end so that getting the driver to recover from a tpgt change is probably not as important anymore. Not sure if you were working on it or not but you can probably close this or at least lower the priority until further notice and I can re-open if need be (unless you're getting other vendors that want this fixed). Thanks.
Thank you for submitting this issue for consideration in Red Hat Enterprise Linux. The release for which you requested us to review is now End of Life. Please See https://access.redhat.com/support/policy/updates/errata/ If you would like Red Hat to re-consider your feature request for an active release, please re-open the request via appropriate support channels and provide additional supporting details about the importance of this issue.