Bug 168057

Summary: linux-iscsi initiator in rhel4 u2 does not recover from tpgt change via "iscsi reload"
Product: Red Hat Enterprise Linux 4 Reporter: Dave Wysochanski <davidw>
Component: iscsi-initiator-utilsAssignee: Mike Christie <mchristi>
Status: CLOSED WONTFIX QA Contact: Brock Organ <borgan>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.0CC: 157070.alewis, coughlan
Target Milestone: ---   
Target Release: ---   
Hardware: alpha   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-20 16:11:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
iscsi-kill-session script
none
messages file snippit showing scsi_lib error after iscsi-kill-session when tpgt changed none

Description Dave Wysochanski 2005-09-11 21:41:16 UTC 
Description of problem:
The linux-iscsi initiator does not recover well from a change to the tpgt in a
session.  This can happen in certain upgrading scenarios for our target, and
there may be other targets that have similar needs.  The change in tpgt does not
really affect much other than a number, so in previous versions of the
linux-iscsi driver an "iscsi reload" would cause the driver to update the tpgt
during the session reconnection process.  In the rewrite for RHEL4 though, this
functionality was lost.

Bugzilla 167946 was filed to track another related issue.  I've filed this
bugzilla to specifically track the change to tpgt.
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=167946


Version-Release number of selected component (if applicable):
iscsi-initiator-utils

How reproducible:
Every time.


Steps to Reproduce:
1. Connect session to target
2. On target side, change tpgt (on netapp filers, you can do this with the
"iscsi tpgroup" command) for in progress session - this should terminate the
session with the initiator and the initiator should try to reconnect
3. There is nothing you can do right now to recover the session on the initiator
side except destroy the session and create a new one
  
Actual results:
Driver tries to reconnect the session and continuously fails because the tpgt
has changed.  Errors seen in /var/log/messages are like:
Sep  9 16:31:27 tso kernel: iscsi-sfnet:host47: Portal group tag mismatch,
expected 1, received 1001
Sep  9 16:31:27 tso kernel: iscsi-sfnet:host48: Portal group tag mismatch,
expected 1, received 1002
Sep  9 16:31:27 tso kernel: iscsi-sfnet:host48: Waiting 1 seconds before next
login attempt
Sep  9 16:31:27 tso kernel: iscsi-sfnet:host47: Waiting 1 seconds before next
login attempt

The only way to recover from this is to blow the session away via sysfs.


Expected results:
Consider having "iscsi reload" tell the driver to pick up the new tpgt and
reconnect the session without having to destroy it via sysfs.


Additional info:
Cut/pasting from previous bugzilla so we can have a context for this bugzilla:

Comment #9 From Dave Wysochanski  	 on 2005-09-09 17:39 EST  	[reply]  	   	 

If you want, I can file another bugzilla to track the tpgt scenario since it's
really different than this one.  New "iscsi-kill-session" script in attachment
of this bugzilla applies to that case, not the general case of "iscsi reload"
hanging.


Comment #10 From Mike Christie 	on 2005-09-09 18:09 EST 	[reply] 	  	 

I think it is becuase of how we match sessions. If the portal group changes and
iscsid uses the new value then it will never be able to update the old one.


Comment #11 From Mike Christie 	on 2005-09-09 18:27 EST 	[reply] 	  	 

I cannot find the mail where we discussed the update problem. But if I remember
correctly, we assumed iscsid would stay alive and that it would detect the
change and kill the old bad session and start a new one. But if iscsid was not
up for some reason then you were out of luck since we could not rebuild iscsids
old state reliably.


Comment #12 From Mike Christie 	on 2005-09-11 02:45 EST 	[reply] 	  	 

actually we did not want to kill the old session for updates since it will fail
all the IO in flight and disrupt a bunc of other things.


Comment #13 From Mike Christie 	on 2005-09-11 03:11 EST 	[reply] 	  	 

Yeah so we discussed this on the list. Krishna removed it and I asked him to add
this functionality back. Obviosly he did not. It turns out to add it back now we
will have to change the interfacae in U3 which I think would violate some RHEL
thing. Problem is, for Cisco's ipv6 addition we will have to update the ioctl too.

Tom what can I do in this case?

Also Dave, can the targetname ever change?


Comment #14 From Dave Wysochanski 	on 2005-09-11 12:01 EST 	[reply] 	  	 

Yes, conceivably it can change.  But we need to be careful here.

From the protocol perspective, I think the right thing to do in the case of a
tpgt change (or anything that changes the I_T nexus identifiers), is to treat it
as a different session.  Here's the definition from RFC 3720:
"   - I_T nexus: According to [SAM2], the I_T nexus is a relationship
     between a SCSI Initiator Port and a SCSI Target Port.  For iSCSI,
     this relationship is a session, defined as a relationship between
     an iSCSI Initiator's end of the session (SCSI Initiator Port) and
     the iSCSI Target's Portal Group.  The I_T nexus can be identified
     by the conjunction of the SCSI port names; that is, the I_T nexus
     identifier is the tuple (iSCSI Initiator Name + ',i,'+ ISID, iSCSI
     Target Name + ',t,'+ Portal Group Tag)."

I personally don't know why exactly Cisco didn't want to do that.  Being a
target vendor, I can guess at why they wanted the initiator to just "look the
other way", but I'm not sure that's the right thing to do - seems unsafe for
certain conceivable (though perhaps arguably pathalogical) cases.

I guess that's why I went ahead and just said maybe we should just add the
"iscsi-kill-session" thing in this case (for us, it'll probably be pretty rare,
though we need to handle it).  I need to ask though for sure.  I think at one
point we had an upgrade scenario where the tpgt could change even though nothing
else changed, but then they went back and made the tpgt's backward compatible
across upgrades.  I need to check though.

This also might be a good question for the ips reflector if we want to be sure.
Comment 1 Dave Wysochanski 2005-09-11 21:41:16 UTC 
Created attachment 118697 [details]
iscsi-kill-session script
Comment 2 Dave Wysochanski 2005-09-12 14:17:32 UTC 
I heard back from someone internally that has dealt with this issue a lot
before.  As I thought, the right behavior is to tear down the session, fail any
I/Os, then start all over.  I don't think this was the case in previous
linux-iscsi drivers (I/Os would not fail, but would just be resubmitted under
the covers w/out LUN map revalidation, etc).  This previous behavior was not
really technically correct, but is probably practically ok.  Here's the precise
wording from the developer:

       My expectation of an initiator when the TPGT changes is for it to 
recognize that something has changed and re-discover its iSCSI targets and LUNs. 
My expectation of the target is to close all existing sessions tied to the 
target portal whose TPGT has changed (which we do).

       If the initiator doesn't do the right thing, the next best thing it can 
do is ignore the TPGT change altogether. Ignoring TPGT is not an expected 
behavior according to the RFC and we shouldn't ask any initiator developers to 
implement it as such. If possible, we should encourage them to do the right thing.
Comment 3 Dave Wysochanski 2005-09-12 14:43:33 UTC 
Just confirmed that on RHEL3 linux-iscsi, an "iscsi reload" will just cause an
update of the tpgt, not an invalidation of the LUNs (I had a LUN mounted with
I/O going to it, then changed the tpgt on the target, which shutdown the
session, initiator was trying to reconnect and couldn't b/c of tpgt change, then
issued "iscsi reload", and initiator reconnected and I/O resumed w/out any I/O
errors).  Looks like if we change the target name though, the LUN gets removed
and I/Os will fail (see below).  Not sure if you change an IP address if you'll
get the tpgt change behavior or the target nodename change behavior - my guess
is the former.


Sep 12 15:28:46 rachman kernel: iSCSI: session dfff0000 portal group tag
mismatch, expected 1001, received 1
Sep 12 15:28:46 rachman kernel: iSCSI: session dfff0000 retrying login to portal
1 at 6744844
Sep 12 15:28:46 rachman kernel: iSCSI: session dfff0000 to
iqn.1992-08.com.netapp:sn.50393227.yanni waiting 1 seconds before next login attempt
Sep 12 15:28:46 rachman iscsid[28350]: Connected to Discovery Address 10.60.155.94
Sep 12 15:28:46 rachman iscsid[28108]: updating bus 0 target 3 to configuration #2
Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 updating configuration of
session dfff0000 to iqn.1992-08.com.netapp:sn.50393227.yanni
Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 =
iqn.1992-08.com.netapp:sn.50393227.yanni
Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 portal 0 = address
10.60.155.94 port 3260 group 1
Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 portal 1 = address
10.60.155.95 port 3260 group 1
Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 portals have changed,
failed to find a new portal in portal group 1002, session dfff0000 trying portal
0 group 1
Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 configuration updated at
6744961 while session dfff0000 to iqn.1992-08.com.netapp:sn.50393227.yanni is
not established
Sep 12 15:28:47 rachman kernel: iSCSI: bus 0 target 3 trying to establish
session dfff0000 to portal 0, address 10.60.155.94 port 3260 group 1
Sep 12 15:28:49 rachman kernel: iSCSI: bus 0 target 3 established session
dfff0000 #3, portal 0, address 10.60.155.94 port 3260 group 1
Sep 12 15:28:49 rachman kernel: iSCSI: session dfff0000 recv_cmd d35a6600, cdb
0x2a, status 0x2, response 0x0, senselen 22, key 06, ASC/ASCQ 29/00, itt 140554
task ce6ec358 to (5 0 3 0), iqn.1992-08.com.netapp:sn.50393227.yanni
Sep 12 15:28:49 rachman kernel: iSCSI: Sense f0000600 0000000e 00000000 29000000
0000
Sep 12 15:28:49 rachman kernel: iSCSI: session dfff0000 recv_cmd d35a6600, itt
140554, task ce6ec358 to (5 0 3 0), cdb 0x2a,  U underflow, received 0, residual
8192, expected 8192
Sep 12 15:42:50 rachman kernel: iSCSI: session dfff024c target dropping all
connections, reconnect min 0 max 0
Sep 12 15:42:50 rachman iscsid[28350]: Connection to Discovery Address
10.60.155.94 closed
Sep 12 15:42:50 rachman kernel: iSCSI: session dfff0000 closed by target
iqn.1992-08.com.netapp:sn.50393227.yanni at 6829312
Sep 12 15:42:50 rachman kernel: iSCSI: session dfff0000 to
iqn.1992-08.com.netapp:sn.50393227.yanni dropped
Sep 12 15:42:51 rachman kernel: iSCSI: bus 0 target 3 trying to establish
session dfff0000 to portal 0, address 10.60.155.94 port 3260 group 1
Sep 12 15:42:51 rachman kernel: iSCSI: session dfff0000 to
iqn.1992-08.com.netapp:sn.50393227.yanni failed to connect, rc -111, Connection
refused
Sep 12 15:42:51 rachman kernel: iSCSI: session dfff0000 connect failed at 6829358
Sep 12 15:42:51 rachman kernel: iSCSI: session dfff0000 to
iqn.1992-08.com.netapp:sn.50393227.yanni waiting 1 seconds before next login attempt
Sep 12 15:42:52 rachman kernel: iSCSI: bus 0 target 3 trying to establish
session dfff0000 to portal 0, address 10.60.155.94 port 3260 group 1
Sep 12 15:42:52 rachman kernel: iSCSI: session dfff0000 login rejected:
initiator error - target not found (02/03)
Sep 12 15:42:52 rachman kernel: iSCSI: session dfff0000 giving up at 6829499
Sep 12 15:42:52 rachman kernel: iSCSI: session dfff0000 terminating, failing all
SCSI commands
Sep 12 15:42:52 rachman kernel: iSCSI: session dfff0000 failing command d35a6600
cdb 0x28 to (5 0 3 0) at 6829512
Sep 12 15:42:52 rachman kernel: iSCSI: session dfff0000 failing command cdb1d600
cdb 0x28 to (5 0 3 0) at 6829512
Sep 12 15:42:52 rachman kernel: iSCSI: session dfff0000 failing command d0831400
cdb 0x2a to (5 0 3 0) at 6829512
Sep 12 15:42:52 rachman kernel: Device 08:30 not ready.
Sep 12 15:42:52 rachman kernel:  I/O error: dev 08:30, sector 4460560
Sep 12 15:42:53 rachman kernel: iSCSI: session dfff0000 terminating, failing to
queue d0831000 cdb 0x28 and any following commands to (5 0 3 0),
iqn.1992-08.com.netapp:sn.50393227.yanni
Sep 12 15:42:53 rachman kernel: Device 08:30 not ready.
Sep 12 15:42:53 rachman kernel:  I/O error: dev 08:30, sector 4460768
Sep 12 15:42:53 rachman kernel: Device 08:30 not ready.
Sep 12 15:42:53 rachman kernel:  I/O error: dev 08:30, sector 61160
Sep 12 15:42:53 rachman kernel: scsi5: remove-single-device 0 3 0 failed, device
busy(1).
Sep 12 15:42:53 rachman kernel: iSCSI: session dfff0000 error -16 writing 'scsi
remove-single-device 5 0 3 0' to /proc/scsi/scsi
Sep 12 15:42:53 rachman kernel: Device 08:30 not ready.
Sep 12 15:42:53 rachman kernel:  I/O error: dev 08:30, sector 61208
Sep 12 15:42:53 rachman kernel: Device 08:30 not ready.
Sep 12 15:42:53 rachman kernel:  I/O error: dev 08:30, sector 4460560
Sep 12 15:42:53 rachman kernel: Device 08:30 not ready.
Sep 12 15:42:53 rachman kernel:  I/O error: dev 08:30, sector 4460560
Sep 12 15:42:56 rachman kernel: iSCSI: queuecommand d0831400 failed to find a
session for HBA cdb1db00, (5 0 3 0)
Sep 12 15:42:56 rachman kernel: Device 08:30 not ready.
Sep 12 15:42:56 rachman kernel:  I/O error: dev 08:30, sector 61216
Sep 12 15:42:56 rachman kernel: Device 08:30 not ready.
Sep 12 15:42:56 rachman kernel:  I/O error: dev 08:30, sector 61232
Sep 12 15:43:10 rachman kernel: Device 08:30 not ready.
Sep 12 15:43:10 rachman kernel:  I/O error: dev 08:30, sector 4419952
Sep 12 15:43:11 rachman kernel: Device 08:30 not ready.
Sep 12 15:43:11 rachman kernel:  I/O error: dev 08:30, sector 4456448
Sep 12 15:43:11 rachman kernel: Device 08:30 not ready.
Sep 12 15:43:11 rachman kernel:  I/O error: dev 08:30, sector 6815744
Comment 4 Dave Wysochanski 2005-09-12 14:51:04 UTC 
Looks like in RHEL3 as long as the target nodename doesn't change, the IPs or
tpgts can change without it affecting I/Os (mainly just an FYI).
Comment 5 Mike Christie 2005-09-12 18:31:34 UTC 
Things like ips and tpgts just cause a logout then the commands are internally
failed (like they are failed for our erl=0 handling) and setup for a requeue. 

Beucase that driver does scanning internally it can also probe luns without
failing IO to upper layers and causing sd? to change. If you turn on debugging
in iscsid you can tell if we are doing a a probe luns on this reconfig (iscsid
-d 8) for sure.
Comment 6 Mike Christie 2005-09-12 18:32:48 UTC 
I mean fo
Comment 7 Dave Wysochanski 2005-09-12 18:42:52 UTC 
I'd be ok with that if it wasn't for the fact that the scanning it does
internally is just based on LUN # - if "LUN 0" was mapped before, and "LUN 0" is
still mapped, it might be a different LUN but the rhel3 driver (or other
drivers) won't recognize it (they do LUN map validation based on bitmaps, not
INQUIRY pg 0x83 data or something like that).  It's probably mostly a
pathalogical case, but something to be aware of for rhel3 and prior drivers.  We
hit this when doing LUN resizes (in rhel3 if you resize a LUN, then do an "iscsi
reload", the kernel will still think the LUN is the old size b/c the iscsi
driver just sees that LU's are still mapped at the same #'s so it doesn't tell
the kernel anything new).
Comment 8 Mike Christie 2005-09-12 18:47:53 UTC 
ok with what? I was only describing the code and I thought we were only trying
to fix RHEL4 to do something. Are you also wanting to fix RHEL3?
Comment 9 Dave Wysochanski 2005-09-12 20:03:18 UTC 
Sorry - was mainly just pointing out differences between rhel4 and rhel3 - not
suggesting we change rhel3 behavior.
Comment 10 Dave Wysochanski 2005-09-12 20:14:23 UTC 
Created attachment 118738 [details]
messages file snippit showing scsi_lib error after iscsi-kill-session when tpgt changed
Comment 11 Dave Wysochanski 2005-09-12 20:18:23 UTC 
In rhel4 I'm just looking for the best way to recover from this scenario.  Right
now the best I can come up with is just "iscsi stop/start" or something like
that. Maybe that's the best answer right now since "iscsi reload" doesn't work
like it used to and maybe it's too hard to get the old behavior back.  It might
be nice to be able to kill the one session and keep going but I'm not sure it's
worth the effort - Mike you can probably make the call.  Maybe it would be nice
to make "iscsi-kill-session" safe and fix errors like the scsi_lib one but then
again maybe not.
Comment 12 Mike Christie 2005-10-24 16:29:52 UTC 
Changing Component to iscsi-initiator-utils for now. In the future please try to
select iscsi-initiator-utils for userspace problems and kernel for driver
problems. iSCSI is legacy Component from when it was all bundled in one rpm. Thanks.
Comment 13 Dave Wysochanski 2005-12-01 03:04:44 UTC 
Mike, just to let you know we're probably going to go with a workaround on our
end so that getting the driver to recover from a tpgt change is probably not
as important anymore.  Not sure if you were working on it or not but you
can probably close this or at least lower the priority until further notice
and I can re-open if need be (unless you're getting other vendors that want
this fixed).

Thanks.
Comment 14 Jiri Pallich 2012-06-20 16:11:05 UTC 
Thank you for submitting this issue for consideration in Red Hat Enterprise Linux. The release for which you requested us to review is now End of Life. 
Please See https://access.redhat.com/support/policy/updates/errata/

If you would like Red Hat to re-consider your feature request for an active release, please re-open the request via appropriate support channels and provide additional supporting details about the importance of this issue.