Bug 601420 - iscsi fails to stop, which prevents reconnecting
iscsi fails to stop, which prevents reconnecting
Status: CLOSED WORKSFORME
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: iscsi-initiator-utils (Show other bugs)
5.5
All Linux
medium Severity medium
: rc
: ---
Assigned To: Mike Christie
Storage QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-06-07 18:07 EDT by David Kovalsky
Modified: 2014-03-31 19:45 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-11-23 10:41:00 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
iscsiadm -m node -u (627 bytes, text/plain)
2010-06-09 10:55 EDT, David Kovalsky
no flags Details
iscsiadm -m session -P 3 (1.99 KB, text/plain)
2010-06-09 10:55 EDT, David Kovalsky
no flags Details

  None (edit)
Description David Kovalsky 2010-06-07 18:07:05 EDT
To give feedback for bug 598941 I tried stopping the iscsi service on the client. It fails, but I can't find anything interesting in the logs.

Logging out of session [sid: 1, target: $CUT_TARGET_1, portal: 10.0.0.1,3260]
Logging out of session [sid: 2, target: $CUT_TARGET_2, portal: 10.0.0.1,3260]
iscsiadm: Could not logout of [sid: 1, target: $CUT_TARGET_1, portal: 10.0.0.1,3260]: 
iscsiadm: initiator reported error (9 - internal error)
iscsiadm: Could not logout of [sid: 2, target: $CUT_TARGET2, portal: 10.0.0.1,3260]:
iscsiadm: initiator reported error (2 - not found)
Stopping iSCSI daemon: 
(and now it hangs)


/var/log/messages:
iscsid: session in invalid state for logout. Try again later
iscsid: iscsid shutting down.
iscsid: semop up failed 22


Target shows in /var/log/messages:
tgtd: conn_close(100) connection closed, 0x12227058 1
tgtd: conn_close(106) sesson 0x12218360 1
tgtd: conn_close(100) connection closed, 0x12227058 1
tgtd: conn_close(106) sesson 0x12218360 1



If you tell me what information I should capture to make it possible to diagnose the issue, I'll add it here. 

By the way, I 'sesson' is likely a typo and should be 'session', right? :)
tgtd: conn_close(106) sesson 0x1222bae0 1
Comment 1 Mike Christie 2010-06-07 19:41:00 EDT
The invalid state message is interesting. It means that the session is partially setup. The daemon does not know what state it is in exactly so it does not know how to proceed and takes the easy way out.


Does this happen every time you try to stop the iscsi service? If so it is probably related to your other bug. In that the failures we see could be preventing the session from getting completely setup. The retries can take a long time.

Or are you doing iscsi boot?



Could you do the following.

1. Stop iscsi from starting automatically when the system starts.

chkconfig --del iscsi
chkconfig --del iscsid

2. Restart the box.

3. Start wireshark.

4. start iscsid service

service iscsid start

(note we are just starting iscsid here)

5. log into target manually

iscsiadm -m node -T target -p ip -l


Does the command in #5 complete. If so send the wireshark trace for when it does. If it does not complete then wait until you get the datalen error from your other bugzilla then send the wireshark trace.
Comment 2 Mike Christie 2010-06-07 19:48:16 EDT
(In reply to comment #1)
> The invalid state message is interesting. It means that the session is
> partially setup. The daemon does not know what state it is in exactly so it
> does not know how to proceed and takes the easy way out.
> 
> 
> Does this happen every time you try to stop the iscsi service? If so it is
> probably related to your other bug. In that the failures we see could be


Oops. Rereading your other bz it looks like we login ok and you get the datalen errors when you do IO tests right? If you can login ok then do the test/steps from comment 1 to get the wireshark trace for the other bug.



For this shutdown bug, let me know if this always happens? And then try this:

run iscsiadm -m session -P 3

before running the service iscsi stop. Then run

iscsiadm -m node -u

Does that last work? If not send the output of the iscsiam session command. Also if you could send the outpout of the iscsiadm -m session command when you do service iscsi stop that would be helpful.
Comment 3 David Kovalsky 2010-06-09 10:55:04 EDT
Created attachment 422582 [details]
iscsiadm -m node -u
Comment 4 David Kovalsky 2010-06-09 10:55:38 EDT
Created attachment 422584 [details]
iscsiadm -m session -P 3
Comment 5 David Kovalsky 2010-06-09 10:57:02 EDT
Still can't shutdown. I will reboot the machine tonight to have a fresh start. 

As for the login, yes, it passes (no password, private network) and I can access the remote drives. The IO fails only under special circumstances.
Comment 6 Mike Christie 2010-06-10 16:58:35 EDT
The iscsiadm -m session -P 3 outpout shows that iqn.2010-06.$FQDN_REMOVED:storage.bigbang.storage1.vmprivate is in a unknown state:
		iSCSI Connection State: Unknown
		iSCSI Session State: FREE
		Internal iscsid Session State: Unknown

We would only see this if iscsid or the kernel was really old and did not support being able to provide that info. The kernel and tools in RHEL5 should have that. The only other reason we would see that is when the login command does not complete ok but you mentioned it the login command ran ok. Also seeing


iscsiadm: Could not logout of [sid: 2, target: iqn.2010-06.$FQDN_REMOVED:storage.bigbang.storage1.vmprivate, portal: 10.0.0.1,3260]: 
iscsiadm: initiator reported error (2 - not found)

when iscsiadm -m session -P 3 clearly shows the target is not right.


If you have not done a fresh start on the box, check that you do not have multiple copies of the iscsi tools. Do a:

whereis iscsiadm
whereis iscsid
Comment 7 Mike Christie 2010-06-10 16:59:09 EDT
Oh yeah, are you by any chance using a 32 bit userspace iscsi-initiator-tools rpm with a 64 bit kernel?
Comment 8 David Kovalsky 2010-06-11 11:56:54 EDT
Hi Mike, 

both setups (target, initiator) are pure 64bit only. You won't find a 32bit package on the system.

iscsiadm: /sbin/iscsiadm /usr/share/man/man8/iscsiadm.8.gz
iscsid: /sbin/iscsid /usr/share/man/man8/iscsid.8.gz

# rpm -q iscsi-initiator-utils 
iscsi-initiator-utils-6.2.0.871-0.16.el5

Target:
scsi-target-utils-0.0-6.20091205snap.el5_5.2

Kernel (on both) kernel-2.6.18-194.3.1.el5

After I've rebooted the initiator the issue is not appearing (yet).
Comment 10 RHEL Product and Program Management 2010-08-09 15:35:42 EDT
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 11 Ondrej Hudlicky 2010-11-23 10:41:00 EST
Closing, not reproducible according to Comment 8

Note You need to log in before you can comment on or make changes to this bug.