To give feedback for bug 598941 I tried stopping the iscsi service on the client. It fails, but I can't find anything interesting in the logs. Logging out of session [sid: 1, target: $CUT_TARGET_1, portal: 10.0.0.1,3260] Logging out of session [sid: 2, target: $CUT_TARGET_2, portal: 10.0.0.1,3260] iscsiadm: Could not logout of [sid: 1, target: $CUT_TARGET_1, portal: 10.0.0.1,3260]: iscsiadm: initiator reported error (9 - internal error) iscsiadm: Could not logout of [sid: 2, target: $CUT_TARGET2, portal: 10.0.0.1,3260]: iscsiadm: initiator reported error (2 - not found) Stopping iSCSI daemon: (and now it hangs) /var/log/messages: iscsid: session in invalid state for logout. Try again later iscsid: iscsid shutting down. iscsid: semop up failed 22 Target shows in /var/log/messages: tgtd: conn_close(100) connection closed, 0x12227058 1 tgtd: conn_close(106) sesson 0x12218360 1 tgtd: conn_close(100) connection closed, 0x12227058 1 tgtd: conn_close(106) sesson 0x12218360 1 If you tell me what information I should capture to make it possible to diagnose the issue, I'll add it here. By the way, I 'sesson' is likely a typo and should be 'session', right? :) tgtd: conn_close(106) sesson 0x1222bae0 1
The invalid state message is interesting. It means that the session is partially setup. The daemon does not know what state it is in exactly so it does not know how to proceed and takes the easy way out. Does this happen every time you try to stop the iscsi service? If so it is probably related to your other bug. In that the failures we see could be preventing the session from getting completely setup. The retries can take a long time. Or are you doing iscsi boot? Could you do the following. 1. Stop iscsi from starting automatically when the system starts. chkconfig --del iscsi chkconfig --del iscsid 2. Restart the box. 3. Start wireshark. 4. start iscsid service service iscsid start (note we are just starting iscsid here) 5. log into target manually iscsiadm -m node -T target -p ip -l Does the command in #5 complete. If so send the wireshark trace for when it does. If it does not complete then wait until you get the datalen error from your other bugzilla then send the wireshark trace.
(In reply to comment #1) > The invalid state message is interesting. It means that the session is > partially setup. The daemon does not know what state it is in exactly so it > does not know how to proceed and takes the easy way out. > > > Does this happen every time you try to stop the iscsi service? If so it is > probably related to your other bug. In that the failures we see could be Oops. Rereading your other bz it looks like we login ok and you get the datalen errors when you do IO tests right? If you can login ok then do the test/steps from comment 1 to get the wireshark trace for the other bug. For this shutdown bug, let me know if this always happens? And then try this: run iscsiadm -m session -P 3 before running the service iscsi stop. Then run iscsiadm -m node -u Does that last work? If not send the output of the iscsiam session command. Also if you could send the outpout of the iscsiadm -m session command when you do service iscsi stop that would be helpful.
Created attachment 422582 [details] iscsiadm -m node -u
Created attachment 422584 [details] iscsiadm -m session -P 3
Still can't shutdown. I will reboot the machine tonight to have a fresh start. As for the login, yes, it passes (no password, private network) and I can access the remote drives. The IO fails only under special circumstances.
The iscsiadm -m session -P 3 outpout shows that iqn.2010-06.$FQDN_REMOVED:storage.bigbang.storage1.vmprivate is in a unknown state: iSCSI Connection State: Unknown iSCSI Session State: FREE Internal iscsid Session State: Unknown We would only see this if iscsid or the kernel was really old and did not support being able to provide that info. The kernel and tools in RHEL5 should have that. The only other reason we would see that is when the login command does not complete ok but you mentioned it the login command ran ok. Also seeing iscsiadm: Could not logout of [sid: 2, target: iqn.2010-06.$FQDN_REMOVED:storage.bigbang.storage1.vmprivate, portal: 10.0.0.1,3260]: iscsiadm: initiator reported error (2 - not found) when iscsiadm -m session -P 3 clearly shows the target is not right. If you have not done a fresh start on the box, check that you do not have multiple copies of the iscsi tools. Do a: whereis iscsiadm whereis iscsid
Oh yeah, are you by any chance using a 32 bit userspace iscsi-initiator-tools rpm with a 64 bit kernel?
Hi Mike, both setups (target, initiator) are pure 64bit only. You won't find a 32bit package on the system. iscsiadm: /sbin/iscsiadm /usr/share/man/man8/iscsiadm.8.gz iscsid: /sbin/iscsid /usr/share/man/man8/iscsid.8.gz # rpm -q iscsi-initiator-utils iscsi-initiator-utils-6.2.0.871-0.16.el5 Target: scsi-target-utils-0.0-6.20091205snap.el5_5.2 Kernel (on both) kernel-2.6.18-194.3.1.el5 After I've rebooted the initiator the issue is not appearing (yet).
This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux.
Closing, not reproducible according to Comment 8