Bug 676804

Summary: iscsid: semop down failed; tgtd: semop up failed
Product: Red Hat Enterprise Linux 5 Reporter: Roland Friedwagner <roland.friedwagner>
Component: scsi-target-utilsAssignee: Andy Grover <agrover>
Status: CLOSED ERRATA QA Contact: Zhouping Liu <zliu>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 5.6CC: agrover, nobody+295318, qcai, roland.friedwagner
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 677475 (view as bug list) Environment:
Last Closed: 2011-07-21 11:36:24 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 677475    
Attachments:
Description Flags
SEMKEY clash with iscsid fix
none
drbd control socket move2 /var/run
none
SPEC File diff none

Description Roland Friedwagner 2011-02-11 10:48:24 UTC
Description of problem:

Running iscsid and tgtd on same machine 
the daemons logs semaphore errors

Version-Release number of selected component (if applicable):
scsi-target-utils-1.0.8-0.el5
iscsi-initiator-utils-6.2.0.872-6.el5

How reproducible: Everytime 

Steps to Reproduce:

1. Stop both services::

     $ service tgtd stop
     $ service iscsid stop

2. Start iscsid::

     $ service iscsid start

3. Start tgtd::

     $ service tgtd start

4. Stop iscsid::

     $ service stop iscsid

5. Stop tgtd::

     $ service tgtd stop

>> syslog error: **tgtd: semop up failed**

6. Start iscsid again::

     $ service iscsid start

7. Start tgtd again::

     $ service tgtd start

8. Stop tgtd::

     $ service tgtd stop

9. Stop iscsid::

     $ service iscsid stop

>> syslog error: **iscsid: semop down failed 22**

  
Actual results:

  Syslog Errors: tgtd: semop up failed
                 iscsid: semop down failed 22

Expected results: No Error Messages


Additional info:

Both services daemons does influence the semaphores of the other one?
Is it save to run both services on same machine at the same time?

Seen also this iscsid messages in syslog: **iscsid: semop down failed 43**
when doing iscisadm login/logout tests :-(But I can not reproduce)

Increased kernel.sem values to "1024 64000 256 1024" but
does not changed behavor.

Comment 1 Roland Friedwagner 2011-02-11 12:38:44 UTC
Ok further testing reveals how to reproduce 
the **iscsid: semop down failed 43** error:

1. service tgtd stop
2. service iscsid restart

3. /usr/sbin/tgtd &>/dev/null
4. tgtadm --op delete --mode system
3. /usr/sbin/tgtd &>/dev/null
5. login to any iscsi target (xxx):
   iscsiadm --mode node --targetname=iqn.xxx --login
>> Syslog Error: iscsid: semop down failed 43
6. logout to any iscsi target (xxx):
   iscsiadm --mode node --targetname=iqn.xxx --logout
>> Syslog Error: iscsid: semop down failed 43

Until you did not restart iscsid you get the **iscsid: semop down failed 43**
on each login/logout of an iscsi target.

==> tgtd has brocken icsid semaphores??

Add.Info: same behavor with scsi-target-utils-0.0-6.20091205snap.el5_5.3

Comment 2 Roland Friedwagner 2011-02-11 16:25:25 UTC
Created attachment 478273 [details]
SEMKEY clash with iscsid fix

Comment 3 Roland Friedwagner 2011-02-11 16:26:24 UTC
Created attachment 478274 [details]
drbd control socket move2 /var/run

Comment 4 Roland Friedwagner 2011-02-11 16:27:33 UTC
Created attachment 478275 [details]
SPEC File diff

Comment 5 Roland Friedwagner 2011-02-11 16:28:57 UTC
The reason is a id collision with same SEMKEY in log.c used in 
scsi-target-utils-1.0.8-0.el5 and iscsi-initiator-utils-6.2.0.872-6.el5

I uploaded patch scsi-target-utils-fix-semkey-clash.patch to fix this!
It may be a very good thing that this fix makes it to RHN very soon,
because until than logging of iscid and tgtd is undefined/broken.

If Mike digs into sources, I suggest also to fix the nasty location
for the tgtd control socket being placed in /tmp but should be IMHO /var/run.
(Uploaded scsi-target-utils-namespace-socket-path.patch for this).

Kind Regards,
Roland

Comment 6 Mike Christie 2011-02-14 22:28:25 UTC
Hey Roland,

Thanks for both patches and problem analysis. Patches look ok to me.

Comment 7 Andy Grover 2011-03-01 19:16:16 UTC
Hi Roland,

I'd like to get both of these upstream as well. Can you submit them to stgt list? Or, if you want to just add checkin comments and signoff here, I can do it.

Thanks -- Andy

Comment 8 Roland Friedwagner 2011-03-04 07:12:35 UTC
Hello Andy,

I posted both issues (sem key clash and ipc socket path) and
the patches to stgt project mailing list stgt.org 
yesterday. BTW: the issue was still in latest(1.0.14) release.

Kind Regards, Roland

Comment 9 Andy Grover 2011-03-08 01:13:56 UTC
Much thanks, Roland!

Comment 11 Zhouping Liu 2011-04-26 10:10:55 UTC
reproduced it on the version packages:
iscsi-initiator-utils-6.2.0.872-6.el5.x86_64.rpm
scsi-target-utils-1.0.8-0.el5.x86_64.rpm
verified it on:
iscsi-initiator-utils-6.2.0.872-9.el5
scsi-target-utils-1.0.14-1.el5

Comment 12 Roland Friedwagner 2011-05-27 08:36:52 UTC
An Errata was released for RHEL6 -
http://rhn.redhat.com/errata/RHBA-2011-0734.html

What is planned for RHEL5?

Comment 13 Andy Grover 2011-05-27 16:41:39 UTC
check out the rhel 5.7 beta, it's in there.

Comment 14 errata-xmlrpc 2011-07-21 11:36:24 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-1049.html