Problem Description -------------------------------------------------------------------------- iscsi target daemon (tgtd) fails to start on F18. Following is the relevant entries in /var/log/messages Jul 16 12:02:23 ltcfbl9cb systemd[1]: Starting tgtd iSCSI target daemon... Jul 16 12:02:23 ltcfbl9cb tgtd[10366]: librdmacm: Warning: couldn't read ABI version. Jul 16 12:02:23 ltcfbl9cb kernel: [ 7849.180057] tgtd (10366): /proc/10366/oom_adj is deprecated, please use /proc/10366/oom_score_adj instead. Jul 16 12:02:23 ltcfbl9cb tgtd[10366]: librdmacm: Warning: assuming: 4 Jul 16 12:02:23 ltcfbl9cb tgtd[10366]: librdmacm: Fatal: unable to get RDMA device list Jul 16 12:02:23 ltcfbl9cb tgtd[10366]: tgtd: iser_ib_init(3376) Failed to initialize RDMA; load kernel modules? Jul 16 12:02:23 ltcfbl9cb tgtd[10366]: tgtd: work_timer_start(146) use timer_fd based scheduler Jul 16 12:02:23 ltcfbl9cb tgtd[10366]: tgtd: bs_init(313) use signalfd notification Jul 16 12:02:29 ltcfbl9cb tgtadm[10369]: tgtadm: tgtd closed the socket Jul 16 12:02:29 ltcfbl9cb systemd[1]: tgtd.service: main process exited, code=killed, status=11/SEGV Jul 16 12:02:29 ltcfbl9cb tgt-admin[10372]: tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected Jul 16 12:02:29 ltcfbl9cb tgt-admin[10372]: tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected Jul 16 12:02:29 ltcfbl9cb tgt-admin[10372]: tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected Jul 16 12:02:29 ltcfbl9cb tgt-admin[10372]: tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected Jul 16 12:02:30 ltcfbl9cb tgtadm[10379]: tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected Jul 16 12:02:30 ltcfbl9cb systemd[1]: tgtd.service: control process exited, code=exited status=107 Jul 16 12:02:30 ltcfbl9cb tgtadm[10384]: tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected Jul 16 12:02:30 ltcfbl9cb systemd[1]: tgtd.service: control process exited, code=exited status=107 Jul 16 12:02:30 ltcfbl9cb systemd[1]: Failed to start tgtd iSCSI target daemon. Jul 16 12:02:30 ltcfbl9cb systemd[1]: Unit tgtd.service entered failed state Configuration file - /etc/tgt/tgtd.conf # The default config file include /etc/tgt/targets.conf # Config files from other packages etc. #include /etc/tgt/conf.d/*.conf include /etc/tgt/conf.d/cinder.conf RPM package version scsi-target-utils-1.0.32-5.fc18.ppc64 == Comment: - Reshma Sheik <reshmasheik.com> - == The same issue is seen on F19. [root@ ~]# cat /etc/issue Fedora release 19 (Schr?dinger?s Cat) Kernel \r on an \m (\l) [root@ ~]# service tgtd status Redirecting to /bin/systemctl status tgtd.service tgtd.service - tgtd iSCSI target daemon Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled) Active: failed (Result: exit-code) since Sat 1978-07-29 10:36:21 EDT; 9min ago Process: 7811 ExecStop=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=107) Process: 7808 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=107) Process: 7800 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS) Process: 7791 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS) Process: 7787 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS) Process: 7786 ExecStart=/usr/sbin/tgtd -f $TGTD_OPTS (code=dumped, signal=SEGV) == Comment: - Vaishnavi Bhat <vaish123.com> - == FYI, https://bugzilla.redhat.com/show_bug.cgi?id=972272 describes the same issue and is expected to fix in scsi-target-utils-1.0.32-5.fc19 F19 has scsi-target-utils-1.0.32-5.fc19 and still found buggy. Mirroring the bug for RH awareness.
------- Comment From rfolco.com 2013-08-20 18:54 EDT------- Adding more info to this bug report. The segfault is generated by strncmp call at mgmt.c, as shown below: Core was generated by `/usr/sbin/tgtd -f '. Program terminated with signal 11, Segmentation fault. #0 .strncmp () at ../sysdeps/powerpc/powerpc64/power7/strncmp.S:141 141 lbz rWORD1,0(rSTR1) (gdb) up #1 0x000000001002139c in sys_mgmt (mtask=0x1002f6c03a0, lld_no=0) at mgmt.c:325 325 if (!strncmp(mtask->req_buf, "debug=", 6)) { (gdb) up #2 mtask_execute (mtask=0x1002f6c03a0) at mgmt.c:504 504 adm_err = sys_mgmt(lld_no, mtask); (gdb) up #3 mtask_received (mtask=0x1002f6c03a0, fd=<optimized out>) at mgmt.c:605 605 adm_err = mtask_execute(mtask); (gdb) up #4 0x0000000010021894 in mtask_recv_send_handler (fd=<optimized out>, events=<optimized out>, data=0x1002f6c03a0) at mgmt.c:667 667 err = mtask_received(mtask, fd); (gdb) up #5 0x000000001001fcb0 in event_loop () at tgtd.c:417 417 tev->handler(tev->fd, events[i].events, tev->data); (gdb) up #6 0x00000000100046f8 in main (argc=<optimized out>, argv=<optimized out>) at tgtd.c:600 600 event_loop(); (gdb) up Initial frame selected; you cannot go up. (gdb) frame 0 #0 .strncmp () at ../sysdeps/powerpc/powerpc64/power7/strncmp.S:141 141 lbz rWORD1,0(rSTR1) (gdb) list 136 cmpldi cr1,rN,0 137 L(unaligned): 138 mtctr rN 139 ble cr1,L(ux) 140 L(uz): 141 lbz rWORD1,0(rSTR1) 142 lbz rWORD2,0(rSTR2) 143 .align 4 144 L(u1): 145 cmpdi cr1,rWORD1,0
------- Comment From rfolco.com 2013-08-20 19:54 EDT------- Looks like the pointer mtask->req_buf is NULL: https://github.com/fujita/tgt/blob/master/usr/mgmt.c Line 325: if (!strncmp(mtask->req_buf, "debug=", 6)) { (gdb) up #1 0x000000001002139c in sys_mgmt (mtask=0x1002f6c03a0, lld_no=0) at mgmt.c:325 325 if (!strncmp(mtask->req_buf, "debug=", 6)) { (gdb) print mtask->req_buf $1 = 0x0 (gdb) print mtask $2 = (struct mgmt_task *) 0x1002f6c03a0
------- Comment From rfolco.com 2013-08-21 18:02 EDT------- FYI this bug seems to have disappeared in the latest scsi-target-utils package version 1.0.38-1. http://dl.fedoraproject.org/pub/fedora-secondary/updates/19/ppc64/scsi-target-utils-1.0.38-1.fc19.ppc64.rpm # systemctl status tgtd.service Loaded: loaded (/usr/lib/systemd/system/tgtd.service; enabled) Active: active (running) since Wed 2013-08-21 13:04:30 EDT; 56min ago Main PID: 52595 (tgtd) CGroup: name=systemd:/system/tgtd.service ??52595 /usr/sbin/tgtd -f
Okay, closing. Please feel free to reopen if it pops up again.
------- Comment From reshmasheik.com 2013-09-06 11:29 EDT------- Based on comment #11 the issue is fixed in scsi-target-utils-1.0.38-1.fc19.ppc64.rpm. Hence closing the bug on IBM side Thanks Reshma