Bug 797913

Summary: tgtd hangs at startup - systemd related
Product: [Fedora] Fedora Reporter: Pádraig Brady <pbrady>
Component: scsi-target-utilsAssignee: Andy Grover <agrover>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 16CC: agrover, apevec, dallan, derekh, johannbg, mcermak, mchristi, metherid, mschmidt, notting, plautrba, systemd-maint, terje.rosten
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: scsi-target-utils-1.0.18-5.fc16 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-03-06 19:43:48 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pádraig Brady 2012-02-27 14:21:53 UTC
This hangs: /etc/init.d/tgtd start
This works: SYSTEMCTL_SKIP_REDIRECT=1 /etc/init.d/tgtd start

Multiple people in the openstack dev team have seen this issue

I'm fairly sure tgtd used to work on F16, so it's probably a recent update.

Comment 1 Derek Higgins 2012-02-27 14:34:09 UTC
On F16 /etc/init.d/tgtd start works on a fresh install 

the problem only appears after 
> yum update

By updating packages individually I narrowed it down to this update
> yum update systemd
Updating:
 systemd         x86_64   37-13.fc16   updates           728 k
Updating for dependencies:
 systemd-sysv    x86_64   37-13.fc16   updates            15 k
 systemd-units   x86_64   37-13.fc16   updates           150 k

Trying to narrow it down further the problem appears to have been introduced in 
37-6.fc16.x86_64
http://pkgs.fedoraproject.org/gitweb/?p=systemd.git;a=commit;h=e43452dfe678264ac81c087d2a8735c973f99686

thats all I have

Comment 2 Pádraig Brady 2012-02-27 14:49:45 UTC
Same issue with systemd-37-14.fc16

Comment 3 Michal Schmidt 2012-02-27 15:03:11 UTC
What is the status of the service when it hangs?:
systemctl status tgtd.service

Do you get anything interesting in /var/log/messages after the attempt to start the service?

Comment 4 Jóhann B. Guðmundsson 2012-02-27 15:20:12 UTC
Wondering if those remount order patches changes might be causing this anyway if you can it would be good if you could test with the latest release in koji [1] which contains native systemd unit files ( it's enough to just copy the units from git into the /etc/systemd/system directory and run systemctl daemon-reload if upstream did not do any systemd specific patching ). 

We probably need you to follow 2. and provide the requested info from there.


1.http://koji.fedoraproject.org/koji/buildinfo?buildID=296816
2.http://fedoraproject.org/wiki/How_to_debug_Systemd_problems

Comment 5 Derek Higgins 2012-02-27 15:54:30 UTC
Michal More info below

====== While on systemd-37-5

[root@test systemd]# rpm -qa | grep -i -e systemd -e scsi-target-utils
systemd-37-5.fc16.x86_64
systemd-sysv-37-5.fc16.x86_64
scsi-target-utils-1.0.18-4.fc16.x86_64
systemd-units-37-5.fc16.x86_64

[root@test systemd]# /etc/init.d/tgtd start
Starting tgtd (via systemctl):                             [  OK  ]

[root@test systemd]# /etc/init.d/tgtd status
tgtd.service - LSB: Starts and stops the generic storage target daemon
	  Loaded: loaded (/etc/rc.d/init.d/tgtd)
	  Active: active (running) since Mon, 27 Feb 2012 15:20:41 +0000; 3s ago
	 Process: 18361 ExecStart=/etc/rc.d/init.d/tgtd start (code=exited, status=0/SUCCESS)
	  CGroup: name=systemd:/system/tgtd.service
		  ├ 18369 tgtd
		  └ 18370 tgtd

====== While on systemd-37-6

[root@test systemd]# rpm -qa | grep -i -e systemd -e scsi-target-utils
systemd-sysv-37-6.fc16.x86_64
scsi-target-utils-1.0.18-4.fc16.x86_64
systemd-37-6.fc16.x86_64
systemd-units-37-6.fc16.x86_64

[root@test systemd]# date
Mon Feb 27 15:23:12 UTC 2012

[root@test systemd]# /etc/init.d/tgtd start ; date
Starting tgtd (via systemctl):  Job failed. See system logs and 'systemctl status' for details.
                                                           [FAILED]
Mon Feb 27 15:28:21 UTC 2012

[root@test systemd]# /etc/init.d/tgtd status
tgtd.service - LSB: Starts and stops the generic storage target daemon
	  Loaded: loaded (/etc/rc.d/init.d/tgtd)
	  Active: failed since Mon, 27 Feb 2012 15:28:20 +0000; 7min ago
	 Process: 18518 ExecStop=/etc/rc.d/init.d/tgtd stop (code=exited, status=0/SUCCESS)
	 Process: 18536 ExecStart=/etc/rc.d/init.d/tgtd start (code=exited, status=0/SUCCESS)
	  CGroup: name=systemd:/system/tgtd.service
		  ├ 18542 tgtd
		  └ 18543 tgtd


[root@test systemd]# systemctl start tgtd.service
< doesn't return for ~5 minutes >
[root@test systemd]# systemctl status tgtd.service                                                                                                            
tgtd.service - LSB: Starts and stops the generic storage target daemon
	  Loaded: loaded (/etc/rc.d/init.d/tgtd)
	  Active: activating (start) since Mon, 27 Feb 2012 15:36:08 +0000; 7min ago
	  CGroup: name=systemd:/system/tgtd.service
		  ├ 18542 tgtd
		  └ 18543 tgtd

[root@test systemd]# tail -n 26 /var/log/messages 
Feb 27 15:22:28 test dbus-daemon[470]: dbus[470]: [system] Reloaded configuration
Feb 27 15:22:28 test systemd[1]: Reexecuting.
Feb 27 15:22:29 test systemd[1]: systemd 37 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +SYSVINIT +LIBCRYPTSETUP; fedora)
Feb 27 15:22:29 test yum[18449]: Updated: systemd-37-6.fc16.x86_64
Feb 27 15:22:29 test yum[18449]: Updated: systemd-sysv-37-6.fc16.x86_64
Feb 27 15:22:29 test systemd[1]: Reloading.
Feb 27 15:22:29 test systemd-logind[18468]: New seat seat0.
Feb 27 15:22:29 test systemd-logind[18468]: New user root logged in.
Feb 27 15:22:29 test systemd-logind[18468]: New session 1 of user root.
Feb 27 15:23:01 test tgtd: tgtd logger stopped, pid:18370
Feb 27 15:23:01 test tgtd[18518]: Stopping SCSI target daemon: [  OK  ]
Feb 27 15:23:21 test tgtd: libibverbs.so: cannot open shared object file: No such file or directory - iser transport not used
Feb 27 15:23:21 test tgtd: semkey 0x6110a50a
Feb 27 15:23:21 test tgtd: tgtd daemon started, pid:18542
Feb 27 15:23:21 test tgtd: tgtd logger started, pid:18543 debug:0
Feb 27 15:23:21 test tgtd[18536]: Starting SCSI target daemon: [  OK  ]
Feb 27 15:23:21 test tgtd: work_timer_start(146) use timer_fd based scheduler
Feb 27 15:23:21 test tgtd: bs_init(312) use signalfd notification
Feb 27 15:23:21 test systemd[1]: PID file /var/run/tgtd.pid not readable (yet?) after start.
Feb 27 15:28:20 test systemd[1]: tgtd.service operation timed out. Terminating.
Feb 27 15:28:20 test systemd[1]: Unit tgtd.service entered failed state.
Feb 27 15:36:08 test systemd[1]: PID file /var/run/tgtd.pid not readable (yet?) after start.
Feb 27 15:40:38 test systemd[1]: Reexecuting.
Feb 27 15:40:38 test systemd[1]: systemd 37 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +SYSVINIT +LIBCRYPTSETUP; fedora)
Feb 27 15:45:38 test systemd[1]: tgtd.service operation timed out. Terminating.
Feb 27 15:45:38 test systemd[1]: Unit tgtd.service entered failed state.
[root@test systemd]#

Comment 6 Michal Schmidt 2012-02-27 16:04:50 UTC
(In reply to comment #5)
> Feb 27 15:23:21 test systemd[1]: PID file /var/run/tgtd.pid not readable (yet?)
> after start.
> Feb 27 15:28:20 test systemd[1]: tgtd.service operation timed out. Terminating.

tgtd failed to write its declared PID file. Its initscript declares:
# pidfile: /var/run/tgtd.pid

It is required that the pidfile information given in initscripts (or PIDFile=... in native systemd units) is correct.

For more information see a similar bug in another package: bug 786625

Comment 7 Derek Higgins 2012-02-28 12:19:24 UTC
Thanks Michal,

When I take the line 
# pidfile: /var/run/tgtd.pid
out of the init script, tgtd comes up and appears to do so correctly (at least for my case)

as far as I can see there is no option in tgtd to write out a pid file

Would removing the line above be an acceptable solution for the init script? Or is there a better option (in the absence of a pid file)?

Comment 8 Michal Schmidt 2012-02-28 14:38:51 UTC
If the daemon really cannot create a pid file, then removing the "# pidfile:" header is correct, yes.

Comment 9 Dave Allan 2012-03-01 20:03:56 UTC
(In reply to comment #7)
> When I take the line 
> # pidfile: /var/run/tgtd.pid
> out of the init script, tgtd comes up and appears to do so correctly (at least
> for my case)

FWIW taking out the pidfile line fixes the hang on my system as well.  I have no idea what the implications are, however.

Comment 10 Fedora Update System 2012-03-02 02:29:00 UTC
scsi-target-utils-1.0.24-2.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/scsi-target-utils-1.0.24-2.fc17

Comment 11 Fedora Update System 2012-03-02 02:30:39 UTC
scsi-target-utils-1.0.18-5.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/scsi-target-utils-1.0.18-5.fc16

Comment 12 Andy Grover 2012-03-06 16:50:14 UTC
*** Bug 800386 has been marked as a duplicate of this bug. ***

Comment 13 Fedora Update System 2012-03-06 19:43:48 UTC
scsi-target-utils-1.0.18-5.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.