Bug 1731420

Summary:	Hitch service file not configured to detect error during initialization (Type=simple)
Product:	[Fedora] Fedora EPEL	Reporter:	Thue Janus Kristensen <thuejk>
Component:	hitch	Assignee:	Ingvar Hagelund <ingvar>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	epel7	CC:	ingvar, smooge
Target Milestone:	---	Keywords:	Reopened
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	hitch-1.5.0-4.fc30 hitch-1.5.0-4.el6 hitch-1.5.0-4.el7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-12-12 00:21:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Thue Janus Kristensen 2019-07-19 11:37:09 UTC

Description of problem:
The hitch systemd service file is not configured to allow systemd to detect errors during hitch initialization, because the service file says "Type=simple". This means that initialization errors are not reported to the user when doing "systemctl start hitch" on the console, and services depending on hitch are (I assume) started without first waiting to see if Hitch has correctly initialized.

How reproducible:
Write an invalid hitch configuration file (just put in some random characters at the start of a line). Then doing "systemctl start hitch" on the command line should detect that initialization of hitch failed, and print the errors to console. But it doesn't, and you can only find out the error by doing "systemctl status hitch".

Additional info:
The solution is to add "Type=forking" to hitch.service and "daemon=on" to hitch.conf . The process forking and then exiting successfully is the traditional way for sysV-style services to tell the init system that initialization had completed successfully, and systemd will wait for this fork+exit to occur.

An alternative solution would be the more elegant and explicit "Type=notify" and having the hitch process do "systemd-notify READY=1". But I assume that hitch is not written to call systemd-notify.

When using an init system to start a service on the console, it is basic functionality for the init system to automatically print any error status to the user.

This bug report is inspired by gquintard's comment at https://github.com/varnish/hitch/issues/188

Comment 1 Thue Janus Kristensen 2019-07-23 14:05:23 UTC

I just noticed. Downloading and "rpm2cpio ../hitch-1.5.0-1.el7.x86_64.rpm | cpio -idmv" the newest package ( https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/h/hitch-1.5.0-1.el7.x86_64.rpm ), in hitch.conf it has

> daemon = on

But in hitch.service it has

> Type=simple

This is definitely a bug, and caused hitch to not come back up on rebooting. While easy to fix if you are aware of it, this caused my website to be down for hours, so I changed the severity from low to high.

Comment 2 Ingvar Hagelund 2019-11-12 21:28:40 UTC

This is now fixed in rawhide (the package is building as I write this). I will also push to f31 and f30.

Now, for those that use hitch in production on EPEL, I'm a bit unsure how to proceeed. If I push a change to the systemd service file, it may break users' systems where the bug is hot-fixed by a workaround.

There were two ways to get around the bug:

- Set daemon=off in hitch.conf. That file is marked with noreplace, so the package should not overwrite the workaround. Probably safe.
- Set type=forking in hitch.service. If that is done the correct way, ie. by making the changes in a copy of hitch.service under /etc/systemd/system, the updated package would not overwrite the workaround.

If a user has edited /lib/systemd/system/hitch.service , the workaround will be overwritten by the update, and hitch will not start on next restart.

Is it safe to push this update to epel7 (and epel6)?

Ingvar

Comment 3 Ingvar Hagelund 2019-11-12 21:36:25 UTC

argh, I'm wrong. Let me try again:

There were two ways to get around the bug:

- Set daemon=off in hitch.conf. That file is marked with noreplace, so the update will not overwrite this fix. As this does not match the updated Type=forking in hitch.service, so hitch will not start after the update.

- Set Type=forking in hitch.service. This is the same fix as in the update, so this should be safe.

Also, the Fedora update adds a (missing) systemd limits.conf including LimitNOFILE that is important, as the default value (1024) would trig network problems on a medium busy site.

Is it safe to push this update to epel7 (and epel6)?

Ingvar

Comment 4 Fedora Update System 2019-11-12 22:38:14 UTC

FEDORA-2019-e2fa1ba359 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-e2fa1ba359

Comment 5 Fedora Update System 2019-11-13 10:47:19 UTC

hitch-1.5.0-4.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-e2fa1ba359

Comment 6 Fedora Update System 2019-11-13 10:53:14 UTC

hitch-1.5.0-4.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-acb707c900

Comment 7 Fedora Update System 2019-11-21 00:55:31 UTC

hitch-1.5.0-4.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 8 Ingvar Hagelund 2019-11-25 14:30:19 UTC

I'll keep this open, as this is "fixed" as stated above in fedora, but I have not pushed the similar change for epel yet.

Comment 9 Fedora Update System 2019-11-25 15:41:39 UTC

FEDORA-EPEL-2019-d72e8adb23 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-d72e8adb23

Comment 10 Fedora Update System 2019-11-25 15:41:40 UTC

FEDORA-EPEL-2019-c3f292fb76 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-c3f292fb76

Comment 11 Ingvar Hagelund 2019-11-25 15:43:00 UTC

Changes pushed to epel7 testing. Please test and leave feedback.

Comment 12 Fedora Update System 2019-11-27 00:40:15 UTC

hitch-1.5.0-4.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-d72e8adb23

Comment 13 Fedora Update System 2019-11-27 00:42:51 UTC

hitch-1.5.0-4.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-c3f292fb76

Comment 14 Fedora Update System 2019-12-12 00:21:24 UTC

hitch-1.5.0-4.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.

Comment 15 Fedora Update System 2019-12-12 00:39:13 UTC

hitch-1.5.0-4.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.