Bug 1211777

Summary:	Reboot get's stuck
Product:	[Fedora] Fedora	Reporter:	Thomas Meyer <thomas.mey>
Component:	jetty	Assignee:	Mikolaj Izdebski <mizdebsk>
Status:	CLOSED NOTABUG	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	21	CC:	agk, bmarzins, bmr, dwysocha, eclipse-sig, heinzm, java-sig-commits, jonathan, krzysztof.daniel, lvm-team, mat.booth, mcsontos, mizdebsk, msimacek, msnitzer, msrb, prajnoha, prockai, thomas.mey, zkabelac
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-05-07 21:03:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Thomas Meyer 2015-04-14 20:38:06 UTC

Description of problem:
The reboot get's stuck on my server, I'm not sure why:

Very likely because of this error message:
"Apr 12 12:42:49 server swapoff[23808]: swapoff: /dev/mapper/vg_static-lv_swap: swapoff fehlgeschlag"

swapoff for the vg_static_lv_swap fails because of too few memory.

A few seconds later jetty get's killed and the main memory is freed again:
"
Apr 12 12:42:56 server systemd[1]: jetty.service: main process exited, code=exited, status=143/n/a
Apr 12 12:42:56 server systemd[1]: Unit jetty.service entered failed state.
Apr 12 12:42:56 server systemd[1]: jetty.service failed."

Any ideas what's goind on here?
Who to make the reboot reliable?

Log extract:
Apr 12 12:42:47 server systemd[22770]: Received SIGRTMIN+24 from PID 23834 (kill).
Apr 12 12:42:48 server systemd[1]: firewalld.service: main process exited, code=exited, status=1/FA
Apr 12 12:42:48 server systemd[1]: Unit firewalld.service entered failed state.
Apr 12 12:42:48 server systemd[1]: firewalld.service failed.
Apr 12 12:42:49 server systemd[1]: Unit mcelog.service entered failed state.
Apr 12 12:42:49 server systemd[1]: mcelog.service failed.
Apr 12 12:42:48 server NetworkManager[644]: <warn>  disconnected by the system bus.
Apr 12 12:42:49 server swapoff[23808]: swapoff: /dev/mapper/vg_static-lv_swap: swapoff fehlgeschlag
Apr 12 12:42:51 server NetworkManager[644]: <info>  Could not connect to the system bus; only the p
Apr 12 12:42:54 server NetworkManager[644]: <info>  Could not connect to the system bus; only the p
Apr 12 12:42:56 server systemd[1]: jetty.service: main process exited, code=exited, status=143/n/a
Apr 12 12:42:56 server systemd[1]: Unit jetty.service entered failed state.
Apr 12 12:42:56 server systemd[1]: jetty.service failed.
Apr 12 12:42:56 server mysqld_safe[23612]: 150412 12:42:56 mysqld_safe mysqld from pid file /var/ru
Apr 12 12:42:56 server NetworkManager[644]: <info>  caught signal 15, shutting down normally.
Apr 12 12:42:56 server NetworkManager[644]: <info>  exiting (success)
Apr 12 12:42:57 server auditd[548]: The audit daemon is exiting.
Apr 12 12:42:57 server kernel: audit: type=1305 audit(1428835377.224:6865): audit_pid=0 old=548 aui
Apr 12 12:42:57 server kernel: audit: type=1130 audit(1428835377.234:6866): pid=1 uid=0 auid=429496
Apr 12 12:42:57 server kernel: audit: type=1131 audit(1428835377.234:6867): pid=1 uid=0 auid=429496
Apr 12 12:42:57 server kernel: audit: type=1130 audit(1428835377.235:6868): pid=1 uid=0 auid=429496
Apr 12 12:42:57 server kernel: audit: type=1131 audit(1428835377.235:6869): pid=1 uid=0 auid=429496
Apr 12 12:42:57 server kernel: audit: type=1130 audit(1428835377.236:6870): pid=1 uid=0 auid=429496
Apr 12 12:42:57 server kernel: audit: type=1131 audit(1428835377.236:6871): pid=1 uid=0 auid=429496
Apr 12 12:42:57 server kernel: audit: type=1130 audit(1428835377.239:6872): pid=1 uid=0 auid=429496
Apr 12 12:42:57 server kernel: audit: type=1131 audit(1428835377.239:6873): pid=1 uid=0 auid=429496


Version-Release number of selected component (if applicable):
lvm2 - 2.02.116-3.fc21
systemd - 216-24.fc21

the jetty service is a self made unit file that is socket activated.
I can provide those unit files if necessary.

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Marian Csontos 2015-05-07 10:13:47 UTC

Whatever jetty is, stop it earlier during shutdown. You can not turn swap off while there are processes using more memory than is available on the host.

Assigning to jetty which should modify it's systemd unit to stop before turning off the swap. Consult systemd documentation/experts how to achieve that.

Comment 2 Mat Booth 2015-05-07 10:42:21 UTC

(In reply to Marian Csontos from comment #1)
> Whatever jetty is, stop it earlier during shutdown. You can not turn swap
> off while there are processes using more memory than is available on the
> host.
> 
> Assigning to jetty which should modify it's systemd unit to stop before
> turning off the swap. Consult systemd documentation/experts how to achieve
> that.

I don't think the jetty maintainers can do much to fix the user's "self made unit file" ;-)

Thomas, does changing your unit file to shut down jetty earlier, following Marian's advice, fix the problem for you?

Comment 3 Thomas Meyer 2015-05-07 21:03:12 UTC

Hi Marian, Mat,

I actually did copy the packaged systemd.unit file from Fedora. I just checked again and I did kill the After= line in my custom unit file.
I think the missing After= is the explanation for the wrong ordering dependency in the shutdown.

I added the line again.

Sorry for the noise and thank for your help.