Bug 1239035

Summary: Update using UI fails to auto-start the server back up
Product: Red Hat CloudForms Management Engine Reporter: Jan Krocil <jkrocil>
Component: ApplianceAssignee: Nick Carboni <ncarboni>
Status: CLOSED ERRATA QA Contact: Jan Krocil <jkrocil>
Severity: medium Docs Contact:
Priority: high    
Version: 5.3.0CC: abellott, cpelland, gtanzill, jhardy, kzvyahin, mfeifer, obarenbo, simaishi
Target Milestone: GAKeywords: ZStream
Target Release: 5.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 5.5.0.13 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1241541 1241542 (view as bug list) Environment:
Last Closed: 2015-12-08 13:21:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1241541, 1241542    

Description Jan Krocil 2015-07-03 10:08:23 UTC
Description of problem:
CFME doesn't auto-start after update executed through UI like it does when updated directly from ssh, using yum.

Version-Release number of selected component (if applicable):
5.3.3.2 -> 5.3.4.2 and 5.3.4.2->5.3.5.1

How reproducible:
Always

Steps to Reproduce:
1. Register to RHSM
2. Add the necessary cfme repos
3. Click "Apply CFME updates" in the UI

Actual results:
Server is shut down, it updates the packages successfully but fails to run the "update_start" procedure on its own.
What seems to be happening is that the pretrans script is run, then the cfme appliance is stopped and when that happens, it takes the yum update process down with it.
This means that the posttrans script, running "update_start" is not run.
But at the same time, the yum update itself returns "Success" return code.

Expected results:
CFME should auto-start back up after update.

Additional info:
Simple workaround is to either update directly using ssh+yum (it does auto-start in that case with no issues) or update using the UI and then start the appliance using "service evmserverd update_start" command manually.

Thanks, Brandon (bdunne).

Comment 3 CFME Bot 2015-07-08 21:16:42 UTC
New commit detected on cfme/5.4.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=9cd777d0a680312a7e97f0935942e77bdbe4d002

commit 9cd777d0a680312a7e97f0935942e77bdbe4d002
Author:     Nick Carboni <ncarboni>
AuthorDate: Wed Jul 8 16:47:11 2015 -0400
Commit:     Nick Carboni <ncarboni>
CommitDate: Wed Jul 8 16:47:11 2015 -0400

    Added logging redirect to evmserver script.
    
    During the update process evmserver.sh update_start is called from
    within a yum posttrans and yum's parent process has already been killed as
    a part of server shutdown.
    This caused the rake task started by evmserver to be killed when
    attempting to write to stdout or stderr.
    For this reason the redirect was added for rake tasks in evmserver.sh
    that are called during an update.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1239035

 system/LINK/etc/init.d/evmserverd | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comment 4 CFME Bot 2015-07-08 21:16:45 UTC
New commit detected on cfme/5.4.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=0d44f0be624ac0e228c9adfcf6a34f37663e91f8

commit 0d44f0be624ac0e228c9adfcf6a34f37663e91f8
Merge: a23b94c 9cd777d
Author:     Joe Rafaniello <jrafanie>
AuthorDate: Wed Jul 8 17:15:54 2015 -0400
Commit:     Joe Rafaniello <jrafanie>
CommitDate: Wed Jul 8 17:15:54 2015 -0400

    Merge branch 'start_server_after_update' into '5.4.z'
    
    Added logging redirect to evmserver script.
    
    During the update process evmserver.sh update_start is called from
    within a yum posttrans and yum's parent process has already been killed as
    a part of server shutdown.
    This caused the rake task started by evmserver to be killed when
    attempting to write to stdout or stderr.
    For this reason the redirect was added for rake tasks in evmserver.sh
    that are called during an update.
    
    Clean copy from upstream https://github.com/ManageIQ/manageiq-appliance/pull/4
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1239035
    
    See merge request !165

 system/LINK/etc/init.d/evmserverd | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comment 5 CFME Bot 2015-07-08 21:34:06 UTC
New commit detected on cfme/5.3.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=f68c08315bc0faeb99e61535dc6e4f48f9e481b6

commit f68c08315bc0faeb99e61535dc6e4f48f9e481b6
Author:     Nick Carboni <ncarboni>
AuthorDate: Wed Jul 8 16:47:11 2015 -0400
Commit:     Nick Carboni <ncarboni>
CommitDate: Wed Jul 8 16:59:38 2015 -0400

    Added logging redirect to evmserver script.
    
    During the update process evmserver.sh update_start is called from
    within a yum posttrans and yum's parent process has already been killed as
    a part of server shutdown.
    This caused the rake task started by evmserver to be killed when
    attempting to write to stdout or stderr.
    For this reason the redirect was added for rake tasks in evmserver.sh
    that are called during an update.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1239035

 system/LINK/etc/init.d/evmserverd | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comment 6 CFME Bot 2015-07-08 21:34:10 UTC
New commit detected on cfme/5.3.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=a80ec6ae261f23ff4f90d71b9766cd2d3a0c67c5

commit a80ec6ae261f23ff4f90d71b9766cd2d3a0c67c5
Merge: b9a7f54 f68c083
Author:     Joe Rafaniello <jrafanie>
AuthorDate: Wed Jul 8 17:17:08 2015 -0400
Commit:     Joe Rafaniello <jrafanie>
CommitDate: Wed Jul 8 17:17:08 2015 -0400

    Merge branch '5.3.z.start_server_after_update' into '5.3.z'
    
    Added logging redirect to evmserver script.
    
    During the update process evmserver.sh update_start is called from
    within a yum posttrans and yum's parent process has already been killed as
    a part of server shutdown.
    This caused the rake task started by evmserver to be killed when
    attempting to write to stdout or stderr.
    For this reason the redirect was added for rake tasks in evmserver.sh
    that are called during an update.
    
    Clean cherry pick from http://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/merge_requests/165
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1239035
    
    See merge request !166

 system/LINK/etc/init.d/evmserverd | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comment 7 CFME Bot 2015-07-08 22:01:34 UTC
New commit detected on manageiq-appliance/master:
https://github.com/ManageIQ/manageiq-appliance/commit/913da2f98b77c06d4258f6c1f55f006835200efa

commit 913da2f98b77c06d4258f6c1f55f006835200efa
Author:     Nick Carboni <ncarboni>
AuthorDate: Wed Jul 8 15:49:53 2015 -0400
Commit:     Nick Carboni <ncarboni>
CommitDate: Wed Jul 8 15:49:53 2015 -0400

    Added logging redirect to evmserver script.
    
    During the update process evmserver.sh update_start is called from
    within a yum posttrans and yum's parent process has already been killed as
    a part of server shutdown.
    This caused the rake task started by evmserver to be killed when
    attempting to write to stdout or stderr.
    For this reason the redirect was added for rake tasks in evmserver.sh
    that are called during an update.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1239035

 LINK/bin/evmserver.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comment 8 CFME Bot 2015-07-09 15:45:52 UTC
New commit detected on cfme/5.3.5:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=783af961ebd6d72b4f7193349dd17b9b4b6e2c23

commit 783af961ebd6d72b4f7193349dd17b9b4b6e2c23
Author:     Nick Carboni <ncarboni>
AuthorDate: Wed Jul 8 16:47:11 2015 -0400
Commit:     Nick Carboni <ncarboni>
CommitDate: Wed Jul 8 17:10:30 2015 -0400

    Added logging redirect to evmserver script.
    
    During the update process evmserver.sh update_start is called from
    within a yum posttrans and yum's parent process has already been killed as
    a part of server shutdown.
    This caused the rake task started by evmserver to be killed when
    attempting to write to stdout or stderr.
    For this reason the redirect was added for rake tasks in evmserver.sh
    that are called during an update.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1239035

 system/LINK/etc/init.d/evmserverd | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comment 9 CFME Bot 2015-07-09 15:45:56 UTC
New commit detected on cfme/5.3.5:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=9104766c214dd3e9084d269109c515d69b8fa3ef

commit 9104766c214dd3e9084d269109c515d69b8fa3ef
Merge: b69b688 783af96
Author:     Joe Rafaniello <jrafanie>
AuthorDate: Thu Jul 9 11:41:58 2015 -0400
Commit:     Joe Rafaniello <jrafanie>
CommitDate: Thu Jul 9 11:41:58 2015 -0400

    Merge branch '5.3.5.start_server_after_update' into '5.3.5'
    
    Added logging redirect to evmserver script.
    
    During the update process evmserver.sh update_start is called from
    within a yum posttrans and yum's parent process has already been killed as
    a part of server shutdown.
    This caused the rake task started by evmserver to be killed when
    attempting to write to stdout or stderr.
    For this reason the redirect was added for rake tasks in evmserver.sh
    that are called during an update.
    
    Clean cherry pick from http://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/merge_requests/166
    
    Waiting on blocker flag.
    https://bugzilla.redhat.com/show_bug.cgi?id=1239035
    
    See merge request !167

 system/LINK/etc/init.d/evmserverd | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comment 16 CFME Bot 2015-12-02 20:21:56 UTC
New commit detected on ManageIQ/manageiq-appliance/master:
https://github.com/ManageIQ/manageiq-appliance/commit/a4f8415ad143fec4be3e5938b81f29d3c18a41e1

commit a4f8415ad143fec4be3e5938b81f29d3c18a41e1
Author:     Nick Carboni <ncarboni>
AuthorDate: Mon Nov 30 17:59:48 2015 -0500
Commit:     Nick Carboni <ncarboni>
CommitDate: Mon Nov 30 17:59:48 2015 -0500

    Change systemd service KillMode to process
    
    We need the evmserverd process to exit while performing
    an update.
    To do this we run `systemctl stop evmserverd` which brings
    down the main process in the tree managed by systemd.
    When the main process exits, systemd sends a sigterm to all
    other processes in the tree, killing the update process.
    
    This change will tell systemd to not kill other processes in
    the tree after the main one has exited.
    
    See `man systemd.kill` for more info.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1239035

 COPY/usr/lib/systemd/system/evmserverd.service | 1 +
 1 file changed, 1 insertion(+)

Comment 17 Jan Krocil 2015-12-03 10:43:48 UTC
Verified fixed in 5.5.0.12->5.5.0.13 update by patching the "evmserverd.service" file manually before running the update itself from the UI.

Comment 19 errata-xmlrpc 2015-12-08 13:21:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2551